From fdrake@acm.org  Mon Oct  1 15:17:06 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Mon, 1 Oct 2001 10:17:06 -0400
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de>
Message-ID: <15288.31458.735886.325225@grendel.zope.com>

Martin von Loewis writes:
 > [I know I've asked this before, but Fred wanted me to ask it again :-]

  Actually, I think I simply suggested the forum so that others could
comment as well.  ;-)

 > What do you think about an integration of Expat into Python, to be
 > always able to build pyexpat (and with the same version also)?
 > Which version of Expat would you use? Would you put the expat files
 > into a separate directory, or all into modules?

  I have mixed feelings.  There are really two things that we could
do:  We could add Expat to our CVS repository, which means syncing a
bunch of files everytime a new Expat release comes out, or we could
bundle the Expat sources with the Python source distribution when the
distribution is built, but not add them to CVS.  This avoids the extra
files in CVS, but complicates construction of the distribution and
adds a new wrinkle to the configuration management.

 > Here is my proposal: Integrate Expat 2.95.2 for release together with
 > Python 2.2; into an expat subdirectory of Modules (taking only the lib
 > files of expat).
 > 
 > This would affect build procedures on all targets; in particular,
 > pyexpat would not link to a shared expat DLL, but incorporate the
 > object files.

  For the "Parsed XML" Zope product, we included the sources for the
Expat library in our CVS, but added our own configure.in and other
build-control files, which are simpler than those included with Expat
(since it only needs to build the static library).  This seems to work
reasonably well, and does not introduce new wrinkles to the
configuration management.
  So I think we agree on the approach to take.

M.-A. Lemburg writes:
 > Are you sure that we should choose expat as "native" XML parser ?
 > 
 > There are other candidates which would fit this role just
 > as well (in particular, Fredrik's sgmlop looks like a nice
 > extension since it not only works with XML but also many
 > other meta languages).

  See Martin's comments about this.  I think this precludes inclusion
of sgmlop until the problems it has have been addressed in the
implementation.
  I'm not sure what "meta languages" it handles; I thought it only
dealt with XML/XHTML and HTML document markup.

 > If you want a very fast validating XML parser, RXP would also
 > be a good choice -- AFAIK, the RXP folks would allow us to
 > ship RXP under a different license than GPL which is then
 > bound to Python.

  Agreed.  I think it would be really nice to have an interface for
RXP that was easy to build and use.  I haven't looked at PyLTXML in a
long time, so I'm not sure what state it's in.

 > Given the many alternatives, I am not sure whether going with
 > expat is the right path... may be wrong though.

  As Martin said, RXP and Expat together don't really qualify as
"many".  sgmlop just isn't robust enough (yet), and it's not clear
there are other alternatives.
  There is libxml (a.k.a. gnome-xml), which is licensed under the
LGPL; Python bindings for that are described as being in the alpha
stage, but I haven't had time to play with them myself.  


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From mal@lemburg.com  Mon Oct  1 15:50:45 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 01 Oct 2001 16:50:45 +0200
Subject: [Python-Dev] Integrating Expat
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de>
Message-ID: <3BB882C5.D1BDFE79@lemburg.com>

Martin von Loewis wrote:
> 
> > Are you sure that we should choose expat as "native" XML parser ?
> 
> It wouldn't necessarily be the only parser. To process XML, different
> applications have different needs. However, since the expatreader is
> the only SAX reader included in the standard library at the moment,
> guaranteeing presence of pyexpat is oft-requested. Notice that
> pyexpat.c is also in the standard library already.

Just wanted to make sure that we still have the option of including
other parsers as well :-)
 
> > There are other candidates which would fit this role just
> > as well (in particular, Fredrik's sgmlop looks like a nice
> > extension since it not only works with XML but also many
> > other meta languages).
> 
> Not that many candidates would work as well. For example, sgmlop has a
> number of known bugs, and a few unknown ones. Guido once complained
> that it is easy to crash sgmlop with ill-formed input, and rejected
> inclusion of sgmlop when xmlrpclib was integrated. A known problem is
> that entity references are not expanded in attributes.

Well, let's put it this way: if someone finds a need to fix these
bugs, it is more likely to happen in the Python core, e.g. xmlrpclib
has already received a few tweaks (by yourself ;-) after it was
checked into the core.

I think that the sgmlop design is sufficiently simple and easy
to extend to make it a good candidate for inclusion. Sure, we'll
get bug reports, but why not add sgmlop marked as experimental
to the core in order to get it stabilized and bug-fixed ?!

I would very much like a sandbox like part in the Python standard
dist to encourage stabilizing of proposed-to-be-included std
lib extensions, e.g. how about a sandbox package in the std lib ?!
 
> Beyond that, I'm not aware of many more pure-C parsers that could be
> reasonably be integrated into the core. There are many XML parsers,
> but many of the are written in C++ or Java.

Me neither... except RXP which is written in plain C.
 
> > If you want a very fast validating XML parser, RXP would also
> > be a good choice -- AFAIK, the RXP folks would allow us to
> > ship RXP under a different license than GPL which is then
> > bound to Python.
> 
> RXP would indeed be a choice. Of course, integrating it is much
> harder; you'd have to write the C module first, plus documentation,
> plus a SAX driver, plus test cases. I'm not sure how much code you can
> inherit from PyLTXML.

Sure; the question I wanted to raise was: given that we have such
an interface, would RXP also be a candidate for inclusion ?
 
> On performance: Please have a look at
> 
> http://www.xml.com/lpt/a/Benchmark/exec.html
> 
> which suggests that expat still has a speed advantage over rxp
> (assuming that the measurements where done carefully, i.e. disabling
> validation in RXP).

Hmm, I know that at least one company has been having great
success in using RXP with Python; from their experience RXP
is faster on average XML than any of the other available
(validating) parsers. May be due to their application, though, 
so YMMV.
 
> > Given the many alternatives, I am not sure whether going with
> > expat is the right path... may be wrong though.
> 
> It shouldn't be the only path. pyexpat is already integrated into the
> Python library, all I'm suggesting to give the promise that it will be
> available on every 2.2 Python installation.
> 
> Any volunteers working on RXP integration are certainly welcome to do
> so; code contributions to PyXML will be welcome (provided the GPL
> issue gets resolved). Code contributions to the Python core would
> require some review, of course - it took quite some time to get
> pyexpat stable, and I guess any other C-integrated parser won't work
> from scratch, either.

True.

Thanks,
-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From mal@lemburg.com  Mon Oct  1 16:02:22 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 01 Oct 2001 17:02:22 +0200
Subject: [Python-Dev] Integrating Expat
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de>
Message-ID: <3BB8857E.4C370439@lemburg.com>

Martin von Loewis wrote:
> 
> > If you want a very fast validating XML parser, RXP would also
> > be a good choice -- AFAIK, the RXP folks would allow us to
> > ship RXP under a different license than GPL which is then
> > bound to Python.
> 
> RXP would indeed be a choice. Of course, integrating it is much
> harder; you'd have to write the C module first, plus documentation,
> plus a SAX driver, plus test cases. I'm not sure how much code you can
> inherit from PyLTXML.
> 
> On performance: Please have a look at
> 
> http://www.xml.com/lpt/a/Benchmark/exec.html
> 
> which suggests that expat still has a speed advantage over rxp
> (assuming that the measurements where done carefully, i.e. disabling
> validation in RXP).

How would libxml fit into this picture ?

	http://xmlsoft.org/

libxml is written in C as well and under the LGPL.

There's also Apache's Xerces which is written in a portable subset
of C++ (is probably to big though to be intergated into Python):

	http://xml.apache.org/xerces-c/

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From fredrik@pythonware.com  Mon Oct  1 16:23:56 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 1 Oct 2001 17:23:56 +0200
Subject: [Python-Dev] Integrating Expat
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de><3BB74B99.3B230398@lemburg.com><200109301708.TAA22863@paros.informatik.hu-berlin.de> <15288.31458.735886.325225@grendel.zope.com>
Message-ID: <019a01c14a8d$178d39a0$b3fa42d5@hagrid>

> I have mixed feelings.  There are really two things that we could
> do:  We could add Expat to our CVS repository, which means syncing a
> bunch of files everytime a new Expat release comes out

I thought MvL had already volunteered to do this?

>  > There are other candidates which would fit this role just
>  > as well (in particular, Fredrik's sgmlop looks like a nice
>  > extension since it not only works with XML but also many
>  > other meta languages).
> 
>   See Martin's comments about this.  I think this precludes inclusion
> of sgmlop until the problems it has have been addressed in the
> implementation.

cannot fix bugs if nobody bothers to report them ;-)

(the crash issue appears to be a rumour; there was a bug when
running in SGML mode, but that was fixed long ago.  people using
the current release in real-life applications haven't reported any
stability problems...)

on the other hand, sgmlop itself will never be anything but a "fast
but sloppy" XML tokenizer.  if you risk running into xml compliance
nazis <0.1 wink>, you shouldn't use it.

</F>



From fdrake@acm.org  Mon Oct  1 16:23:59 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Mon, 1 Oct 2001 11:23:59 -0400
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <019a01c14a8d$178d39a0$b3fa42d5@hagrid>
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de>
 <15288.31458.735886.325225@grendel.zope.com>
 <019a01c14a8d$178d39a0$b3fa42d5@hagrid>
Message-ID: <15288.35471.300115.993620@grendel.zope.com>

Fredrik Lundh writes:
 > I thought MvL had already volunteered to do this?

  I didn't state this was a huge issue or that it didn't have a nice
solution.  ;-)  It also isn't something that happens all that often,
given that I don't have a lot of time to make Expat releases.

 > cannot fix bugs if nobody bothers to report them ;-)
 > 
 > (the crash issue appears to be a rumour; there was a bug when
 > running in SGML mode, but that was fixed long ago.  people using
 > the current release in real-life applications haven't reported any
 > stability problems...)

  Glad to hear this!  Perhaps someone (not implying you) should
start writing a substantial test suite for it to ferret out any
remaining bugs?  I don't see a test_sgmlop.py in the PyXML package; if
you already have something perhaps you could contribute it?  It might
help you unload maintenance if anyone does manage to find a bug.

 > on the other hand, sgmlop itself will never be anything but a "fast
 > but sloppy" XML tokenizer.  if you risk running into xml compliance
 > nazis <0.1 wink>, you shouldn't use it.

  "Nazi" would not have been my word for it, but ... Wham!  ;-)


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From jepler@inetnebr.com  Mon Oct  1 15:00:29 2001
From: jepler@inetnebr.com (Jeff Epler)
Date: Mon, 1 Oct 2001 09:00:29 -0500
Subject: [Python-Dev] Python 2.2a* getattr suggestion and question
In-Reply-To: <200109301837.OAA32659@cj20424-a.reston1.va.home.com>
References: <Pine.LNX.4.30.0109301748460.8078-100000@rnd.onego.ru> <200109301837.OAA32659@cj20424-a.reston1.va.home.com>
Message-ID: <20011001090026.B1755@inetnebr.com>

On Sun, Sep 30, 2001 at 02:37:00PM -0400, Guido van Rossum wrote:
> > Sure. Still, I think interpreter diagnostics should be pointing to the
> > exact place of trouble. At least, __getattribute__ must appear somewhere
> > in the traceback to give a hint where from __repr__ was attempted to be
> > called.
> 
> When you write a faulty __getattribute__ that returns None instead of
> raising AttributeError, it's not realistic to expect __getattribute__
> to be in the stack trace.

PyChecker might want to give an error if flow in __getattr[ibute]__
doesn't either pass through a return or a 'raise AttributeError'.
Even if the intent is to have attributes default to None, an explicit
'return None' could be added.

Jeff


From skip@pobox.com (Skip Montanaro)  Mon Oct  1 18:08:00 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 12:08:00 -0500
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <3BB882C5.D1BDFE79@lemburg.com>
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de>
 <3BB882C5.D1BDFE79@lemburg.com>
Message-ID: <15288.41712.751198.83416@beluga.mojam.com>

    mal> I think that the sgmlop design is sufficiently simple and easy to
    mal> extend to make it a good candidate for inclusion. Sure, we'll get
    mal> bug reports, but why not add sgmlop marked as experimental to the
    mal> core in order to get it stabilized and bug-fixed ?!

I would be happy to sgmlop added to the core.  The xmlrpclib encoding and
decoding do need some sort of C-based acceleration to be usable:

    % python testxmlrpc.py
    testing with xmlrpclib 0.9.8
    using FastParser
    415 dumps per second
    106 loads per second
    disabling fast parsers in xmlrpclib
    using SlowParser
    412 dumps per second
    16.1 loads per second

FWIW, the xmlrpclib delivered with Python is substantially slower dumping
data than the 0.9.x versions that have been around awhile, though its
decoding performance degrades less without sgmlop.  Compare the above with
this:

    % PYTHONPATH=~/misc/python/python2 python testxmlrpc.py
    testing with xmlrpclib 1.0b3
    using SgmlopParser
    229 dumps per second
    94.3 loads per second
    disabling fast parsers in xmlrpclib
    using ExpatParser
    231 dumps per second
    76.8 loads per second

I haven't had or taken the time to investigate the difference yet.

Skip


From loewis@informatik.hu-berlin.de  Mon Oct  1 18:09:25 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Mon, 1 Oct 2001 19:09:25 +0200 (MEST)
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <15288.31458.735886.325225@grendel.zope.com> (fdrake@acm.org)
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de> <15288.31458.735886.325225@grendel.zope.com>
Message-ID: <200110011709.TAA13228@paros.informatik.hu-berlin.de>

>   I have mixed feelings.  There are really two things that we could
> do:  We could add Expat to our CVS repository, which means syncing a
> bunch of files everytime a new Expat release comes out, or we could
> bundle the Expat sources with the Python source distribution when the
> distribution is built, but not add them to CVS.  This avoids the extra
> files in CVS, but complicates construction of the distribution and
> adds a new wrinkle to the configuration management.

I'm in favour of putting it into the CVS. This will allow CVS users to
build it, and it will allow proper integration and testing of required
changes into setup.py (and Setup.dist; the current text has caused a
lot of confusion in the past).

Regards,
Martin


From mal@lemburg.com  Mon Oct  1 18:50:31 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 01 Oct 2001 19:50:31 +0200
Subject: [Python-Dev] Integrating Expat
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de>
 <3BB882C5.D1BDFE79@lemburg.com> <15288.41712.751198.83416@beluga.mojam.com>
Message-ID: <3BB8ACE7.B61A4881@lemburg.com>

Skip Montanaro wrote:
> 
>     mal> I think that the sgmlop design is sufficiently simple and easy to
>     mal> extend to make it a good candidate for inclusion. Sure, we'll get
>     mal> bug reports, but why not add sgmlop marked as experimental to the
>     mal> core in order to get it stabilized and bug-fixed ?!
> 
> I would be happy to sgmlop added to the core.  The xmlrpclib encoding and
> decoding do need some sort of C-based acceleration to be usable:
> 
>     % python testxmlrpc.py
>     testing with xmlrpclib 0.9.8
>     using FastParser
>     415 dumps per second
>     106 loads per second
>     disabling fast parsers in xmlrpclib
>     using SlowParser
>     412 dumps per second
>     16.1 loads per second
> 
> FWIW, the xmlrpclib delivered with Python is substantially slower dumping
> data than the 0.9.x versions that have been around awhile, though its
> decoding performance degrades less without sgmlop.  Compare the above with
> this:
> 
>     % PYTHONPATH=~/misc/python/python2 python testxmlrpc.py
>     testing with xmlrpclib 1.0b3
>     using SgmlopParser
>     229 dumps per second
>     94.3 loads per second
>     disabling fast parsers in xmlrpclib
>     using ExpatParser
>     231 dumps per second
>     76.8 loads per second
> 
> I haven't had or taken the time to investigate the difference yet.

Hmm, you cannot really compare these numbers though, since the two
runs use two different sets of parsers. Have you checked using
SgmlopParser with the 0.9.8 version of xmlrpclib ?

There's also a project on SF called py-xmlrpc which uses a C
implementation as basis and is said to be much faster than 
xmlrpclib (at least that's what they quote on their web-page).

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From skip@pobox.com (Skip Montanaro)  Mon Oct  1 19:05:35 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 13:05:35 -0500
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <3BB8ACE7.B61A4881@lemburg.com>
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de>
 <3BB882C5.D1BDFE79@lemburg.com>
 <15288.41712.751198.83416@beluga.mojam.com>
 <3BB8ACE7.B61A4881@lemburg.com>
Message-ID: <15288.45167.465025.917304@beluga.mojam.com>

    mal> Hmm, you cannot really compare these numbers though, since the two
    mal> runs use two different sets of parsers. Have you checked using
    mal> SgmlopParser with the 0.9.8 version of xmlrpclib ?

They are the same parser.  I forgot to mention that.  What is called
"FastParser" in 0.9.8 is called SgmlopParser in the CVS version.  That has a
different thing called "FastParser".  I believe it is the thing you can get
by contacting Pythonware.

    mal> There's also a project on SF called py-xmlrpc which uses a C
    mal> implementation as basis and is said to be much faster than
    mal> xmlrpclib (at least that's what they quote on their web-page).

Yes, it is.  Amazingly enough, the guy who wrote it (Shilad Sen at
Sourcelight Technologies) works in the same building I do (and it's a pretty
small building).  We had lunch last week and talked a bit about it.  It
doesn't yet do Unicode.  I sent Shilad my little test script.  He modified
it to use his parser.  His results suggest that py-xmlrpc is about as fast
as cPickle. 

Skip


From Samuele Pedroni <pedroni@inf.ethz.ch>  Mon Oct  1 10:20:10 2001
From: Samuele Pedroni <pedroni@inf.ethz.ch> (Samuele Pedroni)
Date: Mon, 1 Oct 2001 11:20:10 +0200 (MET DST)
Subject: [Python-Dev] Re: descr PLAN.txt
Message-ID: <200110010920.LAA07635@core.inf.ethz.ch>

> > I think and it seems it contains some useful information
> > for when we try to port the descr changes to jython,
> > or will all this stuff be  completely documented somewhere else?
> 
> Hopefully the PEPs will be much more complete records.  PLAN.txt was
> just my own to-do list, indicating how far I was along the realization
> of the PEPs.  I realize that the PEPs are currently behind, but it is
> in the PLAN.txt to remedy this. :-)

I see.

> I'm glad you are planning to copy this effort in Jython.  

Yes I'm planning to do try that.
I don't think that without keeping up with Python Jython
can survive. Not that we are very fast at keeping up OTOH.

(yes for very practical reasons I'm among
those who would prefer a slower development for Python :))

> (It should
> actually be simpler in Jython as it doesn't have so much of a
> dichotomy between types and classes as C-Python does.)

I take this as an  encouragement but actually the good point
is that Java is bit more productive than multi-platform C, that's
all.

You should consider that we have to integrate the changes
with the whole Java interoperability architecture and
that we should deal with the limitations of java reflection
and probably invent some new kind of inheriting from java,
different from what is already there,
that means new code that should produce dynamically
some java bytecode ...

>  If there's
> anything I can do to help (short of coding in Java :-) please let me
> know.  In particular, if there's anything unclear in the PEPs or
> PLAN.txt or in python.org/2.2/descrintro.html or anywhere else
> (including the C code), I'd be happy to clarify it at your (or anybody
> else's) request.  

Thanks, I imagine we will have some questions :)

> (Is Finn Bock still around in Jython?)

AFAIK yes. For 2.1 (yet not out) he has done:
- rich comparison
- new coercion model
- weak ref support
- importing from jars on sys.path
...

I have done:
static nested scopes
fixed some old nasty bugs
improved the compatibility of importing wrt CPython

And in any case I hope so:

given that for 2.2 we have to do:
iterators
generators
descr

both in the runtime and jythonc ...

and our user base so far has been quite passive.

Sorry for the rants.

regards.



From skip@pobox.com (Skip Montanaro)  Mon Oct  1 18:55:37 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 12:55:37 -0500
Subject: [Python-Dev] xmlrpclib speed boost
Message-ID: <15288.44569.922160.604530@beluga.mojam.com>

Regarding slowdown in xmlrpclib dump performance, I wrote:

    Skip> I haven't had or taken the time to investigate the difference yet.

Well, it turned out to be pretty trivial.  Somewhere along the way, the
import of the escape function from the cgi module got moved out of the top
level into the various functions that call it.  Moving it back out to top
level fixed that.  I just checked in this change and a fairly trivial test
case for xmlrpclib.

This change may be deemed not to be the correct fix as far backwards
compatibility is concerned (it uses the "from m import x as y" feature which
was new with 2.0 I think).  If someone alters this fix, please don't put the
import back into the functions that call cgi.escape.

Skip



From thomas@xs4all.net  Mon Oct  1 19:45:49 2001
From: thomas@xs4all.net (Thomas Wouters)
Date: Mon, 1 Oct 2001 20:45:49 +0200
Subject: [Python-Dev] xmlrpclib speed boost
In-Reply-To: <15288.44569.922160.604530@beluga.mojam.com>
References: <15288.44569.922160.604530@beluga.mojam.com>
Message-ID: <20011001204548.B846@xs4all.nl>

On Mon, Oct 01, 2001 at 12:55:37PM -0500, Skip Montanaro wrote:

> This change may be deemed not to be the correct fix as far backwards
> compatibility is concerned (it uses the "from m import x as y" feature which
> was new with 2.0 I think).  If someone alters this fix, please don't put the
> import back into the functions that call cgi.escape.

Why not ? Moving the import to the top level just causes the slowdown to
occur at a different moment. If this is really the problem, the slowdown
should occur only the first time you use a particular function (unless you
explicitly un-import cgi somehow ?)

Importing cgi only in the functions that actually use it, in order to avoid
the slowdown unless it's really necessary, sure seems like a sensible
solution to me :-)

-- 
Thomas Wouters <thomas@xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!


From mal@lemburg.com  Mon Oct  1 19:44:37 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 01 Oct 2001 20:44:37 +0200
Subject: [Python-Dev] Integrating Expat
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
 <3BB74B99.3B230398@lemburg.com>
 <200109301708.TAA22863@paros.informatik.hu-berlin.de>
 <3BB882C5.D1BDFE79@lemburg.com>
 <15288.41712.751198.83416@beluga.mojam.com>
 <3BB8ACE7.B61A4881@lemburg.com> <15288.45167.465025.917304@beluga.mojam.com>
Message-ID: <3BB8B995.3EA05D66@lemburg.com>

Skip Montanaro wrote:
> 
>     mal> Hmm, you cannot really compare these numbers though, since the two
>     mal> runs use two different sets of parsers. Have you checked using
>     mal> SgmlopParser with the 0.9.8 version of xmlrpclib ?
> 
> They are the same parser.  I forgot to mention that.  What is called
> "FastParser" in 0.9.8 is called SgmlopParser in the CVS version.  That has a
> different thing called "FastParser".  I believe it is the thing you can get
> by contacting Pythonware.

I see. 

Rereading your numbers suggests that only dumps got slower. Now that
you've fixed this in CVS the reason is obvious... from xyz import abc
is slow. OTOH, Fredrik mentions that he put in this change in order
to decrease startup time for the lib. I guess you can't win 'em all :-)
 
>     mal> There's also a project on SF called py-xmlrpc which uses a C
>     mal> implementation as basis and is said to be much faster than
>     mal> xmlrpclib (at least that's what they quote on their web-page).
> 
> Yes, it is.  Amazingly enough, the guy who wrote it (Shilad Sen at
> Sourcelight Technologies) works in the same building I do (and it's a pretty
> small building).  We had lunch last week and talked a bit about it.  It
> doesn't yet do Unicode.  I sent Shilad my little test script.  He modified
> it to use his parser.  His results suggest that py-xmlrpc is about as fast
> as cPickle.

Cool !

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From tim.one@home.com  Mon Oct  1 19:47:04 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 1 Oct 2001 14:47:04 -0400
Subject: [Python-Dev] xmlrpclib speed boost
In-Reply-To: <15288.44569.922160.604530@beluga.mojam.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIELHLOAA.tim.one@home.com>

[Skip Montanaro]
> ...
> Somewhere along the way, the import of the escape function from the cgi
> module got moved out of the top level into the various functions that
> call it.  Moving it back out to top level fixed that.  I just checked in
> this change and a fairly trivial test case for xmlrpclib.
>
> This change may be deemed not to be the correct fix as far backwards
> compatibility is concerned (it uses the "from m import x as y"
> feature which was new with 2.0 I think).

I pay no attention to "backwards compatibility" issues in the libraries we
ship, and have Guido's blessing for that ornery attitude:  the libs we ship
should do the best possible job in the context of the release they're
shipped in.  It's explicitly not a PythonLabs goal that people be able to
pick modules out of a release and use them in an earlier release (although
it may be a goal of some developers for some modules -- but then the burden
is on them to ensure it works for the module and release combinations they
care about).



From skip@pobox.com (Skip Montanaro)  Mon Oct  1 19:56:58 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 13:56:58 -0500
Subject: [Python-Dev] xmlrpclib speed boost
In-Reply-To: <20011001204548.B846@xs4all.nl>
References: <15288.44569.922160.604530@beluga.mojam.com>
 <20011001204548.B846@xs4all.nl>
Message-ID: <15288.48250.206337.702825@beluga.mojam.com>

    >> This change may be deemed not to be the correct fix as far backwards
    >> compatibility is concerned (it uses the "from m import x as y"
    >> feature which was new with 2.0 I think).  If someone alters this fix,
    >> please don't put the import back into the functions that call
    >> cgi.escape.

    Thomas> Why not ? Moving the import to the top level just causes the
    Thomas> slowdown to occur at a different moment. If this is really the
    Thomas> problem, the slowdown should occur only the first time you use a
    Thomas> particular function (unless you explicitly un-import cgi somehow
    Thomas> ?)

Trust me, this really *is* the problem.  Instead of getting imported once,
it gets imported once for every string and once for every key in every
dictionary.  I timed it before and after.  Compare Marshaller.dump_string
before

    def dump_string(self, value):
        from cgi import escape
        self.write("<value><string>%s</string></value>\n" % escape(value))

and after

    def dump_string(self, value):
        self.write("<value><string>%s</string></value>\n" % _escape(value))

I don't care how little work it is to import a module a second time, it is
probably going to be on the same order of magnitude as that write call.  In
Marshaller.dump_struct the import was *inside* the for loop.  I originally
pulled it up out of the for loop but left it inside the method.  That got me
about a 25% boost in my simple test.  I was ready to check in that one
change and thought, "aw hell, might as well see what happens if I pull the
import all the way out to the top level".  Dump performance went all the way
back up to where 0.9.8 is.

    Thomas> Importing cgi only in the functions that actually use it, in
    Thomas> order to avoid the slowdown unless it's really necessary, sure
    Thomas> seems like a sensible solution to me :-)

Except dumping structs (dictionaries) and strings happens a lot.  Even if
all we ever did was dump ints, floats and lists, the total cost of my change
would be one import.

Skip


From guido@python.org  Mon Oct  1 19:54:34 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 01 Oct 2001 14:54:34 -0400
Subject: [Python-Dev] xmlrpclib speed boost
In-Reply-To: Your message of "Mon, 01 Oct 2001 20:45:49 +0200."
 <20011001204548.B846@xs4all.nl>
References: <15288.44569.922160.604530@beluga.mojam.com>
 <20011001204548.B846@xs4all.nl>
Message-ID: <200110011854.OAA07942@cj20424-a.reston1.va.home.com>

> > This change may be deemed not to be the correct fix as far
> > backwards compatibility is concerned (it uses the "from m import x
> > as y" feature which was new with 2.0 I think).  If someone alters
> > this fix, please don't put the import back into the functions that
> > call cgi.escape.
> 
> Why not ? Moving the import to the top level just causes the slowdown to
> occur at a different moment. If this is really the problem, the slowdown
> should occur only the first time you use a particular function (unless you
> explicitly un-import cgi somehow ?)
> 
> Importing cgi only in the functions that actually use it, in order to avoid
> the slowdown unless it's really necessary, sure seems like a sensible
> solution to me :-)

There's a cost to an import inside a function even if the module is
already imported.  If the function is very small and heavily used (as
it apparently was in this case) the function can slow down
considerably.

It's kind of sad that the escape() function, which is only a few
lines, comes from the cgi module which is thousands of lines that are
not normally used except by actual CGI scripts.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Mon Oct  1 20:20:37 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Mon, 1 Oct 2001 15:20:37 -0400
Subject: [Python-Dev] xmlrpclib speed boost
In-Reply-To: <200110011854.OAA07942@cj20424-a.reston1.va.home.com>
References: <15288.44569.922160.604530@beluga.mojam.com>
 <20011001204548.B846@xs4all.nl>
 <200110011854.OAA07942@cj20424-a.reston1.va.home.com>
Message-ID: <15288.49669.875132.906282@grendel.zope.com>

Guido van Rossum writes:
 > It's kind of sad that the escape() function, which is only a few
 > lines, comes from the cgi module which is thousands of lines that are
 > not normally used except by actual CGI scripts.

  Yep.
  In this case, it probably makes more sense to call the escape()
function from the xml.sax.saxutils module, though.  The cgi version is
for HTML and the xml.sax.saxutils version is for XML (big surprise).
With only one argument, though, there's no difference.
  If we really want to deal with the startup time problem (regardless
of which module we get the escape function from), we could do this:


def _escape(s):
    global _escape
    import xml.sax.saxutils
    _escape = xml.sax.saxutils.escape
    return _escape(s)


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From fredrik@pythonware.com  Mon Oct  1 21:01:31 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 1 Oct 2001 22:01:31 +0200
Subject: [Python-Dev] xmlrpclib speed boost
References: <15288.44569.922160.604530@beluga.mojam.com>
Message-ID: <02cc01c14ab3$e0124660$b3fa42d5@hagrid>

skip wrote:

> This change may be deemed not to be the correct fix as far backwards
> compatibility is concerned (it uses the "from m import x as y" feature which
> was new with 2.0 I think).  If someone alters this fix, please don't put the
> import back into the functions that call cgi.escape.

you forgot to remove the comment at the top of the file that says
it works with 1.5.2 and later.

I've restored backwards compatibility, and tweaked the marshaller
a little bit more.  the new code is about 80% faster than 1.0b3 on
my machine.  ymmv, as usual.

</F>



From skip@pobox.com (Skip Montanaro)  Mon Oct  1 21:04:11 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 15:04:11 -0500
Subject: [Python-Dev] xmlrpclib speed boost
In-Reply-To: <02cc01c14ab3$e0124660$b3fa42d5@hagrid>
References: <15288.44569.922160.604530@beluga.mojam.com>
 <02cc01c14ab3$e0124660$b3fa42d5@hagrid>
Message-ID: <15288.52283.381456.704181@beluga.mojam.com>

    Fredrik> skip wrote:
    >> This change may be deemed not to be the correct fix as far backwards
    >> compatibility is concerned (it uses the "from m import x as y"
    >> feature which was new with 2.0 I think).  If someone alters this fix,
    >> please don't put the import back into the functions that call
    >> cgi.escape.

    Fredrik> you forgot to remove the comment at the top of the file that
    Fredrik> says it works with 1.5.2 and later.

Thanks, this is what I was referring to (of course, I didn't see the
comment). 

    Fredrik> I've restored backwards compatibility, and tweaked the
    Fredrik> marshaller a little bit more.  the new code is about 80% faster
    Fredrik> than 1.0b3 on my machine.  ymmv, as usual.

Cool.

Skip



From gward@python.net  Tue Oct  2 01:54:17 2001
From: gward@python.net (Greg Ward)
Date: Mon, 1 Oct 2001 20:54:17 -0400
Subject: [Python-Dev] Demo/dns
Message-ID: <20011001205417.A6563@cthulhu.mems-exchange.org>

Demo/dns is pretty old, and has been superseded by Anthony Baxter's
pydns project on SourceForge.  At the very least, Demo/dns/README should
point to
  http://sourceforge.net/projects/pydns
instead of Anthony's old page for this project (dead URL).

More radically, we could delete all the code in Demo/dns, and just leave
behind a little README pointing at Anthony's project.  I've *just*
checked out his code, haven't dug in seriously yet, so I don't know if
this is a good idea yet.

Opinions?  If I hear nothing, I'll just fix the URL in the README.

        Greg
-- 
Greg Ward - Unix nerd                                   gward@python.net
http://starship.python.net/~gward/
A day for firm decisions!!!!!  Or is it?


From skip@pobox.com (Skip Montanaro)  Tue Oct  2 02:45:10 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 20:45:10 -0500
Subject: [Python-Dev] Performance of various marshallers
Message-ID: <15289.7206.266140.820041@beluga.mojam.com>


I just tested each of the marshallers readily available to me.  I dumped and
loaded this object:

    ['MusicEntry',
     'email': 'foo@bar.baz.spam',
     'time': '7:30pm',
     'tickets': '',
     'program': '',
     'state': 'MA',
     'start': '2002-01-26',
     'venueurl': '',
     'country': '',
     'performers': ['An Evening with Karen Savoca'],
     'addressid': 7283,
     'name': '',
     'zip': '',
     'city': 'Sudbury',
     'info': 'Reservations required. Please call (978)443-3253 or e-mail Laurie at lalcorn@ultranet.com.',
     'merchandise': [],
     'event': '',
     'keywords': ['.zyx.41'],
     'submit_time': '2001-08-28',
     'key': 325629,
     'active': 1,
     'end': '2002-01-26',
     'address1': '',
     'venue': 'Fox Run House Concerts',
     'price': '$17',
     'address3': '',
     'address2': '',
     'update_time': '2001-09-22:19:28:44'}]

I don't claim this is typical data, but it is typical of the type of data I
push through XML-RPC, so it's important to me.  You can see why moving
imports out of dump_string was so worthwhile.  I would be happy to change
the object being marshalled to better reflect what people think is
"typical".

All numbers in the following table are in encodings or decodings per second.
All times were measured using time.clock.  The number of times the
encoding/decoding operation was performed was varied to give a reasonable
total test time (approximately 5 seconds).  Each test was run 3 times.  The
largest number is recorded below, rounded to three significant digits.

                        encode          decode
                        ------          ------
    marshal             25900           7830
    cPickle              1230            149
    xmlrpclib 0.9.8
        w/ sgmlop         416            107
        w/o sgmlop        415             16.3
    xmlrpclib 1.0b4
        w/ sgmlop         365             92.0
        w/o sgmlop        363             74.9
    py-xmlrpc            2780           2260

Skip


From fdrake@acm.org  Tue Oct  2 03:42:38 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Mon, 1 Oct 2001 22:42:38 -0400
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <15289.7206.266140.820041@beluga.mojam.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
Message-ID: <15289.10654.162126.780312@grendel.zope.com>

Skip Montanaro writes:
 > total test time (approximately 5 seconds).  Each test was run 3 times.  The
 > largest number is recorded below, rounded to three significant digits.
 > 
 >                         encode          decode
 >                         ------          ------
 >     marshal             25900           7830
 >     cPickle              1230            149

  Were the cPickle tests run in binary or original flavor?

 >     xmlrpclib 0.9.8
 >         w/ sgmlop         416            107
 >         w/o sgmlop        415             16.3  <--+
 >     xmlrpclib 1.0b4                                |
 >         w/ sgmlop         365             92.0     |
 >         w/o sgmlop        363             74.9  <--+
 >     py-xmlrpc            2780           2260       |
                                                      |
  +---------------------------------------------------+
  |
  +----> I presume that Expat was available for the second run and not
         for the first?  These should probably be broken into three
         categories: sgmlop, expat, and xmllib.

         I also presume that py-xmlrpc never calls from C->Python
         during the parse phase, but I've not yet had a chance to look
         at this code.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From skip@pobox.com (Skip Montanaro)  Tue Oct  2 03:52:58 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 21:52:58 -0500
Subject: [Python-Dev] is htmllib broken in 2.2a4?
Message-ID: <15289.11274.567758.60276@beluga.mojam.com>

Responding to a question in python-help about extracting links from web
pages, I wrote a simple href printer:

    import htmllib, formatter

    class MyParser(htmllib.HTMLParser):
        def anchor_bgn(self, href, name, type):
            print href

    fmt = formatter.NullFormatter()
    parser = MyParser(fmt, verbose=1)
    parser.feed(open("tour01.html").read())
    parser.close()

When run using 2.2a4, it never prints anything.  It outputs a list of hrefs
when run with 2.1 or 1.6.  Either there's a bug somewhere (in my code
possibly, though it's pretty simple) or some semantics changed that I
missed.

I thought maybe the method resolution order change affected things, but
htmllib.HTMLParser only uses single inheritance.  When displaying help about
htmllib.HTMLParser, pydoc.help does emit the method resolution order, which
it doesn't generally seem to do:

    class HTMLParser(sgmllib.SGMLParser)
     |  Method resolution order:
     |      HTMLParser
     |      sgmllib.SGMLParser
     |      markupbase.ParserBase
     ...

Skip



From fdrake@acm.org  Tue Oct  2 03:55:54 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Mon, 1 Oct 2001 22:55:54 -0400
Subject: [Python-Dev] is htmllib broken in 2.2a4?
In-Reply-To: <15289.11274.567758.60276@beluga.mojam.com>
References: <15289.11274.567758.60276@beluga.mojam.com>
Message-ID: <15289.11450.952344.374616@grendel.zope.com>

Skip Montanaro writes:
 > When run using 2.2a4, it never prints anything.  It outputs a list of hrefs
 > when run with 2.1 or 1.6.  Either there's a bug somewhere (in my code
 > possibly, though it's pretty simple) or some semantics changed that I

  Sounds like a bug to me.  Please file a bug report on SF including
your code, and assign to me.
  Thanks!


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From tim.one@home.com  Tue Oct  2 04:05:11 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 1 Oct 2001 23:05:11 -0400
Subject: [Python-Dev] is htmllib broken in 2.2a4?
In-Reply-To: <15289.11274.567758.60276@beluga.mojam.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMLLOAA.tim.one@home.com>

[Skip Montanaro]
> Responding to a question in python-help about extracting links from web
> pages, I wrote a simple href printer:
>
>     import htmllib, formatter
>
>     class MyParser(htmllib.HTMLParser):
>         def anchor_bgn(self, href, name, type):
>             print href
>
>     fmt = formatter.NullFormatter()
>     parser = MyParser(fmt, verbose=1)
>     parser.feed(open("tour01.html").read())
>     parser.close()
>
> When run using 2.2a4, it never prints anything.  It outputs a
> list of hrefs when run with 2.1 or 1.6.  Either there's a bug somewhere
> (in my code possibly, though it's pretty simple) or some semantics
> changed that I missed.

Sorry, I don't know anything about that, and I don't know that code.  Open a
bug report!  Sure doesn't sound right to me.

> I thought maybe the method resolution order change affected things,

The MRO hasn't changed for classic classes.  Only for new classes (so if you
don't derive from object, nothing about MRO changed).

> but htmllib.HTMLParser only uses single inheritance.  When displaying
> help about htmllib.HTMLParser, pydoc.help does emit the method
> resolution order, which it doesn't generally seem to do:

I recently changed pydoc to display MRO if and only if there are more than
two classes an attribute *could* come from (if there are no more than two
classe involved, there's no possibility of confusion; but if there are more
than two, confusion is possible).

>     class HTMLParser(sgmllib.SGMLParser)
>      |  Method resolution order:
>      |      HTMLParser
>      |      sgmllib.SGMLParser
>      |      markupbase.ParserBase
>      ...

It listed MRO simply because more than 2 classes are possible attribute
sources.



From skip@pobox.com (Skip Montanaro)  Tue Oct  2 04:06:04 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 22:06:04 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <15289.10654.162126.780312@grendel.zope.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
Message-ID: <15289.12060.493061.267430@beluga.mojam.com>

 > total test time (approximately 5 seconds).  Each test was run 3 times.  The
 > largest number is recorded below, rounded to three significant digits.
 > 
 >                         encode          decode
 >                         ------          ------
 >     marshal             25900           7830
 >     cPickle              1230            149

  Were the cPickle tests run in binary or original flavor?

I wasn't aware of a "binary flavor".  It's not mentioned in the online docs.
I just called cPickle.dumps or cPickle.loads as appropriate.  It looks like
I should call them with a second binary flag.  

 >     xmlrpclib 0.9.8
 >         w/ sgmlop         416            107
 >         w/o sgmlop        415             16.3  <--+
 >     xmlrpclib 1.0b4                                |
 >         w/ sgmlop         365             92.0     |
 >         w/o sgmlop        363             74.9  <--+
 >     py-xmlrpc            2780           2260       |
                                                      |
  +---------------------------------------------------+
  |
  +----> I presume that Expat was available for the second run and not
         for the first?  These should probably be broken into three
         categories: sgmlop, expat, and xmllib.

In 0.9.8 there are two parsers, fast (with sgmlop) and slow (without).  I
believe the ExpatParser was used in the second version.  It doesn't really
matter to me because they are all perform so abysmally.

         I also presume that py-xmlrpc never calls from C->Python
         during the parse phase, but I've not yet had a chance to look
         at this code.

I don't know.  I've not looked at the code, only the output.  I have cc'd
Shilad Sen on this thread.  He should be able to tell us how py-xmlrpc gets
such good performance.

Skip


From skip@pobox.com (Skip Montanaro)  Tue Oct  2 04:21:16 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 1 Oct 2001 22:21:16 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <15289.10654.162126.780312@grendel.zope.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
Message-ID: <15289.12972.375469.537213@beluga.mojam.com>

 >                         encode          decode
 >                         ------          ------
 >     cPickle              1230            149

  Were the cPickle tests run in binary or original flavor?

Setting the binary flag when dumping using cPickle boosts the encode rate to
4450 and the decode rate to 3940.

Skip


From andy@reportlab.com  Tue Oct  2 06:53:40 2001
From: andy@reportlab.com (Andy Robinson)
Date: Tue, 2 Oct 2001 06:53:40 +0100
Subject: [Python-Dev] RE: Python-Dev digest, Vol 1 #1637 - 11 msgs
In-Reply-To: <E15o5WL-0005Y6-00@mail.python.org>
Message-ID: <LKENLBBMDHMKBECHIAIAEEGJCAAA.andy@reportlab.com>

> If you want a very fast validating XML parser, RXP would also
> be a good choice -- AFAIK, the RXP folks would allow us to
> ship RXP under a different license than GPL which is then
> bound to Python.
> 
> Given the many alternatives, I am not sure whether going with
> expat is the right path... may be wrong though.
> 

Lucky I tuned in.  Reportlab has had great success with RXP.
We have a python wrapper, pyRXP, with binaries available for
several platforms.

It is GPLed at present.  They wish to keep GPL just in
case someone big comes along and wants their code for ten million
set-top boxes or something.  However, I persuaded them to grant
a license to let it be used through the Python binding under 
Python-like terms, as long as we invent the words and save 
them having to waste time on lawyers.  They would even be
happy for it to go into the Python distribution.  And 
we're happy to maintain the wrapper and binaries for 
several platforms, which we have to do for our customers 
anyway.

If one of the core Python team, who I know have long and 
painful experience of this stuff, would like to drop me a line, 
we can probably sort this out in a night.  

The other thing we found very useful was our representation.
We make reports, and ML is a common data source; so our goal
is typically to slurp XML into memory as fast as possible,
with validation.  We eventually hit on a 'tuple tree':
each tag is represented as 
  (tagname, attrs, list-of-children, spare)
We get there about 6x faster than the fastest alternative
parser we know, because all the work is done in C; with
typical use of other parsers you call back into Python
on every tag.  The tree structure is a fraction of the
size in memory of what gets created by models using
objects for every node.  It would be very easy to add
this as an alternative interface to expat as well.  So
then Python users could have a choice of tree or events,
and validating or non-validating, all done in C and
in the standard distribution.

Andy Robinson
CEO/Chief Architect, Reportlab Inc.



From effbot@telia.com  Tue Oct  2 08:43:04 2001
From: effbot@telia.com (Fredrik Lundh)
Date: Tue, 2 Oct 2001 09:43:04 +0200
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com>
Message-ID: <03aa01c14b15$e329ac00$b3fa42d5@hagrid>

fred wrote:
>  >     xmlrpclib 0.9.8
>  >         w/ sgmlop         416            107
>  >         w/o sgmlop        415             16.3  <--+
>  >     xmlrpclib 1.0b4                                |
>  >         w/ sgmlop         365             92.0     |
>  >         w/o sgmlop        363             74.9  <--+
>  >     py-xmlrpc            2780           2260       |
>                                                       |
>   +---------------------------------------------------+
>   |
>   +----> I presume that Expat was available for the second run and not
>          for the first?  These should probably be broken into three
>          categories: sgmlop, expat, and xmllib.

footnote: 0.9.8 didn't support pyexpat.

>          I also presume that py-xmlrpc never calls from C->Python
>          during the parse phase, but I've not yet had a chance to look
>          at this code.

does py-xmlrpc use a real XML parser?

</F>



From skip@pobox.com (Skip Montanaro)  Tue Oct  2 13:29:45 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 07:29:45 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
Message-ID: <15289.45881.920165.427067@beluga.mojam.com>

    >> I also presume that py-xmlrpc never calls from C->Python during the
    >> parse phase, but I've not yet had a chance to look at this code.

    Fredrik> does py-xmlrpc use a real XML parser?

I suspect not.  It's special purpose is to parse or generate XML-RPC, so you
know ahead of time that the end result is the only thing you need.

Skip


From akuchlin@mems-exchange.org  Tue Oct  2 18:57:24 2001
From: akuchlin@mems-exchange.org (Andrew Kuchling)
Date: Tue, 02 Oct 2001 13:57:24 -0400
Subject: [Python-Dev] Bookstore proceeds
Message-ID: <E15oTnI-0005pO-00@ute.cnri.reston.va.us>

I've received another cheque from Amazon with the last quarter's
proceeds for the Python bookstore.  The amount is just a bit less than
$500; I'll contribute the few dollars to bring it up to $500 flat.

Question: what do with the money?  Right now the best candidate is to
pay the Python10 conference fees for some of Jeffrey Elkner's
students.  I can't think of anything else to do with the money; anyone
have any brilliant suggestions?

--amk


From guido@python.org  Tue Oct  2 19:28:33 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 02 Oct 2001 14:28:33 -0400
Subject: [Python-Dev] Bookstore proceeds
In-Reply-To: Your message of "Tue, 02 Oct 2001 13:57:24 EDT."
 <E15oTnI-0005pO-00@ute.cnri.reston.va.us>
References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us>
Message-ID: <200110021828.f92ISXk25410@odiug.digicool.com>

> I've received another cheque from Amazon with the last quarter's
> proceeds for the Python bookstore.  The amount is just a bit less than
> $500; I'll contribute the few dollars to bring it up to $500 flat.
> 
> Question: what do with the money?  Right now the best candidate is to
> pay the Python10 conference fees for some of Jeffrey Elkner's
> students.  I can't think of anything else to do with the money; anyone
> have any brilliant suggestions?

Subsidize Finn Bock and Samuele Pedroni coming to Python10?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From paul@ActiveState.com  Tue Oct  2 19:34:45 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 11:34:45 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com>
Message-ID: <3BBA08C5.240EB4BC@ActiveState.com>

Skip Montanaro wrote:
> 
>...
> 
> I suspect not.  It's special purpose is to parse or generate XML-RPC, so you
> know ahead of time that the end result is the only thing you need.

One reason to use a full XML parser is you get Unicode cheaply. I don't
see Unicode as a feature that you add in a weekend at the end...

 Paul Prescod


From akuchlin@mems-exchange.org  Tue Oct  2 19:37:34 2001
From: akuchlin@mems-exchange.org (Andrew Kuchling)
Date: Tue, 2 Oct 2001 14:37:34 -0400
Subject: [Python-Dev] Bookstore proceeds
In-Reply-To: <200110021828.f92ISXk25410@odiug.digicool.com>; from guido@python.org on Tue, Oct 02, 2001 at 02:28:33PM -0400
References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> <200110021828.f92ISXk25410@odiug.digicool.com>
Message-ID: <20011002143734.J21687@ute.mems-exchange.org>

On Tue, Oct 02, 2001 at 02:28:33PM -0400, Guido van Rossum wrote:
>Subsidize Finn Bock and Samuele Pedroni coming to Python10?

Will $500 be of significant assistance?  

Note on cost: student fees seem to be around $250, so that's two
students.  Developer's Day registration is $295, so it could almost
pay for their admission for that day; it's not enough to help with
their hotel or flight.

--amk


From guido@python.org  Tue Oct  2 19:41:09 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 02 Oct 2001 14:41:09 -0400
Subject: [Python-Dev] Bookstore proceeds
In-Reply-To: Your message of "Tue, 02 Oct 2001 14:37:34 EDT."
 <20011002143734.J21687@ute.mems-exchange.org>
References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> <200110021828.f92ISXk25410@odiug.digicool.com>
 <20011002143734.J21687@ute.mems-exchange.org>
Message-ID: <200110021841.f92If9T26208@odiug.digicool.com>

> Will $500 be of significant assistance?  

Ask them.

> Note on cost: student fees seem to be around $250, so that's two
> students.  Developer's Day registration is $295, so it could almost
> pay for their admission for that day; it's not enough to help with
> their hotel or flight.

Flights are cheap.  Somone locally might be able to put them up.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From skip@pobox.com (Skip Montanaro)  Tue Oct  2 19:57:25 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 13:57:25 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA08C5.240EB4BC@ActiveState.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
Message-ID: <15290.3605.641031.46951@beluga.mojam.com>

    >> I suspect not.  It's special purpose is to parse or generate XML-RPC,
    >> so you know ahead of time that the end result is the only thing you
    >> need.

    Paul> One reason to use a full XML parser is you get Unicode cheaply. I
    Paul> don't see Unicode as a feature that you add in a weekend at the
    Paul> end...

XML-RPC's relationship to Unicode is ill-defined.  The spec that Dave Winer
wrote requires all data to be US-ASCII, so XML-RPC isn't really
XML-compliant.  (You'll have to take up issues of standards compliance with
Dave.)

Still, Unicode or not, the notion that XML-RPC is a data serialization
mechanism instead of a compound data markup language means you don't need to
provide hooks for processing each element, so full-blown XML parsers tend to
be overkill as py-xmlrpc demonstrates.  No matter how hard Shilad finds it
to add Unicode support to his package, it's still likely to be miles ahead
of other XML parsers.

Skip


From bckfnn@worldonline.dk  Tue Oct  2 20:12:53 2001
From: bckfnn@worldonline.dk (Finn Bock)
Date: Tue, 02 Oct 2001 19:12:53 GMT
Subject: [Python-Dev] Bookstore proceeds
In-Reply-To: <200110021828.f92ISXk25410@odiug.digicool.com>
References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us>  <200110021828.f92ISXk25410@odiug.digicool.com>
Message-ID: <3bba10cd.14259694@mail.wanadoo.dk>

[Andrew Kuchling]

> Question: what do with the money?  Right now the best candidate is to
> pay the Python10 conference fees for some of Jeffrey Elkner's
> students.  I can't think of anything else to do with the money; anyone
> have any brilliant suggestions?

[Guido]

>Subsidize Finn Bock and Samuele Pedroni coming to Python10?

"Thanks, but No Thanks" from me.

While I would dearly like to meet you all, the travel is too much of a
pain for my taste.

regards,
finn


From martin@loewis.home.cs.tu-berlin.de  Tue Oct  2 21:09:51 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Tue, 2 Oct 2001 22:09:51 +0200
Subject: [Python-Dev] Demo/dns
Message-ID: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de>

> More radically, we could delete all the code in Demo/dns, and just
> leave behind a little README pointing at Anthony's project.

+1.

Martin


From guido@python.org  Tue Oct  2 21:18:59 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 02 Oct 2001 16:18:59 -0400
Subject: [Python-Dev] Demo/dns
In-Reply-To: Your message of "Tue, 02 Oct 2001 22:09:51 +0200."
 <200110022009.f92K9p301782@mira.informatik.hu-berlin.de>
References: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de>
Message-ID: <200110022018.f92KIxV29996@odiug.digicool.com>

> > More radically, we could delete all the code in Demo/dns, and just
> > leave behind a little README pointing at Anthony's project.

[MvL]
> +1.

If nobody disagrees, I'll do this.  It's unclear where to leave a
pointer -- a Demo/dns directory without code feels a little strange.
Maybe we don't need to leave a pointer?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From paul@ActiveState.com  Tue Oct  2 21:44:11 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 13:44:11 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com>
Message-ID: <3BBA271B.63B07CEF@ActiveState.com>

Skip Montanaro wrote:
> 
>...
> 
> XML-RPC's relationship to Unicode is ill-defined.  The spec that Dave Winer
> wrote requires all data to be US-ASCII, so XML-RPC isn't really
> XML-compliant.  (You'll have to take up issues of standards compliance with
> Dave.)

Most XML-RPC implementations support Unicode, Dave Winer
notwithstanding. Plus, the XML-RPC spec says nothing to indicate that
XML-RPC documents may not be encoded in either of XML's two built-in
encodings (even if the data is restricted to ASCII values).

> Still, Unicode or not, the notion that XML-RPC is a data serialization
> mechanism instead of a compound data markup language means you don't need to
> provide hooks for processing each element, so full-blown XML parsers tend to
> be overkill as py-xmlrpc demonstrates. 

I don't see how that follows. py-xmlrpc needs to handle <struct>
different than <array> so it needs to have a "hook" for each of those
element types. Having a fixed list of hooks or an extensible array of
them should not be much different from a performance point of view.

Yes, an incomplete XML parser could be faster if it ignores Unicode,
ignores character references, and does not do all of the error checking
required by the spec. I'm not sure if this would really improve
performance anyhow.

py-xmlrpc is probably faster because it doesn't call out to Python code
until the entire message has been parsed. xmlrpclib on the other hand,
is entirely written in Python. Is there a Python XML-RPC implementation
that uses no Python code but does use a true XML parser?

> ...  No matter how hard Shilad finds it
> to add Unicode support to his package, it's still likely to be miles ahead
> of other XML parsers.

I think you are exaggerating the benefit of having a fixed vocabulary.
There is hardly any performance boost possible based on that one detail.

 Paul Prescod


From martin@loewis.home.cs.tu-berlin.de  Tue Oct  2 22:30:13 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Tue, 2 Oct 2001 23:30:13 +0200
Subject: [Python-Dev] Integrating Expat
Message-ID: <200110022130.f92LUDt02414@mira.informatik.hu-berlin.de>

> How would libxml fit into this picture ?

Uncertain. I could find out the following facts:

Python Wrappers: Dave Kuhlmann has written some, see
http://www.rexx.com/~dkuhlman/

Licensing: Available through either LGPL or W3C IPR (Daniel Veillard
from Redhat is the maintainer, he used to work for the W3C).

Portability: Apparently tested on Unix and Win32. Uses zlib and iconv
when available.

Size: 85kLOC C source code (compare to 11kLOC in Expat)

Performance: I could not find any results on this subject, comparing
libxml with other parsers.

Regards,
Martin


From nas@python.ca  Tue Oct  2 22:51:14 2001
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 2 Oct 2001 14:51:14 -0700
Subject: [Python-Dev] Demo/dns
In-Reply-To: <200110022018.f92KIxV29996@odiug.digicool.com>; from guido@python.org on Tue, Oct 02, 2001 at 04:18:59PM -0400
References: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> <200110022018.f92KIxV29996@odiug.digicool.com>
Message-ID: <20011002145114.A20139@glacier.arctrix.com>

Guido van Rossum wrote:
> It's unclear where to leave a pointer -- a Demo/dns directory without
> code feels a little strange.

Misc/NEWS?

  Neil


From skip@pobox.com (Skip Montanaro)  Tue Oct  2 22:53:15 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 16:53:15 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA271B.63B07CEF@ActiveState.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com>
Message-ID: <15290.14155.432881.855383@beluga.mojam.com>

    >> Still, Unicode or not, the notion that XML-RPC is a data
    >> serialization mechanism instead of a compound data markup language
    >> means you don't need to provide hooks for processing each element, so
    >> full-blown XML parsers tend to be overkill as py-xmlrpc demonstrates.

    Paul> I don't see how that follows. py-xmlrpc needs to handle <struct>
    Paul> different than <array> so it needs to have a "hook" for each of
    Paul> those element types. Having a fixed list of hooks or an extensible
    Paul> array of them should not be much different from a performance
    Paul> point of view.

Sure, <struct> and <array> mean different things, but <struct> will always
mean the same thing in an XML-RPC context.  There's no need to provide any
hooks.  Once you've successfully parsed a <struct> you get a Python
dictionary.

As far as I can tell sgmlop is always going to be slower than py-xmlrpc
because it must callback to an Unmarshaller instance for each tag.  The only
option currently available is the Unmarshaller class written in Python.
Pythonware has a FastParser/FastUnmarshaller pair available now which I
don't have access to.  Perhaps it exhibits encode/decode speeds similar to
py-xmlrpc.  You'll have to ask Fredrik.  Py-xmlrpc was written with the
knowledge that intermediate results aren't useful and that as you put it, it
has a fixed vocabulary.  Why structure a parser to accommodate situations
that aren't needed?

    Paul> Yes, an incomplete XML parser could be faster if it ignores
    Paul> Unicode, ignores character references, and does not do all of the
    Paul> error checking required by the spec. I'm not sure if this would
    Paul> really improve performance anyhow.

Does py-xmlrpc have a ways to go?  Sure.  It's still pretty new software, so
give it time.  You seem to be dismissing it completely because it's not as
mature as, say, Expat.  I doubt it will lose a factor of 8 in encoding speed
or a factor of 24 in decoding speed (the current speed advantages I measure
over xmlrpclib 1.0b4 using sgmlop) when those things are all added.  I'm not
sure all those things will ever be needed, but you're welcome to think they
will.

    Paul> py-xmlrpc is probably faster because it doesn't call out to Python
    Paul> code until the entire message has been parsed. xmlrpclib on the
    Paul> other hand, is entirely written in Python. Is there a Python
    Paul> XML-RPC implementation that uses no Python code but does use a
    Paul> true XML parser?

That's precisely why py-xmlrpc is faster.  Should it behave some other way?
I don't think there is another XML-RPC parser out there that is available
from Python but that doesn't use Python.

    >> ...  No matter how hard Shilad finds it to add Unicode support to his
    >> package, it's still likely to be miles ahead of other XML parsers.

    Paul> I think you are exaggerating the benefit of having a fixed
    Paul> vocabulary.  There is hardly any performance boost possible based
    Paul> on that one detail.

I don't understand see how you can't make that connection.  XML-RPC has a
fixed vocabulary and never needs to look at intermediate results.  It sounds
to me like all you have is a hammer so everything looks like a nail.  There
are places for general-purpose XML parsers and places for special-purpose
XML parsers.  In this particular context I only care about how fast I can
push objects between a client and server using XML-RPC.

I apologize if the subject seems more general than I intended.  My only
intention was to compare the data serialization performance of various
tools.  I didn't include "XML-RPC" in the subject of this thread because I
tossed in marshal and cPickle results as well, simply for comparison.

Skip


From pedroni@inf.ethz.ch  Tue Oct  2 22:57:35 2001
From: pedroni@inf.ethz.ch (Samuele Pedroni)
Date: Tue, 2 Oct 2001 23:57:35 +0200
Subject: [Python-Dev] Bookstore proceeds
References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us>  <200110021828.f92ISXk25410@odiug.digicool.com>
Message-ID: <00d601c14b8d$40049a80$8a73fea9@newmexico>

[Guido]
> Subsidize Finn Bock and Samuele Pedroni coming to Python10?

Thanks for the proposal.

Surely I would like to meet you all but - financial issues aside -
while nice for me, I don't think my presence would 
be that much constructive and Finn -  who's the main mantainer 
and has done a whole lot - deserves more than me to eventually meet you
and the users and receive their thanks.

I'm not your guy and Finn don't want to fly :(

Samuele.



From fredrik@pythonware.com  Tue Oct  2 23:18:02 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 3 Oct 2001 00:18:02 +0200
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>        <15289.10654.162126.780312@grendel.zope.com>        <03aa01c14b15$e329ac00$b3fa42d5@hagrid>        <15289.45881.920165.427067@beluga.mojam.com>        <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com>
Message-ID: <050d01c14b90$213834b0$b3fa42d5@hagrid>

paul wrote:
> > XML-RPC's relationship to Unicode is ill-defined.  The spec that Dave Winer
> > wrote requires all data to be US-ASCII, so XML-RPC isn't really
> > XML-compliant.  (You'll have to take up issues of standards compliance with
> > Dave.)
> 
> Most XML-RPC implementations support Unicode, Dave Winer
> notwithstanding. Plus, the XML-RPC spec says nothing to indicate that
> XML-RPC documents may not be encoded in either of XML's two built-in
> encodings (even if the data is restricted to ASCII values).

the specification says that XML-RPC uses XML and HTTP.  it
doesn't say anything about a Dave-specific subset of XML or
HTTP...

(like so many other parts of the specification, the "string" type
isn't exactly well-specified. the specification first says that strings
contains ASCII characters, and later that "any characters are
allowed in a string" and that "a string can be used to encode
binary data")

> Yes, an incomplete XML parser could be faster if it ignores Unicode,
> ignores character references, and does not do all of the error checking
> required by the spec.  I'm not sure if this would really improve
> performance anyhow.

well, sgmlop is a bit faster than expat (up to 50%, in some tests).
expat does a bit more error checking.

> xmlrpclib on the other hand, is entirely written in Python. Is there a
> Python XML-RPC implementation that uses no Python code but does
> use a true XML parser?

the _xmlrpclib accelerator (see the xmlrpclib.py source) uses expat,
with a really fast C layer.  judging from Skip's benchmarks, expat is a
bit slower than the py-xmlrpc parser (which is why I asked).

</F>



From paul@pfdubois.com  Tue Oct  2 23:16:16 2001
From: paul@pfdubois.com (Paul F. Dubois)
Date: Tue, 2 Oct 2001 15:16:16 -0700
Subject: [Python-Dev] Pmw broken under 2.2a4?
Message-ID: <000101c14b8f$db4938a0$3d01a8c0@plstn1.sfba.home.com>

Building Pmw 0.8.5 under 2.2a4 I got syntax errors of the 'inconsistent
use of tabs and spaces' kind. I'm not sure who I should tell but I
speculate that I just did.



From paul@ActiveState.com  Tue Oct  2 23:15:49 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 15:15:49 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com> <15290.14155.432881.855383@beluga.mojam.com>
Message-ID: <3BBA3C95.1CC094CC@ActiveState.com>

Skip Montanaro wrote:
> 
>...
> 
> Sure, <struct> and <array> mean different things, but <struct> will always
> mean the same thing in an XML-RPC context.  There's no need to provide any
> hooks.  

There are two different issues. One is parsing: taking a string of bytes
and interpreting them as XML. The other is passing this information to
the Python programmer. The handling of "hooks" is on the backend,
passing the information to the Python programmer. I interpreted
Fredrick's question as being about the front end: does it use a real XML
parser or not.

>...
> Does py-xmlrpc have a ways to go?  Sure.  It's still pretty new software, so
> give it time.  You seem to be dismissing it completely because it's not as
> mature as, say, Expat.  

I'm not asking it to be as mature as Expat. I'm asking why it didn't
*use* Expat or some other parser. Expat would recognize structs and
arrays and pass them to C code which builds Python objects. Then those
Python objects can be passed to Python.

>...
>     Paul> py-xmlrpc is probably faster because it doesn't call out to Python
>     Paul> code until the entire message has been parsed. xmlrpclib on the
>     Paul> other hand, is entirely written in Python. Is there a Python
>     Paul> XML-RPC implementation that uses no Python code but does use a
>     Paul> true XML parser?
> 
> That's precisely why py-xmlrpc is faster.  Should it behave some other way?
> I don't think there is another XML-RPC parser out there that is available
> from Python but that doesn't use Python.

Okay, so we agree that the fast part is probably not so much the parser
but the handing of data to Python. So why rewrite a parser? Nothing
requires an Expat-using XML-RPC implementation to call back into Python
for every element. It can collect the results in C and then call Python
when it has values.

>...
> I don't understand see how you can't make that connection.  XML-RPC has a
> fixed vocabulary and never needs to look at intermediate results.  

Let me suggest an analogy. Someone writes "CGIPython". It uses a
specially optimized parser designed for parsing only Python CGI scripts.
Do you think it would run much faster than the regular Python parser?

Well, syntactically CGI scripts are basically the same as ordinary
Python programs so why would you *want* a specialized parser? Parsing
angle brackets is the same whether they are in an XML-RPC message or a
Docbook document, just as parsing Python is the same, whether it is a
CGI or a GUI app.

> ... It sounds
> to me like all you have is a hammer so everything looks like a nail.  There
> are places for general-purpose XML parsers and places for special-purpose
> XML parsers.  In this particular context I only care about how fast I can
> push objects between a client and server using XML-RPC.

I don't personally see much benefit using XML if you don't adhere to the
XML spec. Just perusing the code quickly I believe I've found a few bugs
that it would not have had if it built on Expat or some other XML
parser.

 1. It doesn't handle &#63; syntax.

 2. It doesn't handle <methodCall > (extra whitespace)

 3. I strongly suspect it won't handle comments in the XML.

 4. It won't handle the mandatory UTF-16 encoding from XML

 5. It won't handle CDATA sections.

 Paul Prescod


From paul@ActiveState.com  Tue Oct  2 23:19:05 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 15:19:05 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>        <15289.10654.162126.780312@grendel.zope.com>        <03aa01c14b15$e329ac00$b3fa42d5@hagrid>        <15289.45881.920165.427067@beluga.mojam.com>        <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <050d01c14b90$213834b0$b3fa42d5@hagrid>
Message-ID: <3BBA3D59.441625E9@ActiveState.com>

Fredrik Lundh wrote:
> 
>...
> 
> the _xmlrpclib accelerator (see the xmlrpclib.py source) uses expat,
> with a really fast C layer.  judging from Skip's benchmarks, expat is a
> bit slower than the py-xmlrpc parser (which is why I asked).

I have a feeling py-xmlrpc will slow down a bit when it is
internationalized:

	if (strncmp(*cp, "<int>", 5) == 0)
		res = decodeInt(cp, ep, lines);
	else if (strncmp(*cp, "<i4>", 4) == 0)
		res = decodeI4(cp, ep, lines);
   ....

 Paul Prescod


From firstalertnews@yahoo.com  Tue Oct  2 21:51:59 2001
From: firstalertnews@yahoo.com (firstalertnews@yahoo.com)
Date: Tue, 02 Oct 2001  13:51:59 -0700
Subject: [Python-Dev] FIRST ALERT - BTLY: OTCBB
Message-ID: <E15oY1F-0001jj-00@mail.python.org>

FIRSTALERT'S COMPANY PROFILE FOR THE WEEK OF OCTOBER 1st

BENTLEY COMMUNICATIONS CORP: (OTC BB: BTLY)

Good day Investors!  After the last two weeks of trading everyone is looking for new investments.  This week's company profile highlights Bentley Communications Corp. (OTC BB: BTLY).  Bentley brings consumers into the world of cutting-edge international gaming websites. Through proprietary technology and advantageous partnership agreements, Bentley operates licensed and fully-insured online gaming websites and sublicenses I-gaming software to businesses worldwide. The Company's flagship I-property, SunriseCityCasino, offers games such as Blackjack, Roulette, Slots, Video Poker and provides users instant access to online betting lines for major sporting events worldwide. Bentley is poised to capture increasing share of the robust Internet sector of I-gaming. By 2002, the global I-gaming market is projected to exceed $1 billion and increase dramatically to more than $9.8 billion by 2005 (source: Merrill Lynch). Bentley targets its websites and software for the world's most popular I-gaming regions, including the United States, Asia and Australia. The Company's growth strategies are directed by Gordon Lee, a polished entrepreneur who has founded several successful emerging-growth companies. Under his leadership, Bentley is poised to enhance its market capitalization substantially over the next 24-months.

CORPORATE SUMMARY

Bentley Communications Corp. (OTC BB: BTLY) specializes in operating Internet wagering sites. The Company is licensed in both the Commonwealth of Dominica and Antigua to operate sites inclusive of sportsbook, horseracing and bingo. Bentley also sublicenses proprietary technology and software that enable clients to operate their own online gaming sites under the Bentley license. The Company recently announced the launch of a world-class online gaming site SunriseCityCasino.com. Fully licensed and insured, Sunrise offers more than 20 casino games and features a sportsbook with real-time betting lines on worldwide events. Bentley is embarking upon an aggressive growth strategy that comprises geographical expansion into Europe and Asia and new product lines, including the selling of international lottery tickets online.

Bentley is also prepared to benefit from new legislation in the state of California. Effective January 1, 2002, telephone and Internet betting on horse racing will be legal in California. Bentley is developing the site offtrackbettingonline.com for launch on 1/2/02, in order to capitalize on this new market in the United States. With numerous revenue streams in place (online casino and sportsbook betting; sublicensing set-up and maintenance fees) and substantial opportunities to grow revenues and profits in the near-term, Bentley is poised to become a leader in the booming online international gaming market.

COMPANY DATA

Company Name: Bentley Comm Corp (BTLY)

Exchange: OTC BB

Shares Outstanding: 26,660,000

Market Cap: 2.4 Million

52-Week High: 0.875 on Wednesday, September 27, 2000

52-Week Low: 0.039 on Tuesday, July 31, 2001

Average Price: 0.0990 (50-day)   0.1337 (200-day)

Average Volume: 110,100 (50-day)   73,600 (200-day)

Fiscal Year End: June 30

Market Makers: 17

INDUSTRY STATISTICS

According to Merrill Lynch, the number of online bettors is expected to increase dramatically from 12 million in 2001 to 81 million in 2005, translating to a market size of $769 million and $9.8 billion respectively. Bentley's primary offerings-Internet casinos and sports betting, currently generate the highest gross win (i.e. the amount of cash lost by consumers) in the industry, followed by online lotteries (a developing new product line for Bentley). Bentley's distinct competitive advantages, leading-edge security and fraud protection; a diversified product mix; turn-key proprietary software and technologies and strategic partnerships, will enable the Company to capture increasing share of this dynamic international marketplace.

HIGHLIGHTS

High-Growth Market. The worldwide Internet gaming market is expected to grow well over 450% over the next two years. Bentley is well-positioned to capitalize on this high-growth sector through the roll-out of several new product lines and regional expansion worldwide. Strategic Partnerships. Bentley has aligned itself with leading technology, banking and security firms to create world-class I-gaming software and Internet websites The Company is associated with the largest e-mail marketing company in the world, Salesmation and has an exclusive agreement with First International Bank of Dominica to provide real-timesecure online financial transactions.

Sales & Profitability. Online casinos and sports betting are key revenue generators for Bentley. Additional revenues are generated from sub licensing fees, ranging from $30-50K and the management and maintenance payments of 10% of clients' net revenues. As new products are rolled out, Bentley anticipates to exceed 100% annual revenue growth and achieve profitability during 2001 and beyond.

Proven Management. Bentley President and CEO Gordon Lee has a 20-year track record of directing private and public start-ups into successfully operating companies. Mr. Lee has held executive posts including founder, partner, officer, and director at companies such as USA Video Corporation (OTC BB: USVO), Startek.com Inc.(OTC BB: STEK), Future Media Technologies (OTC: FMTF) and American IDC Corp. (OTC BB: ACNI).

Upside Opportunity. Bentley is implementing a multifold growth plan that includes international expansion into the fastest growing I-gaming region of Asia; roll-out of new, higher margin product lines such online lottery sales and the launch of additional websites that capitalize on I-gaming growth trends. These efforts should lead to accelerating revenues and increasing shareholder value.

EXPERIENCED LEADERSHIP

Gordon Lee, president and CEO of Bentley Communications Corp, has been involved with emerging growth companies in the private and public arena for more than 20 years. His astute business acumen has been applied toward a variety of industries, including hi-tech, communications, e-commerce, mining and oil and gas and he has been featured in more than 200 influential media publications.

TO OPT OT PLEASE VISIT THE FOLLOWING SITE: http://65.12.58.131/optout.html

DISCLAIMER

This report is provided as an information service only, and the statements and opinions in this report should not be construed as an offer or solicitation to buy or sell any security. We accept no liability for any loss arising from an investor's reliance on or use of this report. Please do your own due diligence and research before making any investment decision. An investment in the company outlined herein is highly speculative and should not be considered unless you can afford a complete loss of your investment. Please, always invest with caution and within your means. You can find a great resource for investor education at the U.S. Securities and Exchange's website at http://www.sec.gov/investor.shtml. Also, please see the SEC filings of the company profiled herein, including the company's most recent annual and quarterly reports. EquityThunder Investment Corporation received 50,000 free trading shares of Bentley Communications (BTLY) from a third party in exchange for the drafting and distribution of this report. EquityThunder Investment Corporation intends to sell its shares.





From guido@python.org  Wed Oct  3 00:13:00 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 02 Oct 2001 19:13:00 -0400
Subject: [Python-Dev] Demo/dns
In-Reply-To: Your message of "Tue, 02 Oct 2001 14:51:14 PDT."
 <20011002145114.A20139@glacier.arctrix.com>
References: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> <200110022018.f92KIxV29996@odiug.digicool.com>
 <20011002145114.A20139@glacier.arctrix.com>
Message-ID: <200110022313.TAA18864@cj20424-a.reston1.va.home.com>

> Guido van Rossum wrote:
> > It's unclear where to leave a pointer -- a Demo/dns directory without
> > code feels a little strange.
> 
> Misc/NEWS?
> 
>   Neil

Good idea.  Done. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From skip@pobox.com (Skip Montanaro)  Wed Oct  3 01:33:59 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 19:33:59 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA3C95.1CC094CC@ActiveState.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com>
 <15290.14155.432881.855383@beluga.mojam.com>
 <3BBA3C95.1CC094CC@ActiveState.com>
Message-ID: <15290.23799.884095.945168@beluga.mojam.com>

    >> That's precisely why py-xmlrpc is faster.  Should it behave some
    >> other way?  I don't think there is another XML-RPC parser out there
    >> that is available from Python but that doesn't use Python.

    Paul> Okay, so we agree that the fast part is probably not so much the
    Paul> parser but the handing of data to Python. So why rewrite a parser?
    Paul> Nothing requires an Expat-using XML-RPC implementation to call
    Paul> back into Python for every element. It can collect the results in
    Paul> C and then call Python when it has values.

You're asking the wrong person.  Shilad will be the only person who can
describe his motivations.  We happen to work in the same building, but we
don't work for the same company.  That's a coincidence about on par with the
chances of winning the Powerball lottery.  We never met each other formally
until about a week ago.  Not trying to put words in his mouth, but my guess
would be that he was not approaching it as an XML problem, but as a parsing
problem.

    >> I don't understand see how you can't make that connection.  XML-RPC
    >> has a fixed vocabulary and never needs to look at intermediate
    >> results.

    Paul> Let me suggest an analogy. Someone writes "CGIPython". It uses a
    Paul> specially optimized parser designed for parsing only Python CGI
    Paul> scripts.  Do you think it would run much faster than the regular
    Paul> Python parser?

Bad analogy.  CGI scripts can contain the entire realm of "stuff" that goes
into any other Python program.  XML-RPC encodings can't contain arbitrary
XML tags or attributes.  A better analogy would have been (Martin's I think)
hypothetical Swallow - a subset of Python that could be efficiently
compiled.

    Paul> I don't personally see much benefit using XML if you don't adhere
    Paul> to the XML spec.  Just perusing the code quickly I believe I've
    Paul> found a few bugs that it would not have had if it built on Expat
    Paul> or some other XML parser.

Paul, you have to stop looking at XML-RPC with your Elton John-style
XML-colored glasses.  XML-RPC is not meant to be some sort of highly
structured hierarchical data representation that you can sniff around in
with arbitrary XML tools of one sort or another.  That its on-the-wire
representation happens to be XML is almost ridiculously unimportant.  Dave
Winer created an RPC tool that used XML at about the same time every
computer journalist was wetting their pants every time they heard the
letters X-M-L.  Many implementations were able to leverage existing XML
parsing tools to get going quickly, and Dave got some well-deserved
publicity that he and XML-RPC wouldn't have gotten if he'd chosen some other
serliazation format like Pickle, or invented something new.  Next step: make
it go faster.  Can that be done with standard XML tools?  Yeah, I'm sure it
can be.  Not everybody approaches the problem with the same background you
have though.

    Paul>  1. It doesn't handle &#63; syntax.

    Paul>  2. It doesn't handle <methodCall > (extra whitespace)

    Paul>  3. I strongly suspect it won't handle comments in the XML.

    Paul>  4. It won't handle the mandatory UTF-16 encoding from XML

    Paul>  5. It won't handle CDATA sections.

Fine.  I'm sure Shilad appreciates the input.  I think your approach to bug
detection and reporting could have been a bit less heavy handed.

As for handling things like CDATA, UTF-16 and extra whitespace after tag
names, I suspect some other XML-RPC packages would exhibit similar problems
if they were exposed to a standards-toting XML gunslinger like yourself.
That it's not a problem in practice is probably because the set of XML-RPC
encoding and decoding software is fairly small and that the stuff that
encodes into XML-RPC is fairly well-behaved.

XML-RPC's widespread availability and practical interoperability (the
XML-RPC website lists 48 implementations) probably owes more to the
cooperative nature of the people involved than the purity of the parsers.

Skip


From skip@pobox.com (Skip Montanaro)  Wed Oct  3 01:41:04 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 19:41:04 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA3D59.441625E9@ActiveState.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com>
 <050d01c14b90$213834b0$b3fa42d5@hagrid>
 <3BBA3D59.441625E9@ActiveState.com>
Message-ID: <15290.24224.866683.240585@beluga.mojam.com>

    Paul> I have a feeling py-xmlrpc will slow down a bit when it is
    Paul> internationalized:

    Paul>       if (strncmp(*cp, "<int>", 5) == 0)
    Paul>               res = decodeInt(cp, ep, lines);
    Paul>       else if (strncmp(*cp, "<i4>", 4) == 0)
    Paul>               res = decodeI4(cp, ep, lines);
    Paul>    ....

Paul, 

If you want to find and fix bugs in py-xmlrpc or help the author improve the
quality of his tools, please send your reports directly to Shilad Sen
(shilad@sourcelight.com).  Py-xmlrpc has nothing to do with the Python core.
I apologize for even including it in the table I posted.  Shilad didn't
deserve any of the bad press you've given him here.  Sending snickering
notes to python-dev about the code is not helpful, and only serves to lessen
the value I place on your other opinions.

Skip


From paul@ActiveState.com  Wed Oct  3 02:11:47 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 18:11:47 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com>
 <15290.14155.432881.855383@beluga.mojam.com>
 <3BBA3C95.1CC094CC@ActiveState.com> <15290.23799.884095.945168@beluga.mojam.com>
Message-ID: <3BBA65D3.3AE8FAA6@ActiveState.com>

Skip Montanaro wrote:
> 
>...
> 
> Bad analogy.  CGI scripts can contain the entire realm of "stuff" that goes
> into any other Python program.  XML-RPC encodings can't contain arbitrary
> XML tags or attributes.  A better analogy would have been (Martin's I think)
> hypothetical Swallow - a subset of Python that could be efficiently
> compiled.

But there is no evidence that this subset of XML can be more efficiently
parsed than any other. XML parsing consists primarily of recognizing
angle brackets and a few other characters, and passing around some extra
data. Any performance loss from a "full" XML parser will shrink as
people submit bug reports that require a "simplified" XML parser to
conform to the XML spec (Unicode, CDATA, etc.).

I strongly agree that a dedicated C-written XML-RPC implementation can
be faster than one written based on Python and Expat. I haven't yet seen
evidence that you can both conform with the standards and get much of a
speedup over one that is built on a fast XML Parser such as Eric Kidd's
XML-RPC C or xmlrpc-epi (both on SourceForge).

>...
> Paul, you have to stop looking at XML-RPC with your Elton John-style
> XML-colored glasses.  XML-RPC is not meant to be some sort of highly
> structured hierarchical data representation that you can sniff around in
> with arbitrary XML tools of one sort or another. That its on-the-wire
> representation happens to be XML is almost ridiculously unimportant.  

XML-RPC uses XML for exactly the same reason every other application of
XML uses XML. Precisely so that you will not have to write yet another
parser for it. That's the central reason *for* XML. That's the only
advantage XML has over cPickle -- that you can be sure whatever language
you have, it will have an XML parser available built in.

> Fine.  I'm sure Shilad appreciates the input.  I think your 
> approach to bug detection and reporting could have been a bit 
> less heavy handed.

I'm not trying to embarrass Shilad. The software isn't at 1.0 yet. Maybe
he hasn't got around to choosing an XML parser. 

I'm trying to point out (more to you, than to him!) that there is a good
reason to build on the work other people have done. If pyxmlrpc is
faster today it is probably because it doesn't conform to the specs.
When it does conform, it won't be faster anymore.

> As for handling things like CDATA, UTF-16 and extra whitespace after tag
> names, I suspect some other XML-RPC packages would exhibit similar problems
> if they were exposed to a standards-toting XML gunslinger like yourself.
> That it's not a problem in practice is probably because the set of XML-RPC
> encoding and decoding software is fairly small and that the stuff that
> encodes into XML-RPC is fairly well-behaved.

Every XML-RPC implementation I have ever used (Python, Perl, C, C++,
PHP) is based upon one pure XML parser or another. Most use Expat.

 Paul Prescod


From shilad@sourcelight.com  Wed Oct  3 02:28:53 2001
From: shilad@sourcelight.com (Shilad Sen)
Date: Tue, 2 Oct 2001 20:28:53 -0500 (CDT)
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <15290.23799.884095.945168@beluga.mojam.com> from "Skip Montanaro" at Oct 02, 2001 07:33:59 PM
Message-ID: <200110030128.f931Sr005236@jericho.sourcelight.com>

Skip has been kind enough to copy me on the bulk of correspondence
regarding py-xmlrpc versus other xmlrpc parsing options.

py-xmlrpc began as a short hack to accomplish specific things that
xmlrpclib couldn't easily accomodate.  I used a hand build parser
because I thought it would be fun and easy (it was!).

Paul, you are correct in that my library doesn't support the 5 items you
mentioned.  I am aware of these, but they are actually not officially
supported by the spec either.  XML-RPC is a bit strange in that the spec
does not allow or require true XML.

My library has been adopted far more than I would have guessed, and I
have had many questions about things like SSL support (which is not up
to spec either).  As a result, I am almost finished with a rewrite that
has all the transport and protocol components nicely split up.  I have
on my list of todo's switching the hand coded parser to expat.  My own
parser works just fine, though, and I haven't had any complaints so that
is relatively low on the list.

My library is certainly not as flexible as xmlrpclib in it's current
format.  I'm hoping that the rewrite will move it to a nice place in the
performance / flexibility spectrum.  As a side effect, it will have a
nice extensible standalone HTTP client and server that offers better
performance for people who really need it.

I am perfectly aware of py-xmlrpc's shortcomings.  On the other hand it
is exactly what the app we use needs, and I would be surprised if there
aren't others who have similar needs.  My hope is that with the next
major release, the library will move a bit closer to a place that suits
people like Paul.  Meanwhile, it works nicely for applications where
performance requirements are absolutely critical.

Shilad Sen


> 
>     >> That's precisely why py-xmlrpc is faster.  Should it behave some
>     >> other way?  I don't think there is another XML-RPC parser out there
>     >> that is available from Python but that doesn't use Python.
> 
>     Paul> Okay, so we agree that the fast part is probably not so much the
>     Paul> parser but the handing of data to Python. So why rewrite a parser?
>     Paul> Nothing requires an Expat-using XML-RPC implementation to call
>     Paul> back into Python for every element. It can collect the results in
>     Paul> C and then call Python when it has values.
> 
> You're asking the wrong person.  Shilad will be the only person who can
> describe his motivations.  We happen to work in the same building, but we
> don't work for the same company.  That's a coincidence about on par with the
> chances of winning the Powerball lottery.  We never met each other formally
> until about a week ago.  Not trying to put words in his mouth, but my guess
> would be that he was not approaching it as an XML problem, but as a parsing
> problem.
> 
>     >> I don't understand see how you can't make that connection.  XML-RPC
>     >> has a fixed vocabulary and never needs to look at intermediate
>     >> results.
> 
>     Paul> Let me suggest an analogy. Someone writes "CGIPython". It uses a
>     Paul> specially optimized parser designed for parsing only Python CGI
>     Paul> scripts.  Do you think it would run much faster than the regular
>     Paul> Python parser?
> 
> Bad analogy.  CGI scripts can contain the entire realm of "stuff" that goes
> into any other Python program.  XML-RPC encodings can't contain arbitrary
> XML tags or attributes.  A better analogy would have been (Martin's I think)
> hypothetical Swallow - a subset of Python that could be efficiently
> compiled.
> 
>     Paul> I don't personally see much benefit using XML if you don't adhere
>     Paul> to the XML spec.  Just perusing the code quickly I believe I've
>     Paul> found a few bugs that it would not have had if it built on Expat
>     Paul> or some other XML parser.
> 
> Paul, you have to stop looking at XML-RPC with your Elton John-style
> XML-colored glasses.  XML-RPC is not meant to be some sort of highly
> structured hierarchical data representation that you can sniff around in
> with arbitrary XML tools of one sort or another.  That its on-the-wire
> representation happens to be XML is almost ridiculously unimportant.  Dave
> Winer created an RPC tool that used XML at about the same time every
> computer journalist was wetting their pants every time they heard the
> letters X-M-L.  Many implementations were able to leverage existing XML
> parsing tools to get going quickly, and Dave got some well-deserved
> publicity that he and XML-RPC wouldn't have gotten if he'd chosen some other
> serliazation format like Pickle, or invented something new.  Next step: make
> it go faster.  Can that be done with standard XML tools?  Yeah, I'm sure it
> can be.  Not everybody approaches the problem with the same background you
> have though.
> 
>     Paul>  1. It doesn't handle &#63; syntax.
> 
>     Paul>  2. It doesn't handle <methodCall > (extra whitespace)
> 
>     Paul>  3. I strongly suspect it won't handle comments in the XML.
> 
>     Paul>  4. It won't handle the mandatory UTF-16 encoding from XML
> 
>     Paul>  5. It won't handle CDATA sections.
> 
> Fine.  I'm sure Shilad appreciates the input.  I think your approach to bug
> detection and reporting could have been a bit less heavy handed.
> 
> As for handling things like CDATA, UTF-16 and extra whitespace after tag
> names, I suspect some other XML-RPC packages would exhibit similar problems
> if they were exposed to a standards-toting XML gunslinger like yourself.
> That it's not a problem in practice is probably because the set of XML-RPC
> encoding and decoding software is fairly small and that the stuff that
> encodes into XML-RPC is fairly well-behaved.
> 
> XML-RPC's widespread availability and practical interoperability (the
> XML-RPC website lists 48 implementations) probably owes more to the
> cooperative nature of the people involved than the purity of the parsers.
> 
> Skip
> 



From barry@zope.com  Wed Oct  3 02:39:04 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Tue, 2 Oct 2001 21:39:04 -0400
Subject: [Python-Dev] Cookie.py and `:' in the key
Message-ID: <15290.27704.19609.960912@anthem.wooz.org>

In Mailman, I use a version of Cookie.py written by Timothy dated from
1998.  I'm now trying to see if I can get rid of my independent
copy and just use Cookie.py in the Python 2.x standard library.

I've hit a snag; in Mailman's copy, it is legal to have a `:' in the
key name, but in Python's Cookie.py it isn't:

-------------------- snip snip --------------------
% python
Python 2.1.1 (#1, Aug 31 2001, 17:07:00) 
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> import Cookie
>>> c = Cookie.Cookie()
>>> c['foo:bar'] = 'hello'
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/lib/python2.1/Cookie.py", line 583, in __setitem__
    self.__set(key, rval, cval)
  File "/usr/local/lib/python2.1/Cookie.py", line 576, in __set
    M.set(key, real_value, coded_value)
  File "/usr/local/lib/python2.1/Cookie.py", line 456, in set
    raise CookieError("Illegal key value: %s" % key)
Cookie.CookieError: Illegal key value: foo:bar
>>> 
% PYTHONPATH=/path/to/mailman/misc python
Python 2.1.1 (#1, Aug 31 2001, 17:07:00) 
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> import Cookie
>>> c = Cookie.Cookie()
>>> c['foo:bar'] = 'hello'
>>> print c
Set-Cookie: foo:bar=hello;
>>> 
-------------------- snip snip --------------------

I don't see any reason why `:' shouldn't be allowed in Set-Cookie:
value, but maybe I'm missing something in the RFCs.  This patch fixes
the problem but perhaps not in the right way.  Comments?

-Barry

-------------------- snip snip --------------------
Index: Cookie.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/Cookie.py,v
retrieving revision 1.11
diff -u -r1.11 Cookie.py
--- Cookie.py	2001/08/02 07:15:29	1.11
+++ Cookie.py	2001/10/03 01:38:39
@@ -249,7 +249,7 @@
 #       _LegalChars       is the list of chars which don't require "'s
 #       _Translator       hash-table for fast quoting
 #
-_LegalChars       = string.ascii_letters + string.digits + "!#$%&'*+-.^_`|~"
+_LegalChars       = string.ascii_letters + string.digits + "!#$%&'*+-.^_`|~:"
 _Translator       = {
     '\000' : '\\000',  '\001' : '\\001',  '\002' : '\\002',
     '\003' : '\\003',  '\004' : '\\004',  '\005' : '\\005',


From paul@ActiveState.com  Wed Oct  3 02:47:06 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 18:47:06 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com>
 <050d01c14b90$213834b0$b3fa42d5@hagrid>
 <3BBA3D59.441625E9@ActiveState.com> <15290.24224.866683.240585@beluga.mojam.com>
Message-ID: <3BBA6E1A.363F4E79@ActiveState.com>

I apologize if I embarassed Shilad. I don't know him so I don't know how
he will take a public critique of his code. For all I know, he agrees
with me and merely hasn't got around to adding in an XML parser.

On the one hand, I can see how it would be nicer to discuss it directly
with you and him, but on the other, it is a real technical issue that
deserves public discussion.

I felt (and feel) that you've made a technical mistake in attributing
py-xmlrpc's speed to its having a fixed tagset and I only posted code to
demonstrate where the real speedup comes from. I've spent my whole life
working around bugs in hand-rolled XML (and SGML) parsers that are
supposed to be faster than general ones but end up not being so. I react
almost as intemperately when someone tells me that their app embeds a
new scripting language that they invented over the weekend.

Although I do think that the current parsing approach taken in py-xmlrpc
is flawed, I do think that the overall idea is good. It makes sense to
parse XML-RPC purely in C without using Python callbacks.

 Paul Prescod


From paul@ActiveState.com  Wed Oct  3 02:57:35 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 02 Oct 2001 18:57:35 -0700
Subject: [Python-Dev] Performance of various marshallers
References: <200110030128.f931Sr005236@jericho.sourcelight.com>
Message-ID: <3BBA708F.D8554BBC@ActiveState.com>

Shilad Sen wrote:
> 
> Skip has been kind enough to copy me on the bulk of correspondence
> regarding py-xmlrpc versus other xmlrpc parsing options.

Thanks for your good-natured response.

>...
> Paul, you are correct in that my library doesn't support the 5 items you
> mentioned.  I am aware of these, but they are actually not officially
> supported by the spec either.  XML-RPC is a bit strange in that the spec
> does not allow or require true XML.

I think that if a spec claims to be based on XML and does not explicitly
disclaim support for built-in XML features, then it allows them. For
instance if it doesn't say that &#67; syntax is illegal, then there is
no reason to believe it is.

> My library has been adopted far more than I would have guessed, and I
> have had many questions about things like SSL support (which is not up
> to spec either).  As a result, I am almost finished with a rewrite that
> has all the transport and protocol components nicely split up. I have
> on my list of todo's switching the hand coded parser to expat. My own
> parser works just fine, though, and I haven't had any complaints 
> so that is relatively low on the list.

That's fine with me. If your simplified parser turns out to be
significantly faster than Expat (too early to say) then you could even
keep it around as an option when the client and the server are both
known to be using the same subset of XML.

> ...  My hope is that with the next
> major release, the library will move a bit closer to a place that suits
> people like Paul.  Meanwhile, it works nicely for applications where
> performance requirements are absolutely critical.

Did you consider wrapping one of the existing XML-RPC libraries written
in C? When we needed a reentrant XML-RPC library for PHP, we wrapped
Eric Kidd's xmlrpc-c.

 Paul Prescod


From shilad@sourcelight.com  Wed Oct  3 03:12:43 2001
From: shilad@sourcelight.com (Shilad Sen)
Date: Tue, 2 Oct 2001 21:12:43 -0500 (CDT)
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA708F.D8554BBC@ActiveState.com> from "Paul Prescod" at Oct 02, 2001 06:57:35 PM
Message-ID: <200110030212.f932Chn05354@jericho.sourcelight.com>

>From Paul:

> 
> That's fine with me. If your simplified parser turns out to be
> significantly faster than Expat (too early to say) then you could even
> keep it around as an option when the client and the server are both
> known to be using the same subset of XML.
> 

That's exactly what I intended to do.

> 
> Did you consider wrapping one of the existing XML-RPC libraries written
> in C? When we needed a reentrant XML-RPC library for PHP, we wrapped
> Eric Kidd's xmlrpc-c.
> 

I actually seriously considered using Eric Kidd's library.  We have a
need for an event model that would have been pretty painful to support
through the w3c libraries, and that is what made me decide to use my own.
My event system is very similar to medusa's, while w3c's system is quite
a bit more heavyweight.

After spending some time with the w3c libraries, I also decided I really
didn't feel great about the general coding style of the libraries.  Thats
my own personal preference.  I could definitely be persuaded otherwise.

Shilad


From skip@pobox.com (Skip Montanaro)  Wed Oct  3 03:16:40 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 21:16:40 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA65D3.3AE8FAA6@ActiveState.com>
References: <15289.7206.266140.820041@beluga.mojam.com>
 <15289.10654.162126.780312@grendel.zope.com>
 <03aa01c14b15$e329ac00$b3fa42d5@hagrid>
 <15289.45881.920165.427067@beluga.mojam.com>
 <3BBA08C5.240EB4BC@ActiveState.com>
 <15290.3605.641031.46951@beluga.mojam.com>
 <3BBA271B.63B07CEF@ActiveState.com>
 <15290.14155.432881.855383@beluga.mojam.com>
 <3BBA3C95.1CC094CC@ActiveState.com>
 <15290.23799.884095.945168@beluga.mojam.com>
 <3BBA65D3.3AE8FAA6@ActiveState.com>
Message-ID: <15290.29960.899139.647577@beluga.mojam.com>

    >> Paul, you have to stop looking at XML-RPC with your Elton John-style
    >> XML-colored glasses.  XML-RPC is not meant to be some sort of highly
    >> structured hierarchical data representation that you can sniff around
    >> in with arbitrary XML tools of one sort or another. That its
    >> on-the-wire representation happens to be XML is almost ridiculously
    >> unimportant.

    Paul> XML-RPC uses XML for exactly the same reason every other
    Paul> application of XML uses XML.

I disagree with that.  Lots of applications use XML because it's got that
pants-wetting capability I described earlier.

    >> Fine.  I'm sure Shilad appreciates the input.  I think your approach
    >> to bug detection and reporting could have been a bit less heavy
    >> handed.

    Paul> I'm not trying to embarrass Shilad. The software isn't at 1.0
    Paul> yet. Maybe he hasn't got around to choosing an XML parser.

Or maybe he has a different set of constraints than you.

    Paul> I'm trying to point out (more to you, than to him!) that there is
    Paul> a good reason to build on the work other people have done. If
    Paul> pyxmlrpc is faster today it is probably because it doesn't conform
    Paul> to the specs.  When it does conform, it won't be faster anymore.

Why point this out to me?  I am essentially just an XML-RPC user, not an
implementer.  I happen to be interested in making my XML-RPC-using code run
faster.  If I have to make some sacrifices I could care less, as long as my
clients and my servers can talk to one another.

    >> As for handling things like CDATA, UTF-16 and extra whitespace after
    >> tag names, I suspect some other XML-RPC packages would exhibit
    >> similar problems if they were exposed to a standards-toting XML
    >> gunslinger like yourself.  That it's not a problem in practice is
    >> probably because the set of XML-RPC encoding and decoding software is
    >> fairly small and that the stuff that encodes into XML-RPC is fairly
    >> well-behaved.

    Paul> Every XML-RPC implementation I have ever used (Python, Perl, C,
    Paul> C++, PHP) is based upon one pure XML parser or another. Most use
    Paul> Expat.

Oh well.

S


From skip@pobox.com (Skip Montanaro)  Wed Oct  3 03:23:31 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 2 Oct 2001 21:23:31 -0500
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: <3BBA708F.D8554BBC@ActiveState.com>
References: <200110030128.f931Sr005236@jericho.sourcelight.com>
 <3BBA708F.D8554BBC@ActiveState.com>
Message-ID: <15290.30371.513079.764350@beluga.mojam.com>

    Paul> I think that if a spec claims to be based on XML and does not
    Paul> explicitly disclaim support for built-in XML features, then it
    Paul> allows them. For instance if it doesn't say that &#67; syntax is
    Paul> illegal, then there is no reason to believe it is.

Paul, 

You probably know as well as anyone that the one and only person you should
talking to about XML-RPC and its XML compliance (or lack thereof) is Dave
Winer.  Feel free to read through the archives of the xmlrpc@yahoogroups.com
mailing list if you haven't already.  If you can move Dave from his current
position, more power to you.  You'll do something that many other people
have been incapable of doing.

I'm done with this topic.  It's gotten way too far from python-dev-related
topics.  Probably should have cut it out of the cc list awhile back.

Skip


From guido@python.org  Wed Oct  3 04:05:32 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 02 Oct 2001 23:05:32 -0400
Subject: [Python-Dev] Performance of various marshallers
In-Reply-To: Your message of "Tue, 02 Oct 2001 21:23:31 CDT."
 <15290.30371.513079.764350@beluga.mojam.com>
References: <200110030128.f931Sr005236@jericho.sourcelight.com> <3BBA708F.D8554BBC@ActiveState.com>
 <15290.30371.513079.764350@beluga.mojam.com>
Message-ID: <200110030305.XAA26130@cj20424-a.reston1.va.home.com>

> I'm done with this topic.  It's gotten way too far from python-dev-related
> topics.  Probably should have cut it out of the cc list awhile back.

Amen.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From timo@alum.mit.edu  Wed Oct  3 06:18:36 2001
From: timo@alum.mit.edu (Timothy O'Malley)
Date: Wed, 3 Oct 2001 01:18:36 -0400
Subject: [Python-Dev] Re: Cookie.py and `:' in the key
In-Reply-To: <15290.27704.19609.960912@anthem.wooz.org>
Message-ID: <1890D9DE-B7BE-11D5-B9BF-00306586E61E@alum.mit.edu>

hola.

On Tuesday, October 2, 2001, at 09:39 PM, Barry A. Warsaw wrote:
> In Mailman, I use a version of Cookie.py written by Timothy dated from
> 1998.  I'm now trying to see if I can get rid of my independent
> copy and just use Cookie.py in the Python 2.x standard library.

According to a very strict reading of the appropriate
specifications (RFC 2109 for cookies, which in turn
references terms defined in RFC 2068 for HTTP), a colon
is not legal in a value unless it is in a quoted value:

    Many HTTP/1.1 header field values consist of words separated by LWS
    or special characters. These special characters MUST be in a quoted
    string to be used within a parameter value.

           token          = 1*<any CHAR except CTLs or tspecials>

           tspecials      = "(" | ")" | "<" | ">" | "@"
                          | "," | ";" | ":" | "\" | <">
                          | "/" | "[" | "]" | "?" | "="
                          | "{" | "}" | SP | HT


Even so, I think I agree that a strict interpretation isn't very
useful in practice.  In this case, for instance, the intended
value is clear and obvious.

I've gone back and forth on this -- should the implementation
be true to the spec or should it follow its own rules for clear
and obvious?

Should people desire the "clear and obvious" over the "strict
interpretation", I think your fix is dead on.

TimO




From skip@pobox.com (Skip Montanaro)  Wed Oct  3 08:40:28 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 3 Oct 2001 02:40:28 -0500
Subject: [Python-Dev] dummy __del__ in SocketServer.BaseRequestHandler
Message-ID: <15290.49388.86639.812781@beluga.mojam.com>

Given that the presence of a __del__ method keeps the garbage collector from
reclaiming cyclic garbage, should SocketServer.BaseRequestHandler define a
__del__ method that just executes "pass"?  Commenting it out seems to have
gotten rid of some "uncollectable" messages from the garbage collector.
Every other __del__ method in the top level of the standard library actually
does something.

Skip


From guido@python.org  Wed Oct  3 12:53:42 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 03 Oct 2001 07:53:42 -0400
Subject: [Python-Dev] dummy __del__ in SocketServer.BaseRequestHandler
In-Reply-To: Your message of "Wed, 03 Oct 2001 02:40:28 CDT."
 <15290.49388.86639.812781@beluga.mojam.com>
References: <15290.49388.86639.812781@beluga.mojam.com>
Message-ID: <200110031153.HAA27628@cj20424-a.reston1.va.home.com>

> Given that the presence of a __del__ method keeps the garbage collector from
> reclaiming cyclic garbage, should SocketServer.BaseRequestHandler define a
> __del__ method that just executes "pass"?  Commenting it out seems to have
> gotten rid of some "uncollectable" messages from the garbage collector.
> Every other __del__ method in the top level of the standard library actually
> does something.

+1.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From skip@pobox.com (Skip Montanaro)  Wed Oct  3 20:18:05 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 3 Oct 2001 14:18:05 -0500
Subject: [Python-Dev] a bit confused about the new type/class stuff
Message-ID: <15291.25709.109749.875704@beluga.mojam.com>

I have a class hierarchy I'm trying to migrate to 2.2 using the latest stuff
from CVS on both the Gtk and Python sides of the fence.  Conceptually its
skeleton looks like

    class Object:
        ...

    class Widget(Object):
        ...

    class Button(Widget):
        ...

(It's a bunch of wrappers around Gtk widgets.  The wrappers happen to use
delegation instead of instantiation, so my Button class is not subclassed
from gtk.Button.  In particular, it's *not* one of the new subclassable
types.)

At one point in my code I test to see if one of my Button instances is in a
list of Widget instances and I get a TypeError:

    TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance'

If I test for b's inclusion in l, even if I draw b from l:

    >>> l
    [<Button object (GtkButton) at 0x8293134>, <Button object (GtkButton) at 0x8296ac4>, <Button object (GtkButton) at 0x8397fb4>]
    >>> b = l[0]
    >>> b in l
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance'

I get an error.

I don't define a __cmp__ method for the Button class or any of its
ancestors, but if I print b.__cmp__ something is definitely there:

    >>> b.__cmp__
    <method-wrapper object at 0x83aae78>

This was all working using Python 2.1.  I haven't changed anything so far in
my code other than to adapt to the new PyGtk naming scheme (e.g., gtk.Button
vs. gtk.GtkButton) and recompile/install Gtk, PyGtk, Python and friends.  In
particular, I believe I have a hierarchy of so-called "classic" classes.

I'm going to toddle off an reread the various PEPs, but would appreciate an
insight or two on where this error is coming from.

Thx,

-- 
Skip Montanaro (skip@pobox.com)
http://www.mojam.com/
http://www.musi-cal.com/


From skip@pobox.com (Skip Montanaro)  Wed Oct  3 21:05:32 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 3 Oct 2001 15:05:32 -0500
Subject: [Python-Dev] Re: [pygtk] a bit confused about the new type/class stuff
In-Reply-To: <15291.25709.109749.875704@beluga.mojam.com>
References: <15291.25709.109749.875704@beluga.mojam.com>
Message-ID: <15291.28556.67625.136475@beluga.mojam.com>

    ...
    Skip> (It's a bunch of wrappers around Gtk widgets.  The wrappers happen
    Skip> to use delegation instead of instantiation, so my Button class is
    Skip> not subclassed from gtk.Button.  In particular, it's *not* one of
    Skip> the new subclassable types.)

God, I was up way too late last night.  Make that "delegation instead of
inheritance". 

still-stumped-in-evanston-ly, y'rs,

Skip


From gward@python.net  Wed Oct  3 23:13:04 2001
From: gward@python.net (Greg Ward)
Date: Wed, 3 Oct 2001 18:13:04 -0400
Subject: [Python-Dev] Making setup.py a little more flexible
Message-ID: <20011003181304.A17246@cthulhu.mems-exchange.org>

Some months ago, I tried to build the latest Python on an IRIX 6.5
system that has been, shall we say, lightly maintained for some time.
(Basically, its /usr/local/ is mounted from an IRIX 5 system, and it's
full of libraries that you can't link against with the default compiler
flags on IRIX 6.)

Unfortunately, Python's setup.py is a bit too rigid: it assumes that it
should always add /usr/local/{lib,include} to the lib/include search
paths, it assumes it should always search /usr/local/lib (and a few
others) for SSL libraries, etc.  Worse, it assumes the existence of
libssl.a implies the linkability of -lssl -- when /usr/local is broken,
this ain't necessarily so.

Now I know what the Right Answer is: finish implementing the Distutils
"config" command, and use it to properly probe the system for includable
header files and linkable libraries, rather than just assuming existence
== validity.

That would be nice.  But I've got a quick 'n easy fix -- the moral
equivalent of commenting out the code in PyBuildExt.detect_modules()
that makes those blithe assumptions.  Namely, I added --without-usrlocal
and --without-ssl options to the custom build_ext command used by
setup.py.

Now, I can build most extensions just fine on this broken IRIX system
like this:
  ./python setup.py build_ext --without-ssl --without-usrlocal

I've just uploaded patch #467650 (assigned to Andrew) to implement this:
  http://sourceforge.net/tracker/index.php?func=detail&aid=467650&group_id=5470&atid=305470

Whaddya think?  Good enough bandaid solution?  Or should I add a few
more things, eg.:
  --with-ssl-lib=DIR     directory to search for OpenSSL libs
  --with-ssl-inc=DIR     directory to search for OpenSSL headers
  --without-foo          don't try to build foo

The last one could be repeated for various values of "foo": curses, dbm,
bsddb, etc.  I don't think that's too necessary; if some foo doesn't
build, then it doesn't build and that's that.  From my cursory
examination of detect_modules(), the _socket/OpenSSL dependency is the
only obvious module-killer.  (That is, the bogus OpenSSL libs killed the
_socket module completely, when all I really need to do is not use
OpenSSL on this system.)

        Greg
-- 
Greg Ward - Python bigot                                gward@python.net
http://starship.python.net/~gward/
I haven't lost my mind; I know exactly where I left it.


From gward@python.net  Thu Oct  4 02:16:35 2001
From: gward@python.net (Greg Ward)
Date: Wed, 3 Oct 2001 21:16:35 -0400
Subject: [Python-Dev] Fixing compiler warnings (bug #445960)
Message-ID: <01100321143900.18648@cthulhu>

What's the general feeling/policy on fixing very minor problems that cause 
legitimate compiler warnings with some compilers?  ISTR that Tim is adamant 
about keeping Python warning-free on Windows with MSVC, but what about, say, 
the compiler that ships with IRIX 6.5?  It's not a widely-used platform, but 
the compiler does catch some genuine, albeit small, problems -- mostly of the 
"variable set but never used" variety.

It looks like most fixes are along the lines of changing this:

static int
init_builtin(char *name)
{
	struct _inittab *p;
	PyObject *mod;

	if ((mod = _PyImport_FindExtension(name, name)) != NULL)
		return 1;

to this:

static int
init_builtin(char *name)
{
	struct _inittab *p;



From guido@python.org  Thu Oct  4 02:28:19 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 03 Oct 2001 21:28:19 -0400
Subject: [Python-Dev] a bit confused about the new type/class stuff
In-Reply-To: Your message of "Wed, 03 Oct 2001 14:18:05 CDT."
 <15291.25709.109749.875704@beluga.mojam.com>
References: <15291.25709.109749.875704@beluga.mojam.com>
Message-ID: <200110040128.VAA32301@cj20424-a.reston1.va.home.com>

> I have a class hierarchy I'm trying to migrate to 2.2 using the latest stuff
> from CVS on both the Gtk and Python sides of the fence.  Conceptually its
> skeleton looks like
> 
>     class Object:
>         ...
> 
>     class Widget(Object):
>         ...
> 
>     class Button(Widget):
>         ...
> 
> (It's a bunch of wrappers around Gtk widgets.  The wrappers happen to use
> delegation instead of instantiation, so my Button class is not subclassed
> from gtk.Button.  In particular, it's *not* one of the new subclassable
> types.)
> 
> At one point in my code I test to see if one of my Button instances is in a
> list of Widget instances and I get a TypeError:
> 
>     TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance'

This error comes from wrap_cmpfunc() in typeobject.c.  It should only
happen when you are somehow comparing new-style objects.

> If I test for b's inclusion in l, even if I draw b from l:
> 
>     >>> l
>     [<Button object (GtkButton) at 0x8293134>, <Button object (GtkButton) at 0x8296ac4>, <Button object (GtkButton) at 0x8397fb4>]
>     >>> b = l[0]
>     >>> b in l
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance'
> 
> I get an error.

What do you get when you ask for (b == l)?

> I don't define a __cmp__ method for the Button class or any of its
> ancestors, but if I print b.__cmp__ something is definitely there:
> 
>     >>> b.__cmp__
>     <method-wrapper object at 0x83aae78>

That's truly bizarre.  A <method-wrapper object> is an *unbound*
__cmp__ method that wraps the tp_compare slot of a built-in type.  I
have no explanation for how it would show up in your instance.

Can you check whether b.__cmp__ is the same as b.__dict__['__cmp__']
or whether it is the same as b.__class__.__cmp__?  It should be either
one of those.  Then, if it's in b.__class__, try to find out in which
class's __dict__ it lives.

> This was all working using Python 2.1.  I haven't changed anything so far in
> my code other than to adapt to the new PyGtk naming scheme (e.g., gtk.Button
> vs. gtk.GtkButton) and recompile/install Gtk, PyGtk, Python and friends.  In
> particular, I believe I have a hierarchy of so-called "classic" classes.
> 
> I'm going to toddle off an reread the various PEPs, but would appreciate an
> insight or two on where this error is coming from.

I have zero experience with Gtk, but I heard somewhere that it uses
the Don Beaudry hook.  This is enabled in recent 2.2, but it might act
slightly different than expected.  This is the first thing I'd
investigate.

Good luck!  (If you can mail me step-by-step instructions for
reproducing your situation on Linux, including URLs of specific Gtk
source tarballs etc., I might play with this a bit...)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Thu Oct  4 03:04:38 2001
From: tim.one@home.com (Tim Peters)
Date: Wed, 3 Oct 2001 22:04:38 -0400
Subject: [Python-Dev] Fixing compiler warnings (bug #445960)
In-Reply-To: <01100321143900.18648@cthulhu>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEDBLPAA.tim.one@home.com>

[Greg Ward]
> What's the general feeling/policy on fixing very minor problems
> that cause legitimate compiler warnings with some compilers?

If they're truly problems, they should be fixed.

> ISTR that Tim is adamant about keeping Python warning-free on Windows
> with MSVC,

Yes, but note that MSVC doesn't complain about non-problems <0.9 wink -- it
actually does, but not at the wng level we use there>.

> but what about, say, the compiler that ships with IRIX 6.5?

Doesn't have to do with the compiler so much as the specifics of what it's
complaining about.  MSVC and gcc and XXX catch different things, and that's
valuable.

> It's not a widely-used platform, but the compiler does catch some
> genuine, albeit small, problems -- mostly of the "variable set but
> never used" variety.  It looks like most fixes are along the lines of
> changing this:
>
> static int
> init_builtin(char *name)
> {
> 	struct _inittab *p;
> 	PyObject *mod;
>
> 	if ((mod = _PyImport_FindExtension(name, name)) != NULL)
> 		return 1;
>
> to this:
>
> static int
> init_builtin(char *name)
> {
> 	struct _inittab *p;

Seems to be missing a line; I assume

	if (_PImport_FindExtension(name, name) != NULL)
		return 1;

was intended.  I'd make that change, as there's no downside.

Examples of changes I wouldn't make:

1. A compiler warning when an __inline function isn't inlined.

2. A compiler complaining about a vrbl possibly used before
   initialization, when it's instantly obvious that the vrbl is
   initialized on all paths before use (got a report like that
   with two of those yesterday -- won't "fix").

3. Often (but not always) a complaint that an assignment is stomped on
   before use.  This one I'd do something about:

     a = 1;  /* complaint about this */
     a = 2;

   This one I wouldn't:

     PyObject *result = NULL;  /* complaint about this */

     /* 300 lines of excruciating code, every one of the 116 paths
        through which just happens to set result before it's used.
     */

     return result;

A special problem with #2 and #3 is that shutting up a compiler that
complains about one of them often creates code that another compiler will
complain about for the other reason.  That's one reason "no warning from any
compiler" is impossible to achieve.



From skip@pobox.com (Skip Montanaro)  Thu Oct  4 03:07:40 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 3 Oct 2001 21:07:40 -0500
Subject: [Python-Dev] a bit confused about the new type/class stuff
In-Reply-To: <200110040128.VAA32301@cj20424-a.reston1.va.home.com>
References: <15291.25709.109749.875704@beluga.mojam.com>
 <200110040128.VAA32301@cj20424-a.reston1.va.home.com>
Message-ID: <15291.50284.699868.549699@beluga.mojam.com>

Guido,

Thanks for the various tips.  I will check 'em out.  Regarding the Don
Beaudry hook, PyGtk was using ExtensionClass until recently.  James
Henstridge has just released the first cut at using your new type/class
stuff in the past few days.  So if you didn't use it, neither does PyGtk.

    Guido> I have zero experience with Gtk, but I heard somewhere that it
    Guido> uses the Don Beaudry hook.  This is enabled in recent 2.2, but it
    Guido> might act slightly different than expected.  This is the first
    Guido> thing I'd investigate.

    Guido> Good luck!  (If you can mail me step-by-step instructions for
    Guido> reproducing your situation on Linux, including URLs of specific
    Guido> Gtk source tarballs etc., I might play with this a bit...)

This is too new to have tarballs yet (lots of hairballs perhaps...).
Everything is coming out of various CVS repositories.  If you want to waste
a half a day getting all the Gtk build tools and libraries correct, I can
set you up, but I suspect you may have better things to do with your time.

An alternative is for me to give you ssh access to my laptop.  You could be
fiddling around with this stuff in a few minutes.  Even if it wasn't useful
to track my problem down you might like kicking the tires on a pretty large
client for your new type/class code.

Skip




From mal@lemburg.com  Thu Oct  4 09:44:28 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 04 Oct 2001 10:44:28 +0200
Subject: [Python-Dev] Making setup.py a little more flexible
References: <20011003181304.A17246@cthulhu.mems-exchange.org>
Message-ID: <3BBC216C.C38AC411@lemburg.com>

Greg Ward wrote:
> 
> Some months ago, I tried to build the latest Python on an IRIX 6.5
> system that has been, shall we say, lightly maintained for some time.
> (Basically, its /usr/local/ is mounted from an IRIX 5 system, and it's
> full of libraries that you can't link against with the default compiler
> flags on IRIX 6.)
> 
> Unfortunately, Python's setup.py is a bit too rigid: it assumes that it
> should always add /usr/local/{lib,include} to the lib/include search
> paths, it assumes it should always search /usr/local/lib (and a few
> others) for SSL libraries, etc.  Worse, it assumes the existence of
> libssl.a implies the linkability of -lssl -- when /usr/local is broken,
> this ain't necessarily so.
> 
> Now I know what the Right Answer is: finish implementing the Distutils
> "config" command, and use it to properly probe the system for includable
> header files and linkable libraries, rather than just assuming existence
> == validity.

> That would be nice.
That would certainly be nice indeed (for other users of distutils as 
well ;-).
 
> But I've got a quick 'n easy fix -- the moral
> equivalent of commenting out the code in PyBuildExt.detect_modules()
> that makes those blithe assumptions.  Namely, I added --without-usrlocal
> and --without-ssl options to the custom build_ext command used by
> setup.py.
> 
> Now, I can build most extensions just fine on this broken IRIX system
> like this:
>   ./python setup.py build_ext --without-ssl --without-usrlocal
> 
> I've just uploaded patch #467650 (assigned to Andrew) to implement this:
>   http://sourceforge.net/tracker/index.php?func=detail&aid=467650&group_id=5470&atid=305470
> 
> Whaddya think?  Good enough bandaid solution?  Or should I add a few
> more things, eg.:
>   --with-ssl-lib=DIR     directory to search for OpenSSL libs
>   --with-ssl-inc=DIR     directory to search for OpenSSL headers
>   --without-foo          don't try to build foo

I think that these options are more in line with what configure
scripts tend to offer: if the magic doesn't work, let the user
decide.
 
> The last one could be repeated for various values of "foo": curses, dbm,
> bsddb, etc.  I don't think that's too necessary; if some foo doesn't
> build, then it doesn't build and that's that.  From my cursory
> examination of detect_modules(), the _socket/OpenSSL dependency is the
> only obvious module-killer.  (That is, the bogus OpenSSL libs killed the
> _socket module completely, when all I really need to do is not use
> OpenSSL on this system.)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From mal@lemburg.com  Thu Oct  4 09:48:01 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 04 Oct 2001 10:48:01 +0200
Subject: [Python-Dev] Fixing compiler warnings (bug #445960)
References: <01100321143900.18648@cthulhu>
Message-ID: <3BBC2241.7BDFB9B1@lemburg.com>

Greg Ward wrote:
> 
> What's the general feeling/policy on fixing very minor problems that cause
> legitimate compiler warnings with some compilers?  ISTR that Tim is adamant
> about keeping Python warning-free on Windows with MSVC, but what about, say,
> the compiler that ships with IRIX 6.5?  It's not a widely-used platform, but
> the compiler does catch some genuine, albeit small, problems -- mostly of the
> "variable set but never used" variety.
> 
> It looks like most fixes are along the lines of changing this:
> 
> static int
> init_builtin(char *name)
> {
>         struct _inittab *p;
>         PyObject *mod;
> 
>         if ((mod = _PyImport_FindExtension(name, name)) != NULL)
>                 return 1;
> 
> to this:
> 
> static int
> init_builtin(char *name)
> {
>         struct _inittab *p;
>
>         if (_PyImport_FindExtension(name, name) != NULL)
>                 return 1;

If it improves code quality, I'd say that any valid code warning should
be considered and fixed (doesn't really matter what the source 
is ;-).

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From guido@python.org  Thu Oct  4 11:29:18 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 04 Oct 2001 06:29:18 -0400
Subject: [Python-Dev] a bit confused about the new type/class stuff
In-Reply-To: Your message of "Wed, 03 Oct 2001 21:07:40 CDT."
 <15291.50284.699868.549699@beluga.mojam.com>
References: <15291.25709.109749.875704@beluga.mojam.com> <200110040128.VAA32301@cj20424-a.reston1.va.home.com>
 <15291.50284.699868.549699@beluga.mojam.com>
Message-ID: <200110041029.GAA01791@cj20424-a.reston1.va.home.com>

> Thanks for the various tips.  I will check 'em out.  Regarding the Don
> Beaudry hook, PyGtk was using ExtensionClass until recently.  James
> Henstridge has just released the first cut at using your new type/class
> stuff in the past few days.  So if you didn't use it, neither does PyGtk.

Ah, but then it could be that somehow this cutting-edge PyGtk has
caused your immediate problem, right?  I still take responsibility --
the new type stuff isn't documented enough yet. :-(

> This is too new to have tarballs yet (lots of hairballs perhaps...).
> Everything is coming out of various CVS repositories.  If you want to waste
> a half a day getting all the Gtk build tools and libraries correct, I can
> set you up, but I suspect you may have better things to do with your time.

Yup.

> An alternative is for me to give you ssh access to my laptop.  You could be
> fiddling around with this stuff in a few minutes.  Even if it wasn't useful
> to track my problem down you might like kicking the tires on a pretty large
> client for your new type/class code.

If you mail me your phone number in private I'll call you to
coordinate this more, when I have time.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From james@daa.com.au  Thu Oct  4 13:26:45 2001
From: james@daa.com.au (James Henstridge)
Date: Thu, 4 Oct 2001 20:26:45 +0800 (WST)
Subject: [pygtk] Re: [Python-Dev] a bit confused about the new type/class
 stuff
In-Reply-To: <200110041029.GAA01791@cj20424-a.reston1.va.home.com>
Message-ID: <Pine.LNX.4.33.0110042015050.7347-100000@james.daa.com.au>

On Thu, 4 Oct 2001, Guido van Rossum wrote:

> > Thanks for the various tips.  I will check 'em out.  Regarding the Don
> > Beaudry hook, PyGtk was using ExtensionClass until recently.  James
> > Henstridge has just released the first cut at using your new type/class
> > stuff in the past few days.  So if you didn't use it, neither does PyGtk.
>
> Ah, but then it could be that somehow this cutting-edge PyGtk has
> caused your immediate problem, right?  I still take responsibility --
> the new type stuff isn't documented enough yet. :-(

I think I got everything right as far as the new type code goes (and it
works in most cases.  For reference, the gtk.Button instance is a new
style type (subclassed from other new style types implemented in C, single
inheritance).  The base class (GObject) implements the tp_compare slot in
the type object, and doesn't implement rich comparisons:

  >>> import gtk
  >>> gtk.Button.__cmp__
  <slot wrapper '__cmp__' of 'GObject' objects>
  >>> b = gtk.Button()
  >>> b.__cmp__
  <method-wrapper object at 0x8268df8>

I don't think I am doing anything weird in the tp_compare slot (although
the function would only handle GObject instances).

>
> > This is too new to have tarballs yet (lots of hairballs perhaps...).
> > Everything is coming out of various CVS repositories.  If you want to waste
> > a half a day getting all the Gtk build tools and libraries correct, I can
> > set you up, but I suspect you may have better things to do with your time.
>
> Yup.

Well, the 1.3.9 releases of glib and gtk are usable.  The last development
release of pygtk (1.99.2) had some issues, as it was released just after I
merged the branch where I ported things to 2.2.  I will put out a 1.99.3
release soon which should work a bit better.

James.

-- 
Email: james@daa.com.au
WWW:   http://www.daa.com.au/~james/




From skip@pobox.com (Skip Montanaro)  Thu Oct  4 14:41:48 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Thu, 4 Oct 2001 08:41:48 -0500
Subject: [pygtk] Re: [Python-Dev] a bit confused about the new type/class stuff
In-Reply-To: <200110040128.VAA32301@cj20424-a.reston1.va.home.com>
References: <15291.25709.109749.875704@beluga.mojam.com>
 <200110040128.VAA32301@cj20424-a.reston1.va.home.com>
Message-ID: <15292.26396.278940.686089@beluga.mojam.com>

    >> >>> l
    >> [<Button object (GtkButton) at 0x8293134>, <Button object (GtkButton) at 0x8296ac4>, <Button object (GtkButton) at 0x8397fb4>]
    >> >>> b = l[0]
    >> >>> b in l
    >> Traceback (most recent call last):
    >>   File "<stdin>", line 1, in ?
    >> TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance'

A brief conversation with Guido cleared up what was happening.  In my class
hierarchy I didn't implement __cmp__, but I did implement __getattr__, which
was delegating most things to the underlying Gtk widget, so 

    b.__cmp__(l)

became

    b._real_widget.__cmp(l)

which then barfed.

Writing a __cmp__ method that does the right thing cleared up the problem.

Skip


From aahz@rahul.net  Thu Oct  4 16:02:05 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Thu, 4 Oct 2001 08:02:05 -0700 (PDT)
Subject: [Python-Dev] Backward compatibility
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIELHLOAA.tim.one@home.com> from "Tim Peters" at Oct 01, 2001 02:47:04 PM
Message-ID: <20011004150205.9A529E906@waltz.rahul.net>

Tim Peters wrote:
> 
> I pay no attention to "backwards compatibility" issues in the libraries we
> ship, and have Guido's blessing for that ornery attitude:  the libs we ship
> should do the best possible job in the context of the release they're
> shipped in.  It's explicitly not a PythonLabs goal that people be able to
> pick modules out of a release and use them in an earlier release (although
> it may be a goal of some developers for some modules -- but then the burden
> is on them to ensure it works for the module and release combinations they
> care about).

As one of the frequent ranters about compatibility issues, I thoroughly
endorse this approach WRT "batteries included" as long as the APIs stay
the same.
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From aahz@rahul.net  Thu Oct  4 16:32:54 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Thu, 4 Oct 2001 08:32:54 -0700 (PDT)
Subject: [Python-Dev] Pmw broken under 2.2a4?
In-Reply-To: <000101c14b8f$db4938a0$3d01a8c0@plstn1.sfba.home.com> from "Paul F. Dubois" at Oct 02, 2001 03:16:16 PM
Message-ID: <20011004153254.CD682E906@waltz.rahul.net>

Paul F. Dubois wrote:
> 
> Building Pmw 0.8.5 under 2.2a4 I got syntax errors of the 'inconsistent
> use of tabs and spaces' kind. I'm not sure who I should tell but I
> speculate that I just did.

Try http://pmw.sourceforge.net/
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From loewis@informatik.hu-berlin.de  Thu Oct  4 18:12:51 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Thu, 4 Oct 2001 19:12:51 +0200 (MEST)
Subject: [Python-Dev] Making setup.py a little more flexible
Message-ID: <200110041712.TAA25396@pandora.informatik.hu-berlin.de>

> Whaddya think?  Good enough bandaid solution?

-1. First of all, a bandaid for the specific problem is already in place:
you can configure everything through Modules/Setup, in particular if you
find that setup.py doesn't do the right thing. Modules/Setup allow much
more detailed changes than what your changes to setup.py offers, plus it
is an established solution to this problem.

Furthermore, your specific patch seems incomplete: You have to run setup.py
manually, which means that you cannot use a plain "make". There should be
atleast minimal interfacing to Makefile.pre.in for this patch to be acceptable.

Regards,
Martin



From walter@livinglogic.de  Thu Oct  4 19:22:01 2001
From: walter@livinglogic.de (Walter =?ISO-8859-1?Q?D=F6rwald?=)
Date: Thu, 04 Oct 2001 20:22:01 +0200
Subject: [Python-Dev] Proposed PEP: Codec error handling callbacks
Message-ID: <3BBCA8C9.7050901@livinglogic.de>

Here's a first try at a PEP for making Python's codec error
handling customizable. This has been briefly discussed before
on the I18N-SIG mailing list.

A sample implementation of the proposed feature is available
as a SourceForge patch.

Bye,
    Walter D=F6rwald

-------------------------------------------------------------------------=
--
PEP: ???
Title: Codec error handling callbacks
Version: $Revision$
Last-Modified: $Date$
Author: walter@livinglogic.de (Walter D=F6rwald)
Status: Draft
Type: Standards Track
Python-Version: 2.3
Created: ??-???-2001
Post-History: 04-Oct-2001


Abstract

     This PEP aims at extending Python's fixed codec error handling
     schemes with a more flexible callback based approach.

     Python currently uses a fixed error handling for codec error
     handlers. This PEP describes a mechanism which allows Python to
     use function callbacks as error handlers.  With these more flexible
     error handlers it is possible to add new functionality to existing
     codecs by e.g. providing fallback solutions or different encodings
     for cases where the standard codec mapping does not apply.


Problem

     A typical case is where a character encoding does not support
     the full range of Unicode characters.  For these cases many high
     level protocols support a way of escaping a Unicode character
     (e.g. Python itself support the \x, \u and \U convention, XML
     supports character references via &#xxxx; etc.).

     When implementing such an encoding algorithm, a problem with the
     current implementation of the encode method of Unicode objects
     becomes apparent: For determining which characters are unencodable
     by a certain encoding, every single character has to be tried,
     because encode does not provide any information about the location
     of the error(s), so

         # (1)
         us =3D u"xxx"
         s =3D us.encode(encoding)

     has to be replaced by

         # (2)
         us =3D u"xxx"
         v =3D []
         for c in us:
             try:
                 v.append(c.encode(encoding))
             except UnicodeError:
                 v.append("&#" + ord(c) + ";")
         s =3D "".join(v)

     This slows down encoding dramatically as now the loop through
     the string is done in Python code and no longer in C code.

     Furthermore this solution poses problems with stateful encodings.
     For example UTF16 uses a Byte Order Mark at the start of the
     encoded byte string to specify the byte order.  Using (2) with
     UTF16, results in an 8 bit string with a BOM between every
     character.

     To work around this problem, a stream writer - which keeps
     state between calls to the encoding function - has to be used:

         # (3)
         us =3D u"xxx"
         import codecs, cStringIO as StringIO
         writer =3D codecs.lookup(encoding)[3]

         v =3D StringIO.StringIO()
         uv =3D writer(v)
         for c in us:
             try:
                 uv.write(c)
             except UnicodeError:
                 uv.write(u"&#%d;" % ord(c))
         s =3D str(v)

     To compare the speed of (1) and (3) the following test script
     has been used:

         # (4)
         import time
         us =3D u"=E4a"*1000000
         encoding =3D "ascii"
         import codecs, cStringIO as StringIO

         t1 =3D time.time()

         s1 =3D us.encode(encoding, "replace")

         t2 =3D time.time()

         writer =3D codecs.lookup(encoding)[3]

         v =3D StringIO.StringIO()
         uv =3D writer(v)
         for c in us:
             try:
                 uv.write(c)
             except UnicodeError:
                 uv.write(u"?")
         s2 =3D v.getvalue()

         t3 =3D time.time()

         assert(s1=3D=3Ds2)
         print "1:", t2-t1
         print "2:", t3-t2
         print "factor:", (t3-t2)/(t2-t1)

     On Linux with Python 2.1 this gives the
     following output:
         1: 0.395456075668
         2: 126.909595966
         factor: 320.919575586
     i.e. (3) is 320 times slower than (1).


Solution 1

     Add a position attribute to UnicodeError instances.

     When the encoder encounters an unencodable character
     it raises an exception that specifies the position in the
     Unicode object where the unencodable character occurs.
     The application can then reencode the Unicode object up
     to the offending character, replace the offending character
     with something appropriate and retry encoding with the
     rest of the Unicode string until encoding is finished.

     A stream writer will write everything up to the offending
     character and then raise an exception, so the application
     only has to replace the offending character and retry the rest
     of the string.

     Advantage

     Requires minor changes to all the encoders/stream writers.

     Disadvantage

     As the encoder has to be called multiple times, this
     won't work with stateful encoding, so a stream writer
     has to be used in all cases.

     If unencodable characters occur very often Solution 1
     will probably not be much faster than (3).  E.g. for the
     string u"a"*10000+u"=E4" all the characters but the last one
     will have to be encoded twice when using an encoder
     (but only once when using a stream writer).

     This solution is specific to encoding and can't
     be extended to decoding easily.


Solution 2

     Add additional error handling names for every needed
     replacement scheme (e.g. "xmlcharrefreplace" for "&#%d;"
     or "escapereplace" for "\\x%02x" / "\\u%04x" / "\\U%08x")

     Advantage

     Requires minor changes to all encoders/stream writers.

     As the replacement scheme is implemented directly in
     the encoder this should be the fastest solution.

     Disadvantages

     The available replacement schemes are hardwired.
     Additional replacement schemes require that all
     encoders/decoders are changed again.  This is especially
     problematic for decoding where users might want to
     implement various error handlers for handling broken
     text files.


Solution 3

     The errors argument is no longer a string, but a callback
     function: This callback function gets passed the original
     unicode object and the position of the unencodable character
     and returns a new unicode object that should be encoded instead
     of the unencodable one. (Note that this requires that the encoder
     *must* be able to encode what is returned from the handler.  If
     not a normal UnicodeError will be raised.)

     Example code could look like this:

         us =3D u"xxx"

         def xmlescape(uni, pos):
             return u"&#%d;" % ord(uni[pos])

         s =3D us.encode(encode, xmlescape)

     Advantages

     This makes the error handling completely customizable. The error
     handler may choose to raise an exception or do any kind of
     replacement required.

     The interface between the encoder and the error handler can be
     designed in a way that this mechanism can be used for decoding too:
     The original 8bit string is passed to the error handler and the
     error handler returns a replacement unicode object and a
     resyncronization position where decoding should continue.

     Disadvantages

     This solutions requires changes to every C function
     that has "const char *errors" arguments. (e.g.

         PyUnicode_EncodeLatin1(
             const Py_UNICODE *p,
             int size,
             const char *errors)

     has to be changed to

         PyUnicode_EncodeLatin1(
             const Py_UNICODE *p,
             int size,
             PyObject *errors)

     (To provide backwards compatibility the PyUnicode_EncodeLatin1
     signature remains the same, a new function
     PyUnicode_EncodeLatin1Ex is introduced with the new
     signature.  PyUnicode_EncodeLatin1 simply calls the new function.)


Solution 4

     The errors argument is still a string, but this string
     is used to lookup a error handling callback function
     from a callback registry.

     Advantage

     No changes to the Python C API are required.  Well known
     error handling schemes could be implemented directly in
     the encoder/decoder for maximum speed.  Like solution 3
     this can be done for encoders and decoders.

     Disadvantages

     Currently all encoding/decoding functions have arguments
         const Py_UNICODE *p, int size
     or
         const char *p, int size
     to specify the unicode characters/8bit characters to be
     encoded/decoded.  In case of a error a new unicode or str
     object has to be created from this information and passed
     to the error handler.  This has to be done either for every
     occuring error or once on the first error and the result object
     must be kept until the end of the function in case another
     error occurs. Most functions that call the codec functions
     work with unicode/str objects anyway, so they have to extract
     the const Py_UNICODE */int arguments and pass it to the functions,
     which has to reconstruct the object in case of an error.


Sample implementation

     A sample implementation is available on SourceForge [1].  This
     patch implements a combination of solutions 3 and 4, i.e. it
     is possible to pass functions *and* registered callback names
     to unicode.encode.

     The interface between the encoder and the handler has been
     extended to be able to support the same interface for encoding
     and decoding.  The encoder/decoder passes a tuple to the callback
     with the following entries:
         0: the name of the encoding
         1: the original Unicode object/the original 8bit string
         2: the position of the unencodable character/byte
         3: the reason why the character/byte can't be
            encoded/decoded as a string (e.g.
            "character not in range(128)" for encoding or
            "unexpected end of data", "invalid code byte" etc.
            for decoding)
         4: an additional object that can be used to pass
            state information to the callback. All implemented
            encoders/decoders currently pass None for this.

     The callback must return a tuple with the following info:

         0: a Unicode object which will be encoded instead
            of the offending character (for encoders).
            A Unicode object that will be used as a replacement
            for the undecodable bytes in the decoded Unicode
            object (for decoders)
         1: a position where the encoding/decoding process should
            continue after processing the replacement string.

     The patch includes several preregistered encode
     error handlers schemes with the following names:
         "strict", "ignore", "replace", "xmlcharrefreplace",
         "escapereplace"
     and decode error handlers with the names:
         "strict", "ignore", "replace"

     The patch includes the other change to the C API described in
     Solution 4.  The parameters
         const Py_UNICODE *p, int size
     have been replace by
         PyObject *p
     so that all functions get the Unicode object directly
     as a PyObject * and pass this directly along to the error
     callback.

     For further details see the patch on SourceForge.


References

     [1] Python patch #432401 "unicode encoding error callbacks"
=20
http://sourceforge.net/tracker/?group_id=3D5470&atid=3D305470&func=3Ddeta=
il&aid=3D432401

     [2] Previous discussions on this topics in the I18N-SIG mailing list
         http://mail.python.org/pipermail/i18n-sig/2000-December/000587.h=
tml
         http://mail.python.org/pipermail/i18n-sig/2001-June/001173.html


Copyright

     This document has been placed in the public domain.


=0C
Local Variables:
mode: indented-text
indent-tabs-mode: nil
End:



From skip@pobox.com (Skip Montanaro)  Fri Oct  5 03:12:21 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Thu, 4 Oct 2001 21:12:21 -0500
Subject: [Python-Dev] Backward compatibility
Message-ID: <15293.5893.817970.754724@beluga.mojam.com>

    Aahz> As one of the frequent ranters about compatibility issues, I
    Aahz> thoroughly endorse this approach WRT "batteries included" as long
    Aahz> as the APIs stay the same.

I don't think APIs have much to do with this particular item.  No API
changed in xmlrpclib.py, but the code became incompatible with pre-2.0
releases.  I checked in a change to xmlrpclib.py that used "import foo as
bar".  The question I raised implicitly by making an incompatible change to
xmlrpclib.py was "is it okay to use all the features of the current release
in a core module?".  Tim's answer was a qualified "yes", the exception being
for modules maintained by other authors.  In PythonLabs view it's ultimately
Fredrik's responsibility to maintain backward compatibility if he so
chooses.  I realized it about 30 seconds after I checked in the change,
which was why I mentioned it.  I didn't know if he maintains a separate copy
for backwards compatible releases or relies on the code now in the Python
CVS repository.  In retrospect, I should probably have sent him a note
instead of checking in the change.

Skip



From ask@valueclick.com  Fri Oct  5 09:40:28 2001
From: ask@valueclick.com (Ask Bjoern Hansen)
Date: Fri, 5 Oct 2001 01:40:28 -0700 (PDT)
Subject: [Python-Dev] Crude Python->Parrot compiler
In-Reply-To: <200109261751.f8QHppn23941@odiug.digicool.com>
Message-ID: <Pine.LNX.4.33.0110050138460.4496-100000@impatience.valueclick.com>

On Wed, 26 Sep 2001, Guido van Rossum wrote:

> > Question for Guido and Simon: Should I check this into sandbox/ in the
> > Python CVS tree, or do you want to pull it into the Parrot CVS?  I'd
> > like to have it in a public CVS tree someplace, preferably one where I
> > have checkin privileges.  sandbox/ is fine with me.
>
> If Simon can give you write access in a corner of the Parrot tree,
> that might be the best solution; if not, sandbox is fine with me too.

I didn't see Simon follow up on this.  I would think that we can get
Andrew an account.  Andrew, sign up for an account at
http://dev.perl.org/auth/account, then if Simon will just say the
word I can flip the bits to give you commit access to
parrot/languages/python/ or something like that. :-)


 - ask

-- 
ask bjoern hansen, http://ask.netcetera.dk/         !try; do();
more than a billion impressions per week, http://valueclick.com




From thomas.heller@ion-tof.com  Fri Oct  5 10:58:12 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 11:58:12 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost>
Message-ID: <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook>

From: "Gordon McMillan" <gmcm@hypernet.com>
> Dev-er's
> 
> After numerous false-starts and dead-ends, I've come up with 
> two modules that do what imputil and modulefinder do, but 
> better.
> 
> Code and detailed descriptions are available here:
> 
> http://www.mcmillan-inc.com/importhooks.html
> 
Gordon, I've read your descriptions and IMO this is a great design
for a new import-utility and a new module-finder.

Some comments:
To fully simulate python's behaviour for import, the case of the
filename must be checked on windows. I found only two possibilities
to do this - the first one is to use os.listdir() which is probably 
expensive, but it returns filenames with the actual case. Second would
be to use Mark's win32api.FindFiles(), but this may not be available.

Your metapath models Python's import policy. One part of the policy
is that already loaded modules are fetched from sys.modules instead
of imported again. Should this behaviour also be modeled by a
SysModulesCacheDirector on your metapath?

What about reload? Shouldn't a new import-util module also implement
a reload-replacement?

> I would like to propose these (or something quite like them) as 
> replacements for the official versions. The code is quite similar 
> (in fact, the modulefinder code could have been written by 
> subclassing the imputil stuff, but I wrote them the other way 
> 'round). 
> 
It seems noone cares about this. imputil is in the distribution,
but is it really 'official'?

> If the charter of the Import-SIG is not as dead as the list is, I 
> would promote the basic structure as a potential model for a 
> reformed import.
> 
> For now, though, it's enough to consider the code. The 
> differences are too extreme to consider these patches, but the 
> subject hardly seems PEPable so I bring it up here.
> 
> 
> - Gordon
> 
Thomas



From simon@netthink.co.uk  Fri Oct  5 12:12:14 2001
From: simon@netthink.co.uk (Simon Cozens)
Date: Fri, 5 Oct 2001 12:12:14 +0100
Subject: [Python-Dev] Crude Python->Parrot compiler
In-Reply-To: <Pine.LNX.4.33.0110050138460.4496-100000@impatience.valueclick.com>
References: <200109261751.f8QHppn23941@odiug.digicool.com> <Pine.LNX.4.33.0110050138460.4496-100000@impatience.valueclick.com>
Message-ID: <20011005121214.B743@netthink.co.uk>

On Fri, Oct 05, 2001 at 01:40:28AM -0700, Ask Bjoern Hansen wrote:
> I didn't see Simon follow up on this.  I would think that we can get
> Andrew an account. 

I think we decided off-list that Guido would be OK with it going in
Python CVS for now.

Also, www.parrotcode.org

-- 
"He was a modest, good-humored boy.  It was Oxford that made him insufferable."


From gmcm@hypernet.com  Fri Oct  5 12:58:05 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Fri, 5 Oct 2001 07:58:05 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook>
Message-ID: <3BBD680D.8967.671A64B6@localhost>

Thomas,

> Some comments:
> To fully simulate python's behaviour for import, the case of the
> filename must be checked on windows.

True. Sigh.

> I found only two
> possibilities to do this - the first one is to use os.listdir()
> which is probably expensive, but it returns filenames with the
> actual case. Second would be to use Mark's win32api.FindFiles(),
> but this may not be available.
> 
> Your metapath models Python's import policy. One part of the
> policy is that already loaded modules are fetched from
> sys.modules instead of imported again. Should this behaviour also
> be modeled by a SysModulesCacheDirector on your metapath?

That's already done by the ImportManager as soon as he's got 
a candidate fqname.
 
> What about reload? Shouldn't a new import-util module also
> implement a reload-replacement?

Yes, it should.
 
> > I would like to propose these (or something quite like them) as
> > replacements for the official versions. The code is quite
> > similar (in fact, the modulefinder code could have been written
> > by subclassing the imputil stuff, but I wrote them the other
> > way 'round). 
> > 
> It seems noone cares about this. imputil is in the distribution,
> but is it really 'official'?

The entire subject seems to have dropped off radar, after 
starting out as highly controversial (the Import-SIG was started 
so the ihooks-partisans could hash it out with the imputil-
partisans). Import hacks are more common than ever, but 
they're all home-grown now.

- Gordon


From thomas.heller@ion-tof.com  Fri Oct  5 13:39:33 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 14:39:33 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BBD680D.8967.671A64B6@localhost>
Message-ID: <067f01c14d9a$ca0f30a0$e000a8c0@thomasnotebook>

[Thomas]
> > Your metapath models Python's import policy. One part of the
> > policy is that already loaded modules are fetched from
> > sys.modules instead of imported again. Should this behaviour also
> > be modeled by a SysModulesCacheDirector on your metapath?
> 
[Gordon]
> That's already done by the ImportManager as soon as he's got 
> a candidate fqname.
I know this. I thought of moving this from ImportManager into
an ImportDirector. Probably doesn't matter too much.

> The entire subject seems to have dropped off radar, after 
> starting out as highly controversial (the Import-SIG was started 
> so the ihooks-partisans could hash it out with the imputil-
> partisans). Import hacks are more common than ever, but 
> they're all home-grown now.
You thought of replacing the import hacks by custom import
policies implemented with imputil or iu4? No chance, IMO.

Several reasons I can think of:
- Import hacks have this cool touch...
- Import hacks start out to help the developers, but they never
get cleaned up for the poor user (pmw being an exception!)

Thomas



From guido@python.org  Fri Oct  5 14:06:32 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 09:06:32 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: Your message of "Fri, 05 Oct 2001 11:58:12 +0200."
 <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook>
References: <3BB204E3.23339.3A9F068C@localhost>
 <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook>
Message-ID: <200110051306.JAA24439@cj20424-a.reston1.va.home.com>

> It seems noone cares about this. imputil is in the distribution,
> but is it really 'official'?

I want to care, but I'm already spread too thin.  I don't find
'imputil' particularly official -- but Greg Stein pushed hard to get
it in the distribution.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gmcm@hypernet.com  Fri Oct  5 14:25:39 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Fri, 5 Oct 2001 09:25:39 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <067f01c14d9a$ca0f30a0$e000a8c0@thomasnotebook>
Message-ID: <3BBD7C93.29505.676A8D8C@localhost>

[Gordon]
> > The entire subject seems to have dropped off radar, after
> > starting out as highly controversial (the Import-SIG was
> > started so the ihooks-partisans could hash it out with the
> > imputil- partisans). Import hacks are more common than ever,
> > but they're all home-grown now.

[Thomas]
> You thought of replacing the import hacks by custom import
> policies implemented with imputil or iu4? No chance, IMO.

That's what ihooks was written for, and nobody ever used it, 
either.
 
> Several reasons I can think of:
> - Import hacks have this cool touch...

Ironically, they usually are justified as "making it easier on the 
user" (where they really mean programmer). Which is crap, 
because when the programmer has trouble with X.A.fribble(), 
he goes looking for the source. But there is no X/A.py.

> - Import hacks start out to help the developers, but they never
> get cleaned up for the poor user (pmw being an exception!)

And they help constrain Python to being a developer-only 
language.

The existing import mechanism is poorly understood (just see 
how many wrong answers an import question on c.l.py gets) 
and home to some major warts (such as the fact that relative 
and absolute imports are spelled the same way). Greg wrote 
imputil as a first step towards fixing those kinds of problems.

under-the-spreading-entropy-ly y'rs

- Gordon


From thomas.heller@ion-tof.com  Fri Oct  5 14:36:25 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 15:36:25 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BBD7C93.29505.676A8D8C@localhost>
Message-ID: <073001c14da2$baa54ac0$e000a8c0@thomasnotebook>

> The existing import mechanism is poorly understood (just see 
> how many wrong answers an import question on c.l.py gets) ...

Note that this question has recently been asked by Michael Abbot
on c.l.p, for now there are no answers:

  Is there up to date documentation for package support in Python 2?

  Section 6.11 of the "Python Reference Manual" has the following nice quote:

  [XXX Can't be bothered to spell this out right now; see the URL 
  http://www.python.org/doc/essays/packages.html for more details, also about 
  how the module search works from inside a package.] 

  and the referred URL documents Python 1.5.  I presume that nothing 
  significant has changed recently, but it's certainly disconcerting for 
  something as fundamental as module importing to not actually be part of the 
  core language documentation!

Thomas



From guido@python.org  Fri Oct  5 15:36:25 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 10:36:25 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
Message-ID: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>

I'm finding more and more the need to test whether an object is a
string, including both 8-bit and Unicode strings.  Current practice
seems to be:

    if type(x) in (str, unicode):

or (more verbose but more b/w compatible):

    if type(x) in (types.StringType, types.UnicodeType):

or (using a variable recently added to types -- I'm not sure this was
a good idea):

    if type(x) in types.StringTypes:

all of which break if type(x) is a *subclass* of str or unicode.  The
alternative:

    if isinstance(x, str) or isinstance(x, unicode):

is apparently too much typing.

Some alternatives that have been proposed already:

- Create a common base class of str and unicode, which should be an
  abstract class.  This is the most OO solution, but I can't think of
  a good name; abstractstring is too long, AbstractString or String
  are uncommon naming conventions for built-in types, 'string' would
  almost work except that it's already the name of a very common
  module.[*]

- Make str a subclass of unicode (or vice versa).  This can't be done
  because subclassing requires implementation inheritance, in
  particular the instance structure layout must overlap.  Also, this
  would make it hard to check for either str or unicode.

- Create a new service function, IsString(x) or isString(x) or
  isstring(x), that's a shortcut for "isinstance(x, str) or
  isinstance(x, unicode)".  The question them becomes where to put
  this: as a builtin, in types.py, or somewhere else...

Preferences please?

--Guido van Rossum (home page: http://www.python.org/~guido/)

[*] For a while I toyed with the idea of calling the abstract base
class 'string', and hacking import so that sys.modules['string'] is
the string class.  The abstract base class should then have methods
that invoke the concrete implementations, so that string.split(s)
would be the same as s.split().  This would be compatible with
previous uses of the string module!  string.letters etc. could then be
class variables.  Unfortunately this broke down when I realized that
the signature of string.join() is wrong for the string module: the the
string.join function is string.join(sequence, stringobject) while the
signature of the string method is join(stringobject, sequence).  So
much for that idea... :-)  (BTW this shows to me again that the method
signature is right and the function signature is wrong.  But even my
time machine isn't powerful enough to fix this.)  (Hm, it could be
saved by making string.join() accept the arguments in either order.
Gross. :-)


From nas@python.ca  Fri Oct  5 16:11:36 2001
From: nas@python.ca (Neil Schemenauer)
Date: Fri, 5 Oct 2001 08:11:36 -0700
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 05, 2001 at 10:36:25AM -0400
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
Message-ID: <20011005081136.A25134@glacier.arctrix.com>

Guido van Rossum wrote:
> 
>     if isinstance(x, str) or isinstance(x, unicode):
> 
> is apparently too much typing.

Perhaps we could extend isinstance().  How about

    isinstance(x, str, unicode)

or

    isinstance(x, (str, unicode))

This is a common problem not limited to string types.  I often want to
test if something is a tuple or a list for example.

  Neil


From guido@python.org  Fri Oct  5 16:20:30 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 11:20:30 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 08:11:36 PDT."
 <20011005081136.A25134@glacier.arctrix.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <20011005081136.A25134@glacier.arctrix.com>
Message-ID: <200110051520.LAA25303@cj20424-a.reston1.va.home.com>

> Guido van Rossum wrote:
> > 
> >     if isinstance(x, str) or isinstance(x, unicode):
> > 
> > is apparently too much typing.
> 
> Perhaps we could extend isinstance().  How about
> 
>     isinstance(x, str, unicode)
> 
> or
> 
>     isinstance(x, (str, unicode))
> 
> This is a common problem not limited to string types.  I often want to
> test if something is a tuple or a list for example.

You may be on to something.  It certainly saves me inventing a new
name!  I'll think about it a bit more...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Fri Oct  5 16:27:01 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 5 Oct 2001 11:27:01 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <20011005081136.A25134@glacier.arctrix.com>
Message-ID: <15293.53573.251652.664773@anthem.wooz.org>

    GvR> if isinstance(x, str) or isinstance(x, unicode): > > is
    GvR> apparently too much typing.

    NS> Perhaps we could extend isinstance().  How about

    NS>     isinstance(x, str, unicode)

    NS> or

    NS>     isinstance(x, (str, unicode))

    NS> This is a common problem not limited to string types.  I often
    NS> want to test if something is a tuple or a list for example.

That's an interesting idea, but please use the former signature, not
the latter.

And what would it return?  It needs to return a true value on success,
but maybe instead of returning 1, it might be more useful to return
the type argument that matched, e.g.:

>>> isinstance('', str, unicode)
<type 'str'>
>>> isinstance(u'', str, unicode)
<type 'unicode'>
>>> isinstance((), list, dictionary, tuple)
<type 'tuple'>
>>> isinstance(7, list, dictionary, tuple)
0

-Barry


From fdrake@acm.org  Fri Oct  5 16:21:50 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 11:21:50 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051520.LAA25303@cj20424-a.reston1.va.home.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <20011005081136.A25134@glacier.arctrix.com>
 <200110051520.LAA25303@cj20424-a.reston1.va.home.com>
Message-ID: <15293.53262.479355.538873@grendel.zope.com>

Neil Schemenauer wrote:
 >     isinstance(x, str, unicode)
...
 > This is a common problem not limited to string types.  I often want to
 > test if something is a tuple or a list for example.

Guido van Rossum writes:
 > You may be on to something.  It certainly saves me inventing a new
 > name!  I'll think about it a bit more...

  I like this approach.  It also doesn't preclude the other changes.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From jim@interet.com  Fri Oct  5 16:31:32 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 05 Oct 2001 11:31:32 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook>
Message-ID: <3BBDD254.32DC4071@interet.com>

Thomas Heller wrote:
> 
> From: "Gordon McMillan" <gmcm@hypernet.com>
> > I would like to propose these (or something quite like them) as
> > replacements for the official versions. The code is quite similar
> > (in fact, the modulefinder code could have been written by
> > subclassing the imputil stuff, but I wrote them the other way
> > 'round).
> >
> It seems noone cares about this. imputil is in the distribution,
> but is it really 'official'?

I care about it.  I am using imputil to implement "jar" files,
zip files of Python libraries.  Greg worked hard to get imputil
into the distribution, and I appreciate his effort.  But I always
wondered if imputil was 'official'.

Users on c.l.p frequently want to ship a stand-alone
Python.  The current answer to this is freeze, but this requires a
C compiler and some work.  A standard jar file implementation would
satisfy this need.  What I am thinking of is documentation which
says something like: "Zip up the Python libraries into pylibs.zip
and put that file in the same directory as python[.exe] and
everything Just Works."

My interest in imputil and ImportManager is to implement jar files,
and neither does this well.  The problem is always how to get the
SpecialImporter.py imported first, and how to hack the imports that
the SpecialImporter.py may need to import.  In iu4.py we see the use
of posix.stat to replace the os module, and the import of only
built-in modules sys, imp, and marshal.  The same is true of imputil.py.

I admit I don't really understand package imports.  But I feel they
are an unsolved problem, because package authors always seem to
write custom ones.

A first-class solution would have these features IMHO:
1) Access to libraries in zip files is implemented in C (I volunteer).
2) All Python library modules can be imported by C code,
   either from directories or zip files.
3) Package import is not implemented in C, it is implemented
   by a Python module which is part of the standard library.
4) If an import of A.B is attempted, and a function
   A.MyCustomImporter() exists, then that is used instead of
   the default.
5) More or better ideas from you all.

Jim Ahlstrom


From fdrake@acm.org  Fri Oct  5 16:24:22 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 11:24:22 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <15293.53573.251652.664773@anthem.wooz.org>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <20011005081136.A25134@glacier.arctrix.com>
 <15293.53573.251652.664773@anthem.wooz.org>
Message-ID: <15293.53414.472587.851330@grendel.zope.com>

Barry A. Warsaw writes:
 > And what would it return?  It needs to return a true value on success,
 > but maybe instead of returning 1, it might be more useful to return
 > the type argument that matched, e.g.:

  Ugh!  If that's what you want, make it explicit in the code:


    for t in ListType, TupleType, ...:
        if isinstance(obj, t):
            break
    else:
        raise Exception('no match!')


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Fri Oct  5 16:43:44 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 11:43:44 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 11:27:01 EDT."
 <15293.53573.251652.664773@anthem.wooz.org>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com>
 <15293.53573.251652.664773@anthem.wooz.org>
Message-ID: <200110051543.LAA25454@cj20424-a.reston1.va.home.com>

>     GvR> if isinstance(x, str) or isinstance(x, unicode): > > is
>     GvR> apparently too much typing.
> 
>     NS> Perhaps we could extend isinstance().  How about
> 
>     NS>     isinstance(x, str, unicode)
> 
>     NS> or
> 
>     NS>     isinstance(x, (str, unicode))
> 
>     NS> This is a common problem not limited to string types.  I often
>     NS> want to test if something is a tuple or a list for example.
> 
> That's an interesting idea, but please use the former signature, not
> the latter.

Why?  The varargs signature makes it less convenient (and less
efficient!) to pre-calculate the list of types.  It also makes it
harder to implement this in PyObject_IsInstance(), so that C code can
use this convenience.  Finally, if I already know the meaning of
isinstance(x, y), then when I encounter isinstance(x, (y, z)) for the
first time, it's very easy to guess the meaning.  The meaning of
isinstance(x, y, z) is much more murky: z could be an optional
argument specifying some other modification of the basic isinstance().

> And what would it return?  It needs to return a true value on success,
> but maybe instead of returning 1, it might be more useful to return
> the type argument that matched, e.g.:
> 
> >>> isinstance('', str, unicode)
> <type 'str'>
> >>> isinstance(u'', str, unicode)
> <type 'unicode'>
> >>> isinstance((), list, dictionary, tuple)
> <type 'tuple'>
> >>> isinstance(7, list, dictionary, tuple)
> 0

Here I agree with Fred: too much hackery combined in one function.
And what's the use case?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jeremy@zope.com  Fri Oct  5 16:47:11 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Fri, 5 Oct 2001 11:47:11 -0400 (EDT)
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <15293.53414.472587.851330@grendel.zope.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <20011005081136.A25134@glacier.arctrix.com>
 <15293.53573.251652.664773@anthem.wooz.org>
 <15293.53414.472587.851330@grendel.zope.com>
Message-ID: <15293.54783.743905.190860@slothrop.digicool.com>

>>>>> "FLD" == Fred L Drake, <fdrake@acm.org> writes:

  FLD>     for t in ListType, TupleType, ...:
  FLD>         if isinstance(obj, t):
  FLD>             break
  FLD>     else:
  FLD>         raise Exception('no match!')

Or:

try:
    raise obj
except (ListType, TupleType):
    pass
except:
    raise TypeError, ...

with-apologies-to-tim-ly y'rs,
Jeremy



From guido@python.org  Fri Oct  5 16:46:57 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 11:46:57 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 11:27:01 EDT."
 <15293.53573.251652.664773@anthem.wooz.org>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com>
 <15293.53573.251652.664773@anthem.wooz.org>
Message-ID: <200110051546.LAA25549@cj20424-a.reston1.va.home.com>

Here's an implementation of Neil's idea.

*** abstract.c	2001/10/01 17:10:18	2.83
--- abstract.c	2001/10/05 15:49:04
***************
*** 1805,1810 ****
--- 1805,1829 ----
  	else if (PyType_Check(cls)) {
  		retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls);
  	}
+ 	else if (PySequence_Check(cls)) {
+ 		int i, n;
+ 
+ 		cls = PySequence_Fast(
+ 			cls, "isinstance() arg 2 is an unacceptable sequence");
+ 		if (cls == NULL)
+ 			return -1;
+ 		n = PySequence_Size(cls);
+ 		if (n < 0)
+ 			retval = -1;
+ 		for (i = 0; i < n; i++) {
+ 			retval = PyObject_IsInstance(
+ 				inst, PySequence_Fast_GET_ITEM(cls, i));
+ 			if (retval != 0)
+ 				break;
+ 		}
+ 		Py_DECREF(cls);
+ 		return retval;
+ 	}
  	else if (!PyInstance_Check(inst)) {
  		if (__class__ == NULL) {
  			__class__ = PyString_FromString("__class__");
***************
*** 1827,1833 ****
  
  	if (retval < 0) {
  		PyErr_SetString(PyExc_TypeError,
! 				"isinstance() arg 2 must be a class or type");
  	}
  	return retval;
  }
--- 1846,1853 ----
  
  	if (retval < 0) {
  		PyErr_SetString(PyExc_TypeError,
! 				"isinstance() arg 2 must be a class or type "
! 				"or tuple of those");
  	}
  	return retval;
  }

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Fri Oct  5 16:46:48 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 11:46:48 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051546.LAA25549@cj20424-a.reston1.va.home.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <20011005081136.A25134@glacier.arctrix.com>
 <15293.53573.251652.664773@anthem.wooz.org>
 <200110051546.LAA25549@cj20424-a.reston1.va.home.com>
Message-ID: <15293.54760.113875.151084@grendel.zope.com>

Guido van Rossum writes:
 > Here's an implementation of Neil's idea.

  Commit this and I'll take care of the documentation.  ;-)


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From mal@lemburg.com  Fri Oct  5 16:54:05 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 05 Oct 2001 17:54:05 +0200
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
Message-ID: <3BBDD79D.C26D1646@lemburg.com>

Guido van Rossum wrote:
> 
> I'm finding more and more the need to test whether an object is a
> string, including both 8-bit and Unicode strings.  Current practice
> seems to be:
> 
>     if type(x) in (str, unicode):
> 
> or (more verbose but more b/w compatible):
> 
>     if type(x) in (types.StringType, types.UnicodeType):
> 
> or (using a variable recently added to types -- I'm not sure this was
> a good idea):
> 
>     if type(x) in types.StringTypes:
> 
> all of which break if type(x) is a *subclass* of str or unicode.  The
> alternative:
> 
>     if isinstance(x, str) or isinstance(x, unicode):
> 
> is apparently too much typing.
> 
> Some alternatives that have been proposed already:
> 
> - Create a common base class of str and unicode, which should be an
>   abstract class.  This is the most OO solution, but I can't think of
>   a good name; abstractstring is too long, AbstractString or String
>   are uncommon naming conventions for built-in types, 'string' would
>   almost work except that it's already the name of a very common
>   module.[*]

+1. This would be nice and the same could be done for sequences, file-like
objects and other common currently interface-defined object categories.

About the naming: how about numberclass, stringclass, sequenceclass, 
fileclass ?!

Then you could write:

if isinstance(obj, stringclass): ...

which looks OK and is not too much typing.

The advantage of this approach is that it can be extended to other
types and classes as well (much like you can currently do with the
Python exceptions).

With the new type logic in place, how hard would it be making
the existing built-in types subclasses of these base types ?
(also: is there a run-time penalty for this ?)
 
> - Make str a subclass of unicode (or vice versa).  This can't be done
>   because subclassing requires implementation inheritance, in
>   particular the instance structure layout must overlap.  Also, this
>   would make it hard to check for either str or unicode.

-0. This would be hard to get right because the two objects use a 
very different struct layout. Could be an option in the long run though.
 
> - Create a new service function, IsString(x) or isString(x) or
>   isstring(x), that's a shortcut for "isinstance(x, str) or
>   isinstance(x, unicode)".  The question them becomes where to put
>   this: as a builtin, in types.py, or somewhere else...

-1. This mechanism can not be extended by e.g. UserStrings. 
 
> Preferences please?
> 
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> 
> [*] For a while I toyed with the idea of calling the abstract base
> class 'string', and hacking import so that sys.modules['string'] is
> the string class.  The abstract base class should then have methods
> that invoke the concrete implementations, so that string.split(s)
> would be the same as s.split().  This would be compatible with
> previous uses of the string module!  string.letters etc. could then be
> class variables.  Unfortunately this broke down when I realized that
> the signature of string.join() is wrong for the string module: the the
> string.join function is string.join(sequence, stringobject) while the
> signature of the string method is join(stringobject, sequence).  So
> much for that idea... :-)  (BTW this shows to me again that the method
> signature is right and the function signature is wrong.  But even my
> time machine isn't powerful enough to fix this.)  (Hm, it could be
> saved by making string.join() accept the arguments in either order.
> Gross. :-)

Indeed. :-)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From guido@python.org  Fri Oct  5 16:57:53 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 11:57:53 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 17:54:05 +0200."
 <3BBDD79D.C26D1646@lemburg.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
Message-ID: <200110051557.LAA25625@cj20424-a.reston1.va.home.com>

> > - Create a common base class of str and unicode, which should be an
> >   abstract class.  This is the most OO solution, but I can't think of
> >   a good name; abstractstring is too long, AbstractString or String
> >   are uncommon naming conventions for built-in types, 'string' would
> >   almost work except that it's already the name of a very common
> >   module.[*]
> 
> +1. This would be nice and the same could be done for sequences, file-like
> objects and other common currently interface-defined object categories.
> 
> About the naming: how about numberclass, stringclass, sequenceclass, 
> fileclass ?!
> 
> Then you could write:
> 
> if isinstance(obj, stringclass): ...
> 
> which looks OK and is not too much typing.

Hm, I don't know if "class" is the proper suffix to mean "abstract
base class".  I think I'll either call it "numberbase" or
"abstractnumber".

> The advantage of this approach is that it can be extended to other
> types and classes as well (much like you can currently do with the
> Python exceptions).

Yup, that's why it's the most OO solution. :-)

> With the new type logic in place, how hard would it be making
> the existing built-in types subclasses of these base types ?

That would be a prerequisite!

> (also: is there a run-time penalty for this ?)

None.

> > - Make str a subclass of unicode (or vice versa).  This can't be done
> >   because subclassing requires implementation inheritance, in
> >   particular the instance structure layout must overlap.  Also, this
> >   would make it hard to check for either str or unicode.
> 
> -0. This would be hard to get right because the two objects use a 
> very different struct layout. Could be an option in the long run though.

I tried to explany that I already gave it a -1. ;-)

> > - Create a new service function, IsString(x) or isString(x) or
> >   isstring(x), that's a shortcut for "isinstance(x, str) or
> >   isinstance(x, unicode)".  The question them becomes where to put
> >   this: as a builtin, in types.py, or somewhere else...
> 
> -1. This mechanism can not be extended by e.g. UserStrings. 

Good point.

There's still Neil's isinstance(x, (str, unicode)) which gives the
programmer more freedom: maybe some function wants to support lists
and tuples but not all sequences.

I'm currently +1 on introducing names for abstract base classes *and*
extensing isinstance()'s API.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Fri Oct  5 17:01:41 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 18:01:41 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com>
Message-ID: <085f01c14db7$06704720$e000a8c0@thomasnotebook>

From: "James C. Ahlstrom" <jim@interet.com>
> Thomas Heller wrote:
> > 
> > From: "Gordon McMillan" <gmcm@hypernet.com>
> > > I would like to propose these (or something quite like them) as
> > > replacements for the official versions. The code is quite similar
> > > (in fact, the modulefinder code could have been written by
> > > subclassing the imputil stuff, but I wrote them the other way
> > > 'round).
> > >
> > It seems noone cares about this. imputil is in the distribution,
> > but is it really 'official'?
> 
> I care about it.  I am using imputil to implement "jar" files,
> zip files of Python libraries.  Greg worked hard to get imputil
> into the distribution, and I appreciate his effort.  But I always
> wondered if imputil was 'official'.
> 
> Users on c.l.p frequently want to ship a stand-alone
> Python.  The current answer to this is freeze, but this requires a
> C compiler and some work.  A standard jar file implementation would
> satisfy this need.  What I am thinking of is documentation which
> says something like: "Zip up the Python libraries into pylibs.zip
> and put that file in the same directory as python[.exe] and
> everything Just Works."
Do you know about py2exe and installer? This is a variation of what they do.
py2exe basically builds a zip-file containing the libary modules,
appends them to a special python.exe. This special exe 'boots' itself
by using imputil, and exposes a 'get_code' function into the __main__
namespace. This function is called by a module name contained in the
(embedded) zip, and returns the stored bytes. The installed imputil
code then creates a module from these bytes.
Gordon's installer works very similar, except that he uses a custom
archive format (IIRC).

> 
> My interest in imputil and ImportManager is to implement jar files,
> and neither does this well.  The problem is always how to get the
> SpecialImporter.py imported first, and how to hack the imports that
> the SpecialImporter.py may need to import.  In iu4.py we see the use
> of posix.stat to replace the os module, and the import of only
> built-in modules sys, imp, and marshal.  The same is true of imputil.py.
> 
> I admit I don't really understand package imports.  But I feel they
> are an unsolved problem, because package authors always seem to
> write custom ones.
> 
> A first-class solution would have these features IMHO:
> 1) Access to libraries in zip files is implemented in C (I volunteer).
You could take the code from py2exe...
> 2) All Python library modules can be imported by C code,
>    either from directories or zip files.
> 3) Package import is not implemented in C, it is implemented
>    by a Python module which is part of the standard library.
This could be imputil, or the new iu4.
> 4) If an import of A.B is attempted, and a function
>    A.MyCustomImporter() exists, then that is used instead of
>    the default.
> 5) More or better ideas from you all.

BTW: I remember I having seen a ZipFileImporter using imputil, but
cannot find it any more. Am I dreaming?

Thomas



From gward@python.net  Fri Oct  5 17:05:35 2001
From: gward@python.net (Greg Ward)
Date: Fri, 5 Oct 2001 12:05:35 -0400
Subject: [Python-Dev] Making setup.py a little more flexible
In-Reply-To: <200110041712.TAA25396@pandora.informatik.hu-berlin.de>
References: <200110041712.TAA25396@pandora.informatik.hu-berlin.de>
Message-ID: <20011005120535.A1801@cthulhu.mems-exchange.org>

[Martin von Loewis, on my band-aid patch to setup.py]
> -1. First of all, a bandaid for the specific problem is already in place:
> you can configure everything through Modules/Setup, in particular if you
> find that setup.py doesn't do the right thing. Modules/Setup allow much
> more detailed changes than what your changes to setup.py offers, plus it
> is an established solution to this problem.

OK, I did not realize that Modules/Setup still had its old effect.  I
thought that the old system was history; I guess not.  Oops.

In discovering this, I think I have found some bits of README that need
updating.  Eg.

  (When working inside the Modules directory, use "make Makefile;
  make".)

-- this is untrue.  There is no Modules/Makefile anymore, right?

  In addition to the file Setup, you can also edit the file Setup.local.
  (the makesetup script processes both).  You may find it more
  convenient to edit Setup.local and leave Setup alone.  Then, when
  installing a new Python version, you can copy your old Setup.local
  file.

I'm not sure if this is correct either.  I tried to make my changes in
Setup.local -- ie. add lines to define readline and _socket, with no
reference to the broken libraries on this system.  It had no effect.  I
made this same changes in Setup, and it worked.  Is this the expected
behaviour?  From the above paragraph, it sounds like editing Setup.local
should have worked for me.

> Furthermore, your specific patch seems incomplete: You have to run setup.py
> manually, which means that you cannot use a plain "make". There should be
> atleast minimal interfacing to Makefile.pre.in for this patch to be acceptable.

Correct -- I wanted to see what sort of reaction I would get before
propagating the effects of the patch upwards.  It now looks like I can
withdraw the patch, so I'm glad I didn't waste the time on doing that.

        Greg
-- 
Greg Ward - Unix weenie                                 gward@python.net
http://starship.python.net/~gward/
"I know a lot about art, but I don't know what I like!"


From guido@python.org  Fri Oct  5 17:02:36 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 12:02:36 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 11:46:48 EDT."
 <15293.54760.113875.151084@grendel.zope.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com>
 <15293.54760.113875.151084@grendel.zope.com>
Message-ID: <200110051602.MAA25705@cj20424-a.reston1.va.home.com>

> Guido van Rossum writes:
>  > Here's an implementation of Neil's idea.
> 
>   Commit this and I'll take care of the documentation.  ;-)

I'll wait 24 hours.  Who knows what I think after the anesthetics in
my jaw wear off... :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Fri Oct  5 17:06:27 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 05 Oct 2001 18:06:27 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com>
Message-ID: <3BBDDA83.B28556CB@lemburg.com>

"James C. Ahlstrom" wrote:
> 
> Thomas Heller wrote:
> >
> > From: "Gordon McMillan" <gmcm@hypernet.com>
> > > I would like to propose these (or something quite like them) as
> > > replacements for the official versions. The code is quite similar
> > > (in fact, the modulefinder code could have been written by
> > > subclassing the imputil stuff, but I wrote them the other way
> > > 'round).
> > >
> > It seems noone cares about this. imputil is in the distribution,
> > but is it really 'official'?
> 
> I care about it.  I am using imputil to implement "jar" files,
> zip files of Python libraries.  Greg worked hard to get imputil
> into the distribution, and I appreciate his effort.  But I always
> wondered if imputil was 'official'.
> 
> Users on c.l.p frequently want to ship a stand-alone
> Python.  The current answer to this is freeze, but this requires a
> C compiler and some work.  A standard jar file implementation would
> satisfy this need.  What I am thinking of is documentation which
> says something like: "Zip up the Python libraries into pylibs.zip
> and put that file in the same directory as python[.exe] and
> everything Just Works."
> 
> My interest in imputil and ImportManager is to implement jar files,
> and neither does this well.  The problem is always how to get the
> SpecialImporter.py imported first, and how to hack the imports that
> the SpecialImporter.py may need to import.  In iu4.py we see the use
> of posix.stat to replace the os module, and the import of only
> built-in modules sys, imp, and marshal.  The same is true of imputil.py.
> 
> I admit I don't really understand package imports.  But I feel they
> are an unsolved problem, because package authors always seem to
> write custom ones.

Huh ? The builtin mechanism in Python works just fine for packages.
I think that custom importers are not very common these days. Still,
I do agree that the current implementation is a gross hack and should
in the long run be replaced by something more OO-style.

For the short-term however, I have a feeling that there are not
that many features actually being requested out there -- the most
wanted one is certainly the ability to import modules from some form
of archive, so perhaps we should start thinking about providing
this kind of support in the core ?!

> A first-class solution would have these features IMHO:
> 1) Access to libraries in zip files is implemented in C (I volunteer).

Great !

> 2) All Python library modules can be imported by C code,
>    either from directories or zip files.

Right.

> 3) Package import is not implemented in C, it is implemented
>    by a Python module which is part of the standard library.

Why ? I don't see a need for that and it would slow down 
package imports which are becoming increasingly popular as the
Python code base grows.

> 4) If an import of A.B is attempted, and a function
>    A.MyCustomImporter() exists, then that is used instead of
>    the default.

Better postpone this one...

> 5) More or better ideas from you all.

One question: should these ZIP-archives filenames be placed in 
sys.path or should Python scan for ZIP-archives within the
dirs on sys.path ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From gward@python.net  Fri Oct  5 17:19:31 2001
From: gward@python.net (Greg Ward)
Date: Fri, 5 Oct 2001 12:19:31 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <3BBDDA83.B28556CB@lemburg.com>
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com>
Message-ID: <20011005121931.B1801@cthulhu.mems-exchange.org>

On 05 October 2001, M.-A. Lemburg said:
> Huh ? The builtin mechanism in Python works just fine for packages.
[...]
> > A first-class solution would have these features IMHO:
> > 1) Access to libraries in zip files is implemented in C (I volunteer).
> 
> Great !
> 
> > 2) All Python library modules can be imported by C code,
> >    either from directories or zip files.
> 
> Right.
> 
> > 3) Package import is not implemented in C, it is implemented
> >    by a Python module which is part of the standard library.
> 
> Why ? I don't see a need for that and it would slow down 
> package imports which are becoming increasingly popular as the
> Python code base grows.

For the record, I agree with you on all points: there's no need to
rewrite package import in Python, but it would be nice to have "import
from archive" available in the core.  Quite possibly adding this feature
would silence 80% of the desire for genreic import hooks.

The only import hook I'm familiar with is Quixote, where we have defined
a Python dialect called PTL (Python Template Language, used to embed
HTML [or other text] in Python).  Install Quixote's import hook, and you
can import .ptl files just like .py files.  It's very handy.  Out of
curiosity, does anyone know of any other import hooks like this out
there -- ie. import something that is not strictly Python.  (It sounds
like most import hooks/hacks deal with the location of the .py files to
import.  Quixote doesn't touch that, but it adds the ability to import
not-quite-Python source files.)

> One question: should these ZIP-archives filenames be placed in 
> sys.path or should Python scan for ZIP-archives within the
> dirs on sys.path ?

I think the archive file should be listed in sys.path.

Note that my experience of this in Java was largely negative, because
Java doesn't have a standard way of putting .class files in the
filesystem (AFAIK) -- so everything has to be in a .jar file, and those
.jar files can be anywhere you please.  So you end up with a mile-long
CLASSPATH that's very fragile and forever needing fixing.  As long as
most Python modules are accessed the ordinary way (files in a
directory), then Python won't have a problem.  But if somebody makes a
Python installation with "stdlib.zip", "distutils.zip",
"mxDateTime.zip", etc. etc., then the poor users will be in the same
boat as Java users.

The main concerns I have with scanning sys.path directories for ZIP
files are performance and transparence.  (It ain't necessarily obvious
where a particular module will come from if sys.path can be searched in
two ways.)

        Greg
-- 
Greg Ward - Unix weenie                                 gward@python.net
http://starship.python.net/~gward/
Pointers are Arrays; Code is Data; Time is Money


From akuchlin@mems-exchange.org  Fri Oct  5 17:27:36 2001
From: akuchlin@mems-exchange.org (Andrew Kuchling)
Date: Fri, 5 Oct 2001 12:27:36 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <20011005121931.B1801@cthulhu.mems-exchange.org>; from gward@python.net on Fri, Oct 05, 2001 at 12:19:31PM -0400
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org>
Message-ID: <20011005122736.A1213@ute.mems-exchange.org>

On Fri, Oct 05, 2001 at 12:19:31PM -0400, Greg Ward wrote:
>can import .ptl files just like .py files.  It's very handy.  Out of
>curiosity, does anyone know of any other import hooks like this out
>there -- ie. import something that is not strictly Python.  (It sounds

I vaguely remember an import hook that looked for ILU or Fnorb foo.idl
files on the path and automatically compiled them to foo.py if found.

--amk



From gward@python.net  Fri Oct  5 17:28:49 2001
From: gward@python.net (Greg Ward)
Date: Fri, 5 Oct 2001 12:28:49 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
Message-ID: <20011005122849.C1801@cthulhu.mems-exchange.org>

On 05 October 2001, Guido van Rossum said:
> - Create a new service function, IsString(x) or isString(x) or
>   isstring(x), that's a shortcut for "isinstance(x, str) or
>   isinstance(x, unicode)".  The question them becomes where to put
>   this: as a builtin, in types.py, or somewhere else...

I've thought Python should have this for a long time.  But Marc-Andre's
point pretty much shoots it down, unless you make isstring(x) sugar for
  isinstance(x, (str, unicode))

Hmmm.  It's a very small dose of sugar, but I like it all the same.

> [...] (BTW this shows to me again that the method
> signature is right and the function signature is wrong.  But even my
> time machine isn't powerful enough to fix this.)  (Hm, it could be
> saved by making string.join() accept the arguments in either order.
> Gross. :-)

Note that a certain infamous April Fool's post from a few years back
said that that was what Python 1.6 would do.  I wasn't serious,
either.  ;-)

        Greg
-- 
Greg Ward - just another /P(erl|ython)/ hacker          gward@python.net
http://starship.python.net/~gward/
LILO boot: linux init=/usr/bin/emacs


From barry@zope.com  Fri Oct  5 17:30:34 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 5 Oct 2001 12:30:34 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
Message-ID: <15293.57386.311536.729065@anthem.wooz.org>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> I'm currently +1 on introducing names for abstract base
    GvR> classes *and* extensing isinstance()'s API.

I'm worried about the abstract base class idea, because it might lead
us down a path of having a complicated class hierarchy in order to
provide completeness.

E.g. should "strings" be a sequencebase, as well as a stringbase?  And
what does a sequencebase /mean/?

Will we push to try to include protocols/interfaces into this mix, so
that we'll end up defining abstract classes for all the deep-mojo
interfaces?  Maybe we'll want to have abstract base classes for file
objects so things like file and StringIO's can have them as a common
base class?

In any event, I think the isinstance() extension is a simple, clean,
and (mostly :) uncontroversial change, so I'm also +1 on doing that
now.

Adding a set of abstract base classes and hooking them up with the
existing concrete data types seems much murkier to me, and of greater
long range impact, so I'm -1 on the idea, at least until it gets
PEP'd.

-Barry


From jeremy@zope.com  Fri Oct  5 17:31:30 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Fri, 5 Oct 2001 12:31:30 -0400 (EDT)
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
Message-ID: <15293.57442.799246.716550@slothrop.digicool.com>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

  GvR> There's still Neil's isinstance(x, (str, unicode)) which gives
  GvR> the programmer more freedom: maybe some function wants to
  GvR> support lists and tuples but not all sequences.

  GvR> I'm currently +1 on introducing names for abstract base classes
  GvR> *and* extensing isinstance()'s API.

+1.

Jeremy




From nas@python.ca  Fri Oct  5 17:42:09 2001
From: nas@python.ca (Neil Schemenauer)
Date: Fri, 5 Oct 2001 09:42:09 -0700
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051546.LAA25549@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 05, 2001 at 11:46:57AM -0400
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com>
Message-ID: <20011005094209.A25437@glacier.arctrix.com>

Guido van Rossum wrote:
> Here's an implementation of Neil's idea.
> 
> *** abstract.c	2001/10/01 17:10:18	2.83
> --- abstract.c	2001/10/05 15:49:04
> ***************
> *** 1805,1810 ****
> --- 1805,1829 ----
>   	else if (PyType_Check(cls)) {
>   		retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls);
>   	}
> + 	else if (PySequence_Check(cls)) {
> + 		int i, n;

Is is possible that a type also passes PySequence_Check?  If so, that
could lead to confusing behavior.

  Neil


From guido@python.org  Fri Oct  5 17:50:48 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 12:50:48 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 09:42:09 PDT."
 <20011005094209.A25437@glacier.arctrix.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com>
 <20011005094209.A25437@glacier.arctrix.com>
Message-ID: <200110051650.MAA26943@cj20424-a.reston1.va.home.com>

> > *** 1805,1810 ****
> > --- 1805,1829 ----
> >   	else if (PyType_Check(cls)) {
> >   		retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls);
> >   	}
> > + 	else if (PySequence_Check(cls)) {
> > + 		int i, n;
> 
> Is is possible that a type also passes PySequence_Check?  If so, that
> could lead to confusing behavior.

The only way a type could pass PySequence_Check is if someone
subclasses 'type' and adds a __getitem__ method.  In that case, it's
appropriate that PyType_Check() has prevalence.  Also, adding
__getitem__ to a type seems to serve no purpose, so I'm not worried
about this (and yes, I had thought about this :-).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Fri Oct  5 17:51:22 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Fri,  5 Oct 2001 12:51:22 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011005165122.33A8A28697@beowolf.digicool.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Removed reference material for some long-obsolete Mac OS modules that have
not been included in recent versions of MacPython.

Added material on the proposed C API for weak reference objects; see the
Python/C API Reference Manual:

    http://python.sourceforge.net/devel-docs/api/weakref-objects.html



From mal@lemburg.com  Fri Oct  5 17:57:20 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 05 Oct 2001 18:57:20 +0200
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org>
Message-ID: <3BBDE670.52DFB7F0@lemburg.com>

"Barry A. Warsaw" wrote:
> 
> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes:
> 
>     GvR> I'm currently +1 on introducing names for abstract base
>     GvR> classes *and* extensing isinstance()'s API.
> 
> I'm worried about the abstract base class idea, because it might lead
> us down a path of having a complicated class hierarchy in order to
> provide completeness.

Since these are abstract classes (basically names for what
we now loosly call interface, e.g. file-like object, sequence-like
object etc.) this very shallow hierarchy wouldn't hurt all that
much.

Sure, strings are sequences, so they stringbase would have
to be a subclass of sequencebase, but I don't think we'll ever
get more than 3 levels deep in the abstract class hierarchy.
 
> E.g. should "strings" be a sequencebase, as well as a stringbase?  And
> what does a sequencebase /mean/?

Good point. There's PySequence_Check(), but we'll have to discuss
this at some point anyway and come up with definitions for the currently
used fuzzy terms "sequence", "number", "string", and so on.
 
> Will we push to try to include protocols/interfaces into this mix, so
> that we'll end up defining abstract classes for all the deep-mojo
> interfaces?  Maybe we'll want to have abstract base classes for file
> objects so things like file and StringIO's can have them as a common
> base class?
> 
> In any event, I think the isinstance() extension is a simple, clean,
> and (mostly :) uncontroversial change, so I'm also +1 on doing that
> now.
> 
> Adding a set of abstract base classes and hooking them up with the
> existing concrete data types seems much murkier to me, and of greater
> long range impact, so I'm -1 on the idea, at least until it gets
> PEP'd.

Are you sure about the -1 ? Using abstract base classes seems like
a natural thing to do in an OO-language like Python while the
isinstance() hack would not allow you to use your own UserList
subclass for code which was written with sequences as input.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From guido@python.org  Fri Oct  5 17:55:31 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 12:55:31 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 12:30:34 EDT."
 <15293.57386.311536.729065@anthem.wooz.org>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
 <15293.57386.311536.729065@anthem.wooz.org>
Message-ID: <200110051655.MAA26969@cj20424-a.reston1.va.home.com>

>     GvR> I'm currently +1 on introducing names for abstract base
>     GvR> classes *and* extensing isinstance()'s API.
> 
> I'm worried about the abstract base class idea, because it might lead
> us down a path of having a complicated class hierarchy in order to
> provide completeness.

Yes, that's a danger.  I'm currently thinking about adding a small
number of abstract base classes, corresponding to the groups of basic
concrete data types in Python: numberbase, integerbase, sequencebase,
stringbase, mappingbase, filebase.  And I'm not even sure of these!
Practically, I only see a use for stringbase and integerbase, since
these have two concrete subclasses; the others are just to make the OO
zealots happy. :-)

> E.g. should "strings" be a sequencebase, as well as a stringbase?

Yes.  Any way you can think of that checks for a "sequence" will
consider strings to be sequences, unless you make explicit exceptions.

> And what does a sequencebase /mean/?

Whatever I want it to mean. :-)

The latest code for sequence-ness tests for sq_getitem != NULL, and I
think that's reasonable.

> Will we push to try to include protocols/interfaces into this mix, so
> that we'll end up defining abstract classes for all the deep-mojo
> interfaces?

I'd prefer not to.

> Maybe we'll want to have abstract base classes for file
> objects so things like file and StringIO's can have them as a common
> base class?

That's currently hard.  I don't think we can make StringIO a new-style
class, and classic classes can't inherit from new-style classes (not
even abstract ones).

> In any event, I think the isinstance() extension is a simple, clean,
> and (mostly :) uncontroversial change, so I'm also +1 on doing that
> now.

Yes.

> Adding a set of abstract base classes and hooking them up with the
> existing concrete data types seems much murkier to me, and of greater
> long range impact, so I'm -1 on the idea, at least until it gets
> PEP'd.

I'm at best +0 myself -- I have more thinking to do...

I've *mostly* convinced myself that stringbase is a good idea.  But
even there, the existence of the buffer API makes the semantics
slightly murky.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Fri Oct  5 18:04:39 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 5 Oct 2001 13:04:39 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
 <15293.57386.311536.729065@anthem.wooz.org>
 <3BBDE670.52DFB7F0@lemburg.com>
Message-ID: <15293.59431.1516.201622@anthem.wooz.org>

>>>>> "M" == M  <mal@lemburg.com> writes:

    M> Are you sure about the -1 ? Using abstract base classes seems
    M> like a natural thing to do in an OO-language like Python while
    M> the isinstance() hack would not allow you to use your own
    M> UserList subclass for code which was written with sequences as
    M> input.

Yes, I'm sure: we need a PEP first.  Ideally it would answer some of
the questions raised above, and it would lay out the class hierarchy.

-Barry


From guido@python.org  Fri Oct  5 18:02:07 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 13:02:07 -0400
Subject: [Python-Dev] Making setup.py a little more flexible
In-Reply-To: Your message of "Fri, 05 Oct 2001 12:05:35 EDT."
 <20011005120535.A1801@cthulhu.mems-exchange.org>
References: <200110041712.TAA25396@pandora.informatik.hu-berlin.de>
 <20011005120535.A1801@cthulhu.mems-exchange.org>
Message-ID: <200110051702.NAA27256@cj20424-a.reston1.va.home.com>

> In discovering this, I think I have found some bits of README that need
> updating.  Eg.
> 
>   (When working inside the Modules directory, use "make Makefile;
>   make".)
> 
> -- this is untrue.  There is no Modules/Makefile anymore, right?

Correct.  Fixed.

>   In addition to the file Setup, you can also edit the file Setup.local.
>   (the makesetup script processes both).  You may find it more
>   convenient to edit Setup.local and leave Setup alone.  Then, when
>   installing a new Python version, you can copy your old Setup.local
>   file.
> 
> I'm not sure if this is correct either.  I tried to make my changes in
> Setup.local -- ie. add lines to define readline and _socket, with no
> reference to the broken libraries on this system.  It had no effect.  I
> made this same changes in Setup, and it worked.  Is this the expected
> behaviour?  From the above paragraph, it sounds like editing Setup.local
> should have worked for me.

Strange.  When I make changes to Setup.local, they definitely work!

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Fri Oct  5 18:01:29 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 13:01:29 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <15293.59431.1516.201622@anthem.wooz.org>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
 <15293.57386.311536.729065@anthem.wooz.org>
 <3BBDE670.52DFB7F0@lemburg.com>
 <15293.59431.1516.201622@anthem.wooz.org>
Message-ID: <15293.59241.954894.415782@grendel.zope.com>

Barry A. Warsaw writes:
 > Yes, I'm sure: we need a PEP first.  Ideally it would answer some of
 > the questions raised above, and it would lay out the class hierarchy.

  Sounds to me like you just volunteered.
  And on that note, its time for some cheap Chinese food...


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Fri Oct  5 18:07:32 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 13:07:32 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 13:04:39 EDT."
 <15293.59431.1516.201622@anthem.wooz.org>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com>
 <15293.59431.1516.201622@anthem.wooz.org>
Message-ID: <200110051707.NAA27421@cj20424-a.reston1.va.home.com>

> Yes, I'm sure: we need a PEP first.  Ideally it would answer some of
> the questions raised above, and it would lay out the class hierarchy.

The more I think about this, the more I agree that we need a PEP
first: there are lots of potential traps.  Maybe the biggest danger:
declaring an abstract base class for e.g. "sequence-ness" would
encourage programmers to test isinstance(x, sequencebase) when all
they *really* need is something that supports the __getitem__
protocol.  This would encourage writing code that is less polymorphic
than it could be, and that would be a real loss.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Fri Oct  5 18:18:17 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 5 Oct 2001 13:18:17 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
 <15293.57386.311536.729065@anthem.wooz.org>
 <3BBDE670.52DFB7F0@lemburg.com>
 <15293.59431.1516.201622@anthem.wooz.org>
 <15293.59241.954894.415782@grendel.zope.com>
Message-ID: <15293.60249.661659.388298@anthem.wooz.org>

>>>>> "Fred" == Fred L Drake, Jr <fdrake@acm.org> writes:

    >> Yes, I'm sure: we need a PEP first.  Ideally it would answer
    >> some of the questions raised above, and it would lay out the
    >> class hierarchy.

    Fred>   Sounds to me like you just volunteered.
    Fred>   And on that note, its time for some cheap Chinese food...

I think the drop in your BKPL (blood Kung Pao level) has affected your
hearing.  Please, hurry to boost it back up!

sound-of-one-chopstick-clapping-ly y'rs,
-Barry


From mclay@nist.gov  Fri Oct  5 18:28:06 2001
From: mclay@nist.gov (Michael McLay)
Date: Fri, 5 Oct 2001 13:28:06 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <20011005121931.B1801@cthulhu.mems-exchange.org>
References: <3BB204E3.23339.3A9F068C@localhost> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org>
Message-ID: <0110051328060C.02197@fermi.eeel.nist.gov>

On Friday 05 October 2001 12:19 pm, Greg Ward wrote:
> On 05 October 2001, M.-A. Lemburg said:

> The only import hook I'm familiar with is Quixote, where we have defined
> a Python dialect called PTL (Python Template Language, used to embed
> HTML [or other text] in Python).  Install Quixote's import hook, and you
> can import .ptl files just like .py files.  It's very handy.  Out of
> curiosity, does anyone know of any other import hooks like this out
> there -- ie. import something that is not strictly Python.  

The dpython interpreter I wrote this summer would tokenize .dp files that 
were in the Python path. If the alternative suffix was detected the Python 
interpreter would use the dpython tokenizer.  The dpython tokenizer 
would generate decimal numbers instead of binary numbers as the default for 
float and int literals.  I didn't use the import hooks.  I built the .dp file 
recognition into the interpreter at the point where it evaluates the suffix 
of the file name and compairs it to the filedesc struct.  

When I discussed the implementation of dpython at the ZPUG meeting I 
joked about extending the concept to add bytecode generators for JavaScript 
and Visual Basic modules.  The work required to add JavaScript would probably 
be reasonable.  It might help get Python used anywhere JavaScript is used.  

There has also been some interest at NIST and NASA in building modules to 
parse meta languages such as XML Schema, KIF, and Express and directly 
generate Python bytecodes.




From pedroni@inf.ethz.ch  Fri Oct  5 19:06:23 2001
From: pedroni@inf.ethz.ch (Samuele Pedroni)
Date: Fri, 5 Oct 2001 20:06:23 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org>
Message-ID: <001101c14dc8$754468a0$8a73fea9@newmexico>

Hi

[Greg Ward]
> > One question: should these ZIP-archives filenames be placed in 
> > sys.path or should Python scan for ZIP-archives within the
> > dirs on sys.path ?
> 
> I think the archive file should be listed in sys.path.

the latest jython code base (2.1a3) allow to put jar/zip
in the sys.path or reference of the form

jarfile!pkgdir1/pkgdir2

This are used to set meaningful __path__ attrs for the packages
and should allow to deal with code that change __path__ 
pointing to some relative subdir according to some context
property.

> Note that my experience of this in Java was largely negative, because
> Java doesn't have a standard way of putting .class files in the
> filesystem (AFAIK) -- so everything has to be in a .jar file, and those
> .jar files can be anywhere you please.  

I don't get the point :(

> So you end up with a mile-long
> CLASSPATH that's very fragile and forever needing fixing.  As long as
> most Python modules are accessed the ordinary way (files in a
> directory), then Python won't have a problem.  But if somebody makes a
> Python installation with "stdlib.zip", "distutils.zip",
> "mxDateTime.zip", etc. etc., then the poor users will be in the same
> boat as Java users.
> 


> The main concerns I have with scanning sys.path directories for ZIP
> files are performance and transparence.  (It ain't necessarily obvious
> where a particular module will come from if sys.path can be searched in
> two ways.)

I agree.

regards.



From mal@lemburg.com  Fri Oct  5 19:21:29 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 05 Oct 2001 20:21:29 +0200
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
 <15293.57386.311536.729065@anthem.wooz.org>
 <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org>
Message-ID: <3BBDFA29.A5A4DCBA@lemburg.com>


"Barry A. Warsaw" wrote:
> 
> >>>>> "M" == M  <mal@lemburg.com> writes:
> 
>     M> Are you sure about the -1 ? Using abstract base classes seems
>     M> like a natural thing to do in an OO-language like Python while
>     M> the isinstance() hack would not allow you to use your own
>     M> UserList subclass for code which was written with sequences as
>     M> input.
> 
> Yes, I'm sure: we need a PEP first.  Ideally it would answer some of
> the questions raised above, and it would lay out the class hierarchy.

Maybe I wasn't clear enough:

Of course we need a PEP for these names, their intended meaning
as object category abstraction and interface placeholder. Too late
for Python 2.2, but probably a nice project for 2.3.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From jim@interet.com  Fri Oct  5 19:47:31 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 05 Oct 2001 14:47:31 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org>
Message-ID: <3BBE0043.663D0411@interet.com>

Greg Ward wrote:
> 
> On 05 October 2001, M.-A. Lemburg said:

> > One question: should these ZIP-archives filenames be placed in
> > sys.path or should Python scan for ZIP-archives within the
> > dirs on sys.path ?
> 
> I think the archive file should be listed in sys.path.

I think that is correct.

At one time there was talk of sys.path containing importer
instances.  But currently (please check me here) sys.path
is purely strings, and these strings must be directory
names.  Python depends on directories and subdirectories
for its import semantics.  So any zip archive must support
subdirectories to support general imports.

Therefore, a string in sys.path which is a zip file is
equivalent to hanging a nameless subdirectory at that point.
"Nameless" means zip archive files *.pyc occur at the top level,
and the zip file may contain relative subdirectories with
files sub1/file1.pyc, etc.  So a zip file can always substitute
for an arbitrary subdirectory tree, and is functionally equivalent
to a directory name at the same place in sys.path.

JimA


From thomas.heller@ion-tof.com  Fri Oct  5 19:51:19 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 20:51:19 +0200
Subject: [Python-Dev] threading bug?
Message-ID: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook>

[Apologies, I also asked this on c.l.p, but got no
answer so far. I have not too much time left before
I will be away for nearly 2 weeks]

I wonder if this is a bug. I'm running a deamon thread doing
some work in the background, and sometimes when Python exits
I get tracebacks like this:

Unhandled exception in thread:
Traceback (most recent call last):
  File "c:\python21\lib\threading.py", line 393, in __bootstrap
    self.__stop()
  File "c:\python21\lib\threading.py", line 399, in __stop
    self.__block.notifyAll()
  File "c:\python21\lib\threading.py", line 235, in notifyAll
    self.notify(len(self.__waiters))
  File "c:\python21\lib\threading.py", line 217, in notify
    me = currentThread()
TypeError: object of type 'None' is not callable

This traceback does not appear when I register an atexit
function to stop the thread.

Is this a bug? Aren't daemon threads supposed to clean up
themselves?

Thomas



From guido@python.org  Fri Oct  5 19:54:26 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 14:54:26 -0400
Subject: [Python-Dev] threading bug?
In-Reply-To: Your message of "Fri, 05 Oct 2001 20:51:19 +0200."
 <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook>
References: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook>
Message-ID: <200110051854.OAA28426@cj20424-a.reston1.va.home.com>

> [Apologies, I also asked this on c.l.p, but got no
> answer so far. I have not too much time left before
> I will be away for nearly 2 weeks]
> 
> I wonder if this is a bug. I'm running a deamon thread doing
> some work in the background, and sometimes when Python exits
> I get tracebacks like this:
> 
> Unhandled exception in thread:
> Traceback (most recent call last):
>   File "c:\python21\lib\threading.py", line 393, in __bootstrap
>     self.__stop()
>   File "c:\python21\lib\threading.py", line 399, in __stop
>     self.__block.notifyAll()
>   File "c:\python21\lib\threading.py", line 235, in notifyAll
>     self.notify(len(self.__waiters))
>   File "c:\python21\lib\threading.py", line 217, in notify
>     me = currentThread()
> TypeError: object of type 'None' is not callable
> 
> This traceback does not appear when I register an atexit
> function to stop the thread.
> 
> Is this a bug? Aren't daemon threads supposed to clean up
> themselves?

This is probably a result of the destructive module finalization.
When Py_Finalize() is called, threads that are still running are not
killed.  But Py_Finalize() replaces the values of all globals in
modules with None (it could just clear the module's dictionary but
there was some reason for doing it this way) and that is what you are
probably seeing.

If there was a primitive to kill a thread, we might use it in
Py_Finalize() -- but there isn't (it's been requested but it's hard to
see how to do it safely).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Fri Oct  5 19:58:44 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 05 Oct 2001 14:58:44 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <085f01c14db7$06704720$e000a8c0@thomasnotebook>
Message-ID: <3BBE02E4.555D64BF@interet.com>

Thomas Heller wrote:

> Do you know about py2exe and installer? This is a variation of what they do.

I've heard of it, but where is it?  I never searched out the URL.
I just use freeze.  Anyway, I think we something in the core.

> BTW: I remember I having seen a ZipFileImporter using imputil, but
> cannot find it any more. Am I dreaming?

Either Greg Stein or I (I don't remember) wrote a ZipFileImporter
for imputil, and that is what I am currently using.

Jim


From thomas.heller@ion-tof.com  Fri Oct  5 20:04:00 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 21:04:00 +0200
Subject: [Python-Dev] threading bug?
References: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook>  <200110051854.OAA28426@cj20424-a.reston1.va.home.com>
Message-ID: <0afd01c14dd0$7e170a20$e000a8c0@thomasnotebook>

From: "Guido van Rossum" <guido@python.org>
> > [Apologies, I also asked this on c.l.p, but got no
> > answer so far. I have not too much time left before
> > I will be away for nearly 2 weeks]
> > 
> > I wonder if this is a bug. I'm running a deamon thread doing
> > some work in the background, and sometimes when Python exits
> > I get tracebacks like this:
> > 
> > Unhandled exception in thread:
> > Traceback (most recent call last):
> >   File "c:\python21\lib\threading.py", line 393, in __bootstrap
> >     self.__stop()
> >   File "c:\python21\lib\threading.py", line 399, in __stop
> >     self.__block.notifyAll()
> >   File "c:\python21\lib\threading.py", line 235, in notifyAll
> >     self.notify(len(self.__waiters))
> >   File "c:\python21\lib\threading.py", line 217, in notify
> >     me = currentThread()
> > TypeError: object of type 'None' is not callable
> > 
> > This traceback does not appear when I register an atexit
> > function to stop the thread.
> > 
> > Is this a bug? Aren't daemon threads supposed to clean up
> > themselves?
> 
> This is probably a result of the destructive module finalization.
> When Py_Finalize() is called, threads that are still running are not
> killed.  But Py_Finalize() replaces the values of all globals in
> modules with None (it could just clear the module's dictionary but
> there was some reason for doing it this way) and that is what you are
> probably seeing.
> 
> If there was a primitive to kill a thread, we might use it in
> Py_Finalize() -- but there isn't (it's been requested but it's hard to
> see how to do it safely).

But it seems the system is trying to stop the thread(s),
maybe too late. In my case, atexit.register(self.stop)
makes a clean shutdown.
My point is: should the threading module do this?
I can easily live with doing it myself - I just don't know
if this is intended or required.

Thomas



From gward@python.net  Fri Oct  5 20:07:04 2001
From: gward@python.net (Greg Ward)
Date: Fri, 5 Oct 2001 15:07:04 -0400
Subject: [Python-Dev] Making setup.py a little more flexible
In-Reply-To: <200110051702.NAA27256@cj20424-a.reston1.va.home.com>
References: <200110041712.TAA25396@pandora.informatik.hu-berlin.de> <20011005120535.A1801@cthulhu.mems-exchange.org> <200110051702.NAA27256@cj20424-a.reston1.va.home.com>
Message-ID: <20011005150704.A2640@cthulhu.mems-exchange.org>

On 05 October 2001, Guido van Rossum said:
> Strange.  When I make changes to Setup.local, they definitely work!

OK, I'll try to investigate more next week.  Off for Thanksgiving long
weekend now!

        Greg
-- 
Greg Ward - geek                                        gward@python.net
http://starship.python.net/~gward/
No animals were harmed in transmitting this message.


From thomas.heller@ion-tof.com  Fri Oct  5 20:09:58 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 21:09:58 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <085f01c14db7$06704720$e000a8c0@thomasnotebook> <3BBE02E4.555D64BF@interet.com>
Message-ID: <0b0501c14dd1$539d83e0$e000a8c0@thomasnotebook>

From: "James C. Ahlstrom" <jim@interet.com>
> Thomas Heller wrote:
> 
> > Do you know about py2exe and installer? This is a variation of what they do.
> 
> I've heard of it, but where is it?  I never searched out the URL.
> I just use freeze.  Anyway, I think we something in the core.
http://starship.python.net/crew/theller/py2exe, the project
is on sourceforge: http://sourceforge.net/projects/py2exe, so you
can browse the CVS repository.

Installer: http://www.mcmillan-inc.com/install1.html

Um, BTW: py2exe's code originated from bdist_wininst (the distutils
windows installer), which also contains C-code to unpack zip-files:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/distutils/misc/

Thomas



From guido@python.org  Fri Oct  5 20:09:43 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 15:09:43 -0400
Subject: [Python-Dev] threading bug?
In-Reply-To: Your message of "Fri, 05 Oct 2001 21:04:00 +0200."
 <0afd01c14dd0$7e170a20$e000a8c0@thomasnotebook>
References: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> <200110051854.OAA28426@cj20424-a.reston1.va.home.com>
 <0afd01c14dd0$7e170a20$e000a8c0@thomasnotebook>
Message-ID: <200110051909.PAA28498@cj20424-a.reston1.va.home.com>

> > > Unhandled exception in thread:
> > > Traceback (most recent call last):
> > >   File "c:\python21\lib\threading.py", line 393, in __bootstrap
> > >     self.__stop()
> > >   File "c:\python21\lib\threading.py", line 399, in __stop
> > >     self.__block.notifyAll()
> > >   File "c:\python21\lib\threading.py", line 235, in notifyAll
> > >     self.notify(len(self.__waiters))
> > >   File "c:\python21\lib\threading.py", line 217, in notify
> > >     me = currentThread()
> > > TypeError: object of type 'None' is not callable
> > > 
> > > This traceback does not appear when I register an atexit
> > > function to stop the thread.
> > > 
> > > Is this a bug? Aren't daemon threads supposed to clean up
> > > themselves?

> > This is probably a result of the destructive module finalization.
> > When Py_Finalize() is called, threads that are still running are not
> > killed.  But Py_Finalize() replaces the values of all globals in
> > modules with None (it could just clear the module's dictionary but
> > there was some reason for doing it this way) and that is what you are
> > probably seeing.
> > 
> > If there was a primitive to kill a thread, we might use it in
> > Py_Finalize() -- but there isn't (it's been requested but it's hard to
> > see how to do it safely).

> But it seems the system is trying to stop the thread(s),
> maybe too late. In my case, atexit.register(self.stop)
> makes a clean shutdown.
> My point is: should the threading module do this?
> I can easily live with doing it myself - I just don't know
> if this is intended or required.

Python doesn't attempt to stop daemon threads at all.  Note that the
__stop() call is in a finally clause.  This probably gets hit because
some other exception happened (caused by the same problem of globals
being replaced by None).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Fri Oct  5 20:13:44 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 05 Oct 2001 15:13:44 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com>
Message-ID: <3BBE0668.4DF5038C@interet.com>

"M.-A. Lemburg" wrote:
> 
> "James C. Ahlstrom" wrote:
> > 3) Package import is not implemented in C, it is implemented
> >    by a Python module which is part of the standard library.
> 
> Why ? I don't see a need for that and it would slow down
> package imports which are becoming increasingly popular as the
> Python code base grows.

Well, I looked at import.c and adding zip files with subdirectories
will be a mess.  The path searches would have to be extended to zip files.
Moving out the package code would be a great simplification.

Or maybe I am lazy.  It doesn't help that I don't really
understand packages.  Your point about efficiency is a good one
though, but I am not sure of the magnitude of the slow down.

Anyone else have an opinion?

JimA


From skip@pobox.com (Skip Montanaro)  Fri Oct  5 20:13:49 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 5 Oct 2001 14:13:49 -0500
Subject: [Python-Dev] os.spawnv[e] and related constants
Message-ID: <15294.1645.63995.884978@beluga.mojam.com>

The spawnv and spawnve functions in the os module look useful, but the lib
manual documentation is rather sparse about their semantics, referring the
reader to the VC++ Runtime Library for more info.  Perhaps this is poetic
justice for us Unix bigots (since many people often complain that the Python
library and docs are too Unix-centric), but it would be useful to know what
the return values (if any) of these functions are and what the related P_*
constants mean.  I can guess about P_WAIT and P_NOWAIT, but the meanings of
the others aren't obvious to a person without Windows experience.

Skip


From tim.one@home.com  Fri Oct  5 20:16:29 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 5 Oct 2001 15:16:29 -0400
Subject: [Python-Dev] threading bug?
In-Reply-To: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEIFLPAA.tim.one@home.com>

[Thomas Heller]
> Unhandled exception in thread:
> Traceback (most recent call last):
>   File "c:\python21\lib\threading.py", line 393, in __bootstrap
>     self.__stop()
>   File "c:\python21\lib\threading.py", line 399, in __stop
>     self.__block.notifyAll()
>   File "c:\python21\lib\threading.py", line 235, in notifyAll
>     self.notify(len(self.__waiters))
>   File "c:\python21\lib\threading.py", line 217, in notify
>     me = currentThread()
> TypeError: object of type 'None' is not callable
>
> This traceback does not appear when I register an atexit
> function to stop the thread.

We're missing basic info, like which OS and which Python release.  More than
one shutdown race has been fixed over the decades <wink>.

I've seen reports of this before, but under older Pythons, and/or in
conjunction with running GUIs where the problem went away if the GUI was
taken out of the equation.  AFAIK, nobody ever stuck with it long enough to
track down the real cause, though.

This relatively recent comment in pythonrun.c's Py_Finalize() is probably
relevant:

/* The interpreter is still entirely intact at this point, and the
 * exit funcs may be relying on that.  In particular, if some thread
 * or exit func is still waiting to do an import, the import machinery
 * expects Py_IsInitialized() to return true.  So don't say the
 * interpreter is uninitialized until after the exit funcs have run.
 * Note that Threading.py uses an exit func to do a join on all the
 * threads created thru it, so this also protects pending imports in
 * the threads created via Threading.
 */
call_sys_exitfunc();
initialized = 0;


I expect that a *foreign* thread (== one not created by threading.py) isn't
going to be waited for by threading.py's exit func .join(), and if so won't
be protected by this mechanism (that's a shutdown race we can't really do
anything about -- we've got no handle on what foreign threads do).



From gmcm@hypernet.com  Fri Oct  5 20:20:33 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Fri, 5 Oct 2001 15:20:33 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <20011005121931.B1801@cthulhu.mems-exchange.org>
References: <3BBDDA83.B28556CB@lemburg.com>
Message-ID: <3BBDCFC1.26732.68AF7945@localhost>

[Greg Ward]

> The only import hook I'm familiar with is Quixote, where we have
> defined a Python dialect called PTL (Python Template Language,
> used to embed HTML [or other text] in Python).  Install Quixote's
> import hook, and you can import .ptl files just like .py files. 
> It's very handy.  Out of curiosity, does anyone know of any other
> import hooks like this out there -- ie. import something that is
> not strictly Python.  (It sounds like most import hooks/hacks
> deal with the location of the .py files to import.  Quixote
> doesn't touch that, but it adds the ability to import
> not-quite-Python source files.)

I know of import hooks like that, but in 1000+ emails from 
Installer users, no one has ever attempted to distribute an app 
that uses one. If the hook writes out a .pyc, then the hook is 
pretty much developer-only, anyway.
 

> Note that my experience of this in Java was largely negative,
> because Java doesn't have a standard way of putting .class files
> in the filesystem (AFAIK) -- so everything has to be in a .jar
> file, and those .jar files can be anywhere you please.  

Huh? Jar files don't date from day 1 - for .class files it works 
almost like Python.

> So you
> end up with a mile-long CLASSPATH that's very fragile and forever
> needing fixing.  As long as most Python modules are accessed the
> ordinary way (files in a directory), then Python won't have a
> problem.  But if somebody makes a Python installation with
> "stdlib.zip", "distutils.zip", "mxDateTime.zip", etc. etc., then
> the poor users will be in the same boat as Java users.

I think the only difference is that Python goes to considerable 
effort (and expense) to work out a sys.path before considering 
PYTHONPATH.
 


- Gordon


From gmcm@hypernet.com  Fri Oct  5 20:20:32 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Fri, 5 Oct 2001 15:20:32 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <3BBDD254.32DC4071@interet.com>
Message-ID: <3BBDCFC0.18517.68AF780F@localhost>

[James C. Ahlstrom]

> My interest in imputil and ImportManager is to implement jar
> files, and neither does this well.  The problem is always how to
> get the SpecialImporter.py imported first, and how to hack the
> imports that the SpecialImporter.py may need to import.  In
> iu4.py we see the use of posix.stat to replace the os module, and
> the import of only built-in modules sys, imp, and marshal.  The
> same is true of imputil.py.

Here's my bootstrap code (archive_rt knows how to import 
from .pyz archives):

import archive_rt, iu4
iu4._globalownertypes.insert(0, archive_rt.PYZOwner)
iu4.ImportManager().install()

Now a .pyz on sys.path will be searched. Nothing hard about 
doing this for a .zip. MacPython & Jython do it.

> I admit I don't really understand package imports.  But I feel
> they are an unsolved problem, because package authors always seem
> to write custom ones.

Most of what I see is namespace trickery, not custom 
importers.
 
> A first-class solution would have these features IMHO:
> 1) Access to libraries in zip files is implemented in C (I
> volunteer). 2) All Python library modules can be imported by C
> code,
>    either from directories or zip files.

With Thomas's changes to PyImport_ImportModuleEx, this is 
largely taken care of (it gets routed through __import__). So 
with the above bootstrap code, C code will import from my 
.pyz as well.

> 3) Package import is not implemented in C, it is implemented
>    by a Python module which is part of the standard library.

The rules aren't documented, but the essentials are pretty 
straightforward. I'd like to see them changed to draw a 
distinction between relative imports and absolute imports 
(since that's a constant source of gotcha's).

However, there are import hacks out there that rely on 
undocumented (and probably accidental) features of 
implementation.

> 4)
> If an import of A.B is attempted, and a function
>    A.MyCustomImporter() exists, then that is used instead of the
>    default.

In iu4, A is welcome to take over all imports below A. If 
A.__init__ does not do so, the standard mechanism is used.

> 5) More or better ideas from you all.



- Gordon


From fdrake@acm.org  Fri Oct  5 20:17:57 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 15:17:57 -0400 (EDT)
Subject: [Python-Dev] os.spawnv[e] and related constants
In-Reply-To: <15294.1645.63995.884978@beluga.mojam.com>
References: <15294.1645.63995.884978@beluga.mojam.com>
Message-ID: <15294.1893.967184.997647@cj42289-a.reston1.va.home.com>

Skip Montanaro writes:
 > The spawnv and spawnve functions in the os module look useful, but the lib
 > manual documentation is rather sparse about their semantics, referring the
 > reader to the VC++ Runtime Library for more info.  Perhaps this is poetic

  Are you looking at the development docs, or the 2.1.1 docs?

    http://python.sourceforge.net/devel-docs/lib/lib.html

  If you're looking at the devel docs and still find things are
missing, please file a bug report with specific info about what you
think is missing, and assign it to me.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From thomas.heller@ion-tof.com  Fri Oct  5 20:34:14 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 5 Oct 2001 21:34:14 +0200
Subject: [Python-Dev] threading bug?
References: <LNBBLJKPBEHFEDALKOLCOEIFLPAA.tim.one@home.com>
Message-ID: <0b7e01c14dd4$b799a5b0$e000a8c0@thomasnotebook>

From: "Tim Peters" <tim.one@home.com>
> [Thomas Heller]
> > Unhandled exception in thread:
> > Traceback (most recent call last):
> >   File "c:\python21\lib\threading.py", line 393, in __bootstrap
> >     self.__stop()
> >   File "c:\python21\lib\threading.py", line 399, in __stop
> >     self.__block.notifyAll()
> >   File "c:\python21\lib\threading.py", line 235, in notifyAll
> >     self.notify(len(self.__waiters))
> >   File "c:\python21\lib\threading.py", line 217, in notify
> >     me = currentThread()
> > TypeError: object of type 'None' is not callable
> >
> > This traceback does not appear when I register an atexit
> > function to stop the thread.
> 
> We're missing basic info, like which OS and which Python release.  More than
> one shutdown race has been fixed over the decades <wink>.
> 
> I've seen reports of this before, but under older Pythons, and/or in
> conjunction with running GUIs where the problem went away if the GUI was
> taken out of the equation.  AFAIK, nobody ever stuck with it long enough to
> track down the real cause, though.
Here's the short info (I can provide everything to produce this,
but not today).
ActiveState's Python 2.1.1, running under Win2k SP1.
Haven't tried in other Python versions.
No gui involved.
The thread basically calls os.stat, not sure if it also calls reload().

Later,

Thomas



From tim.one@home.com  Fri Oct  5 20:42:48 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 5 Oct 2001 15:42:48 -0400
Subject: [Python-Dev] os.spawnv[e] and related constants
In-Reply-To: <15294.1645.63995.884978@beluga.mojam.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEIJLPAA.tim.one@home.com>

[Skip Montanaro]
> The spawnv and spawnve functions in the os module look useful, but the
> lib manual documentation is rather sparse about their semantics,
> referring the reader to the VC++ Runtime Library for more info.

Gigabytes of MS docs (including all the VC docs) are available online for
free:

    http://msdn.microsoft.com/library/

Since the MS docs are copyrighted, we can't just copy them into our docs.

> ...
> I can guess about P_WAIT and P_NOWAIT, but the
> meanings of the others aren't obvious to a person without Windows
> experience.

They aren't obvious to anyone, and if you want to use the non-obvious ones,
there are reams of MS docs you should understand first:

<http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/H
TML/_crt__spawn.2c_._wspawn_functions.asp>

How is Fred going to summarize all that (all of it is necessary in the
end!), without running afoul of copyright?

Alas, it can take a long time to learn how to find this stuff (the doc
collection is massive).  In general, the place to start for C library docs
is:

Visual Tools and Languages ->
Visual Studio 6.0 ->
Vlsual C and C++ 6.0 ->
Product Documentation ->
Visual C++ Programmer's Guide ->
Run-Time Library Reference



From tim.one@home.com  Fri Oct  5 20:48:20 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 5 Oct 2001 15:48:20 -0400
Subject: [Python-Dev] threading bug?
In-Reply-To: <0b7e01c14dd4$b799a5b0$e000a8c0@thomasnotebook>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEIKLPAA.tim.one@home.com>

[Thomas Heller]
> Here's the short info (I can provide everything to produce this,
> but not today).

You overestimate my interest <wink>.

> ActiveState's Python 2.1.1, running under Win2k SP1.
> Haven't tried in other Python versions.
> No gui involved.
> The thread basically calls os.stat, not sure if it also calls reload().

If you can produce a reasonably small failing program, that doesn't involve
foreign threads (as before, we've got no handle on those), attach it to a
new bug report on SourceForge.  If I can reproduce it in core Python, I'll
find a way to fix it; else it's unique to ActiveState's distro, and they'll
have to tackle it.



From guido@python.org  Fri Oct  5 20:50:23 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 15:50:23 -0400
Subject: [Python-Dev] threading bug?
In-Reply-To: Your message of "Fri, 05 Oct 2001 15:48:20 EDT."
 <LNBBLJKPBEHFEDALKOLCAEIKLPAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCAEIKLPAA.tim.one@home.com>
Message-ID: <200110051950.PAA31207@cj20424-a.reston1.va.home.com>

> If you can produce a reasonably small failing program, that doesn't involve
> foreign threads (as before, we've got no handle on those), attach it to a
> new bug report on SourceForge.  If I can reproduce it in core Python, I'll
> find a way to fix it; else it's unique to ActiveState's distro, and they'll
> have to tackle it.

I think he mentioned it was a daemon thread, and those are not stopped
-- they are let run just like foreign threads.  I still believe in my
theory which I posted earlier today.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Fri Oct  5 20:58:44 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 5 Oct 2001 15:58:44 -0400
Subject: [Python-Dev] threading bug?
In-Reply-To: <200110051950.PAA31207@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEIKLPAA.tim.one@home.com>

[Guido]
> I think he mentioned it was a daemon thread,

He wasn't clear about that.

> and those are not stopped -- they are let run just like foreign threads.

I'm going to redefine what I mean by "foreign" then, to cover this too.

> I still believe in my theory which I posted earlier today.

So do I.


From skip@pobox.com (Skip Montanaro)  Fri Oct  5 21:37:59 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 5 Oct 2001 15:37:59 -0500
Subject: [Python-Dev] os.spawnv[e] and related constants
In-Reply-To: <15294.1893.967184.997647@cj42289-a.reston1.va.home.com>
References: <15294.1645.63995.884978@beluga.mojam.com>
 <15294.1893.967184.997647@cj42289-a.reston1.va.home.com>
Message-ID: <15294.6695.493437.770967@beluga.mojam.com>

    >> The spawnv and spawnve functions in the os module look useful, ...

    Fred>   Are you looking at the development docs, or the 2.1.1 docs?

I was looking at the 2.1.1 docs, but the development docs have the same
problem w.r.t. the P_* constants.  (The spawn* docs look much better.)  I
filed a bug report about the P_* documentation.

Skip


From skip@pobox.com (Skip Montanaro)  Fri Oct  5 21:50:13 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 5 Oct 2001 15:50:13 -0500
Subject: [Python-Dev] os.spawnv[e] and related constants
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIJLPAA.tim.one@home.com>
References: <15294.1645.63995.884978@beluga.mojam.com>
 <LNBBLJKPBEHFEDALKOLCEEIJLPAA.tim.one@home.com>
Message-ID: <15294.7429.716150.204682@beluga.mojam.com>

>>>>> "Tim" == Tim Peters <tim.one@home.com> writes:

    Tim> [Skip Montanaro]
    >> The spawnv and spawnve functions in the os module look useful, but
    >> the lib manual documentation is rather sparse about their semantics,
    >> referring the reader to the VC++ Runtime Library for more info.

    Tim> Gigabytes of MS docs (including all the VC docs) are available
    Tim> online for free:

    Tim>     http://msdn.microsoft.com/library/

    Tim> Since the MS docs are copyrighted, we can't just copy them into our
    Tim> docs.
    ...
    Tim> <http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/HTML/_crt__spawn.2c_._wspawn_functions.asp>

Perhaps the Python docs could have a "See Also" that refers to this page?

    Tim> How is Fred going to summarize all that (all of it is necessary in
    Tim> the end!), without running afoul of copyright?

It appears that in the development docs Fred has done a good job of
summarizing the spawn functions' semantics.  It's only the P_* constants
whose definition is still left unclear.  Copyright has the notion of "fair
use".  Since the information extracted would be so small w.r.t. the whole
work from which it was extracted and it would not be used for commercial
gain (at least, not directly), I would think that describing the five
constants would fall under fair use guidelines.

    Tim> Alas, it can take a long time to learn how to find this stuff (the
    Tim> doc collection is massive).  In general, the place to start for C
    Tim> library docs is:

Thanks, but I'm not really interested in programming Windows, just knowing
what the stuff imported from Windows means. ;-)

Skip


From guido@python.org  Fri Oct  5 22:01:22 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 05 Oct 2001 17:01:22 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Fri, 05 Oct 2001 20:21:29 +0200."
 <3BBDFA29.A5A4DCBA@lemburg.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org>
 <3BBDFA29.A5A4DCBA@lemburg.com>
Message-ID: <200110052101.RAA02367@cj20424-a.reston1.va.home.com>

Now that the introduction of a class hierarchy is dead, I'd like to
move forward with the isinstance(x, (A, B, C...)) patch.

Followup question: should I also implement issubclass(X, (A, B, C...))?

It's consistent to do so, but it's more work (issubclass is a bit
murkier due to the "abstract subclass" support), and I'm not sure that
it has the same benefits...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Fri Oct  5 23:33:47 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 5 Oct 2001 18:33:47 -0400
Subject: [Python-Dev] Painful death in debug build
Message-ID: <LNBBLJKPBEHFEDALKOLCKEJGLPAA.tim.one@home.com>

test_descr.py crashes under the debug build now.  I've got too many things
piled on my stack to look into it right now, so it would be good if someone
could try to feel debug pain on Linux too.

The top of the call stack is

_free_dbg_lk(void * 0x009e5fd0, int 1) line 1066 + 60 bytes
_free_dbg(void * 0x009e5fd0, int 1) line 1001 + 13 bytes
free(void * 0x009e5fd0) line 956 + 11 bytes
_PyObject_GC_Del(_object * 0x009e5fdc) line 873 + 10 bytes
long_dealloc(_object * 0x009e5fdc) line 1230 + 16 bytes
subtype_dealloc(_object * 0x009e5fdc) line 301 + 7 bytes
_Py_Dealloc(_object * 0x009e5fdc) line 1738 + 7 bytes
eval_frame(_frame * 0x00803d7c) line 845 + 59 bytes
PyEval_EvalCodeEx(...)

at this point, so first guess is that it has to do with adding new-style
instances to GC.  The MS debug library free() is complaining because it
detects damage (corruption) in the pad bytes debug malloc() writes *between*
the blocks it hands out.  So it's unlikely that it points to the direct
cause; it's just detecting that user code, somewhere, sometime, wrote over
memory it had no business writing to.



From martin@loewis.home.cs.tu-berlin.de  Sat Oct  6 00:54:03 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Sat, 6 Oct 2001 01:54:03 +0200
Subject: [Python-Dev] Painful death in debug build
Message-ID: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de>

> test_descr.py crashes under the debug build now.  I've got too many
> things piled on my stack to look into it right now, so it would be
> good if someone could try to feel debug pain on Linux too.

Configuring Python with --with-pydebug, I cannot get test_descr to
crash on Linux, using the current CVS.

Regards,
Martin


From dariob@millic.com.ar  Sat Oct  6 01:19:01 2001
From: dariob@millic.com.ar (dariob@millic.com.ar)
Date: Fri, 05 Oct 2001 21:19:01 -0300
Subject: [Python-Dev] VCD Y MP3.
Message-ID: <04fc632160006a1MAIL1@smtp1.millic.com.ar>

This is a Multipart MIME message.

------=_NextPart_000_001__2771585_76741,02
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 7bit













------=_NextPart_000_001__2771585_76741,02
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: base64

PGh0bWw+DQoNCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1MYW5ndWFnZSIg
Y29udGVudD0iZXMiPg0KPG1ldGEgbmFtZT0iR0VORVJBVE9SIiBjb250ZW50PSJNaWNyb3Nv
ZnQgRnJvbnRQYWdlIDUuMCI+DQo8bWV0YSBuYW1lPSJQcm9nSWQiIGNvbnRlbnQ9IkZyb250
UGFnZS5FZGl0b3IuRG9jdW1lbnQiPg0KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBl
IiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9d2luZG93cy0xMjUyIj4NCjx0aXRsZT5Q
YWdpbmEgbnVldmEgMTwvdGl0bGU+DQo8L2hlYWQ+DQoNCjxib2R5Pg0KDQo8dGFibGUgaGVp
Z2h0PSI4MSIgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0iMTAwJSIg
YmdDb2xvcj0iIzAwMDAwMCIgYm9yZGVyPSIwIj4NCiAgPGZvbnQgZmFjZT0iQXJpYWwsIEhl
bHZldGljYSwgc2Fucy1zZXJpZiIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjQiPjxiPg0KICA8
Y2FwdGlvbj48Zm9udCBmYWNlPSJUYWhvbWEiIGNvbG9yPSIjZmYwMDAwIiBzaXplPSI2Ij48
c3Ryb25nPqGhoaEgTk8gVEUgUE9ERVMgDQogIFBFUkRFUiBFU1RBIE9QT1JUVU5JREFEISEh
ITwvc3Ryb25nPjwvZm9udD4gPHN0cm9uZz4NCiAgPGZvbnQgZmFjZT0iVGFob21hIiBjb2xv
cj0iIzAwMDBmZiIgc2l6ZT0iNSI+PGZvbnQgc2l6ZT0iNSIgZmFjZT0iVGFob21hIj4NCiAg
RVNDVUNIQSA8L2ZvbnQ+PC9mb250Pjwvc3Ryb25nPjxmb250IHNpemU9IisyIiBjb2xvcj0i
IzAwMDBmZiIgZmFjZT0iVGFob21hIj4NCiAgTdpTSUNBIE1QMyBFTiBUVSBBVVRPLCBDQVNB
LCBDQU1JTkFORE8gTyBFTiBET05ERSBQUkVGSUVSQVMsPC9mb250PjxzdHJvbmc+PGZvbnQg
ZmFjZT0iVGFob21hIiBjb2xvcj0iIzAwMDBmZiIgc2l6ZT0iNSI+PGZvbnQgc2l6ZT0iNSIg
ZmFjZT0iVGFob21hIj4NCiAgWSBNSVJBIExBUyBNRUpPUkVTIDwvZm9udD48Zm9udCBzaXpl
PSI1IiBmYWNlPSJUYWhvbWEiPg0KICBQRUzNQ1VMQVMgRU4gRUwgVFYgREUgVFUgQ0FTQS48
L2ZvbnQ+PC9mb250Pjwvc3Ryb25nPg0KICA8Zm9udCBmYWNlPSJUYWhvbWEiIGNvbG9yPSIj
MDAwMGZmIiBzaXplPSI1Ij48c3Ryb25nPjx1PlPaUEVSIE9GRVJUQSBFTiBMQSANCiAgUFVF
UlRBIERFIFRVIENBU0E8L3U+PC9zdHJvbmc+PC9mb250PjwvYj48L2ZvbnQ+DQogIDxwIGFs
aWduPSJjZW50ZXIiPg0KICA8Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNl
cmlmIiBjb2xvcj0iIzAwODBmZiIgc2l6ZT0iNCI+PGI+TG8gTWVqb3IgDQogIGRlbCBNZXJj
YWRvIGEgUHJlY2lvIGRlIExPQ09TPC9iPjwvZm9udD48L3A+DQogIDwvY2FwdGlvbj4NCiAg
PHRyPg0KICAgIDx0ZCBoZWlnaHQ9IjgxIj4NCiAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAi
IGNlbGxQYWRkaW5nPSI2IiB3aWR0aD0iMTAwJSIgYm9yZGVyPSIwIiBoZWlnaHQ9IjcxIj4N
CiAgICAgIDx0cj4NCiAgICAgICAgPHRkIHdpZHRoPSI1MyUiIGhlaWdodD0iNTkiPg0KICAg
ICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIzIiB3aWR0aD0iMTAw
JSIgYmdDb2xvcj0iIzAwODAwMCIgYm9yZGVyPSIwIj4NCiAgICAgICAgICA8dHI+DQogICAg
ICAgICAgICA8dGQgd2lkdGg9IjEwMCUiPg0KICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQt
c3BhY2luZzogMHB4OyBtYXJnaW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFy
Z2luLXRvcDogMHB4OyBtYXJnaW4tYm90dG9tOiA2cHgiIGFsaWduPSJjZW50ZXIiPg0KICAg
ICAgICAgICAgPGZvbnQgZmFjZT0iQXJpYWwgQmxhY2siIGNvbG9yPSIjZmZmZjAwIj5PZmVy
dGEgbGltaXRhZGEsIGhhc3RhIA0KICAgICAgICAgICAgYWdvdGFyIFN0b2NrLjwvZm9udD48
L3A+DQogICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdpbi1s
ZWZ0OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdpbi1i
b3R0b206IDZweCIgYWxpZ249ImNlbnRlciI+DQogICAgICAgICAgICA8Yj48Zm9udCBmYWNl
PSJBcmlhbCIgY29sb3I9IiNmZmZmMDAiIHNpemU9IjIiPk11Y2hvcyB5YSBsbyB0aWVuZW4s
IA0KICAgICAgICAgICAgZXN0YSBvcG9ydHVuaWRhZCBubyBzZSByZXBldGly4SB5IGxhcyB1
bmlkYWRlcyBzZSBlc3RhbiBhY2FiYW5kbywgbm8gDQogICAgICAgICAgICBzZSB2ZW5kZSBw
b3Igb3RybyBtZWRpbywgaGF6IHR1IG9mZXJ0YSBwdWVzIG1h8WFuYSBwdWVkZSBzZXIgdGFy
ZGUuPC9mb250PjwvYj48L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+
DQogICAgICAgIDwvdGQ+DQogICAgICA8L3RyPg0KICAgIDwvdGFibGU+DQogICAgPC90ZD4N
CiAgPC90cj4NCjwvdGFibGU+DQo8dGFibGUgaGVpZ2h0PSIxNTQiIGNlbGxTcGFjaW5nPSIw
IiBjZWxsUGFkZGluZz0iMCIgd2lkdGg9IjU4NCIgYm9yZGVyPSIwIj4NCiAgPHRyPg0KICAg
IDx0ZCB2QWxpZ249InRvcCIgd2lkdGg9IjI3MSIgaGVpZ2h0PSIxNTQiPg0KICAgIDx0YWJs
ZSBoZWlnaHQ9IjQxNSIgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0i
MjA3IiBib3JkZXI9IjAiPg0KICAgICAgPHRyPg0KICAgICAgICA8dGQgd2lkdGg9IjE5NyIg
aGVpZ2h0PSI0MTUiPg0KICAgICAgICA8dGFibGUgYm9yZGVyQ29sb3I9IiMwMDgwZmYiIGhl
aWdodD0iMzQxIiBjZWxsU3BhY2luZz0iMCIgY2VsbFBhZGRpbmc9IjAiIHdpZHRoPSIxMDAl
IiBib3JkZXI9IjIiPg0KICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgIDx0ZCB3aWR0aD0i
MTAwJSIgaGVpZ2h0PSIzMSI+DQogICAgICAgICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAi
IGNlbGxQYWRkaW5nPSI1IiB3aWR0aD0iMTAwJSIgYmdDb2xvcj0iIzAwODBmZiIgYm9yZGVy
PSIwIj4NCiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0i
MTAwJSI+PGZvbnQgZmFjZT0iVGFob21hIiBjb2xvcj0iI2ZmZmZmZiIgc2l6ZT0iMyI+DQog
ICAgICAgICAgICAgICAgPGI+Q2FyYWN0ZXLtc3RpY2FzPC9iPjwvZm9udD48L3RkPg0KICAg
ICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgPC90YWJsZT4NCiAgICAgICAgICAgIDwv
dGQ+DQogICAgICAgICAgPC90cj4NCiAgICAgICAgICA8dHI+DQogICAgICAgICAgICA8dGQg
d2lkdGg9IjEwMCUiIGhlaWdodD0iMzA2Ij4NCiAgICAgICAgICAgIDx0YWJsZSBoZWlnaHQ9
IjMwMiIgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSI1IiB3aWR0aD0iMTAwJSIgYm9y
ZGVyPSIwIj4NCiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0
aD0iMTAwJSIgaGVpZ2h0PSIyOTYiPg0KICAgICAgICAgICAgICAgIDx0YWJsZSBjZWxsU3Bh
Y2luZz0iMCIgY2VsbFBhZGRpbmc9IjAiIHdpZHRoPSIxMDAlIiBib3JkZXI9IjAiPg0KICAg
ICAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgICAgICA8dGQgd2lkdGg9IjEw
MCUiPg0KICAgICAgICAgICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7
IG1hcmdpbi1sZWZ0OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7
IG1hcmdpbi1ib3R0b206IDVweCI+DQogICAgICAgICAgICAgICAgICAgIDxmb250IGZhY2U9
IlRhaG9tYSIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjIiPjxiPlVuaWNvIGVuIA0KICAgICAg
ICAgICAgICAgICAgICBBcmdlbnRpbmEgcXVlIGxlZSBjZCBjb24gTVAzIHkgYWRlbWFzIGxl
ZSBWQ0QgY29uIHNhbGlkYSBkZSANCiAgICAgICAgICAgICAgICAgICAgdmlkZW8geSBhdWRp
byBvIHNlYSBxdWUmbmJzcDtzZSB2ZW4gZW4gZWwgdGVsZXZpc29yLCBlcyBsbyB1bHRpbW8g
DQogICAgICAgICAgICAgICAgICAgIGVuIHRlY25vbG9naWEgcGFyYSBtcDMgeSB2Y2Q8L2I+
PC9mb250PjwvcD4NCiAgICAgICAgICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQtc3BhY2lu
ZzogMHB4OyBtYXJnaW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFyZ2luLXRv
cDogMHB4OyBtYXJnaW4tYm90dG9tOiA1cHgiPg0KICAgICAgICAgICAgICAgICAgICA8Zm9u
dCBmYWNlPSJUYWhvbWEiIGNvbG9yPSIjMDA4MGZmIiBzaXplPSIyIj48Yj5FbCANCiAgICAg
ICAgICAgICAgICAgICAgcmVwcm9kdWN0b3IgZGUgTVAzIHkgVkNEIGluY2x1eWU6PC9iPjwv
Zm9udD48L3A+DQogICAgICAgICAgICAgICAgICAgIDxwIHN0eWxlPSJ3b3JkLXNwYWNpbmc6
IDBweDsgbWFyZ2luLWxlZnQ6IDBweDsgbWFyZ2luLXJpZ2h0OiAwcHg7IG1hcmdpbi10b3A6
IDBweDsgbWFyZ2luLWJvdHRvbTogNXB4Ij4NCiAgICAgICAgICAgICAgICAgICAgPGZvbnQg
ZmFjZT0iVGFob21hIiBjb2xvcj0iIzAwODBmZiIgc2l6ZT0iMiI+PGI+LSBDb250cm9sIA0K
ICAgICAgICAgICAgICAgICAgICByZW1vdG8gaW5hbGFtYnJpY288L2I+PC9mb250PjwvcD4N
CiAgICAgICAgICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQtc3BhY2luZzogMHB4OyBtYXJn
aW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFyZ2luLXRvcDogMHB4OyBtYXJn
aW4tYm90dG9tOiA1cHgiPg0KICAgICAgICAgICAgICAgICAgICA8Zm9udCBmYWNlPSJUYWhv
bWEiIGNvbG9yPSIjMDA4MGZmIiBzaXplPSIyIj48Yj4tIEJhdGVyaWFzIA0KICAgICAgICAg
ICAgICAgICAgICAoUGlsYXMpIFJlY2FyZ2FibGVzIExpdGl1bTwvYj48L2ZvbnQ+PC9wPg0K
ICAgICAgICAgICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdp
bi1sZWZ0OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdp
bi1ib3R0b206IDVweCI+DQogICAgICAgICAgICAgICAgICAgIDxmb250IGZhY2U9IlRhaG9t
YSIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjIiPjxiPi0gQ2FibGVzIGRlIA0KICAgICAgICAg
ICAgICAgICAgICBBdWRpbyB5IGRlIFZpZGVvPC9iPjwvZm9udD48L3A+DQogICAgICAgICAg
ICAgICAgICAgIDxwIHN0eWxlPSJ3b3JkLXNwYWNpbmc6IDBweDsgbWFyZ2luLWxlZnQ6IDBw
eDsgbWFyZ2luLXJpZ2h0OiAwcHg7IG1hcmdpbi10b3A6IDBweDsgbWFyZ2luLWJvdHRvbTog
NXB4Ij4NCiAgICAgICAgICAgICAgICAgICAgPGZvbnQgZmFjZT0iVGFob21hIiBjb2xvcj0i
IzAwODBmZiIgc2l6ZT0iMiI+PGI+LSBBdWRpZm9ubzwvYj48L2ZvbnQ+PC9wPg0KICAgICAg
ICAgICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdpbi1sZWZ0
OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdpbi1ib3R0
b206IDVweCI+DQogICAgICAgICAgICAgICAgICAgIDxmb250IGZhY2U9IlRhaG9tYSIgY29s
b3I9IiMwMDgwZmYiIHNpemU9IjIiPjxiPi0gTWFudWFsIGRlIA0KICAgICAgICAgICAgICAg
ICAgICBpbnN0cnVjY2lvbjwvYj48L2ZvbnQ+PC9wPg0KICAgICAgICAgICAgICAgICAgICA8
cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdpbi1sZWZ0OiAwcHg7IG1hcmdpbi1y
aWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdpbi1ib3R0b206IDVweCI+DQogICAg
ICAgICAgICAgICAgICAgIDxmb250IGZhY2U9IlRhaG9tYSIgY29sb3I9IiMwMDgwZmYiIHNp
emU9IjIiPjxiPi0gDQogICAgICAgICAgICAgICAgICAgIFRyYW5zZm9ybWFkb3I8L2I+PC9m
b250PjwvcD4NCiAgICAgICAgICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQtc3BhY2luZzog
MHB4OyBtYXJnaW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFyZ2luLXRvcDog
MHB4OyBtYXJnaW4tYm90dG9tOiA1cHgiPg0KICAgICAgICAgICAgICAgICAgICA8Zm9udCBm
YWNlPSJUYWhvbWEiIGNvbG9yPSIjMDA4MGZmIiBzaXplPSIyIj48Yj4tIENhcmdhZG9yPC9i
PjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAg
ICA8L3RhYmxlPg0KICAgICAgICAgICAgICAgIDwvdGQ+DQogICAgICAgICAgICAgIDwvdHI+
DQogICAgICAgICAgICA8L3RhYmxlPg0KICAgICAgICAgICAgPC90ZD4NCiAgICAgICAgICA8
L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L3RkPg0KICAgICAgPC90cj4NCiAg
ICA8L3RhYmxlPg0KICAgIDwvdGQ+DQogICAgPHRkIHdpZHRoPSIzMjciIGhlaWdodD0iMTU0
Ij4NCiAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0i
MTAwJSIgYm9yZGVyPSIwIj4NCiAgICAgIDx0cj4NCiAgICAgICAgPHRkIHdpZHRoPSIxMDAl
Ij4NCiAgICAgICAgPHRhYmxlIGNlbGxTcGFjaW5nPSIwIiBjZWxsUGFkZGluZz0iMCIgd2lk
dGg9IjEwMCUiIGJvcmRlcj0iMCI+DQogICAgICAgICAgPHRyPg0KICAgICAgICAgICAgPHRk
IHdpZHRoPSI3MiUiPg0KICAgICAgICAgICAgPGltZyBoZWlnaHQ9IjIxMiIgc3JjPSJodHRw
Oi8vaW1hZ2Vob3N0LmF1Y3Rpb253YXRjaC5jb20vcHJldmlldy9jYy9jY29ycmVhL21wdHJl
cDUuanBnIiB3aWR0aD0iMjIyIiBib3JkZXI9IjAiPjwvdGQ+DQogICAgICAgICAgICA8dGQg
d2lkdGg9IjI4JSI+DQogICAgICAgICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQ
YWRkaW5nPSIwIiB3aWR0aD0iMTAwJSIgYm9yZGVyPSIwIj4NCiAgICAgICAgICAgICAgPHRy
Pg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMTAwJSI+DQogICAgICAgICAgICAgICAg
PGltZyBzcmM9Imh0dHA6Ly9pbWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2Nj
L2Njb3JyZWEvbXB0cmVwMS5qcGciIHdpZHRoPSIxMjYiIGJvcmRlcj0iMCI+PC90ZD4NCiAg
ICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAg
IDx0ZCB3aWR0aD0iMTAwJSI+DQogICAgICAgICAgICAgICAgPGltZyBzcmM9Imh0dHA6Ly9p
bWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2NjL2Njb3JyZWEvbXB0cmVwMi5q
cGciIHdpZHRoPSIxMjYiIGJvcmRlcj0iMCI+PC90ZD4NCiAgICAgICAgICAgICAgPC90cj4N
CiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMTAwJSI+
DQogICAgICAgICAgICAgICAgPGltZyBzcmM9Imh0dHA6Ly9pbWFnZWhvc3QuYXVjdGlvbndh
dGNoLmNvbS9wcmV2aWV3L2NjL2Njb3JyZWEvbXB0cmVwNy5qcGciIHdpZHRoPSIxMjYiIGJv
cmRlcj0iMCI+PC90ZD4NCiAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAgPHRy
Pg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMTAwJSI+DQogICAgICAgICAgICAgICAg
PGltZyBzcmM9Imh0dHA6Ly9pbWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2Nj
L2Njb3JyZWEvbXB0cmVwOC5qcGciIHdpZHRoPSIxMjYiIGJvcmRlcj0iMCI+PC90ZD4NCiAg
ICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgIDwvdGFibGU+DQogICAgICAgICAgICA8
L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+DQogICAgICAgIDwvdGQ+
DQogICAgICA8L3RyPg0KICAgICAgPHRyPg0KICAgICAgICA8dGQgd2lkdGg9IjEwMCUiPg0K
ICAgICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0i
MTAwJSIgYm9yZGVyPSIwIj4NCiAgICAgICAgICA8dHI+DQogICAgICAgICAgICA8dGQgd2lk
dGg9IjUwJSI+DQogICAgICAgICAgICA8aW1nIGhlaWdodD0iMTMyIiBzcmM9Imh0dHA6Ly9p
bWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2NjL2Njb3JyZWEvbXB0cmVwMy5q
cGciIHdpZHRoPSIxNjMiIGJvcmRlcj0iMCI+PC90ZD4NCiAgICAgICAgICAgIDx0ZCB3aWR0
aD0iNTAlIj4NCiAgICAgICAgICAgIDxpbWcgaGVpZ2h0PSIxMzMiIHNyYz0iaHR0cDovL2lt
YWdlaG9zdC5hdWN0aW9ud2F0Y2guY29tL3ByZXZpZXcvY2MvY2NvcnJlYS9tcHRyZXA2Lmpw
ZyIgd2lkdGg9IjE2OCIgYm9yZGVyPSIwIj48L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAg
ICAgIDwvdGFibGU+DQogICAgICAgIDwvdGQ+DQogICAgICA8L3RyPg0KICAgIDwvdGFibGU+
DQogICAgPC90ZD4NCiAgPC90cj4NCjwvdGFibGU+DQo8dGFibGUgaGVpZ2h0PSI0MCIgY2Vs
bFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0iNTg0IiBib3JkZXI9IjAiPg0K
ICA8dHI+DQogICAgPHRkIGhlaWdodD0iNDAiPg0KICAgIDx0YWJsZSBjZWxsU3BhY2luZz0i
MCIgY2VsbFBhZGRpbmc9IjAiIHdpZHRoPSIxMDAlIiBib3JkZXI9IjAiPg0KICAgICAgPHRy
Pg0KICAgICAgICA8dGQgd2lkdGg9IjEwMCUiPg0KICAgICAgICA8dGFibGUgY2VsbFNwYWNp
bmc9IjAiIGNlbGxQYWRkaW5nPSI1IiB3aWR0aD0iMTAwJSIgYmdDb2xvcj0iIzAwODBmZiIg
Ym9yZGVyPSIwIj4NCiAgICAgICAgICA8dHI+DQogICAgICAgICAgICA8dGQgd2lkdGg9IjEw
MCUiPjxiPjxmb250IGZhY2U9IlRhaG9tYSIgY29sb3I9IiNmZmZmZmYiIHNpemU9IjMiPg0K
ICAgICAgICAgICAgRXNwZWNpZmljYWNpb25lcyB0ZWNuaWNhczwvZm9udD48L2I+PC90ZD4N
CiAgICAgICAgICA8L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L3RkPg0KICAg
ICAgPC90cj4NCiAgICAgIDx0cj4NCiAgICAgICAgPHRkIHdpZHRoPSIxMDAlIj4NCiAgICAg
ICAgPHRhYmxlIGNlbGxTcGFjaW5nPSIwIiBjZWxsUGFkZGluZz0iNSIgd2lkdGg9IjEwMCUi
IGJvcmRlcj0iMCI+DQogICAgICAgICAgPHRyPg0KICAgICAgICAgICAgPHRkIHdpZHRoPSI1
MCUiPjxiPjxmb250IGZhY2U9IlRhaG9tYSIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjIiPg0K
ICAgICAgICAgICAgU3Vtc3VuZyBMZW5zPGJyPg0KICAgICAgICAgICAgU29ueSBTZXJ2byBz
eXN0ZW08YnI+DQogICAgICAgICAgICBTdXBwb3J0IFJlZ3VsYXIgQXVkaW8sIE1QMyxWQ0Qs
RFZDRCw4MCBtaW4gQ0QtUixDRC1SVyBhbmQgOCBjbSBDRC1SIA0KICAgICAgICAgICAgZGlz
Yzxicj4NCiAgICAgICAgICAgIGNvbXBhdGlibGUgVkNEIDIuMCwzLjAgYW5kIERWQ0QgZGlz
Yzxicj4NCiAgICAgICAgICAgIDYgViBwb3dlciBvdXRwdXQ8YnI+DQogICAgICAgICAgICBi
dWlsZC1pbiByZWNoYXJnZWFibGUgbGl0aGl1bSBiYXR0ZXJ5PGJyPg0KICAgICAgICAgICAg
Q29uc3VtcHRpb24gb2YgYmF0dGVyeSAoc2kgZXMgcXVlIG5vIGxhIG9yaWdpbmFsKSAyLjUg
LSAzIGhvdXJzIGZvciANCiAgICAgICAgICAgIE1QMyBkaXNjPGJyPg0KICAgICAgICAgICAg
MS41IC0gMiBob3VycyBmb3IgVkNEIGRpc2M8YnI+DQombmJzcDs8L2ZvbnQ+PC9iPjwvdGQ+
DQogICAgICAgICAgICA8dGQgd2lkdGg9IjUwJSI+PGI+PGZvbnQgZmFjZT0iVGFob21hIiBj
b2xvcj0iIzAwODBmZiIgc2l6ZT0iMiI+DQogICAgICAgICAgICBBY2Nlc3Nvcmllczo8YnI+
DQogICAgICAgICAgICBQb3dlciBzdXBwbHkoMTAwViAtIDI0MFYpPGJyPg0KICAgICAgICAg
ICAgSW5zdHJ1Y3Rpb24gbWFudWFsPGJyPg0KICAgICAgICAgICAgRWFycGhvbmU8YnI+DQog
ICAgICAgICAgICBSZWNoYXJnZWFibGUgTGl0aGl1bSBiYXR0ZXJpZXM8YnI+DQogICAgICAg
ICAgICBBL1YgQ2FibGU8YnI+DQogICAgICAgICAgICBSZW1vdGUgY29udHJvbDxicj4NCiAg
ICAgICAgICAgIEVzdGFzIHNvbiBsYXMgbm9ybWFzIGEgbGFzIHF1ZSBzZSBzb21ldGllcm9u
IHkgcGFzYXJvbiBlc3RvIGF2YWxhIGxhIA0KICAgICAgICAgICAgY2FsaWRhZDxicj4NCiAg
ICAgICAgICAgIGFwcHJvdmFsOiBGQ0MsIFVMLCBGREEsIENFLCBDQ0VFPC9mb250PjwvYj48
L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+DQogICAgICAgIDwvdGQ+
DQogICAgICA8L3RyPg0KICAgIDwvdGFibGU+DQogICAgPGRpdiBhbGlnbj0iY2VudGVyIj4N
CiAgICAgIDxmb250IGZhY2U9IlRhaG9tYSIgc2l6ZT0iNSI+PHN0cm9uZz5Db24gbGEgY29t
cHJhIGRlbCBT+nBlciBEaXNjbWFuIHRlIA0KICAgICAgbGxldmFzIGRlIHJlZ2FsbyA8L3N0
cm9uZz48L2ZvbnQ+DQogICAgPC9kaXY+DQogICAgPGRpdiBhbGlnbj0iY2VudGVyIj4NCiAg
ICAgIDxmb250IGZhY2U9IlRhaG9tYSIgc2l6ZT0iNSI+PHN0cm9uZz51bmEgcOlsaWN1bGEg
eSB1biBDRCBkZSBtcDMgYSANCiAgICAgIGVsZWNjafNuLjwvc3Ryb25nPjwvZm9udD48L2Rp
dj4NCiAgICA8ZGl2IGFsaWduPSJjZW50ZXIiPg0KICAgICAgPGRpdiBhbGlnbj0iY2VudGVy
Ij4NCiAgICAgICAgPGZvbnQgZmFjZT0iVGFob21hIiBjb2xvcj0iIzgwMDA4MCIgc2l6ZT0i
NSI+PHN0cm9uZz4oTk8gVEUgUE9ERVMgUFJJVkFSIA0KICAgICAgICBERSBMQVMgTUVKT1JF
UyBDT1NBUyk8L3N0cm9uZz48L2ZvbnQ+PC9kaXY+DQogICAgPC9kaXY+DQogICAgPC90ZD4N
CiAgPC90cj4NCjwvdGFibGU+DQo8cD48Zm9udCBzaXplPSIrMiIgY29sb3I9IiM4MDAwODAi
IGZhY2U9IlRhaG9tYSI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7IA0KRU5WSU9TIEEgVE9ETyBFTCBQQUlTPC9mb250PjwvcD4NCjxw
Pjxmb250IHNpemU9IisyIiBjb2xvcj0iIzgwMDA4MCIgZmFjZT0iVGFob21hIj5QQVJBIE1B
UyANCklORk9STUFDSU9OIFNPTE8gRU5WSUEgVU4gTUFJTCBBPC9mb250PjxzdHJvbmc+PGZv
bnQgZmFjZT0iVGFob21hIiBzaXplPSI1IiBjb2xvcj0iIzgwMDA4MCI+DQo8YSBocmVmPSJt
YWlsdG86ZGFyaW9iZXJAc3BlZWR5LmNvbS5hciI+ZGFyaW9iZXJAc3BlZWR5LmNvbS5hcjwv
YT48L2ZvbnQ+PC9zdHJvbmc+PC9wPg0KDQo8L2JvZHk+DQoNCjwvaHRtbD4=

------=_NextPart_000_001__2771585_76741,02--



From mark.favas@csiro.au  Sat Oct  6 03:43:22 2001
From: mark.favas@csiro.au (Mark Favas)
Date: Sat, 06 Oct 2001 10:43:22 +0800
Subject: [Python-Dev] Painful death in debug build
Message-ID: <3BBE6FCA.DEDF4DD6@csiro.au>

Following Martin, and configuring current CVS with --with-pydebug on
Tru64 Unix (wonder what HP will call it... <wink>), I can't get
test_descr.py to crash either. Tried also with --with-pymalloc and
different compiler optimizations.

Cheers,
Mark
-- 
Mark Favas  -   m.favas@per.dem.csiro.au
CSIRO, Private Bag No 5, Wembley, Western Australia 6913, AUSTRALIA


From skip@pobox.com (Skip Montanaro)  Sat Oct  6 03:58:03 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 5 Oct 2001 21:58:03 -0500
Subject: [Python-Dev] Painful death in debug build
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEJGLPAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCKEJGLPAA.tim.one@home.com>
Message-ID: <15294.29499.710313.179455@beluga.mojam.com>

    Tim> test_descr.py crashes under the debug build now.  I've got too many
    Tim> things piled on my stack to look into it right now, so it would be
    Tim> good if someone could try to feel debug pain on Linux too.

Works for me.  (Mandrake 8.0, GCC 3.0.1) test_fcntl failed during the first
pass of "make test", but succeeded with the -O test.  When I ran it
manually, it also succeeded.  When I ran make quicktest it crashed in the
non -O run as well.  Don't know if it's related or not.  I haven't seen this
before.

    test test_fcntl crashed -- exceptions.IOError: [Errno 22] Invalid argument

Skip


From tim.one@home.com  Sat Oct  6 04:01:22 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 5 Oct 2001 23:01:22 -0400
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com>

[Martin v. Loewis]
> Configuring Python with --with-pydebug, I cannot get test_descr to
> crash on Linux, using the current CVS.

I believe that <wink>.  It's not dying with a memory fault, it's dying
because code in MS's debug libraries is proactively searching for memory
corruption, finding it, then deliberately causing a crash as its way of
spelling "hey! your program is insane".  I expect you'd need something like
Insure++ or efence to have a good chance of catching it on Linux.  Sooner or
later it will manifest as insane behavior in the release build -- this is
"wild store" stuff, like bad pointer arithmetic, or out-of-bounds array
indexing.  Resulting problems usually occur long after the bad store, and
can be symptom-free by luck (until you're doing a demo <0.9 wink>).



From fdrake@acm.org  Sat Oct  6 04:14:22 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 23:14:22 -0400
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com>
References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de>
 <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com>
Message-ID: <15294.30478.165439.151351@grendel.zope.com>

Tim Peters writes:
 > spelling "hey! your program is insane".  I expect you'd need something like
 > Insure++ or efence to have a good chance of catching it on Linux.  Sooner or

  I've started an Electric Fence run on a Mandrake 7.2 box, running
the regression tests.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From fdrake@acm.org  Sat Oct  6 04:38:22 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 23:38:22 -0400
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com>
References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de>
 <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com>
Message-ID: <15294.31918.124898.851098@grendel.zope.com>

  Given that Guido's already done the efence run, I'll ignore the
weird result I'm getting.  ;-)
  On a laptop running Mandrake 7.2, Python configured --with-pydebug,
and lots of free memory, I get this:


beowolf(.../python/debug); LD_PRELOAD=/usr/lib/libefence.so.0.0 ./python ../Lib/test/regrtest.py -l

  Electric Fence 2.2.0 Copyright (C) 1987-1999 Bruce Perens <bruce@perens.com>
Adding parser accelerators ...
Done.
test_grammar
test_opcodes
test_operations
test_builtin
test_exceptions
test_types
test_MimeWriter
test_StringIO
test___all__

ElectricFence Exiting: mmap() failed: Cannot allocate memory
[255] beowolf(.../python/debug); free
             total       used       free     shared    buffers     cached
Mem:        523808     204256     319552          0      29112     116824
-/+ buffers/cache:      58320     465488
Swap:      1050800          0    1050800



  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From fdrake@acm.org  Sat Oct  6 04:44:48 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Oct 2001 23:44:48 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110052101.RAA02367@cj20424-a.reston1.va.home.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
 <3BBDD79D.C26D1646@lemburg.com>
 <200110051557.LAA25625@cj20424-a.reston1.va.home.com>
 <15293.57386.311536.729065@anthem.wooz.org>
 <3BBDE670.52DFB7F0@lemburg.com>
 <15293.59431.1516.201622@anthem.wooz.org>
 <3BBDFA29.A5A4DCBA@lemburg.com>
 <200110052101.RAA02367@cj20424-a.reston1.va.home.com>
Message-ID: <15294.32304.20074.289952@grendel.zope.com>

Guido van Rossum writes:
 > Followup question: should I also implement issubclass(X, (A, B, C...))?
 > 
 > It's consistent to do so, but it's more work (issubclass is a bit
 > murkier due to the "abstract subclass" support), and I'm not sure that
 > it has the same benefits...

  They're similar, but they're usage is not.  Let's keep issubclass()
simple until we know there's a reason to change it.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From nas@python.ca  Sat Oct  6 05:44:55 2001
From: nas@python.ca (Neil Schemenauer)
Date: Fri, 5 Oct 2001 21:44:55 -0700
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: <15294.31918.124898.851098@grendel.zope.com>; from fdrake@acm.org on Fri, Oct 05, 2001 at 11:38:22PM -0400
References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> <15294.31918.124898.851098@grendel.zope.com>
Message-ID: <20011005214455.A26548@glacier.arctrix.com>

Fred L. Drake, Jr. wrote:
> 
>   Given that Guido's already done the efence run, I'll ignore the
> weird result I'm getting.  ;-)
>   On a laptop running Mandrake 7.2, Python configured --with-pydebug,
> and lots of free memory, I get this:
> 
> 
> beowolf(.../python/debug); LD_PRELOAD=/usr/lib/libefence.so.0.0 ./python ../Lib/test/regrtest.py -l
...
> ElectricFence Exiting: mmap() failed: Cannot allocate memory

Using electric fence requires huge amounts of (virtual) memory.  Every
allocation requires two pages of memory.  I doubt running the entire
test suite would ever work.  I'm having trouble running just test_descr
alone with all the tests commented out!  I think I'm running into
MAX_MAP_COUNT (which defaults to 64k).

I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a
dynamically configurable max_map_count.  We'll see how that goes.

Someone should really write a portable debug version of malloc and free
for Python that works like the Microsoft implementation.  It can't be
too hard to lay down poison before and after the hunk of memory on
allocation and then check it on deallocation.

  Neil


From tim.one@home.com  Sat Oct  6 06:02:09 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 6 Oct 2001 01:02:09 -0400
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: <20011005214455.A26548@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEKDLPAA.tim.one@home.com>

[Neil Schemenauer]
> Using electric fence requires huge amounts of (virtual) memory.  Every
> allocation requires two pages of memory.

Yikes!
[NealS]
> I doubt running the entire test suite would ever work.  I'm having
> trouble running just test_descr alone with all the tests commented out!

FWIW, I can provoke the bug by running test_descr.inherits() alone:

C:\Code\python\PCbuild>python_d
Adding parser accelerators ...
Done.
Python 2.2a4+ (#24, Oct  5 2001, 18:16:47) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_descr
[21862 refs]
>>> test_descr.inherits()
Testing inheritance from basic types...

And then it blows up.

> I think I'm running into MAX_MAP_COUNT (which defaults to 64k).
>
> I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a
> dynamically configurable max_map_count.  We'll see how that goes.

You may as well rewrite the kernel while you're at it <wink>.

> Someone should really write a portable debug version of malloc and free
> for Python that works like the Microsoft implementation.  It can't be
> too hard to lay down poison before and after the hunk of memory on
> allocation and then check it on deallocation.

It's extremely useful for establishing the presence of off-by-one stores,
but this is the first time I've seen this particular check trigger in
Python.  Another trick it plays is to fill free()'ed memory with a different
unlikely byte pattern, then delay recycling the memory.  When things blow up
this "dead landfill" value is amazingly often present (indicating that
someone is still trying to read from free'd memory).  The third trick is
that newly malloc'ed memory is filled with a third "clean landfill" byte,
which helps catch use of uninitialized memory.  All the special byte values
are picked so that they're unlikely to yield valid addresses, or integers.
In bang for the buck, these tricks are huge wins.

if-only-they-pointed-to-the-true-causes-too-ly y'rs  - tim



From tim.one@home.com  Sat Oct  6 06:10:02 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 6 Oct 2001 01:10:02 -0400
Subject: FW: [PythonLabs] Re: [Python-Dev] RE: Painful death in debug build
Message-ID: <LNBBLJKPBEHFEDALKOLCAEKELPAA.tim.one@home.com>

FYI, for anyone else looking at this.  Guido tricked me into sending this to
an internal PythonLabs address.

-----Original Message-----
From: Zope.Com Python Labs
Sent: Saturday, October 06, 2001 12:47 AM
To: Guido van Rossum
Cc: Zope.Com Python Labs
Subject: [PythonLabs] Re: [Python-Dev] RE: Painful death in debug build


[Guido]
> I tried efence on Linux (which traps bad stores using VM tricks) and
> couldn't get it to fail either.

Thanks for trying! I'll pursue here then. VM tricks may not be enough,
unless they're very fine-grained.  The MS debug malloc overallocates by 8
bytes, 4 on each side of the blocks it returns.  These get filled with 0xFD
bytes.  When the debug free() gets a block back, it first checks these
bytes.  The four before the block are OK (still 0xFD).  The first two bytes
after the block have been zeroed, but the two bytes after those are still
0xFD.  So who's storing 0 into a short via a computed address suffering an
off-by-one error <0.9 wink>? (*)

When I crank up enough energy, it should be more-or-less straightforward to
set a data breakpoint on one of the byte addresses getting corrupted, and so
catch the culprit in the act (the potential problem is that memory gets
recycled, and on occasion I've had to give up on this approach after wading
thru 100s of false alarms).


(*) Hmm!  Cogitating:  the traceback pointed at a subclass of long, and the
internal representation of longs is indeed one of the few places Python uses
an array of shorts (Unicode is another -- others?).  A dollar bets this will
prove to be relevant.



From tim.one@home.com  Sat Oct  6 06:32:44 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 6 Oct 2001 01:32:44 -0400
Subject: : [Python-Dev] RE: Painful death in debug build
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEKELPAA.tim.one@home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEKFLPAA.tim.one@home.com>

This is enough to reproduce the problem under a debug Windows build:

---------------------------------------------------------------
class X(long):
    pass

x = X(0xffffL)
print "before del"
del x
print "after del"
---------------------------------------------------------------

It does not fail if the instance is created via

    x = X(0x7fffL)

instead.  It does fail on

    x = X(0x8000L)

The relevant difference is that Python uses 15-bit "digits" internally for
longs, so 0x8000 may be the smallest value that will cause it to allocate
more memory than already comes for free with the _longobject header:

struct _longobject {
	PyObject_HEAD
	int ob_size;
	digit ob_digit[1];
};

but-too-tired-to-do-more-now-ly y'rs  - tim



From fdrake@acm.org  Sat Oct  6 07:09:45 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Sat,  6 Oct 2001 02:09:45 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011006060945.4C32828697@beowolf.digicool.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Updates to add ReferenceError to the standard exceptions.


From tim.one@home.com  Sat Oct  6 09:35:01 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 6 Oct 2001 04:35:01 -0400
Subject: : [Python-Dev] RE: Painful death in debug build
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEKFLPAA.tim.one@home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEKILPAA.tim.one@home.com>

[Tim]
> This is enough to reproduce the problem under a debug Windows build:
>
> ---------------------------------------------------------------
> class X(long):
>     pass
>
> x = X(0xffffL)
> print "before del"
> del x
> print "after del"
> ---------------------------------------------------------------
>
> It does not fail if the instance is created via
>
>     x = X(0x7fffL)
>
> instead.  It does fail on
>
>     x = X(0x8000L)
>
> The relevant difference is that Python uses 15-bit "digits" internally
> for longs, so 0x8000 may be the smallest value that will cause it to
> allocate more memory than already comes for free with the _longobject
> header

Now I'm in:

PyObject *
PyType_GenericAlloc(PyTypeObject *type, int nitems)
{
#define PTRSIZE (sizeof(PyObject *))

	int size;
	PyObject *obj;

	/* Inline PyObject_New() so we can zero the memory */

That comment doesn't seem to make much sense; the code following does a
whole bunch that PyObject_New() doesn't do.

	size = _PyObject_VAR_SIZE(type, nitems);

Then size = basicsize + 2 * itemsize = 26 + 2*2 = 30.

	/* Round up size, if necessary, so we fully zero out __dict__ */
	if (type->tp_itemsize % PTRSIZE != 0) {
		size += PTRSIZE - 1;
		size /= PTRSIZE;
		size *= PTRSIZE;
	}

I don't understand what that comment is trying to say; the code following it
bumps size up to 32, and that turns out to be "the problem":

	if (PyType_IS_GC(type)) {
		obj = _PyObject_GC_Malloc(type, nitems);
	}
	else {
		obj = PyObject_MALLOC(size);
	}

The subtype is a GC type, so _PyObject_GC_Malloc allocates

   sizeof(PyGC_Head) + (size_t)_PyObject_VAR_SIZE(tp, size) =
   12                + 30 =

42 bytes.  Note that the second term is the 30 originally computed for size,
not the 32 that size was later bumped up to.

	if (obj == NULL)
		return PyErr_NoMemory();

Nope.

	memset(obj, '\0', size);

Bingo!  obj is at offset 12 from the allocated block, and size is 32, so we
zero out the last 32 of the trailing 30 (=42-12) bytes allocated; i.e., we
zero out two bytes beyond the end of the allocated block.  Everything else
follows from that.  Seems pretty clear that this doesn't have anything
specific to do with longs.  Yup:

>>> class X(str):
...     pass
...
[6960 refs]
>>> x = X("ab")
[6969 refs]
>>> x = 1

That also blows up.

Since I'm not sure what the roundup of size is trying to accomplish, I'm not
sure what to do about it.  If I ignore the comment, I *imagine* we want
subtype pointer fields (like tp_dict) following the base-type prefix to be
naturally aligned for the architecture, and then padding makes sense to me.



From guido@python.org  Sat Oct  6 18:49:13 2001
From: guido@python.org (Guido van Rossum)
Date: Sat, 06 Oct 2001 13:49:13 -0400
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: Your message of "Fri, 05 Oct 2001 21:44:55 PDT."
 <20011005214455.A26548@glacier.arctrix.com>
References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> <15294.31918.124898.851098@grendel.zope.com>
 <20011005214455.A26548@glacier.arctrix.com>
Message-ID: <200110061749.NAA09260@cj20424-a.reston1.va.home.com>

> Using electric fence requires huge amounts of (virtual) memory.  Every
> allocation requires two pages of memory.  I doubt running the entire
> test suite would ever work.  I'm having trouble running just test_descr
> alone with all the tests commented out!  I think I'm running into
> MAX_MAP_COUNT (which defaults to 64k).

The trick is to compile it to a .pyc file without efence.  The
compiler is vastly malloc-happy.  Once you have the .pyc, test_descr
runs to completion without errors using efence.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Sat Oct  6 19:42:55 2001
From: guido@python.org (Guido van Rossum)
Date: Sat, 06 Oct 2001 14:42:55 -0400
Subject: : [Python-Dev] RE: Painful death in debug build
In-Reply-To: Your message of "Sat, 06 Oct 2001 04:35:01 EDT."
 <LNBBLJKPBEHFEDALKOLCMEKILPAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCMEKILPAA.tim.one@home.com>
Message-ID: <200110061842.OAA09418@cj20424-a.reston1.va.home.com>

So the problem is that one size calculation rounds up to multiples of
a pointer size and the other doesn't; unfortunately the one that
doesn't gets passed to malloc.  The reason for the rounding up --a
recent addition-- is that when you derive a subclass from tuple,
string or long, the __dict__ of the subclass instance (and its weakref
list head) are allocated after the variable-length portion of the
instance.  I propose to fix the other size calculation to also round
up.  I'll leave it to you to fix this, since you've almost nailed this
already (and since I'm not feeling so great right now).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From nas@python.ca  Sat Oct  6 21:50:26 2001
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 6 Oct 2001 13:50:26 -0700
Subject: [Python-Dev] RE: Painful death in debug build
In-Reply-To: <200110061749.NAA09260@cj20424-a.reston1.va.home.com>; from guido@python.org on Sat, Oct 06, 2001 at 01:49:13PM -0400
References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> <15294.31918.124898.851098@grendel.zope.com> <20011005214455.A26548@glacier.arctrix.com> <200110061749.NAA09260@cj20424-a.reston1.va.home.com>
Message-ID: <20011006135026.A27373@glacier.arctrix.com>

Guido van Rossum wrote:
> The trick is to compile it to a .pyc file without efence.  The
> compiler is vastly malloc-happy.  Once you have the .pyc, test_descr
> runs to completion without errors using efence.

I figured that out after watching the process grow to 800 MB. :-)

  Neil


From aahz@rahul.net  Sun Oct  7 17:00:31 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Sun, 7 Oct 2001 09:00:31 -0700 (PDT)
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <15293.53573.251652.664773@anthem.wooz.org> from "Barry A. Warsaw" at Oct 05, 2001 11:27:01 AM
Message-ID: <20011007160032.39A87E8C6@waltz.rahul.net>

[I know we've already reached agreement, but I feel the need to stick my
oar in.]

Barry A. Warsaw wrote:
> 
>     NS> Perhaps we could extend isinstance().  How about
> 
>     NS>     isinstance(x, str, unicode)
> 
>     NS> or
> 
>     NS>     isinstance(x, (str, unicode))
> 
>     NS> This is a common problem not limited to string types.  I often
>     NS> want to test if something is a tuple or a list for example.
> 
> That's an interesting idea, but please use the former signature, not
> the latter.

Ewww, gross, spit spit spit.  ;-)

At least with the latter, you can do:

    stringtype = (str, unicode)
    isinstance(x, stringtype)

Hmmmm...  Or can you?
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From gstein@lyra.org  Sun Oct  7 23:17:47 2001
From: gstein@lyra.org (Greg Stein)
Date: Sun, 7 Oct 2001 15:17:47 -0700
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <200110051306.JAA24439@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 05, 2001 at 09:06:32AM -0400
References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <200110051306.JAA24439@cj20424-a.reston1.va.home.com>
Message-ID: <20011007151746.A4216@lyra.org>

On Fri, Oct 05, 2001 at 09:06:32AM -0400, Guido van Rossum wrote:
> > It seems noone cares about this. imputil is in the distribution,
> > but is it really 'official'?
> 
> I want to care, but I'm already spread too thin.  I don't find
> 'imputil' particularly official -- but Greg Stein pushed hard to get
> it in the distribution.

Don't blame me :-)

I thought it was much better than ihooks, and a good number of people wanted
to see it in the distro.

I wrote imputil to solve some of my problems, and it was a great step
forward over the ihooks stuff. It was reasonably clean, very easy to extend,
and helped to clarify the existing import and package mechanisms (by virtue
of needing to replace them). Gordon picked up imputil and did a lot more
work with it than I ever did. At this point, if he says that iu4 is better
than imputil, then I'd be a mind to trust that :-) I'm just happy that my
imputil was able to show a way for even better improvements.

Personally, I don't have a particular attachment to preferring it over
Gordon's new iu4. iu4 probably needs a bit wider usage to see whether
*other* people can use it like Gordon :-), but it looks like he has
documented it quite well.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/


From gstein@lyra.org  Mon Oct  8 00:59:39 2001
From: gstein@lyra.org (Greg Stein)
Date: Sun, 7 Oct 2001 16:59:39 -0700
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <200109301453.QAA21436@paros.informatik.hu-berlin.de>; from loewis@informatik.hu-berlin.de on Sun, Sep 30, 2001 at 04:53:06PM +0200
References: <200109301453.QAA21436@paros.informatik.hu-berlin.de>
Message-ID: <20011007165939.B4216@lyra.org>

On Sun, Sep 30, 2001 at 04:53:06PM +0200, Martin von Loewis wrote:
> [I know I've asked this before, but Fred wanted me to ask it again :-]
> 
> What do you think about an integration of Expat into Python, to be
> always able to build pyexpat (and with the same version also)?
> Which version of Expat would you use? Would you put the expat files
> into a separate directory, or all into modules?
> 
> Here is my proposal: Integrate Expat 2.95.2 for release together with
> Python 2.2; into an expat subdirectory of Modules (taking only the lib
> files of expat).
> 
> This would affect build procedures on all targets; in particular,
> pyexpat would not link to a shared expat DLL, but incorporate the
> object files.


Speaking from the experience of bundling Expat directly into the Apache
binaries (also using a subset of the original source) ...

    I think bundling the sources is fine, but it should *ONLY* be a fallback
    if you do not find the Expat library installed on the system. *ALWAYS*
    link against a system-installed library first.


We ran into a problem that has bothered some Perl users for a long while
now. Specifically: Apache 1.3 would get loaded and export the Expat symbols
to the rest of the process. Any third-party module that was built *against
Apache* (obviously the case since they are Apache modules) and needed Expat
would immediately resolve upon loading and be happy. But! What we ran into
is mod_perl (linked against Apache) running a Perl script which, in turn,
loaded XML::Parsers::Expat. That Perl module linked against *Expat*, not
Apache (it is a standard module and has nothing to do with Apache). Well...
when the Perl module was loaded, you now had *two* sets of Expat symbols in
the process space. Segfaults, bugs, and madness ensued.

I just made some fixes this past week to Apache 1.3 to fix the situation
somewhat. The basic answer is to always grab a system (.so) library when
possible. When the shared lib is present, then both Apache and
XML::Parsers::Expat would link against the same thing about loading. And
Apache still has the feature of exposing XML to its third-party modules.


This situation could easily happen to Python, too. Imagine building Expat
directly into pyexpat. Some Python script loads pyexpat and the Expat
symbols come with it. Now, some *other* module is loaded and dynamically
links against /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols
and crashes are going to start happening.


-1 on *always* using bundles sources -- they should only be a fallback. +1
on including it as a fallback.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/


From tim.one@home.com  Mon Oct  8 08:36:19 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 8 Oct 2001 03:36:19 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <20011007160032.39A87E8C6@waltz.rahul.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIENDLPAA.tim.one@home.com>

[Aahz]
> At least with the latter, you can do:
>
>     stringtype = (str, unicode)
>     isinstance(x, stringtype)
>
> Hmmmm...  Or can you?

Yes -- it's the value of the expression that counts, not how it's spelled.
You can even nest these tuples:

>>> int_types = int, long
>>> file_types = file,
>>> def fori(x):
...     return isinstance(x, (int_types, file_types))
...
>>> fori(43)
1
>>> fori(43L)
1
>>> import sys
>>> fori(sys.stdin)
1
>>> fori(43.0)
0
>>>



From tim.one@home.com  Mon Oct  8 08:54:59 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 8 Oct 2001 03:54:59 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMENELPAA.tim.one@home.com>

[Guido]
> ...
> [*] For a while I toyed with the idea of calling the abstract base
> class 'string', and hacking import so that sys.modules['string'] is
> the string class.  The abstract base class should then have methods
> that invoke the concrete implementations, so that string.split(s)
> would be the same as s.split().
> ...
> Unfortunately this broke down when I realized that the signature of
> string.join() is wrong for the string module: the string.join function
> is string.join(sequence, stringobject) while the signature of the
> string method is join(stringobject, sequence).
> ...
> (Hm, it could be saved by making string.join() accept the arguments
> in either order.  Gross. :-)

Luckily, it's also hopeless:

>>> string.join('abc', ' - ')
'a - b - c'
>>>

That is, since strings are also sequences, there's no way to disambiguate
the pass-2-strings case.  Although, if you ask me, it's obvious I wanted the
result shown instead of ' abc-abc ' <wink>.



From martin@loewis.home.cs.tu-berlin.de  Mon Oct  8 09:03:54 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Mon, 8 Oct 2001 10:03:54 +0200
Subject: [Python-Dev] Integrating Expat
Message-ID: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de>

> I think bundling the sources is fine, but it should *ONLY* be a fallback
> if you do not find the Expat library installed on the system. *ALWAYS*
> link against a system-installed library first.

I'm not so sure about that. There are many problems doing that:

- if there is no shared libexpat, a static one is only usable if it is
  compiled either as PIC, or if pyexpat is statically linked into
  the Python interpreter. These conditions are impossible to test.

- if the installed version of expat is older than the one we ship,
  I think Python should still use the included one.

I think the problem you had with Apache does not apply to Python:

> Apache 1.3 would get loaded and export the Expat symbols to the rest
> of the process.

In Python, symbols of one extension module are not exported to
anything else. We would not build a libexpat.so (not sure whether
Apache did), and nobody would link with pyexpat.so. Furthermore,
dynloading pyexpat.so will use RTLD_LOCAL, unless that is overridden
by the application (and you really have to know what you're doing when
changing the dlopen flags).

> Now, some *other* module is loaded and dynamically links against
> /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and
> crashes are going to start happening.

No, that can't happen. pyexpat.so does not export its symbols (due to
RTLD_LOCAL), so the other module will still use libexpat.so.

Regards,
Martin



From mal@lemburg.com  Mon Oct  8 09:40:04 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 08 Oct 2001 10:40:04 +0200
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org>
 <3BBDFA29.A5A4DCBA@lemburg.com> <200110052101.RAA02367@cj20424-a.reston1.va.home.com>
Message-ID: <3BC16664.5BE1E0B8@lemburg.com>

Guido van Rossum wrote:
> 
> Now that the introduction of a class hierarchy is dead, 

Is it ? I didn't see any messages on the list saying that the
idea is dead -- only that a PEP is needed and that it cannot
be done for 2.2.

> I'd like to
> move forward with the isinstance(x, (A, B, C...)) patch.
> 
> Followup question: should I also implement issubclass(X, (A, B, C...))?
> 
> It's consistent to do so, but it's more work (issubclass is a bit
> murkier due to the "abstract subclass" support), and I'm not sure that
> it has the same benefits...

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From mwh@python.net  Mon Oct  8 10:55:33 2001
From: mwh@python.net (Michael Hudson)
Date: 08 Oct 2001 05:55:33 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: aahz@rahul.net's message of "Sun, 7 Oct 2001 09:00:31 -0700 (PDT)"
References: <20011007160032.39A87E8C6@waltz.rahul.net>
Message-ID: <2mk7y6lalm.fsf@starship.python.net>

aahz@rahul.net (Aahz Maruch) writes:

> [I know we've already reached agreement, but I feel the need to stick my
> oar in.]
> 
> Barry A. Warsaw wrote:
> > 
> >     NS> Perhaps we could extend isinstance().  How about
> > 
> >     NS>     isinstance(x, str, unicode)
> > 
> >     NS> or
> > 
> >     NS>     isinstance(x, (str, unicode))
> > 
> >     NS> This is a common problem not limited to string types.  I often
> >     NS> want to test if something is a tuple or a list for example.
> > 
> > That's an interesting idea, but please use the former signature, not
> > the latter.
> 
> Ewww, gross, spit spit spit.  ;-)
> 
> At least with the latter, you can do:
> 
>     stringtype = (str, unicode)
>     isinstance(x, stringtype)
> 
> Hmmmm...  Or can you?

Yes, and with the former, you can do

    stringtype = (str, unicode)
    isinstance(x, *stringtype)

not-that-I-care-ly y'rs
M.

-- 
  Indeed, when I design my killer language, the identifiers "foo" and
  "bar" will be reserved words, never used, and not even mentioned in
  the reference manual. Any program using one will simply dump core
  without comment. Multitudes will rejoice. -- Tim Peters, 29 Apr 1998


From gstein@lyra.org  Mon Oct  8 12:29:32 2001
From: gstein@lyra.org (Greg Stein)
Date: Mon, 8 Oct 2001 04:29:32 -0700
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de>; from martin@loewis.home.cs.tu-berlin.de on Mon, Oct 08, 2001 at 10:03:54AM +0200
References: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de>
Message-ID: <20011008042932.H4216@lyra.org>

On Mon, Oct 08, 2001 at 10:03:54AM +0200, Martin v. Loewis wrote:
> > I think bundling the sources is fine, but it should *ONLY* be a fallback
> > if you do not find the Expat library installed on the system. *ALWAYS*
> > link against a system-installed library first.
> 
> I'm not so sure about that. There are many problems doing that:
> 
> - if there is no shared libexpat, a static one is only usable if it is
>   compiled either as PIC, or if pyexpat is statically linked into
>   the Python interpreter. These conditions are impossible to test.

What? I've never seen a .a installed on a system that isn't usable. What
makes Python modules so special that they could not use a library? I find
your statement rather surprising, and I'm quite doubtful of the issue.

I've linked .a files into .so modules often. Never a problem.

> - if the installed version of expat is older than the one we ship,
>   I think Python should still use the included one.

Every time you use the included one, you run into the possibility of
symbol conflict problems. If you refuse to use the one installed in the
system, then you increase the chance of problems.

Granted: there isn't much you can do if pyexpat *requires* something newer
than what is on the system (does it require the latest? or can pyexpat work
with all versions of expat?).

> I think the problem you had with Apache does not apply to Python:
> 
> > Apache 1.3 would get loaded and export the Expat symbols to the rest
> > of the process.
> 
> In Python, symbols of one extension module are not exported to
> anything else.

Eh? Again: what makes Python so special? How can you guarantee this rule? On
*all* platforms?

> We would not build a libexpat.so (not sure whether
> Apache did),

We never built a .so. The library was linked right into the Apache httpd
executable.

> and nobody would link with pyexpat.so. Furthermore,
> dynloading pyexpat.so will use RTLD_LOCAL, unless that is overridden
> by the application (and you really have to know what you're doing when
> changing the dlopen flags).

This argument doesn't hold. dlopen() is not used on every platform. Thus,
you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee
that symbols in a library will remain private.

> > Now, some *other* module is loaded and dynamically links against
> > /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and
> > crashes are going to start happening.
> 
> No, that can't happen. pyexpat.so does not export its symbols (due to
> RTLD_LOCAL), so the other module will still use libexpat.so.

See above.


Incorporating Expat directly into Python will definitely be "nice", but I've
been through this already. It isn't pretty. If that shared library exists,
then use the darned thing, or you're gonna get burned.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/


From guido@python.org  Mon Oct  8 14:14:17 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 08 Oct 2001 09:14:17 -0400
Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string
In-Reply-To: Your message of "Mon, 08 Oct 2001 10:40:04 +0200."
 <3BC16664.5BE1E0B8@lemburg.com>
References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> <3BBDFA29.A5A4DCBA@lemburg.com> <200110052101.RAA02367@cj20424-a.reston1.va.home.com>
 <3BC16664.5BE1E0B8@lemburg.com>
Message-ID: <200110081314.JAA30586@cj20424-a.reston1.va.home.com>

> Guido van Rossum wrote:
> > 
> > Now that the introduction of a class hierarchy is dead, 
> 
> Is it ? I didn't see any messages on the list saying that the
> idea is dead -- only that a PEP is needed and that it cannot
> be done for 2.2.

Sorry, I meant that the introduction of a hierarchy in 2.2 is dead.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Mon Oct  8 16:09:40 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 08 Oct 2001 11:09:40 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BBDCFC0.18517.68AF780F@localhost>
Message-ID: <3BC1C1B4.5EAEECAE@interet.com>

Gordon McMillan wrote:

> With Thomas's changes to PyImport_ImportModuleEx, this is
> largely taken care of (it gets routed through __import__). So
> with the above bootstrap code, C code will import from my
> .pyz as well.

I am not sure if you are saying that code already exists to
import from zip archives, or that iu4.py should be used as
the basis.

I think the best approach is for me to write up a PEP based
on the discussions so far, and let everyone criticize.  The
proposal is only to import from zip archives, not
whether to replace imputil.py with iu4.py.

I will submit a PEP after I write some code.  I often find
that my initial design is poor once I implement it.  So
expect the PEP in a week or so.  This won't make it into
2.2, so there is no rush, right?

Jim


From martin@loewis.home.cs.tu-berlin.de  Tue Oct  9 02:55:15 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Tue, 9 Oct 2001 03:55:15 +0200
Subject: [Python-Dev] Integrating Expat
In-Reply-To: <20011008042932.H4216@lyra.org> (message from Greg Stein on Mon,
 8 Oct 2001 04:29:32 -0700)
References: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de> <20011008042932.H4216@lyra.org>
Message-ID: <200110090155.f991tFW04878@mira.informatik.hu-berlin.de>

> What? I've never seen a .a installed on a system that isn't usable. What
> makes Python modules so special that they could not use a library? I find
> your statement rather surprising, and I'm quite doubtful of the issue.
> 
> I've linked .a files into .so modules often. Never a problem.

It happens on Solaris with gcc as the compiler and /usr/ccs/bin/ld as
the linker. Watch this

pandora loewis 39 ( ~/tmp ) > cat a.c
int i;
int f(){return i;}
pandora loewis 40 ( ~/tmp ) > gcc -c a.c
pandora loewis 41 ( ~/tmp ) > gcc -o a.so -shared a.o
Text relocation remains                         referenced
    against symbol                  offset      in file
i                                   0x4         a.o
i                                   0x8         a.o
ld: fatal: relocations remain against allocatable but non-writable sections
collect2: ld returned 1 exit status

The problem here is that gcc passes -z text to the linker, unless
-mimpure-text is specified. ld(1) then complains if the code was not
compiled with -fPIC/-KPIC, i.e. that relocations remain.

Python is not special at all in this respect; that happens every time
when you integrate a non-PIC static library (or object file) into a
shared library, and
- the system is Solaris (or probably any SysV-derived system)
- the linker is the system linker
- the compiler passes -z text to the linker (i.e. it is gcc)

If the linker is binutils ld, it won't happen since the linker ignores
-z text. If the compiler is SUNWspro, it won't happen because cc does
not pass -z text by default. On a Linux system, it won't happen
because relocations against the text segment are not considered a
problem in the first place - COW will take care of that.

Of course, relocations against the text segment *are* a problem: they
cause the shared library not to be shared at run-time.

I don't know whether the problem is specific to Python; perhaps we
deal with more varying configurations than anybody else. Please see
the bug reports #232783, #404322, and #405329 for prior reports of
this issue.


> > - if the installed version of expat is older than the one we ship,
> >   I think Python should still use the included one.
> 
> Every time you use the included one, you run into the possibility of
> symbol conflict problems. If you refuse to use the one installed in the
> system, then you increase the chance of problems.

How would such a conflict occur using the current mechanisms for
dynamic loading in Python?

> > I think the problem you had with Apache does not apply to Python:
> > 
> > > Apache 1.3 would get loaded and export the Expat symbols to the rest
> > > of the process.
> > 
> > In Python, symbols of one extension module are not exported to
> > anything else.
>
> Eh? Again: what makes Python so special? How can you guarantee this
> rule? On *all* platforms?

Unlike everybody else, Python does *not* use RTLD_GLOBAL in the flags
passed to dlopen. This alone is a guarantee that shared libraries
won't interfere with each other:

  Optionally, RTLD_GLOBAL may be or'ed with flag, in which case the
  external symbols defined in the library will be made available to
  subsequently loaded libraries. [from the Linux dlopen man page]

It will work on all systems using dynload_shlib, plus it will work on
Windows (where you have to explicitly specify the symbols to export
and the libraries you import symbols from). Please also note the
Misc/HISTORY item

- Revert a new feature in Unix dynamic loading: for one or two
revisions, modules were loaded using the RTLD_GLOBAL flag.  It turned
out to be a bad idea.

The specific problem was that somebody was using a library that
exposed an initsocket function, which conflicted with the init
function of the socket module. So we cannot do much for systems that
cannot protect independent shared libraries from each other.


> This argument doesn't hold. dlopen() is not used on every platform. Thus,
> you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee
> that symbols in a library will remain private.

Systems that do not support local symbols in shared libraries have
bigger problems than competing libexpat implementations. BTW, which of
the systems in today's use do not have dlopen, and aren't Windows?
[How does MacOS X operate in this area?]

> Incorporating Expat directly into Python will definitely be "nice",
> but I've been through this already. It isn't pretty. If that shared
> library exists, then use the darned thing, or you're gonna get
> burned.

OTOH, I'm not looking forward to bug reports which turn out problems
with obsolete libexpat installations. You have to judge the risks.
Any user who is concerned about using the installed libexpat can still
modify Modules/Setup to use that instead of the one coming with
Python.

Regards,
Martin


From gmcm@hypernet.com  Tue Oct  9 03:26:04 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 8 Oct 2001 22:26:04 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: <3BC1C1B4.5EAEECAE@interet.com>
Message-ID: <3BC227FC.32750.79A82249@localhost>

[Gordon]
> > With Thomas's changes to PyImport_ImportModuleEx, this is
> > largely taken care of (it gets routed through __import__). So
> > with the above bootstrap code, C code will import from my .pyz
> > as well.
[Jim Ahsltrom]
> I am not sure if you are saying that code already exists to
> import from zip archives, or that iu4.py should be used as
> the basis.
>

Neither. I thought you were referring to the Py 1.5-timeframe 
problem of C code using PyImport_ImportModule* and 
bypassing all hooks.

> I think the best approach is for me to write up a PEP based
> on the discussions so far, and let everyone criticize.  The
> proposal is only to import from zip archives, not
> whether to replace imputil.py with iu4.py.

Does "importing from zip archives" really need a PEP? It's 
already quasi-official in that Jython and MacPython implement 
it. The problem it exposes (the fact that the C code makes it 
difficult to do for packages) is certainly a good topic for 
discussion and is part of the reason Greg wrote imputil and I 
wrote iu4.
 


- Gordon


From pedroni@inf.ethz.ch  Tue Oct  9 12:15:08 2001
From: pedroni@inf.ethz.ch (Samuele Pedroni)
Date: Tue, 09 Oct 2001 13:15:08 +0200
Subject: [Python-Dev] imputil and modulefinder replacements
References: <3BC227FC.32750.79A82249@localhost>
Message-ID: <3BC2DC3C.AEEB3379@inf.ethz.ch>

Hi.

>
>
> Does "importing from zip archives" really need a PEP? It's
> already quasi-official in that Jython and MacPython implement
> it.

Personally, I think so, a PEP would be indeed a good thing,
mostly to fix the details. AFAIK MacPython and Jython
are already different about the details, at least in Jython
the feature is still experimental so there is space for
changes, don't know in general.

regards, Samule Pedroni.




From guido@python.org  Tue Oct  9 12:15:57 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 09 Oct 2001 07:15:57 -0400
Subject: [Python-Dev] imputil and modulefinder replacements
In-Reply-To: Your message of "Tue, 09 Oct 2001 13:15:08 +0200."
 <3BC2DC3C.AEEB3379@inf.ethz.ch>
References: <3BC227FC.32750.79A82249@localhost>
 <3BC2DC3C.AEEB3379@inf.ethz.ch>
Message-ID: <200110091115.HAA16907@cj20424-a.reston1.va.home.com>

> > Does "importing from zip archives" really need a PEP? It's
> > already quasi-official in that Jython and MacPython implement
> > it.
> 
> Personally, I think so, a PEP would be indeed a good thing,
> mostly to fix the details. AFAIK MacPython and Jython
> are already different about the details, at least in Jython
> the feature is still experimental so there is space for
> changes, don't know in general.
> 
> regards, Samule Pedroni.

+1.  A PEP is the only way to get this to be a standard feature.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Greg.Wilson@baltimore.com  Tue Oct  9 15:17:31 2001
From: Greg.Wilson@baltimore.com (Greg Wilson)
Date: Tue, 9 Oct 2001 10:17:31 -0400
Subject: [Python-Dev] Byte revived!
Message-ID: <930BBCA4CEBBD411BE6500508BB3328F432204@nsamcanms1.ca.baltimore.com>

Hi, all.  It's finally public --- Byte magazine is being revived:

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-05
-2001/0001585919

The easiest way to make sure that the new Byte is as good as the old is to
contribute content.  If you are working on something that's technically
sweet, or if you think you know enough about a topic to write a good
overview, we would love to hear from you.  (In particular, I'd *still* like
articles about garbage collection in Python, how Stackless works, and a
bunch of other things that I've nagged people about over the past year and a
half...)

Thanks,

Greg Wilson
gvwilson at ddj.com


-----------------------------------------------------------------------------------------------------------------
The information contained in this message is confidential and is intended 
for the addressee(s) only.  If you have received this message in error or 
there are any problems please notify the originator immediately.  The 
unauthorized use, disclosure, copying or alteration of this message is 
strictly forbidden. Baltimore Technologies plc will not be liable for direct, 
special, indirect or consequential damages arising from alteration of the 
contents of this message by a third party or as a result of any virus being 
passed on.

In addition, certain Marketing collateral may be added from time to time to 
promote Baltimore Technologies products, services, Global e-Security or 
appearance at trade shows and conferences.
 
This footnote confirms that this email message has been swept by 
Baltimore MIMEsweeper for Content Security threats, including
computer viruses.


From paul@ActiveState.com  Tue Oct  9 20:36:55 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 09 Oct 2001 12:36:55 -0700
Subject: [Python-Dev] Fame, Fortune and the Python Conference
Message-ID: <3BC351D7.DF81B466@ActiveState.com>

Many of you will have submitted papers for the Python conference.
Whether you did or didn't, please consider another way to contribute to
the conference. You could submit a talks to the Web Services and
Protocols track, Zope or Tools tracks.

The Web Services and Protocols track is for anyone who is using Python
to communicate information between computers in an innovative way. You
can get involved with a simple email back to me saying you are
interested with a couple of sentences about your area of interest. I
will work with you to turn that into an abstract. You do not have to
submit a paper or anything else in advance. Just work with me on your
abstract and then show up at the conference with a fascinating talk. We
already have some well-known speakers signed up so you will be in good
company!

Here are some areas of interest:

 * Jabber peer-to-peer protocol 
 * XML-based Web Services protocols (SOAP, XML-RPC) 
 * CORBA Distributed Computing Protocol 
 * Web Services Description Language 
 * Business Integration techniques ("B2B") 
 * Enterprise Application Integration techniques ("EAI")
 * Proprietary Protocols

Just send me a note this week (why not reply right row!) describing your
project with Python and protocols or web services and we'll work out
whether there is an abstract in there.

More info is available here:

 http://www.python10.org/p10-conferenceEvents.html#webServices

Conference info (dates, location,etc.) is here:

 http://www.python10.org/

 Paul Prescod


From arigo@ulb.ac.be  Wed Oct 10 11:54:46 2001
From: arigo@ulb.ac.be (Armin Rigo)
Date: Wed, 10 Oct 2001 12:54:46 +0200 (MET DST)
Subject: [Python-Dev] Psyco3 is working
Message-ID: <Pine.OSF.3.96.1011010125034.3330A-100000@aster1.ulb.ac.be>

Hello everybody,

Psyco, the Python Specializing Comiler, is working --- currently only on
very simple examples, but it works well. The previous acceleration of 3 to
5 times is peanuts; now I get a factor 100, including the compilation
time!

If you don't believe me have a look at it:

   http://homepages.ulb.ac.be/~arigo/psyco/


Promizing-to-add-a-lot-of-comments-in-the-source'ly yours,

Armin



From mahler@cyc.com  Wed Oct 10 14:22:52 2001
From: mahler@cyc.com (Daniel Mahler)
Date: Wed, 10 Oct 2001 08:22:52 -0500
Subject: [Python-Dev] Psyco3 is working
In-Reply-To: <Pine.OSF.3.96.1011010125034.3330A-100000@aster1.ulb.ac.be>
References: <Pine.OSF.3.96.1011010125034.3330A-100000@aster1.ulb.ac.be>
Message-ID: <15300.19372.877863.62136@mcallister.cyc.com>

g'day y'all, (this is what happens when ozzies live in texas :))

I was wandering how projects like Psyco, Stackless and Parrot
affect each other, since they seem to change aspects of the execution
model.

regards
D

Armin Rigo writes:
 > Hello everybody,
 > 
 > Psyco, the Python Specializing Comiler, is working --- currently only on
 > very simple examples, but it works well. The previous acceleration of 3 to
 > 5 times is peanuts; now I get a factor 100, including the compilation
 > time!
 > 
 > If you don't believe me have a look at it:
 > 
 >    http://homepages.ulb.ac.be/~arigo/psyco/
 > 
 > 
 > Promizing-to-add-a-lot-of-comments-in-the-source'ly yours,
 > 
 > Armin
 > 
 > 
 > _______________________________________________
 > Python-Dev mailing list
 > Python-Dev@python.org
 > http://mail.python.org/mailman/listinfo/python-dev



From msw@redhat.com  Wed Oct 10 18:27:53 2001
From: msw@redhat.com (Matt Wilson)
Date: Wed, 10 Oct 2001 13:27:53 -0400
Subject: [Python-Dev] __del__ and subclassed types
Message-ID: <20011010132753.F12576@devserv.devel.redhat.com>

Python 2.2a4:

class myList(type([])):
    def __del__(self):
        print "list dealloced"

list = myList()
del list

never prints "list dealloced".  Is this supposed to work?

Cheers,

Matt


From mclay@nist.gov  Wed Oct 10 18:35:58 2001
From: mclay@nist.gov (Michael McLay)
Date: Wed, 10 Oct 2001 13:35:58 -0400
Subject: [Python-Dev] Type checks of instance variables
Message-ID: <01101013355800.02052@fermi.eeel.nist.gov>

Many applications require constraining the contents of instance variables to 
specific types.  For instance, the attributes of an element in a validating 
XML Schema implementation would require constraining the types allowed in 
each of the instance variables of the class implementing the element 
definition. This testing for  type constraints can be accomplished by adding 
an isinstance test to the __setattr__ of the class that contains the 
constrained attributes. 

Prior to Python 2.2 the resulting class definitions would be bloated with 
many lines of type checking code. The new builtin property class has 
simplified the task of constraint checking somewhat.  There are some 
improvements that would make it easier and less error prone to selectively 
restrict the types of instance variables.

The following change to the property class adds a  test for the instance 
variable type to the __set__ method.

class property2(object):

    def __init__(self, fget=None, fset=None, doc=None, type=None):
        self.fget = fget
        self.fset = fset
        self._type = type
        self.__doc__ = doc
         
    def __get__(self, inst, type=None):
        if self.fget is None:
            raise AttributeError, "this attribute is write-only"
        return self.fget(inst)

    def __set__(self, inst, value):
        if self.fset is None:
            raise AttributeError, "this attribute is read-only"
        if not isinstance(value,self._type):
            raise AttributeError,"attribute not of type '%s'"%\
                                                        self._type.__name__
        return self.fset(inst, value)

The following example illustrates the use of the property2 type checking 
capability.   When the assignment of a value is made to c.b the type checks 
are performed as desired.  Unfortunately this approach is not sufficient to 
guard against incorrect types being assigned to the instance variables. There 
are several examples demonstrating the "hidden" _b instance variable being 
populated with data of the wrong type.  

################# demonstration of property2 ##############################
from property2 import property2

class unsignedByte(int):
    """ An unsignedByte is a constrained integer with a range of 0-255
    """
    def __init__(self,value):
	if value < 0 or value > 255:
	    raise ValueError, "unsignedByte must be between 0 and 255"
	self = value

class Color(object):
    def __init__(self,b= 0 ):
        # the following assignment does not check the type of "_b".
        # The default value of "0" is not of the proper type for 
        # the instance variable b as defined by the property2 statement.
        self._b = b
        
    __slots__ =  ["_b"]
    def setb(self, value):
        self._b = value
    def getb(self):
        return self._b
    def messItUp(self):
        self._b = 2000

    b = property2(getb,setb,type=unsignedByte)

c = Color()

x = unsignedByte(34)

c.b = x

print "The value of c.b is ",c.b

c.messItUp()
print "A method can accidentally mess up the variable", c.b

c.b = x
c._b = 4000
print "you can override the hidden value as well", c.b

# Proper assignment to the instance variable does run the type check.
# An exception is raised by the following incorrectly typed variable 
# assignment

c.b = 25

In order for the property2 type checking to work well the mechanism needs to 
forbid access to the "hidden" container for the value of the instance 
variable.  

The type checking could also be combined with the new __slots__ mechanism for 
declaring variable names.  In this approach a TypedSlots class defines the 
type constraints for the slot names. The following example  demonstrates the 
use of the TypedSlots class (a subclassed dictionary) to define the types 
allowed for each slot in a class definition. 

class unsignedByte(int):
    """ An unsignedByte is a constrained integer with a range of 0-255
    """
    def __init__(self,value):
	if value < 0 or value > 255:
	    raise ValueError, "unsignedByte must be between 0 and 255"
	self = value

class TypedSlots(dictionary):
    """A TypedSlot object defines an object that is assigned to a
    class.__slot__ definition. The TypedSlot object constrains each slot to
    a specific set of types.  The allowed types are associated with the name
    at the time the slot names are defined.
    """
    def defineSlot(self,name,types):
	"""Add a new slot to the TypedSlots object.  All slot definitions
	can be added as part of the TypedSlots constructor, or they can
	be added one at a time using the definedSlot method
	"""
	self["_" + name] = (name,types)

# build an instance of the TypedSlots object by adding the slots
# one at a time.
	
t = TypedSlots()
t.defineSlot("r",unsignedByte)
t.defineSlot("g",unsignedByte)
t.defineSlot("b",unsignedByte)
t.defineSlot("name",str)

# The following class definition uses the TypedSlot instance example
# The dictionary type can be used in place of a list of
# strings for defining slots because iterating over a dictionary returns 
# a list of strings (the keys to the dictionary).

class Color(object):
    __slots__ = t

    # The bytecodes generated by the following setX and getX methods and
    # assignment of property to X would be automatically generated when
    # an object of type TypedSlots is detected as the type for __slots__.

    def setb(self,value):
	if isinstance(value,self.__slots__["_b"][1]):
	    self._b = value
	else:
	    raise TypeError, "b must be of type unsignedByte"
	
    def getb(self):
	if isinstance(self._b,self.__slots__["_b"][1]):
	    return self._b
	else:
	    raise TypeError, "b must be set to a value of type unsignedByte"
	    
    b = property(getb,setb)

    def setr(self,value):
	if isinstance(value,self.__slots__["_r"][1]):
	    self._r = value
	else:
	    raise TypeError, "r must be of type unsignedByte"
	
    def getr(self):
	if isinstance(self._r,self.__slots__["_r"][1]):
	    return self._r
	else:
	    raise TypeError, "r must be set to a value of type unsignedByte"
	    
    r = property(getr,setr)

    def setg(self,value):
	if isinstance(value,self.__slots__["_g"][1]):
	    self._g = value
	else:
	    raise TypeError, "g must be of type unsignedByte"
	
    def getg(self):
	if isinstance(self._g,self.__slots__["_g"][1]):
	    return self._g
	else:
	    raise TypeError, "g must be set to a value of type unsignedByte"
	    
    g = property(getg,setg)

    def setname(self,value):
	if isinstance(value,self.__slots__["_name"][1]):
	    self._name = value
	else:
	    raise TypeError, "name must be of type str"
	
    def getname(self):
	if isinstance(self._name,self.__slots__["_name"][1]):
	    return self._name
	else:
	    raise TypeError, "name must be set to a value of type str"
	    
    name = property(getname,setname)


c = Color()

c.name = "aqua"
print "The color name is", c.name

x = unsignedByte(254)
c.b= x
print "The value of the blue component is", c.b

# the following statement will fail because the value is not 
# of type unsignedByte
c.g= 255


This implementation also cannot guard the "hidden" names for the slots from 
access.  This creates a security hole in the type constraint system.  This 
hole complicates any attempts to write optimizations that take advantage of 
the typed slots. The proposed change to the __slots__ implementation would 
force type checks to be made on all accesses to the variables defined by 
TypedSlots.  The implementation would automatically generate the bytecode for 
testing the instance type as defined by the TypeSlots definition.

The implementation of the example required the slot to have "hidden" names 
(i.e. the names _r, _b _g, and _name) in order to implement the type checks 
on the slots for the instance variables.  The proposed TypedSlots mechanism 
would eliminate the "hidden" name and calculate the address of the slot 
directly.

The addition of this mechanism to the __slots__ definition would add 
protections against violations of a type constraint system for slots.  The 
mechanism would also reduce the code required to define the constraints.  
Using the TypedSlots definition, the constrained type of the example would be 
written as follows:

class Color(object):
    __slots__ =TypedSlots({ "r":unsignedByte,
                            "g":unsignedByte,
                            "b":unsignedByte,
                            "name":str})

The contents of the slot would only be accessible though the exposed instance 
variable name.  For example:

c = Color()
c.r = unsignedByte(126)

An optional default value could be defined during the creation of the slots 
by making the value a tuple:

class Color(object):
    __slots__ =TypedSlots({ "r":(unsignedByte,unsignedByte(0),
                            "g":(unsignedByte,unsignedByte(0),
                            "b":(unsignedByte,unsignedByte(0),
                            "name":str})

If default values are defined the type test on __get__ could be skipped.



From jeremy@zope.com  Wed Oct 10 19:16:53 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Wed, 10 Oct 2001 14:16:53 -0400 (EDT)
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <15300.37013.942228.195940@slothrop.digicool.com>

>>>>> "SM" == Skip Montanaro <montanaro@users.sourceforge.net> writes:

  SM> Update of /cvsroot/python/python/dist/src/Lib In directory
  SM> usw-pr-cvs1:/tmp/cvs-serv25433

  SM> Modified Files:
  SM> 	xmlrpclib.py
  SM> Log Message: allow long ints to be marshalled as ints - no check
  SM> is made to the incoming value, so the programmer will have to
  SM> catch OverflowError.

How does the other end unmarshal it?  You shouldn't get overflow
errors any more.

Oh, well, actually you do still get overflow and value errors from the
builtin int().  I wonder if there should be a generic number-from-string 
conversion function that returns either a long or an int as needed.
If an integer literal in your program text can produce a long, why
can't builtin int()?

Jeremy




From skip@pobox.com (Skip Montanaro)  Wed Oct 10 20:09:41 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 10 Oct 2001 14:09:41 -0500
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <15300.37013.942228.195940@slothrop.digicool.com>
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
 <15300.37013.942228.195940@slothrop.digicool.com>
Message-ID: <15300.40181.548940.510295@beluga.mojam.com>

    SM> xmlrpclib.py
    SM> Log Message: allow long ints to be marshalled as ints - no check
    SM> is made to the incoming value, so the programmer will have to
    SM> catch OverflowError.

    Jeremy> How does the other end unmarshal it?  You shouldn't get overflow
    Jeremy> errors any more.

    Jeremy> Oh, well, actually you do still get overflow and value errors
    Jeremy> from the builtin int().

The receiving end will unmarshal the object into an int.  This is no worse
than the way xmlrpclib treats tuples.  XML-RPC only has an <array> type,
which both lists and tuples marshal into.  On the receiving end <array>
values are always unmarshalled into lists.

    Jeremy> I wonder if there should be a generic number-from-string
    Jeremy> conversion function that returns either a long or an int as
    Jeremy> needed.  If an integer literal in your program text can produce
    Jeremy> a long, why can't builtin int()?

Not sure what you're getting at here. You do have to pass in something that
smells like an int.  "123" would obviously smell like a string and get
marshalled as such.  The XML-RPC spec also indicates that the <int> tag is
restricted to the value that can be represented by a four-byte signed
integer.  It has no type that would marshal a Python long or a native int
that is larger than sys.maxint.  Adding that would be difficult to get past
Dave Winer on the grounds of interoperability.

Skip




From jeremy@zope.com  Wed Oct 10 20:17:08 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Wed, 10 Oct 2001 15:17:08 -0400 (EDT)
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <15300.40181.548940.510295@beluga.mojam.com>
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
 <15300.37013.942228.195940@slothrop.digicool.com>
 <15300.40181.548940.510295@beluga.mojam.com>
Message-ID: <15300.40628.922232.776798@slothrop.digicool.com>

>>>>> "SM" == Skip Montanaro <skip@pobox.com> writes:

  SM> xmlrpclib.py Log Message: allow long ints to be marshalled as
  SM> ints - no check is made to the incoming value, so the programmer
  SM> will have to catch OverflowError.

  Jeremy> How does the other end unmarshal it?  You shouldn't get
  Jeremy> overflow errors any more.

  Jeremy> Oh, well, actually you do still get overflow and value
  Jeremy> errors from the builtin int().

  SM> The receiving end will unmarshal the object into an int.

Sorry to repeat myself but:

  "Jeremy> How does the other end unmarshal it?"

  Jeremy> I wonder if there should be a generic number-from-string
  Jeremy> conversion function that returns either a long or an int as
  Jeremy> needed.  If an integer literal in your program text can
  Jeremy> produce a long, why can't builtin int()?

  SM> Not sure what you're getting at here.

I meant to say "I wonder if Python should provide a generic
string-to-number conversion routine that returns either an int or a
long." 

  SM>                                            The XML-RPC spec also
  SM> indicates that the <int> tag is restricted to the value that can
  SM> be represented by a four-byte signed integer.

It sounds like such a function would not be particularly useful for
XML-RPC, but I can imagine it would be useful in a world where
programmers don't worry about whether they have an int or a long.

Incidentally, it looks like the XML-RPC code won't work with
subclasses of built-in types.  It's does dispatch on the type() of the
object, but a subclass of string won't have type StringType.  It seems to
me, though, that it should be marshallable using XML-RPC.

Jeremy







From skip@pobox.com (Skip Montanaro)  Wed Oct 10 20:21:21 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 10 Oct 2001 14:21:21 -0500
Subject: [Python-Dev] Re: [pygtk] How to delete Gtk objects
In-Reply-To: <20011010132134.E12576@devserv.devel.redhat.com>
References: <3BC46024.80B87D73@ijs.si>
 <20011010130011.C12576@devserv.devel.redhat.com>
 <20011010132134.E12576@devserv.devel.redhat.com>
Message-ID: <15300.40881.489481.1219@beluga.mojam.com>

(adding python-dev to this conversation because decisions about missing
special methods vis a vis PEP 252/253 seem to me to definitely belong in
that space.)

    >> I think that there may be some problems with extended types and
    >> __del__?

    Matt> For example:

    Matt> class myList(type([])):
    Matt>     def __del__(self):
    Matt>         print "list dealloced"

    Matt> list = myList()
    Matt> list.append("foo")
    Matt> list.append("bar")
    Matt> del list

    Matt> never prints list delloced.

My guess (without examining any code) is that __del__ is not implemented for
new-style classes.  There is this rather cryptic note at the end of PEP 253:

      - open issues:
         ...
         - do we need __coerce__, __del__? 
         ...

which leads me to believe that __del__ would not be called in the current
implementation.  If people are going to be subclassing more than the
traditional builtin types I think __del__ methods will have to be supported.

Skip


From paul@ActiveState.com  Wed Oct 10 20:33:18 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Wed, 10 Oct 2001 12:33:18 -0700
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib
 xmlrpclib.py,1.11,1.12
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
 <15300.37013.942228.195940@slothrop.digicool.com>
 <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com>
Message-ID: <3BC4A27E.1E135B01@ActiveState.com>

Jeremy Hylton wrote:
> 
>...
> 
> It sounds like such a function would not be particularly useful for
> XML-RPC, but I can imagine it would be useful in a world where
> programmers don't worry about whether they have an int or a long.

In such a case I would use "long()" and not worry about a slight
performance hit.

 Paul Prescod


From skip@pobox.com (Skip Montanaro)  Wed Oct 10 20:39:26 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 10 Oct 2001 14:39:26 -0500
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <15300.40628.922232.776798@slothrop.digicool.com>
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
 <15300.37013.942228.195940@slothrop.digicool.com>
 <15300.40181.548940.510295@beluga.mojam.com>
 <15300.40628.922232.776798@slothrop.digicool.com>
Message-ID: <15300.41966.749814.1125@beluga.mojam.com>

    Jeremy> I meant to say "I wonder if Python should provide a generic
    Jeremy> string-to-number conversion routine that returns either an int
    Jeremy> or a long."

    SM> The XML-RPC spec also indicates that the <int> tag is restricted to
    SM> the value that can be represented by a four-byte signed integer.

    Jeremy> It sounds like such a function would not be particularly useful
    Jeremy> for XML-RPC, but I can imagine it would be useful in a world
    Jeremy> where programmers don't worry about whether they have an int or
    Jeremy> a long.

Ah, okay.  I was only looking at your note through XML-RPC-colored glasses.
I thought PEP 237 was supposed to handle the int/long unification.
Presumably, after it is fully in effect, int("9999999999999999999999999")
will just return a long, maybe not.

    Jeremy> Incidentally, it looks like the XML-RPC code won't work with
    Jeremy> subclasses of built-in types.  It's does dispatch on the type()
    Jeremy> of the object, but a subclass of string won't have type
    Jeremy> StringType.  It seems to me, though, that it should be
    Jeremy> marshallable using XML-RPC.

I filed a bug report just so this thought wouldn't get lost:

    http://sourceforge.net/tracker/index.php?func=detail&aid=469972&group_id=5470&atid=105470

I don't know if that's a bug report or a feature request though.  Xmlrpclib
doesn't currently marshal instances of classic classes, so I'm not so sure
it should marshal instances of new-style classes either.  It's Fredrik's and
Martin's call.

Skip








From jeremy@zope.com  Wed Oct 10 20:49:21 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Wed, 10 Oct 2001 15:49:21 -0400 (EDT)
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <15300.41966.749814.1125@beluga.mojam.com>
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
 <15300.37013.942228.195940@slothrop.digicool.com>
 <15300.40181.548940.510295@beluga.mojam.com>
 <15300.40628.922232.776798@slothrop.digicool.com>
 <15300.41966.749814.1125@beluga.mojam.com>
Message-ID: <15300.42561.319722.570229@slothrop.digicool.com>

>>>>> "SM" == Skip Montanaro <skip@pobox.com> writes:

  Jeremy> It sounds like such a function would not be particularly
  Jeremy> useful for XML-RPC, but I can imagine it would be useful in
  Jeremy> a world where programmers don't worry about whether they
  Jeremy> have an int or a long.

  SM> Ah, okay.  I was only looking at your note through
  SM> XML-RPC-colored glasses.  I thought PEP 237 was supposed to
  SM> handle the int/long unification.  Presumably, after it is fully
  SM> in effect, int("9999999999999999999999999") will just return a
  SM> long, maybe not.

If that happens, we'll need to fix xmlrpclib so that it doesn't
marshal int that don't fit in 4 bytes :-).

  Jeremy> Incidentally, it looks like the XML-RPC code won't work with
  Jeremy> subclasses of built-in types.  It's does dispatch on the
  Jeremy> type() of the object, but a subclass of string won't have
  Jeremy> type StringType.  It seems to me, though, that it should be
  Jeremy> marshallable using XML-RPC.

  SM> I filed a bug report just so this thought wouldn't get lost:

  SM>     http://sourceforge.net/tracker/index.php?func=detail&aid=469972&group_id=5470&atid=105470

  SM> I don't know if that's a bug report or a feature request though.
  SM> Xmlrpclib doesn't currently marshal instances of classic
  SM> classes, so I'm not so sure it should marshal instances of
  SM> new-style classes either.  It's Fredrik's and Martin's call.

It seems like it ought to handle subclasses of builtin classes like
string.

Jeremy









From martin@loewis.home.cs.tu-berlin.de  Wed Oct 10 21:59:09 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Wed, 10 Oct 2001 22:59:09 +0200
Subject: [Python-Dev] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
Message-ID: <200110102059.f9AKx9O02319@mira.informatik.hu-berlin.de>

> It seems like it ought to handle subclasses of builtin classes like
> string.

That's desirable, indeed. I think Skip's concern is primarily how to
implement that; that is my concern, atleast.

I can think of an implementation that replaces

    def __dump(self, value):
        try:
            f = self.dispatch[type(value)]
        except KeyError:
            raise TypeError, "cannot marshal %s objects" % type(value)
        else:
            f(self, value)

with

    def __dump(self, value):
        for t in type(value).__mro__:
            try:
                f = self.dispatch[t]
            except KeyError:
                pass
            else:
                f(self, value)
                return
        raise TypeError, "cannot marshal %s objects" % type(value)

That has several draw-backs, though:

- it does not work with Py 1.5, which /F still requires for xmlrpclib,
- it may give ambiguous results, in case an object could act as
  either of the XML-RPC types (it works deterministic, but may
  still give the undesired outcome)
- it doesn't work for classic classes and instances

Can you come up with anything better?

Regards,
Martin

P.S. I don't know whether usage of __mro__ itself would be a bug or a
feature. I certainly take pride in coming up with that idea :-)


From skip@pobox.com (Skip Montanaro)  Wed Oct 10 22:17:16 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 10 Oct 2001 16:17:16 -0500
Subject: [Python-Dev] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <200110102059.f9AKx9O02319@mira.informatik.hu-berlin.de>
References: <200110102059.f9AKx9O02319@mira.informatik.hu-berlin.de>
Message-ID: <15300.47836.277081.93224@beluga.mojam.com>

    >> It seems like it ought to handle subclasses of builtin classes like
    >> string.

    Martin> That's desirable, indeed. I think Skip's concern is primarily
    Martin> how to implement that; that is my concern, atleast.

No, I'm not worried about the implementation.  I know one of you wizards can
figure that out.  I'm worried about the silent loss of information.  Given
this class definition:

    >>> class MyString(str):
    ...   def __init__(self, s):
    ...     str.__init__(s)
    ...     self.foo = "xyz"
    ... 
    >>> s = MyString("sdfsdf")
    >>> s
    'sdfsdf'
    >>> s.foo
    'xyz'

Marshalling it into XML-RPC form in the obvious way will only transmit the
string "sdfsdf" to the remote end and will lose the object's foo attribute,
which might matter to the client and/or the server.

XML-RPC does not provide any mechanism for automagically marshalling scalar
objects other than ints, booleans, strings, doubles, ISO8601 date/time
objects and base64 strings.  The only compound objects it will marshal are
structs (dicts, and only dicts with strings as keys) and arrays (tuples or
lists).  If you want to marshal other things, you have to deal with that at
the application level.  You could subclass xmlrpclib.Marshaller to encode
class instances by encoding their __dict__ or whatever __getstate__ returns,
but the the marshaller at the other end will have to be aware of your spiffy
encoding and recreate an instance of the appropriate class (whatever that
might be - remember the remote end just might not be written in Python).

Skip


From michel@zope.com  Wed Oct 10 22:33:15 2001
From: michel@zope.com (Michel Pelletier)
Date: Wed, 10 Oct 2001 17:33:15 -0400
Subject: [Python-Dev] __del__ and subclassed types
In-Reply-To: <20011010132753.F12576@devserv.devel.redhat.com>
Message-ID: <web-2797649@digicool.com>

On Wed, 10 Oct 2001 13:27:53 -0400
 Matt Wilson <msw@redhat.com> wrote:
> Python 2.2a4:
> 
> class myList(type([])):
>     def __del__(self):
>         print "list dealloced"
> 
> list = myList()
> del list
> 
> never prints "list dealloced".  Is this supposed to work?

I don't know, but it does work if you *don't* subclass from type([]).  But you may already know that.  Neither PEP 252 or 253 mention specifics on __del__, so maybe it's a bug?

-Michel


From andy@reportlab.com  Wed Oct 10 22:58:52 2001
From: andy@reportlab.com (Andy Robinson)
Date: Wed, 10 Oct 2001 22:58:52 +0100
Subject: [Python-Dev] Type checks of instance variables
In-Reply-To: <E15rQkn-0006Ep-00@mail.python.org>
Message-ID: <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com>

> class Color(object):
>     __slots__ =TypedSlots({ "r":(unsignedByte,unsignedByte(0),
>                             "g":(unsignedByte,unsignedByte(0),
>                             "b":(unsignedByte,unsignedByte(0),
>                             "name":str})
>

This sounds great.  In fact, it's almost exactly
what we have been trying to do in reportlab/graphics for
the last six months:  a graphical widget hierarchy
which has to check the type of every assignment
and maybe trigger recalculations, because it supports
a GUI. It was one of the few occasions when I wished I
was using Delphi instead of Python :-)  Reportlab would
certainly be interested in a standardised framework for
property-checking.

Let me throw in a few more ideas:

(1) __slots__ could hold optional attribute doc strings
(2) 'one of a list' is a mightily useful type,
and we have GUIs that can infer enough from this
to build smart editing interfaces.
(3) frequently you want the __slots__ for a class
to 'inherit' from a parent, which we do by allowing
one to pass in the parent's class; it then looks
for the __slots__ in that as a starting point.
(4) you may also sometimes want to delete things
from the __slots__ in a derived class, and to reduce
visibility.
(5) things can get slow. I'm also wondering if support
for type-checking classes could be enhanced by a C
extension, so that if one stuck to a basic bunch of types,
the penalty of checking every attribute assignment
would not be too great.  It would still need to
call out to Python if the 'checking function' was
written in Python.

Here's one of our classes:

class Marker(Widget):
    '''A polymorphic class of markers'''
    _attrMap = AttrMap(BASE=Widget,
                    kind = AttrMapValue(
                            OneOf(None, 'Square', 'Diamond',
                    'Circle', 'Cross', 'Triangle', 'StarSix',
                    'Pentagon', 'Hexagon','Heptagon', 'Octagon',
                    'FilledSquare','FilledCircle', 'FilledDiamond',
                    'FilledCross','FilledTriangle','FilledStarSix',
                    'FilledPentagon', 'FilledHexagon', 'FilledHeptagon',
                    'FilledOctagon', 'Smiley'),
                            desc='marker type name'),
                    size = AttrMapValue(isNumber,desc='marker size'),
                    x = AttrMapValue(isNumber,desc='marker x coordinate'),
                    y = AttrMapValue(isNumber,desc='marker y coordinate'),
                    dx = AttrMapValue(isNumber,desc='marker x coordinate
adjustment'),
                    dy = AttrMapValue(isNumber,desc='marker y coordinate
adjustment'),
                    angle = AttrMapValue(isNumber,desc='marker rotation'),
                    fillColor = AttrMapValue(isColorOrNone, desc='marker
fill colour'),
                    strokeColor = AttrMapValue(isColorOrNone, desc='marker
stroke colour'),
                    strokeWidth = AttrMapValue(isNumber, desc='marker stroke
width'),
                    )

Regards,

Andy Robinson
CEO/Chief Architect, Reportlab Inc.



From tim@zope.com  Wed Oct 10 22:58:45 2001
From: tim@zope.com (Tim Peters)
Date: Wed, 10 Oct 2001 17:58:45 -0400
Subject: [Python-Dev] __del__ and subclassed types
In-Reply-To: <web-2797649@digicool.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEFJCIAA.tim@zope.com>

__del__ isn't implemented for new-style classes yet.



From guido@python.org  Wed Oct 10 23:05:12 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 10 Oct 2001 18:05:12 -0400
Subject: [Python-Dev] Re: [pygtk] How to delete Gtk objects
In-Reply-To: Your message of "Wed, 10 Oct 2001 14:21:21 CDT."
 <15300.40881.489481.1219@beluga.mojam.com>
References: <3BC46024.80B87D73@ijs.si> <20011010130011.C12576@devserv.devel.redhat.com> <20011010132134.E12576@devserv.devel.redhat.com>
 <15300.40881.489481.1219@beluga.mojam.com>
Message-ID: <200110102205.f9AM5Ck21449@odiug.digicool.com>

> which leads me to believe that __del__ would not be called in the current
> implementation.  If people are going to be subclassing more than the
> traditional builtin types I think __del__ methods will have to be supported.

Correct, __del__ is not yet supported.  (__coerce__ is, as of 2.2a4 at
least).  I'm not sure that __del__ should be supported though: it
causes problems with the garbage collector.  The GC won't destroy
objects with a __del__ method because it's impossible to decide in
which order to finalize mutually referencing objects.

Can somebody (e.g. whoever started this thread) present a good reason
why __del__ should be supported?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim@zope.com  Wed Oct 10 23:43:16 2001
From: tim@zope.com (Tim Peters)
Date: Wed, 10 Oct 2001 18:43:16 -0400
Subject: [Python-Dev] Bug or feature?  Unicode vs t#
Message-ID: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com>

Here in current CVS Python:

>>> import binascii
>>> binascii.hexlify(u'a')
'6100'
>>>

In 2.1.1, it's '61' instead.  I don't know what ought to happen, but 2.1.1
suprised me less.  binascii inherits its view of the argument via
PyArg_ParseTuple's t# code.



From paul@ActiveState.com  Wed Oct 10 23:53:30 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Wed, 10 Oct 2001 15:53:30 -0700
Subject: [Python-Dev] Bug or feature?  Unicode vs t#
References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com>
Message-ID: <3BC4D16A.A6C01C14@ActiveState.com>

Tim Peters wrote:
> 
>...
> 
> In 2.1.1, it's '61' instead.  I don't know what ought to happen, but 2.1.1
> suprised me less.  binascii inherits its view of the argument via
> PyArg_ParseTuple's t# code.

The documentation for hexlify says: "Return the hexadecimal
representation of the binary data." I don't think it should accept a
Unicode string as input at all. 

 Paul Prescod


From tim@zope.com  Thu Oct 11 00:07:39 2001
From: tim@zope.com (Tim Peters)
Date: Wed, 10 Oct 2001 19:07:39 -0400
Subject: [Python-Dev] Bug or feature?  Unicode vs t#
In-Reply-To: <3BC4D16A.A6C01C14@ActiveState.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEFPCIAA.tim@zope.com>

[Paul Prescod]
> The documentation for hexlify says: "Return the hexadecimal
> representation of the binary data." I don't think it should accept a
> Unicode string as input at all.

So change the docs.  hexlify() hasn't changed, yet the result has.  So long
as Unicode objects support the buffer interface, the question is why they've
changed how they respond to that interface.  This is a question about t#,
not hexlify().



From tim.one@home.com  Thu Oct 11 03:25:46 2001
From: tim.one@home.com (Tim Peters)
Date: Wed, 10 Oct 2001 22:25:46 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <15300.42561.319722.570229@slothrop.digicool.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEEPMAAA.tim.one@home.com>

[Skip]
> Presumably, after it is fully in effect, int("9999999999999999999999999")
> will just return a long, maybe not.

[Jeremy]
> If that happens, we'll need to fix xmlrpclib so that it doesn't
> marshal int that don't fit in 4 bytes :-).

If someone really cares about this, xmlrcplib is already broken in this
respect, at least on 64-bit Linux boxes (where Python ints have always been
64 bits).

def fits_in_n_bit_signed_int(i, n):
    signs = i >> (n-1)
    return signs == 0 or signs == -1



From skip@pobox.com (Skip Montanaro)  Thu Oct 11 03:36:06 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 10 Oct 2001 21:36:06 -0500
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEEPMAAA.tim.one@home.com>
References: <15300.42561.319722.570229@slothrop.digicool.com>
 <LNBBLJKPBEHFEDALKOLCKEEPMAAA.tim.one@home.com>
Message-ID: <15301.1430.548615.437567@beluga.mojam.com>

    Tim> If someone really cares about this, xmlrcplib is already broken in
    Tim> this respect, at least on 64-bit Linux boxes (where Python ints
    Tim> have always been 64 bits).

So instead of simply int()ing longs to marshal them, we should do something
like

    val = int(val)
    if val > 2L**31-1 or val < -2L**31:
        raise OverflowError

?  (Same test needed for ints...)

Skip


From gward@python.net  Thu Oct 11 03:47:36 2001
From: gward@python.net (Greg Ward)
Date: Wed, 10 Oct 2001 22:47:36 -0400
Subject: [Python-Dev] Type checks of instance variables
In-Reply-To: <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com>
References: <E15rQkn-0006Ep-00@mail.python.org> <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com>
Message-ID: <20011010224736.A5252@cthulhu.mems-exchange.org>

On 10 October 2001, Andy Robinson said:
> > class Color(object):
> >     __slots__ =TypedSlots({ "r":(unsignedByte,unsignedByte(0),
> >                             "g":(unsignedByte,unsignedByte(0),
> >                             "b":(unsignedByte,unsignedByte(0),
> >                             "name":str})
> >
> 
> This sounds great.  In fact, it's almost exactly
> what we have been trying to do in reportlab/graphics for
> the last six months:  a graphical widget hierarchy
> which has to check the type of every assignment
> and maybe trigger recalculations, because it supports
> a GUI. It was one of the few occasions when I wished I
> was using Delphi instead of Python :-)  Reportlab would
> certainly be interested in a standardised framework for
> property-checking.

Allow me to plug Grouch, my attempt at a type-description language and
type-enforcement tool for Python.  See
  http://www.mems-exchange.org/software/grouch/
for some mild hype.

Grouch works by parsing type descriptions (currently found in class
docstrings) and creating an ObjectSchema instance.  An object schema
consists of:
  * a collection of atomic types (default: int, long, float, complex,
    string); all other types in the schema are built from these
  * a collection of type aliases (eg. "number" could be an alias for
    "int | float | long", or "FooBar" for "mypkg.foobar.FooBar")
  * a collection of ClassDefinition objects

A class definition, in turn, consists (mainly) of a list of instance
attributes and the type of each instance attribute.

One of the things you can do with an object schema is use it to walk an
object graph, enforcing the attribute types in the class definitions in
the schema.  The two scripts currently distributed with Grouch do just
that: gen_schema parses docstrings, creates an ObjectSchema, and pickles
it; check_data loads the pickled schema, and uses it to typecheck some
stored object graph (currently either another pickle or a ZODB).

The major things missing from Grouch right now are:
  * simple value constraints (eg. "notnone", "notempty")
  * fancy value constraints (eg. "oneof")
  * any knowledge of new-style classes, eg. the __slots__ attribute

I would love for some other python-dev'ers to download Grouch and give
it a look.  In my non-at-all-humble opinion, Grouch is damned neat and
provides a lot of cool ideas for anyone interested in deep, rigorous
type-checking of Python objects.

        Greg
-- 
Greg Ward - Unix bigot                                  gward@python.net
http://starship.python.net/~gward/
All programmers are playwrights and all computers are lousy actors.


From tim.one@home.com  Thu Oct 11 03:56:49 2001
From: tim.one@home.com (Tim Peters)
Date: Wed, 10 Oct 2001 22:56:49 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <15301.1430.548615.437567@beluga.mojam.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEFAMAAA.tim.one@home.com>

[Skip]
> So instead of simply int()ing longs to marshal them, we should do
> something like
>
>     val = int(val)
>     if val > 2L**31-1 or val < -2L**31:
>         raise OverflowError
>
> ?  (Same test needed for ints...)

I don't know -- I'm not an xmlrpc guy.  My guess, though, is that we should
refuse to produce anything that doesn't meet the letter of the spec
(although "the spec" in this case leaves oodles of stuff un-spec-ified, it
seemed clear about the limitation to 4 bytes for ints; OTOH, IIRC it didn't
specify 2's-complement ints, and if not we can only guess at the intended
limits).



From skip@pobox.com (Skip Montanaro)  Thu Oct 11 04:30:19 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 10 Oct 2001 22:30:19 -0500
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEFAMAAA.tim.one@home.com>
References: <15301.1430.548615.437567@beluga.mojam.com>
 <LNBBLJKPBEHFEDALKOLCMEFAMAAA.tim.one@home.com>
Message-ID: <15301.4683.666174.785104@beluga.mojam.com>

    Tim> I don't know -- I'm not an xmlrpc guy.  My guess, though, is that
    Tim> we should refuse to produce anything that doesn't meet the letter
    Tim> of the spec (although "the spec" in this case leaves oodles of
    Tim> stuff un-spec-ified, it seemed clear about the limitation to 4
    Tim> bytes for ints; OTOH, IIRC it didn't specify 2's-complement ints,
    Tim> and if not we can only guess at the intended limits).

yeah, the spec is pretty casual.  As for twos-complement, it's not needed.
Everything is ascii... ;-)

Skip



From mal@lemburg.com  Thu Oct 11 09:02:20 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 11 Oct 2001 10:02:20 +0200
Subject: [Python-Dev] pickling subclasses of types (Re: [Python-checkins]
 CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12)
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net>
 <15300.37013.942228.195940@slothrop.digicool.com>
 <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com>
Message-ID: <3BC5520C.7749FDB@lemburg.com>

Jeremy Hylton wrote:
> 
> Incidentally, it looks like the XML-RPC code won't work with
> subclasses of built-in types.  It's does dispatch on the type() of the
> object, but a subclass of string won't have type StringType.  It seems to
> me, though, that it should be marshallable using XML-RPC.

You have a good point there: AFAIK, pickle and marshal both
do the same thing. Perhaps at least pickle should be adapted 
to treat all objects like instances or at least treat subclasses 
as instances so they make it across to the unpickling end (Guido,
perhaps you already have code in place which does this ?!).

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From mal@lemburg.com  Thu Oct 11 09:13:49 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 11 Oct 2001 10:13:49 +0200
Subject: [Python-Dev] Bug or feature?  Unicode vs t#
References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com>
Message-ID: <3BC554BD.A94FE62B@lemburg.com>

Tim Peters wrote:
> 
> Here in current CVS Python:
> 
> >>> import binascii
> >>> binascii.hexlify(u'a')
> '6100'
> >>>
> 
> In 2.1.1, it's '61' instead.  I don't know what ought to happen, but 2.1.1
> suprised me less.  binascii inherits its view of the argument via
> PyArg_ParseTuple's t# code.

Neither Unicode nor binascii has changed. For some reason the
code in Python/getargs.c was changed in a way which removes the
distinctin between "t#" and "s#" -- by delegating the actual
buffer conversion to convertbuffer() only the bf_getreadbuffer
slot gets used for both cases.

Whether this is right or wrong is a different topic (binascii.hexlify()
should probably work on the bf_getreadbuffer slot anyway since
its normal usage is to encode binary data using HEX), but
the change in getargs.c will certainly break existing code out 
there...

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From mal@lemburg.com  Thu Oct 11 09:28:50 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 11 Oct 2001 10:28:50 +0200
Subject: [Python-Dev] Bug or feature?  Unicode vs t#
References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com> <3BC554BD.A94FE62B@lemburg.com>
Message-ID: <3BC55842.5D10E385@lemburg.com>

[Changes to getargs.c]

The CVS log indicates that Jeremy's refactoring of the code 
introduced the mentioned change. Jeremy, was this intended or
a simple oversight ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From martin@loewis.home.cs.tu-berlin.de  Thu Oct 11 09:25:28 2001
From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis)
Date: Thu, 11 Oct 2001 10:25:28 +0200
Subject: [Python-Dev] Bug or feature? Unicode vs t#
Message-ID: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de>

> So change the docs.  hexlify() hasn't changed, yet the result has.
> So long as Unicode objects support the buffer interface, the
> question is why they've changed how they respond to that interface.

Unicode objects haven't changed in their response to the buffer
interface. What has changed is how t# uses this interface.

AFAICT, the change has occured in version 2.59 of getargs.c, in
response to patch #426072. In 2.1.1, t# used to call the getcharbuffer
operation, now it calls the getreadbuffer function. 

For Unicode objects, these are different: getcharbuffer converts the
string using the default encoding into a character string, whereas
getreadbuffer returns a pointer to the internal representation
(depending on the platform, this makes 2 or 4 bytes per Unicode
character, and they may come out either little or big endian).

It seems that the problem is in the simplification attempted with said
patch.

Regards,
Martin



From guido@python.org  Thu Oct 11 10:02:05 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 05:02:05 -0400
Subject: [Python-Dev] pickling subclasses of types (Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12)
In-Reply-To: Your message of "Thu, 11 Oct 2001 10:02:20 +0200."
 <3BC5520C.7749FDB@lemburg.com>
References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com>
 <3BC5520C.7749FDB@lemburg.com>
Message-ID: <200110110902.FAA25731@cj20424-a.reston1.va.home.com>

> > Incidentally, it looks like the XML-RPC code won't work with
> > subclasses of built-in types.  It's does dispatch on the type() of the
> > object, but a subclass of string won't have type StringType.  It seems to
> > me, though, that it should be marshallable using XML-RPC.
> 
> You have a good point there: AFAIK, pickle and marshal both
> do the same thing. Perhaps at least pickle should be adapted 
> to treat all objects like instances or at least treat subclasses 
> as instances so they make it across to the unpickling end (Guido,
> perhaps you already have code in place which does this ?!).

As of 2.2a4, pickle fully supports subclasses of built-in types.  This
is done without any changes to pickle or cPickle: the 'object' class
has a __reduce__ method that "does the right thing" (it is implemented
by calling copy_reg._reduce).  The only unsupported feature is
__slots__: attributes described by __slots__ aren't saved (this could
be fixed by adding more code to copy_reg._reduce).

Marshal happens to use a series of PyXxx_Check(obj) tests rather than
a switch on obj->ob_type, so it already supports subclasses of
built-in types it supports.

Note that marshal and pickle make different choices about what to do
for instances of subclasses of built-in types: marshal treats these as
the corresponding base type, while pickle attempts to save *all* the
object's state.

I'm not sure what choice XMLRPC should make: it attempts to marshal
the state of classic instances as as structures, but these are
unmarshalled as dicts, not as instances.  It should probably be
extended to do the same for instances of subclasses of 'object', but
I'm not sure I'm comfortable with it losing some of the object's state
in the case of subclasses of built-in types: some state would be lost
in either case.  For example:

  class C(str):
    def __init__(self):
      self.encoding = "ascii"
    def set_encoding(self, enc):
      self.encoding = enc

  a = C("hello world")
  a.set_encoding("latin-1")

If we choose to marshal 'a' as a string, the encoding is lost; if we
choose to marshal it as a record, the data is lost!

This actually answers the question for me: *if* a new-style class is a
subclass of a built-in type that xmlrpclib supports, its instances
should be marshalled as the corresponding base type; otherwise it
should be marshalled as a struct.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jeremy@zope.com  Thu Oct 11 15:05:25 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Thu, 11 Oct 2001 10:05:25 -0400 (EDT)
Subject: [Python-Dev] Bug or feature?  Unicode vs t#
In-Reply-To: <3BC55842.5D10E385@lemburg.com>
References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com>
 <3BC554BD.A94FE62B@lemburg.com>
 <3BC55842.5D10E385@lemburg.com>
Message-ID: <15301.42789.685596.895422@slothrop.digicool.com>

>>>>> "MAL" == M -A Lemburg <mal@lemburg.com> writes:

  MAL> [Changes to getargs.c] The CVS log indicates that Jeremy's
  MAL> refactoring of the code introduced the mentioned
  MAL> change. Jeremy, was this intended or a simple oversight ?

An oversight.  I'll try to set it right.

Jeremy



From guido@python.org  Thu Oct 11 15:08:32 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 10:08:32 -0400
Subject: [Python-Dev] GC head alignment issue
Message-ID: <200110111408.KAA26702@cj20424-a.reston1.va.home.com>

See this bug report:

http://sourceforge.net/tracker/?func=detail&atid=105470&aid=467145&group_id=5470

The GC header adds 12 bytes to the start of the object, but for
objects (like complex) that contain a double and require double
alignment, that messes up the alignment.  Something needs to be done
about this: I'm guessing a configuration test for the required
alignment of double, and a rounding up of the GC head size to a
multiple of that.  Should affect only the AS_GC and FROM_GC macros in
gcmodule.c and the two uses of sizeof(PyGC_Head) there -- unless you
can figure out a way to add a pad field to the GC head only when
needed, then a change to objimpl.h might be sufficient.

So far this only affects HP hardware, but we can't trust that!

(It hasn't affected 2.0 or 2.1 because it only breaks when I *subclass*
complex.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Thu Oct 11 15:48:45 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Thu, 11 Oct 2001 10:48:45 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
Message-ID: <3BC5B14D.FC2E673F@interet.com>

I am looking at Python 2.2a3.

In Include/, marshal.h declares both PyMarshal_ReadLongFromFile()
and PyMarshal_ReadShortFromFile(), but the second seems to be
missing from marshal.c.  Or is it somewhere else?

Here is the code; put in marshal.c just before ReadLongFromFile at line 638:

int
PyMarshal_ReadShortFromFile(FILE *fp)
{
        RFILE rf;
        rf.fp = fp;
        return r_short(&rf);
}

Jim Ahlstrom


From guido@python.org  Thu Oct 11 16:47:10 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 11:47:10 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: Your message of "Thu, 11 Oct 2001 10:48:45 EDT."
 <3BC5B14D.FC2E673F@interet.com>
References: <3BC5B14D.FC2E673F@interet.com>
Message-ID: <200110111547.LAA28304@cj20424-a.reston1.va.home.com>

> I am looking at Python 2.2a3.
> 
> In Include/, marshal.h declares both PyMarshal_ReadLongFromFile()
> and PyMarshal_ReadShortFromFile(), but the second seems to be
> missing from marshal.c.  Or is it somewhere else?
> 
> Here is the code; put in marshal.c just before ReadLongFromFile at line 638:
> 
> int
> PyMarshal_ReadShortFromFile(FILE *fp)
> {
>         RFILE rf;
>         rf.fp = fp;
>         return r_short(&rf);
> }

+1

--Guido van Rossum (home page: http://www.python.org/~guido/)


From paul@ActiveState.com  Thu Oct 11 17:36:51 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Thu, 11 Oct 2001 09:36:51 -0700
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de>
Message-ID: <3BC5CAA3.6A07502F@ActiveState.com>

"Martin v. Loewis" wrote:
> 
>...
> 
> For Unicode objects, these are different: getcharbuffer converts the
> string using the default encoding into a character string, whereas

What is the use case for treating a Unicode object as if it were a
buffer of bytes by translating it to the default encoding?

"Python objects implemented in C can export a group of functions called
the ``buffer interface.'' These functions can be used by an object to
expose its data in a raw, byte-oriented format. Clients of the object
can use the buffer interface to access the object data directly, without
needing to copy it first."

Translating the data before handing it to the user violates the
"raw"-ness of the byte-oriented format and the principle about not
needing to copy it. It behaves quite differently than other
implementations of getcharbuffer. 

 Paul Prescod


From tim.one@home.com  Thu Oct 11 18:04:46 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 13:04:46 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <200110111408.KAA26702@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEGNMAAA.tim.one@home.com>

[Guido]
> ...
> So far this only affects HP hardware, but we can't trust that!

Doubt it, and the status quo may be inefficient even on HW where "it works"
(it's actually common for HW to *allow* for some suboptimal alignments, but
at reduced speed).

My head hurts, so I'm not sure this is strictly legit C (drop-in replacement
for the current PyGC_Head declaration):

/* GC information is stored BEFORE the object structure */
typedef union _gc_head {
	struct {
		union _gc_head *gc_next; /* not NULL if object is tracked */
		union _gc_head *gc_prev;
		int gc_refs;
	};
	double dummy;  /* force worst-case alignment */
} PyGC_Head;

Works fine under MSVC-for-Intel, and indeed MSVC-for-Intel prefers (but does
not insist upon) 8-byte alignment for doubles.

About "strictly legit":  I'm not positive it's OK to (A) have an anonymous
struct, and then (B) directly reference members of the anonymous struct
within a union, e.g.

    PyGC_Head *p;
    p->gc_refs = 1; /* guaranteed legit? */

If that's not legit, we could make it legit by naming the struct member and
interpolating the name into all existing references.



From tim.one@home.com  Thu Oct 11 18:11:18 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 13:11:18 -0400
Subject: [Python-Dev] Bug or feature? Unicode vs t#
In-Reply-To: <3BC5CAA3.6A07502F@ActiveState.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com>

[Paul Prescod]
> What is the use case for treating a Unicode object as if it were a
> buffer of bytes by translating it to the default encoding?

Obviously, so that you can pass it to binascii.hexlify() <wink>.

uselessly y'rs  - tim



From mclay@nist.gov  Thu Oct 11 18:15:09 2001
From: mclay@nist.gov (Michael McLay)
Date: Thu, 11 Oct 2001 13:15:09 -0400
Subject: [Python-Dev] Type checks of instance variables
Message-ID: <01101113144801.02241@fermi.eeel.nist.gov>

On Wednesday 10 October 2001 05:58 pm, Andy Robinson wrote:

> Let me throw in a few more ideas:
>
> (1) __slots__ could hold optional attribute doc strings
> (2) 'one of a list' is a mightily useful type,
> and we have GUIs that can infer enough from this
> to build smart editing interfaces.
> (3) frequently you want the __slots__ for a class
> to 'inherit' from a parent, which we do by allowing
> one to pass in the parent's class; it then looks
> for the __slots__ in that as a starting point.
> (4) you may also sometimes want to delete things
> from the __slots__ in a derived class, and to reduce
> visibility.

I hope we can use the TypedSlots proposal as a starting point and refine it 
until it also meets the requirements for ReportLab and Grouch.

The following example attempts to add the features you suggested.  It extends 
the original TypedSlots definition to include actions that would be applied 
to slots. Adding type checking actions would simplify writing a number of 
applications, such as validation software for XML Schema. The XML Schema type 
system inspired the Restrict action in the example.  The XML Schema type 
system could be used to identify candidate actions to be added to TypedSlots. 
class Widget(object):
    __slots__ = TypedSlots({ "name": CheckType(str,
                          doc="The name by which a widget can be 
referenced")})

class Color(Widget):
    __slots__ = TypedSlots({ "r": CheckType(unsignedByte, 
                                            default = unsignedByte(0)),
                             "g": CheckType(unsignedByte, 
                                            default = unsignedByte(0)),
                             "b": CheckType(unsignedByte, 
                                            default = unsignedByte(0)),})

class Shape(Widget):
    __slots__ = TypedSlots({ "colorRef": CheckType(Color)})

class Circle(Shape):
    __slots__ = TypedSlots({ "diameter": CheckType(Number,default=1)})
                             

class RedCircle(Circle):
    """Define a type for red Circles.  The Restrict declaration allows
       the "colorRef" name to be restricted to a subset of the allowed
       values of the colorRef name.  In this case a ReadOnly
       flag is set to keep the Circle color from being changed and the
       value is set.
    """
    __slots__ = TypedSlots({"colorRef": Restrict( ReadOnly( Color(r=255)))})

class Drawable(Widget):
    """The OneOf action allows one of several types to be assigned to the
       name.  Most likely all the types share a common interface.
    """
    __slots__ = TypedSlots({"shapeRef":OneOf(Circle, RedCircle,Rectangle)})

    
> (5) things can get slow. I'm also wondering if support
> for type-checking classes could be enhanced by a C
> extension, so that if one stuck to a basic bunch of types,
> the penalty of checking every attribute assignment
> would not be too great.  It would still need to
> call out to Python if the 'checking function' was
> written in Python.

Performance will benefit from making C enhancements. Creating a type 
verification system that can't be hacked (from Python) will require writing 
some of the implementation in C.   The danger in extending the actions of the 
TypedSlots to more than a simple type check on __set__ is that it will become 
to complected to implement in a timely manner.  What is the bare minimum 
required?  Specification languages tend to grow to be ugly monsters. Can we 
start small and add other actions as needed? Could some of the actions be 
defined in a Python module?

>
> Here's one of our classes:
>
> class Marker(Widget):
>     '''A polymorphic class of markers'''
>     _attrMap = AttrMap(BASE=Widget,
>                     kind = AttrMapValue(
>                             OneOf(None, 'Square', 'Diamond',
>                     'Circle', 'Cross', 'Triangle', 'StarSix',
> ...

My project involves capturing the product information for printed circuit 
boards and printed circuit assemblies.  You r example looks like it came 
right out of the definition of the graphics required to by tools that draw 
the layers of a printed circuit board.


From mclay@nist.gov  Thu Oct 11 18:20:19 2001
From: mclay@nist.gov (Michael McLay)
Date: Thu, 11 Oct 2001 13:20:19 -0400
Subject: [Python-Dev] Type checks of instance variables
In-Reply-To: <20011010224736.A5252@cthulhu.mems-exchange.org>
References: <E15rQkn-0006Ep-00@mail.python.org> <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> <20011010224736.A5252@cthulhu.mems-exchange.org>
Message-ID: <01101113201903.02241@fermi.eeel.nist.gov>

On Wednesday 10 October 2001 10:47 pm, Greg Ward wrote:
> Grouch works by parsing type descriptions (currently found in class
> docstrings) and creating an ObjectSchema instance.  An object schema
> consists of:

Extending the magic of __slots__  was intended to provide a minimally 
intrusive and pythonic solution to adding type verification to Python.  
Parsing docstrings to get the type definitions is a pragmatic work-around, 
but the features of Grouch would benefit by giving them direct language 
support.

>   * a collection of atomic types (default: int, long, float, complex,
>     string); all other types in the schema are built from these
>   * a collection of type aliases (eg. "number" could be an alias for
>     "int | float | long", or "FooBar" for "mypkg.foobar.FooBar")
>   * a collection of ClassDefinition objects
>
> A class definition, in turn, consists (mainly) of a list of instance
> attributes and the type of each instance attribute.

That's what the TypedSlots class does, so I think we are on the same page 
regarding what features are needed. 

> One of the things you can do with an object schema is use it to walk an
> object graph, enforcing the attribute types in the class definitions in
> the schema.  The two scripts currently distributed with Grouch do just
> that: gen_schema parses docstrings, creates an ObjectSchema, and pickles
> it; check_data loads the pickled schema, and uses it to typecheck some
> stored object graph (currently either another pickle or a ZODB).

Are you walking the object graph to enforce the type because it isn't 
possible to enforce them at creation time in the current Python language?  
The TypedSlots concept never allows the instance variables to be populated 
with anything but the allowed types.  

Adding this concept would have been difficult prior to the addition of 
__slots__ because it won't require changes that restrict access to the 
__dict__ of an object.  The proposed TypedSlots will require some changes to 
the C code that implements the __slots__ interpretation.  The interpretation 
of the __slots__ definition would be determined by the type of the __slot__ 
value.  If a list type is detected the interpreter would follow Guido's 
existing semantics.  If the value is a TypedSlots type the interpreter would 
enforce type checking.  The features of the Grouch schema will be helpful in 
shaping the definition of the type checking actions.
>
> The major things missing from Grouch right now are:
>   * simple value constraints (eg. "notnone", "notempty")
>   * fancy value constraints (eg. "oneof")
>   * any knowledge of new-style classes, eg. the __slots__ attribute

The primary advantage of __slots__ in Python is a reduction in memory usage.  
The TypedSlots extension is also an efficiency feature.  It would simplify 
the process of accessing the value of slots that are expected to be of a 
particular type and it may facilitate optimizations based on variable type 
information.   The TypedSlots mechanism would only apply to the variables 
defined with the new __slots__  mechanism.  Would this the constraint be a 
problem for the types of applications you are building with Grouch? 


From guido@python.org  Thu Oct 11 18:15:49 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 13:15:49 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: Your message of "Thu, 11 Oct 2001 13:04:46 EDT."
 <LNBBLJKPBEHFEDALKOLCMEGNMAAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCMEGNMAAA.tim.one@home.com>
Message-ID: <200110111715.NAA29098@cj20424-a.reston1.va.home.com>

> My head hurts, so I'm not sure this is strictly legit C (drop-in replacement
> for the current PyGC_Head declaration):
> 
> /* GC information is stored BEFORE the object structure */
> typedef union _gc_head {
> 	struct {
> 		union _gc_head *gc_next; /* not NULL if object is tracked */
> 		union _gc_head *gc_prev;
> 		int gc_refs;
> 	};
> 	double dummy;  /* force worst-case alignment */
> } PyGC_Head;

Alas, gcc gives me warnings about this:

../Include/objimpl.h:274: warning: unnamed struct/union that defines no instances

and references to the fields fail with errors:

../Objects/cellobject.c: In function `PyCell_New':
../Objects/cellobject.c:14: union has no member named `gc_next'
[and many more]

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jeremy@zope.com  Thu Oct 11 18:31:08 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Thu, 11 Oct 2001 13:31:08 -0400 (EDT)
Subject: [Python-Dev] Bug or feature? Unicode vs t#
In-Reply-To: <3BC5CAA3.6A07502F@ActiveState.com>
References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de>
 <3BC5CAA3.6A07502F@ActiveState.com>
Message-ID: <15301.55132.722342.122465@slothrop.digicool.com>

>>>>> "PP" == Paul Prescod <paul@ActiveState.com> writes:

  PP> Translating the data before handing it to the user violates the
  PP> "raw"-ness of the byte-oriented format and the principle about
  PP> not needing to copy it. It behaves quite differently than other
  PP> implementations of getcharbuffer.

As far as I can tell, all other implementations of getcharbuffer are
exact duplicates of getreadbuffer.  If the Unicode object doesn't have
an appropriate implementation of getcharbuffer, one wonders why the
method exists at all:  In every case it would be redundant or
incorrect. 

Jeremy





From tim.one@home.com  Thu Oct 11 18:49:55 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 13:49:55 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <200110111715.NAA29098@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>

[Tim]
> My head hurts, so I'm not sure this is strictly legit C
> (drop-in replacement for the current PyGC_Head declaration):
>
> /* GC information is stored BEFORE the object structure */
> typedef union _gc_head {
> 	struct {
> 		union _gc_head *gc_next; /* not NULL if object is tracked */
> 		union _gc_head *gc_prev;
> 		int gc_refs;
> 	};
> 	double dummy;  /* force worst-case alignment */
> } PyGC_Head;

[Guido]
> Alas, gcc gives me warnings about this:
>
> ../Include/objimpl.h:274: warning: unnamed struct/union that
> defines no instances
>
> and references to the fields fail with errors:
>
> ../Objects/cellobject.c: In function `PyCell_New':
> ../Objects/cellobject.c:14: union has no member named `gc_next'
> [and many more]

Then:

    If that's not legit, we could make it legit by naming the struct
    member and interpolating the name into all existing references.

This is straightforward, if a bit tedious ... OK, I did that.  Requires
changes to the track and untrack macros, plus a ton of edits inside
gcmodule.c.  But that's it, and they're all straightforward changes (i.e.,
low chance of breakage).  Should I check it in?



From guido@python.org  Thu Oct 11 19:01:29 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 14:01:29 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: Your message of "Thu, 11 Oct 2001 13:49:55 EDT."
 <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>
Message-ID: <200110111801.OAA29448@cj20424-a.reston1.va.home.com>

>     If that's not legit, we could make it legit by naming the struct
>     member and interpolating the name into all existing references.
> 
> This is straightforward, if a bit tedious ... OK, I did that.  Requires
> changes to the track and untrack macros, plus a ton of edits inside
> gcmodule.c.  But that's it, and they're all straightforward changes (i.e.,
> low chance of breakage).  Should I check it in?

+1

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Thu Oct 11 19:35:57 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 14:35:57 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <200110111801.OAA29448@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEHFMAAA.tim.one@home.com>

[Tim]
> If that's not legit, we could make it legit by naming the struct
> member and interpolating the name into all existing references.
> ... OK, I did that ... Should I check it in?

[Guido]
> +1

It's checked in.



From tim.one@home.com  Thu Oct 11 20:15:32 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 15:15:32 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: <3BC5B14D.FC2E673F@interet.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEHJMAAA.tim.one@home.com>

[James C. Ahlstrom]
> I am looking at Python 2.2a3.
>
> In Include/, marshal.h declares both PyMarshal_ReadLongFromFile()
> and PyMarshal_ReadShortFromFile(), but the second seems to be
> missing from marshal.c.  Or is it somewhere else?

Doesn't appear to be anywhere.

> Here is the code; put in marshal.c just before ReadLongFromFile
> at line 638:

Please submit as a patch to SourceForge, including doc and test case
changes.



From nas@python.ca  Thu Oct 11 20:24:49 2001
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 11 Oct 2001 12:24:49 -0700
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>; from tim.one@home.com on Thu, Oct 11, 2001 at 01:49:55PM -0400
References: <200110111715.NAA29098@cj20424-a.reston1.va.home.com> <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>
Message-ID: <20011011122449.B6852@glacier.arctrix.com>

Tim Peters wrote:
> This is straightforward, if a bit tedious ... OK, I did that.  Requires
> changes to the track and untrack macros, plus a ton of edits inside
> gcmodule.c.  But that's it, and they're all straightforward changes (i.e.,
> low chance of breakage).  Should I check it in?

Sounds good to me.  Hopefully you can cover most of it up with macros.

  Neil


From mal@lemburg.com  Thu Oct 11 22:08:49 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 11 Oct 2001 23:08:49 +0200
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de>
 <3BC5CAA3.6A07502F@ActiveState.com> <15301.55132.722342.122465@slothrop.digicool.com>
Message-ID: <3BC60A61.21822A13@lemburg.com>

Jeremy Hylton wrote:
> 
> >>>>> "PP" == Paul Prescod <paul@ActiveState.com> writes:
> 
>   PP> Translating the data before handing it to the user violates the
>   PP> "raw"-ness of the byte-oriented format and the principle about
>   PP> not needing to copy it. It behaves quite differently than other
>   PP> implementations of getcharbuffer.
> 
> As far as I can tell, all other implementations of getcharbuffer are
> exact duplicates of getreadbuffer.  If the Unicode object doesn't have
> an appropriate implementation of getcharbuffer, one wonders why the
> method exists at all:  In every case it would be redundant or
> incorrect.

Well, the question is whether hexlify() should use "t#" which
is specifically intended to return *character* data and not "s#"
which means *binary* data. I think the latter is more appropriate
for hexlify() as its intention is to encode binary data.

Regarding Paul's question: the getcharbuffer interface was designed
with Unicode in mind before a Unicode implementation even existed.
It turned out to be a more or less useless design :-( since "character
data" is not enough -- you also need to be able to specify an
encoding. Unicode only supports this feature because some IO objects
rely on it (e.g. the file object when opened in text mode).

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From gward@python.net  Thu Oct 11 22:12:18 2001
From: gward@python.net (Greg Ward)
Date: Thu, 11 Oct 2001 17:12:18 -0400
Subject: [Python-Dev] Type checks of instance variables
In-Reply-To: <01101113201903.02241@fermi.eeel.nist.gov>
References: <E15rQkn-0006Ep-00@mail.python.org> <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> <20011010224736.A5252@cthulhu.mems-exchange.org> <01101113201903.02241@fermi.eeel.nist.gov>
Message-ID: <20011011171218.A2474@cthulhu.mems-exchange.org>

On 11 October 2001, Michael McLay said:
> Parsing docstrings to get the type definitions is a pragmatic work-around, 
> but the features of Grouch would benefit by giving them direct language 
> support.

Yes indeed.

> That's what the TypedSlots class does, so I think we are on the same page 
> regarding what features are needed. 

I'm sorry, I must have missed something.  What is TypedSlots and where
is it defined/documented?

> Are you walking the object graph to enforce the type because it isn't 
> possible to enforce them at creation time in the current Python language?  

More-or-less.  I think it's possible to do that with a classic
__setattr__, but I avoided going that way out of performance concerns.
We run Grouch on our database nightly to detect any type errors that may
have crept in in the last 24 hours.  That's kinda weird, and specific to
applications that work primarily on a large, persistent data store.
Using Grouch in another context would require custom coding to query the
object schema API for type-correctness of an object or whole object
graph.

> The TypedSlots concept never allows the instance variables to be populated 
> with anything but the allowed types.  

Let me see if I have this straight: __slots__, a built-in feature of
new-style classes, catches "self.nubmer = 3" when you meant
"self.number = 3".  TypedSlots, which I've never heard of before,
adds to this by catching "self.number = '3'".  Correct?

Question: does use of __slots__ guarantee that self.foo is always there?
Or do you still have to create it in __init__?

> The primary advantage of __slots__ in Python is a reduction in memory
> usage.

Gee, I would have said the primary advantage is catching
"self.nubmer = 3".

> The TypedSlots mechanism would only apply to the variables 
> defined with the new __slots__  mechanism.  Would this the constraint be a 
> problem for the types of applications you are building with Grouch? 

Not at all.  I nearly leapt out of my seat with joy when I saw __slots__
for the first time.

In the short term, we (the MEMS Exchange virtual fab, the project for
which I wrote Grouch) are utterly dependent on ZODB and ExtensionClass,
so won't be able to use new-style classes until ExtensionClass is dead
and ZODB is switched to use new-style classes.  So Grouch will continue
to work as-is for a while.  (It already knows about two types of Python
classes, namely classic classes and ExtensionClass.  Adding a third
shouldn't be *too* hard .... >famous last words<)

        Greg
-- 
Greg Ward - Linux weenie                                gward@python.net
http://starship.python.net/~gward/
If you and a friend are being chased by a lion, it is not necessary to
outrun the lion.  It is only necessary to outrun your friend.


From paulp@ActiveState.com  Thu Oct 11 22:15:14 2001
From: paulp@ActiveState.com (Paul Prescod)
Date: Thu, 11 Oct 2001 14:15:14 -0700
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com>
Message-ID: <3BC60BE2.D9182D64@ActiveState.com>

Tim Peters wrote:
> 
> [Paul Prescod]
> > What is the use case for treating a Unicode object as if it were a
> > buffer of bytes by translating it to the default encoding?
> 
> Obviously, so that you can pass it to binascii.hexlify() <wink>.

This is NOT a question about hexlify. It is a question about t#! Or
getcharbuffer. Or something.

My real question is whether there is any value in having Unicode objects
expose their internal representation to Python programmers through the
buffer interface?

 Paul Prescod


From barry@zope.com  Thu Oct 11 22:24:40 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 11 Oct 2001 17:24:40 -0400
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de>
 <3BC5CAA3.6A07502F@ActiveState.com>
 <15301.55132.722342.122465@slothrop.digicool.com>
 <3BC60A61.21822A13@lemburg.com>
Message-ID: <15302.3608.196270.320265@anthem.wooz.org>

>>>>> "M" == M  <mal@lemburg.com> writes:

    M> Well, the question is whether hexlify() should use "t#" which
    M> is specifically intended to return *character* data and not
    M> "s#" which means *binary* data. I think the latter is more
    M> appropriate for hexlify() as its intention is to encode binary
    M> data.

And I think I agree, even though that /could/ break code.  Then again,
maybe Paul's suggestion that hexlify() should reject Unicode strings
is the better approach.

-Barry


From guido@python.org  Thu Oct 11 22:22:33 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 17:22:33 -0400
Subject: [Python-Dev] Bug or feature? Unicode vs t#
In-Reply-To: Your message of "Thu, 11 Oct 2001 14:15:14 PDT."
 <3BC60BE2.D9182D64@ActiveState.com>
References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com>
 <3BC60BE2.D9182D64@ActiveState.com>
Message-ID: <200110112122.RAA00321@cj20424-a.reston1.va.home.com>

> My real question is whether there is any value in having Unicode objects
> expose their internal representation to Python programmers through the
> buffer interface?

I used to think so, but I no longer believe this.  UTF-16 should be an
encoding and that's that.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Oct 11 22:25:42 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 11 Oct 2001 17:25:42 -0400
Subject: [Python-Dev] Bug or feature? Unicode vs t#
In-Reply-To: Your message of "Thu, 11 Oct 2001 17:24:40 EDT."
 <15302.3608.196270.320265@anthem.wooz.org>
References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> <3BC5CAA3.6A07502F@ActiveState.com> <15301.55132.722342.122465@slothrop.digicool.com> <3BC60A61.21822A13@lemburg.com>
 <15302.3608.196270.320265@anthem.wooz.org>
Message-ID: <200110112125.RAA00370@cj20424-a.reston1.va.home.com>

> And I think I agree, even though that /could/ break code.  Then again,
> maybe Paul's suggestion that hexlify() should reject Unicode strings
> is the better approach.

+1

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Thu Oct 11 22:42:30 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Thu, 11 Oct 2001 17:42:30 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
References: <LNBBLJKPBEHFEDALKOLCKEHJMAAA.tim.one@home.com>
Message-ID: <3BC61246.6EF6CFA@interet.com>

Tim Peters wrote:

> Please submit as a patch to SourceForge, including doc and test case
> changes.

OK, done.  But both functions are undocumented and neither
is in test_marshal.py.  They are internal C utilities.

Jim


From tim.one@home.com  Thu Oct 11 23:04:15 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 18:04:15 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: <3BC61246.6EF6CFA@interet.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>

[Tim]
> Please submit as a patch to SourceForge, including doc and test case
> changes.

[James C. Ahlstrom]
> OK, done.

Thanks!

> But both functions are undocumented and neither is in test_marshal.py.

Then you've included the doc and test changes <wink>.

> They are internal C utilities.

Not so:  you're making a bad situation worse.  If an extern name doesn't
begin with "_", it's part of Python's public C API, and *should* be
documented and tested.  I don't think anyone will let that hold up your
patch, although everyone probably should.

including-me-on-both-counts-ly y'rs  - tim



From fdrake@acm.org  Thu Oct 11 23:16:56 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Thu, 11 Oct 2001 18:16:56 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
References: <3BC61246.6EF6CFA@interet.com>
 <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
Message-ID: <15302.6744.531834.714184@grendel.zope.com>

Tim Peters writes:
 > documented and tested.  I don't think anyone will let that hold up your
 > patch, although everyone probably should.

  Of course, it would be easy to write the little bit of LaTeX needed
to document the new function.  It would have the added benefit that
I'd need to document the rest of the PyMarshal_*() functions so the
new function would have a place to be listed.
  Or all sorts of PyMarshal_*() functions could magically grow a
leading underscore.  ;-)
  Guido, were all of these really intended to be public, or can the
list be trimmed back?  They really only appear to be used in import.c
and pythonrun.c, with the module init also appear in PC/config.c and
Modules/config.c.in (and the generated Modules/config.c).


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From paul@svensson.org  Fri Oct 12 00:07:08 2001
From: paul@svensson.org (Paul Svensson)
Date: Thu, 11 Oct 2001 19:07:08 -0400 (EDT)
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>
Message-ID: <Pine.LNX.4.21.0110111900440.2552-100000@familjen.svensson.org>

On Thu, 11 Oct 2001, Tim Peters wrote:

>[Tim]
>> My head hurts, so I'm not sure this is strictly legit C
>> (drop-in replacement for the current PyGC_Head declaration):
>>
>> /* GC information is stored BEFORE the object structure */
>> typedef union _gc_head {
>> 	struct {
>> 		union _gc_head *gc_next; /* not NULL if object is tracked */
>> 		union _gc_head *gc_prev;
>> 		int gc_refs;
>> 	};
>> 	double dummy;  /* force worst-case alignment */
>> } PyGC_Head;
>
>[Guido]
>> Alas, gcc gives me warnings about this:
>>
>> ../Include/objimpl.h:274: warning: unnamed struct/union that
>> defines no instances
>>
>> and references to the fields fail with errors:
>>
>> ../Objects/cellobject.c: In function `PyCell_New':
>> ../Objects/cellobject.c:14: union has no member named `gc_next'
>> [and many more]
>
>Then:
>
>    If that's not legit, we could make it legit by naming the struct
>    member and interpolating the name into all existing references.
>
>This is straightforward, if a bit tedious ... OK, I did that.  Requires
>changes to the track and untrack macros, plus a ton of edits inside
>gcmodule.c.  But that's it, and they're all straightforward changes (i.e.,
>low chance of breakage).  Should I check it in?

The usual thing to do in this situation would be:

typedef union _gc_head {
	struct {
		union _gc_head *internal_gc_next;
		union _gc_head *internal_gc_prev;
		int internal_gc_refs;
	} gc_internals;
	double dummy;  /* force worst-case alignment */
} PyGC_Head;

#define gc_next	gc_internals.internal_gc_head
#define gc_next	gc_internals.internal_gc_head
#define gc_refs	gc_internals.internal_gc_refs


No need to change any other code.

	/Paul




From tim.one@home.com  Fri Oct 12 01:54:29 2001
From: tim.one@home.com (Tim Peters)
Date: Thu, 11 Oct 2001 20:54:29 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <Pine.LNX.4.21.0110111900440.2552-100000@familjen.svensson.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEIPMAAA.tim.one@home.com>

[Paul Svensson]
> The usual thing to do in this situation would be:
>
> typedef union _gc_head {
> 	struct {
> 		union _gc_head *internal_gc_next;
> 		union _gc_head *internal_gc_prev;
> 		int internal_gc_refs;
> 	} gc_internals;
> 	double dummy;  /* force worst-case alignment */
> } PyGC_Head;
>
> #define gc_next	gc_internals.internal_gc_head
> #define gc_next	gc_internals.internal_gc_head
> #define gc_refs	gc_internals.internal_gc_refs

Yuck.  The less we abuse the preprocessor, the brighter the future of the
Python codebase, and that's a good concrete example of why.

and-that's-after-ignoring-the-redundant-redefinition-of-
    gc_next<wink>-ly y'rs  - tim



From mal@lemburg.com  Fri Oct 12 09:19:18 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 12 Oct 2001 10:19:18 +0200
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com>
 <3BC60BE2.D9182D64@ActiveState.com> <200110112122.RAA00321@cj20424-a.reston1.va.home.com>
Message-ID: <3BC6A786.BCB4A4D1@lemburg.com>

Guido van Rossum wrote:
> 
> > My real question is whether there is any value in having Unicode objects
> > expose their internal representation to Python programmers through the
> > buffer interface?
> 
> I used to think so, but I no longer believe this.  UTF-16 should be an
> encoding and that's that.

... and later ...

> > And I think I agree, even though that /could/ break code.  Then again,
> > maybe Paul's suggestion that hexlify() should reject Unicode strings
> > is the better approach.
> 
> +1

Since hexlify() uses a parser marker which does not involve a
type check, there's no way to have it reject Unicode objects.

BTW, the "s#" parser marker does *not* map to getreadbuffer
for Unicode objects. Long ago we decided that the difference
between "s#" and "t#" does not make sense for Unicode objects
and, in order to increase compatibility of Unicode objects to
existing code which uses "s#", to have both parser markers
map to getcharbuffer.

As a result, both parser markers return the default encoded
version of the Unicode object. The getreadbuffer interface
is still in place, though... perhaps we ought to consider
removing it ?!

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From paulp@ActiveState.com  Fri Oct 12 09:48:11 2001
From: paulp@ActiveState.com (Paul Prescod)
Date: Fri, 12 Oct 2001 01:48:11 -0700
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com>
 <3BC60BE2.D9182D64@ActiveState.com> <200110112122.RAA00321@cj20424-a.reston1.va.home.com> <3BC6A786.BCB4A4D1@lemburg.com>
Message-ID: <3BC6AE4B.F6B75744@ActiveState.com>

"M.-A. Lemburg" wrote:
> 
>...
> 
> Since hexlify() uses a parser marker which does not involve a
> type check, there's no way to have it reject Unicode objects.

Well, we do have the option of changing the code!

We could have hexlify check that its argument type is not Unicode or,
more likely, remove the buffer interface from Unicode objects. I think
that's the logical outcome of Guido's belief that Python programmers do
NOT need access to the internal representation of string objects. The
buffer interface seems to be only in existence to give access to
internal representations of objects.

 Paul Prescod


From paul@svensson.org  Fri Oct 12 13:10:33 2001
From: paul@svensson.org (Paul Svensson)
Date: Fri, 12 Oct 2001 08:10:33 -0400 (EDT)
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEIPMAAA.tim.one@home.com>
Message-ID: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>

On Thu, 11 Oct 2001, Tim Peters wrote:

>[Paul Svensson]
>> The usual thing to do in this situation would be:
>>
>> typedef union _gc_head {
>> 	struct {
>> 		union _gc_head *internal_gc_next;
>> 		union _gc_head *internal_gc_prev;
>> 		int internal_gc_refs;
>> 	} gc_internals;
>> 	double dummy;  /* force worst-case alignment */
>> } PyGC_Head;
>>
(fixed some typos)
>> #define gc_next	gc_internals.internal_gc_next
>> #define gc_prev	gc_internals.internal_gc_prev
>> #define gc_refs	gc_internals.internal_gc_refs
>
>Yuck.  The less we abuse the preprocessor, the brighter the future of the
>Python codebase, and that's a good concrete example of why.

Tim,
Why do you consider this abuse ?  This pattern is used all over the place,
you can study it for example in the system headers for linux of glibc.
It's perfectly legitimate to not want to spread knowledge of the innards
of a struct (it should be wrapped another layer, yes) all over the place.
Now, if you claim that for this particular struct,
hiding the internals is irrelevant, that's an argument I can buy.

I have anuther issue with the change tho: as written, it forces
the struct (it's only a union by accident) _gc_head to be aligned,
but is it obvious that it really has to pad up the size too ?
If for example sizeof (int) is 4, sizeof (struct *) is 8 and
sizeof (double) is 16, with 8 byte alignment required for doubles,
would a compiler be allowed to make sizeof (struct _gc_head[1]) 24,
while still keeping sizeof (struct _gc_head) at 20 ?
This would make a difference when making it part of another struct.

Try this on for size:
--------------------------------------------------------------------------
typedef struct _gc_head {
	struct _gc_head *internal_gc_next;
	struct _gc_head *internal_gc_prev;
	union _gc_head_internals {
		int _gc_head_internal_gc_refs;
		double dummy;	  /* force worst-case alignment */
	}
} PyGC_Head;

#define gc_refs	_gc_head_internals._gc_head_internal_gc_refs
--------------------------------------------------------------------------

	/Paul




From mal@lemburg.com  Fri Oct 12 12:20:10 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 12 Oct 2001 13:20:10 +0200
Subject: [Python-Dev] Bug or feature? Unicode vs t#
References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com>
 <3BC60BE2.D9182D64@ActiveState.com> <200110112122.RAA00321@cj20424-a.reston1.va.home.com> <3BC6A786.BCB4A4D1@lemburg.com> <3BC6AE4B.F6B75744@ActiveState.com>
Message-ID: <3BC6D1EA.4201B3C9@lemburg.com>

Paul Prescod wrote:
> 
> "M.-A. Lemburg" wrote:
> >
> >...
> >
> > Since hexlify() uses a parser marker which does not involve a
> > type check, there's no way to have it reject Unicode objects.
> 
> Well, we do have the option of changing the code!
> 
> We could have hexlify check that its argument type is not Unicode or,
> more likely, remove the buffer interface from Unicode objects.

As mentioned in that email, Unicode objects do not expose their
internals via the "s#" or "t#" parser markers. Removing the buffer
interface from Unicode would not solve the problem for these.

> I think
> that's the logical outcome of Guido's belief that Python programmers do
> NOT need access to the internal representation of string objects. The
> buffer interface seems to be only in existence to give access to
> internal representations of objects.

I can't follow you here: access to the internals is always possible
at C level and even at Python level they can use the unicode-internal
codec to peek at the internals. 8-bit strings don't differ between 
internal and external at all.

The question Guido raised really boils down to whether Unicode objects
should provide the getreadbuffer interface or not. I don't think it
is used in many places since most Python programmers writing
C extensions will use the parser markers to get at their arguments.
OTOH, getreadbuffer is meant to access the internals of an object,
e.g. arrays have this interface too, so this would be an argument
for not removing it.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From jim@interet.com  Fri Oct 12 14:04:08 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 12 Oct 2001 09:04:08 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
Message-ID: <3BC6EA48.DB476F16@interet.com>

Tim Peters wrote:

> Not so:  you're making a bad situation worse.  If an extern name doesn't
> begin with "_", it's part of Python's public C API, and *should* be
> documented and tested.  I don't think anyone will let that hold up your
> patch, although everyone probably should.

Yes, I think PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile()
are indeed part of Python's public C API, and that is why they were in
marshal.c with public declarations.

But they can't be tested by Lib/test_marshal.py because they are not
available in the marshal module.  I don't know of an existing C API
tester, and the names are highly descriptive in lieu of documentation.
And marshal.c has been relegated to supporting module.c anyway.
In short, I think we are getting a bit carried away.

JimA


From fdrake@acm.org  Fri Oct 12 14:15:03 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 12 Oct 2001 09:15:03 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: <3BC6EA48.DB476F16@interet.com>
References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
 <3BC6EA48.DB476F16@interet.com>
Message-ID: <15302.60631.186267.843484@grendel.zope.com>

James C. Ahlstrom writes:
 > Yes, I think PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile()
 > are indeed part of Python's public C API, and that is why they were in
 > marshal.c with public declarations.

  I think my "should they be?" question remains valid.

 > But they can't be tested by Lib/test_marshal.py because they are not
 > available in the marshal module.  I don't know of an existing C API
 > tester, and the names are highly descriptive in lieu of documentation.

  Modules/_testcapimodule.c.
  The long names we use for the C API make reading code that uses the
API very comfortable, but doesn't help with discoverability.  The
reference documentation could use some improvements for that, but
generally does much better than the headers, if only because you don't
need to be familiar with the arrangement of all the different headers.
(API users should not need to be familiar with all the headers; they
should only need to know to #include Python.h.)

 > And marshal.c has been relegated to supporting module.c anyway.
 > In short, I think we are getting a bit carried away.

  Is this a comment on the testing or the documentation requirements?
If the function is being sufficiently covered by other tests, then we
don't need to add new tests.  Since it was missing, I have to presume
it wasn't being called.  That tells me that it either isn't needed at
all or some test needs to exercise it.
  As for the documentation, I think the discoverability issue I
brought up indicates that we're doing the right thing -- its a lot
easier to learn the public API with documentation than without it.  I
don't know about the rest of the PythonLabs group, but I certainly
find the C API documentaiton useful.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From jim@interet.com  Fri Oct 12 14:45:39 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 12 Oct 2001 09:45:39 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
 <3BC6EA48.DB476F16@interet.com> <15302.60631.186267.843484@grendel.zope.com>
Message-ID: <3BC6F403.FE3A2454@interet.com>

"Fred L. Drake, Jr." wrote:
> 
> James C. Ahlstrom writes:
>  > Yes, I think PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile()
>  > are indeed part of Python's public C API, and that is why they were in
>  > marshal.c with public declarations.
> 
>   I think my "should they be?" question remains valid.

Ooops, I meant "they were in marshal.h".

Yes they should be public, because PyMarshal_ReadLongFromFile() is used
in import.c to read the Python byte code signature MAGIC from a *.pyc file.
I need a PyMarshal_ReadShortFromFile() to read zip archives in import.c.

>   Modules/_testcapimodule.c.

Thanks, I knew it had to be somewhere but I couldn't find it.

> If the function is being sufficiently covered by other tests, then we
> don't need to add new tests.  Since it was missing, I have to presume
> it wasn't being called.  That tells me that it either isn't needed at
> all or some test needs to exercise it.

Not true.  PyMarshal_ReadLongFromFile() is just a wraper function
for the function r_long() which is used internally by marshal.
Similarly, there is already an r_short() used internally by marshal.
Presumably, marshal would't work if these fail, so I assume they
work.  The PyMarshal_ReadShortFromFile() is just a wraper for r_short(),
not new funtionality.

>   As for the documentation, I think the discoverability issue I
> brought up indicates that we're doing the right thing -- its a lot
> easier to learn the public API with documentation than without it.  I
> don't know about the rest of the PythonLabs group, but I certainly
> find the C API documentaiton useful.

Yes, I find it essential, and wish there were more.

JimA


From nas@python.ca  Fri Oct 12 15:08:07 2001
From: nas@python.ca (Neil Schemenauer)
Date: Fri, 12 Oct 2001 07:08:07 -0700
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>; from paul@svensson.org on Fri, Oct 12, 2001 at 08:10:33AM -0400
References: <LNBBLJKPBEHFEDALKOLCGEIPMAAA.tim.one@home.com> <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>
Message-ID: <20011012070807.A8457@glacier.arctrix.com>

Paul Svensson wrote:
> On Thu, 11 Oct 2001, Tim Peters wrote:
> >[Paul Svensson]
> >> #define gc_next	gc_internals.internal_gc_next
> >Yuck.
> Why do you consider this abuse ?  This pattern is used all over the place,
> you can study it for example in the system headers for linux of glibc.
> It's perfectly legitimate to not want to spread knowledge of the innards
> of a struct (it should be wrapped another layer, yes) all over the place.

Huh?  Say I include a header that abomination in it.  Now consider the
following line of code:

    PyGC_Head *gc_next;

Oops.  The only reasonable alternative to Tim's approach that I can
think of would be:

  #define GC_NEXT(op) ((AS_GC(op))->gc.gc_next)
  #define GC_PREV(op) ((AS_GC(op))->gc.gc_prev)
  #define GC_REFS(op) ((AS_GC(op))->gc.gc_refs)

I don't think that's any better.

  Neil


From guido@python.org  Fri Oct 12 15:54:48 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 12 Oct 2001 10:54:48 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: Your message of "Thu, 11 Oct 2001 18:16:56 EDT."
 <15302.6744.531834.714184@grendel.zope.com>
References: <3BC61246.6EF6CFA@interet.com> <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
 <15302.6744.531834.714184@grendel.zope.com>
Message-ID: <200110121454.KAA04951@cj20424-a.reston1.va.home.com>

>   Of course, it would be easy to write the little bit of LaTeX needed
> to document the new function.  It would have the added benefit that
> I'd need to document the rest of the PyMarshal_*() functions so the
> new function would have a place to be listed.
>   Or all sorts of PyMarshal_*() functions could magically grow a
> leading underscore.  ;-)
>   Guido, were all of these really intended to be public, or can the
> list be trimmed back?  They really only appear to be used in import.c
> and pythonrun.c, with the module init also appear in PC/config.c and
> Modules/config.c.in (and the generated Modules/config.c).

Hm.  I consider them pretty internal, but on the other hand there are
uses for these outside the core (e.g. a hypothetical extension that
might want to manipulate .pyc files).  So let's document them.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Fri Oct 12 16:41:09 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 12 Oct 2001 11:41:09 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
 <3BC6EA48.DB476F16@interet.com> <15302.60631.186267.843484@grendel.zope.com>
Message-ID: <3BC70F15.206D6C24@interet.com>

"Fred L. Drake, Jr." wrote:
> don't know about the rest of the PythonLabs group, but I certainly
> find the C API documentaiton useful.

Hmmm.  It occurs to me that I should write some.  I just
submitted another patch with the docs.

JimA


From fdrake@acm.org  Fri Oct 12 16:53:45 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 12 Oct 2001 11:53:45 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: <3BC70F15.206D6C24@interet.com>
References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com>
 <3BC6EA48.DB476F16@interet.com>
 <15302.60631.186267.843484@grendel.zope.com>
 <3BC70F15.206D6C24@interet.com>
Message-ID: <15303.4617.550167.403762@grendel.zope.com>

James C. Ahlstrom writes:
 > Hmmm.  It occurs to me that I should write some.  I just
 > submitted another patch with the docs.

  Thanks!


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Fri Oct 12 20:06:12 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 12 Oct 2001 15:06:12 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: Your message of "Fri, 12 Oct 2001 08:10:33 EDT."
 <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>
References: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>
Message-ID: <200110121906.PAA13531@cj20424-a.reston1.va.home.com>

> I have anuther issue with the change tho: as written, it forces
> the struct (it's only a union by accident) _gc_head to be aligned,
> but is it obvious that it really has to pad up the size too ?

Yes.

> If for example sizeof (int) is 4, sizeof (struct *) is 8 and
> sizeof (double) is 16, with 8 byte alignment required for doubles,
> would a compiler be allowed to make sizeof (struct _gc_head[1]) 24,
> while still keeping sizeof (struct _gc_head) at 20 ?

No.  Each item in an array has to be aligned, so the sizeof a struct
must be a multiple of the largest required alignment for any of its
items.  It's all spelled out in the C standard, so don't worry.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Fri Oct 12 20:15:53 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 12 Oct 2001 15:15:53 -0400
Subject: [Python-Dev] profiler hooks
Message-ID: <15303.16745.570827.948656@grendel.zope.com>

  For those who aren't already aware of it, I've been working on a new
profiler for Python, implemented in C.  It includes features that will
allow per-line profiling if desired.  I should have the implementation
checked in shortly.
  One thing that I'd like to change is the way the profiler/tracer are
dispatched from the main eval loop.  The changes are easy enough, but
I can simplify things further if we don't need to support both a
profiler and tracer simultaneously.  (The trace hook is used by
sys.settrace(), needed for the debugger and 3rd-party coverage tools;
the profile hook is used for (you guessed it!) the profiler, via
sys.setprofile().)
  Would anyone object to working things so that only one of the two
can be installed at a time?  They can both operate, but it doesn't
make sense to use the trace hook while a profiler is active.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From skip@pobox.com (Skip Montanaro)  Fri Oct 12 20:46:56 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 12 Oct 2001 14:46:56 -0500
Subject: [Python-Dev] profiler hooks
In-Reply-To: <15303.16745.570827.948656@grendel.zope.com>
References: <15303.16745.570827.948656@grendel.zope.com>
Message-ID: <15303.18608.490130.307506@beluga.mojam.com>

    Fred> ... I can simplify things further if we don't need to support both
    Fred> a profiler and tracer simultaneously.  Would anyone object to
    Fred> working things so that only one of the two can be installed at a
    Fred> time?

+1

Skip


From pedroni@inf.ethz.ch  Fri Oct 12 21:16:16 2001
From: pedroni@inf.ethz.ch (Samuele Pedroni)
Date: Fri, 12 Oct 2001 22:16:16 +0200
Subject: [Python-Dev] profiler hooks
References: <15303.16745.570827.948656@grendel.zope.com>
Message-ID: <007401c1535a$c0977d00$8e0bfea9@newmexico>

> 
>   For those who aren't already aware of it, I've been working on a new
> profiler for Python, implemented in C.  It includes features that will
> allow per-line profiling if desired.  I should have the implementation
> checked in shortly.
>   One thing that I'd like to change is the way the profiler/tracer are
> dispatched from the main eval loop.  The changes are easy enough, but
> I can simplify things further if we don't need to support both a
> profiler and tracer simultaneously.  (The trace hook is used by
> sys.settrace(), needed for the debugger and 3rd-party coverage tools;
> the profile hook is used for (you guessed it!) the profiler, via
> sys.setprofile().)

Will your changes propagate to changes to pdb etc?
because in that case they will need to be ported
to jython, that actually supports both tracers (pdb ) and profilers 
(profile ).

regards.



From fdrake@acm.org  Fri Oct 12 22:03:36 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 12 Oct 2001 17:03:36 -0400
Subject: [Python-Dev] profiler hooks
In-Reply-To: <007401c1535a$c0977d00$8e0bfea9@newmexico>
References: <15303.16745.570827.948656@grendel.zope.com>
 <007401c1535a$c0977d00$8e0bfea9@newmexico>
Message-ID: <15303.23208.240933.689397@grendel.zope.com>

Samuele Pedroni writes:
 > Will your changes propagate to changes to pdb etc?
 > because in that case they will need to be ported
 > to jython, that actually supports both tracers (pdb ) and profilers 
 > (profile ).

  Changes to bdb/pdb will probably be required, but I'm not sure yet.
The biggest change I expect for debugging is that a 'return' event
will be generated even when an exception is being propogated, but what
this means is that the code that runs for the 'exception' event will
not need to be so magical (this was why the old profiler results were
so terribly difficult to interpret -- the times were often wrong
depending on just how exceptions were handled in the application
code).
  The recent changes to ceval.c for the profiler ensure that the
profiler receives a 'return' event when control leaves a frame, even
if an exception was set, and expception events are no longer reported
to the profiler.  This avoids the messy code needed to ensure the
profiler accounted for timimg properly for 'exception' events, which
is still quite fragile -- it works very hard just to make sure it
doesn't lose track of the stack.  Making the tracer also receive
matching call/return events means that the exception events should no
longer need to worry about stack management, which should make the
code simpler.
  I don't know if Jython uses the same pdb.py as CPython, so I don't
know what affects will show up there.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From pedroni@inf.ethz.ch  Fri Oct 12 22:19:14 2001
From: pedroni@inf.ethz.ch (Samuele Pedroni)
Date: Fri, 12 Oct 2001 23:19:14 +0200
Subject: [Python-Dev] profiler hooks
References: <15303.16745.570827.948656@grendel.zope.com><007401c1535a$c0977d00$8e0bfea9@newmexico> <15303.23208.240933.689397@grendel.zope.com>
Message-ID: <004101c15363$8c9dbce0$8e0bfea9@newmexico>

> 
>   Changes to bdb/pdb will probably be required, but I'm not sure yet.
> The biggest change I expect for debugging is that a 'return' event
> will be generated even when an exception is being propogated, but what
> this means is that the code that runs for the 'exception' event will
> not need to be so magical (this was why the old profiler results were
> so terribly difficult to interpret -- the times were often wrong
> depending on just how exceptions were handled in the application
> code).
>   The recent changes to ceval.c for the profiler ensure that the
> profiler receives a 'return' event when control leaves a frame, even
> if an exception was set, and expception events are no longer reported
> to the profiler.  This avoids the messy code needed to ensure the
> profiler accounted for timimg properly for 'exception' events, which
> is still quite fragile -- it works very hard just to make sure it
> doesn't lose track of the stack.  Making the tracer also receive
> matching call/return events means that the exception events should no
> longer need to worry about stack management, which should make the
> code simpler.
>   I don't know if Jython uses the same pdb.py as CPython, so I don't
> know what affects will show up there.
> 

I know, and yes pdb.py worked with jython too, that's why I asked
about the changes in the first place ... another to-do for jython 2.2

regards. 



From tim@zope.com  Fri Oct 12 23:55:40 2001
From: tim@zope.com (Tim Peters)
Date: Fri, 12 Oct 2001 18:55:40 -0400
Subject: [Python-Dev] GC head alignment issue
In-Reply-To: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>
Message-ID: <BIEJKCLHCIOIHAGOKOLHCEIICIAA.tim@zope.com>

[Paul Svensson]
> ...
> #define gc_next	gc_internals.internal_gc_next
> ...
> Why do you consider this abuse ?

Such macros would be visible to every file #include'ing Python.h, and it's
certainly not our intent to remove 3 ordinary identifiers from the namespace
of every program extending or embedding Python.  That brittleness makes it
"abuse" on its own.

Short of that, it complicates life for developers more than the
straightforward alternate I checked in.  For example, ask a random debugger
for the current value of p->gc_next, and chances are great it will complain
that there is no such member (not doing macro expansion itself); or, on the
flip side, ask for a dump of a PyGC_Head*, and you'll see fields with names
that are utterly surprising because never mentioned in the code using
PyGC_Head pointers.  Telling and using the plain truth doesn't cause any
difficulties, save a tiny bit of extra typing I was happy to do (if I didn't
know better, I'd think *you* had to insert "gc." a hundred times yourself
<wink>).

> This pattern is used all over the place, you can study it for example in
> the system headers for linux of glibc.

Perhaps one reason Python is more stable than, and spends thousands of lines
worming around bugs in, the C libraries it relies on <wink>.

> It's perfectly legitimate to not want to spread knowledge of the innards
> of a struct (it should be wrapped another layer, yes) all over the place.

It's not all over the place -- only the track and untrack macros, and the gc
module, have any business mucking with this union (nee struct); indeed, it's
in the header file only so that the macros can get at the guts.

> ...
> I have anuther issue with the change tho: as written, it forces
> the struct (it's only a union by accident) _gc_head to be aligned,
> but is it obvious that it really has to pad up the size too ?

It may not have to, and that's "a feature".

> If for example sizeof (int) is 4, sizeof (struct *) is 8 and
> sizeof (double) is 16, with 8 byte alignment required for doubles,
> would a compiler be allowed to make sizeof (struct _gc_head[1]) 24,
> while still keeping sizeof (struct _gc_head) at 20 ?

A std-conforming C compiler is not allowed to do that, as, in general, the
compiler can't know whether a struct S* points to the first S in an array of
S, or to an "isolated" S.  So all Ss must have the same layout, regardless
of context.  As Guido said, the C std goes on at length about this stuff.

> This would make a difference when making it part of another struct.

Again, the compiler generally can't know whether a struct S* points to an
instance of S embedded in another struct either, so again it can't change
layout depending on context.

> Try this on for size:
> ----------------------------------------------------------------------
> typedef struct _gc_head {
> 	struct _gc_head *internal_gc_next;
> 	struct _gc_head *internal_gc_prev;
> 	union _gc_head_internals {
> 		int _gc_head_internal_gc_refs;
> 		double dummy;	  /* force worst-case alignment */
> 	}
> } PyGC_Head;
>
> #define gc_refs	_gc_head_internals._gc_head_internal_gc_refs
> -------------------------------------------------------------------------

As before, I've got no use for the macro.  Given that, I'd rather be
consistent in sticking a "gc." prefix on every struct member than on just
one of them.



From tim.one@home.com  Sat Oct 13 04:56:58 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 12 Oct 2001 23:56:58 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
In-Reply-To: <3BC6F403.FE3A2454@interet.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKELMMAAA.tim.one@home.com>

[James C. Ahlstrom]
> Yes they should be public, because PyMarshal_ReadLongFromFile() is used
> in import.c to read the Python byte code signature MAGIC from a *.pyc
> file.  I need a PyMarshal_ReadShortFromFile() to read zip archives in
> import.c.

There's a finer distinction to be drawn:  "public" doesn't mean "C extern",
it means a conscious decision that if the API in question ever changes, we
promise to bump the Python API version number and so force extension modules
to be recompiled (or at least irritate the user with warnings that they
*should* be recompiled).  IOW, "public" means "anybody in the world can use
this, in or out of the core distribution, and we promise it will never break
silently across releases".  If you use a C extern function like (say)
_PyString_FormatLong outside of the core, and we (say) change the calling
sequence, or even get rid of it, tough luck -- we don't promise anything
about functions (or macros) whose names begin with an underscore (they're
part of "the private C API", C extern or not).

import.c is part of the core, so can use all the private API functions it
likes (and we're responsible for making sure it works across releases no
matter how wildly the private API functions change).




From neal@metaslash.com  Sat Oct 13 16:30:56 2001
From: neal@metaslash.com (Neal Norwitz)
Date: Sat, 13 Oct 2001 11:30:56 -0400
Subject: [Python-Dev] pychecker warnings from python-current
Message-ID: <3BC85E30.FACC6C19@metaslash.com>

Here's a new set of warnings from pychecker (current) against 
python-current as of 13 Oct:

cgi.py:250: No global (ib) found
    (no idea which variable is desired, best guess is boundary)

gzip.py:292: No global (f) found
    (f should be self?)

markupbase.py:306: Invalid arguments to (error), got 2, expected 1
    (not sure if this is a problem or not)

rexec.py:262: Using the return value from (execfile) which is always None
    (not really a problem, but implies there is a return value other than None)

smtplib.py:540: No global (SMTPError) found
    (should be SMTPException?)

urllib.py:587: Invalid arguments to (http_error_default), got 5, expected 6
    (needs errcode passed as 4th arg, between fp & errmsg)

webbrowser.py:182: No global (_os) found
    (should be os)

There's also a problem in Objects/stringobject.c on line 2541.  The docstring
for ''.splitlines() has an extra ].

Neal


From fdrake@acm.org  Sat Oct 13 16:54:37 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Sat, 13 Oct 2001 11:54:37 -0400
Subject: [Python-Dev] pychecker warnings from python-current
In-Reply-To: <3BC85E30.FACC6C19@metaslash.com>
References: <3BC85E30.FACC6C19@metaslash.com>
Message-ID: <15304.25533.445048.365261@grendel.zope.com>

Neal Norwitz writes:
 > markupbase.py:306: Invalid arguments to (error), got 2, expected 1
 >     (not sure if this is a problem or not)
...
 > webbrowser.py:182: No global (_os) found
 >     (should be os)
 > 
 > There's also a problem in Objects/stringobject.c on line 2541.  The docstring
 > for ''.splitlines() has an extra ].

  These have been fixed; we'll have to look at the rest later.  (I
have to take a child to a soccor game now!)
  Thanks!


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From fdrake@acm.org  Sat Oct 13 19:33:38 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Sat, 13 Oct 2001 14:33:38 -0400
Subject: [Python-Dev] pychecker warnings from python-current
In-Reply-To: <3BC85E30.FACC6C19@metaslash.com>
References: <3BC85E30.FACC6C19@metaslash.com>
Message-ID: <15304.35074.718737.112730@grendel.zope.com>

Neal Norwitz writes:
 > Here's a new set of warnings from pychecker (current) against 
 > python-current as of 13 Oct:

  Ok, I'm back from soccer.  ;-)  Your suggested changes all look good
to me; I've checked them in.  Now we just need test cases for them!


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From Keila - Curitiba - Pr"<pastorinha@ieg.com.br  Sun Oct 14 03:13:12 2001
From: Keila - Curitiba - Pr"<pastorinha@ieg.com.br (Keila - Curitiba - Pr)
Date: Sat, 13 Oct 2001 23:13:12 -0300
Subject: [Python-Dev] UM SITE DE PROTESTO
Message-ID: <E15saoy-0002f0-00@mail.python.org>

Olá!
Veja meu site pessoal no "Tripod.com.br". Basta clicar no endereço
abaixo. GARANTO SER SUI-GENERIS  -  CLIQUE ABAIXO:
http://pastorinha.tripod.com.br/seminarista
Mais de 61.000 internautas visitaram a PG., existe 7 Álbuns:
Se você quiser, por favor, indique minha Home Page, a outros
Internautas.  Mais detalhes, se comunique, passe um e-mail, que
responderei brevemente. Dentro da Home Page, ao lado das fotos,
você poderá saber muito mais sobre mim!
Obrigada.
e-mail: pastorinha@ieg.com.br
Beijos:-     Keila - Curitiba - Pr
- Podes falar comigo, direto dela. Brevemente uma Carta Aberta.
http://pastorinha.tripod.com.br/seminarista

"Esta mensagem é enviada com a complacência da nova legislação 
sobre correio eletrônico, Seção 301, Parágrafo (a) (2) (c) Decreto S.
1618, Título Terceiro aprovado pelo "105º Congresso Base das 
Normativas Internacionais sobre o SPAM". Este E-mail não poderá
ser considerado SPAM quando incluir uma forma de ser removido.
Para ser removido de futuros correios, simplesmente responda
indicando no Assunto: REMOVER"


From fdrake@acm.org  Sun Oct 14 05:46:12 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Sun, 14 Oct 2001 00:46:12 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011014044612.90DE228697@beowolf.digicool.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Added documentation for the PyMarshal_*() functions in the Python/C API.



From tim.one@home.com  Sun Oct 14 20:50:36 2001
From: tim.one@home.com (Tim Peters)
Date: Sun, 14 Oct 2001 15:50:36 -0400
Subject: [Python-Dev] conditional expressions (RE: Loop-and-a-half (Re: Curious assignment behaviour))
In-Reply-To: <7xhet39ckb.fsf@ruckus.brouhaha.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEOCMAAA.tim.one@home.com>

[Tim]
> If people sign off on taking "then" as a new keyword, I think
> the chances are good that we could get
>
>     x = if e1 then e2 else e3
>
> into 2.2b1.  That's the only obvious spelling, hence the only
> truly Pythonic way to spell it.  Other languages spelling it that
> way range from Algol-60 (Guido's first intense language affair) to
> Haskell.

[Paul Rubin, among others of similar mind]
> This sounds fine to me.

Alas, it didn't to Python's parser -- one-token lookahead isn't enough to
distinguish

    if 1:

from

    if 1 then 2 else 3

let alone

    if a + b / c:

from

    if a + b / c then 2 else 3

etc.

and Python won't grow anything a simple parser can't sort out.

Everything's cool if parens are required around a conditional expression,
though, in which case:

    x = if e1 then e2 else e3 + 1   # SyntaxError
    x = (if e1 then e2 else e3) + 1 # cool
    x = (if e1 then e2 else e3 + 1) # cool
    x = if e1 then e2 else e3       # SyntaxError
    x = (if e1 then e2 else e3)     # cool
    x = if if e1 then e2 else e3 then e4 else e5     # SyntaxError
    x = (if (if e1 then e2 else e3) then e4 else e5) # cool

Seems a mixed bag, but I'm more interested in readability and the
functionality than in minimizing keystrokes; requiring parens doesn't hurt
the goals I care about.

implemented-but-not-checked-in-ly y'rs  - tim



From aahz@panix.com  Mon Oct 15 03:27:34 2001
From: aahz@panix.com (aahz@panix.com)
Date: Sun, 14 Oct 2001 22:27:34 -0400 (EDT)
Subject: [Python-Dev] Will Ware alive? (was Re: where lives uthread (microthreads) now ??)
References: <3b091a1c.0110080739.23205b07@posting.google.com>
Message-ID: <200110150227.f9F2RYm18907@panix2.panix.com>

[x-posted to c.l.py and python-dev]

In article <3b091a1c.0110080739.23205b07@posting.google.com>,
cmkl <cmkleffner@gmx.de> wrote:
>
>I wanted to test a samle code with microthreads needed.
>The download area: http://world.std.com/~wware/uthread.html
>is not available however. 
>
>Is there a mirror to get this code somewhere?

Has anybody heard from Will Ware since 9/12?  E-mail to him doesn't
bounce, but I haven't received any reply.
-- 
                      --- Aahz  <*>  (Copyright 2001 by aahz@pobox.com)

Hugs and backrubs -- I break Rule 6                 http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista   

We must not let the evil of a few trample the freedoms of the many.
-- 
                      --- Aahz  <*>  (Copyright 2001 by aahz@pobox.com)

Hugs and backrubs -- I break Rule 6                 http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista   

We must not let the evil of a few trample the freedoms of the many.


From skip@pobox.com (Skip Montanaro)  Mon Oct 15 13:57:46 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 07:57:46 -0500
Subject: [Python-Dev] Problem with Py_NewInterpreter
Message-ID: <15306.56650.146816.403758@beluga.mojam.com>

Don't know why it got posted where it did (in the mysql-python forum on SF),
but some guy posted a question/problem about executing Py_NewInterpreter
twice.  I have no experience with this sort of stuff, so I figured I'd
forward it along to the people with the smarts in this area:

    http://sourceforge.net/forum/message.php?msg_id=249171

(Need I say, "don't respond to me"? ;-)

Skip


From jim@interet.com  Mon Oct 15 14:40:13 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 15 Oct 2001 09:40:13 -0400
Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ?
References: <LNBBLJKPBEHFEDALKOLCKELMMAAA.tim.one@home.com>
Message-ID: <3BCAE73D.A1935CAE@interet.com>

Tim Peters wrote:

> There's a finer distinction to be drawn:  "public" doesn't mean "C extern",
> ...

Ok, thanks for drawing this distinction.  So perhaps these
functions should have an initial underscore.  I have no
opinion on this.  I assumed the name PyMarshal_ReadLongFromFile()
looked very official and was therefore "public".

JimA


From guido@python.org  Mon Oct 15 16:11:45 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 11:11:45 -0400
Subject: [Python-Dev] Problem with Py_NewInterpreter
In-Reply-To: Your message of "Mon, 15 Oct 2001 07:57:46 CDT."
 <15306.56650.146816.403758@beluga.mojam.com>
References: <15306.56650.146816.403758@beluga.mojam.com>
Message-ID: <200110151511.LAA07344@cj20424-a.reston1.va.home.com>

> Don't know why it got posted where it did (in the mysql-python forum on SF),
> but some guy posted a question/problem about executing Py_NewInterpreter
> twice.  I have no experience with this sort of stuff, so I figured I'd
> forward it along to the people with the smarts in this area:
> 
>     http://sourceforge.net/forum/message.php?msg_id=249171
> 
> (Need I say, "don't respond to me"? ;-)

I've posted a quick pointer to Demo/pysvr which shows how to use
Py_NewInterpreter().

It is possible that mysql contains global state that can't be shared
between interpreters; mysql may have to be changed to fix that.  But I
don't know.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From skip@pobox.com (Skip Montanaro)  Mon Oct 15 19:05:19 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 13:05:19 -0500
Subject: [Python-Dev] conditional expressions?
Message-ID: <15307.9567.187259.557683@beluga.mojam.com>

Is the thread in c.l.py on conditional expressions leading in the direction
of actually getting added to the language, or wer Tim's experiments done
just to show it's possible?

Skip


From tim.one@home.com  Mon Oct 15 19:25:11 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 15 Oct 2001 14:25:11 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <15307.9567.187259.557683@beluga.mojam.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com>

[Skip Montanaro]
> Is the thread in c.l.py on conditional expressions leading in the
> direction of actually getting added to the language, or wer Tim's
> experiments done just to show it's possible?

We (PythonLabs) were wondering whether to reserve any more keywords for 2.2.
"then" was a natural candidate, for this specific use.  Guido and I have
been playing with it.  If it proves to be a low-hassle, low-impact change
that works well, the intent is to get it in for 2.2b1 later this week.
Doesn't look *likely* to me at this point, but don't know.



From paul@svensson.org  Mon Oct 15 19:30:04 2001
From: paul@svensson.org (Paul Svensson)
Date: Mon, 15 Oct 2001 14:30:04 -0400 (EDT)
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <15307.9567.187259.557683@beluga.mojam.com>
Message-ID: <Pine.LNX.4.21.0110151427100.2552-100000@familjen.svensson.org>

On Mon, 15 Oct 2001, Skip Montanaro wrote:

>Is the thread in c.l.py on conditional expressions leading in the direction
>of actually getting added to the language, or wer Tim's experiments done
>just to show it's possible?

Ick. Please don't tell me that's more than a bad joke ?

	/Paul



From skip@pobox.com (Skip Montanaro)  Mon Oct 15 19:46:44 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 13:46:44 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com>
References: <15307.9567.187259.557683@beluga.mojam.com>
 <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com>
Message-ID: <15307.12052.100384.784424@beluga.mojam.com>

    Tim> [Skip Montanaro]
    >> Is the thread in c.l.py on conditional expressions leading in the
    >> direction of actually getting added to the language, or wer Tim's
    >> experiments done just to show it's possible?

    Tim> We (PythonLabs) were wondering whether to reserve any more keywords
    Tim> for 2.2.  "then" was a natural candidate, for this specific use.
    Tim> Guido and I have been playing with it.  If it proves to be a
    Tim> low-hassle, low-impact change that works well, the intent is to get
    Tim> it in for 2.2b1 later this week.  Doesn't look *likely* to me at
    Tim> this point, but don't know.

Regardless whether or not you think this could make it into 2.2b1, I hope if
you proceed it will get a PEP a reasonable amount of time before the CVS
checkin... <0.1 wink>.

It seems downright weird to me that the syntactic baggage necessary to write
a conditional expression is greater the the baggage necessary to write an if
statement (new "then" keyword, parens required for disambiguation).  The
parens function pretty much as "{...}" in C, Java, Perl, etc.  It's a step
away from indentation-based block structure and toward delimiter-based block
structure.  If you add it, I think it will be harder to justify the lack of
delimiter-based block stucture at the statement level.  It will just be one
more argument in the arsenal of people whose knee-jerk reaction to Python's
block structure is to whine about it.

Skip



From tim.one@home.com  Mon Oct 15 19:59:21 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 15 Oct 2001 14:59:21 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <15307.12052.100384.784424@beluga.mojam.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>

[Skip Montanaro]
> Regardless whether or not you think this could make it into
> 2.2b1, I hope if you proceed it will get a PEP a reasonable
> amount of time before the CVS checkin... <0.1 wink>.

Guido can exempt himself from the PEP process if he feels like it.  If he
does, I expect I'll be in the "retroactive PEP" business again.

> It seems downright weird to me that the syntactic baggage
> necessary to write a conditional expression is greater the the
> baggage necessary to write an if statement (new "then" keyword, parens
> required for disambiguation).

There are many ways a conditional expression could be spelled.  This is
aiming at the easiest way to read; harder-to-read ways could be introduced
with less disruption of the kinds you're focusing on.

> The parens function pretty much as "{...}" in C, Java, Perl, etc.

Sorry, I don't see the resemblance.

> It's a step away from indentation-based block structure and toward
> delimiter-based block structure.

Ditto.

> If you add it, I think it will be harder to justify the lack of
> delimiter-based block stucture at the statement level.  It will
> just be one more argument in the arsenal of people whose knee-jerk
> reaction to Python's block structure is to whine about it.

I'm not concerned about 21 bad arguments versus 20 <wink>.



From skip@pobox.com (Skip Montanaro)  Mon Oct 15 20:20:27 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 14:20:27 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>
References: <15307.12052.100384.784424@beluga.mojam.com>
 <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>
Message-ID: <15307.14075.322035.39614@beluga.mojam.com>

    >> Regardless whether or not you think this could make it into 2.2b1, I
    >> hope if you proceed it will get a PEP a reasonable amount of time
    >> before the CVS checkin... <0.1 wink>.

    Tim> Guido can exempt himself from the PEP process if he feels like it.
    Tim> If he does, I expect I'll be in the "retroactive PEP" business
    Tim> again.

I'm sure he can, but that doesn't mean he ought to. <wink>

    >> The parens function pretty much as "{...}" in C, Java, Perl, etc.

    Tim> Sorry, I don't see the resemblance.

Well, the examples I saw looked like

    x = (if (if e1 then e2 else e3) then e4 else e5)

Seems to me that those inner parens are there to separate the inner
conditional from the trailing parts of the outer conditional:

    x = (
        if (
            if (
               e1
            then
               e2
            else
               e3
            )
        then
            e4
        else
            e5
        )

Whether or not that's what they are in a language parser sense, they sure
look like it to the human eye.

    >> If you add it, I think it will be harder to justify the lack of
    >> delimiter-based block stucture at the statement level.  It will just
    >> be one more argument in the arsenal of people whose knee-jerk
    >> reaction to Python's block structure is to whine about it.

    Tim> I'm not concerned about 21 bad arguments versus 20 <wink>.

That 21st argument will be a little stronger than the other, because it will
go something like, "Why can't we have delimiters for block statements?
After all, we have them in conditional expressions."  The reference, instead
of being to another language, will be to Python itself.

Skip



From guido@python.org  Mon Oct 15 20:24:03 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 15:24:03 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 13:46:44 CDT."
 <15307.12052.100384.784424@beluga.mojam.com>
References: <15307.9567.187259.557683@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com>
 <15307.12052.100384.784424@beluga.mojam.com>
Message-ID: <200110151924.PAA25209@cj20424-a.reston1.va.home.com>

[Skip]
>     >> Is the thread in c.l.py on conditional expressions leading in the
>     >> direction of actually getting added to the language, or wer Tim's
>     >> experiments done just to show it's possible?

[Paul Svensson]
> Ick. Please don't tell me that's more than a bad joke ?

[Tim]
>     Tim> We (PythonLabs) were wondering whether to reserve any more
>     Tim> keywords for 2.2.  "then" was a natural candidate, for this
>     Tim> specific use.  Guido and I have been playing with it.  If
>     Tim> it proves to be a low-hassle, low-impact change that works
>     Tim> well, the intent is to get it in for 2.2b1 later this week.
>     Tim> Doesn't look *likely* to me at this point, but don't know.

[Skip]
> Regardless whether or not you think this could make it into 2.2b1, I
> hope if you proceed it will get a PEP a reasonable amount of time
> before the CVS checkin... <0.1 wink>.
> 
> It seems downright weird to me that the syntactic baggage necessary
> to write a conditional expression is greater the the baggage
> necessary to write an if statement (new "then" keyword, parens
> required for disambiguation).  The parens function pretty much as
> "{...}" in C, Java, Perl, etc.  It's a step away from
> indentation-based block structure and toward delimiter-based block
> structure.  If you add it, I think it will be harder to justify the
> lack of delimiter-based block stucture at the statement level.  It
> will just be one more argument in the arsenal of people whose
> knee-jerk reaction to Python's block structure is to whine about it.

I think you must be misunderstanding the proposal, which is to add

    if <expr> then <expr> else <expr>

as an alternative to the expression syntax.  Here's a preliminary
patch (which I won't apply until I have more confidence that this is
acceptable to the community):

    http://sourceforge.net/tracker/index.php?func=detail&aid=471421&group_id=5470&atid=305470

The parens in this proposal (my version) act no different than any
other use of parentheses in Python expressions.  Basically, you need
to add parentheses to disambiguate expressions:

- if otherwise the 'if' keyword would be the start of a statement
  (because 'if' at the start of a statement starts an if *statement*,
  and the parser can't look ahead for the 'then' keyword);

- if the conditional expression is to be combined with a unary or
  binary operator.

Some examples where no parentheses are needed (note that a comma binds
less tight than a conditional expression -- same as for lambda):

    x = if 1 then 2 else 3, y
    f(if 1 then 2 else 3, y)
    a[if 1 then 2 else 3, y]
    `if 1 then 2 else 3`
    lambda: if 1 then 2 else 3

Some examples where parentheses *are* required:

    (if 1 then 2 else 3) + 4
    a[(if i then 2 else 3) : 4]

In some situations I'm not sure what's right; The un-parenthesized
form looks weird although it's not neede to avoid ambiguity:

    if (if 1 then 2 else 3): pass
    print (if 1 then 2 else 3)
    for i in (if 1 then "abc" else "def"): pass

I'd be happy to discuss this more.  I'm *not* happy with responses
like "is this a bad joke?".  I don't understand how this could be an
argument in the arsenal of the anti-Python league.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 15 20:34:19 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 15:34:19 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 14:20:27 CDT."
 <15307.14075.322035.39614@beluga.mojam.com>
References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>
 <15307.14075.322035.39614@beluga.mojam.com>
Message-ID: <200110151934.PAA25299@cj20424-a.reston1.va.home.com>

> Well, the examples I saw looked like
> 
>     x = (if (if e1 then e2 else e3) then e4 else e5)
> 
> Seems to me that those inner parens are there to separate the inner
> conditional from the trailing parts of the outer conditional:
> 
>     x = (
>         if (
>             if (
                 ^
this is not there

>                e1
>             then
>                e2
>             else
>                e3
>             )
>         then
>             e4
>         else
>             e5
>         )
> 
> Whether or not that's what they are in a language parser sense, they sure
> look like it to the human eye.

They are syntactically unnecessary; they are mostly for guidance of
the human reader.  If I saw a piece of code that read

    x = if if if x == 1 then y else z then p else q then a else b

I would get a strong urge to commit illegal violence against the
author.  If on the other hand I saw

    x = if (if (if x == 1 then y else z) then p else q) then a else b

I might be willing to sit down and figure out what it meant -- maybe
with the help of a parentheses-balancing command in my editor.

>     Tim> I'm not concerned about 21 bad arguments versus 20 <wink>.
> 
> That 21st argument will be a little stronger than the other, because
> it will go something like, "Why can't we have delimiters for block
> statements?  After all, we have them in conditional expressions."
> The reference, instead of being to another language, will be to
> Python itself.

Sorry, I still don't get this at all.  A conditional expression is
still an expression.  Curly brances are a statement-level concept.
What am I missing?  Where is the similarity between the use of { } and
( ) in C/Java/C++/Perl?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From paul-python@svensson.org  Mon Oct 15 21:04:59 2001
From: paul-python@svensson.org (Paul Svensson)
Date: Mon, 15 Oct 2001 16:04:59 -0400 (EDT)
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110151924.PAA25209@cj20424-a.reston1.va.home.com>
Message-ID: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>

On Mon, 15 Oct 2001, Guido van Rossum wrote:

>I'd be happy to discuss this more.  I'm *not* happy with responses
>like "is this a bad joke?".  I don't understand how this could be an
>argument in the arsenal of the anti-Python league.

I'm sorry if it hit a bad nerve,
but I did skip this whole thread on c.l.py,
thinking that this could surely not be seriously meant,
so I guess don't have the understanding of the motivation
behind the proposal that I could have had.

I know there's been a long time request for this feature,
supposedly from people coming to Python from C,
but what's the real reason for adding it now,
without even a PEP ?
Does it really add enough to the language to justify the bloat ?

	/Paul



From skip@pobox.com (Skip Montanaro)  Mon Oct 15 21:13:48 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 15:13:48 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110151934.PAA25299@cj20424-a.reston1.va.home.com>
References: <15307.12052.100384.784424@beluga.mojam.com>
 <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>
 <15307.14075.322035.39614@beluga.mojam.com>
 <200110151934.PAA25299@cj20424-a.reston1.va.home.com>
Message-ID: <15307.17276.738359.631642@beluga.mojam.com>

I misunderstood about the need for parens.  I had only been watching the
thread in python-list with one eye.  I thought it was just some python-list
riff-raff throwing around ideas.  Then I saw that Tim was part of the rabble
;-).  I still didn't notice the "I've implemented it but not yet checked it
in" bit until someone else did and asked about it.  I then went looking for
a PEP but didn't see one.  I hadn't seen any discussion on python-dev
either.  (Maybe I missed it because python-list and python-dev mail wind up
in the same mailbox.)

    Guido> If I saw a piece of code that read

    Guido>     x = if if if x == 1 then y else z then p else q then a else b

    Guido> I would get a strong urge to commit illegal violence against the
    Guido> author.  If on the other hand I saw

    Guido>     x = if (if (if x == 1 then y else z) then p else q) then a else b

    Guido> I might be willing to sit down and figure out what it meant --
    Guido> maybe with the help of a parentheses-balancing command in my
    Guido> editor.

Python's block structured statement syntax keep you from composing just
monstrosities at the statement level.  I'm not sure what makes them more
appealing at the expression level.  How many levels of parens would be
required before you had a "strong urge to commit illegal violence" against
someone using a parenthesized conditional expression?

I guess that's what it boils down to for me.  You can easily write much more
inscrutable code with the new conditional expression than you could before.
I think that will make Python somewhat less of a CP4E language.  I don't
think it falls into the same category of the more esoteric features of the
language (generators, metaclasses, __*__ methods, etc) that beginners can
safely ignore.  Conditional expressions are pretty basic stuff, and this
makes it easier to write fairly inscrutable conditional expressions.

    Guido> Sorry, I still don't get this at all.  A conditional expression
    Guido> is still an expression.  Curly brances are a statement-level
    Guido> concept.  What am I missing?  Where is the similarity between the
    Guido> use of { } and ( ) in C/Java/C++/Perl?

>From the computer's viewpoint, they are syntactically and semantically,
different, I agree.  From the point of view of a human trying to understand
what's written I think they will serve much the same purpose, however.
White space is viewed is being essential enough to understanding that Python
uses it for block structure.  In particular, I can't nest two compound
statements on the same line.  With this new addition to expressions, it
appears you will be able to create arbitrarily complex expressions all on
one line.

Skip



From guido@python.org  Mon Oct 15 21:14:48 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 16:14:48 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 16:04:59 EDT."
 <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>
References: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>
Message-ID: <200110152014.QAA25765@cj20424-a.reston1.va.home.com>

[Paul]
> I know there's been a long time request for this feature,
> supposedly from people coming to Python from C,

or any other language -- it's in Java, Perl, C++, Icon, Haskell,
Lisp...  People have been arguing for years how to do this.

> but what's the real reason for adding it now,
> without even a PEP ?
> Does it really add enough to the language to justify the bloat ?

We'll see about the PEP (I think it's a good idea to motivate the
change).

The language bloat is very small IMO -- much less than augmented
assignment or list comprehensions, actually, and the benefits are
quite substantial, when used judiciously.  Like augmented assignment,
the conditional expression matches how I *think* about certain
situations; it's nice to finally be able to write them down right.
It's especially useful in function arguments, in string formatting
operations, or in lambdas (if you like lambdas :-).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 15 21:18:48 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 16:18:48 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 15:13:48 CDT."
 <15307.17276.738359.631642@beluga.mojam.com>
References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> <15307.14075.322035.39614@beluga.mojam.com> <200110151934.PAA25299@cj20424-a.reston1.va.home.com>
 <15307.17276.738359.631642@beluga.mojam.com>
Message-ID: <200110152018.QAA25787@cj20424-a.reston1.va.home.com>

> I misunderstood about the need for parens.  I had only been watching the
> thread in python-list with one eye.  I thought it was just some python-list
> riff-raff throwing around ideas.  Then I saw that Tim was part of the rabble
> ;-).  I still didn't notice the "I've implemented it but not yet checked it
> in" bit until someone else did and asked about it.  I then went looking for
> a PEP but didn't see one.  I hadn't seen any discussion on python-dev
> either.  (Maybe I missed it because python-list and python-dev mail wind up
> in the same mailbox.)

OK.  I guess you were thinking that the parentheses were required as
for 'if' in C?

>     Guido> If I saw a piece of code that read
> 
>     Guido>     x = if if if x == 1 then y else z then p else q then a else b
> 
>     Guido> I would get a strong urge to commit illegal violence against the
>     Guido> author.  If on the other hand I saw
> 
>     Guido>     x = if (if (if x == 1 then y else z) then p else q) then a else b
> 
>     Guido> I might be willing to sit down and figure out what it meant --
>     Guido> maybe with the help of a parentheses-balancing command in my
>     Guido> editor.
> 
> Python's block structured statement syntax keep you from composing just
> monstrosities at the statement level.  I'm not sure what makes them more
> appealing at the expression level.  How many levels of parens would be
> required before you had a "strong urge to commit illegal violence" against
> someone using a parenthesized conditional expression?

As soon as there is an if inside the condition of the first if.  Even
parentheses don't fix this, really.

> I guess that's what it boils down to for me.  You can easily write
> much more inscrutable code with the new conditional expression than
> you could before.

"You can write Fortran in any language" -- we've been through this
before.  Potential abuse is not enough to prevent it -- likely abuse
is.

> I think that will make Python somewhat less of a CP4E language.  I
> don't think it falls into the same category of the more esoteric
> features of the language (generators, metaclasses, __*__ methods,
> etc) that beginners can safely ignore.  Conditional expressions are
> pretty basic stuff, and this makes it easier to write fairly
> inscrutable conditional expressions.

It also makes it easier to write what you thinkk, IMO.

>     Guido> Sorry, I still don't get this at all.  A conditional
>     Guido> expression is still an expression.  Curly brances are a
>     Guido> statement-level concept.  What am I missing?  Where is
>     Guido> the similarity between the use of { } and ( ) in
>     Guido> C/Java/C++/Perl?
> 
> From the computer's viewpoint, they are syntactically and
> semantically, different, I agree.  From the point of view of a human
> trying to understand what's written I think they will serve much the
> same purpose, however.  White space is viewed is being essential
> enough to understanding that Python uses it for block structure.  In
> particular, I can't nest two compound statements on the same line.
> With this new addition to expressions, it appears you will be able
> to create arbitrarily complex expressions all on one line.

Previous versions of Python in no way impaired your ability to create
arbitrarily complex expressions all on one line.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Mon Oct 15 21:24:05 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 15 Oct 2001 16:24:05 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110151924.PAA25209@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEBGMBAA.tim.one@home.com>

[Guido, to Skip]
> I think you must be misunderstanding the proposal

Skip was responding to the c.l.py discussion, which you probably haven't seen.  I started it there sometime over the weekend, and
"the rules" at that time required parens in all cases (which is just what was implemented at the time).



From paul-python@svensson.org  Mon Oct 15 21:39:30 2001
From: paul-python@svensson.org (Paul Svensson)
Date: Mon, 15 Oct 2001 16:39:30 -0400 (EDT)
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110151934.PAA25299@cj20424-a.reston1.va.home.com>
Message-ID: <Pine.LNX.4.21.0110151608190.2552-100000@familjen.svensson.org>

On Mon, 15 Oct 2001, Guido van Rossum wrote:

>Sorry, I still don't get this at all.  A conditional expression is
>still an expression.  Curly brances are a statement-level concept.
>What am I missing?  Where is the similarity between the use of { } and
>( ) in C/Java/C++/Perl?

Trying to put a finger on why I think this proposed new syntax looks ugly,
I would say that my biggest problem is that it blurs some of the visual
distictions between statements and expressions that I find helpful
for quickly locating the interesting parts of a piece of code.

I assume this has some parallel to list comprehensions that way,
but I fail to see the same blatant conceptual beauty in this proposal,
motivating the unfamiliar form.

	/Paul



From guido@python.org  Mon Oct 15 21:42:57 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 16:42:57 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 16:39:30 EDT."
 <Pine.LNX.4.21.0110151608190.2552-100000@familjen.svensson.org>
References: <Pine.LNX.4.21.0110151608190.2552-100000@familjen.svensson.org>
Message-ID: <200110152042.QAA28987@cj20424-a.reston1.va.home.com>

> Trying to put a finger on why I think this proposed new syntax looks ugly,
> I would say that my biggest problem is that it blurs some of the visual
> distictions between statements and expressions that I find helpful
> for quickly locating the interesting parts of a piece of code.
> 
> I assume this has some parallel to list comprehensions that way,
> but I fail to see the same blatant conceptual beauty in this proposal,
> motivating the unfamiliar form.

And what exactly qualifies you as a language designer?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From skip@pobox.com (Skip Montanaro)  Mon Oct 15 21:50:40 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 15:50:40 -0500
Subject: [Python-Dev] New socketmodule.c warnings
Message-ID: <15307.19488.702413.550932@beluga.mojam.com>

Just came across these new warnings (gcc 2.96, Linux Mandrake):

    .../Modules/socketmodule.c:1625: warning: initialization from incompatible pointer type
    .../Modules/socketmodule.c:1629: warning: initialization from incompatible pointer type
    .../Modules/socketmodule.c:1636: warning: initialization from incompatible pointer type
    .../Modules/socketmodule.c:1639: warning: initialization from incompatible pointer type
    .../Modules/socketmodule.c:1642: warning: initialization from incompatible pointer type
    .../Modules/socketmodule.c:1645: warning: initialization from incompatible pointer type

I see a lot of interface changes in socketmodule between 1.175 and 1.176.
Is this a spurious error or something significant?

Skip


From skip@pobox.com (Skip Montanaro)  Mon Oct 15 21:57:47 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Mon, 15 Oct 2001 15:57:47 -0500
Subject: [Python-Dev] New socketmodule.c warnings
In-Reply-To: <15307.19488.702413.550932@beluga.mojam.com>
References: <15307.19488.702413.550932@beluga.mojam.com>
Message-ID: <15307.19915.436629.47120@beluga.mojam.com>

    Skip> Is this a spurious error or something significant?

Oops...  s/error/warning/

S


From gmcm@hypernet.com  Mon Oct 15 22:24:29 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 15 Oct 2001 17:24:29 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>
References: <200110151924.PAA25209@cj20424-a.reston1.va.home.com>
Message-ID: <3BCB1BCD.7229.9CA08A91@localhost>

Paul Svensson wrote:

> so I guess don't have the understanding of the motivation
> behind the proposal that I could have had.

Because:
 a and b or c
burns you when b evaluates to false, but
(a and [b] or [c])[0]
is a monstrosity. And yes, people do that.

- Gordon


From tim.one@home.com  Mon Oct 15 22:25:43 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 15 Oct 2001 17:25:43 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110151924.PAA25209@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com>

[Guido]
> ...
> Some examples where no parentheses are needed (note that a comma binds
> less tight than a conditional expression -- same as for lambda):
>
>     x = if 1 then 2 else 3, y

Also

    x = if 1 then 2 else 3,

which sets x to the singleton tuple (2,).  This is an old potential
confusion, though, much the same as the current

    x = 2 or 3,

(which also sets x to (2,)).

> ...
> Some examples where parentheses *are* required:
>
>     (if 1 then 2 else 3) + 4

But that's because it's a top-level expression beginning with "if", not
because of the "+", right?  In the patched Python, this works fine:

>>> x = if 0 then 2 else 3 + 4
>>> x
7
>>>

OTOH, these seem odd:

>>> 4 + if 1 then 2 else 3
  File "<stdin>", line 1
    4 + if 1 then 2 else 3
         ^
SyntaxError: invalid syntax

>>> x = 4 + if 1 then 2 else 3
  File "<stdin>", line 1
    x = 4 + if 1 then 2 else 3
             ^
SyntaxError: invalid syntax
>>>

I say "odd" because there's no question about what they mean.

>     a[(if i then 2 else 3) : 4]
>
> In some situations I'm not sure what's right; The un-parenthesized
> form looks weird although it's not neede to avoid ambiguity:
>
>     if (if 1 then 2 else 3): pass
>     print (if 1 then 2 else 3)

OTOH,

    print >> sys.stderr, if 1 then 2 else 3
and
    print 4, if 1 then 2 else 3

don't look particularly weird but also raise SyntaxError.

As a matter of taste, I was most surprised by the

    4 + if 1 then 2 else 3

examples.



From fredrik@pythonware.com  Mon Oct 15 22:47:54 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 15 Oct 2001 23:47:54 +0200
Subject: [Python-Dev] conditional expressions?
References: <15307.12052.100384.784424@beluga.mojam.com>        <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>        <15307.14075.322035.39614@beluga.mojam.com>        <200110151934.PAA25299@cj20424-a.reston1.va.home.com> <15307.17276.738359.631642@beluga.mojam.com>
Message-ID: <00b001c155c3$0ca44380$ced241d5@hagrid>

skip wrote:
> I think that will make Python somewhat less of a CP4E language.

I doubt Python 2.2 will qualify as a CP4E language, even if
conditional expressions are left out.

</F>



From guido@python.org  Mon Oct 15 22:42:39 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 17:42:39 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 17:25:43 EDT."
 <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com>
Message-ID: <200110152142.RAA02911@cj20424-a.reston1.va.home.com>

> [Guido]
> > ...
> > Some examples where no parentheses are needed (note that a comma binds
> > less tight than a conditional expression -- same as for lambda):
> >
> >     x = if 1 then 2 else 3, y

[Tim]
> Also
> 
>     x = if 1 then 2 else 3,
> 
> which sets x to the singleton tuple (2,).  This is an old potential
> confusion, though, much the same as the current
> 
>     x = 2 or 3,
> 
> (which also sets x to (2,)).

These don't bother me any more than

    x = 2,

does.

> > ...
> > Some examples where parentheses *are* required:
> >
> >     (if 1 then 2 else 3) + 4
> 
> But that's because it's a top-level expression beginning with "if", not
> because of the "+", right?  In the patched Python, this works fine:
> 
> >>> x = if 0 then 2 else 3 + 4
> >>> x
> 7
> >>>

But it groups differently: the 'else' part is '3 + 4' here.  Try 'if
1'. :-)

> OTOH, these seem odd:
> 
> >>> 4 + if 1 then 2 else 3
>   File "<stdin>", line 1
>     4 + if 1 then 2 else 3
>          ^
> SyntaxError: invalid syntax

That's too much work for the parser to get right.  And you'd get
anomalies like

    3 * if 1 then 2 else 3 + 4

-- how should this group?

> >>> x = 4 + if 1 then 2 else 3
>   File "<stdin>", line 1
>     x = 4 + if 1 then 2 else 3
>              ^
> SyntaxError: invalid syntax
> >>>
> 
> I say "odd" because there's no question about what they mean.

Only because the examples are so simply.

> >     a[(if i then 2 else 3) : 4]
> >
> > In some situations I'm not sure what's right; The un-parenthesized
> > form looks weird although it's not neede to avoid ambiguity:
> >
> >     if (if 1 then 2 else 3): pass
> >     print (if 1 then 2 else 3)
> 
> OTOH,
> 
>     print >> sys.stderr, if 1 then 2 else 3
> and
>     print 4, if 1 then 2 else 3
> 
> don't look particularly weird but also raise SyntaxError.
> 
> As a matter of taste, I was most surprised by the
> 
>     4 + if 1 then 2 else 3
> 
> examples.

They act the same as lambda in this respect.

Because the conditional *ends* in something that "pulls in" all
following binary operators and their operands until it hits a ")", 
I find it safer not to let it accept an operator at the front.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 15 23:02:31 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 15 Oct 2001 18:02:31 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Mon, 15 Oct 2001 17:42:39 EDT."
 <200110152142.RAA02911@cj20424-a.reston1.va.home.com>
References: <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com>
 <200110152142.RAA02911@cj20424-a.reston1.va.home.com>
Message-ID: <200110152202.SAA04326@cj20424-a.reston1.va.home.com>

> Because the conditional *ends* in something that "pulls in" all
> following binary operators and their operands until it hits a ")", 
> I find it safer not to let it accept an operator at the front.

To explain this perhaps a bit better, the 'else' keyword acts as the
right side of a low-priority binary operator with respect to the
expression that follows.  It would be unpleasant if the 'if' keyword
acted as the left side of a *high*-priority operator to the expression
that preceeds it.  Again, think of what this would mean:

    (3 * if x then y else z + 4)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pedroni@inf.ethz.ch  Mon Oct 15 23:29:54 2001
From: pedroni@inf.ethz.ch (Samuele Pedroni)
Date: Tue, 16 Oct 2001 00:29:54 +0200
Subject: [Python-Dev] conditional expressions?
References: <15307.12052.100384.784424@beluga.mojam.com>        <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com>        <15307.14075.322035.39614@beluga.mojam.com>        <200110151934.PAA25299@cj20424-a.reston1.va.home.com> <15307.17276.738359.631642@beluga.mojam.com> <00b001c155c3$0ca44380$ced241d5@hagrid>
Message-ID: <003601c155c8$eae0b660$8e0bfea9@newmexico>

[Fredrik Lundh]
> 
> I doubt Python 2.2 will qualify as a CP4E language, even if
> conditional expressions are left out.
> 

[sorry but I feel I should say this]
I heartly agree, i.e. I'm fine with that but it is true.

regards.



From tim.one@home.com  Tue Oct 16 00:30:58 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 15 Oct 2001 19:30:58 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110152142.RAA02911@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEECFMBAA.tim.one@home.com>

||| ...
||| Some examples where parentheses *are* required:
|||
|||     (if 1 then 2 else 3) + 4

|| But that's because it's a top-level expression beginning with "if", not
|| because of the "+", right?  In the patched Python, this works fine:
||
|| >>> x = if 0 then 2 else 3 + 4
|| >>> x
|| 7
|| >>>

| But it groups differently: the 'else' part is '3 + 4' here.  Try 'if
| 1'. :-)

Yes, I understand how it groups, and am fine with it.  The question I asked
is the one I asked <wink>:  the paretheses aren't required because of the
"+", but solely because it's a "top-level expression".  This is by way of
contrast with the example that was next to it in the original:

    a[(if i then 2 else 3) : 4]

where, unlike the "+" example, the parens are also required if you stick
that on the RHS of an assignment.

I thought both were *intended* to be examples of:

    you need to add parentheses to disambiguate expressions:
    ...
    - if the conditional expression is to be combined with a unary or
      binary operator.

as "+" and ":" both feel like <wink> binary operators in these examples.

|| OTOH, these seem odd:
||
|| >>> 4 + if 1 then 2 else 3
||   File "<stdin>", line 1
||     4 + if 1 then 2 else 3
||          ^
|| SyntaxError: invalid syntax

> That's too much work for the parser to get right.  And you'd get
> anomalies like
>
>     3 * if 1 then 2 else 3 + 4
>
> -- how should this group?

That's a good point.

> ...
> Because the conditional *ends* in something that "pulls in" all
> following binary operators and their operands until it hits a ")",
> I find it safer not to let it accept an operator at the front.

I agree.  The "precedence pull" of "*" does make it more tempting to misread

    x = 3 * if 1 then 2 else 3 + 4

than to misread

    x = if 1 then 2 else 3 + 4

and I'm personally happier with "excess" parentheses anyway.



From tim.one@home.com  Tue Oct 16 04:22:51 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 15 Oct 2001 23:22:51 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com>

[Paul Svensson]
> ...
> I know there's been a long time request for this feature,
> supposedly from people coming to Python from C,
> but what's the real reason for adding it now,
> without even a PEP ?

The reason for *considering* it right now is a rapidly closing window of
opportunity -- 2.2b1 goes out later this week.  A fellow on c.l.py has been
plugging away at a PEP, Eric Raymond submitted a working patch for C's ?: in
January of 2000, Jim Fulton loved the idea then (but not particularly the ?:
spelling), David Ascher reported that lack of a conditional was a FAQ in his
classes then, etc etc.  Most replies hated the ?: syntax, and Guido
eventually remembered (although I think he forgot it this time around) that
Algol-60 used the obvious if/then/else.  His reply on Mon, 31 Jan 2000
09:36:52 -0500:

    The solution can be the same as what Algol used:  'if' outside
    parentheses is a statement, and inside parentheses is an expression.
    It's a bit of a grammar rearrangement, but totally unambiguous.

    However, the added keyword means it won't be in 1.6.  The lively
    discussion means that Eric's patch will have a hard time getting
    in too...

The only difference between then and now is s/1.6/2.2/.

> Does it really add enough to the language to justify the bloat ?

I expect most people would say it does; by definition I'll adjust my beliefs
to whatever Guido decides <wink>.



From mahler@cyc.com  Tue Oct 16 04:40:47 2001
From: mahler@cyc.com (Daniel Mahler)
Date: Mon, 15 Oct 2001 22:40:47 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com>
References: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>
 <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com>
Message-ID: <15307.44095.944381.833422@mcallister.cyc.com>

Has an if(cond,exp1,exp2) syntax been considered?
It would have the advantage that existing
syntax sensitive editors would handle without change.
I think conditional exprs badly needed,
but the if/then/else syntax seems too verbose for use inside
expressions.

regards
Daniel Mahler

Tim Peters writes:
 > [Paul Svensson]
 > > ...
 > > I know there's been a long time request for this feature,
 > > supposedly from people coming to Python from C,
 > > but what's the real reason for adding it now,
 > > without even a PEP ?
 > 
 > The reason for *considering* it right now is a rapidly closing window of
 > opportunity -- 2.2b1 goes out later this week.  A fellow on c.l.py has been
 > plugging away at a PEP, Eric Raymond submitted a working patch for C's ?: in
 > January of 2000, Jim Fulton loved the idea then (but not particularly the ?:
 > spelling), David Ascher reported that lack of a conditional was a FAQ in his
 > classes then, etc etc.  Most replies hated the ?: syntax, and Guido
 > eventually remembered (although I think he forgot it this time around) that
 > Algol-60 used the obvious if/then/else.  His reply on Mon, 31 Jan 2000
 > 09:36:52 -0500:
 > 
 >     The solution can be the same as what Algol used:  'if' outside
 >     parentheses is a statement, and inside parentheses is an expression.
 >     It's a bit of a grammar rearrangement, but totally unambiguous.
 > 
 >     However, the added keyword means it won't be in 1.6.  The lively
 >     discussion means that Eric's patch will have a hard time getting
 >     in too...
 > 
 > The only difference between then and now is s/1.6/2.2/.
 > 
 > > Does it really add enough to the language to justify the bloat ?
 > 
 > I expect most people would say it does; by definition I'll adjust my beliefs
 > to whatever Guido decides <wink>.
 > 
 > 
 > _______________________________________________
 > Python-Dev mailing list
 > Python-Dev@python.org
 > http://mail.python.org/mailman/listinfo/python-dev



From tim.one@home.com  Tue Oct 16 05:41:50 2001
From: tim.one@home.com (Tim Peters)
Date: Tue, 16 Oct 2001 00:41:50 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <15307.44095.944381.833422@mcallister.cyc.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>

[Daniel Mahler]
> Has an if(cond,exp1,exp2) syntax been considered?

Not seriously, no.  That looks like a function call, and function arguments
are always evaluated in Python, and that works against seeing this as a
short-circuiting construct:

    q = if y then x/y else copysign(INFINITY, x)

> It would have the advantage that existing
> syntax sensitive editors would handle without change.

I don't see any problem there (and am responsible for two context sensitive
Python edit gimmicks, and used to be responsible for a third -- all I expect
to need to do is get "then" at token level colored as keyword).

> I think conditional exprs badly needed,
> but the if/then/else syntax seems too verbose for use inside
> expressions.

Good -- then you won't be tempted to abuse it <0.5 wink>.

best-in-small-doses-ly y'rs  - tim



From greg@cosc.canterbury.ac.nz  Tue Oct 16 06:06:40 2001
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 16 Oct 2001 18:06:40 +1300 (NZDT)
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <15307.12052.100384.784424@beluga.mojam.com>
Message-ID: <200110160506.SAA07491@s454.cosc.canterbury.ac.nz>

Skip Montanaro <skip@pobox.com>:

> It seems downright weird to me that the syntactic baggage necessary to write
> a conditional expression is greater the the baggage necessary to write an if
> statement (new "then" keyword, parens required for disambiguation).

Has the "x if c else y" idea been totally rejected?
It doesn't require a new keyword, or parentheses to
disambiguate from an if statement.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From tim.one@home.com  Tue Oct 16 06:15:05 2001
From: tim.one@home.com (Tim Peters)
Date: Tue, 16 Oct 2001 01:15:05 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110160506.SAA07491@s454.cosc.canterbury.ac.nz>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com>

[Greg Ewing]
> Has the "x if c else y" idea been totally rejected?
> It doesn't require a new keyword, or parentheses to
> disambiguate from an if statement.

I was its only fan within PythonLabs, and I was lukewarm.  The trick is to
view "if c else" as a new short-circuiting infix binary operator (for each
c), and then it makes great-- even elegant --sense.  But it's apparently a
major strain to see it that way.  It had at least one sworn mortal enemy,
and I think Fred Drake was so upset at the idea that he couldn't speak.

bad-sign-in-a-wordsmith<wink>-ly y'rs  - tim



From esr@thyrsus.com  Tue Oct 16 06:54:32 2001
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 16 Oct 2001 01:54:32 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>; from paul-python@svensson.org on Mon, Oct 15, 2001 at 04:04:59PM -0400
References: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>
Message-ID: <20011016015432.E3779@thyrsus.com>

Paul Svensson <paul-python@svensson.org>:
> On Mon, 15 Oct 2001, Guido van Rossum wrote:
> 
> >I'd be happy to discuss this more.  I'm *not* happy with responses
> >like "is this a bad joke?".  I don't understand how this could be an
> >argument in the arsenal of the anti-Python league.
> 
> I'm sorry if it hit a bad nerve,
> but I did skip this whole thread on c.l.py,
> thinking that this could surely not be seriously meant,
> so I guess don't have the understanding of the motivation
> behind the proposal that I could have had.
> 
> I know there's been a long time request for this feature,
> supposedly from people coming to Python from C,
> but what's the real reason for adding it now,
> without even a PEP ?
> Does it really add enough to the language to justify the bloat ?

I actually implemented this last year.

The less cumbersome syntax I suggest is:

a ? b ! c
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Everything you know is wrong.  But some of it is a useful first approximation.


From pf@artcom-gmbh.de  Tue Oct 16 08:06:41 2001
From: pf@artcom-gmbh.de (Peter Funk)
Date: Tue, 16 Oct 2001 09:06:41 +0200 (CEST)
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com> from Tim Peters
 at "Oct 15, 2001 11:22:51 pm"
Message-ID: <m15tOJF-002XjiC@artcom0.artcom-gmbh.de>

Hi,

I'm -1 on the concept of conditional expressions.  And I don't give
a damn whether they are spelled x?y:z or (if x then y else z).  

Let me explain why:

IMO the motivation of people asking for conditional expressions is,
that they want to write down complex constructs without having to
name these properly.  Otherwise it is easy to put the code into
a function or method and call this instead.  In most cases this will
improve the readability of the code considerable, since people
are alreaday trained to choose proper identifiers instead of X, Y, Z.

Conditional expressions will be abused as an obfuscation device in 
the same way as lambdas have been used before.  They will make it
far too easy to write obfuscated (perlish?) Python code even inadvertent.

Regards, Peter
-- 
Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260
office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen, Germany)



From mahler@cyc.com  Tue Oct 16 08:51:32 2001
From: mahler@cyc.com (Daniel Mahler)
Date: Tue, 16 Oct 2001 02:51:32 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>
References: <15307.44095.944381.833422@mcallister.cyc.com>
 <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>
Message-ID: <15307.59140.626077.69525@mcallister.cyc.com>

Tim Peters writes:
 > [Daniel Mahler]
 > > Has an if(cond,exp1,exp2) syntax been considered?
 > 
 > Not seriously, no.  That looks like a function call, and function arguments
 > are always evaluated in Python, and that works against seeing this as a
 > short-circuiting construct:
 > 
 >     q = if y then x/y else copysign(INFINITY, x)

I do not see a problem:
almost all of the infix operators also use eager evaluation,
but 2 use lazy evaluation, so operator syntax by itself
does not imply or even suggest nonstandard evaluation order
and there is no systematic syntactic convention to indicate it.
You just have to remember that "and" and "or" are the only
lazy operators (you might count lambda)
out of ~35 (by my count).
-- anyway, I had to check my reference card to see they were the only ones.
I do not see why this is any more problematic for functional syntax.
Since there is no generic way of indicating lazyness
at the expression level (":" kind of indicates it at the statement level),
people will just have to remember that conditionals are special
no matter which syntax is chosen.

OTOH conditionals are like functions in the syntactic sense
(they take expressions as arguments and can be used anywhere
expressions can)
and in the mathematical sense
(in the absence of side effects in the argument expressions
the result is a mathematical function of the arguments).
In lazy languages (Haskell,Miranda,Gofer)
"if" is just a regular function.

Hmmm, now that I think about it,
I wonder why python makes a strong distinction between
statements and expressions, but does not distinguish
between procedures and functions?

 > > It would have the advantage that existing
 > > syntax sensitive editors would handle without change.
 > 
 > I don't see any problem there (and am responsible for two context sensitive
 > Python edit gimmicks, and used to be responsible for a third -- all I expect
 > to need to do is get "then" at token level colored as keyword).

My editor does more than highlighting.
What about autoindentation if you split the line?
Will one need a trailing "\" if when splitting across lines?
-- with the if/then/else syntax,
we will be splitting lines in expressions.
[and deep down inside python does not really like that ;)].
I actually prefer to split even concise syntax,
for readability, and python is friendlier to splitting functional
expressions than to splitting operator expressions.

Functional syntax would do the "right thing" automatically,
even when nested,
because it does not involve changing the syntax of the language.
It just introduces lazy/call-by-name semantics.
[Guido, can we have user defineable call-by-name functions -- pleeeease ;)]

Actually, I lied little -- nested if(_,_,_) confuses emacs python-mode,
but change the name to, say, ifTE (short for ifThenElse)
and it autoindents beautyfully

x = ifTE(a,
         ifTE(b,
              x,
              y),
         ifTE(c,
              w,
              z)
         )

 > > I think conditional exprs badly needed,
 > > but the if/then/else syntax seems too verbose for use inside
 > > expressions.
 > 
 > Good -- then you won't be tempted to abuse it <0.5 wink>.

But one should be tempted to *use* them,
otherwise one can just drop the feature,
preventing abuse entirely.
Note that 
"z = if a then x else y"
is 2/3 syntactic sugar.
I can see that
"z = a ? x : y"
is somewhat cryptic but no more than
"x >> y", "x % y" or "x ^ y"
("^" is exponentiation in most other languages).

Of course, "z = if(a,x,y)" is just ... ;).

Daniel

 > best-in-small-doses-ly y'rs  - tim
 > 
 > 
 > _______________________________________________
 > Python-Dev mailing list
 > Python-Dev@python.org
 > http://mail.python.org/mailman/listinfo/python-dev



From paul@ActiveState.com  Tue Oct 16 11:13:35 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 16 Oct 2001 03:13:35 -0700
Subject: [Python-Dev] conditional expressions?
References: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com>
Message-ID: <3BCC084F.FCBB2CF0@ActiveState.com>

Tim Peters wrote:
> 
> ...
> 
> I was its only fan within PythonLabs, and I was lukewarm.  The trick is to
> view "if c else" as a new short-circuiting infix binary operator (for each
> c), and then it makes great-- even elegant --sense.  But it's apparently a
> major strain to see it that way.  It had at least one sworn mortal enemy,
> and I think Fred Drake was so upset at the idea that he couldn't speak.

It actually read more easily to me the first time. As I said on
python-list:

unit is meters if metric, else feet

versus

unit is, if metric, meters, else feet

At first I didn't like that it split the two options but I think that
that actually helps you to keep clear in your head what the "default" is
and what the "alternative" is.

 Paul Prescod


From gward@python.net  Tue Oct 16 14:00:58 2001
From: gward@python.net (Greg Ward)
Date: Tue, 16 Oct 2001 09:00:58 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <3BCC084F.FCBB2CF0@ActiveState.com>
References: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com> <3BCC084F.FCBB2CF0@ActiveState.com>
Message-ID: <20011016090058.A2546@cthulhu.mems-exchange.org>

On 16 October 2001, Paul Prescod said:
> It actually read more easily to me the first time. As I said on
> python-list:
> 
> unit is meters if metric, else feet
> 
> versus
> 
> unit is, if metric, meters, else feet

That's confusing Python with English.  They're two different languages,
with different syntaxes, different semantics, and very different
purposes.  I wonder if a native speaker of German or Cantonese would
agree with you?

I really don't see what the big fuss is about.  Algol got it right 40
years ago with "if x then y else z", and (nearly?) every language since
then has either made conditional expressions painfully obfuscated (Lisp,
C and its ilk) or forgotten them entirely (Python).  They're very
useful, and just as easily abused as other powerful constructs like list
comprehensions, regular expressions, or lambdas.  But just because
something can be abused doesn't mean it should be banned.  Down that
road lies Nanny Python.

        Greg
-- 
Greg Ward - geek                                        gward@python.net
http://starship.python.net/~gward/
Vote Cthulhu -- why settle for a lesser evil?


From guido@python.org  Tue Oct 16 14:17:37 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 16 Oct 2001 09:17:37 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Tue, 16 Oct 2001 01:15:05 EDT."
 <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com>
Message-ID: <200110161317.JAA07196@cj20424-a.reston1.va.home.com>

> [Greg Ewing]
> > Has the "x if c else y" idea been totally rejected?
> > It doesn't require a new keyword, or parentheses to
> > disambiguate from an if statement.
> 
> I was its only fan within PythonLabs, and I was lukewarm.  The trick is to
> view "if c else" as a new short-circuiting infix binary operator (for each
> c), and then it makes great-- even elegant --sense.  But it's apparently a
> major strain to see it that way.  It had at least one sworn mortal enemy,
> and I think Fred Drake was so upset at the idea that he couldn't speak.

Also I believe it would create a parsing problem in list
comprehensions; compare these two:

    [x for x in range(10) if prime(x)]  # 'if' is part of list compreh.
    [x for x in range(10) if small else range(100)]  # 'if-then' construct

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Tue Oct 16 14:24:40 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 16 Oct 2001 09:24:40 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Tue, 16 Oct 2001 02:51:32 CDT."
 <15307.59140.626077.69525@mcallister.cyc.com>
References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>
 <15307.59140.626077.69525@mcallister.cyc.com>
Message-ID: <200110161324.JAA07249@cj20424-a.reston1.va.home.com>

Even after reading Daniel's passionate plea, if(cond,expr,expr) looks
ugly to me.  It also doesn't solve the parsing ambiguity problem -- it
still can't be allowed in an expression that starts a line because the
parser doesn't know whether it's an if expression or an if statement
without many tokens of lookahead.

So, -1 for if(cond,expr,expr).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From walter@livinglogic.de  Tue Oct 16 14:46:05 2001
From: walter@livinglogic.de (Walter =?ISO-8859-1?Q?D=F6rwald?=)
Date: Tue, 16 Oct 2001 15:46:05 +0200
Subject: [Python-Dev] conditional expressions?
References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>              <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com>
Message-ID: <3BCC3A1D.8000308@livinglogic.de>

Guido van Rossum wrote:

> Even after reading Daniel's passionate plea, if(cond,expr,expr) looks
> ugly to me.  It also doesn't solve the parsing ambiguity problem -- it
> still can't be allowed in an expression that starts a line because the
> parser doesn't know whether it's an if expression or an if statement
> without many tokens of lookahead.


This can be solved by simply choosing a different name:
     iif(cond, expr, expr)
     when(cond, expr, expr)
     cond(cond, expr, expr)
     check(cond, expr, expr)
     ...

> So, -1 for if(cond,expr,expr).


Bye,
    Walter D=F6rwald



From guido@python.org  Tue Oct 16 15:20:59 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 16 Oct 2001 10:20:59 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Tue, 16 Oct 2001 15:46:05 +0200."
 <3BCC3A1D.8000308@livinglogic.de>
References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com>
 <3BCC3A1D.8000308@livinglogic.de>
Message-ID: <200110161420.KAA07533@cj20424-a.reston1.va.home.com>

> This can be solved by simply choosing a different name:
>      iif(cond, expr, expr)
>      when(cond, expr, expr)
>      cond(cond, expr, expr)
>      check(cond, expr, expr)
>      ...

-1 too.

(1) In order to do short-circuiting evaluation, the name would have to
    be recognized by the parser, which means it would have to be a new
    keyword.  None of these keywords sound attractive.

(2) It looks too much like a function call for my comfort.

I still like

    (if cond then expr else expr)

better than any of these.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From paul@pfdubois.com  Tue Oct 16 16:49:13 2001
From: paul@pfdubois.com (Paul F. Dubois)
Date: Tue, 16 Oct 2001 08:49:13 -0700
Subject: [Python-Dev] Conditional expressions and sequences
Message-ID: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com>

One of the most cherished of Numpy operations is:

x = where (condition, t, f)

which creates an array x, shaped like condition. Each element of this
result is set to the corresponding element of t or f depending on
whether the corresponding element of condition is true or false. Either
or both of t and f can be a scalar but otherwise must be the same shape
as condition.

The reason this is cherished is, of course, that it avoids a scalar loop
with an if-test in it. Scalar loops make Nummies cringe.

Now in your new syntax this is essentially

x = if condition then t else f

Therefore it would be cool if the implementation was
x=condition.__where__(t,f). This slot could be overwritten by classes
like Numeric's array. If condition was a list this might be useful too. 

Never mind how I spelled __where__, please. The naming would require
thought.

Ducking-down-behind-the-barricade-'ly yours,

Paul

P.S. I once learned a language where you could say
x = y if c
and it was yummy.




From skip@pobox.com (Skip Montanaro)  Tue Oct 16 17:04:51 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Tue, 16 Oct 2001 11:04:51 -0500
Subject: [Python-Dev] Conditional expressions and sequences
In-Reply-To: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com>
References: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com>
Message-ID: <15308.23203.242343.613940@beluga.mojam.com>

    Paul> Now in your new syntax this is essentially

    Paul> x = if condition then t else f

    Paul> Therefore it would be cool if the implementation was
    Paul> x=condition.__where__(t,f). This slot could be overwritten by
    Paul> classes like Numeric's array. If condition was a list this might
    Paul> be useful too.

Unfortunately, that requires both t and f to be evaluated.  In a NumPy
setting that's probably not a big deal because in most cases both values
will be used in the resulting output.  In a scalar context, the
short-circuit nature of the expression is often important.  If implemented
as any kind of function you've lost that capability.  

-- 
Skip Montanaro (skip@pobox.com)
http://www.mojam.com/
http://www.musi-cal.com/


From mahler@cyc.com  Tue Oct 16 17:11:06 2001
From: mahler@cyc.com (Daniel Mahler)
Date: Tue, 16 Oct 2001 11:11:06 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
References: <15307.44095.944381.833422@mcallister.cyc.com>
 <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>
 <15307.59140.626077.69525@mcallister.cyc.com>
 <200110161324.JAA07249@cj20424-a.reston1.va.home.com>
 <3BCC3A1D.8000308@livinglogic.de>
 <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
Message-ID: <15308.23578.120938.316757@mcallister.cyc.com>

Guido van Rossum writes:

 > (1) In order to do short-circuiting evaluation, the name would have to
 >     be recognized by the parser, which means it would have to be a new
 >     keyword. 

That surprises me.
In a conventional compiler architecture,
I would expect this to be a static analysis and code generation issue.
In languages that have multiple parameter passing disciplines
(eg Pascal, Simula), 
the parser does not track how any given argument is being passed.

 > (2) It looks too much like a function call for my comfort.

I am trying to make the case that it *is* a function call,
with args 2 & 3 being passed by-name.
Or at least it can very sensibly be viewed and implemented that way.

 > 
 > I still like
 > 
 >     (if cond then expr else expr)
 > 
 > better than any of these.

:(

 > --Guido van Rossum (home page: http://www.python.org/~guido/)
 > 
 > _______________________________________________
 > Python-Dev mailing list
 > Python-Dev@python.org
 > http://mail.python.org/mailman/listinfo/python-dev

Daniel



From guido@python.org  Tue Oct 16 17:24:43 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 16 Oct 2001 12:24:43 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Tue, 16 Oct 2001 11:11:06 CDT."
 <15308.23578.120938.316757@mcallister.cyc.com>
References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
 <15308.23578.120938.316757@mcallister.cyc.com>
Message-ID: <200110161624.f9GGOhI19035@odiug.digicool.com>

>  > (1) In order to do short-circuiting evaluation, the name would have to
>  >     be recognized by the parser, which means it would have to be a new
>  >     keyword. 
> 
> That surprises me.
> In a conventional compiler architecture,
> I would expect this to be a static analysis and code generation issue.
> In languages that have multiple parameter passing disciplines
> (eg Pascal, Simula), 
> the parser does not track how any given argument is being passed.

This just shows how little you know about how Python is implemented. ;-(

>  > (2) It looks too much like a function call for my comfort.
> 
> I am trying to make the case that it *is* a function call,
> with args 2 & 3 being passed by-name.
> Or at least it can very sensibly be viewed and implemented that way.

Python has no by-name argument passing mechanisms, and adding one just
for this purpose strikes me as the tail wagging the dog.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From hamish_lawson@yahoo.co.uk  Tue Oct 16 17:59:29 2001
From: hamish_lawson@yahoo.co.uk (=?iso-8859-1?q?Hamish=20Lawson?=)
Date: Tue, 16 Oct 2001 17:59:29 +0100 (BST)
Subject: [Python-Dev] What happened to Guido's plans for (S1; S2; ...; Sn; E) ?
Message-ID: <20011016165929.54794.qmail@web11001.mail.yahoo.com>

The current discussion on conditional expressions has at times touched
on the ever-recurring issue of assignment in expressions. It reminded
me that a proposal by Kevin Digweed back in early 1999 had actually
managed to obtain Guido's approval as a reasonably Pythonic approach to
allowing assignment in expressions, and the proposal seemed intended to
make its eventual way into Python. Is this still the case?

Kevin's original proposal:

| What about allowing an optional "assignment only" suite
| between "while" and the test condition. This suite will be executed 
| before testing the condition each time
|
<http://groups.google.com/groups?hl=en&selm=78naok%242ed%241%40nnrp1.dejanews.com>

Guido then expanded on the proposal and seemed to give it his approval:

| Why stop there?  It could easily be any number of "small" statements
| separated by semicolons with the restriction that the final one must
| be an expression and that the ones before it must not be flow-control
| statements (return, raise, continue, break).  Other useful statements
| to allow are print (for debugging) and assert, and for all I care
| import, del, pass, and exec.
|
| Note that I like this for 2.0, not so much for for 1.6 (if only
| because I don't want 1.6 to make existing books obsolete).
|
<http://groups.google.com/groups?hl=en&selm=199902021501.KAA07333%40eric.cnri.reston.va.us>

Guido also clarified its intended syntax and semantics:

| In my interpretation, the expression (S1; S2; ...; Sn; E) yields the
| value of expression E -- S1-Sn are only there for their side effects.
|
<http://groups.google.com/groups?hl=en&selm=199902031343.IAA09480%40eric.cnri.reston.va.us>


Hamish Lawson


____________________________________________________________
Do You Yahoo!?
Get your free @yahoo.co.uk address at http://mail.yahoo.co.uk
or your free @yahoo.ie address at http://mail.yahoo.ie


From mahler@cyc.com  Tue Oct 16 18:02:00 2001
From: mahler@cyc.com (Daniel Mahler)
Date: Tue, 16 Oct 2001 12:02:00 -0500
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110161624.f9GGOhI19035@odiug.digicool.com>
References: <15307.44095.944381.833422@mcallister.cyc.com>
 <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>
 <15307.59140.626077.69525@mcallister.cyc.com>
 <200110161324.JAA07249@cj20424-a.reston1.va.home.com>
 <3BCC3A1D.8000308@livinglogic.de>
 <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
 <15308.23578.120938.316757@mcallister.cyc.com>
 <200110161624.f9GGOhI19035@odiug.digicool.com>
Message-ID: <15308.26632.664809.870977@mcallister.cyc.com>

Guido van Rossum writes:
 > >  > (1) In order to do short-circuiting evaluation, the name would have to
 > >  >     be recognized by the parser, which means it would have to be a new
 > >  >     keyword. 
 > > 
 > > That surprises me.
 > > In a conventional compiler architecture,
 > > I would expect this to be a static analysis and code generation issue.
 > > In languages that have multiple parameter passing disciplines
 > > (eg Pascal, Simula), 
 > > the parser does not track how any given argument is being passed.
 > 
 > This just shows how little you know about how Python is implemented. ;-(

True, but I am learning.

 > 
 > >  > (2) It looks too much like a function call for my comfort.
 > > 
 > > I am trying to make the case that it *is* a function call,
 > > with args 2 & 3 being passed by-name.
 > > Or at least it can very sensibly be viewed and implemented that way.
 > 
 > Python has no by-name argument passing mechanisms, and adding one just
 > for this purpose strikes me as the tail wagging the dog.

But it need not be the only one ;)

 > --Guido van Rossum (home page: http://www.python.org/~guido/)

regards
Daniel



From paul@ActiveState.com  Tue Oct 16 20:12:40 2001
From: paul@ActiveState.com (Paul Prescod)
Date: Tue, 16 Oct 2001 12:12:40 -0700
Subject: [Python-Dev] Conditional expressions and sequences
References: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com> <15308.23203.242343.613940@beluga.mojam.com>
Message-ID: <3BCC86A8.1CD4B229@ActiveState.com>

Skip Montanaro wrote:
> 
>...
> 
> Unfortunately, that requires both t and f to be evaluated.  

Perhaps not. What if the __where__ function was passed two callable
objects representing closures?

if x then y else z -> x.__where__(lambda: y, lambda: z)

I'm not saying I'm necessarily in favor of __where__. It is a kind of
innovative and I haven't thought through all of the issues.

 Paul Prescod


From greg@cosc.canterbury.ac.nz  Wed Oct 17 01:51:39 2001
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Wed, 17 Oct 2001 13:51:39 +1300 (NZDT)
Subject: [Python-Dev] conditional expressions?
In-Reply-To: <200110161317.JAA07196@cj20424-a.reston1.va.home.com>
Message-ID: <200110170051.NAA07586@s454.cosc.canterbury.ac.nz>

Guido:

> Someone else:
>
> > I was its only fan within PythonLabs, and I was lukewarm.  The trick is to
> > view "if c else" as a new short-circuiting infix binary operator (for each
> > c), and then it makes great-- even elegant --sense.

I think it makes sense in its own right -- no need for
"tricks".

> Also I believe it would create a parsing problem in list
> comprehensions;
>
> [x for x in range(10) if small else range(100)]

Parentheses would need to be required in that case:

  [x for x in (range(10) if small else range(100))]

I don't see that as a big problem -- wanting to put a conditional 
expression in that position is going to be fairly rare.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From neal@metaslash.com  Wed Oct 17 01:57:17 2001
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 16 Oct 2001 20:57:17 -0400
Subject: [Python-Dev] more pychecker warnings from python-current
Message-ID: <3BCCD76D.D1645A1D@metaslash.com>

Well, almost python-current.  The warnings were from a few days ago.

Neal
--

cgi.py:880: Local variable (traceback) not used

copy_reg.py:47: self is argument in function
	not sure if this is really a problem or intended

copy_reg.py:65: Function return types are inconsistent
	line 63 returns tuple of 3, but line 65, returns tuple of 2

doctest.py:528: Parameter (prefix) not used
	docstring says prefix is used, but it isn't

locale.py:358: Redefining attribute (setlocale) original line (68)

markupbase.py:31: No class attribute (rawdata) found
markupbase.py:49: Local variable (sys) not used
markupbase.py:71: No class attribute (unknown_decl) found
markupbase.py:86: No class attribute (error) found
markupbase.py:118: No class attribute (parse_comment) found
markupbase.py:166: Local variable (n) not used

mhlib.py:651: Local variable (messages) not used
	looks like a return in the if clause is necessary, otherwise exception

pickle.py:607: No class attribute (persistent_load) found
pickle.py:616: No class attribute (persistent_load) found

profile.py:356: Local variable (a) not used
	on next line, return is not necessary, should a be returned?
profile.py:368: Local variable (a) not used

xmlrpclib.py:1: Imported module (sys) not used


From guido@python.org  Wed Oct 17 06:57:26 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 17 Oct 2001 01:57:26 -0400
Subject: [Python-Dev] more pychecker warnings from python-current
In-Reply-To: Your message of "Tue, 16 Oct 2001 20:57:17 EDT."
 <3BCCD76D.D1645A1D@metaslash.com>
References: <3BCCD76D.D1645A1D@metaslash.com>
Message-ID: <200110170557.BAA09026@cj20424-a.reston1.va.home.com>

> cgi.py:880: Local variable (traceback) not used

I believe this was to avoid a nasty problem when the 'import
traceback' in print_exception() actually has to load the traceback
module (as opposed to finding it already loaded in sys.modules); this
could lose the pending traceback.  I believe that problem is solved now
so this can safely be removed.

> copy_reg.py:47: self is argument in function
> 	not sure if this is really a problem or intended

This is intended -- _reduce() is called by the C method
object.__reduce__ to do the heavy lifing that I didn't want to code in
C (yet).

> copy_reg.py:65: Function return types are inconsistent
> 	line 63 returns tuple of 3, but line 65, returns tuple of 2

This is unfortunately the spec of the __reduce__ function -- it can
return a 2-tuple or a 3-tuple.

> doctest.py:528: Parameter (prefix) not used
> 	docstring says prefix is used, but it isn't

Tim?

> locale.py:358: Redefining attribute (setlocale) original line (68)

This is intentional.  It confused me too.  Note that the previous
locale is stored in _setlocale -- there's a comment about that there.
Still, it's all very confusing.  Martin?

> markupbase.py:31: No class attribute (rawdata) found
> markupbase.py:49: Local variable (sys) not used
> markupbase.py:71: No class attribute (unknown_decl) found
> markupbase.py:86: No class attribute (error) found
> markupbase.py:118: No class attribute (parse_comment) found
> markupbase.py:166: Local variable (n) not used

This is an abstract base class -- these are references to instance
variables of a derived class.  Except for the two "local variable not
used" warnings -- these are relics from the code refactoring that
created the abstract class.  Looks like a job for Fred.

> mhlib.py:651: Local variable (messages) not used
> 	looks like a return in the if clause is necessary, otherwise exception

No, this is done for the side effect of setting self.last (mentioned
in the comment in listmessages()).  I've fixed this by getting rid of
the "messages =" part.

> pickle.py:607: No class attribute (persistent_load) found
> pickle.py:616: No class attribute (persistent_load) found

AFAIK this is intentional; a derived class would have to define
persistent_load() for this feature to work.

> profile.py:356: Local variable (a) not used
> 	on next line, return is not necessary, should a be returned?
> profile.py:368: Local variable (a) not used

Tim?

> xmlrpclib.py:1: Imported module (sys) not used

Fred already fixed this one.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Wed Oct 17 14:26:43 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Wed, 17 Oct 2001 09:26:43 -0400
Subject: [Python-Dev] more pychecker warnings from python-current
In-Reply-To: <200110170557.BAA09026@cj20424-a.reston1.va.home.com>
References: <3BCCD76D.D1645A1D@metaslash.com>
 <200110170557.BAA09026@cj20424-a.reston1.va.home.com>
Message-ID: <15309.34579.733687.799625@grendel.zope.com>

Guido van Rossum writes:
 > > markupbase.py:31: No class attribute (rawdata) found
 > > markupbase.py:49: Local variable (sys) not used
 > > markupbase.py:71: No class attribute (unknown_decl) found
 > > markupbase.py:86: No class attribute (error) found
 > > markupbase.py:118: No class attribute (parse_comment) found
 > > markupbase.py:166: Local variable (n) not used
 > 
 > This is an abstract base class -- these are references to instance
 > variables of a derived class.  Except for the two "local variable not
 > used" warnings -- these are relics from the code refactoring that
 > created the abstract class.  Looks like a job for Fred.

  I expect to check something in for this tonight.

 > > profile.py:356: Local variable (a) not used
 > > 	on next line, return is not necessary, should a be returned?
 > > profile.py:368: Local variable (a) not used
 > 
 > Tim?

  Sorry; I nailed this one already.  ;-)


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From neal@metaslash.com  Wed Oct 17 15:04:36 2001
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 17 Oct 2001 10:04:36 -0400
Subject: [Python-Dev] bug or feature in imp.load_module()?
Message-ID: <3BCD8FF4.96457AD8@metaslash.com>

module = imp.load_module(path, file, filename, smt)

After returning from imp.load_module(), the file paramter passed in may 
be at the beginning or end depending on whether it was byte compiled.
If the file was already compiled, the file returned will stay at the beginning.
If the file was byte compiled in load_module(), the file will be at the end.

The following program may demonstrate better:

#####################

import imp, sys, os

FILE_BASE = 'x'
PY_FILE = FILE_BASE + '.py'

def load(f):
    file, filename, smt = imp.find_module(f, sys.path)
    imp.load_module('', file, filename, smt)
    print 'File is at byte:', file.tell()

def main():
    try:
        os.unlink(PY_FILE + 'c')
    except:
        pass
    if not os.path.exists(PY_FILE):
        file = open(PY_FILE, 'w')
        file.close()
    load(FILE_BASE)
    load(FILE_BASE)

if __name__ == '__main__':
    main()

#####################

file sometimes being modified was unexpected.  And I couldn't
find the behaviour documented anywhere.

This behaviour goes back to Python 1.5 at least AFAIR.

Neal


From Greg.Wilson@baltimore.com  Wed Oct 17 15:46:45 2001
From: Greg.Wilson@baltimore.com (Greg Wilson)
Date: Wed, 17 Oct 2001 10:46:45 -0400
Subject: [Python-Dev] re: conditional expressions
Message-ID: <930BBCA4CEBBD411BE6500508BB3328F4322C4@nsamcanms1.ca.baltimore.com>

I just showed the proposed conditional expression syntax
to two colleagues, and asked them how they would write
the "sign" function (-1 for negative, 0 for zero, +1 for
positive) using it.  Both of them wrote:

    s = if x < 0 then -1 elif x == 0 then 0 else 1

(Well, OK, one of them tested negative, then positive, then
returned zero as the 'else', but that's a quibble.)  I think
most users will expect 'elif' to work in expressions, just
as it does in statements; is this already part of the new 
syntax?

Thanks,
Greg


-----------------------------------------------------------------------------------------------------------------
The information contained in this message is confidential and is intended 
for the addressee(s) only.  If you have received this message in error or 
there are any problems please notify the originator immediately.  The 
unauthorized use, disclosure, copying or alteration of this message is 
strictly forbidden. Baltimore Technologies plc will not be liable for direct, 
special, indirect or consequential damages arising from alteration of the 
contents of this message by a third party or as a result of any virus being 
passed on.

In addition, certain Marketing collateral may be added from time to time to 
promote Baltimore Technologies products, services, Global e-Security or 
appearance at trade shows and conferences.
 
This footnote confirms that this email message has been swept by 
Baltimore MIMEsweeper for Content Security threats, including
computer viruses.


From fredrik@pythonware.com  Wed Oct 17 16:03:04 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 17 Oct 2001 17:03:04 +0200
Subject: [Python-Dev] conditional expressions?
References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com>              <3BCC3A1D.8000308@livinglogic.de>  <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
Message-ID: <007f01c1571c$d43c7fc0$ced241d5@hagrid>

guido wrote:

> I still like
> 
>     (if cond then expr else expr)
> 
> better than any of these.

I guess I've lost track of all the proposals here; what's
the current main proposal?

    (if cond then expr else expr)

with mandatory parentheses?

not too bad -- but I doubt it will save me much time in
real life...  (probably a net loss: even if I could use this
from time to time, it's not like I have to use much brain-
power to spell it out in today's python...)

</F>



From barry@zope.com  Wed Oct 17 18:27:59 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Wed, 17 Oct 2001 13:27:59 -0400
Subject: [Python-Dev] conditional expressions?
References: <15307.44095.944381.833422@mcallister.cyc.com>
 <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com>
 <15307.59140.626077.69525@mcallister.cyc.com>
 <200110161324.JAA07249@cj20424-a.reston1.va.home.com>
 <3BCC3A1D.8000308@livinglogic.de>
 <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
 <007f01c1571c$d43c7fc0$ced241d5@hagrid>
Message-ID: <15309.49055.970876.74880@anthem.wooz.org>

>>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:

    FL> I guess I've lost track of all the proposals here; what's
    FL> the current main proposal?

Another good reason why there should be a PEP first.  We talked about
it yesterday and decided against trying to get anything like this into
Py2.2.  PEPs for 2.3 would be welcome. ;)

-Barry



From guido@python.org  Wed Oct 17 18:27:04 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 17 Oct 2001 13:27:04 -0400
Subject: [Python-Dev] conditional expressions?
In-Reply-To: Your message of "Wed, 17 Oct 2001 17:03:04 +0200."
 <007f01c1571c$d43c7fc0$ced241d5@hagrid>
References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com>
 <007f01c1571c$d43c7fc0$ced241d5@hagrid>
Message-ID: <200110171727.NAA16770@cj20424-a.reston1.va.home.com>

> I guess I've lost track of all the proposals here; what's
> the current main proposal?
> 
>     (if cond then expr else expr)
> 
> with mandatory parentheses?

Almost -- the parentheses can be left out in some cases, e.g.

    x = if 1 then 2 else 3

I agree with Greg Wilson that elif should be added too.

> not too bad -- but I doubt it will save me much time in
> real life...  (probably a net loss: even if I could use this
> from time to time, it's not like I have to use much brain-
> power to spell it out in today's python...)

That's my conclusion too.  I went looking through the stdandard
library for places that could use it and basically didn't find any in
the 15 minutes before I got bored.

So the patch is still there but this won't make it into 2.2.

--Guido van Rossum (home page: http://www.python.org/~guido/)



From guido@python.org  Wed Oct 17 18:32:00 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 17 Oct 2001 13:32:00 -0400
Subject: [Python-Dev] bug or feature in imp.load_module()?
In-Reply-To: Your message of "Wed, 17 Oct 2001 10:04:36 EDT."
 <3BCD8FF4.96457AD8@metaslash.com>
References: <3BCD8FF4.96457AD8@metaslash.com>
Message-ID: <200110171732.NAA16813@cj20424-a.reston1.va.home.com>

> module = imp.load_module(path, file, filename, smt)
> 
> After returning from imp.load_module(), the file paramter passed in
> may be at the beginning or end depending on whether it was byte
> compiled.  If the file was already compiled, the file returned will
> stay at the beginning.  If the file was byte compiled in
> load_module(), the file will be at the end.

This certainly doesn't surprise me -- in one case it read the source
and in the other case it didn't read the source.

If you pass a file to a function, I don't think that there is any
guarantee to the file position when it returns, so I don't think this
should be documented.  I can't think of why you'd expect it to be at
any specific position.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From neal@metaslash.com  Wed Oct 17 19:22:43 2001
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 17 Oct 2001 14:22:43 -0400
Subject: [Python-Dev] bug or feature in imp.load_module()?
References: <3BCD8FF4.96457AD8@metaslash.com> <200110171732.NAA16813@cj20424-a.reston1.va.home.com>
Message-ID: <3BCDCC73.A516C31E@metaslash.com>

Guido van Rossum wrote:
> 
> > module = imp.load_module(path, file, filename, smt)
> >
> > After returning from imp.load_module(), the file paramter passed in
> > may be at the beginning or end depending on whether it was byte
> > compiled.  If the file was already compiled, the file returned will
> > stay at the beginning.  If the file was byte compiled in
> > load_module(), the file will be at the end.
> 
> This certainly doesn't surprise me -- in one case it read the source
> and in the other case it didn't read the source.
> 
> If you pass a file to a function, I don't think that there is any
> guarantee to the file position when it returns, so I don't think this
> should be documented.  I can't think of why you'd expect it to be at
> any specific position.

I'm not sure either. :-)  It was odd for me to pass both file & filename.
The doc says that:

	The file argument is an open file, and filename is 
	the corresponding file name; these can be None and '', 
	respectively, when the module is not being loaded from a file.

I'm not sure how the module could not be loaded from a file (unless
the implication is that the file argument is None and a new file will
be opened from the filename).

If the file can be None anytime, it makes more sense to me.
But the last phrase in the doc seems to indicate the file argument
should not be None.

Perhaps, it's just me, it wouldn't be the first time. :-)

Neal


From guido@python.org  Wed Oct 17 20:05:35 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 17 Oct 2001 15:05:35 -0400
Subject: [Python-Dev] bug or feature in imp.load_module()?
In-Reply-To: Your message of "Wed, 17 Oct 2001 14:22:43 EDT."
 <3BCDCC73.A516C31E@metaslash.com>
References: <3BCD8FF4.96457AD8@metaslash.com> <200110171732.NAA16813@cj20424-a.reston1.va.home.com>
 <3BCDCC73.A516C31E@metaslash.com>
Message-ID: <200110171905.PAA17035@cj20424-a.reston1.va.home.com>

> > If you pass a file to a function, I don't think that there is any
> > guarantee to the file position when it returns, so I don't think this
> > should be documented.  I can't think of why you'd expect it to be at
> > any specific position.
> 
> I'm not sure either. :-)  It was odd for me to pass both file & filename.

Well, you've got to realize that this is a really low-level interface
designed to make certain optimizations possible.  The filename is
needed (a) for diagnostics, (b) for opening the .pyc file; the open
file is a side effect from the search.  It uses fopen() to check for
file existence, to prevent silly things like existing files that
cannot be opened for whatever reason).  You don't want to have to open
the file twice!

> The doc says that:
> 
> 	The file argument is an open file, and filename is 
> 	the corresponding file name; these can be None and '', 
> 	respectively, when the module is not being loaded from a file.
> 
> I'm not sure how the module could not be loaded from a file (unless
> the implication is that the file argument is None and a new file will
> be opened from the filename).

Read the C source and weep. :-)  Some systems have other places from
which they can load modules, e.g. on the Mac a module can be in a
"resource".  Also there are frozen modules.

> If the file can be None anytime, it makes more sense to me.
> But the last phrase in the doc seems to indicate the file argument
> should not be None.
> 
> Perhaps, it's just me, it wouldn't be the first time. :-)

Why are you trying to use this?  Can't you use __import__?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From neal@metaslash.com  Wed Oct 17 20:24:05 2001
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 17 Oct 2001 15:24:05 -0400
Subject: [Python-Dev] bug or feature in imp.load_module()?
References: <3BCD8FF4.96457AD8@metaslash.com> <200110171732.NAA16813@cj20424-a.reston1.va.home.com>
 <3BCDCC73.A516C31E@metaslash.com> <200110171905.PAA17035@cj20424-a.reston1.va.home.com>
Message-ID: <3BCDDAD5.EAEEE1E4@metaslash.com>

Guido van Rossum wrote:

[stuff about imp.load_module() deleted]

> Why are you trying to use this?  Can't you use __import__?

This was from ancient code.  I think __import__ could be used.
I haven't modified this code for a long time (I don't even think
I wrote the original code).  The code is used to load the modules
so they can be processed by pychecker.

When I change pychecker to not actually import the file, 
using imp.load_module() or __import__ will be moot.

This was just something I noticed and seemed inconsistent to me.
Although it was easy enough to overcome with a file.seek(0).

Neal


From tim.one@home.com  Thu Oct 18 03:08:03 2001
From: tim.one@home.com (Tim Peters)
Date: Wed, 17 Oct 2001 22:08:03 -0400
Subject: [Python-Dev] re: conditional expressions
In-Reply-To: <930BBCA4CEBBD411BE6500508BB3328F4322C4@nsamcanms1.ca.baltimore.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEJIMBAA.tim.one@home.com>

[Greg Wilson]
> I just showed the proposed conditional expression syntax
> to two colleagues, and asked them how they would write
> the "sign" function (-1 for negative, 0 for zero, +1 for
> positive) using it.  Both of them wrote:
>
>     s = if x < 0 then -1 elif x == 0 then 0 else 1
>
> (Well, OK, one of them tested negative, then positive, then
> returned zero as the 'else', but that's a quibble.)

Quibble?  A professor in college took points off one of my programs because
I didn't test for 0 first.  He explained that testing for either negative or
positive first left you vulnerable on 1's-complement machines, where
comparisons for <0 or >0 just checked the sign bit, and either form of
1's-comp 0 would pass one of those tests.  I looked at him like he was
insane.  He looked at me the same way.  He was old enough that I suppose
he's dead now.  Ha!  I win <wink>.

elif-is-file-spelled-backwards-ly y'rs  - tim



From tim.one@home.com  Thu Oct 18 03:30:54 2001
From: tim.one@home.com (Tim Peters)
Date: Wed, 17 Oct 2001 22:30:54 -0400
Subject: [Python-Dev] more pychecker warnings from python-current
In-Reply-To: <200110170557.BAA09026@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEJJMBAA.tim.one@home.com>

>> doctest.py:528: Parameter (prefix) not used
>> 	docstring says prefix is used, but it isn't

> Tim?

No, the docstring says prefix is ignored:

def is_private(prefix, base):
    """prefix, base -> true iff name prefix + "." + base is "private".

    Prefix may be an empty string, and base does not contain a period.
    Prefix is ignored (although functions you write conforming to this
    ^^^^^^^^^^^^^^^^^
    protocol may make use of it).
    ...

is_private is a user-replaceable function, and doctest is careful to *pass*
the correct prefix to it, but the default implementation makes no use of
prefix.  It's there for advanced users who want to factor in, e.g., the name
of a containing class, when making an "is this name private?" decision.

I wouldn't be averse to adding, e.g.,

    if 0:
        prefix  # make prefix appear used to checking tools

but sooner or later checking tools will complain about that too.

In my last job, we resorted to a C++ macro of the flavor

#define UNUSED_VAR_OK(VAR) (void)ignore_var((void*)&(VAR))

to shut up compiler wngs about intentionally ignored vrbls, and actually had
a do-nothing extern void ignore_var(void*) function -- nothing short of that
shut up every compiler we used.

can't-win-ly y'rs  - tim



From neal@metaslash.com  Thu Oct 18 03:53:48 2001
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 17 Oct 2001 22:53:48 -0400
Subject: [Python-Dev] more pychecker warnings from python-current
References: <LNBBLJKPBEHFEDALKOLCGEJJMBAA.tim.one@home.com>
Message-ID: <3BCE443C.E032A033@metaslash.com>

Tim Peters wrote:
> 
> >> doctest.py:528: Parameter (prefix) not used
> >>      docstring says prefix is used, but it isn't
> 
> > Tim?
> 
> No, the docstring says prefix is ignored:
> 
> def is_private(prefix, base):
>     """prefix, base -> true iff name prefix + "." + base is "private".
> 
>     Prefix may be an empty string, and base does not contain a period.
>     Prefix is ignored (although functions you write conforming to this
>     ^^^^^^^^^^^^^^^^^
>     protocol may make use of it).
>     ...

Oops, sorry about that, I saw all the example usages in the docstring and 
thought it was used.  I didn't pay close enough attention.

> I wouldn't be averse to adding, e.g.,
> 
>     if 0:
>         prefix  # make prefix appear used to checking tools
> 
> but sooner or later checking tools will complain about that too.

I wouldn't worry about it.  I think I will eventually create
a suppressions dictionary for the std library.  So any warning
that checker might normally output will not produce a warning,
when it has been determined the code is correct.

The capability exists to do this today.  However, my concern is putting
in a suppression and later that suppression masks a real error.

In general, we (collectively) can define best practices for both code
and checking tools, at least pychecker.

Neal


From thomas.heller@ion-tof.com  Thu Oct 18 09:28:58 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Thu, 18 Oct 2001 10:28:58 +0200
Subject: [Python-Dev] Function arguments
Message-ID: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook>

Call me weird, but sometimes I need functions taking a number of
positional arguments, some named arguments with default values, and
other named arguments as well.

Something like this (which does _not_ work):

def function(*args, defarg1=None, defarg2=0, **kw):
    ...

The usual way to program this is:

def function(*args, ***kw):
    if kw.has_key('defarg1'):
        defarg1 = kw['defarg1']
        del kw['defarg1']
    else:
        defarg1 = None

    if kw.has_key('defarg2'):
        defarg1 = kw['defarg2']
        del kw['defarg2']
    else:
        defarg2 = 0

    # ...process the positional arguments in args
    # ...process the remaining named arguments in kw

Doesn't look very pretty IMO, and using the dictionaries'
get method doesn't help.

I was thinking of a popitem() dictionary method taking
(optionally) 2 arguments: the name of the item to pop,
and the default value to return if the item is not present
in the dictionary:

>>> d = {'a': 2, 'b': 3}
>>> d.popitem('defarg', 0)
0
>>> d
{'a': 2, 'b': 3}
>>> d.popitem('a', 100)
2
>>> d
{'b': 3}
>>>

Opinions?

Thomas






From mwh@python.net  Thu Oct 18 10:42:31 2001
From: mwh@python.net (Michael Hudson)
Date: 18 Oct 2001 05:42:31 -0400
Subject: [Python-Dev] more pychecker warnings from python-current
In-Reply-To: "Tim Peters"'s message of "Wed, 17 Oct 2001 22:30:54 -0400"
References: <LNBBLJKPBEHFEDALKOLCGEJJMBAA.tim.one@home.com>
Message-ID: <2madypp9mg.fsf@starship.python.net>

"Tim Peters" <tim.one@home.com> writes:

> I wouldn't be averse to adding, e.g.,
> 
>     if 0:
>         prefix  # make prefix appear used to checking tools
> 
> but sooner or later checking tools will complain about that too.

If they work by scanning bytecode (as PyChecker does, doesn't it?),
that won't work today.

pointless-ly y'rs,
M.

-- 
  Screaming 14-year-old boys attempting to prove to each other that
  they are more 3133t than j00.
         -- Reason #8 for quitting slashdot today, from
            http://www.cs.washington.edu/homes/klee/misc/slashdot.html


From fdrake@acm.org  Thu Oct 18 13:00:00 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Thu, 18 Oct 2001 08:00:00 -0400
Subject: [Python-Dev] Function arguments
In-Reply-To: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook>
References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook>
Message-ID: <15310.50240.333969.995440@grendel.zope.com>

Thomas Heller writes:
 > I was thinking of a popitem() dictionary method taking
 > (optionally) 2 arguments: the name of the item to pop,
 > and the default value to return if the item is not present

  A one-arg version of the {}.popitem() method was discussed
extensively, but some people did not think it sufficiently useful to
include that variation.  (I don't think anyone suggested a 2-arg
version at the time.)
  This certainly leads me to think the decision not to support a
one-arg version should be considered, or maybe a two-arg version.
However, I do expect the usage with args and without args are
substantially different.  Perhaps there should be a different method
that has the one- and two-arg versions you describe, but does not
support the 0-arg semantics of the current popitem()?  The catch is
that I'd name this popitem() as well.  ;-)
  Anyone else?


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From thomas.heller@ion-tof.com  Thu Oct 18 13:27:46 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Thu, 18 Oct 2001 14:27:46 +0200
Subject: [Python-Dev] Function arguments
References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> <15310.50240.333969.995440@grendel.zope.com>
Message-ID: <037501c157d0$4b1e3710$e000a8c0@thomasnotebook>

> Thomas Heller writes:
>  > I was thinking of a popitem() dictionary method taking
>  > (optionally) 2 arguments: the name of the item to pop,
>  > and the default value to return if the item is not present
> 

From: "Fred L. Drake, Jr." <fdrake@acm.org>
>   A one-arg version of the {}.popitem() method was discussed
> extensively, but some people did not think it sufficiently useful to
> include that variation.  (I don't think anyone suggested a 2-arg
> version at the time.)
I tried to look it up in the archives. Although (because?) there are lots
of messages about popitem() I cannot locate the relevant ones quick.

>   This certainly leads me to think the decision not to support a
> one-arg version should be considered, or maybe a two-arg version.
> However, I do expect the usage with args and without args are
> substantially different.  Perhaps there should be a different method
> that has the one- and two-arg versions you describe, but does not
> support the 0-arg semantics of the current popitem()?  The catch is
> that I'd name this popitem() as well.  ;-)
>   Anyone else?
After posting I noticed that {}.popitem() returns a (key, value) tuple.

To make it absolutely clear (most of the time I'm not very lucky
in explaining what I really want):
What I mean is a {}.get(key, [value=None]) method, which returns the value,
and removes the (key, value) pair from the dict if it is present.
No idea how it should be named - since it does only return the value,
popitem doesn't seem a good idea.

Thomas



From loewis@informatik.hu-berlin.de  Thu Oct 18 15:28:42 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Thu, 18 Oct 2001 16:28:42 +0200 (MEST)
Subject: [Python-Dev] 2.1.1 maintenance branch
Message-ID: <200110181428.QAA16670@paros.informatik.hu-berlin.de>

How should the 2.1 branch be used? Is it acceptable to propagate
patches from the mainline to that branch? Is there somebody that needs
to review changes made to the branch?

Specific case in question: Anthony Baxter found Zope crashing, and
apparently ceval.c 2.277 fixes that crash. I would now like to forward
this patch to the 2.1 branch.

Regards,
Martin


From skip@pobox.com (Skip Montanaro)  Thu Oct 18 15:29:10 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Thu, 18 Oct 2001 09:29:10 -0500
Subject: [Python-Dev] Function arguments
In-Reply-To: <15310.50240.333969.995440@grendel.zope.com>
References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook>
 <15310.50240.333969.995440@grendel.zope.com>
Message-ID: <15310.59190.519026.561865@beluga.mojam.com>

It's perhaps worth pointing out that dicts already have a zero-arg popitem
method.  I don't think I would use Thomas's proposed popitem so frequently
that the fairly obvious

    v = d.get(k, default)
    try: del d[k]
    except KeyError: pass

would be all that much trouble to type.  Those are the semantics you're
proposing, right?

Skip


From nas@python.ca  Thu Oct 18 15:38:15 2001
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 18 Oct 2001 07:38:15 -0700
Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7
In-Reply-To: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net>; from noreply@sourceforge.net on Thu, Oct 18, 2001 at 03:31:42AM -0700
References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net>
Message-ID: <20011018073815.A22506@glacier.arctrix.com>

[I'm moving the discussion here from SF, using the tracker is too
painful.]

Anthony Baxter:
> I've got a Zope installation where python2.1.1 is
> segfaulting on Solaris2.7 - it's running a largish 
> ZEO server.
[..]
> Here's the trace with debugging enabled:
> 
> #0  0xff00 in ?? ()
> #1  0x402f0 in collect (young=0x9b538, old=0x9b544) at
> ./Modules/gcmodule.c:379
> #2  0x405a8 in collect_generations () at
> ./Modules/gcmodule.c:484
> #3  0x40624 in _PyGC_Insert (op=0xbc1f24) at
> ./Modules/gcmodule.c:507
> #4  0x5a224 in PyList_New (size=0) at Objects/listobject.c:61
> #5  0x21bc8 in eval_code2 (co=0x1cb370, globals=0x21bc0,
> locals=0x67,
>     args=0x0, argcount=1, kws=0xf89b24, kwcount=0, defs=0x0,
> defcount=0,
>     closure=0xbc1f24) at Python/ceval.c:1741
> 
> Next trick is to rebuild without any optimisation (sigh)
> as I suspect that it's inlined subtract_refs().

Martin v. Löwis:
> It would be interesting what the value of "gc" is at the 
> time of the crash. It looks like you got an object that 
> claims to support GC but has a null tp_traverse.

Anthony Baxter:
> Ok, I have an intact core file, and a matching binary,
> no optimisations, nothing. This crash is showing the
> crash at line 166 of gcmodule.c
>  traverse = PyObject_FROM_GC(gc)->ob_type->tp_traverse;
> PyObject_FROM_GC(gc)->ob_type in this case is
> 
> $24 = {ob_refcnt = 1, ob_type = 0x0}
> 
> To check my logic, I checked gc_next and gc_prev using 
> the same GDB magic, and they correctly show up as a tuple
> and an instance method. 
> 
> Some fiddling around seems to rule out stack space as the
> problem, as well. We're going to try and see if purify 
> helps here, but the problem looks to be a junk object - 
> I have no idea how to track this down further. Help?
> Would taking the horrible horrible hack of removing the
> object from the gc linked list if ob_type is null help?
> Well, it'd stop the crashes, anyway.

Martin v. Löwis:
> There are two options:
> 
> a) the object isn't really a GC object, i.e. has no GC
> header. In gdb, you can try to cast gc to PyObject*, then
> see if the resulting pointer has a better ob_type (this is
> unlikely, though, since the logic entering the object was
> already using fromgc/togc)
> 
> b) somebody has cleared the ob_type field.
> 
> Can you guarantee that all extension modules have been
> compiled with the 2.1.1 header files?
> 
> Is the problem repeatable in the sense that gc will have the
> same pointer value on each crash? If so, it is relatively
> easy to track down: just set a gdb change watchpoint on the
> address on the ob_type field of that address (note that
> setting watchpoints is not possible until there is really
> mapped memory on that address).
> 
> If you can't analyse it through change breakpoints, I
> recommend to annotate the interpreter in the following way:
> in pyobject_init, put a printf that prints the address and
> the tp_name of the type. In subtract_refs, if the ob_type
> slot is null, print the address of the object and abort.
> Then analyse the log to see whether a object really has been
> allocated on that address, and what its type was (make sure
> you consider the possibility that address are off by the
> delta that FROM_GC adds).

Anthony Baxter:
> It's not a GC object. I'm positive all the extension 
> objects are correct - I just recompiled, without the
> 1.5/2.0 headers around.
> It's a different pointer each time round, unfortunately. It 
> also takes anything from 5 minutes to 2 hours to reproduce.
> I've got about 4 copies of it running now, and I've got a
> bunch of different core files. I've grabbed purify and an
> eval license, and I'm feeding it the binary. 
> 
> The printf approach is probably not going to work - these
> are busy busy Zope servers. Instead, my plan, assuming that
> purify doesn't immediately spot a problem, is to change the
> code so that if it gets a dud GC object, it will just bust
> it out of the tree and let it leak, and print a message 
> saying so. Then I can quit the program, and purify will
> tell me 'hey, you leaked!' and also tell me where it was
> allocated. 
> 
> More concerning, about half the segfaults are not from the
> GC at all, but from realloc in PyFrame_New (line 161 of
> frameobject). These are the only two I'm getting - it's 
> split 50-50 amongst the 10 coredumps I have now. I'm not
> sure whether to open a seperate bug for this. 
> 
> Has python2.1.1 been purified? With Zope and zope's 
> extensions?
> 
> 
> Wow - it's amazing how this SF bug thing is so painful for
> conversations :)

The ob_type pointer must be getting cleared after the object has been
added to the GC lists.  The PyObject_IS_GC call in _PyGC_Insert would
have segfaulted otherwise.  Knowing the type of the object would be
helpful in debugging the problem.  I suggest reconsidering Martin's
printf idea.  You could add something like this to _PyGC_Insert:

    void
    _PyGC_Insert(PyObject *op)
    {
        static int did_open = 0;
        static FILE *log;
        if (!did_open) {
            did_open = 1;
            log = fopen("type.log", "w");
        }
        fprintf(log, "%p %p\n", op, op->ob_type);
    ...
        
Debugging this type of problem is really hard (as you already know)
because the effect of the bug is found so far away from the source.

  Neil


From guido@python.org  Thu Oct 18 15:41:35 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 10:41:35 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: Your message of "Thu, 18 Oct 2001 16:28:42 +0200."
 <200110181428.QAA16670@paros.informatik.hu-berlin.de>
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de>
Message-ID: <200110181441.f9IEfZF02019@odiug.zope.com>

> How should the 2.1 branch be used? Is it acceptable to propagate
> patches from the mainline to that branch? Is there somebody that needs
> to review changes made to the branch?
> 
> Specific case in question: Anthony Baxter found Zope crashing, and
> apparently ceval.c 2.277 fixes that crash. I would now like to forward
> this patch to the 2.1 branch.

I don't know that we're going to do a 2.1.2 release, but *if* we do,
it will be released from the 2.1 branch, so it's a good idea to start
migrating patches to that branch -- VERY selectively.  This is a good
one to start with: an obvious bug that causes a core dump.

If someone (you?) volunteers to act as a 2.1.2 release manager,
chances of a 2.1.2 release go way up.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Anthony Baxter <anthony@interlink.com.au>  Thu Oct 18 16:11:20 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Fri, 19 Oct 2001 01:11:20 +1000
Subject: [Python-Dev] Re: python2.1.1 SEGV in GC on Solaris 2.7
In-Reply-To: Message from Neil Schemenauer <nas@python.ca>
 of "Thu, 18 Oct 2001 07:38:15 MST." <20011018073815.A22506@glacier.arctrix.com>
Message-ID: <200110181511.f9IFBKu01348@mbuna.arbhome.com.au>

>>> Neil Schemenauer wrote
> [I'm moving the discussion here from SF, using the tracker is too
> painful.]

ok. please CC me on any replies. =


current update - I applied the PyFrame_BlockSetup() fix for ceval.c
that Martin suggested, and so far I've had 4 processes running for
a couple of hours each, without any splattage. this doesn't necessarily
mean it's fixed - the time between failures has varied from 5 minutes to
12 hours over the last few days.

> The ob_type pointer must be getting cleared after the object has been
> added to the GC lists.  The PyObject_IS_GC call in _PyGC_Insert would
> have segfaulted otherwise.  Knowing the type of the object would be
> helpful in debugging the problem.  I suggest reconsidering Martin's
> printf idea.  You could add something like this to _PyGC_Insert:

I'll have a poke at this tomorrow, and see if it's possible at all -
the problem is that the boxes are part of a zope cluster that gets =

a _lot_ of hits (something over 1m/day) so that's a lot of objects...

I'm also going to have another go at getting purify to work on our
stripped down production servers.
         =

> Debugging this type of problem is really hard (as you already know)
> because the effect of the bug is found so far away from the source.

*nod* I'm aware of it. =


thanks!
Anthony

-- =

Anthony Baxter     <anthony@interlink.com.au>   =

It's never too late to have a happy childhood.



From nas@python.ca  Thu Oct 18 16:19:06 2001
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 18 Oct 2001 08:19:06 -0700
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: <200110181441.f9IEfZF02019@odiug.zope.com>; from guido@python.org on Thu, Oct 18, 2001 at 10:41:35AM -0400
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com>
Message-ID: <20011018081906.A22654@glacier.arctrix.com>

Guido van Rossum wrote:
> If someone (you?) volunteers to act as a 2.1.2 release manager,
> chances of a 2.1.2 release go way up.

I'm willing to spend some time backporting patches but I pretty sure I
don't have time to be the release manager.

  Neil


From barry@zope.com  Thu Oct 18 16:25:56 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 18 Oct 2001 11:25:56 -0400
Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7
References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net>
 <20011018073815.A22506@glacier.arctrix.com>
Message-ID: <15310.62596.531353.944255@anthem.wooz.org>

    NS> The ob_type pointer must be getting cleared after the object
    NS> has been added to the GC lists.  The PyObject_IS_GC call in
    NS> _PyGC_Insert would have segfaulted otherwise.  Knowing the
    NS> type of the object would be helpful in debugging the problem.
    NS> I suggest reconsidering Martin's printf idea.  You could add
    NS> something like this to _PyGC_Insert:

Do you know about _PyObject_Dump() and _PyGC_Dump()?  See also
Misc/gdbinit.

-Barry


From thomas.heller@ion-tof.com  Thu Oct 18 16:26:51 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Thu, 18 Oct 2001 17:26:51 +0200
Subject: [Python-Dev] Function arguments
References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook>        <15310.50240.333969.995440@grendel.zope.com> <15310.59190.519026.561865@beluga.mojam.com>
Message-ID: <04ea01c157e9$4fc79180$e000a8c0@thomasnotebook>

From: "Skip Montanaro" <skip@pobox.com>
> 
> It's perhaps worth pointing out that dicts already have a zero-arg popitem
> method.  I don't think I would use Thomas's proposed popitem so frequently
> that the fairly obvious
> 
>     v = d.get(k, default)
>     try: del d[k]
>     except KeyError: pass
> 
> would be all that much trouble to type.  Those are the semantics you're
> proposing, right?
Exactly.

Thomas



From loewis@informatik.hu-berlin.de  Thu Oct 18 16:29:48 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Thu, 18 Oct 2001 17:29:48 +0200 (MEST)
Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7
In-Reply-To: <15310.62596.531353.944255@anthem.wooz.org> (barry@zope.com)
References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net>
 <20011018073815.A22506@glacier.arctrix.com> <15310.62596.531353.944255@anthem.wooz.org>
Message-ID: <200110181529.RAA19958@paros.informatik.hu-berlin.de>

> Do you know about _PyObject_Dump() and _PyGC_Dump()?  See also
> Misc/gdbinit.

In the specific problem, ob_type is NULL at the time of the crash, so
_PyObject_Dump won't do anything interesting.

Regards,
Martin


From guido@python.org  Thu Oct 18 16:31:51 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 11:31:51 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: Your message of "Thu, 18 Oct 2001 08:19:06 PDT."
 <20011018081906.A22654@glacier.arctrix.com>
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com>
 <20011018081906.A22654@glacier.arctrix.com>
Message-ID: <200110181532.f9IFVvm02497@odiug.zope.com>

> I'm willing to spend some time backporting patches but I pretty sure I
> don't have time to be the release manager.

OK, we can do it without a release manager for a while.  What kind of
patches did you have in mind?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Oct 18 16:37:14 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 11:37:14 -0400
Subject: [Python-Dev] warnings in dynload_shlib.c and posixmodule.c
Message-ID: <200110181537.f9IFbK204967@odiug.zope.com>

When compiling the latest CVS on Linux (Mandrake 8.0) I get these
warnings:

../Python/dynload_shlib.c: In function `_PyImport_GetDynLoadFunc':
../Python/dynload_shlib.c:72: warning: implicit declaration of function `dlsym'
../Python/dynload_shlib.c:88: warning: implicit declaration of function `dlopen'
../Python/dynload_shlib.c:88: warning: assignment makes pointer from integer without a cast
../Python/dynload_shlib.c:91: warning: implicit declaration of function `dlerror'
../Python/dynload_shlib.c:91: warning: passing arg 2 of `PyErr_SetString' makes pointer from integer without a cast
../Modules/posixmodule.c: In function `posix_setgroups':
../Modules/posixmodule.c:3169: warning: implicit declaration of function `setgroups'

Looks like some recent checkins...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Oct 18 16:38:45 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 11:38:45 -0400
Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7
In-Reply-To: Your message of "Thu, 18 Oct 2001 17:29:48 +0200."
 <200110181529.RAA19958@paros.informatik.hu-berlin.de>
References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net> <20011018073815.A22506@glacier.arctrix.com> <15310.62596.531353.944255@anthem.wooz.org>
 <200110181529.RAA19958@paros.informatik.hu-berlin.de>
Message-ID: <200110181538.f9IFcp505270@odiug.zope.com>

> > Do you know about _PyObject_Dump() and _PyGC_Dump()?  See also
> > Misc/gdbinit.
> 
> In the specific problem, ob_type is NULL at the time of the crash, so
> _PyObject_Dump won't do anything interesting.

Plus the only way he can debug this is by waiting for a core dump to
happen; I don't think those routines work in post-mortem debug
sesions.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Thu Oct 18 17:07:29 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 18 Oct 2001 12:07:29 -0400
Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7
References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net>
 <20011018073815.A22506@glacier.arctrix.com>
 <15310.62596.531353.944255@anthem.wooz.org>
 <200110181529.RAA19958@paros.informatik.hu-berlin.de>
 <200110181538.f9IFcp505270@odiug.zope.com>
Message-ID: <15310.65089.228565.876912@anthem.wooz.org>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Plus the only way he can debug this is by waiting for a core
    GvR> dump to happen; I don't think those routines work in
    GvR> post-mortem debug sesions.

You can run the process under gdb or attach to it early in the startup
period.  Then when gdb gets the SEGV, you can often (but not always!)
_PyObject_Dump() objects on the call stack.

-Barry


From nas@python.ca  Thu Oct 18 17:19:14 2001
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 18 Oct 2001 09:19:14 -0700
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: <200110181532.f9IFVvm02497@odiug.zope.com>; from guido@python.org on Thu, Oct 18, 2001 at 11:31:51AM -0400
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com>
Message-ID: <20011018091913.A22846@glacier.arctrix.com>

Guido van Rossum wrote:
> OK, we can do it without a release manager for a while.  What kind of
> patches did you have in mind?

So far: Thomas's ceval fix, profile.py, abstract_isinstance.  I'm sure
there's lots more if I start looking through the logs.  Is it worthing
fixing the alignment of the GC header?

  Neil


From fredrik@pythonware.com  Thu Oct 18 17:35:52 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 18 Oct 2001 18:35:52 +0200
Subject: [Python-Dev] 2.2b1 release plan?
Message-ID: <003701c157f2$f6ab05f0$ced241d5@hagrid>

the PEP says October 10.

I was planning to do some SRE work tonight, but
I'm not sure I'll be able to squeeze it in.  when is
2.2b1 going out?  will I have time to work on this
over the weekend?

</F>



From guido@python.org  Thu Oct 18 17:31:20 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 12:31:20 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: Your message of "Thu, 18 Oct 2001 09:19:14 PDT."
 <20011018091913.A22846@glacier.arctrix.com>
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com>
 <20011018091913.A22846@glacier.arctrix.com>
Message-ID: <200110181631.f9IGVLR11199@odiug.zope.com>

> > OK, we can do it without a release manager for a while.  What kind of
> > patches did you have in mind?
> 
> So far: Thomas's ceval fix, profile.py, abstract_isinstance.  I'm sure
> there's lots more if I start looking through the logs.  Is it worthing
> fixing the alignment of the GC header?

I think so.

I'n curious about abstract_isinstance -- is that what you checked in
last night?  I thought that was for 2.2 but I may have misunderstood
the issue (I have to admit I have no idea what abstract_isinstance
does any more :-( ).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim@zope.com  Thu Oct 18 17:35:57 2001
From: tim@zope.com (Tim Peters)
Date: Thu, 18 Oct 2001 12:35:57 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: <20011018091913.A22846@glacier.arctrix.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHKENPCIAA.tim@zope.com>

[Neil Schemenauer]
> Is it worthing fixing the alignment of the GC header?

I wouldn't -- there was nothing in the core in 2.1.1 that both participated
in gc and required double alignment, and no reports of extension modules
that care.



From guido@python.org  Thu Oct 18 17:35:22 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 12:35:22 -0400
Subject: [Python-Dev] 2.2b1 release plan?
In-Reply-To: Your message of "Thu, 18 Oct 2001 18:35:52 +0200."
 <003701c157f2$f6ab05f0$ced241d5@hagrid>
References: <003701c157f2$f6ab05f0$ced241d5@hagrid>
Message-ID: <200110181635.f9IGZMM11249@odiug.zope.com>

> the PEP says October 10.

Sorry, we slipped by about the same amount as 2.2a4.  It goes out tomorrow.

> I was planning to do some SRE work tonight, but
> I'm not sure I'll be able to squeeze it in.  when is
> 2.2b1 going out?  will I have time to work on this
> over the weekend?

Argh!  Please squeeze it in tonight.  Fortunately your tonight is
earlier than ours. :)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim@zope.com  Thu Oct 18 17:38:02 2001
From: tim@zope.com (Tim Peters)
Date: Thu, 18 Oct 2001 12:38:02 -0400
Subject: [Python-Dev] 2.2b1 release plan?
In-Reply-To: <003701c157f2$f6ab05f0$ced241d5@hagrid>
Message-ID: <BIEJKCLHCIOIHAGOKOLHCEOACIAA.tim@zope.com>

[/F]
> I was planning to do some SRE work tonight, but
> I'm not sure I'll be able to squeeze it in.  when is
> 2.2b1 going out?

Tomorrow (Friday).

> will I have time to work on this over the weekend?

I hope so <wink>.



From guido@python.org  Thu Oct 18 17:55:44 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 12:55:44 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: Your message of "Thu, 18 Oct 2001 12:35:57 EDT."
 <BIEJKCLHCIOIHAGOKOLHKENPCIAA.tim@zope.com>
References: <BIEJKCLHCIOIHAGOKOLHKENPCIAA.tim@zope.com>
Message-ID: <200110181655.f9IGti111355@odiug.zope.com>

> [Neil Schemenauer]
> > Is it worthing fixing the alignment of the GC header?

[Tim]
> I wouldn't -- there was nothing in the core in 2.1.1 that both participated
> in gc and required double alignment, and no reports of extension modules
> that care.

You're right.  I forgot about this.  Let's not do this one.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Thu Oct 18 18:58:40 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 18 Oct 2001 13:58:40 -0400
Subject: [Python-Dev] 2.2b1 release plan?
References: <003701c157f2$f6ab05f0$ced241d5@hagrid>
Message-ID: <15311.6224.239595.715872@anthem.wooz.org>

>>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:

    FL> the PEP says October 10.

The PEP has been updated.

-Barry


From nas@python.ca  Thu Oct 18 20:53:22 2001
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 18 Oct 2001 12:53:22 -0700
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: <200110181631.f9IGVLR11199@odiug.zope.com>; from guido@python.org on Thu, Oct 18, 2001 at 12:31:20PM -0400
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <20011018091913.A22846@glacier.arctrix.com> <200110181631.f9IGVLR11199@odiug.zope.com>
Message-ID: <20011018125322.A23424@glacier.arctrix.com>

Guido van Rossum wrote:
> I'n curious about abstract_isinstance -- is that what you checked in
> last night?

Yes.

> I thought that was for 2.2 but I may have misunderstood
> the issue (I have to admit I have no idea what abstract_isinstance
> does any more :-( ).

No, abstract_isinstance has been broken for a long time, maybe forever.
It is supposed to work for class-like objects like ExtensionClass
instances as well as for standard classes and types.  One problem was
that if you gave it a normal instance as the first argument but an
ExtensionClass as the second argument it raised a type error.  It's
annoying to us at the MEMS Exchange because we use ExtensionClasses and
isinstance all the time (Greg likes type checking).  We ended up
rewriting isinstance() and issubclass() in Python.  The average Python
user doesn't care.

  Neil


From guido@python.org  Thu Oct 18 20:58:20 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 15:58:20 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: Your message of "Thu, 18 Oct 2001 12:53:22 PDT."
 <20011018125322.A23424@glacier.arctrix.com>
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <20011018091913.A22846@glacier.arctrix.com> <200110181631.f9IGVLR11199@odiug.zope.com>
 <20011018125322.A23424@glacier.arctrix.com>
Message-ID: <200110181958.f9IJwKA29065@odiug.zope.com>

> > I'n curious about abstract_isinstance -- is that what you checked in
> > last night?
> 
> Yes.
> 
> > I thought that was for 2.2 but I may have misunderstood
> > the issue (I have to admit I have no idea what abstract_isinstance
> > does any more :-( ).
> 
> No, abstract_isinstance has been broken for a long time, maybe forever.
> It is supposed to work for class-like objects like ExtensionClass
> instances as well as for standard classes and types.  One problem was
> that if you gave it a normal instance as the first argument but an
> ExtensionClass as the second argument it raised a type error.  It's
> annoying to us at the MEMS Exchange because we use ExtensionClasses and
> isinstance all the time (Greg likes type checking).  We ended up
> rewriting isinstance() and issubclass() in Python.  The average Python
> user doesn't care.

Then I'm not sure if this should be fixed in 2.1.2.  It's almost a new
feature that it doesn't bomb out. :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Thu Oct 18 21:11:07 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Thu, 18 Oct 2001 22:11:07 +0200
Subject: [Python-Dev] 2.1.1 maintenance branch
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com>              <20011018081906.A22654@glacier.arctrix.com>  <200110181532.f9IFVvm02497@odiug.zope.com>
Message-ID: <070b01c15811$05e3d470$e000a8c0@thomasnotebook>

> 
> OK, we can do it without a release manager for a while.  What kind of
> patches did you have in mind?

Another bug which should be repaired in a 2.1.2 release
is that distutils version number is invalid, see

[ 417796 ] illegal version 1.0.2pre (priority 5)
http://sf.net/tracker/?func=detail&aid=417796&group_id=5470&atid=105470

Thomas



From fdrake@acm.org  Thu Oct 18 21:10:34 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Thu, 18 Oct 2001 16:10:34 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011018201034.72D8E28697@cj42289-a.reston1.va.home.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Documentation as packaged with Python 2.2 beta 1.


From skip@pobox.com (Skip Montanaro)  Thu Oct 18 21:17:15 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Thu, 18 Oct 2001 15:17:15 -0500
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: <20011018125322.A23424@glacier.arctrix.com>
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de>
 <200110181441.f9IEfZF02019@odiug.zope.com>
 <20011018081906.A22654@glacier.arctrix.com>
 <200110181532.f9IFVvm02497@odiug.zope.com>
 <20011018091913.A22846@glacier.arctrix.com>
 <200110181631.f9IGVLR11199@odiug.zope.com>
 <20011018125322.A23424@glacier.arctrix.com>
Message-ID: <15311.14539.305244.20323@beluga.mojam.com>

    Neil> No, abstract_isinstance has been broken for a long time, maybe
    Neil> forever.  It is supposed to work for class-like objects like
    Neil> ExtensionClass instances as well as for standard classes and
    Neil> types.  One problem was that if you gave it a normal instance as
    Neil> the first argument but an ExtensionClass as the second argument it
    Neil> raised a type error.  It's annoying to us at the MEMS Exchange
    Neil> because we use ExtensionClasses and isinstance all the time ...

Getting a little off the subject, but a good first test of the new 2.2
type/class stuff is that it only took James Henstridge a couple days to
convert the PyGtk2 stuff from ExtensionClass to pure 2.2 code.  I've
experienced no problems related to that change.

Skip


From fdrake@acm.org  Thu Oct 18 21:11:31 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Thu, 18 Oct 2001 16:11:31 -0400
Subject: [Python-Dev] 2.1.1 maintenance branch
In-Reply-To: <070b01c15811$05e3d470$e000a8c0@thomasnotebook>
References: <200110181428.QAA16670@paros.informatik.hu-berlin.de>
 <200110181441.f9IEfZF02019@odiug.zope.com>
 <20011018081906.A22654@glacier.arctrix.com>
 <200110181532.f9IFVvm02497@odiug.zope.com>
 <070b01c15811$05e3d470$e000a8c0@thomasnotebook>
Message-ID: <15311.14195.862624.167879@grendel.zope.com>

Guido writes:
 > OK, we can do it without a release manager for a while.  What kind of
 > patches did you have in mind?

  The diff between Objects/weakrefobject.c 1.1 and 1.3 should be
applied to Modules/_weakref.c for 2.1.2, since that could cause core
dumps.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From fredrik@pythonware.com  Thu Oct 18 22:06:53 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 18 Oct 2001 23:06:53 +0200
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7
References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <012d01c15818$d62b0ed0$ced241d5@hagrid>

guido wrote:
> But they also have attributes like st_mtime or tm_year.

umm.  do we really need the "st_" and "tm_" prefixes?

</F>



From fdrake@acm.org  Thu Oct 18 22:00:20 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Thu, 18 Oct 2001 17:00:20 -0400
Subject: [Python-Dev] Python 2.2 beta 1 documentation
Message-ID: <15311.17124.615391.305548@grendel.zope.com>

  The Python 2.2 beta 1 documentation is now available in the usual
places.  The HTML packages are there as always, and one new package: a
version for the iSilo document reader for PalmOS devices (see
www.iSilo.com).  It has problems still, but as long as you navigate by
hyperlinks it should work just fine.  ;-)  If people are interested in
this format I can work on solving the problems with it, so please let
me know if you think you'll be interested in actually using docs in
this format.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Thu Oct 18 22:08:50 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 17:08:50 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7
In-Reply-To: Your message of "Thu, 18 Oct 2001 23:06:53 +0200."
 <012d01c15818$d62b0ed0$ced241d5@hagrid>
References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net>
 <012d01c15818$d62b0ed0$ced241d5@hagrid>
Message-ID: <200110182108.f9IL8pg03390@odiug.zope.com>

> > But they also have attributes like st_mtime or tm_year.
> 
> umm.  do we really need the "st_" and "tm_" prefixes?

I think so.  They make it easier to switch between C and Python
code, documentation, brains etc.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Thu Oct 18 22:29:52 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 18 Oct 2001 23:29:52 +0200
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7
References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net>              <012d01c15818$d62b0ed0$ced241d5@hagrid>  <200110182108.f9IL8pg03390@odiug.zope.com>
Message-ID: <001f01c1581c$0c2a6c80$ced241d5@hagrid>

Guido van Rossum wrote:
> > umm.  do we really need the "st_" and "tm_" prefixes?
> 
> I think so.  They make it easier to switch between C and Python
> code, documentation, brains etc.

does that imply that tm.tm_year != tm[0]?

and should really all those of us who wouldn't dream of
writing time or stat code in C have to suffer?  -0.5 from
me.

</F>



From gward@python.net  Fri Oct 19 00:00:06 2001
From: gward@python.net (Greg Ward)
Date: Thu, 18 Oct 2001 19:00:06 -0400
Subject: [Python-Dev] CVS question: checking in old doc update
Message-ID: <20011018190006.A6046@gerg.ca>

How embarassing.  I've just discovered that I added a couple pages of
text to one of the Distutils docs a year ago, and then never checked it
in.  Since then, various people have made various changes, and simply
"cvs up"'ing produced a horrible mess.

For posterity, I'd like to check in my changes relative to the last
revision I worked on.  That is, sometime in October 2000, I made
revision 1.26 to Doc/dist/dist.tex, and on October 29th (according to my
own backups), I added a bunch of text and made some other changes.

So I *think* I'd like to make a "mini-branch" for this one file, where I
check in my year-old changes relative to revision 1.26.  I tried this:

  cvs ci -r1.26 dist.tex

but CVS said

  cvs server: Up-to-date check failed for `dist.tex'

So I said, "Fine, I'll be explicit about working relative to rev 1.26":

  mv dist.tex dist.tex-hacked
  cvs up -r 1.26 dist.tex
  mv dist.tex-hacked dist.tex
  cvs ci dist.tex

And this time CVS said:

  cvs server: sticky tag `1.26' for file `dist.tex' is not a branch

Sigh.  Do I have to explicitly create a branch for this, as in

  cvs tag -r 1.26 -b last-gpw-revision

?  If that's the only thing to do, is it OK for me to do this?  It's
only one file.  Is that a good name for the branch?

        Greg
-- 
Greg Ward - just another /P(erl|ython)/ hacker          gward@python.net
http://starship.python.net/~gward/
Reality is for people who can't handle science fiction.


From guido@python.org  Fri Oct 19 00:27:24 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 19:27:24 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7
In-Reply-To: Your message of "Thu, 18 Oct 2001 23:29:52 +0200."
 <001f01c1581c$0c2a6c80$ced241d5@hagrid>
References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com>
 <001f01c1581c$0c2a6c80$ced241d5@hagrid>
Message-ID: <200110182327.TAA06329@cj20424-a.reston1.va.home.com>

> does that imply that tm.tm_year != tm[0]?

Fortunately not.

> and should really all those of us who wouldn't dream of
> writing time or stat code in C have to suffer?  -0.5 from
> me.

Call me old-fashioned, but I *like* struct fields with unique
prefixes, in Python as well as in C.  It's a hint about what kind of
data structure we're looking at.  And it greps nicely.

--Guido van Rossum (home page: http://www.python.org/~guido/)



From guido@python.org  Fri Oct 19 00:32:44 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 18 Oct 2001 19:32:44 -0400
Subject: [Python-Dev] CVS question: checking in old doc update
In-Reply-To: Your message of "Thu, 18 Oct 2001 19:00:06 EDT."
 <20011018190006.A6046@gerg.ca>
References: <20011018190006.A6046@gerg.ca>
Message-ID: <200110182332.TAA06390@cj20424-a.reston1.va.home.com>

> Sigh.  Do I have to explicitly create a branch for this, as in
> 
>   cvs tag -r 1.26 -b last-gpw-revision
> 
> ?

Yes.

> If that's the only thing to do, is it OK for me to do this?  It's
> only one file.  Is that a good name for the branch?

Sure.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From DavidA@ActiveState.com  Fri Oct 19 01:19:22 2001
From: DavidA@ActiveState.com (David Ascher)
Date: Thu, 18 Oct 2001 17:19:22 -0700
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.
References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com>              <001f01c1581c$0c2a6c80$ced241d5@hagrid> <200110182327.TAA06329@cj20424-a.reston1.va.home.com>
Message-ID: <3BCF718A.1050501@ActiveState.com>

Guido van Rossum wrote:

> Call me old-fashioned, but I *like* struct fields with unique
> prefixes, in Python as well as in C.  It's a hint about what kind of
> data structure we're looking at.  And it greps nicely.


Ok.  "Old fashioned!"

--david

PS: FWIW, I'm w/ /F on this one.  It never made sense to me why I had to 
type:

	traceback.tb_frame.f_code.co_consts

instead of

         traceback.frame.code.constants

Not a huge deal, though, in the scheme of things. =)



From paulp@ActiveState.com  Fri Oct 19 01:48:46 2001
From: paulp@ActiveState.com (Paul Prescod)
Date: Thu, 18 Oct 2001 17:48:46 -0700
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test
 test_os.py,1.6,1.
References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com>              <001f01c1581c$0c2a6c80$ced241d5@hagrid> <200110182327.TAA06329@cj20424-a.reston1.va.home.com> <3BCF718A.1050501@ActiveState.com>
Message-ID: <3BCF786E.A901374A@ActiveState.com>

David Ascher wrote:
> 
>...
> PS: FWIW, I'm w/ /F on this one.  It never made sense to me why I had to
> type:
> 
>         traceback.tb_frame.f_code.co_consts
> 
> instead of
> 
>          traceback.frame.code.constants

I never liked that either. Plus Python doesn't have a real distinction
between "structs" and "objects" so it is just one more thing to
remember: "does this thing use the obsolete C struct convention or the
modern Python object convention?"

 Paul Prescod


From fdrake@acm.org  Fri Oct 19 03:40:27 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Thu, 18 Oct 2001 22:40:27 -0400
Subject: [Python-Dev] CVS question: checking in old doc update
In-Reply-To: <20011018190006.A6046@gerg.ca>
References: <20011018190006.A6046@gerg.ca>
 <200110182332.TAA06390@cj20424-a.reston1.va.home.com>
Message-ID: <15311.37531.900530.578011@grendel.zope.com>

Greg Ward writes:
 > ?  If that's the only thing to do, is it OK for me to do this?  It's
 > only one file.  Is that a good name for the branch?

  Please don't!  Ignore what Guido said -- the docs are mine.  ;-)
Submit the difference between what you have and what you started from
as a context diff on SourceForge; assign it to me.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From greg@cosc.canterbury.ac.nz  Fri Oct 19 04:52:38 2001
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Fri, 19 Oct 2001 16:52:38 +1300 (NZDT)
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7
In-Reply-To: <200110182327.TAA06329@cj20424-a.reston1.va.home.com>
Message-ID: <200110190352.QAA07859@s454.cosc.canterbury.ac.nz>

Guido:

> Call me old-fashioned, but I *like* struct fields with unique
> prefixes, in Python as well as in C.  It's a hint about what kind of
> data structure we're looking at.

The name of whatever is referring to the struct
should be doing that.

Anyway, if it's such a good thing, why don't you
use it in all the Python code you write?

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From Mark.Favas@csiro.au  Fri Oct 19 04:54:41 2001
From: Mark.Favas@csiro.au (Mark.Favas@csiro.au)
Date: Fri, 19 Oct 2001 11:54:41 +0800
Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately?
Message-ID: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU>

For the last couple of days, socketmodule hasn't compiled for me on Tru64
Unix. If I'm the only one seeing this, I'll file a bug report on sf...

building '_socket' extension
cc: Error: Modules/socketmodule.c, line 2820: An unexpected newline
character is present in a string literal. (nlstring)
"RAND_egd(path) -> bytes
^
cc: Warning: Modules/socketmodule.c, line 2821: Invalid token discarded.
(invaltoken)
\n\
^
cc: Warning: Modules/socketmodule.c, line 2822: Invalid token discarded.
(invaltoken)
Queries the entropy gather daemon (EGD) on socket path.  Returns number\n\
-----------------------------------------------------------------------^
cc: Warning: Modules/socketmodule.c, line 2823: Invalid token discarded.
(invaltoken)
of bytes read.  Raises socket.sslerror if connection to EGD fails or\n\
--------------------------------------------------------------------^
cc: Error: Modules/socketmodule.c, line 2824: An unexpected newline
character is present in a string literal. (nlstring)
if it does provide enough data to seed PRNG.";
--------------------------------------------^
cc: Error: Modules/socketmodule.c, line 3037: In this statement,
"PySocket_methods" is not declared. (undeclared)
	m = Py_InitModule3("_socket", PySocket_methods, module_doc);
------------^
/bin/cc -O -Olimit 1500 -DUSE_SSL=1 -I/usr/local/ssl/include -I.
-I/home/erebus2/groucho1/mark/src/python/CVS/python/dist/src/./Include
-I/usr/local/include -IInclude/ -c
/home/erebus2/groucho1/mark/src/python/CVS/python/dist/src/Modules/socketmod
ule.c -o build/temp.osf1-V4.0-alpha-2.2/socketmodule.o
WARNING: building of extension "_socket" failed: command '/bin/cc' failed
with exit status 1


From anthony@interlink.com.au  Fri Oct 19 05:31:37 2001
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 19 Oct 2001 14:31:37 +1000
Subject: [Python-Dev] Goals for patch selection for 2.1.2
Message-ID: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>

Hi there,

I'm happy enough to be the release manager for 2.1.2 - I've a reasonably
high amount of CVS experience, so that shouldn't be a problem.

Here's my current thoughts for selecting patches for a 2.1.2 release.
Feedback would be appreciated. I plan to post this to python-list and
ask people for suggestions for "must have" fixes. This could possibly
be added to PEP 006 once it's nailed down.

  Code that adds 
    new packages
    new modules
    new methods
  is out. (duh)

  Anything that introduces a large chunk of code should be out.

  Anyone who's subclassing a standard library class shouldn't get screwed
  over - even if they're playing with the internals of the classes.

  C code changes should be _very_ _very_ conservative. Bug fixes that fix 
  problems that people are having are good - anything else should be
  regarded with a hell of a lot of suspicion. And the "don't introduce
  large chunks of new code" test would obviously apply here.

  Things that shouldn't be considered:
    Fixes that "make something work better"
    Fixes that "remove a nasty workaround" 

  In general, I'd like the starting point for any patch to be considered
  as "guilty until proven necessary".

An example (from a discussion with Guido) is the recent profiler fixup:

  We made two sets of changes to the profiler.  One fixed profile.py 
  to properly interpret exception events.  This one should go in 
  (I specifically tested it with Python 1.5.2 and 2.1).  

  The other changed the C code that generates profiling events to 
  suppress exception events, and to generate return events even when 
  a frame is exited with an exception.  That one should not go in.
 
Thanks,
Anthony
--
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.


From barry@zope.com  Fri Oct 19 05:39:49 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 19 Oct 2001 00:39:49 -0400
Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately?
References: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU>
Message-ID: <15311.44693.311402.326528@anthem.wooz.org>

>>>>> "MF" == Mark Favas <Mark.Favas@csiro.au> writes:

    MF> For the last couple of days, socketmodule hasn't compiled for
    MF> me on Tru64 Unix. If I'm the only one seeing this, I'll file a
    MF> bug report on sf...

I just did a fresh cvs update, "make test" on Linux and everything
works for me.  I'm about to create the 2.2b1 release branch, so unless
the patch is in my inbox when I wake up about 8 hours from now, it
probably won't make it into 2.2b1.

-Barry


From Anthony Baxter <anthony@interlink.com.au>  Fri Oct 19 05:52:33 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Fri, 19 Oct 2001 14:52:33 +1000
Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7
In-Reply-To: Message from Martin von Loewis <loewis@informatik.hu-berlin.de>
 of "Thu, 18 Oct 2001 17:29:48 +0200." <200110181529.RAA19958@paros.informatik.hu-berlin.de>
Message-ID: <200110190452.f9J4qXX04666@mbuna.arbhome.com.au>

Ok, after more than 16 hours of running, I can say that I'm pretty
confident the crash-in-realloc is gone. 

However, I'm still seeing crashes (although much much less frequent) 
inside the GC. Same bat time, same bat channel, same bat line-of-code. 
I'm thinking I'll have to try the printf approach :-/ I might wait until 
next week, because our local directors will be back on line then, and it 
will be easier to knock servers in and out of service. 

Anthony


From barry@zope.com  Fri Oct 19 06:51:06 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 19 Oct 2001 01:51:06 -0400
Subject: [Python-Dev] Python 2.2b1 - release branch created
Message-ID: <15311.48970.398745.60515@anthem.wooz.org>

I've created the Python 2.2b1 release branch.  The CVS tag is
`r22b1-branch'.  There's also a tag `r22b1-fork' -- that's the point
on the trunk where the branch forks off.  So far the only change in
the branch is the version number.

As usual, no one except Guido and myself (or our authorized bots)
should make checkins to the branch.  Checkins to the trunk are still
okay, but generally will /not/ be merged into the release branch
unless you notify us (a comment in the checkin message would suffice).

I'll be cutting the release tomorrow morning local time (Friday
October 19).  After the release is done, the branch will be merged
back into the trunk.

Please note too that with the release of 2.2 beta 1, there is now a
feature freeze on the trunk until after the 2.2 final release is made.
According to PEP 251, this is currently scheduled for 19-Dec-2001.
Those of you with commit privileges, please respect the feature
freeze!  You can use SourceForge to archive any feature requests or
feature patches for 2.3.  If you're unsure whether a change you want
to make is a feature or a bug fix, contact us first.

Enjoy,
-Barry


From loewis@informatik.hu-berlin.de  Fri Oct 19 07:17:30 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Fri, 19 Oct 2001 08:17:30 +0200 (MEST)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
Message-ID: <200110190617.IAA12488@pandora.informatik.hu-berlin.de>

Hi Anthony,

Your write-up of management principles for 2.1.2 sounds very reasonable,
I think all these principles should be enforced.

One open issue would be to come up with a release schedule. Should
you restrict yourself to patches that also appear in 2.2 (perhaps in
a back-ported fashion), or will you give a deadline for new patches
to come in? If the former, I think releasing 2.1.2 around the time
that 2.2 is released (perhaps earlier) would be desirable.

Thanks for volunteering,
Martin


From Mark.Favas@csiro.au  Fri Oct 19 08:40:33 2001
From: Mark.Favas@csiro.au (Mark.Favas@csiro.au)
Date: Fri, 19 Oct 2001 15:40:33 +0800
Subject: [Python-Dev] Has socketmodule compilation broken for anyone e
 lse lately?
Message-ID: <116D27C8E12BD411B3AB00B0D022B0B87C46D4@yate.wa.CSIRO.AU>

Bug now logged on SourceForge... Not the first time that checkins to
socketmodule have caused compilation errors (but not on Linux <wink>).

Mark

>>>>> "MF" == Mark Favas <Mark.Favas@csiro.au> writes:

    MF> For the last couple of days, socketmodule hasn't compiled for
    MF> me on Tru64 Unix. If I'm the only one seeing this, I'll file a
    MF> bug report on sf...

I just did a fresh cvs update, "make test" on Linux and everything
works for me.  I'm about to create the 2.2b1 release branch, so unless
the patch is in my inbox when I wake up about 8 hours from now, it
probably won't make it into 2.2b1.

-Barry


From mwh@python.net  Fri Oct 19 09:23:52 2001
From: mwh@python.net (Michael Hudson)
Date: 19 Oct 2001 04:23:52 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib __phello__.foo.py,NONE,1.1
In-Reply-To: Guido van Rossum's message of "Thu, 18 Oct 2001 11:49:39 -0700"
References: <E15uIEd-0002gI-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <2mr8s0rqav.fsf@starship.python.net>

Guido van Rossum <gvanrossum@users.sourceforge.net> writes:

> Update of /cvsroot/python/python/dist/src/Lib
> In directory usw-pr-cvs1:/tmp/cvs-serv9628/Lib
> 
> Added Files:
> 	__phello__.foo.py 

$ cvs up && ./configure --prefix=$HOME && make install
...
$ ls ~/lib/python2.2/__phello__.foo.py
/home/crew/mwh/lib/python2.2/__phello__.foo.py

...?  Shouldn't it at least be in 

/home/crew/mwh/lib/python2.2/test

?

Cheers,
M.


From guido@python.org  Fri Oct 19 13:29:07 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 19 Oct 2001 08:29:07 -0400
Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately?
In-Reply-To: Your message of "Fri, 19 Oct 2001 11:54:41 +0800."
 <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU>
References: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU>
Message-ID: <200110191229.IAA26121@cj20424-a.reston1.va.home.com>

> For the last couple of days, socketmodule hasn't compiled for me on Tru64
> Unix. If I'm the only one seeing this, I'll file a bug report on sf...

It's probably a missing \n\ in a string literal.  I've fixed this on
the release branch now.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Fri Oct 19 13:30:48 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 19 Oct 2001 08:30:48 -0400
Subject: [Python-Dev] Has socketmodule compilation broken for anyone e lse lately?
In-Reply-To: Your message of "Fri, 19 Oct 2001 15:40:33 +0800."
 <116D27C8E12BD411B3AB00B0D022B0B87C46D4@yate.wa.CSIRO.AU>
References: <116D27C8E12BD411B3AB00B0D022B0B87C46D4@yate.wa.CSIRO.AU>
Message-ID: <200110191230.IAA26163@cj20424-a.reston1.va.home.com>

> Bug now logged on SourceForge... Not the first time that checkins to
> socketmodule have caused compilation errors (but not on Linux <wink>).

We should really figure out which gcc command line option makes it
complain about bare newlines in string literals.  Usually, the Windows
build catches these, but this time it was inside an #ifdef SSL...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Fri Oct 19 13:31:55 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 19 Oct 2001 08:31:55 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib __phello__.foo.py,NONE,1.1
In-Reply-To: Your message of "19 Oct 2001 04:23:52 EDT."
 <2mr8s0rqav.fsf@starship.python.net>
References: <E15uIEd-0002gI-00@usw-pr-cvs1.sourceforge.net>
 <2mr8s0rqav.fsf@starship.python.net>
Message-ID: <200110191231.IAA26183@cj20424-a.reston1.va.home.com>

> Guido van Rossum <gvanrossum@users.sourceforge.net> writes:
> 
> > Update of /cvsroot/python/python/dist/src/Lib
> > In directory usw-pr-cvs1:/tmp/cvs-serv9628/Lib
> > 
> > Added Files:
> > 	__phello__.foo.py 
> 
> $ cvs up && ./configure --prefix=$HOME && make install
> ...
> $ ls ~/lib/python2.2/__phello__.foo.py
> /home/crew/mwh/lib/python2.2/__phello__.foo.py
> 
> ...?  Shouldn't it at least be in 
> 
> /home/crew/mwh/lib/python2.2/test
> 
> ?

No, because that's inside a package (depending on how you run the test
suite) and the test requires a file on sys.path.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Fri Oct 19 13:49:40 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 19 Oct 2001 14:49:40 +0200
Subject: [Python-Dev] Bug 216405
Message-ID: <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook>

I know most of you are busy with the release, but may I beg again
that SF bug #216405 be finally fixed in Python 2.2?

I can apply the patch myself, if nobody has time, but
I need an OK from Guido ;-)
The bug is currently marked closed, resolution is set to 'later'.

http://sourceforge.net/tracker/index.php?func=detail&aid=216405&group_id=5470&atid=105470

I know that the buffer stuff is somewhat (broken|fragile|dangerous,
pick your favorite here), but this does at least fix a small point.

Here is the complete patch against the current CVS sources, I've
made a build with this patch applied, and the whole test suite runs
the same way as it does without this patch (can only test on windows):

147 tests OK.
34 tests skipped:
    test_al test_bsddb test_cd test_cl test_commands test_crypt
    test_dbm test_dl test_fcntl test_fork1 test_gdbm test_gl test_grp
    test_gzip test_imgfile test_largefile test_linuxaudiodev
    test_mhlib test_minidom test_nis test_openpty test_poll test_pty
    test_pwd test_pyexpat test_sax test_signal test_socket_ssl
    test_socketserver test_sunaudiodev test_sundry test_timing
    test_zipfile test_zlib
8 skips unexpected on win32:
    test_zipfile test_bsddb test_sax test_sundry test_minidom
    test_zlib test_pyexpat test_gzip

Thomas

Index: bufferobject.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Objects/bufferobject.c,v
retrieving revision 2.15
diff -c -r2.15 bufferobject.c
*** bufferobject.c      2001/08/24 18:34:26     2.15
--- bufferobject.c      2001/10/19 12:46:33
***************
*** 73,83 ****
                offset = count;
        if ( offset + size > count )
                size = count - offset;
! 
!       /* if the base object is another buffer, then "deref" it */
!       if ( PyBuffer_Check(base) )
                base = ((PyBufferObject *)base)->b_base;
! 
        return _PyBuffer_FromMemory(base, (char *)p + offset, size, readonly);
  }
  
--- 73,85 ----
                offset = count;
        if ( offset + size > count )
                size = count - offset;
!       
!       /* if the base object is another buffer, then "deref" it,
!        * except if the base of the other buffer is NULL
!        */
!       if ( PyBuffer_Check(base) && (((PyBufferObject *)base)->b_base) )
                base = ((PyBufferObject *)base)->b_base;
!       
        return _PyBuffer_FromMemory(base, (char *)p + offset, size, readonly);
  }
  




From guido@python.org  Fri Oct 19 14:31:28 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 19 Oct 2001 09:31:28 -0400
Subject: [Python-Dev] Bug 216405
In-Reply-To: Your message of "Fri, 19 Oct 2001 14:49:40 +0200."
 <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook>
References: <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook>
Message-ID: <200110191331.JAA26393@cj20424-a.reston1.va.home.com>

> I know most of you are busy with the release, but may I beg again
> that SF bug #216405 be finally fixed in Python 2.2?
> 
> I can apply the patch myself, if nobody has time, but
> I need an OK from Guido ;-)

OK.  It's too late for the b1 release (Tim has already cooked the
Windows installer) but it will go in the next releas.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Fri Oct 19 14:40:54 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 19 Oct 2001 15:40:54 +0200
Subject: [Python-Dev] Bug 216405
References: <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook>  <200110191331.JAA26393@cj20424-a.reston1.va.home.com>
Message-ID: <0c0a01c158a3$ad1a4ca0$e000a8c0@thomasnotebook>

> > I know most of you are busy with the release, but may I beg again
> > that SF bug #216405 be finally fixed in Python 2.2?
> > 
> > I can apply the patch myself, if nobody has time, but
> > I need an OK from Guido ;-)
> 
> OK.  It's too late for the b1 release (Tim has already cooked the
> Windows installer) but it will go in the next releas.
Great. I can live with the CVS version for now.

Thanks,

Thomas



From fdrake@acm.org  Fri Oct 19 14:52:58 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 19 Oct 2001 09:52:58 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110190617.IAA12488@pandora.informatik.hu-berlin.de>
References: <200110190617.IAA12488@pandora.informatik.hu-berlin.de>
Message-ID: <15312.12346.482959.64272@grendel.zope.com>

Martin von Loewis writes:
 > One open issue would be to come up with a release schedule. Should
 > you restrict yourself to patches that also appear in 2.2 (perhaps in
 > a back-ported fashion), or will you give a deadline for new patches

  Since the point of a strictly bug-fix release is to fix bugs, I
would not tie the 2.1.2 release to 2.2 in any way.  The 2.2
developments remain a cheap source of bug fixes, though, so anything
that was an actual bug fix should be considered for 2.1.2.  If an
alternate fix is needed because the codebase changed too much (which
may be very possible this time), a completely new patch may be
needed.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Fri Oct 19 14:59:30 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 19 Oct 2001 09:59:30 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Your message of "Fri, 19 Oct 2001 14:31:37 +1000."
 <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>
References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>
Message-ID: <200110191359.JAA26790@cj20424-a.reston1.va.home.com>

Thanks, Anthony!  It's good to see someone volunteering without too
much pressure.  What's your SF id, so we can give you checkin
permission?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Fri Oct 19 15:13:59 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 19 Oct 2001 10:13:59 -0400
Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately?
References: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU>
 <200110191229.IAA26121@cj20424-a.reston1.va.home.com>
Message-ID: <15312.13607.821357.720224@anthem.wooz.org>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    >> For the last couple of days, socketmodule hasn't compiled for
    >> me on Tru64 Unix. If I'm the only one seeing this, I'll file a
    >> bug report on sf...

    GvR> It's probably a missing \n\ in a string literal.  I've fixed
    GvR> this on the release branch now.

Thanks Guido.
-Barry


From fdrake@acm.org  Fri Oct 19 15:14:59 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 19 Oct 2001 10:14:59 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110191359.JAA26790@cj20424-a.reston1.va.home.com>
References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>
 <200110191359.JAA26790@cj20424-a.reston1.va.home.com>
Message-ID: <15312.13667.946275.569788@grendel.zope.com>

Guido van Rossum writes:
 > Thanks, Anthony!  It's good to see someone volunteering without too
 > much pressure.  What's your SF id, so we can give you checkin
 > permission?

  "anthonybaxter".  I've added him and set up permissions.
  Anthony, if I've left out any permissions, speak up and we'll get
them fixed.  Thanks for helping out!


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From skip@pobox.com (Skip Montanaro)  Fri Oct 19 16:47:48 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 19 Oct 2001 10:47:48 -0500
Subject: [Python-Dev] should I check in this test_xmlrpc test case change?
Message-ID: <15312.19236.314775.530443@beluga.mojam.com>

When I made the latest change to xmlrpclib.py (to test for integer
overflow), I also expanded the test_xmlrpc.py file.  Unfortunately, I forgot
to check it in.  Should I do that now or file it as a separate patch?

Skip


From guido@python.org  Fri Oct 19 16:50:13 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 19 Oct 2001 11:50:13 -0400
Subject: [Python-Dev] should I check in this test_xmlrpc test case change?
In-Reply-To: Your message of "Fri, 19 Oct 2001 10:47:48 CDT."
 <15312.19236.314775.530443@beluga.mojam.com>
References: <15312.19236.314775.530443@beluga.mojam.com>
Message-ID: <200110191550.LAA30349@cj20424-a.reston1.va.home.com>

> When I made the latest change to xmlrpclib.py (to test for integer
> overflow), I also expanded the test_xmlrpc.py file.  Unfortunately, I forgot
> to check it in.  Should I do that now or file it as a separate patch?

Assuming that it works for you :-), check it in to the trunk.  It won't
make it into 2.2b1, but it will be ready for 2.2b2.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Fri Oct 19 16:54:29 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 19 Oct 2001 11:54:29 -0400
Subject: [Python-Dev] should I check in this test_xmlrpc test case change?
References: <15312.19236.314775.530443@beluga.mojam.com>
Message-ID: <15312.19637.671407.555401@anthem.wooz.org>

>>>>> "SM" == Skip Montanaro <skip@pobox.com> writes:

    SM> When I made the latest change to xmlrpclib.py (to test for
    SM> integer overflow), I also expanded the test_xmlrpc.py file.
    SM> Unfortunately, I forgot to check it in.  Should I do that now
    SM> or file it as a separate patch?

You can check it into the trunk, but it's too late for anything else
to make it in 2.2b1 now.

-Barry


From Martin.v.Loewis@t-online.de  Fri Oct 19 16:56:00 2001
From: Martin.v.Loewis@t-online.de (Martin v. Loewis)
Date: Fri, 19 Oct 2001 17:56:00 +0200
Subject: [Python-Dev] should I check in this test_xmlrpc test case change?
Message-ID: <200110191556.f9JFu0J01595@mira.informatik.hu-berlin.de>

> Should I do that now or file it as a separate patch?

Please check it in now, so it appears in 2.2b2.

Thanks,
Martin


From skip@pobox.com (Skip Montanaro)  Fri Oct 19 17:02:27 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Fri, 19 Oct 2001 11:02:27 -0500
Subject: [Python-Dev] should I check in this test_xmlrpc test case change?
In-Reply-To: <15312.19637.671407.555401@anthem.wooz.org>
References: <15312.19236.314775.530443@beluga.mojam.com>
 <15312.19637.671407.555401@anthem.wooz.org>
Message-ID: <15312.20115.92879.794630@beluga.mojam.com>

    BAW> You can check it into the trunk, but it's too late for anything
    BAW> else to make it in 2.2b1 now.

Thanks, will do.  I didn't care about making 2.2b1 (it is just a test case,
after all).  I just didn't want to botch something up related to the
mechanics of the release.

Skip



From barry@zope.com  Fri Oct 19 17:07:25 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 19 Oct 2001 12:07:25 -0400
Subject: [Python-Dev] should I check in this test_xmlrpc test case change?
References: <15312.19236.314775.530443@beluga.mojam.com>
 <15312.19637.671407.555401@anthem.wooz.org>
 <15312.20115.92879.794630@beluga.mojam.com>
Message-ID: <15312.20413.437682.308810@anthem.wooz.org>

>>>>> "SM" == Skip Montanaro <skip@pobox.com> writes:

    BAW> You can check it into the trunk, but it's too late for
    BAW> anything else to make it in 2.2b1 now.

    SM> Thanks, will do.  I didn't care about making 2.2b1 (it is just
    SM> a test case, after all).  I just didn't want to botch
    SM> something up related to the mechanics of the release.

The trunk is always safe to make commits to.  That's why we release
from a branch.

-Barry



From barry@zope.com  Fri Oct 19 18:00:26 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 19 Oct 2001 13:00:26 -0400
Subject: [Python-Dev] RELEASED: Python 2.2b1 is out!
Message-ID: <15312.23594.31057.586571@anthem.wooz.org>

We've released Python 2.2b1, the first beta release of Python 2.2, for
your ebullience, ecdysis, and embarcation.  As this is the first beta
release, we are now in feature freeze for Python 2.2.

    http://www.python.org/2.2/

Give it a good try, and report what breaks to the bug tracker:

  http://sourceforge.net/bugs/?group_id=5470

New features in this release include:

- New-style classes are now always dynamic, except for built-in and
  extension types

- A very subtle syntactical pitfall in list comprehensions was fixed

- getattr(obj, name, default) now only catches AttributeError

- Weak reference objects are now part of the core and offers a C API.

- unicode(obj) now behaves more like str(obj)

- isinstance() now accepts more objects as its first and second
  arguments

- os and posix supports chroot(), setgroups() and unsetenv() where
  available.  The stat(), fstat(), statvfs() and fstatvfs() functions
  now return "pseudo-sequences" whose elements can be accessed by name
  or index.  The time module's localtime(), gmtime() and strptime()
  are pseudo-sequences too.

- Many improvements to the profiler

As usual, Andrew Kuchling is writing a gentle introduction to the most
important changes (currently excluding type/class unification), titled
"What's New in Python 2.2":

  http://www.amk.ca/python/2.2/

There is an introduction to the type/class unification at:

  http://www.python.org/2.2/descrintro.html

Thanks to everybody who contributed to this release, including all the
alpha testers!

brought-to-you-by-the-letter-"E"-ly y'rs,
-Barry


From fdrake@acm.org  Fri Oct 19 21:46:13 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 19 Oct 2001 16:46:13 -0400
Subject: [Python-Dev] Python documentation for iSilo
Message-ID: <15312.37141.139330.26322@grendel.zope.com>

  I've re-worked the iSilo version of the documenation for the first
beta of Python 2.2; each manual is now a separate document for iSilo,
so you can pick & choose the documents you need.  They are also
smaller than the first test version (by about 1/3 of a megabyte),
which is a nice savings.  This version includes a few post-beta
documentation fixes.
  To try it out, download the file

	isilo-2.2b1+.zip

from:

	ftp://ftp.python.org/pub/python/doc/2.2b1/

  Please send comments on this version of the docs to me at:

	python-docs@python.org

  Enjoy!


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From fdrake@acm.org  Fri Oct 19 21:59:04 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Fri, 19 Oct 2001 16:59:04 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011019205904.1F95528697@beowolf.digicool.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Documentation update beyond the Python 2.2 beta 1 docs.
This matches the revised iSilo document conversion.



From barry.alan.scott@ntlworld.com  Fri Oct 19 23:54:28 2001
From: barry.alan.scott@ntlworld.com (Barry Scott)
Date: Fri, 19 Oct 2001 23:54:28 +0100
Subject: [Python-Dev] Imports very slow on one NT4 system - any idea what to check for
Message-ID: <000d01c158f1$020e9fc0$060210ac@private>

I helping a friend get up to speed with Python. He has installed the
ActiveState 2.1.1 on a number of machines. All run nice and fast except
for one machine. It is one of group of identical desktop Dual 866MHz PIII
systems with 1G RAM, SCSI hard drive running Windows NT4.0/SP6a.

Importing getopt, re, sys, string and os takes 7 seconds. Using python -v
just shows a uniform slowness, all files are slow. python -vv he tells me
did not show a particular slow spot.

His laptop Crusoe 700MHz machine, 192MB, IDE hard drive runs the same
imports in 1.1 seconds.

I've asked that he checks the python -v output to make sure that the .pyc
files are being loaded and not the .py.

Any idea what else to check for?

		Barry



From davida@ActiveState.com  Sat Oct 20 00:15:28 2001
From: davida@ActiveState.com (David Ascher)
Date: Fri, 19 Oct 2001 16:15:28 -0700 (PDT)
Subject: [Python-Dev] Imports very slow on one NT4 system - any idea what
 to check for
In-Reply-To: <000d01c158f1$020e9fc0$060210ac@private>
Message-ID: <Pine.LNX.4.30.0110191614320.19978-100000@latte.ActiveState.com>

On Fri, 19 Oct 2001, Barry Scott wrote:

> I helping a friend get up to speed with Python. He has installed the
> ActiveState 2.1.1 on a number of machines. All run nice and fast except
> for one machine. It is one of group of identical desktop Dual 866MHz PIII
> systems with 1G RAM, SCSI hard drive running Windows NT4.0/SP6a.
>
> Importing getopt, re, sys, string and os takes 7 seconds. Using python -v
> just shows a uniform slowness, all files are slow. python -vv he tells me
> did not show a particular slow spot.
>
> His laptop Crusoe 700MHz machine, 192MB, IDE hard drive runs the same
> imports in 1.1 seconds.
>
> I've asked that he checks the python -v output to make sure that the .pyc
> files are being loaded and not the .py.
>
> Any idea what else to check for?

This is probably not a good topic for python-dev.  However, before I
suggest that you take it up on python-list or activepython-feedback, let
me suggest you check for virus-checking software.  Some versions of Norton
AntiVirus slow down file opens drastically.

-- David Ascher



From Anthony Baxter <anthony@interlink.com.au>  Sun Oct 21 03:00:48 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Sun, 21 Oct 2001 12:00:48 +1000
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Message from "Fred L. Drake, Jr." <fdrake@acm.org>
 of "Fri, 19 Oct 2001 09:52:58 -0400." <15312.12346.482959.64272@grendel.zope.com>
Message-ID: <200110210201.f9L20mX29206@mbuna.arbhome.com.au>

>>> "Fred L. Drake, Jr." wrote
>   Since the point of a strictly bug-fix release is to fix bugs, I
> would not tie the 2.1.2 release to 2.2 in any way.  The 2.2
> developments remain a cheap source of bug fixes, though, so anything
> that was an actual bug fix should be considered for 2.1.2.  If an
> alternate fix is needed because the codebase changed too much (which
> may be very possible this time), a completely new patch may be
> needed.

I agree  - there's also the possibility that there will be additional 
patches needed to fix a bug in code that's just not relevant for 2.2 
(this is less likely), or an alternate fix might be needed that's
not as thorough as one in 2.2 (where there's a large fix, a smaller 
workaround might be more appropriate for 2.1.2)


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.



From Anthony Baxter <anthony@interlink.com.au>  Sun Oct 21 06:47:18 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Sun, 21 Oct 2001 15:47:18 +1000
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Message from Guido van Rossum <guido@python.org>
 of "Fri, 19 Oct 2001 09:59:30 -0400." <200110191359.JAA26790@cj20424-a.reston1.va.home.com>
Message-ID: <200110210547.f9L5lIP30307@mbuna.arbhome.com.au>

Hm. What's the general consensus on portability fixes w.r.t. 2.1.2? 
I'm leaning toward "yes", but my concern is that they stand a chance 
of causing some other weird freakshow platform to fail to compile,
where it previously worked... obviously I don't mean HP/UX threading 
here, as I think it's a lost cause :) 

But on the other hand, if it helps a platform build, where it previously
did not... Hm hm hm.

Anthony


From tim.one@home.com  Sun Oct 21 07:33:39 2001
From: tim.one@home.com (Tim Peters)
Date: Sun, 21 Oct 2001 02:33:39 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110210547.f9L5lIP30307@mbuna.arbhome.com.au>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEDOMCAA.tim.one@home.com>

[Anthony Baxter]
> Hm. What's the general consensus on portability fixes w.r.t. 2.1.2?

If it's for a platform where Python used to run OK, it's probably "a bug" if
2.1.2 no longer runs on it (I say "probably" because you really don't want
all the changes it would take to get Python working correctly under, e.g.,
WinCE again ...).

If it's for a platform where Python has never run, then running on that
platform is a new feature, and I wouldn't touch that in a bugfix release
even if you stood between us and blocked most of the exploding guts <wink>.



From aahz@rahul.net  Sun Oct 21 15:35:48 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Sun, 21 Oct 2001 07:35:48 -0700 (PDT)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> from "Anthony Baxter" at Oct 19, 2001 02:31:37 PM
Message-ID: <20011021143548.D66E7E8DF@waltz.rahul.net>

Fred, could you update PEP 6 to make Anthony the 2.1.2 manager, or
should I submit a patch to you?  (Yeah, yeah, one of these days I'll
actually do enough work to make it worth SF privileges.)

Anthony Baxter wrote:
> 
> Here's my current thoughts for selecting patches for a 2.1.2 release.
> Feedback would be appreciated. I plan to post this to python-list and
> ask people for suggestions for "must have" fixes. This could possibly
> be added to PEP 006 once it's nailed down.

I have zero objection to any single release using stricter guidelines
than PEP 6 requires, but PEP 6 started life as a set of guidelines for
both "bugfix" and "patch" releases.  While I don't have a lot of
objection to tightening up PEP 6 a bit more to reflect the fact that it
focuses entirely on bugfix releases now, there are enough gray areas
about what is a bug vs. feature that I don't want to make the PEP quite
as strict as you plan to for 2.1.2.

I don't want to get into a long discussion about this on python-dev;
I'll respond to your post on c.l.py.  I just wanted to make a short
statement of principle in case anyone wants to get involved later.
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From BrainFletcher90@hotmail.com  Sun Oct 21 17:27:57 2001
From: BrainFletcher90@hotmail.com (Brian Fletcher)
Date: Sun, 21 Oct 2001 17:27:57 +0100
Subject: [Python-Dev] Fwd: Play a hilarious prank call
Message-ID: <E15vLQQ-0004gI-00@mail.python.org>

Computerized prank call service. Wind-up your mates! Click on this link.
www.pranks.gq.nu (Works on any phone in the UK including mobiles) Or call 020
8387 8754 (24hrs)


From tim.one@home.com  Sun Oct 21 19:25:58 2001
From: tim.one@home.com (Tim Peters)
Date: Sun, 21 Oct 2001 14:25:58 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <20011021143548.D66E7E8DF@waltz.rahul.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEELMCAA.tim.one@home.com>

[Aahz]
> Fred, could you update PEP 6 to make Anthony the 2.1.2 manager,

Done.

i-am-my-fred's-keeper-ly y'rs  - tim



From guido@python.org  Mon Oct 22 02:23:06 2001
From: guido@python.org (Guido van Rossum)
Date: Sun, 21 Oct 2001 21:23:06 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Your message of "Sun, 21 Oct 2001 15:47:18 +1000."
 <200110210547.f9L5lIP30307@mbuna.arbhome.com.au>
References: <200110210547.f9L5lIP30307@mbuna.arbhome.com.au>
Message-ID: <200110220123.VAA18017@cj20424-a.reston1.va.home.com>

> obviously I don't mean HP/UX threading 
> here, as I think it's a lost cause :) 

Actually, reportedly 2.2b1 builds with threads and runs the thread
tests correctly on HP-UX 11.00.  See 

http://sf.net/tracker/?func=detail&atid=105470&aid=473150&group_id=5470

It's not clear what made this work though, so the backport may be
muddy.  You might try to enlist the help of the guy who reported that
bug, he seems knowledgeable.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 22 02:33:38 2001
From: guido@python.org (Guido van Rossum)
Date: Sun, 21 Oct 2001 21:33:38 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Your message of "Sun, 21 Oct 2001 12:00:48 +1000."
 <200110210201.f9L20mX29206@mbuna.arbhome.com.au>
References: <200110210201.f9L20mX29206@mbuna.arbhome.com.au>
Message-ID: <200110220133.VAA19032@cj20424-a.reston1.va.home.com>

> I agree  - there's also the possibility that there will be additional 
> patches needed to fix a bug in code that's just not relevant for 2.2 
> (this is less likely), or an alternate fix might be needed that's
> not as thorough as one in 2.2 (where there's a large fix, a smaller 
> workaround might be more appropriate for 2.1.2)

Yup -- this was the case for the profile.py patch (which took a life
of its own in 2.2 :-).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Mon Oct 22 06:36:38 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 22 Oct 2001 01:36:38 -0400
Subject: [Python-Dev] forwarded message from Guido van Rossum
Message-ID: <15315.45158.289660.559852@anthem.wooz.org>

With a cvs update, test_email.py fails because the behavior of
re.split() has changed, presumably due to /F's latest sre checkins.
Here's a boiled down example:

-------------------- snip snip --------------------
import re

ecre = re.compile(r'''
  =\?                   # literal =?
  (?P<charset>[^?]*?)   # non-greedy up to the next ? is the charset
  \?                    # literal ?
  (?P<encoding>[qb])    # either a "q" or a "b", case insensitive
  \?                    # literal ?
  (?P<atom>.*?)         # non-greedy up to the next ?= is the atom
  \?=                   # literal ?=
  ''', re.VERBOSE | re.IGNORECASE)

print ecre.split('=?iso-8859-1?q?this=20is=20some=20text?=', 1)
-------------------- snip snip --------------------

In Python 2.2b1 this prints

['', 'iso-8859-1', 'q', 'this=20is=20some=20text', '']

but in today's cvs this prints

['', 'iso-8859-1', 'q', 'this=20is=20some=20text']

Notice the missing trailing empty string.
:(

-Barry


From tim.one@home.com  Mon Oct 22 07:36:04 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 22 Oct 2001 02:36:04 -0400
Subject: [Python-Dev] forwarded message from Guido van Rossum
In-Reply-To: <15315.45158.289660.559852@anthem.wooz.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEGCMCAA.tim.one@home.com>

[Barry]
> With a cvs update, test_email.py fails because the behavior of
> re.split() has changed, presumably due to /F's latest sre checkins.

I believe /F has since fixed this (and test_email.py works for me again).



From barry@zope.com  Mon Oct 22 07:35:46 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 22 Oct 2001 02:35:46 -0400
Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Modules _sre.c,2.71,2.72
References: <E15vY9u-0003L6-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <15315.48706.655213.385913@anthem.wooz.org>

>>>>> "FL" == Fredrik Lundh <effbot@users.sourceforge.net> writes:

    FL> sre.split should return the last segment, even if empty
    FL> (sorry, barry)

Thank you effbot!


From loewis@informatik.hu-berlin.de  Mon Oct 22 09:02:35 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Mon, 22 Oct 2001 10:02:35 +0200 (MEST)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110210201.f9L20mX29206@mbuna.arbhome.com.au> (message from
 Anthony Baxter on Sun, 21 Oct 2001 12:00:48 +1000)
References: <200110210201.f9L20mX29206@mbuna.arbhome.com.au>
Message-ID: <200110220802.KAA27251@paros.informatik.hu-berlin.de>

> I agree  - there's also the possibility that there will be additional 
> patches needed to fix a bug in code that's just not relevant for 2.2 
> (this is less likely), or an alternate fix might be needed that's
> not as thorough as one in 2.2 (where there's a large fix, a smaller 
> workaround might be more appropriate for 2.1.2)

Then I come back to my original question: What is your schedule for
2.1.2 then? I don't mind if it is either very tight or very lose, but
I think there should be a schedule (which can be adjusted as
needed). I think the following milestones should appear:
- deadline for contributing new patches
- deadline for selecting and integrating CVS mainline patches
- release of a release candidate
- release

IMO, it isn't important that *all* patches that are desirable for
2.1.2 really get integrated by the second deadline. Instead, what gets
done is done, everything else is for 2.1.3 (if there is interest in
such a release). E.g. I wouldn't delay the release indefinitely just
because somebody promised to indicate all, say, pyexpat patches, and
never did.

Regards,
Martin

P.S. Although I certainly do promise to indicate all pyexpat patches
that should go into 2.1.2 :-)


From Anthony Baxter <anthony@interlink.com.au>  Mon Oct 22 15:20:34 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Tue, 23 Oct 2001 00:20:34 +1000
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Message from Martin von Loewis <loewis@informatik.hu-berlin.de>
 of "Mon, 22 Oct 2001 10:02:35 +0200." <200110220802.KAA27251@paros.informatik.hu-berlin.de>
Message-ID: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au>

>>> Martin von Loewis wrote
> Then I come back to my original question: What is your schedule for
> 2.1.2 then? 

Darn. Was hoping you'd missed that I'd put that question off :)
At this point, I'm still trawling CVS - I hope to have an estimate
in a day or so (work stuff permitting), or the end of the week at
the latest. My gut feel right now is that it's a couple of weeks
to finish trawling the CVS logs (geez you guys commit a lot of
changes) and checking them in carefully).

> I don't mind if it is either very tight or very lose, but
> I think there should be a schedule (which can be adjusted as
> needed). I think the following milestones should appear:
> - deadline for contributing new patches
> - deadline for selecting and integrating CVS mainline patches
> - release of a release candidate
> - release

Sounds like a good set of milestones. As I said, though, right now
I'm just not sure.

> IMO, it isn't important that *all* patches that are desirable for
> 2.1.2 really get integrated by the second deadline. Instead, what gets

Nope. I'd like to be able to say, though, that "for this release, 
there are some known bugs in pyexpat/the floating point overflow/
whatever" - at least let people know.


> P.S. Although I certainly do promise to indicate all pyexpat patches
> that should go into 2.1.2 :-)

I'm glad to hear it :)

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From loewis@informatik.hu-berlin.de  Mon Oct 22 15:36:04 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Mon, 22 Oct 2001 16:36:04 +0200 (MEST)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au> (message from
 Anthony Baxter on Tue, 23 Oct 2001 00:20:34 +1000)
References: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au>
Message-ID: <200110221436.QAA27751@paros.informatik.hu-berlin.de>

> Darn. Was hoping you'd missed that I'd put that question off :)
> At this point, I'm still trawling CVS - I hope to have an estimate
> in a day or so (work stuff permitting), or the end of the week at
> the latest. My gut feel right now is that it's a couple of weeks
> to finish trawling the CVS logs (geez you guys commit a lot of
> changes) and checking them in carefully).

Remember you are Patch Czar, not the slave of patches. You could
restrict yourself to incorporate only patches that people propose.

If you want systematic scanning, we could try to come up with a script
that automatically reviews all commit messages of patches since 2.1.1
whether these mention an SF bug. We could then further automate
scanning to produce the number of changed lines, to exclude 'large'
changes.

Perhaps anybody in the readership of this software already has code
that does such things?

Regards,
Martin


From Anthony Baxter <anthony@interlink.com.au>  Mon Oct 22 15:39:25 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Tue, 23 Oct 2001 00:39:25 +1000
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Message from Martin von Loewis <loewis@informatik.hu-berlin.de>
 of "Mon, 22 Oct 2001 16:36:04 +0200." <200110221436.QAA27751@paros.informatik.hu-berlin.de>
Message-ID: <200110221439.f9MEdP105195@mbuna.arbhome.com.au>

> Remember you are Patch Czar, not the slave of patches. You could
> restrict yourself to incorporate only patches that people propose.

Heh. No, the problem at this point as far as making estimates go is
even knowing a vague idea of how much there is to do.

> If you want systematic scanning, we could try to come up with a script
> that automatically reviews all commit messages of patches since 2.1.1
> whether these mention an SF bug. We could then further automate
> scanning to produce the number of changed lines, to exclude 'large'
> changes.

Hrm. That might be an idea - at the moment I've got the trunk and the 
2.1 branch checked out in parallel directories, and have a teeny zsh
alias to do diffs and look at the tree. I'm then checking the commit 
logs since the 2.1 branch to see the diffs.

> Perhaps anybody in the readership of this software already has code
> that does such things?

We can hope... 


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.



From guido@python.org  Mon Oct 22 15:41:16 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 22 Oct 2001 10:41:16 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Your message of "Mon, 22 Oct 2001 16:36:04 +0200."
 <200110221436.QAA27751@paros.informatik.hu-berlin.de>
References: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au>
 <200110221436.QAA27751@paros.informatik.hu-berlin.de>
Message-ID: <200110221441.KAA30687@cj20424-a.reston1.va.home.com>

> If you want systematic scanning, we could try to come up with a script
> that automatically reviews all commit messages of patches since 2.1.1
> whether these mention an SF bug. We could then further automate
> scanning to produce the number of changed lines, to exclude 'large'
> changes.

Slightly dangerous though -- some bugfixes were never reported to SF.

> Perhaps anybody in the readership of this software already has code
> that does such things?

No, but you can start with Tools/scripts/logmerge.py which parses CVS
log output and sorts it by date.

The best way to scan for SF patch refs is probably to search for
6-digit numbers...  There's too much diversity otherwise in the SF
references :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)


From aahz@rahul.net  Mon Oct 22 16:41:10 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Mon, 22 Oct 2001 08:41:10 -0700 (PDT)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110221436.QAA27751@paros.informatik.hu-berlin.de> from "Martin von Loewis" at Oct 22, 2001 04:36:04 PM
Message-ID: <20011022154110.6E372E8C1@waltz.rahul.net>

Martin von Loewis wrote:
> 
> Remember you are Patch Czar, not the slave of patches. You could
> restrict yourself to incorporate only patches that people propose.
> 
> If you want systematic scanning, we could try to come up with a script
> that automatically reviews all commit messages of patches since 2.1.1
> whether these mention an SF bug. We could then further automate
> scanning to produce the number of changed lines, to exclude 'large'
> changes.

As it says in PEP 6, one thing I was hoping for was that people
submitting patches would indicate whether a patch ought to be considered
for inclusion in patch releases for prior formal releases.  One way to
do that would be for people to check in patches to the prior version's
trunk, and then the Patch Czar could decide which patches belong in a
release branch.  Of course, that's assuming I understand how CVS works,
which is probably false.  ;-)

Alternatively, we could agree on some text tag format for use in SF
comments.
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From aahz@rahul.net  Mon Oct 22 16:47:00 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Mon, 22 Oct 2001 08:47:00 -0700 (PDT)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110220802.KAA27251@paros.informatik.hu-berlin.de> from "Martin von Loewis" at Oct 22, 2001 10:02:35 AM
Message-ID: <20011022154700.A0C65E8C1@waltz.rahul.net>

Martin von Loewis wrote:
> 
> I think the following milestones should appear:
> - deadline for contributing new patches
> - deadline for selecting and integrating CVS mainline patches
> - release of a release candidate
> - release

I can't remember whether it was Tim or Guido, but it has been suggested
that a true bugfix release shouldn't need a release candidate.  Me, I'm
always in favor of release candidates.


[skip this next bit if you don't want an anecdote]


(Back when I knew less about Unix than I do now, I was two weeks into a
new job, dealing with a customer who was complaining about being unable
to install the latest release of our software.  I tracked it down to a
syntax error in the install shell script -- proving that *NOBODY* from
QA had bothered to do an actual test install with the released CD.)
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From barry@zope.com  Mon Oct 22 16:58:14 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 22 Oct 2001 11:58:14 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
References: <200110220802.KAA27251@paros.informatik.hu-berlin.de>
 <20011022154700.A0C65E8C1@waltz.rahul.net>
Message-ID: <15316.16918.889688.286867@anthem.wooz.org>

>>>>> "AM" == Aahz Maruch <aahz@rahul.net> writes:

    AM> I can't remember whether it was Tim or Guido, but it has been
    AM> suggested that a true bugfix release shouldn't need a release
    AM> candidate.  Me, I'm always in favor of release candidates.

Possibly me, because I don't think release candidates on
micro-releases are that important.  If you royally screwed up 2.1.2 it
should be low-cost to turnaround and do another 2.1.3 release very
quickly.  I'd rather opt for quick, low-overhead micro/patch releases.

-Barry


From fdrake@acm.org  Mon Oct 22 18:24:35 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Mon, 22 Oct 2001 13:24:35 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011022172435.28AF528697@cj42289-a.reston1.va.home.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Various updates, including support for the "Site Navigation Bar" in
Mozilla 0.9.5.



From tim.one@home.com  Mon Oct 22 19:08:32 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 22 Oct 2001 14:08:32 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEHLMCAA.tim.one@home.com>

[Anthony Baxter]
> Darn. Was hoping you'd missed that I'd put that question off :)
> ...

Aren't you glad you asked an innocent question about bugfix releases on
c.l.py?  Aren't you glad I replied?  Heh heh -- that's how all Python work
gets done in the end <wink>.

tag-you're-it-ly y'rs  - tim



From tim.one@home.com  Mon Oct 22 21:10:36 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 22 Oct 2001 16:10:36 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <20011022154700.A0C65E8C1@waltz.rahul.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEIHMCAA.tim.one@home.com>

[Aahz]
> ...
> (Back when I knew less about Unix than I do now, I was two weeks into a
> new job, dealing with a customer who was complaining about being unable
> to install the latest release of our software.  I tracked it down to a
> syntax error in the install shell script -- proving that *NOBODY* from
> QA had bothered to do an actual test install with the released CD.)

In my Cray Research days, releases were shipped on magnetic tape.  A day
after one major release, customers started to report that the tapes they got
were blank -- the tape unit write head was in fact utterly dead, but nobody
had bothered to even trying reading the release media, let alone install
from it.

PythonLabs has also shipped damaged tarballs.  The only thing we haven't
done is ship an empty Windows installer <wink>.



From DavidA@ActiveState.com  Mon Oct 22 21:46:50 2001
From: DavidA@ActiveState.com (David Ascher)
Date: 22 Oct 2001 13:46:50 -0700
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEIHMCAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCOEIHMCAA.tim.one@home.com>
Message-ID: <1003783610.23693.44.camel@loom.activestate.com>

On Mon, 2001-10-22 at 13:10, Tim Peters wrote:
> [Aahz]
> > ...
> > (Back when I knew less about Unix than I do now, I was two weeks into a
> > new job, dealing with a customer who was complaining about being unable
> > to install the latest release of our software.  I tracked it down to a
> > syntax error in the install shell script -- proving that *NOBODY* from
> > QA had bothered to do an actual test install with the released CD.)
> 
> In my Cray Research days, releases were shipped on magnetic tape.  A day
> after one major release, customers started to report that the tapes they got
> were blank -- the tape unit write head was in fact utterly dead, but nobody
> had bothered to even trying reading the release media, let alone install
> from it.
> 
> PythonLabs has also shipped damaged tarballs.  The only thing we haven't
> done is ship an empty Windows installer <wink>.

Myself, I'll never admit anything in public. 

--david



From tim.one@home.com  Mon Oct 22 21:56:42 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 22 Oct 2001 16:56:42 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <1003783610.23693.44.camel@loom.activestate.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEILMCAA.tim.one@home.com>

[David Ascher]
> Myself, I'll never admit anything in public.

Of course, but we have to:  it's the difference between working for an Open
Source company and working for Microsoft <wink>.



From DavidA@ActiveState.com  Mon Oct 22 22:02:35 2001
From: DavidA@ActiveState.com (David Ascher)
Date: 22 Oct 2001 14:02:35 -0700
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEILMCAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCAEILMCAA.tim.one@home.com>
Message-ID: <1003784555.23700.52.camel@loom.activestate.com>

On Mon, 2001-10-22 at 13:56, Tim Peters wrote:
> [David Ascher]
> > Myself, I'll never admit anything in public.
> 
> Of course, but we have to:  it's the difference between working for an Open
> Source company and working for Microsoft <wink>.

Yowza!

If I were working for Microsoft, I wouldn't be dealing with GTK+ issues
these days =).

--david



From jeremy@zope.com  Tue Oct 23 01:52:09 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Mon, 22 Oct 2001 20:52:09 -0400 (EDT)
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>
References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>
Message-ID: <15316.48953.976319.490902@slothrop.digicool.com>

>>>>> "AB" == Anthony Baxter <anthony@interlink.com.au> writes:

  AB>   Things that shouldn't be considered:
  [...]
  AB>     Fixes that "remove a nasty workaround"

  AB>   In general, I'd like the starting point for any patch to be
  AB>   considered as "guilty until proven necessary".

Do you have an opinion on the recent change to traceback.c (make it
collectable)?  I think it's a good candidate for 2.1.2 even though it
doesn't meet the criteria above :-).

It removes the nasty workaround that you *never* exit a function with
a local variable that points to a traceback that references the
function.  There's a bit of C code (most of it probably extending the
type structure for tracebacks), but it's simple stuff, thoroughly
tested by the current test suite, and it's a *nasty* problem in
practice. 

Two other things I thought of on this thread:

    We got out of the habit of marking changes in 2.2 as potential
    bug fixes, because we didn't expect to have a 2.1.2 release.  I
    think we should continue marking checkins this way indefinitely.
    We may not have a 2.1.3 and a 2.2.2, but it's pretty easy to mark
    the checkin just in case we do make such a release.

    I'd like to update the compiler package in Tools/compiler, because
    I've fixed a number of nasty bugs, e.g. computing a stack size
    that is too small for a function.  But I don't know if I'll have
    time.  Once you've got a schedule, I'll let you know if I can
    manage it.

Jeremy



From guido@python.org  Tue Oct 23 02:14:46 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 22 Oct 2001 21:14:46 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: Your message of "Mon, 22 Oct 2001 20:52:09 EDT."
 <15316.48953.976319.490902@slothrop.digicool.com>
References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au>
 <15316.48953.976319.490902@slothrop.digicool.com>
Message-ID: <200110230114.VAA03866@cj20424-a.reston1.va.home.com>

>   AB>   In general, I'd like the starting point for any patch to be
>   AB>   considered as "guilty until proven necessary".
> 
> Do you have an opinion on the recent change to traceback.c (make it
> collectable)?  I think it's a good candidate for 2.1.2 even though it
> doesn't meet the criteria above :-).

Hm.  In 2.2, the C API for the garbage collector is different from
2.1, so backporting this would require extra work.  Also, you'd
probably have to add GC to frames as well to reap the benefits.

To me this particular problem feels like a bug that's almost a feature
(everybody knows about it, it's documented, etc.), and I think we'd be
better off not fixing this in 2.1.2.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Tue Oct 23 06:05:59 2001
From: tim.one@home.com (Tim Peters)
Date: Tue, 23 Oct 2001 01:05:59 -0400
Subject: [Python-Dev] Goals for patch selection for 2.1.2
In-Reply-To: <15316.48953.976319.490902@slothrop.digicool.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEKDMCAA.tim.one@home.com>

[Jeremy]
> Do you have an opinion on the recent change to traceback.c (make it
> collectable)?  I think it's a good candidate for 2.1.2 even though it
> doesn't meet the criteria above :-).

Guido's right:  that change alone won't do any good.  Frame objects need to
be chased by gc too, else the cycle would survive.  However, adding frame
objects to gc caused a measurable slowdown, which I judged tolerable in 2.2
because 2.2 got enough non-bugfix speedup work to cancel it out (at the
time -- I expect we're losing again by now).

if-it-ain't-a-pure-self-contained-bugfix-it's-dubious-at-best-ly y'rs  - tim



From akuchlin@mems-exchange.org  Tue Oct 23 14:11:26 2001
From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org)
Date: Tue, 23 Oct 2001 09:11:26 -0400
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
Message-ID: <20011023091126.A24293@mems-exchange.org>

There's a discussion of SourceForge alternatives at
http://www.advogato.org/article/357.html, and in a comment someone
mentions that there's an XML export of the data in the tracker.
Unfortunately I can't get it to work following his instructions, but
perhaps someone else can experiment with it more than I can.

--amk


From barry@zope.com  Tue Oct 23 15:02:34 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Tue, 23 Oct 2001 10:02:34 -0400
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
References: <20011023091126.A24293@mems-exchange.org>
Message-ID: <15317.30842.636138.922307@anthem.wooz.org>

>>>>> "akuchlin" ==   <akuchlin@mems-exchange.org> writes:

    akuchlin> There's a discussion of SourceForge alternatives at
    akuchlin> http://www.advogato.org/article/357.html, and in a
    akuchlin> comment someone mentions that there's an XML export of
    akuchlin> the data in the tracker.  Unfortunately I can't get it
    akuchlin> to work following his instructions, but perhaps someone
    akuchlin> else can experiment with it more than I can.

I can't get it to work either, but if someone comes up with the right
recipe, I can add it to our backup script.

Every night I pull down the CVS repositories for Python, Mailman,
Jython, and mimelib.  I also grab the Jython FAQ and moinmoin.  These
I backup nightly on a Pythonlabs/ZC server.  So I feel fairly
confident about this data.  But I'd love to find a solution for the
tracker data too.

-Barry


From fredrik@pythonware.com  Tue Oct 23 16:35:44 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 23 Oct 2001 17:35:44 +0200
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
References: <20011023091126.A24293@mems-exchange.org> <15317.30842.636138.922307@anthem.wooz.org>
Message-ID: <003b01c15bd8$6cffede0$ced241d5@hagrid>

barry wrote:

>     akuchlin> There's a discussion of SourceForge alternatives at
>     akuchlin> http://www.advogato.org/article/357.html, and in a
>     akuchlin> comment someone mentions that there's an XML export of
>     akuchlin> the data in the tracker.  Unfortunately I can't get it
>     akuchlin> to work following his instructions, but perhaps someone
>     akuchlin> else can experiment with it more than I can.
>
> I can't get it to work either, but if someone comes up with the right
> recipe, I can add it to our backup script.

works fine from MSIE, using this URL:

url = (
    "http://sourceforge.net/export/sf_tracker_export.php" +
    "?atid=105470&group_id=5470"
    )

returns a 1666445 byte large document (with no content
type), starting with:

<tracker version="1.0" xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://sourceforge.net/export/sf_tracker_export.xsd">
	<artifact id="473985">
		<submitted_by>doerwalter</submitted_by>
		<submitted_date>2001-10-23 03:39</submitted_date>
		<artifact_type id="105470">105470</artifact_type>
		<category id="347131">Type/class unification</category>
		<artifact_group id="136128"></artifact_group>
		<assigned_to>Guido van Rossum</assigned_to>
		<priority id="5">5</priority>
		<status>Open</status>
		<resolution>Remind</resolution>
		<summary>str, __getitem__ and slices</summary>
		<detail>Using slices with __getitem__ doesn't work with

(this returns the bugs database.  see
http://sourceforge.net/tracker/?group_id=5470
for atid's for other parts of the tracker)

doing the same through urllib doesn't appear work, though.

(no time to dig further into this right now)

</F>



From aahz@rahul.net  Tue Oct 23 16:35:04 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Tue, 23 Oct 2001 08:35:04 -0700 (PDT)
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
In-Reply-To: <003b01c15bd8$6cffede0$ced241d5@hagrid> from "Fredrik Lundh" at Oct 23, 2001 05:35:44 PM
Message-ID: <20011023153504.D66C6E8C5@waltz.rahul.net>

Fredrik Lundh wrote:
> 
> works fine from MSIE, using this URL:
>    [....]
> doing the same through urllib doesn't appear work, though.

Well, that's a good reason for leaving SF.  :-/
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From loewis@informatik.hu-berlin.de  Tue Oct 23 18:25:30 2001
From: loewis@informatik.hu-berlin.de (Martin von Loewis)
Date: Tue, 23 Oct 2001 19:25:30 +0200 (MEST)
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
Message-ID: <200110231725.TAA11514@paros.informatik.hu-berlin.de>

> doing the same through urllib doesn't appear work, though.

Nor does it work with Konqueror or Netscape, see

http://sourceforge.net/tracker/?func=detail&aid=469987&group_id=1&atid=100001

It's good to know it works with IE, though.

Regards,
Martin


From mal@lemburg.com  Tue Oct 23 19:28:55 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 23 Oct 2001 20:28:55 +0200
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
References: <20011023091126.A24293@mems-exchange.org> <15317.30842.636138.922307@anthem.wooz.org> <003b01c15bd8$6cffede0$ced241d5@hagrid>
Message-ID: <3BD5B6E7.CF511C1@lemburg.com>

Fredrik Lundh wrote:
> 
> barry wrote:
> 
> >     akuchlin> There's a discussion of SourceForge alternatives at
> >     akuchlin> http://www.advogato.org/article/357.html, and in a
> >     akuchlin> comment someone mentions that there's an XML export of
> >     akuchlin> the data in the tracker.  Unfortunately I can't get it
> >     akuchlin> to work following his instructions, but perhaps someone
> >     akuchlin> else can experiment with it more than I can.
> >
> > I can't get it to work either, but if someone comes up with the right
> > recipe, I can add it to our backup script.
> 
> works fine from MSIE, using this URL:
> 
> url = (
>     "http://sourceforge.net/export/sf_tracker_export.php" +
>     "?atid=105470&group_id=5470"
>     )
> 
> returns a 1666445 byte large document (with no content
> type), starting with:
> ...
> (this returns the bugs database.  see
> http://sourceforge.net/tracker/?group_id=5470
> for atid's for other parts of the tracker)
> 
> doing the same through urllib doesn't appear work, though.
> 
> (no time to dig further into this right now)

FYI, It seems that you have to be logged into SF in order for the 
download to work. Netscape does the download too if you create a
web page and then use the right-click-download feature.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From akuchlin@mems-exchange.org  Tue Oct 23 21:13:29 2001
From: akuchlin@mems-exchange.org (Andrew Kuchling)
Date: 23 Oct 2001 20:13:29 -0000
Subject: [Python-Dev] Dictionary subclasses and exec
Message-ID: <20011023201329.27143.qmail@mozart.cnri.reston.va.us>

The following bit of code defines a dictionary that folds all its keys
to lowercase before inserting them, but 'exec "Name = 1" in
LowerCaseDict()' doesn't seem to call my __getitem__(), as the listed
keys include 'Name'.  Should this be expected to work?

--amk

class LowerCaseDict(dictionary):
    def _fold_key (self, key):
        if not isinstance(key, str):
            raise TypeError, "All keys must be strings"
        return key.lower()
    
    def __getitem__ (self, key):
        key = self._fold_key(key)
        return dictionary.__getitem__(self, key)

    def __setitem__ (self, key, value):
        key = self._fold_key(key)
        dictionary.__setitem__(self, key, value)

    def __delitem__ (self, key):
        key = self._fold_key(key)
        dictionary.__delitem__(self, key, value)

d = LowerCaseDict()
exec 'Name = 1' in d
print d.keys()


From guido@python.org  Tue Oct 23 22:08:15 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 23 Oct 2001 17:08:15 -0400
Subject: [Python-Dev] Dictionary subclasses and exec
In-Reply-To: Your message of "23 Oct 2001 20:13:29 -0000."
 <20011023201329.27143.qmail@mozart.cnri.reston.va.us>
References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us>
Message-ID: <200110232108.f9NL8Fi25565@odiug.zope.com>

> The following bit of code defines a dictionary that folds all its keys
> to lowercase before inserting them, but 'exec "Name = 1" in
> LowerCaseDict()' doesn't seem to call my __getitem__(), as the listed
> keys include 'Name'.  Should this be expected to work?
> 
> --amk
> 
> class LowerCaseDict(dictionary):
>     def _fold_key (self, key):
>         if not isinstance(key, str):
>             raise TypeError, "All keys must be strings"
>         return key.lower()
>     
>     def __getitem__ (self, key):
>         key = self._fold_key(key)
>         return dictionary.__getitem__(self, key)
> 
>     def __setitem__ (self, key, value):
>         key = self._fold_key(key)
>         dictionary.__setitem__(self, key, value)
> 
>     def __delitem__ (self, key):
>         key = self._fold_key(key)
>         dictionary.__delitem__(self, key, value)
> 
> d = LowerCaseDict()
> exec 'Name = 1' in d
> print d.keys()

Alas, this is one of the things that don't work yet.  To set and get
local variables, exec uses lower-level APIs (PyDict_SetItem and
PyDict_GetItem) that you can't override.  I've thought about what it
would take to make this work as desired, but I haven't found a way yet
that wouldn't (a) slow down the normal case, or (b) create subtle
reference count bugs.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From akuchlin@mems-exchange.org  Tue Oct 23 23:17:37 2001
From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org)
Date: Tue, 23 Oct 2001 18:17:37 -0400
Subject: [Python-Dev] Dictionary subclasses and exec
In-Reply-To: <200110232108.f9NL8Fi25565@odiug.zope.com>; from guido@python.org on Tue, Oct 23, 2001 at 05:08:15PM -0400
References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com>
Message-ID: <20011023181737.B27568@mems-exchange.org>

On Tue, Oct 23, 2001 at 05:08:15PM -0400, Guido van Rossum wrote:
>Alas, this is one of the things that don't work yet.  To set and get

Oh, well; I need to come up with a different example for subclassing a
built-in type, then.  Thanks!   <puts on thinking cap>

--amk



From guido@python.org  Wed Oct 24 00:58:27 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 23 Oct 2001 19:58:27 -0400
Subject: [Python-Dev] Dictionary subclasses and exec
In-Reply-To: Your message of "Tue, 23 Oct 2001 18:17:37 EDT."
 <20011023181737.B27568@mems-exchange.org>
References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com>
 <20011023181737.B27568@mems-exchange.org>
Message-ID: <200110232358.TAA06401@cj20424-a.reston1.va.home.com>

> >Alas, this is one of the things that don't work yet.  To set and get
> 
> Oh, well; I need to come up with a different example for subclassing a
> built-in type, then.  Thanks!   <puts on thinking cap>

You can use a similar example but not override __getitem__ /
__setitem__; instead, add some new methods (e.g. merge() -- like
update() but the existing key wins).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Wed Oct 24 01:00:44 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 23 Oct 2001 20:00:44 -0400
Subject: [Python-Dev] patchlevel.h
In-Reply-To: Your message of "Tue, 23 Oct 2001 15:21:00 PDT."
 <E15w9uu-0004HJ-00@usw-pr-cvs1.sourceforge.net>
References: <E15w9uu-0004HJ-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <200110240000.UAA06439@cj20424-a.reston1.va.home.com>

Jack, you can't do this on the trunk.  If you *have* to have a
different patch level, please use a branch!

--Guido van Rossum (home page: http://www.python.org/~guido/)

> Update of /cvsroot/python/python/dist/src/Include
> In directory usw-pr-cvs1:/tmp/cvs-serv16428/Python/Include
> 
> Modified Files:
> 	patchlevel.h 
> Log Message:
> Tweaks for MacPython 2.2b1
> 
> Index: patchlevel.h
> ===================================================================
> RCS file: /cvsroot/python/python/dist/src/Include/patchlevel.h,v
> retrieving revision 2.56
> retrieving revision 2.57
> diff -C2 -d -r2.56 -r2.57
> *** patchlevel.h	2001/10/19 17:11:58	2.56
> --- patchlevel.h	2001/10/23 22:20:58	2.57
> ***************
> *** 27,31 ****
>   
>   /* Version as a string */
> ! #define PY_VERSION		"2.2b1+"
>   
>   /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
> --- 27,31 ----
>   
>   /* Version as a string */
> ! #define PY_VERSION		"2.2b1"
>   
>   /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
> 
> 
> _______________________________________________
> Python-checkins mailing list
> Python-checkins@python.org
> http://mail.python.org/mailman/listinfo/python-checkins



From guido@python.org  Wed Oct 24 02:23:42 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 23 Oct 2001 21:23:42 -0400
Subject: [Python-Dev] Fixing send()
Message-ID: <200110240123.VAA06720@cj20424-a.reston1.va.home.com>

I've got a possible patch for the send() problem.  I'd like review of
the patch and the related issues before I apply it.  I'd also like
more guidance regarding the suitability of this patch for the 2.1.2
bugfix release.  (It fixes a real bug, but can sometime lead to
different behavior, and adds a new interpretation of the absence of an
optional argument, which can be considered a feature.)

http://sf.net/tracker/?group_id=5470&atid=305470&func=detail&aid=474307

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Wed Oct 24 02:30:27 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 23 Oct 2001 21:30:27 -0400
Subject: [Python-Dev] Dictionary subclasses and exec
In-Reply-To: <20011023181737.B27568@mems-exchange.org>
References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us>
 <200110232108.f9NL8Fi25565@odiug.zope.com>
 <20011023181737.B27568@mems-exchange.org>
Message-ID: <15318.6579.803083.815943@grendel.zope.com>

akuchlin@mems-exchange.org writes:
 > Oh, well; I need to come up with a different example for subclassing a
 > built-in type, then.  Thanks!   <puts on thinking cap>

Andrew,
  You can take a look at that implementation of
xml.dom.minidom.NodeList; for Python 2.2, the implementation goes like
this:

----------------------------------------------------------------------
class NodeList(list):
    def item(self, index):
        if 0 <= index < len(self):
            return self[index]

    length = property(lambda self: len(self),
                      doc="The number of nodes in the NodeList.")
----------------------------------------------------------------------


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From akuchlin@mems-exchange.org  Wed Oct 24 02:42:42 2001
From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org)
Date: Tue, 23 Oct 2001 21:42:42 -0400
Subject: [Python-Dev] Dictionary subclasses and exec
In-Reply-To: <15318.6579.803083.815943@grendel.zope.com>; from fdrake@acm.org on Tue, Oct 23, 2001 at 09:30:27PM -0400
References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> <15318.6579.803083.815943@grendel.zope.com>
Message-ID: <20011023214242.A28096@mems-exchange.org>

On Tue, Oct 23, 2001 at 09:30:27PM -0400, Fred L. Drake, Jr. wrote:
>  You can take a look at that implementation of
>xml.dom.minidom.NodeList; for Python 2.2, the implementation goes like
>this:
   <vanishingly short implementation deleted>

Thanks; maybe something like that will do, and it sparks a stray
thought: would it be worth making the file wrapper class in posixfile.py
a subclass of file?

--amk



From fdrake@acm.org  Wed Oct 24 02:57:51 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 23 Oct 2001 21:57:51 -0400
Subject: [Python-Dev] Dictionary subclasses and exec
In-Reply-To: <20011023214242.A28096@mems-exchange.org>
References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us>
 <200110232108.f9NL8Fi25565@odiug.zope.com>
 <20011023181737.B27568@mems-exchange.org>
 <15318.6579.803083.815943@grendel.zope.com>
 <20011023214242.A28096@mems-exchange.org>
Message-ID: <15318.8223.646037.669260@grendel.zope.com>

akuchlin@mems-exchange.org writes:
 > Thanks; maybe something like that will do, and it sparks a stray
 > thought: would it be worth making the file wrapper class in posixfile.py
 > a subclass of file?

  Given that the documentation for posixfile delares it to be
(essentially) obsolete, I thnk the answer is "no".  Code that is
obsolete should not be touched as long as it continues to pass any
relevant tests and there are no reported bugs.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From Anthony Baxter <anthony@interlink.com.au>  Wed Oct 24 03:23:30 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Wed, 24 Oct 2001 12:23:30 +1000
Subject: [Python-Dev] Fixing send()
In-Reply-To: Message from Guido van Rossum <guido@python.org>
 of "Tue, 23 Oct 2001 21:23:42 -0400." <200110240123.VAA06720@cj20424-a.reston1.va.home.com>
Message-ID: <200110240223.f9O2NUg29502@mbuna.arbhome.com.au>

>>> Guido van Rossum wrote
> I've got a possible patch for the send() problem.  I'd like review of
> the patch and the related issues before I apply it.  I'd also like
> more guidance regarding the suitability of this patch for the 2.1.2
> bugfix release.  (It fixes a real bug, but can sometime lead to
> different behavior, and adds a new interpretation of the absence of an
> optional argument, which can be considered a feature.)

It's a tough decision, because this really _is_ the cleaner solution,
but it does stand a slight chance of breaking code, and it's also something
that can be worked around in the std library. At the moment I'm planning to
just fix the std library modules instead - as these are definitely broken.
The added functionality of python's send() is very useful, but it's a 
feature, not a bug :)

Anthony


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.



From jack@oratrix.nl  Wed Oct 24 10:25:46 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Wed, 24 Oct 2001 11:25:46 +0200
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Message by Guido van Rossum <guido@python.org> ,
 Tue, 23 Oct 2001 20:00:44 -0400 , <200110240000.UAA06439@cj20424-a.reston1.va.home.com>
Message-ID: <20011024092547.54106303181@snelboot.oratrix.nl>

> Jack, you can't do this on the trunk.  If you *have* to have a
> different patch level, please use a branch!

Well, apparently I _can_, I just did. Maybe I _shouldn't_.... :-)

I undid it this morning, this mod wasn't supposed to escape from my machine. 
Maybe I should start using the release branch for Mac releases after you are 
done with it for the Unix/Windows release. As there is an r22b1 tag to point 
to the exact revision used for the release there should be no adverse 
consequences. What do you think?
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From fdrake@acm.org  Wed Oct 24 12:02:10 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Wed, 24 Oct 2001 07:02:10 -0400
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: <20011024092547.54106303181@snelboot.oratrix.nl>
References: <guido@python.org>
 <200110240000.UAA06439@cj20424-a.reston1.va.home.com>
 <20011024092547.54106303181@snelboot.oratrix.nl>
Message-ID: <15318.40882.444530.755092@grendel.zope.com>

Jack Jansen writes:
 > Maybe I should start using the release branch for Mac releases
 > after you are done with it for the Unix/Windows release. As there
 > is an r22b1 tag to point to the exact revision used for the release
 > there should be no adverse consequences. What do you think?

  That sounds like the right thing to me.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Wed Oct 24 13:50:08 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 24 Oct 2001 08:50:08 -0400
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Your message of "Wed, 24 Oct 2001 11:25:46 +0200."
 <20011024092547.54106303181@snelboot.oratrix.nl>
References: <20011024092547.54106303181@snelboot.oratrix.nl>
Message-ID: <200110241250.IAA09678@cj20424-a.reston1.va.home.com>

> Well, apparently I _can_, I just did. Maybe I _shouldn't_.... :-)
> 
> I undid it this morning, this mod wasn't supposed to escape from my machine. 

Thanks. :-)

> Maybe I should start using the release branch for Mac releases after you are 
> done with it for the Unix/Windows release. As there is an r22b1 tag to point 
> to the exact revision used for the release there should be no adverse 
> consequences. What do you think?

Fine with me.  For you, it might mean more merging of changes though
(unless you start working on the Mac port a few days *before* the next
beta release :-).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Anthony Baxter <anthony@interlink.com.au>  Wed Oct 24 14:09:59 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Wed, 24 Oct 2001 23:09:59 +1000
Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato
In-Reply-To: Message from akuchlin@mems-exchange.org
 of "Tue, 23 Oct 2001 09:11:26 -0400." <20011023091126.A24293@mems-exchange.org>
Message-ID: <200110241309.f9OD9xv32763@mbuna.arbhome.com.au>

It's worth re-visiting the advogato page again - there's a post from 
Pat McGovern at SF there now.

>>> akuchlin@mems-exchange.org wrote
> There's a discussion of SourceForge alternatives at
> http://www.advogato.org/article/357.html, and in a comment someone
> mentions that there's an XML export of the data in the tracker.
> Unfortunately I can't get it to work following his instructions, but
> perhaps someone else can experiment with it more than I can.
> 
> --amk
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> 

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.



From barry@zope.com  Wed Oct 24 15:10:56 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Wed, 24 Oct 2001 10:10:56 -0400
Subject: [Python-Dev] Re: patchlevel.h
References: <guido@python.org>
 <200110240000.UAA06439@cj20424-a.reston1.va.home.com>
 <20011024092547.54106303181@snelboot.oratrix.nl>
Message-ID: <15318.52208.962415.759362@anthem.wooz.org>

>>>>> "JJ" == Jack Jansen <jack@oratrix.nl> writes:

    JJ> I undid it this morning, this mod wasn't supposed to escape
    JJ> from my machine.  Maybe I should start using the release
    JJ> branch for Mac releases after you are done with it for the
    JJ> Unix/Windows release. As there is an r22b1 tag to point to the
    JJ> exact revision used for the release there should be no adverse
    JJ> consequences. What do you think?

Should be fine, although I'd like to add something about it to PEP
101, just to make it official.  How about in the "What Next?" section,
something like:

"Turn ownership of the branch tag over to Jack so he can make the Mac
release.  He'll typically twiddle with patchlevel.h and may make a few
other checkins as needed for the Mac.

When he's done, he'll add another tag to the branch: X.YaZ-mac.
"

That way we'll have the source tarball tag in the branch and the mac
release in the branch and we could recreate either.

-Barry


From jack@oratrix.nl  Wed Oct 24 15:16:33 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Wed, 24 Oct 2001 16:16:33 +0200
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Message by barry@zope.com (Barry A. Warsaw) ,
 Wed, 24 Oct 2001 10:10:56 -0400 , <15318.52208.962415.759362@anthem.wooz.org>
Message-ID: <20011024141633.96B6F303181@snelboot.oratrix.nl>

> Should be fine, although I'd like to add something about it to PEP
> 101, just to make it official.  How about in the "What Next?" section,
> something like:
> 
> "Turn ownership of the branch tag over to Jack so he can make the Mac
> release.  He'll typically twiddle with patchlevel.h and may make a few
> other checkins as needed for the Mac.
> 
> When he's done, he'll add another tag to the branch: X.YaZ-mac.
> "

Fine with me, except for the middle sentence, which should go: one of the 
reasons to go on the branch is that I *don't* have to twiddle patchlevel.h:-)
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From barry@zope.com  Wed Oct 24 15:22:52 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Wed, 24 Oct 2001 10:22:52 -0400
Subject: [Python-Dev] Re: patchlevel.h
References: <15318.52208.962415.759362@anthem.wooz.org>
 <20011024141633.96B6F303181@snelboot.oratrix.nl>
Message-ID: <15318.52924.633767.193832@anthem.wooz.org>

>>>>> "JJ" == Jack Jansen <jack@oratrix.nl> writes:

    JJ> Fine with me, except for the middle sentence, which should go:
    JJ> one of the reasons to go on the branch is that I *don't* have
    JJ> to twiddle patchlevel.h:-)

D'oh!  Of course. ;)


From martin@v.loewis.de  Wed Oct 24 19:18:32 2001
From: martin@v.loewis.de (Martin v. Loewis)
Date: Wed, 24 Oct 2001 20:18:32 +0200
Subject: [Python-Dev] Re: patchlevel.h
Message-ID: <200110241818.f9OIIWp01350@mira.informatik.hu-berlin.de>

> Maybe I should start using the release branch for Mac releases after you are 
> done with it for the Unix/Windows release. As there is an r22b1 tag to point 
> to the exact revision used for the release there should be no adverse 
> consequences. What do you think?

As before, I think there should be only a single 2.2b1 release. If you
need to provide binaries, use what was released in the 2.2b1 source
distribution (which is *not* the Unix release, it is the source
distribution).

What do your users lose if they have to get 2.2b1 as-is (ie. without
further patches)? It may have bugs, sure - but so does the 2.2b1
release for everybody else. There will be a 2.2b2 release, so whatever
you commit now will appear in 2.2b2.

I think these release schedules are there for a purpose.

Regards,
Martin



From guido@python.org  Wed Oct 24 20:13:44 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 24 Oct 2001 15:13:44 -0400
Subject: [Python-Dev] Fixing send()
In-Reply-To: Your message of "Wed, 24 Oct 2001 12:23:30 +1000."
 <200110240223.f9O2NUg29502@mbuna.arbhome.com.au>
References: <200110240223.f9O2NUg29502@mbuna.arbhome.com.au>
Message-ID: <200110241913.f9OJDiM10434@odiug.zope.com>

> >>> Guido van Rossum wrote
> > I've got a possible patch for the send() problem.  I'd like review of
> > the patch and the related issues before I apply it.  I'd also like
> > more guidance regarding the suitability of this patch for the 2.1.2
> > bugfix release.  (It fixes a real bug, but can sometime lead to
> > different behavior, and adds a new interpretation of the absence of an
> > optional argument, which can be considered a feature.)

> It's a tough decision, because this really _is_ the cleaner solution,
> but it does stand a slight chance of breaking code, and it's also something
> that can be worked around in the std library. At the moment I'm planning to
> just fix the std library modules instead - as these are definitely broken.
> The added functionality of python's send() is very useful, but it's a 
> feature, not a bug :)
> 
> Anthony

After ample discussion, the PythonLabs crew didn't think my patch
would be safe enough -- see the SF tracker entry for a full
discussion.

Instead, we decided to fix the standard library to always check the
return value of send().  Can you help?  (If you check in a fix to the
2.1 maintenance branch, you can mark it as "propagate to trunk please".)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Wed Oct 24 22:52:51 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 24 Oct 2001 23:52:51 +0200
Subject: [Python-Dev] iterator support for SRE?
Message-ID: <014801c15cd6$3d18b270$ced241d5@hagrid>

a common wish for SRE is a variant of findall that returns
all match objects, instead of the strings.

recently, I realized that the internal scanner type (originally
added to speed up Python versions of findall/sub/split) can
be wrapped in an iterator, allowing the user to loop over all
matches.

>>> import re
>>> p = re.compile(somepattern)
>>> for match in iter(p.scanner(somestring).search, None):
>>>     print match.groups()

how about adding a "finditer" method, which takes care of
the low-level setup:

>>> for match in p.finditer(somestring):
>>>     print match.groups()

or should we just make the scanner factory an official part
of the SRE interface?

</F>



From guido@python.org  Thu Oct 25 04:35:56 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 24 Oct 2001 23:35:56 -0400
Subject: [Python-Dev] iterator support for SRE?
In-Reply-To: Your message of "Wed, 24 Oct 2001 23:52:51 +0200."
 <014801c15cd6$3d18b270$ced241d5@hagrid>
References: <014801c15cd6$3d18b270$ced241d5@hagrid>
Message-ID: <200110250335.XAA10393@cj20424-a.reston1.va.home.com>

> a common wish for SRE is a variant of findall that returns
> all match objects, instead of the strings.
> 
> recently, I realized that the internal scanner type (originally
> added to speed up Python versions of findall/sub/split) can
> be wrapped in an iterator, allowing the user to loop over all
> matches.
> 
> >>> import re
> >>> p = re.compile(somepattern)
> >>> for match in iter(p.scanner(somestring).search, None):
> >>>     print match.groups()
> 
> how about adding a "finditer" method, which takes care of
> the low-level setup:
> 
> >>> for match in p.finditer(somestring):
> >>>     print match.groups()
> 
> or should we just make the scanner factory an official part
> of the SRE interface?

Or both?  The scanner interface seems vaguely useful as a low-level
tool to create other higher-level variants; but the finditer() call
seems to nicely capture a common case.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Anthony Baxter <anthony@interlink.com.au>  Thu Oct 25 07:24:20 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Thu, 25 Oct 2001 16:24:20 +1000
Subject: [Python-Dev] Fixing send()
In-Reply-To: Message from Guido van Rossum <guido@python.org>
 of "Wed, 24 Oct 2001 15:13:44 -0400." <200110241913.f9OJDiM10434@odiug.zope.com>
Message-ID: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au>

>>> Guido van Rossum wrote
> Instead, we decided to fix the standard library to always check the
> return value of send().  Can you help?  (If you check in a fix to the
> 2.1 maintenance branch, you can mark it as "propagate to trunk please".)

Will do - I plan to check in a bunch of fixes tomorrow morning. (continuing
gdb hell today).

One to consider is the status of 'socket.py' - patched, or not patched...?

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.



From sjoerd.mullender@oratrix.com  Thu Oct 25 11:36:57 2001
From: sjoerd.mullender@oratrix.com (Sjoerd Mullender)
Date: Thu, 25 Oct 2001 12:36:57 +0200
Subject: [Python-Dev] suggestion for PEP 101
Message-ID: <20011025103658.82DCD3021E6@bireme.oratrix.nl>

I noticed that the first step in creating a release is

    ___ Do a CVS update with the -A flag, e.g.
        % cvs update -A

I would suggest adding the -d and -P flags to this command (so cvs
update -APd).
The -d flag says to create directories that are in the repository but
not in the checked-out version (handy for new directories), and the -P
flag says to delete (prune) empty directories from the checked-out
version (handy for directories that have become empty because all
files were deleted).

-- Sjoerd Mullender <sjoerd.mullender@oratrix.com>


From guido@python.org  Thu Oct 25 15:59:34 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 25 Oct 2001 10:59:34 -0400
Subject: [Python-Dev] Fixing send()
In-Reply-To: Your message of "Thu, 25 Oct 2001 16:24:20 +1000."
 <200110250624.f9P6OKE11623@mbuna.arbhome.com.au>
References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au>
Message-ID: <200110251459.f9PExZg21822@odiug.zope.com>

I'm planning to add a sendall() method to the socket object.  See the
SF patch:

http://sf.net/tracker/index.php?func=detail&aid=474307&group_id=5470&atid=305470

If you have an objection, please let me know.  I think this is
important enough to bypass the feature freeze -- it's the best way to
fix the very real bug (on FreeBSD) of unchecked send() calls.  (The
alternative would be to add a loop to every use of send().)

The sendall() call allows us to fix the standard library modules that
currently don't check for send()'s return value, without breaking any
existing code, by changing sock.send(data) into sock.sendall(data).
Other alternatives intended to fix the problem of unchecked send()
calls in standard library module without having to change all send()
calls were deemed to dangerous (see the SF patch discussion).

I think Anthony is going to add this to the 2.1.2 bugfix release as
well -- with the same argument for breaking the feature freeze.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Anthony Baxter <anthony@interlink.com.au>  Thu Oct 25 16:14:01 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Fri, 26 Oct 2001 01:14:01 +1000
Subject: [Python-Dev] Fixing send()
In-Reply-To: Message from Guido van Rossum <guido@python.org>
 of "Thu, 25 Oct 2001 10:59:34 -0400." <200110251459.f9PExZg21822@odiug.zope.com>
Message-ID: <200110251514.f9PFE1229217@mbuna.arbhome.com.au>

>>> Guido van Rossum wrote
> I think Anthony is going to add this to the 2.1.2 bugfix release as
> well -- with the same argument for breaking the feature freeze.

That's correct. I think it's the appropriate way forward.

I'm pretty sure the "goals for patch selection" had an implicit 
"except where footnote 1 applies" in there :)

I'd be curious to know if anyone disagrees violently with this...

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.



From guido@python.org  Thu Oct 25 16:16:29 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 25 Oct 2001 11:16:29 -0400
Subject: [Python-Dev] Fixing send()
In-Reply-To: Your message of "Thu, 25 Oct 2001 11:12:23 EDT."
 <15320.11223.688433.641613@slothrop.digicool.com>
References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <200110251459.f9PExZg21822@odiug.zope.com>
 <15320.11223.688433.641613@slothrop.digicool.com>
Message-ID: <200110251516.f9PFGTf24686@odiug.zope.com>

[Jeremy]
> Can you offer a brief explanation of why we're doing this after all?
> I thought we concluded yesterday that we would fix this library.
> (That's what the last message before this one said.)

Fixing the library would mean adding a loop like this to each
application that uses send and currently doesn't check the return
value:

<    sock.send(data)
---
>    while data:
>        n = sock.send(data)
>        data = data[n:]

I find this more attractive:

<    sock.send(data)
---
>    sock.sendall(data)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gmcm@hypernet.com  Thu Oct 25 16:36:49 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 25 Oct 2001 11:36:49 -0400
Subject: [Python-Dev] Fixing send()
In-Reply-To: <200110251459.f9PExZg21822@odiug.zope.com>
References: Your message of "Thu, 25 Oct 2001 16:24:20 +1000."             <200110250624.f9P6OKE11623@mbuna.arbhome.com.au>
Message-ID: <3BD7F951.26534.24848D4A@localhost>

[Guido]
> I'm planning to add a sendall() method to the socket object.  See
> the SF patch:
> 
> http://sf.net/tracker/index.php?func=detail&aid=474307&group_id=5
> 470&atid=305470
> 
> If you have an objection, please let me know.  I think this is
> important enough to bypass the feature freeze 

The bonehead who uses sendall with a non-blocking socket 
will, of course, get an EWOULDBLOCK (or platform 
equivalent) on an oversize send. Maybe EWOULDBLOCK 
should get turned into ButtHeadProgrammerError("Don't use 
sendall with a non-blocking socket")?

[I agree that this is the best way to fix the std lib, and that it is 
very important that it be fixed.]

- Gordon


From guido@python.org  Thu Oct 25 16:42:45 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 25 Oct 2001 11:42:45 -0400
Subject: [Python-Dev] Fixing send()
In-Reply-To: Your message of "Thu, 25 Oct 2001 11:36:49 EDT."
 <3BD7F951.26534.24848D4A@localhost>
References: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au>
 <3BD7F951.26534.24848D4A@localhost>
Message-ID: <200110251542.f9PFgjF28022@odiug.zope.com>

> The bonehead who uses sendall with a non-blocking socket 
> will, of course, get an EWOULDBLOCK (or platform 
> equivalent) on an oversize send. Maybe EWOULDBLOCK 
> should get turned into ButtHeadProgrammerError("Don't use 
> sendall with a non-blocking socket")?

I think natural selection will take care of this breed. :-)

> [I agree that this is the best way to fix the std lib, and that it is 
> very important that it be fixed.]

Thanks!

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Thu Oct 25 16:45:02 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 25 Oct 2001 11:45:02 -0400
Subject: [Python-Dev] suggestion for PEP 101
References: <20011025103658.82DCD3021E6@bireme.oratrix.nl>
Message-ID: <15320.13182.670145.233045@anthem.wooz.org>

>>>>> "SM" == Sjoerd Mullender <sjoerd.mullender@oratrix.com> writes:

    SM> I noticed that the first step in creating a release is

    |     ___ Do a CVS update with the -A flag, e.g.
    |         % cvs update -A

    SM> I would suggest adding the -d and -P flags to this command (so
    SM> cvs update -APd).

Good point, thanks.
-Barry


From fdrake@acm.org  Thu Oct 25 16:56:18 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Thu, 25 Oct 2001 11:56:18 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011025155618.C21CA28697@cj42289-a.reston1.va.home.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Experimental stylesheet changes -- code is now presented using proportional
fonts (mostly since Guido really dislikes Courier).  Please send comments
on this change to me at python-docs@python.org.



From jeremy@zope.com  Thu Oct 25 17:18:40 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Thu, 25 Oct 2001 12:18:40 -0400 (EDT)
Subject: [Python-Dev] Fixing send()
In-Reply-To: <3BD7F951.26534.24848D4A@localhost>
References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au>
 <3BD7F951.26534.24848D4A@localhost>
Message-ID: <15320.15200.184832.18268@slothrop.digicool.com>

If we're making the std library routines that use sockets robust, I
expect we ought to think about exceptions that should be dealt with.
My Linux man page suggests that ENOBUFS and EINTR are errors that
indicate "try again."  There are probably other such errors on other
platforms. 

We use Python sockets and asyncore in ZEO (a Zope subsystem) and I've
found that every weird error that I've never heard of seems to occur
in the wild.  If we don't deal with these errors, then we've haven't
fully succeeded in making the calls robust.

Jeremy



From guido@python.org  Thu Oct 25 19:03:13 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 25 Oct 2001 14:03:13 -0400
Subject: [Python-Dev] Fixing send()
In-Reply-To: Your message of "Thu, 25 Oct 2001 12:18:40 EDT."
 <15320.15200.184832.18268@slothrop.digicool.com>
References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost>
 <15320.15200.184832.18268@slothrop.digicool.com>
Message-ID: <200110251803.f9PI3D428392@odiug.zope.com>

> If we're making the std library routines that use sockets robust, I
> expect we ought to think about exceptions that should be dealt with.
> My Linux man page suggests that ENOBUFS and EINTR are errors that
> indicate "try again."  There are probably other such errors on other
> platforms. 
> 
> We use Python sockets and asyncore in ZEO (a Zope subsystem) and I've
> found that every weird error that I've never heard of seems to occur
> in the wild.  If we don't deal with these errors, then we've haven't
> fully succeeded in making the calls robust.

There's a difference though.  The bug I'm trying to fix with sendall()
was a silent failure, causing at best protocol failures later, at
worst silently lost data.  ENOBUFS or EINTR etc. cause clear
exceptions when they happen.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jeremy@zope.com  Thu Oct 25 22:15:39 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Thu, 25 Oct 2001 17:15:39 -0400 (EDT)
Subject: [Python-Dev] dict comps
Message-ID: <15320.33019.723537.580070@slothrop.digicool.com>

We agreed yesterday that the dictionary() constructor would accept a
a list of two-tuples (strictly speaking an iterable object of iterable
objects of length 2).  That plus list comprehensions pretty much
covers the territory of dict comprehensions:

>>> print dictionary([(i, chr(65 + i)) for i in range(4)])
{0: 'A', 1: 'B', 2: 'C', 3: 'D'}

Jeremy



From andymac@bullseye.apana.org.au  Thu Oct 25 14:15:24 2001
From: andymac@bullseye.apana.org.au (Andrew MacIntyre)
Date: Thu, 25 Oct 2001 23:15:24 +1000 (EST)
Subject: [Python-Dev] status of OS/2 ports of Python
In-Reply-To: <323.1003777312.538787.209802503@lilbastard>
Message-ID: <Pine.OS2.4.32.0110252249260.577-100000@central>

Unfortunately I haven't had the bandwidth to vigorously pursue completing
integration of my previously submitted patches, and at this time don't
expect to be able to do much until about the time 2.2b2 is scheduled to
hit the streets :-(.  So I don't expect that 2.2 will have OS/2+EMX build
support in the sourceball...

At this stage, my plan is to finalise a binary release of 2.2 before
submitting a new set of patches, taking into account Martin von Loewis'
comments and suggestions (much appreciated thank you!) among others.  I
also plan to try and coordinate my patches with those of Michael Muller
(473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2.

I will try and review Michael's patches with a view to getting them into
2.2, as the VAC++ build has been part of the sourceball for a few
versions.

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
        andymac@pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia



From guido@python.org  Thu Oct 25 22:30:22 2001
From: guido@python.org (Guido van Rossum)
Date: Thu, 25 Oct 2001 17:30:22 -0400
Subject: [Python-Dev] status of OS/2 ports of Python
In-Reply-To: Your message of "Thu, 25 Oct 2001 23:15:24 +1000."
 <Pine.OS2.4.32.0110252249260.577-100000@central>
References: <Pine.OS2.4.32.0110252249260.577-100000@central>
Message-ID: <200110252130.f9PLUM329494@odiug.zope.com>

> Unfortunately I haven't had the bandwidth to vigorously pursue completing
> integration of my previously submitted patches, and at this time don't
> expect to be able to do much until about the time 2.2b2 is scheduled to
> hit the streets :-(.  So I don't expect that 2.2 will have OS/2+EMX build
> support in the sourceball...
> 
> At this stage, my plan is to finalise a binary release of 2.2 before
> submitting a new set of patches, taking into account Martin von Loewis'
> comments and suggestions (much appreciated thank you!) among others.  I
> also plan to try and coordinate my patches with those of Michael Muller
> (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2.
> 
> I will try and review Michael's patches with a view to getting them into
> 2.2, as the VAC++ build has been part of the sourceball for a few
> versions.

Andy, would it help if you had SourceForge commit privileges?  You'd
be obliged to only check in code that doesn't break the build on other
platforms, so you would be required to test anything you want to check
in on another platform (Linux or Windows) before you commit to save
you the embarrassment of breaking the build, but if (as I expect) you
would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a
big burden.  It would save you the trouble of uploading patches to
SourceForge and it would save *us* the trouble of reviewing your
patches and checking them in.

All your code would be owned by the PSF, but that's what you want
anyway, right?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From skip@pobox.com (Skip Montanaro)  Thu Oct 25 22:53:27 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Thu, 25 Oct 2001 16:53:27 -0500
Subject: [Python-Dev] weird gdb/python/gtk/pango interaction?
Message-ID: <15320.35287.392870.291275@beluga.mojam.com>

Folks,

I'm no longer able to run PyGtk scripts from gdb under some circumstances.
I am using the CVS HEAD of glib, atk, pango, gtk+ and pygtk.  The following
simple Python script works fine when run from gdb:

    #!/usr/bin/env python

    import gtk

    w = gtk.Window()
    w.connect("destroy", gtk.mainquit)
    #lbl = gtk.Label("hi")
    #w.add(lbl)

    w.show_all()
    gtk.mainloop()

with the following output:

    (gdb) r basic.py
    Starting program: /usr/local/bin/python basic.py
    [New Thread 1024 (LWP 19044)]
    Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY
                    UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES.                if you don't know why you are getting this, you probably want to                use the stable branch which can be retrieved from
                    ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with
                    cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+

    Program exited normally.

If I uncomment the two lines involving lbl, however, I get this mess:

    (gdb) r basic.py
    Starting program: /usr/local/bin/python basic.py
    [New Thread 1024 (LWP 19127)]
    Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY
                    UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES.                if you don't know why you are getting this, you probably want to                use the stable branch which can be retrieved from
                    ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with
                    cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+

    basic.py (pid:19127): GRuntime-CRITICAL **: file gparamspecs.c: line 1687 (g_param_spec_float): assertion `default_value >= minimum && default_value <= maximum' failed

    basic.py (pid:19127): GRuntime-CRITICAL **: file gobject.c: line 270 (g_object_class_install_property): assertion `G_IS_PARAM_SPEC (pspec)' failed

    basic.py (pid:19127): ** WARNING **: Couldn't load font "Sans -2.09715e+06" falling back to "Sans -2.09715e+06"

    basic.py (pid:19127): ** WARNING **: Couldn't load font "Sans -2.09715e+06" falling back to "Sans -2.09715e+06"

    basic.py (pid:19127): ** WARNING **: All font failbacks failed!!!!

    Program exited with code 01.

If I change the Label to an EventBox, everything works, presumably because
no font rendering is involved.  Also when I run the script from the bash
prompt it works as expected (no Gtk warnings).  The equivalent C program:

    #include <gtk/gtk.h>

    int main( int   argc,
              char *argv[] )
    {
        GtkWidget *window;
        GtkWidget *label;

        gtk_init(&argc, &argv);

        window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
        gtk_signal_connect (GTK_OBJECT (window), "destroy",
                            GTK_SIGNAL_FUNC(gtk_main_quit),
                            NULL);
        gtk_window_set_title (GTK_WINDOW (window), "");

        label = gtk_label_new("hi");
        gtk_container_add(GTK_CONTAINER(window), label);

        gtk_widget_show_all (GTK_WIDGET(window));
        gtk_main();

        return(0);
    }

works as expected, both from the command line and if run from the gdb
prompt.

This wouldn't be such a big deal, except in a more complex script I get
segfaults in some circumstances, and this makes debugging that problem a bit
difficult.

Any idea what's going on?

-- 
Skip Montanaro (skip@pobox.com)
http://www.mojam.com/
http://www.musi-cal.com/


From martin@v.loewis.de  Thu Oct 25 23:31:57 2001
From: martin@v.loewis.de (Martin v. Loewis)
Date: Fri, 26 Oct 2001 00:31:57 +0200
Subject: [Python-Dev] weird gdb/python/gtk/pango interaction?
Message-ID: <200110252231.f9PMVvW03121@mira.informatik.hu-berlin.de>

> Any idea what's going on?

When you see that a program behaves differently in gdb than it does on
the shell, the most likely cause is that gdb starts another shell when
running the program, I believe essentially through system("$SHELL -c
command") (see gdb documentation and source for details). For example,
some of the $HOME/.something files may set PYTHONPATH to some
interesting value.

If you see that a Python program behaves differently from an
"equivalent" C code, well, the most likely cause is that the programs
are not equivalent. One obvious (to me) difference is that one calls
set_title, when the other doesn't.

Regards,
Martin


From aahz@rahul.net  Fri Oct 26 02:12:45 2001
From: aahz@rahul.net (Aahz Maruch)
Date: Thu, 25 Oct 2001 18:12:45 -0700 (PDT)
Subject: [Python-Dev] Fixing send()
In-Reply-To: <200110251514.f9PFE1229217@mbuna.arbhome.com.au> from "Anthony Baxter" at Oct 26, 2001 01:14:01 AM
Message-ID: <20011026011245.65372E8C3@waltz.rahul.net>

Anthony Baxter wrote:
>Guido van Rossum wrote
>>
>> I think Anthony is going to add this to the 2.1.2 bugfix release as
>> well -- with the same argument for breaking the feature freeze.
> 
> That's correct. I think it's the appropriate way forward.
> 
> I'm pretty sure the "goals for patch selection" had an implicit 
> "except where footnote 1 applies" in there :)

PEP 6 has an explicit exception for BDFL Pronouncements.  ;-)

Given that this is "guaranteed" to not break any code, it's fair game.
-- 
                      --- Aahz (@pobox.com)

Hugs and backrubs -- I break Rule 6       <*>       http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

We must not let the evil of a few trample the freedoms of the many.


From fdrake@acm.org  Fri Oct 26 04:13:46 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Thu, 25 Oct 2001 23:13:46 -0400 (EDT)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011026031346.8550628697@beowolf.digicool.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Yet another experimental change to the presentation -- ditch the
proportional font for code since some displays (especially interactive
sessions and tabular displays) get messed up with proportional fonts.
We do try to use monospaced fonts that are less ugly than Courier.

As before, feedback on the fonts is welcome at python-docs@python.org.



From tim.one@home.com  Fri Oct 26 06:34:16 2001
From: tim.one@home.com (Tim Peters)
Date: Fri, 26 Oct 2001 01:34:16 -0400
Subject: [Python-Dev] dict comps
In-Reply-To: <15320.33019.723537.580070@slothrop.digicool.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOECOMDAA.tim.one@home.com>

[Jeremy Hylton]
> We agreed yesterday that the dictionary() constructor would accept a
> a list of two-tuples (strictly speaking an iterable object of iterable
> objects of length 2).

FYI, this is checked in now.

> That plus list comprehensions pretty much covers the territory of
> dict comprehensions:
>
> >>> print dictionary([(i, chr(65 + i)) for i in range(4)])
> {0: 'A', 1: 'B', 2: 'C', 3: 'D'}

Wow -- that's *exactly* what it prints.  You got your own time machine now?

While it covers the semantics, the pragmatics may be off, since listcomps
produce genuine lists, and so e.g.

    dictionary([(key, f(key)) for key in file('huge')])

may require constructing an unboundedly large list of twoples before
dictionary() sees the first pair.  dictionary() per se doesn't require
materializing a giant list in one gulp.



From mal@lemburg.com  Fri Oct 26 10:02:02 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 26 Oct 2001 11:02:02 +0200
Subject: [Python-Dev] dict comps
References: <LNBBLJKPBEHFEDALKOLCOECOMDAA.tim.one@home.com>
Message-ID: <3BD9268A.72AD71E9@lemburg.com>

Tim Peters wrote:
> 
> [Jeremy Hylton]
> > We agreed yesterday that the dictionary() constructor would accept a
> > a list of two-tuples (strictly speaking an iterable object of iterable
> > objects of length 2).
> 
> FYI, this is checked in now.
> 
> > That plus list comprehensions pretty much covers the territory of
> > dict comprehensions:
> >
> > >>> print dictionary([(i, chr(65 + i)) for i in range(4)])
> > {0: 'A', 1: 'B', 2: 'C', 3: 'D'}

Cool !
 
> Wow -- that's *exactly* what it prints.  You got your own time machine now?
> 
> While it covers the semantics, the pragmatics may be off, since listcomps
> produce genuine lists, and so e.g.
> 
>     dictionary([(key, f(key)) for key in file('huge')])
> 
> may require constructing an unboundedly large list of twoples before
> dictionary() sees the first pair.  dictionary() per se doesn't require
> materializing a giant list in one gulp.

One way or another, you'll use up a giant chunk or two of data 
on the heap... I'd suggest adding a new builtin 
huge_file_as_mapping_apply('file', f) ;-)

Seriously, this goes down the path of lazy evaluation of expressions.
Not sure whether this is the right path to follow though (can cause
brain damage due to overloaded builtin debuggers, IMHO).

BTW, looks like I can finally get rid off the dict() builtin I 
have in mxTools which is Good News !

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From mal@lemburg.com  Fri Oct 26 09:52:50 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 26 Oct 2001 10:52:50 +0200
Subject: [Python-Dev] weird gdb/python/gtk/pango interaction?
References: <15320.35287.392870.291275@beluga.mojam.com>
Message-ID: <3BD92462.8A8E2C4F@lemburg.com>

Skip Montanaro wrote:
> 
> Folks,
> 
> I'm no longer able to run PyGtk scripts from gdb under some circumstances.
> I am using the CVS HEAD of glib, atk, pango, gtk+ and pygtk.  The following
> simple Python script works fine when run from gdb:
> 
>     #!/usr/bin/env python
> 
>     import gtk
> 
>     w = gtk.Window()
>     w.connect("destroy", gtk.mainquit)
>     #lbl = gtk.Label("hi")
>     #w.add(lbl)
> 
>     w.show_all()
>     gtk.mainloop()
> 
> with the following output:
> 
>     (gdb) r basic.py
>     Starting program: /usr/local/bin/python basic.py
>     [New Thread 1024 (LWP 19044)]
>     Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY
>                     UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES.                if you don't know why you are getting this, you probably want to                use the stable branch which can be retrieved from
>                     ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with
>                     cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+

Just curious: does the proposed stable branch show the same behaviour ?
If so, I'd suggest to ask the GTK guys for help. If not, then perhaps
you have to rebuild the Python extensions involved and make
sure that they match the Python version you are using.

What you're seeing looks like there are some buffer overruns going
on... I've never seen such a strange font name ;-)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From fredrik@pythonware.com  Fri Oct 26 10:38:29 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 26 Oct 2001 11:38:29 +0200
Subject: [Python-Dev] iterator support for SRE?
References: <014801c15cd6$3d18b270$ced241d5@hagrid>  <200110250335.XAA10393@cj20424-a.reston1.va.home.com>
Message-ID: <015a01c15e01$f99f4ef0$0900a8c0@spiff>

guido wrote:
> > how about adding a "finditer" method, which takes care of
> > the low-level setup:
> >
> > >>> for match in p.finditer(somestring):
> > >>>     print match.groups()
> >
> > or should we just make the scanner factory an official part
> > of the SRE interface?
>
> Or both?  The scanner interface seems vaguely useful as a low-level
> tool to create other higher-level variants; but the finditer() call
> seems to nicely capture a common case.

as you might have noticed, I went ahead and checked in code
for "finditer".  nobody has complained this far ;-)

isn't "iterfind" a better name, btw?  more like "iterkeys" (etc).

if nobody complains, I'll change the name, and check in some
documentation.

:::

will have to think about scanner; I agree that it may seem to
be vaguely useful, but cannot think of a use case that isn't
better handled by findall/split/sub or iterfind.  can anyone?

</F>




From jack@oratrix.nl  Fri Oct 26 11:25:39 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Fri, 26 Oct 2001 12:25:39 +0200
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Message by "Martin v. Loewis" <martin@v.loewis.de> ,
 Wed, 24 Oct 2001 20:18:32 +0200 , <200110241818.f9OIIWp01350@mira.informatik.hu-berlin.de>
Message-ID: <20011026102540.3DBEB303181@snelboot.oratrix.nl>

> > Maybe I should start using the release branch for Mac releases after you are 
> > done with it for the Unix/Windows release. As there is an r22b1 tag to point 
> > to the exact revision used for the release there should be no adverse 
> > consequences. What do you think?
> 
> As before, I think there should be only a single 2.2b1 release. If you
> need to provide binaries, use what was released in the 2.2b1 source
> distribution (which is *not* the Unix release, it is the source
> distribution).

Martin,
while I completely agree with you in principle, in practice this isn't 
possible.
When the release branch is made there's usually a flurry of minor fixes, and 
these are almost guaranteed to break something on the mac. Tiny things, like 
missing casts, but breaks nonetheless. Moreover, the "release loop" is now 
about 24 hours long, according to PEP101, and even extending it seriously 
(like to about a week) still wouldn't guarantee that I could react timely. Not 
only am I 6 hours away from the unix/win release folks, but I also have a 
paying job that is less and less MacPython-related, so I can't firmly commit 
myself.

And it has happened to me already (twice, I think) that there was a 
showstopper bug on the Mac that has caused me to either be very late with a 
release or skip one altogether. This happens on the Mac more often than on 
Unix/Windows, because MacPython relies on a third party unix I/O emulation 
library.

--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From thomas.heller@ion-tof.com  Fri Oct 26 14:29:03 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 26 Oct 2001 15:29:03 +0200
Subject: [Python-Dev] innocent question ;-)
Message-ID: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook>

... about my beloved buffer object:

Would a patch to make the buffer object subclassable
be still accepted for 2.2, or does this fall under
the feature freeze?

Another question:
It is my understanding that makeing an object
subclassable also exposes its constructor to Python.

What is the signature of this constructor? Is the
three argument version xxx(name, bases, dict) always used
to return a new type object (Don Beaudry hook), and can other
signatures freely be used to creates objects/instances
of this type/class?

Thomas



From guido@python.org  Fri Oct 26 14:30:55 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 09:30:55 -0400
Subject: [Python-Dev] iterator support for SRE?
In-Reply-To: Your message of "Fri, 26 Oct 2001 11:38:29 +0200."
 <015a01c15e01$f99f4ef0$0900a8c0@spiff>
References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com>
 <015a01c15e01$f99f4ef0$0900a8c0@spiff>
Message-ID: <200110261330.JAA18598@cj20424-a.reston1.va.home.com>

> as you might have noticed, I went ahead and checked in code
> for "finditer".  nobody has complained this far ;-)

Far from it. :-)

> isn't "iterfind" a better name, btw?  more like "iterkeys" (etc).
> 
> if nobody complains, I'll change the name, and check in some
> documentation.

Hm, but finditer() is more like findall(), and that counts more I'd say.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Fri Oct 26 14:42:53 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 09:42:53 -0400
Subject: [Python-Dev] innocent question ;-)
In-Reply-To: Your message of "Fri, 26 Oct 2001 15:29:03 +0200."
 <024501c15e22$2dd753f0$e000a8c0@thomasnotebook>
References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook>
Message-ID: <200110261342.JAA18712@cj20424-a.reston1.va.home.com>

> ... about my beloved buffer object:
> 
> Would a patch to make the buffer object subclassable
> be still accepted for 2.2, or does this fall under
> the feature freeze?

NO.  Go away. :-)

I think the buffer() builtin should be deprecated, not improved.

> Another question:
> It is my understanding that makeing an object
> subclassable also exposes its constructor to Python.

Yes, since otherwise there would be no way to instantiate the
subclass either.

> What is the signature of this constructor? Is the
> three argument version xxx(name, bases, dict) always used
> to return a new type object (Don Beaudry hook), and can other
> signatures freely be used to creates objects/instances
> of this type/class?

I'm sorry, I don't understand the question.  This is the constructor
signature for type object, IOW for metaclasses.  It has nothing to do
with the constructor signature for builtin classes.

Also, what do you mean by constructor?  __new__ or __init__?

Read test_descr.py for examples.  Also xxsubtype.c and various
built-in types if you're interested in doing this at the C level.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Fri Oct 26 14:57:23 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 26 Oct 2001 15:57:23 +0200
Subject: [Python-Dev] innocent question ;-)
References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook>  <200110261342.JAA18712@cj20424-a.reston1.va.home.com>
Message-ID: <02e901c15e26$2314abd0$e000a8c0@thomasnotebook>

From: "Guido van Rossum" <guido@python.org>
> > ... about my beloved buffer object:
> > 
> > Would a patch to make the buffer object subclassable
> > be still accepted for 2.2, or does this fall under
> > the feature freeze?
> 
> NO.  Go away. :-)
Ok. ;-(
Is the list of subtypable types now complete (for 2.2),
or do you indend to do more?

> 
> I think the buffer() builtin should be deprecated, not improved.
> 
> > Another question:
> > It is my understanding that makeing an object
> > subclassable also exposes its constructor to Python.
> 
> Yes, since otherwise there would be no way to instantiate the
> subclass either.
So I understood this...
> 
> > What is the signature of this constructor? Is the
> > three argument version xxx(name, bases, dict) always used
> > to return a new type object (Don Beaudry hook), and can other
> > signatures freely be used to creates objects/instances
> > of this type/class?
> 
> I'm sorry, I don't understand the question.  This is the constructor
> signature for type object, IOW for metaclasses.  It has nothing to do
> with the constructor signature for builtin classes.
...but not this, sorry.
I've confused this:

  >>> int(10)
  10

with this:
  >>> type(int)("name", (), {})
  <class '__main__.name'>

Sorry again.

Thomas




From paul-python@svensson.org  Fri Oct 26 15:01:56 2001
From: paul-python@svensson.org (Paul Svensson)
Date: Fri, 26 Oct 2001 10:01:56 -0400 (EDT)
Subject: [Python-Dev] dict comps
In-Reply-To: <15320.33019.723537.580070@slothrop.digicool.com>
Message-ID: <Pine.LNX.4.21.0110260958270.27030-100000@familjen.svensson.org>

On Thu, 25 Oct 2001, Jeremy Hylton wrote:

>We agreed yesterday that the dictionary() constructor would accept a
>a list of two-tuples (strictly speaking an iterable object of iterable
>objects of length 2).  That plus list comprehensions pretty much
>covers the territory of dict comprehensions:
>
>>>> print dictionary([(i, chr(65 + i)) for i in range(4)])
>{0: 'A', 1: 'B', 2: 'C', 3: 'D'}
>

Wouldn't it be nice to be able to spell that

	{ i: chr(65 + i) for i in range(4) }

		/Paul



From guido@python.org  Fri Oct 26 15:03:25 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 10:03:25 -0400
Subject: [Python-Dev] innocent question ;-)
In-Reply-To: Your message of "Fri, 26 Oct 2001 15:57:23 +0200."
 <02e901c15e26$2314abd0$e000a8c0@thomasnotebook>
References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> <200110261342.JAA18712@cj20424-a.reston1.va.home.com>
 <02e901c15e26$2314abd0$e000a8c0@thomasnotebook>
Message-ID: <200110261403.KAA18833@cj20424-a.reston1.va.home.com>

> Is the list of subtypable types now complete (for 2.2),
> or do you indend to do more?

I just found a few extension modules that define their own types, like
SocketType.  I'd like to see those converted.  I don't know if I have
time.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jack@oratrix.nl  Fri Oct 26 15:23:31 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Fri, 26 Oct 2001 16:23:31 +0200
Subject: [Python-Dev] innocent question ;-)
In-Reply-To: Message by Guido van Rossum <guido@python.org> ,
 Fri, 26 Oct 2001 10:03:25 -0400 , <200110261403.KAA18833@cj20424-a.reston1.va.home.com>
Message-ID: <20011026142331.C8E84303181@snelboot.oratrix.nl>

> > Is the list of subtypable types now complete (for 2.2),
> > or do you indend to do more?
> 
> I just found a few extension modules that define their own types, like
> SocketType.  I'd like to see those converted.  I don't know if I have
> time.

At what point can we consider the API stable? All this inheriting and 
subtyping seems like it would be absolutely brilliant for the Mac toolbox APIs 
(and I suspect also for the PythonWin API's, Mark?) but I've refrained from 
even looking at it (well, I did have a quick peek:-), because (a) I have no 
time, (b) I don't want to chase a changing API and (c) there's some 
methodchain-based poor-mans-inheritance in there already that will undoubtedly 
break.
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From webmaster@garymgordon.com  Fri Oct 26 15:30:10 2001
From: webmaster@garymgordon.com (Gary M. Gordon)
Date: Fri, 26 Oct 2001 09:30:10 -0500
Subject: [Python-Dev] Which one of our FREE website enhancements would you like?
Message-ID: <E15x7zu-0005Pa-00@thunder5.cwihosting.com>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Gary M. Gordon - FREE Website Enhancement Promotional Offer</TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
</HEAD>
<BODY bgcolor="#ffffee">
<hr solid color="#336699" width="475" align="left">
<br>
<DIV>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <FONT face=Verdana  size="4" color="#ff0000"><STRONG>FREE WEBSITE 
ENHANCEMENT OFFER<br>&nbsp; &nbsp; &nbsp;  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; yes, ... absolutely FREE</strong></font><STRONG> 
!</STRONG><BR><br></DIV><SPAN class=288043613-22102001></SPAN>
<DIV><FONT face=Verdana color=#000040 size=2><SPAN 
class=288043613-22102001>&nbsp;</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
(No contracts!&nbsp;&nbsp; No commitments!&nbsp;&nbsp; No cost!)</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2><SPAN 
class=288043613-22102001>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
<EM>- Links to sample enhancements are provided below. 
-</EM></SPAN></FONT></DIV><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana 
color=#000040 size=2><STRONG>As a way to introduce&nbsp;<SPAN 
class=288043613-22102001>our</SPAN> company to you, we are going to 
give</STRONG></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2><STRONG>you<SPAN 
class=288043613-22102001> </SPAN>an enhancement that you can add to your 
website<SPAN class=288043613-22102001> ............</SPAN></STRONG></FONT></DIV>
<DIV><FONT size=+0><STRONG><SPAN class=288043613-22102001></SPAN><FONT 
face=Verdana><FONT color=#000040><FONT size=2><SPAN 
class=288043613-22102001>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
</SPAN>&nbsp;absolutely <FONT 
color=#ff0000>FREE</FONT>!</FONT></FONT></FONT></STRONG></FONT></DIV>
<DIV><FONT size=+0><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><BR><FONT face=Verdana color=#000040 
size=2>==========================================<SPAN 
class=288043613-22102001>===</SPAN>======</FONT></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp; Just select from one 
of our <STRONG><FONT color=#ff0000>FREE</FONT></STRONG> website enhancement 
features<BR>&nbsp;&nbsp; that are listed below.&nbsp; <STRONG>There are 'No 
contracts required!'</STRONG> ..<BR>
</FONT><BLOCKQUOTE><FONT face=Verdana color=#000040 size=2><STRONG>There are ...
  </STRONG></font><UL><FONT face=Verdana color=#000040 size=2><LI>
    <STRONG>No commitments!<BR></strong><STRONG>No costs!</STRONG><BR>and 
    ...<BR><STRONG>we will also set everything up ... for <FONT 
    color=#ff0000>FREE</FONT>!</STRONG></LI></font> </UL></BLOCKQUOTE></DIV>

<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp; Let me introduce 
myself ...</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp; My name is Gary 
Gordon. I am the owner of Gary M. Gordon, LLC<BR>&nbsp;&nbsp; a WEB DEVELOPMENT 
COMPANY which provides complete solutions<BR>&nbsp;&nbsp; for the needs of small 
to large businesses. (No project is ever to<BR>&nbsp;&nbsp; small or to 
large.)&nbsp; If you need a&nbsp;<SPAN class=288043613-22102001>'</SPAN>fix<SPAN 
class=288043613-22102001>'</SPAN> or&nbsp;<SPAN 
class=288043613-22102001>'</SPAN>enhancement<SPAN 
class=288043613-22102001>'</SPAN> added to<BR>&nbsp;&nbsp; a current website ... 
that's&nbsp;<SPAN class=288043613-22102001>perfect</SPAN>.&nbsp; If you need to 
have a new<BR>&nbsp;&nbsp; website developed from scratch ... that's&nbsp;<SPAN 
class=288043613-22102001>great</SPAN>.&nbsp;&nbsp;<SPAN 
class=288043613-22102001> </SPAN><STRONG><EM>We do it 
all!</EM></STRONG></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT color=#000040><FONT face=Verdana><FONT size=2>&nbsp;&nbsp; 
<STRONG><EM><FONT color=#000000><FONT color=#ff0000>FREE</FONT> 
WEBSITE&nbsp;<SPAN class=288043613-22102001>ENHANCEMENTS</SPAN> YOU CAN CHOOSE 
FROM:</FONT></EM></STRONG></FONT></FONT></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp; (Simply click on the 
'http:// link' provided below the title of<BR>&nbsp;&nbsp; each item to view 
just a sample of the FREE enhancement.)</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp; 
<STRONG>1)</STRONG>&nbsp; <STRONG><FONT color=#ff0000>FREE 'Macromedia Flash 
(.SWF FORMAT) Animation'<BR>
  </FONT></STRONG></FONT><A 
href="http://www.garymgordon.com/flash_samples/demo/logo_flash_sample.html" target="_blank"><FONT 
face=Verdana color=#000040 
size=2>http://www.garymgordon.com/flash_samples/demo/logo_flash_sample.html</FONT></A><BR>
  <FONT 
face=Verdana color=#000040 size=2>Simply provide us with any text, logo and/or 
photo(s) that you would<BR>like to have animated and developed as a FLASH (.SWF) 
ANIMATION.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana 
color=#000040 size=2>&nbsp;&nbsp; <STRONG>2)</STRONG>&nbsp; <STRONG><FONT 
color=#ff0000>FREE 'Cursor Trailer - Animated Clock 
(Javascript)'<BR>
  </FONT></STRONG></FONT><A 
href="http://www.garymgordon.com/javascript_samples/clock_cursor/index.html" target="_blank"><FONT 
face=Verdana color=#000040 
size=2>http://www.garymgordon.com/javascript_samples/clock_cursor/index.html</FONT></A><BR>
  <FONT 
face=Verdana color=#000040 size=2>We will insert and set up this attractive and 
eye catching feature to<BR>work on any webpage of your website.</FONT></DIV>
<FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT>
<FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana 
color=#000040 size=2>&nbsp;&nbsp; <STRONG>3)</STRONG>&nbsp; <STRONG><FONT 
color=#ff0000>FREE 'your choice of any of the following Javascripts<SPAN 
class=288043613-22102001>'</SPAN></FONT></STRONG></FONT></DIV>
<DIV><FONT size=+0><FONT face=Verdana><FONT color=#000040><FONT size=2><SPAN 
class=288043613-22102001>W</SPAN>e will insert and set<SPAN 
class=288043613-22102001> them up</SPAN>&nbsp;to work in any webpage on your 
website'.</FONT></FONT></FONT></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
<STRONG>a)</STRONG>&nbsp;&nbsp;<STRONG><FONT color=#ff0000><SPAN 
class=288043613-22102001>&nbsp; </SPAN>Page Content Management</FONT></STRONG> 
that automatically allows content<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; to be 
displayed on a page -&nbsp;<SPAN class=288043613-22102001>from and 
</SPAN>through a specified date, then</FONT></DIV>
<DIV><FONT size=+0><FONT face=Verdana><FONT color=#000040><FONT size=2><SPAN 
class=288043613-22102001>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
</SPAN>it&nbsp;<SPAN class=288043613-22102001>is </SPAN>automatically&nbsp;<SPAN 
class=288043613-22102001>r</SPAN>emoved from the 
page. <BR>
  </FONT></FONT></FONT></FONT><A 
href="http://www.garymgordon.com/javascript_samples/content_mgmt/index.html" target="_blank"><FONT 
face=Verdana color=#000040 
size=2>http://www.garymgordon.com/javascript_samples/content_mgmt/index.html</FONT></A></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
<STRONG>b)</STRONG>&nbsp;&nbsp;<STRONG><FONT color=#ff0000><SPAN 
class=288043613-22102001>&nbsp; </SPAN>Auto Scroller</FONT></STRONG> for Text 
and Images.&nbsp; This provides a<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
scrolling message that can include text and images (with 
links<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; to other pages or 
websites).<BR>
</FONT><A 
href="http://www.garymgordon.com/javascript_samples/scroller_box/index.html" target="_blank"><FONT 
face=Verdana color=#000040 
size=2>http://www.garymgordon.com/javascript_samples/scroller_box/index.html</FONT></A>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <STRONG>c)</STRONG>&nbsp;<SPAN 
class=288043613-22102001><STRONG>&nbsp; </STRONG></SPAN>&nbsp;<STRONG><FONT 
color=#ff0000>Navigational Menu System</FONT></STRONG> that allows you to 
utilize a drop<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; down menu that lists any 
number of pages, websites, etc. It is<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; a 
convenient and space saving way to add lots of links 
within<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; a small place on a 
webpage.<BR>
</FONT><A 
href="http://www.garymgordon.com/javascript_samples/dropdownmenu/index.html" target="_blank"><FONT 
face=Verdana color=#000040 
size=2>http://www.garymgordon.com/javascript_samples/dropdownmenu/index.html</FONT></A>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana 
color=#000040 size=2><STRONG>Now ... you can choose from any of the above 
website enhancements<BR>absolutely FREE!</STRONG></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>Then, simply contact us&nbsp;and we 
will set everything up<SPAN class=288043613-22102001> </SPAN>for you<SPAN 
class=288043613-22102001> </SPAN><SPAN class=288043613-22102001>-</SPAN> NO 
CHARGE!</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>(This is simply our way of 
introducing our company to you.)</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2><STRONG>E-mail us 
at:</STRONG>&nbsp;&nbsp; </FONT><A href="mailto:webmaster@garymgordon.com"><FONT 
face=Verdana color=#000040 size=2>webmaster@garymgordon.com</FONT></A><BR><FONT 
face=Verdana color=#000040 size=2>or<BR><STRONG>Phone us 
at:</STRONG>&nbsp;&nbsp;&nbsp; 1-856-482-8320&nbsp; (New Jersey, 
USA)</FONT></DIV>
<FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><BR><FONT face=Verdana color=#000040 
size=2>==========================================<SPAN 
class=288043613-22102001>===</SPAN>======</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT><FONT face=Verdana 
color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
To request your FREE website 
enhancement,<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
please include the following two lines 
of<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
information in the BODY of your reply email:</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT><STRONG><FONT 
face=Verdana color=#000040 size=2>FREE WEBSITE ENHANCEMENT 
OFFER<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<SPAN 
class=288043613-22102001>&nbsp;&nbsp;&nbsp;</SPAN><SPAN 
class=288043613-22102001>&nbsp;&nbsp;&nbsp;&nbsp;</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
WE-149-359-1001</FONT></STRONG></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Also provide us with the following:</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
1)&nbsp; Your company name and 
address.<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
2)&nbsp; Your first and last 
name.<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
3)&nbsp; Your title/position with the 
company.<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
4)&nbsp; Your telephone number - and the 
best<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
time to reach you by 
phone.<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
5)&nbsp; Your website URL (if you have one).</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
We will email&nbsp;<SPAN class=288043613-22102001>or call </SPAN>you back 
shortly to 
discuss<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
the free enhancement of your choice. </FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>==========================================<SPAN 
class=288043613-22102001>===</SPAN>======</FONT><FONT face=Verdana color=#000040 
size=2></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; <STRONG>OUR 
CORPORATE INFORMATION:</STRONG></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; Gary M. Gordon, 
LLC<BR>&nbsp;&nbsp;&nbsp; - A Global Web Development 
Corporation<BR>&nbsp;&nbsp;&nbsp; 421 Jamaica Drive<BR>&nbsp;&nbsp;&nbsp; Cherry 
Hill, New Jersey&nbsp; 08002</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; (Available 
-&nbsp; 7 days a week.)</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 
size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; E-mail ::&nbsp; 
</FONT><A href="mailto:webmaster@garymgordon.com"><FONT face=Verdana 
color=#000040 size=2>webmaster@garymgordon.com</FONT></A><BR><FONT face=Verdana 
color=#000040 size=2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Phone&nbsp; ::&nbsp; 
856-482-8320<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Fax&nbsp;&nbsp;&nbsp; ::&nbsp; 801-858-4271</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana 
color=#000040 size=2>&nbsp;&nbsp;&nbsp; Background Information of GARY M. GORDON 
(Owner):</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; Certified Senior 
Web Developer since 1996<BR>&nbsp;&nbsp;&nbsp; Certified Web Master Instructor 
since 1999<BR>&nbsp;&nbsp;&nbsp; 20+ years experience in Sales/Marketing/Graphic 
Design<BR>&nbsp;&nbsp;&nbsp; and Development</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp; <BR>&nbsp;&nbsp;&nbsp; 
Additional Comments:</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; Gary M. Gordon, 
LLC (Web Development Services) was created<BR>&nbsp;&nbsp;&nbsp; in an effort to 
provide quality 'web development' work on a<BR>&nbsp;&nbsp;&nbsp; world-wide 
scale through the joint efforts of a comprehensive<BR>&nbsp;&nbsp;&nbsp; and 
world-wide network (that includes a selected group of<BR>&nbsp;&nbsp;&nbsp; 
professional and certified application programmers, 
website<BR>&nbsp;&nbsp;&nbsp; developers, graphic designers, etc. - from the 
USA, Canada,<BR>&nbsp;&nbsp;&nbsp; India, Romania, Japan, Australia, and 
others).</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT color=#000040><FONT face=Verdana size=2>&nbsp;&nbsp;&nbsp;<SPAN 
class=288043613-22102001> 
</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
</FONT><STRONG><FONT face=Verdana color=#ff0000 size=2>ALL OF OUR WORK IS 
PROVIDED<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
WITH A 100% SATISFACTION GUARANTEE!</FONT></STRONG></FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; So, if you're 
not satisfied with our work, ... you don't pay.<BR>&nbsp;&nbsp;&nbsp; It's as 
simple as that!</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; Please feel free 
to visit our website or email us at your<BR>&nbsp;&nbsp;&nbsp; convenience, and 
we will be happy to help you in any way<BR>&nbsp;&nbsp;&nbsp; 
possible.</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial 
size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana 
color=#000040 size=2>&nbsp;&nbsp;&nbsp; * Certain restrictions or requirements 
may apply to all work<BR>&nbsp;&nbsp;&nbsp; requests, free offers, etc.&nbsp; 
Please ask for complete details.</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV>
<DIV><FONT face=Verdana color=#000040 size=2>&nbsp;&nbsp;&nbsp; Free offers are 
also limited to one per website domain and/or<BR>&nbsp;&nbsp;&nbsp; 
company.&nbsp; Gary M. Gordon, LLC reserves the right to 
discontinue,<BR>&nbsp;&nbsp;&nbsp; change, or modify this free offer at any time 
without notice.</FONT></DIV>
<DIV><FONT face=Verdana color=#000040 size=2></FONT>&nbsp;</DIV><FONT face=Arial 
size=2></FONT>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT 
face=Arial size=2></FONT><BR>
</div>
<hr solid color="#336699" width="475" align="left">
<br>


If your browser doesn't support the display of HTML, you can access the same information  as shown above by following the link to take advange of our Free Website Enhancement Offer at: http://www.garymgordon.com/services.html
(then, on this page, look for the link that says "Free Current Promotional Offer).


<br>
<br>
</BODY></HTML>


<br>-------------------------------------------------------------------------------------<br>
<font face='arial,verdana' size='1' color='#336699'>If you have joined Gary M. Gordon, LLC by accident or someone else has joined you without your permission,<br>
or if you ever want to remove yourself from Gary M. Gordon, LLC, simply visit:<br>
<a href='http://www.garymgordon.com/easylist/easylist.cgi?action=unsubscribe&submitemail=python-dev@python.org'>http://www.garymgordon.com/easylist/easylist.cgi?action=unsubscribe&submitemail=python-dev@python.org</a><br>
and you will be automatically removed.</font><br>



From guido@python.org  Fri Oct 26 15:32:59 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 10:32:59 -0400
Subject: [Python-Dev] innocent question ;-)
In-Reply-To: Your message of "Fri, 26 Oct 2001 16:23:31 +0200."
 <20011026142331.C8E84303181@snelboot.oratrix.nl>
References: <20011026142331.C8E84303181@snelboot.oratrix.nl>
Message-ID: <200110261432.KAA18981@cj20424-a.reston1.va.home.com>

> At what point can we consider the API stable? All this inheriting
> and subtyping seems like it would be absolutely brilliant for the
> Mac toolbox APIs (and I suspect also for the PythonWin API's, Mark?)
> but I've refrained from even looking at it (well, I did have a quick
> peek:-), because (a) I have no time, (b) I don't want to chase a
> changing API and (c) there's some methodchain-based
> poor-mans-inheritance in there already that will undoubtedly break.

I don't plan to change the C API that was released in 2.2b1
incompatibly.  Besides, I won't have time to touch it after next Friday. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Fri Oct 26 15:45:13 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Fri, 26 Oct 2001 10:45:13 -0400
Subject: [Python-Dev] iterator support for SRE?
References: <014801c15cd6$3d18b270$ced241d5@hagrid>
 <200110250335.XAA10393@cj20424-a.reston1.va.home.com>
 <015a01c15e01$f99f4ef0$0900a8c0@spiff>
Message-ID: <15321.30457.83985.982168@anthem.wooz.org>

>>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:

    FL> as you might have noticed, I went ahead and checked in code
    FL> for "finditer".  nobody has complained this far ;-)

Please add something to Misc/NEWS file too!

-Barry


From martin@v.loewis.de  Fri Oct 26 19:31:50 2001
From: martin@v.loewis.de (Martin v. Loewis)
Date: Fri, 26 Oct 2001 20:31:50 +0200
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: <20011026102540.3DBEB303181@snelboot.oratrix.nl> (message from
 Jack Jansen on Fri, 26 Oct 2001 12:25:39 +0200)
References: <20011026102540.3DBEB303181@snelboot.oratrix.nl>
Message-ID: <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de>

> When the release branch is made there's usually a flurry of minor
> fixes, and these are almost guaranteed to break something on the
> mac. Tiny things, like missing casts, but breaks nonetheless.

While this may happen in principle, and out of curiosity: Was this a
problem in 2.2b1 also? Looking at the changes made on the 2.2b1
release branch, I see that a total of 7 files was changed. Except for
patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes
to C code.

So I can't see why the changes on the release branch could have any
effect on the Mac port, atleast in this release.

> Moreover, the "release loop" is now about 24 hours long, according
> to PEP101, and even extending it seriously (like to about a week)
> still wouldn't guarantee that I could react timely. Not only am I 6
> hours away from the unix/win release folks, but I also have a paying
> job that is less and less MacPython-related, so I can't firmly
> commit myself.

I'm not asking that you work harder on Python, I'm asking that you
work less :-) Seriously, I think there is a much larger set of people
testing the CVS regularly, so I doubt any breakage atleast on OS X
wouldn't be noticed within hours.

> And it has happened to me already (twice, I think) that there was a
> showstopper bug on the Mac that has caused me to either be very late
> with a release or skip one altogether. This happens on the Mac more
> often than on Unix/Windows, because MacPython relies on a third
> party unix I/O emulation library.

I sympathize with this problem, and I can't really suggest a good
solution to it. I'm also not so much concerned about concerned about
beta releases; nobody will care whether they can build 2.2b1 from the
sources on the Mac two months from now.

It's just that I think very strict principles must be applied for the
final release. If that means that the 2.2 release can't go without ok
from you (or somebody else who has produced MacOS 9 binaries), I think
we should add that to the release procedure. That check would be to
avoid show-stopper bugs only, of course - anything complex needs to be
detected long before the final release.

Regards,
Martin


From guido@python.org  Fri Oct 26 19:49:28 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 14:49:28 -0400
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Your message of "Fri, 26 Oct 2001 20:31:50 +0200."
 <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de>
References: <20011026102540.3DBEB303181@snelboot.oratrix.nl>
 <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de>
Message-ID: <200110261849.OAA20387@cj20424-a.reston1.va.home.com>


From guido@python.org  Fri Oct 26 19:56:20 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 14:56:20 -0400
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Your message of "Fri, 26 Oct 2001 20:31:50 +0200."
 <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de>
References: <20011026102540.3DBEB303181@snelboot.oratrix.nl>
 <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de>
Message-ID: <200110261856.OAA20424@cj20424-a.reston1.va.home.com>

> > When the release branch is made there's usually a flurry of minor
> > fixes, and these are almost guaranteed to break something on the
> > mac. Tiny things, like missing casts, but breaks nonetheless.
> 
> While this may happen in principle, and out of curiosity: Was this a
> problem in 2.2b1 also? Looking at the changes made on the 2.2b1
> release branch, I see that a total of 7 files was changed. Except for
> patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes
> to C code.

What's more common is actually a flurry of checkins just *before* the
release branch is made (say in the last two days).  These often
include a somewhat hasty commit of a project that's valuable to have
but introduces new bugs -- either Mac-specific or generic.  The patch
that introduced the missing \n\ problem was an example of this.

I don't know what to do about this.  Branching several days earlier
doesn't prevent it, because usually there's a really good reason to
copy the last-minute changes into the release branch.  (We don't have
the problem that some other projects appear to have, which is that
there's a lot of tentative development on the trunk -- then it would
make sense to branch earlier, but we tend to use the patch manager for
experiments.)

> It's just that I think very strict principles must be applied for the
> final release. If that means that the 2.2 release can't go without ok
> from you (or somebody else who has produced MacOS 9 binaries), I think
> we should add that to the release procedure. That check would be to
> avoid show-stopper bugs only, of course - anything complex needs to be
> detected long before the final release.

I think that for the final release we should definitely attempt this.
Hopefully the speed and quality of change will reduce closer to the
final release.  Also, the release candidate should help.

Jack, We'll try to work with you.  Barry, please add something to this
effect in the release procedure PEP (hold up the *final* release until
Jack approves or until we lose patience).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Fri Oct 26 20:13:48 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 26 Oct 2001 15:13:48 -0400
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
Message-ID: <3BD9B5EC.119319A5@interet.com>

The PEP for zip import is 273.  Please take a look and comment.

        http://python.sourceforge.net/peps/pep-0273.html

Jim Ahlstrom


From guido@python.org  Fri Oct 26 21:34:15 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 16:34:15 -0400
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Fri, 26 Oct 2001 15:13:48 EDT."
 <3BD9B5EC.119319A5@interet.com>
References: <3BD9B5EC.119319A5@interet.com>
Message-ID: <200110262034.QAA20802@cj20424-a.reston1.va.home.com>

> The PEP for zip import is 273.  Please take a look and comment.
> 
>         http://python.sourceforge.net/peps/pep-0273.html

OK, I'll shoot.  But I expect Gordon McMillan and Greg Stein to
provide more useful feedback.

|     Currently, sys.path is a list of directory names as strings.  If
|     this PEP is implemented, an item of sys.path can be a string
|     naming a zip file archive.  The zip archive can contain a
|     subdirectory structure to support package imports.  The zip
|     archive satisfies imports exactly as a subdirectory would.

I like this.

|     The implementation is in C code in the Python core and works on
|     all supported Python platforms.

This is good too, as it provides a bootstrap.  OTOH I also would like
to see a prototype in Python, using either ihooks or imputil.

|     Any files may be present in the zip archive, but only files *.pyc,
|     *.pyo and __init__.py[co] are available for import.  Zip import of
|     *.py and dynamic modules (*.pyd, *.so) is disallowed.
| 
|     Just as sys.path currently has default directory names, default
|     zip archive names are added too.  Otherwise there is no way to
|     import all Python library files from an archive.

More bootstrap goodness.

|     Reading compressed zip archives requires the zlib module.  An
|     import of zlib will be attempted prior to any other imports.  If
|     zlib is not available at that time, only uncompressed archives
|     will be readable, even if zlib subsequently becomes available.

Hm, I wonder if we couldn't just link with the libz.a C library and
use the C interface, if you're implementing this in C anyway.

| Subdirectory Equivalence
| 
|     The zip archive must be treated exactly as a subdirectory tree so
|     we can support package imports based on current and future rules.
|     Zip archive files must be created with relative path names.  That
|     is, archive file names are of the form: file1, file2, dir1/file3,
|     dir2/dir3/file4.
| 
|     Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip",
|     and we are trying to import modfoo from the Q package.  Then
|     import.c will generate a list of paths and extensions and will
|     look for the file.  The list of generated paths does not change
|     for zip imports.

(Very clever.)

                        Suppose import.c generates the path
|     "/A/B/SubDir/Q/R/modfoo.pyc".  Then it will also generate the path
|     "/C/D/E/Archive.zip/Q/R/modfoo.pyc".  Finding the SubDir path is
|     exactly equivalent to finding "Q/R/modfoo.pyc" in the archive.

Nice.

|     Suppose you zip up /A/B/SubDir/* and all its subdirectories.  Then
|     your zip file will satisfy imports just as your subdirectory did.
| 
|     Well, not quite.  You can't satisfy dynamic modules from a zip
|     file.  Dynamic modules have extensions like .dll, .pyd, and .so.
|     They are operating system dependent, and probably can't be loaded
|     except from a file.  It might be possible to extract the dynamic
|     module from the zip file, write it to a plain file and load it.
|     But that would mean creating temporary files, and dealing with all
|     the dynload_*.c, and that's probably not a good idea.

Agreed.

|     You also can't import source files *.py from a zip archive.  The
|     problem here is what to do with the compiled files.  Python would
|     normally write these to the same directory as *.py, but surely we
|     don't want to write to the zip file.  We could write to the
|     directory of the zip archive, but that would clutter it up, not
|     good if it is /usr/bin for example.  We could just fail to write
|     the compiled files, but that makes zip imports very slow, and the
|     user would probably not figure out what is wrong.  It is probably
|     best for users to put *.pyc into zip archives in the first place,
|     and this PEP enforces that rule.

I agree.  But it would still be good if the .py files were also in the
zip file, so the source can be used in tracebacks etc.  A C API to get
a source line from a filename might be a good idea (plus a Python API).

|     So the only imports zip archives support are *.pyc and *.pyo, plus
|     the import of __init__.py[co] for packages, and the search of the
|     subdirectory structure for the same.

I wonder if we need to make an additional rule that allows a .pyc file
to satisfy a module request even if we're in optimized mode (where
normally only .pyo files are searched).  Otherwise, if someone ships a
zipfile with only .pyc files, their modules can't be imported at all
when python -O is used.


| Efficiency
| 
|     The only way to find files in a zip archive is linear search.

But there's an index record at the end that provides quick access.

                                                                     So
|     for each zip file in sys.path, we search for its names once, and
|     put the names plus other relevant data into a static Python
|     dictionary.  The key is the archive name from sys.path joined with
|     the file name (including any subdirectories) within the archive.
|     This is exactly the name generated by import.c, and makes lookup
|     easy.

We could do this kind of pre-scanning for regular dictionaries on
sys.path too.  I found out very long ago (around '93 or '94) that this
saves a *lot* of startup time; I presume it still does.  (And even
more if the info can be cached in a file.)  The only problem is how to
detect when the cache becomes out of date.  Of course, you could say
"if you want faster startup time, put all your files in a zip
archive", and I couldn't really argue with that. :-)


| zlib
| 
|     Compressed zip archives require zlib for decompression.  Prior to
|     any other imports, we attempt an import of zlib, and set a flag if
|     it is available.  All compressed files are invisible unless this
|     flag is true.

Do we get an "module not found" error or something better, like
"compressed module found as <filename> but zlib unavailable"?

|     It could happen that zlib was available later.  For example, the
|     import of site.py might add the correct directory to sys.path so a
|     dynamic load succeeds.  But compressed files will still be
|     invisible.  It is unknown if it can happen that importing site.py
|     can cause zlib to appear, so maybe we're worrying about nothing.
|     On Windows and Linux, the early import of zlib succeeds without
|     site.py.

Yes, site.py isn't needed to make standard library modules available;
it's intended to make non-standare library modules available. :-)

|     The problem here is the confusion caused by the reverse.  Either a
|     zip file satisfies imports or it doesn't.  It is silly to say that
|     site.py needs to be uncompressed, and that maybe imports will
|     succeed later.  If you don't like this, create uncompressed zip
|     archives or make sure zlib is available, for example, as a
|     built-in module.  Or we can write special search logic during zip
|     initialization.

I don't think we need anything special here.  site.py shouldn't be
needed.


| Booting
| 
|     Python imports site.py itself, and this imports os, nt, ntpath,
|     stat, and UserDict.  It also imports sitecustomize.py which may
|     import more modules.  Zip imports must be available before site.py
|     is imported.
| 
|     Just as there are default directories in sys.path, there must be
|     one or more default zip archives too.
| 
|     The problem is what the name should be.  The name should be linked
|     with the Python version, so the Python executable can correctly
|     find its corresponding libraries even when there are multiple
|     Python versions on the same machine.
| 
|     This PEP suggests a zip archive name equal to the Python
|     interpreter path with extension ".zip" (eg, /usr/bin/python.zip)
|     which is always prepended to sys.path.  So a directory with python
|     and python.zip is complete.  This would work fine on Windows, as
|     it is common to put supporting files in the directory of the
|     executable.  But it may offend Unix fans, who dislike bin
|     directories being used for libraries.  It might be fine to
|     generate different defaults for Windows and Unix if necessary, but
|     the code will be in C, and there is no sense getting complicated.


Well, this is the domain of getpath.c, and that's got a different
implementation for Unix and Windows anyway (Windows has PC/getpathp.c).


| Implementation
| 
|     A C implementation exists which works, but which can be made better.

Upload as a patch please?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From bckfnn@worldonline.dk  Fri Oct 26 23:22:05 2001
From: bckfnn@worldonline.dk (Finn Bock)
Date: Fri, 26 Oct 2001 22:22:05 GMT
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BD9B5EC.119319A5@interet.com>
References: <3BD9B5EC.119319A5@interet.com>
Message-ID: <3bd9d419.10386565@mail.wanadoo.dk>

[James C. Ahlstrom]

>The PEP for zip import is 273.  Please take a look and comment.
>
>        http://python.sourceforge.net/peps/pep-0273.html

>    Any files may be present in the zip archive, but only files *.pyc,
>    *.pyo and __init__.py[co] are available for import.  Zip import of
>    *.py and dynamic modules (*.pyd, *.so) is disallowed.

Why can't .py be allowed? If a more recent .py[co] (or $py.class)
exists, it is used. Otherwise the .py file is compiled and discarded
when the process ends. Sure, it is slower, but a zip files with only
.py[co] entries would be of little use with jython.

>    Just as sys.path currently has default directory names, default
>    zip archive names are added too.  Otherwise there is no way to
>    import all Python library files from an archive.

A standard for this would be really cool.

>    Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip",
>    and we are trying to import modfoo from the Q package.  Then
>    import.c will generate a list of paths and extensions and will
>    look for the file.  The list of generated paths does not change
>    for zip imports.  Suppose import.c generates the path
>    "/A/B/SubDir/Q/R/modfoo.pyc".  Then it will also generate the path
>    "/C/D/E/Archive.zip/Q/R/modfoo.pyc".  Finding the SubDir path is
>    exactly equivalent to finding "Q/R/modfoo.pyc" in the archive.

So python packages and modules can exists *only* at the top level? That
would conflict somewhat with jython where at would be common to put
python modules into the same .zip file as java classes and java classes
also wants to own the root of a zip file.

In the current implementaion in jython, we can put python modules under
any path and add the zipfile!path to sys.path:

   sys.path.append("/path/to/zipfile.zip!Lib")

which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in
the archive.

>Efficiency
>    The only way to find files in a zip archive is linear search.  So
>    for each zip file in sys.path, we search for its names once, and
>    put the names plus other relevant data into a static Python
>    dictionary.  The key is the archive name from sys.path joined with
>    the file name (including any subdirectories) within the archive.
>    This is exactly the name generated by import.c, and makes lookup
>    easy.

[I found efficiency hard to achieve in jython because opening a zipfile
in java also cause the zip index to be read into a dictionary. So we
did not want to reopen a zipfile if it can be avoided. Instead we hide a
reference to the opened file in sys.path so that when removing a zipfile
name from sys.path, the file is eventually closed.]

Would entries in the static python dict be removed when a zipfile is
removed from sys.path? 

What is the __path__ vrbl set to in a module imported from a zipfile?
Can the module make changes to __path__ and will be changes to used when
importing submodules?

What value should __file__ have?

regards,
finn


From gerhard@bigfoot.de  Sat Oct 27 00:42:12 2001
From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=)
Date: Sat, 27 Oct 2001 01:42:12 +0200
Subject: [Python-Dev] Future of SSL
Message-ID: <20011027014211.A11092@lilith.hqd-internal>

First I'd like to say hello. I'm new to this list :-)

I've some concerns with the latest development wrt SSL development in
Python.

The SSL support in the Python core seems to have been added with Python
2.0, and Guido told me in a SF patch discussion, that Pythonlabs doesn't
know the code very well. Meanwhile it seems like Jeremy has tried fixing
bugs. I've also submitted several patches to the SSL code.

As you might know, there are several third-party SSL implementatations
for Python (m2crypto, pyOpenSSL, POW). I've contacted the maintainers of
these packages and asked them if they'd like to review my patches.

The two that answered told me that the reason they implemented their own
packages was that the SSL support in Python itself was very incomplete
and, to sum it up, Python's SSL is just plain broken.

The current state is that there is a single ssl() method in the socket
module that can be used to instantiate client-side SSL connections.
There's also a patch on Sourceforge that adds server-side SSL
functionality.

Several points why the current API is broken:
- no support for specifying the SSL protocol (SSL v2/SSL v3/SSL v2|3)
- no proper certicate checking
- SSL objects are neither compatible with sockets nor are they file-like
  objects
- no advanced features like SSL callbacks (to allow the user to accept a
  cerficicate, for example)

What I'd suggest for Python 2.2 is to *not* add any new features, like
server-side SSL but only accept bugfixes for the current client-side
code.

As the current implementation is broken and there is probably little SSL
knowledge in the Python core team, I propose to "outsource" the problem:

It should be possible to define a "Python SSL interface" that describes
an API for SSL. The various modules in Python that use SSL (urllib,
smtp, ...) would then be rewritten to use the new API. The
socketmodule.c would be rewritten to use the new API instead.

Then, wrappers could be written for the various SSL modules that wrap
them into the new "Python SSL interface" API.

As for me, I'm not an expert in SSL, but I'd be willing to try
coordinate the efforts, write a PEP, talk with the module maintainers
and such.

I'd be grateful to hear your opinions about this newbie proposal :-)

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))


From guido@python.org  Sat Oct 27 03:23:30 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 22:23:30 -0400
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Fri, 26 Oct 2001 22:22:05 GMT."
 <3bd9d419.10386565@mail.wanadoo.dk>
References: <3BD9B5EC.119319A5@interet.com>
 <3bd9d419.10386565@mail.wanadoo.dk>
Message-ID: <200110270223.WAA23737@cj20424-a.reston1.va.home.com>

> >    Just as sys.path currently has default directory names, default
> >    zip archive names are added too.  Otherwise there is no way to
> >    import all Python library files from an archive.
> 
> A standard for this would be really cool.

Yup.

> >    Suppose import.c generates the path
> >    "/A/B/SubDir/Q/R/modfoo.pyc".  Then it will also generate the path
> >    "/C/D/E/Archive.zip/Q/R/modfoo.pyc".  Finding the SubDir path is
> >    exactly equivalent to finding "Q/R/modfoo.pyc" in the archive.
> 
> So python packages and modules can exists *only* at the top level? That
> would conflict somewhat with jython where at would be common to put
> python modules into the same .zip file as java classes and java classes
> also wants to own the root of a zip file.
> 
> In the current implementaion in jython, we can put python modules under
> any path and add the zipfile!path to sys.path:
> 
>    sys.path.append("/path/to/zipfile.zip!Lib")
> 
> which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in
> the archive.

Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/"
in sys.path?  That sounds better than picking a random new character
as delimiter.

> [I found efficiency hard to achieve in jython because opening a zipfile
> in java also cause the zip index to be read into a dictionary. So we
> did not want to reopen a zipfile if it can be avoided. Instead we hide a
> reference to the opened file in sys.path so that when removing a zipfile
> name from sys.path, the file is eventually closed.]

I don't think Python has this problem, since we have control over the
zipfile reading code.

> Would entries in the static python dict be removed when a zipfile is
> removed from sys.path? 

It can be arranged that they are removed at some later point
(e.g. when sys.path is next searched).

> What is the __path__ vrbl set to in a module imported from a zipfile?
> Can the module make changes to __path__ and will be changes to used when
> importing submodules?
> 
> What value should __file__ have?

IMO these two questions are answered by the pathname semantics
that Jim proposes: __file__ = "/C/D/E/Archive.zip/Q/R/modfoo.pyc" and
__path__ = ["/C/D/E/Archive.zip/Q/R"].

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Sat Oct 27 03:26:38 2001
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Oct 2001 22:26:38 -0400
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Sat, 27 Oct 2001 01:42:12 +0200."
 <20011027014211.A11092@lilith.hqd-internal>
References: <20011027014211.A11092@lilith.hqd-internal>
Message-ID: <200110270226.WAA23767@cj20424-a.reston1.va.home.com>

> What I'd suggest for Python 2.2 is to *not* add any new features, like
> server-side SSL but only accept bugfixes for the current client-side
> code.

Sounds good to me.

> As the current implementation is broken and there is probably little SSL
> knowledge in the Python core team, I propose to "outsource" the problem:

Thanks!  We can sure use some help here.

> It should be possible to define a "Python SSL interface" that describes
> an API for SSL. The various modules in Python that use SSL (urllib,
> smtp, ...) would then be rewritten to use the new API. The
> socketmodule.c would be rewritten to use the new API instead.

I've just started digging in the socketmodule.c for a different
reason, and I propose to move all the SSL stuff to a separate file and
module.

> Then, wrappers could be written for the various SSL modules that wrap
> them into the new "Python SSL interface" API.

This is a good idea.  The DB API works like this.

> As for me, I'm not an expert in SSL, but I'd be willing to try
> coordinate the efforts, write a PEP, talk with the module maintainers
> and such.

But we do need *an* expert, don't we?  Maybe you can develop expertise
as you go?

> I'd be grateful to hear your opinions about this newbie proposal :-)

You don't sound much like a newbie. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From petrilli@amber.org  Sat Oct 27 04:55:27 2001
From: petrilli@amber.org (Christopher Petrilli)
Date: Fri, 26 Oct 2001 23:55:27 -0400
Subject: [Python-Dev] Future of SSL
In-Reply-To: <200110270226.WAA23767@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 26, 2001 at 10:26:38PM -0400
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com>
Message-ID: <20011026235527.A2919@trump.amber.org>

Guido van Rossum [guido@python.org] wrote:
> > Then, wrappers could be written for the various SSL modules that wrap
> > them into the new "Python SSL interface" API.
> 
> This is a good idea.  The DB API works like this.

I think that if OpenSSL is available, Python should build "out of the
box" with SSL support.  This is becomming more and more important with 
projects I'm working on, especially with SOAP and XML-RPC.  This
doesn't mean someone shouldn't be able to replace it, and we should
always define an API, but... I think we need to work out of the box.

> > As for me, I'm not an expert in SSL, but I'd be willing to try
> > coordinate the efforts, write a PEP, talk with the module maintainers
> > and such.
> 
> But we do need *an* expert, don't we?  Maybe you can develop expertise
> as you go?

I don't have time to provide code right now, but I do know SSL and
X.509 specifically inside and out and would be happy to provide
support from a standards/crypto/certificate perspective.

Chris
-- 
| Christopher Petrilli
| petrilli@amber.org


From guido@python.org  Sat Oct 27 16:56:34 2001
From: guido@python.org (Guido van Rossum)
Date: Sat, 27 Oct 2001 11:56:34 -0400
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Fri, 26 Oct 2001 23:55:27 EDT."
 <20011026235527.A2919@trump.amber.org>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com>
 <20011026235527.A2919@trump.amber.org>
Message-ID: <200110271556.LAA25312@cj20424-a.reston1.va.home.com>

> I think that if OpenSSL is available, Python should build "out of the
> box" with SSL support.  This is becomming more and more important with 
> projects I'm working on, especially with SOAP and XML-RPC.  This
> doesn't mean someone shouldn't be able to replace it, and we should
> always define an API, but... I think we need to work out of the box.

Good point.  That's how the SSL support is configured now, and that's
how it should continue to work.

(Note that, alas, the DB-API never made it this far -- we still
haven't been able to find the time to do the tedious work of moving
the various database adapters in the core distribution. :-( )

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Sat Oct 27 19:41:41 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 27 Oct 2001 14:41:41 -0400
Subject: [Python-Dev] dict comps
In-Reply-To: <3BD9268A.72AD71E9@lemburg.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEHDMDAA.tim.one@home.com>

[M.-A. Lemburg]
> ...
> Seriously, this goes down the path of lazy evaluation of expressions.

Lazy sequences specifically, but that's been a tension in Python all along
(it started with the for/__getitem__ protocol, and nothing we do from here
on in will ever be as much of a hack as xrange() was for that <0.9 wink>).

> Not sure whether this is the right path to follow though (can cause
> brain damage due to overloaded builtin debuggers, IMHO).

We can introduce "L[...]" for explicitly lazy list comprehenions <wink>.

> BTW, looks like I can finally get rid off the dict() builtin I
> have in mxTools which is Good News !

It's not quite the same in the details:  CVS dictionary(s) works just like
CVS

    d = {}
    for k, v in s:
        d[k] = v

In particular, it demands that the elements of s each produce exactly 2
objects, where IIRC the mxTools dict() requires at least 2 (ignoring any
after the second).  Was ignoring excess objects "a feature", or just
exposing an implementation that didn't want to bother to check?



From tim.one@home.com  Sat Oct 27 19:55:36 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 27 Oct 2001 14:55:36 -0400
Subject: [Python-Dev] iterator support for SRE?
In-Reply-To: <015a01c15e01$f99f4ef0$0900a8c0@spiff>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEHEMDAA.tim.one@home.com>

+1 on finditer.

[/F]
> ...
> will have to think about scanner; I agree that it may seem to
> be vaguely useful, but cannot think of a use case that isn't
> better handled by findall/split/sub or iterfind.  can anyone?

It's in desperate need of usage docs.  The few times I recall seeing it
mentioned on c.l.py, it was from someone staring at the code unable to
reverse-engineer its intended use.  Therefore I don't believe it's gotten a
good trial, and so I'm at best -0 on making it official now.  In my own
stuff I've built directly on top of .lastindex/.lastgroup instead -- perhaps
only because it's easier to remember how to use your own undocumented stuff
than somebody else's <wink>.

the-folks-most-likely-to-use-a-framework-are-those-most-likely-to-
    write-their-own-ly y'rs  - tim



From tim.one@home.com  Sat Oct 27 20:45:03 2001
From: tim.one@home.com (Tim Peters)
Date: Sat, 27 Oct 2001 15:45:03 -0400
Subject: [Python-Dev] Idle speed freak looking for trouble?
Message-ID: <LNBBLJKPBEHFEDALKOLCAEHHMDAA.tim.one@home.com>

SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown
from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1.  Also major
slowdown from enabling gc.  I *believe* Frederic ran it on Tru64.

no-time-here-ly y'rs  - tim



From fredrik@pythonware.com  Sat Oct 27 21:45:36 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sat, 27 Oct 2001 22:45:36 +0200
Subject: [Python-Dev] Idle speed freak looking for trouble?
References: <LNBBLJKPBEHFEDALKOLCAEHHMDAA.tim.one@home.com>
Message-ID: <004501c15f28$56148340$ced241d5@hagrid>

tim wrote:


> SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown
> from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1.

fwiw, here's the timings I get on a slow windows box (using
PythonWare builds, and time.clock() instead of os.times()):

python 1.5: 2.01
python 2.0: 3.02
python 2.1: 3.00
python 2.2: 3.04

after playing some more, I ended up with this little benchmark:

def dosomething(item):
    return [None, None, None]

data = [None] * 100000

import time
t1 = time.clock();
x = map(dosomething, data)
t2 = time.clock()
print t2 - t1

python 1.5: 1.02 seconds
python 2.0: 1.95 seconds
python 2.1: 1.85 seconds
python 2.2: 1.90 seconds

</F>



From mal@lemburg.com  Sat Oct 27 21:38:03 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sat, 27 Oct 2001 22:38:03 +0200
Subject: [Python-Dev] Idle speed freak looking for trouble?
References: <LNBBLJKPBEHFEDALKOLCAEHHMDAA.tim.one@home.com>
Message-ID: <3BDB1B2B.8705CCCE@lemburg.com>

Tim Peters wrote:
> 
> SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown
> from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1.  Also major
> slowdown from enabling gc.  I *believe* Frederic ran it on Tru64.

... want me to run pybench against the different versions ? 

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From mal@lemburg.com  Sat Oct 27 21:53:04 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sat, 27 Oct 2001 22:53:04 +0200
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com>
Message-ID: <3BDB1EB0.C2A9B1FF@lemburg.com>

"James C. Ahlstrom" wrote:
> 
> The PEP for zip import is 273.  Please take a look and comment.
> 
>         http://python.sourceforge.net/peps/pep-0273.html

Looks good to me. Just three questions:

1. Why are .py files being outlawed ?

2. Where's the C implementation you mention in the PEP ?

3. Would it be possible to ship zlib together with Python ?
   (the zlib license should allow this and I don't think that
   the code size is too big)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From mal@lemburg.com  Sat Oct 27 21:46:04 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sat, 27 Oct 2001 22:46:04 +0200
Subject: [Python-Dev] dict comps
References: <LNBBLJKPBEHFEDALKOLCMEHDMDAA.tim.one@home.com>
Message-ID: <3BDB1D0C.A60F2EF@lemburg.com>

Tim Peters wrote:
> 
> [M.-A. Lemburg]
> > ...
> > Seriously, this goes down the path of lazy evaluation of expressions.
> 
> Lazy sequences specifically, but that's been a tension in Python all along
> (it started with the for/__getitem__ protocol, and nothing we do from here
> on in will ever be as much of a hack as xrange() was for that <0.9 wink>).
> 
> > Not sure whether this is the right path to follow though (can cause
> > brain damage due to overloaded builtin debuggers, IMHO).
> 
> We can introduce "L[...]" for explicitly lazy list comprehenions <wink>.

... and add a lazy list comprehension object (e.g. xlistcompobj) ?
Cool :-)
 
> > BTW, looks like I can finally get rid off the dict() builtin I
> > have in mxTools which is Good News !
> 
> It's not quite the same in the details:  CVS dictionary(s) works just like
> CVS
> 
>     d = {}
>     for k, v in s:
>         d[k] = v
> 
> In particular, it demands that the elements of s each produce exactly 2
> objects, where IIRC the mxTools dict() requires at least 2 (ignoring any
> after the second).  Was ignoring excess objects "a feature", or just
> exposing an implementation that didn't want to bother to check?

It was a feature; it's just that I forgot what I needed it for ;-)

There are quite a few things in mxTools which I've never *really*
needed. The single most used API in the package was and still is
irange() which returns a sequence (i, obj[i]). dict() and invdict()
are also rather popular ones. Most of the others tend to solve
performance problems in some inner loops of some applications
I wrote.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From fredrik@pythonware.com  Sat Oct 27 22:21:56 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sat, 27 Oct 2001 23:21:56 +0200
Subject: [Python-Dev] Idle speed freak looking for trouble?
Message-ID: <006201c15f2d$6b986100$ced241d5@hagrid>

I wrote:

> python 1.5: 1.02 seconds
> python 2.0: 1.95 seconds
> python 2.1: 1.85 seconds
> python 2.2: 1.90 seconds

doh.  forgot to disable GC for 2.0 and later.  with GC switched
off, the difference is a bit smaller.

python 1.5: 1.02 seconds
python 2.0: 1.30 seconds
python 2.1: 1.20 seconds
python 2.2: 1.05 seconds

now replace [None, None, None] with [item, item, item]:

python 1.5: 0.58 seconds
python 2.0: 0.85 seconds
python 2.1: 0.69 seconds
python 2.2: 0.71 seconds

playing with other dosomething bodies (and with GC switched
off), I've noticed that the following things has gotten slower
from 1.5.2 to 2.2:

- concatenating strings (item+item+item is 20% slower)
- calling the float builtin using global lookup (20% slower)
- list and tuple forming using locals (10-20% slower)

</F>



From petrilli@amber.org  Sun Oct 28 01:43:58 2001
From: petrilli@amber.org (Christopher Petrilli)
Date: Sat, 27 Oct 2001 21:43:58 -0400
Subject: [Python-Dev] Future of SSL
In-Reply-To: <200110271556.LAA25312@cj20424-a.reston1.va.home.com>; from guido@python.org on Sat, Oct 27, 2001 at 11:56:34AM -0400
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com>
Message-ID: <20011027214357.B6993@trump.amber.org>

Guido van Rossum [guido@python.org] wrote:
> > I think that if OpenSSL is available, Python should build "out of the
> > box" with SSL support.  This is becomming more and more important with 
> > projects I'm working on, especially with SOAP and XML-RPC.  This
> > doesn't mean someone shouldn't be able to replace it, and we should
> > always define an API, but... I think we need to work out of the box.
> 
> Good point.  That's how the SSL support is configured now, and that's
> how it should continue to work.

Perhaps there is one of the existing modules (M2Crypto?) that can be
integrated, assuming licensing issues can be resolved. Also, I think
that perhaps high level abstractions should be introduced, though I'm
not sure what they are now... that's just hand waving.

The initial goal in my mind would be to have transparent (or nearly
so) SSL session management, but the X.509 manipulation and general
crypto interface could wait until later.  While they are both useful,
the SSL side is the really critical part.

> (Note that, alas, the DB-API never made it this far -- we still
> haven't been able to find the time to do the tedious work of moving
> the various database adapters in the core distribution. :-( )

Actually what this looks more like is not just SSL, but a "crypto"
collection for Python, dependent on the excellent work in OpenSSL.  I
can start outlining some stuff if that would be a good start, though
obviously if there's an existing library that could be integrated,
that would be excellent.

Chris
-- 
| Christopher Petrilli
| petrilli@amber.org


From michel@zope.com  Sun Oct 28 02:02:42 2001
From: michel@zope.com (Michel Pelletier)
Date: Sat, 27 Oct 2001 19:02:42 -0700
Subject: [Python-Dev] Future of SSL
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org>
Message-ID: <3BDB6742.E143BF8B@zope.com>

Christopher Petrilli wrote:
>
> Actually what this looks more like is not just SSL, but a "crypto"
> collection for Python, dependent on the excellent work in OpenSSL.  I
> can start outlining some stuff if that would be a good start, though
> obviously if there's an existing library that could be integrated,
> that would be excellent.

AMK has such a package:

http://www.amk.ca/python/code/crypto.html

which served as an exellent introduction to the world of crypto for me.

-Michel


From guido@python.org  Sun Oct 28 12:20:30 2001
From: guido@python.org (Guido van Rossum)
Date: Sun, 28 Oct 2001 07:20:30 -0500
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Sat, 27 Oct 2001 21:43:58 EDT."
 <20011027214357.B6993@trump.amber.org>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com>
 <20011027214357.B6993@trump.amber.org>
Message-ID: <200110281220.HAA09692@cj20424-a.reston1.va.home.com>

> The initial goal in my mind would be to have transparent (or nearly
> so) SSL session management, but the X.509 manipulation and general
> crypto interface could wait until later.  While they are both useful,
> the SSL side is the really critical part.

Which is of course exactly the goal of the existing socket.ssl
support, primitive though it is.  This was paid for in part by HP;
their goal was to let urllib support the HTTPS protocol, nothing more,
nothing less.  And it does that well enough.

> > (Note that, alas, the DB-API never made it this far -- we still
> > haven't been able to find the time to do the tedious work of moving
> > the various database adapters in the core distribution. :-( )
> 
> Actually what this looks more like is not just SSL, but a "crypto"
> collection for Python, dependent on the excellent work in OpenSSL.  I
> can start outlining some stuff if that would be a good start, though
> obviously if there's an existing library that could be integrated,
> that would be excellent.

I think it would be a big mistake to start from scratch.  Remember the
GUI SIG?

PS.  One issue with adding more crypto to Python could be US export
issues.  It's possible that new export limitations for crypto software
are made law by a congress that doesn't understand the issues, and
then the US Python distribution could be in trouble (even though our
site in the the Netherlands, we build the distributions here in the
US).  Back at CNRI, we couldn't release the SSL wrappers, which don't
contain any crypto code but enable linking with it, before an
extensive and expensive legal review, and then we had to wait until
after a certain date, at which some of the crypto export restrictions
were lifted.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From bckfnn@worldonline.dk  Sun Oct 28 14:30:30 2001
From: bckfnn@worldonline.dk (Finn Bock)
Date: Sun, 28 Oct 2001 14:30:30 GMT
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <200110270223.WAA23737@cj20424-a.reston1.va.home.com>
References: <3BD9B5EC.119319A5@interet.com>   <3bd9d419.10386565@mail.wanadoo.dk>  <200110270223.WAA23737@cj20424-a.reston1.va.home.com>
Message-ID: <3bdbfda4.3697676@mail.wanadoo.dk>

[me]

> In the current implementaion in jython, we can put python modules under
> any path and add the zipfile!path to sys.path:
> 
>    sys.path.append("/path/to/zipfile.zip!Lib")
> 
> which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in
> the archive.

[GvR]

>Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/"
>in sys.path?  That sounds better than picking a random new character
>as delimiter.

Fine by me. From a java POV the bang ("!") was not random:

>>> import java
>>> java.lang.Class.getResource(type(1), "/java/lang/Class.class")
jar:file:/D:/java/jdk1.4/jre/lib/rt.jar!/java/lang/Class.class
>>>

The URL library in java can of course open the string above as an input
stream.

>> Would entries in the static python dict be removed when a zipfile is
>> removed from sys.path? 
>
>It can be arranged that they are removed at some later point
>(e.g. when sys.path is next searched).

An API to do this could be usefull for jython. Right now we depend on
the GC thread to close the file. Since files are a limited resource it
would be a good thing to have an explicit way to clean up the cached
resources.

I don't expect the lack of a cleanup method to be a huge problem in real
life.

>> What is the __path__ vrbl set to in a module imported from a zipfile?
>> Can the module make changes to __path__ and will be changes to used when
>> importing submodules?
>> 
>> What value should __file__ have?
>
>IMO these two questions are answered by the pathname semantics
>that Jim proposes: __file__ = "/C/D/E/Archive.zip/Q/R/modfoo.pyc" and
>__path__ = ["/C/D/E/Archive.zip/Q/R"].

Ok.

I assume that as long as some module exists with a __path__ like this,
it not possible to clear the cached entries for /C/D/E/Archive.zip.

If importing from zip is regarded mainly as a deployment/bootstrapping
feature, then the cleanup issues does not exist. I have no problem
looking at it as a deployment feature (that is all I need it for
myself), I just didn't dare to put such a limit on my jython
implementation.

regards,
finn


From andymac@bullseye.apana.org.au  Sun Oct 28 07:41:46 2001
From: andymac@bullseye.apana.org.au (Andrew MacIntyre)
Date: Sun, 28 Oct 2001 18:41:46 +1100 (EDT)
Subject: [Python-Dev] status of OS/2 ports of Python
In-Reply-To: <200110252130.f9PLUM329494@odiug.zope.com>
Message-ID: <Pine.OS2.4.32.0110281805420.605-100000@central>

On Thu, 25 Oct 2001, Guido van Rossum wrote:

> > At this stage, my plan is to finalise a binary release of 2.2 before
> > submitting a new set of patches, taking into account Martin von Loewis'
> > comments and suggestions (much appreciated thank you!) among others.  I
> > also plan to try and coordinate my patches with those of Michael Muller
> > (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2.
> >
> > I will try and review Michael's patches with a view to getting them into
> > 2.2, as the VAC++ build has been part of the sourceball for a few
> > versions.
>
> Andy, would it help if you had SourceForge commit privileges?  You'd
> be obliged to only check in code that doesn't break the build on other
> platforms, so you would be required to test anything you want to check
> in on another platform (Linux or Windows) before you commit to save
> you the embarrassment of breaking the build, but if (as I expect) you
> would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a
> big burden.  It would save you the trouble of uploading patches to
> SourceForge and it would save *us* the trouble of reviewing your
> patches and checking them in.

I've never used CVS, but am prepared to rectify that.  I am also prepared
to work within the commit rules you specify.

I can work with the requirement for testing on another platform if FreeBSD
is an acceptable test platform.

I would also propose to look after patches to the VAC++ build (such as
Michael Muller's), on the basis that although I can't test the patches
directly, I can test that they don't affect the rest of the system, and I
can work with the submitter to resolve issues that affect OS/2 specific
functionality.

As to whether having commit privs would help - on the time availability
front, no; otherwise more likely than not.

> All your code would be owned by the PSF, but that's what you want
> anyway, right?

Yes.

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
        andymac@pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia



From gerhard@bigfoot.de  Sun Oct 28 17:40:25 2001
From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=)
Date: Sun, 28 Oct 2001 18:40:25 +0100
Subject: [Python-Dev] Future of SSL
In-Reply-To: <3BDB6742.E143BF8B@zope.com>; from michel@zope.com on Sat, Oct 27, 2001 at 07:02:42PM -0700
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com>
Message-ID: <20011028184024.A1066@lilith.hqd-internal>

On Sat, Oct 27, 2001 at 07:02:42PM -0700, Michel Pelletier wrote:
> Christopher Petrilli wrote:
> >
> > Actually what this looks more like is not just SSL, but a "crypto"
> > collection for Python, dependent on the excellent work in OpenSSL.  I
> > can start outlining some stuff if that would be a good start, though
> > obviously if there's an existing library that could be integrated,
> > that would be excellent.
> 
> AMK has such a package:
> 
> http://www.amk.ca/python/code/crypto.html

It doesn't offer SSL, however. I've compiled a list of SSL libraries for
Python:

http://www.cs.fhm.edu/~ifw00065/pyssl/

The "Python OpenSSL wrappers" are BSD licensed and could probably serve
as a good basis for a SSL library in Python itself. It's an early
release, that I can crash easily if I want to ;-) Also, it would have to
be modified to fit Python's coding standards (basically, the docstrings
are all written in DocBook; these would have to be translated to normal
text). But it's a working solution to start from.

The other BSD licensed one is M2Crypto, which is around much longer, but
requires SWIG to build. I think SSL support is not reason enough to
require SWIG for Python builds. So this one is possibly out. The third
one is LGPL licensed, so it's out immediately.

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))


From gerhard@bigfoot.de  Sun Oct 28 17:52:51 2001
From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=)
Date: Sun, 28 Oct 2001 18:52:51 +0100
Subject: [Python-Dev] Future of SSL
In-Reply-To: <20011027214357.B6993@trump.amber.org>; from petrilli@amber.org on Sat, Oct 27, 2001 at 09:43:58PM -0400
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org>
Message-ID: <20011028185250.B1066@lilith.hqd-internal>

On Sat, Oct 27, 2001 at 09:43:58PM -0400, Christopher Petrilli wrote:
> Guido van Rossum [guido@python.org] wrote:
> > > I think that if OpenSSL is available, Python should build "out of the
> > > box" with SSL support.  This is becomming more and more important with 
> > > projects I'm working on, especially with SOAP and XML-RPC.  This
> > > doesn't mean someone shouldn't be able to replace it, and we should
> > > always define an API, but... I think we need to work out of the box.
> > 
> > Good point.  That's how the SSL support is configured now, and that's
> > how it should continue to work.

Ok. I understand completely "outsourcing" SSL is not an option. So we
either build a completely new SSL module or try to integrate an existing
one.

> Perhaps there is one of the existing modules (M2Crypto?) that can be
> integrated, assuming licensing issues can be resolved.

Yup. To save you time finding them all, I've summarized them and put up
a page about them (cf. my other post).

> [...] The initial goal in my mind would be to have transparent (or nearly
> so) SSL session management, [...]

I'm not sure I understand what you mean by transparent session
management. Perhaps that one important feature would be that SSL objects
be interface compatible with socket objects as much as possible?  So
ugly hacks like FakeSocket in httplib and SSLFakeSocket in smtplib are
no longer necessary.

And, btw. one complaint about socketmodule.c I've heard is that it
doesn't have a C API, it might be necessary to expose some of it with
the help of a header file.

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))


From tim.one@home.com  Mon Oct 29 03:14:44 2001
From: tim.one@home.com (Tim Peters)
Date: Sun, 28 Oct 2001 22:14:44 -0500
Subject: [Python-Dev] status of OS/2 ports of Python
In-Reply-To: <Pine.OS2.4.32.0110281805420.605-100000@central>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEKBMDAA.tim.one@home.com>

[Andrew MacIntyre, to Guido]
> I've never used CVS, but am prepared to rectify that.  I am also prepared
> to work within the commit rules you specify.
>
> I can work with the requirement for testing on another platform if FreeBSD
> is an acceptable test platform.

Yes, FreeBSD is fine.  It's "Unix-like", and random collateral damage there
is mostly what we're trying to guard against, as most people building from
CVS are building on Unix-like systems.

> I would also propose to look after patches to the VAC++ build (such as
> Michael Muller's), on the basis that although I can't test the patches
> directly, I can test that they don't affect the rest of the system, and I
> can work with the submitter to resolve issues that affect OS/2 specific
> functionality.

That would be wonderful.  I don't know what to do with OS/2 patches,
although since I'm "the Windows guy", people look at me like I'm *supposed*
to know something about OS/2 <wink>.  I don't.  If you and Michael can
coordinate, it would help Python on OS/2 a lot.

> As to whether having commit privs would help - on the time availability
> front, no; otherwise more likely than not.

It would be more time-efficient for you (and for us).  Waiting for somebody
else to review your patches can drag on for-- literally --months, at which
point the patch likely doesn't even apply anymore.  Checking in directly
from a build tree saves a whole lot of time.

take-it-easy-on-#ifdefs-outside-of-pyport.h-ly y'rs  - tim



From guido@python.org  Mon Oct 29 07:30:13 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 02:30:13 -0500
Subject: [Python-Dev] status of OS/2 ports of Python
In-Reply-To: Your message of "Sun, 28 Oct 2001 22:14:44 EST."
 <LNBBLJKPBEHFEDALKOLCIEKBMDAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCIEKBMDAA.tim.one@home.com>
Message-ID: <200110290730.CAA11349@cj20424-a.reston1.va.home.com>

> > As to whether having commit privs would help - on the time availability
> > front, no; otherwise more likely than not.
> 
> It would be more time-efficient for you (and for us).  Waiting for somebody
> else to review your patches can drag on for-- literally --months, at which
> point the patch likely doesn't even apply anymore.  Checking in directly
> from a build tree saves a whole lot of time.

Plus, if you screw up, you have instant review. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 08:15:37 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 03:15:37 -0500
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Sun, 28 Oct 2001 18:52:51 +0100."
 <20011028185250.B1066@lilith.hqd-internal>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org>
 <20011028185250.B1066@lilith.hqd-internal>
Message-ID: <200110290815.DAA11519@cj20424-a.reston1.va.home.com>

> And, btw. one complaint about socketmodule.c I've heard is that it
> doesn't have a C API, it might be necessary to expose some of it with
> the help of a header file.

What would you need from a C API?  Socket objects currently are pretty
thin wrappers around file descriptors.  I'm not against this (although
defining and using an external C API for a Python extension module is
a bit cumbersome due to the requirement for dynamic loading), but I'd
like to see the requirements first.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 08:19:41 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 03:19:41 -0500
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Sun, 28 Oct 2001 18:40:25 +0100."
 <20011028184024.A1066@lilith.hqd-internal>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com>
 <20011028184024.A1066@lilith.hqd-internal>
Message-ID: <200110290819.DAA11553@cj20424-a.reston1.va.home.com>

> > AMK has such a package:
> > 
> > http://www.amk.ca/python/code/crypto.html
> 
> It doesn't offer SSL, however. I've compiled a list of SSL libraries for
> Python:
> 
> http://www.cs.fhm.edu/~ifw00065/pyssl/
> 
> The "Python OpenSSL wrappers" are BSD licensed and could probably serve
> as a good basis for a SSL library in Python itself. It's an early
> release, that I can crash easily if I want to ;-) Also, it would have to
> be modified to fit Python's coding standards (basically, the docstrings
> are all written in DocBook; these would have to be translated to normal
> text). But it's a working solution to start from.

Do you know the author?

> The other BSD licensed one is M2Crypto, which is around much longer, but
> requires SWIG to build. I think SSL support is not reason enough to
> require SWIG for Python builds. So this one is possibly out.

It's definitely out to require SWIG to build Python from a
distribution, but we may require SWIG to hack on this particular
module; we could ship the SWIG output.  Not ideal, but if otherwise
this module is the best candidate, maybe we could live with this.

> The third
> one is LGPL licensed, so it's out immediately.

Really?  I believe the LGPL is not "viral" as the GPL.  Also, there
may be the possibility to shame the author into changing the
license.  Can you review it a bit more technically?

--Guido van Rossum (home page: http://www.python.org/~guido/)



From guido@python.org  Mon Oct 29 08:31:16 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 03:31:16 -0500
Subject: [Python-Dev] status of OS/2 ports of Python
In-Reply-To: Your message of "Sun, 28 Oct 2001 18:41:46 +1100."
 <Pine.OS2.4.32.0110281805420.605-100000@central>
References: <Pine.OS2.4.32.0110281805420.605-100000@central>
Message-ID: <200110290831.DAA11598@cj20424-a.reston1.va.home.com>

> > Andy, would it help if you had SourceForge commit privileges?  You'd
> > be obliged to only check in code that doesn't break the build on other
> > platforms, so you would be required to test anything you want to check
> > in on another platform (Linux or Windows) before you commit to save
> > you the embarrassment of breaking the build, but if (as I expect) you
> > would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a
> > big burden.  It would save you the trouble of uploading patches to
> > SourceForge and it would save *us* the trouble of reviewing your
> > patches and checking them in.
> 
> I've never used CVS, but am prepared to rectify that.

Be prepared for some long evenings of fun with CVS!  Lots of folks
here on python-dev have developed a deep understanding of CVS theory
and tricks though, so you're in a good place to learn.

> I am also prepared
> to work within the commit rules you specify.

I didn't doubt this for a second.

> I can work with the requirement for testing on another platform if FreeBSD
> is an acceptable test platform.

Sure is -- in fact, we could use more FreeBSD testing. :-)

> I would also propose to look after patches to the VAC++ build (such as
> Michael Muller's), on the basis that although I can't test the patches
> directly, I can test that they don't affect the rest of the system, and I
> can work with the submitter to resolve issues that affect OS/2 specific
> functionality.

That sounds like a very good idea.  Right now I don't really know what
to do with the two sets of OS/2 patches...  Since Michael is cc'ed
here, Michael, what do you think of this?

> As to whether having commit privs would help - on the time availability
> front, no; otherwise more likely than not.

OK.  As soon as you think you're ready to commit anything (try
something small first :-), post a message here and one of the admins
will add you.  (You need to have a SF login first.)  Not me, I'm about
to go underground for a while to take care of my wife and son (to be
born by Friday).

> > All your code would be owned by the PSF, but that's what you want
> > anyway, right?
> 
> Yes.

Good!  The PSF legal team will eventually come up with paperwork to
make this official; in the mean time, this archived email message is
good enough for me.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 08:37:00 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 03:37:00 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Sun, 28 Oct 2001 14:30:30 GMT."
 <3bdbfda4.3697676@mail.wanadoo.dk>
References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com>
 <3bdbfda4.3697676@mail.wanadoo.dk>
Message-ID: <200110290837.DAA11621@cj20424-a.reston1.va.home.com>

> [GvR]
> 
> >Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/"
> >in sys.path?  That sounds better than picking a random new character
> >as delimiter.

[Finn]
> Fine by me. From a java POV the bang ("!") was not random:
> 
> >>> import java
> >>> java.lang.Class.getResource(type(1), "/java/lang/Class.class")
> jar:file:/D:/java/jdk1.4/jre/lib/rt.jar!/java/lang/Class.class
> >>>
> 
> The URL library in java can of course open the string above as an input
> stream.

I guess the advantage of this notation is that it makes a simple check
for "file inside zip archive" possible; I sort of like that.  The
downside is that it limits the use of "!" for other reasons in
pathnames (which seems a mostly but not entirely theoretical problem).

> I assume that as long as some module exists with a __path__ like this,
> it not possible to clear the cached entries for /C/D/E/Archive.zip.

Good point.

> If importing from zip is regarded mainly as a deployment/bootstrapping
> feature, then the cleanup issues does not exist. I have no problem
> looking at it as a deployment feature (that is all I need it for
> myself), I just didn't dare to put such a limit on my jython
> implementation.

I don't see sys.path as a very dynamic thing anyway.  It gets
manipulated briefly at the beginning, and then maybe, rarely, stuff
gets added during the program run.  I've seen temporary additions, but
these were usually in the test suite.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Mon Oct 29 08:36:41 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 29 Oct 2001 09:36:41 +0100
Subject: [Python-Dev] Future of SSL
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal>
Message-ID: <3BDD1519.31496FB@lemburg.com>

[Integrating more crypto code into Python]

Please don't proceed in this direction. Crypto code has a long
tradition of making world-wide distribution hard and painful,
either to export, to import or to usage restrictions.

There's a good reason why e.g. mxCrypto was developed outside the
US, in fact I wrote that package because I couldn't legally export
Andrew's code from the US. OpenSSL itself was developed in large 
parts in Australia for much the same reason.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From mal@lemburg.com  Mon Oct 29 08:49:22 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 29 Oct 2001 09:49:22 +0100
Subject: [Python-Dev] Adding DB modules to the core ? (Re: Future of SSL)
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com>
 <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com>
Message-ID: <3BDD1812.1FEF2E33@lemburg.com>

Guido van Rossum wrote:
> 
> (Note that, alas, the DB-API never made it this far -- we still
> haven't been able to find the time to do the tedious work of moving
> the various database adapters in the core distribution. :-( )

Do you really think that this would be the right approach ?
Sounds like code bloat to me... 

BTW, I'm still waiting for one of the PSF sponsors to create 
the mythical Sumo distribution of Python. Any news in that
direction ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From jack@oratrix.nl  Mon Oct 29 10:02:04 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Mon, 29 Oct 2001 11:02:04 +0100
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Message by Guido van Rossum <guido@python.org> ,
 Fri, 26 Oct 2001 14:56:20 -0400 , <200110261856.OAA20424@cj20424-a.reston1.va.home.com>
Message-ID: <20011029100204.E0766303181@snelboot.oratrix.nl>

> > While this may happen in principle, and out of curiosity: Was this a
> > problem in 2.2b1 also? Looking at the changes made on the 2.2b1
> > release branch, I see that a total of 7 files was changed. Except for
> > patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes
> > to C code.
> 
> What's more common is actually a flurry of checkins just *before* the
> release branch is made (say in the last two days).  These often
> include a somewhat hasty commit of a project that's valuable to have
> but introduces new bugs -- either Mac-specific or generic.  The patch
> that introduced the missing \n\ problem was an example of this.

And, more serious, there's another example for the 2.2b1 release that I found 
just friday (so this bug is going to stay in MacPython 2.2b1, I'm afraid): 
importing packages from frozen programs is broken.

There was a moderately simple fix to import.c, to test that submodules of 
frozen modules were also frozen, and not picked up from the filesystem. 
Unfortunately whoever did the patch wasn't aware that MacPython frozen modules 
work different, and hence broke it.
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From jack@oratrix.nl  Mon Oct 29 10:10:16 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Mon, 29 Oct 2001 11:10:16 +0100
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Message by Guido van Rossum <guido@python.org> ,
 Fri, 26 Oct 2001 14:56:20 -0400 , <200110261856.OAA20424@cj20424-a.reston1.va.home.com>
Message-ID: <20011029101016.E80A6303181@snelboot.oratrix.nl>

> > It's just that I think very strict principles must be applied for the
> > final release. If that means that the 2.2 release can't go without ok
> > from you (or somebody else who has produced MacOS 9 binaries), I think
> > we should add that to the release procedure. That check would be to
> > avoid show-stopper bugs only, of course - anything complex needs to be
> > detected long before the final release.
> 
> I think that for the final release we should definitely attempt this.
> Hopefully the speed and quality of change will reduce closer to the
> final release.  Also, the release candidate should help.

For the final release I agree that this is a good idea. Let's try and make it 
so.
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From just@letterror.com  Mon Oct 29 10:24:33 2001
From: just@letterror.com (Just van Rossum)
Date: Mon, 29 Oct 2001 11:24:33 +0100
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: <20011029100204.E0766303181@snelboot.oratrix.nl>
Message-ID: <20011029112439-r01010800-90687d33-0910-010c@10.0.0.23>

Jack Jansen wrote:

> And, more serious, there's another example for the 2.2b1 release that I found 
> just friday (so this bug is going to stay in MacPython 2.2b1, I'm afraid): 
> importing packages from frozen programs is broken.
> 
> There was a moderately simple fix to import.c, to test that submodules of 
> frozen modules were also frozen, and not picked up from the filesystem. 
> Unfortunately whoever did the patch wasn't aware that MacPython frozen modules 
> work different, and hence broke it.

I remember seeing that, and meaning to look into it but forgot. I think it's a
highly questionable fix, not only for MacPython: I'm not sure if this still
allows C extensions as submodules in frozen apps.

Just


From chimaadams@yahoo.com  Mon Oct 29 10:42:53 2001
From: chimaadams@yahoo.com (chima adams)
Date: Mon, 29 Oct 2001 11:42:53 +0100
Subject: [Python-Dev] (no subject)
Message-ID: <E15y9wC-0000a2-00@mail.python.org>

FROM THE DESK:Dr.CHIMA ADAMS,
ACCOUNT DEPARTMENT (NNPC)
FALOMO OFFICE COMPLEX,
IKOYI – LAGOS, NIGERIA
TEL:234-8033084103
FAX: 234-1-7599186                                                                               


ATTN: GENERAL MANAGER

Dear Sir,

I am Dr.Chima Adams chairman- audit committee, with the Nigerian National Petroleum Corporation (NNPC), I  headed the audit committee whose assignment was to audit, review and recommend payment for all contracts awarded in the NNPC by the corrupt past military administration. 
In the course of our review, we discovered that the sum of USD 30 million (THIRTY MILLION DOLLARS) was floating unclaimed in the corporation account without beneficiary. This money emanated from an over-invoicing in the contract awarded to a foreign firm for the supply, fixing, erecting and computerized optimization of oil pipeline to one of our oil refineries during the past Military administration. 
Base on our status as senior government officials in top sensitive position of trust, we cannot stand to claim this money ourselves, hence, I decided to contact you. All modalities for this transfer have been fully worked out. We have already secured a preliminary order to effect this payment, we now require from you your BANKING PARTICULARS, FAX AND TELEPHONE NUMBER for smooth transfer of this money into your bank account. We intend to share the money in the following manner: 30% for you, 65% for us and 5% for any miscellaneous expenses. This business is 100% risk-free. We look forward to a fruitful business relationship with you.
Best Regards 


Dr. Chima Adams
Account Department (NNPC)






_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com



From fredrik@pythonware.com  Mon Oct 29 11:28:32 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 29 Oct 2001 12:28:32 +0100
Subject: [Python-Dev] (no subject)
References: <E15y9wC-0000a2-00@mail.python.org>
Message-ID: <005f01c1606c$d8dc1140$ced241d5@hagrid>

> In the course of our review, we discovered that the sum of
> USD 30 million (THIRTY MILLION DOLLARS) was floating un-
> claimed in the corporation account without beneficiary.

> We intend to share the money in the following manner: 30%
> for you, 65% for us and 5% for any miscellaneous expenses.

$9 million to the PSF?  not bad.

maybe we should hand them over to VA Linux, to make
sure sourceforge stays up another year or two.

</F>



From gerhard@bigfoot.de  Mon Oct 29 12:36:40 2001
From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=)
Date: Mon, 29 Oct 2001 13:36:40 +0100
Subject: [Python-Dev] Future of SSL
In-Reply-To: <3BDD1519.31496FB@lemburg.com>; from mal@lemburg.com on Mon, Oct 29, 2001 at 09:36:41AM +0100
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com>
Message-ID: <20011029133640.A1804@lilith.hqd-internal>

On Mon, Oct 29, 2001 at 09:36:41AM +0100, M.-A. Lemburg wrote:
> [Integrating more crypto code into Python]
> 
> Please don't proceed in this direction. Crypto code has a long
> tradition of making world-wide distribution hard and painful, either
> to export, to import or to usage restrictions.

True. But there's not much you can do about stupid governments, except
not voting for them next time.

> There's a good reason why e.g. mxCrypto was developed outside the US,
> in fact I wrote that package because I couldn't legally export
> Andrew's code from the US. OpenSSL itself was developed in large parts
> in Australia for much the same reason.

The problem is that there needs to be some crypto support so that HTTPS,
SMTP over TLS, etc. works if the crypto is available. So, at least the
crypto hooks (SSL API) must be there, right?

What would be your suggestions? Would you prefer to go in the direction
of my original proposal - only providing a SSL API, but not the
implementation?

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))


From guido@python.org  Mon Oct 29 12:54:22 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 07:54:22 -0500
Subject: [Python-Dev] Adding DB modules to the core ? (Re: Future of SSL)
In-Reply-To: Your message of "Mon, 29 Oct 2001 09:49:22 +0100."
 <3BDD1812.1FEF2E33@lemburg.com>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com>
 <3BDD1812.1FEF2E33@lemburg.com>
Message-ID: <200110291254.HAA12811@cj20424-a.reston1.va.home.com>

> Guido van Rossum wrote:
> > 
> > (Note that, alas, the DB-API never made it this far -- we still
> > haven't been able to find the time to do the tedious work of moving
> > the various database adapters in the core distribution. :-( )
> 
> Do you really think that this would be the right approach ?
> Sounds like code bloat to me... 

It depends.  This is definitely a frequently requested feature, and
that's what I go by...  Plus, I know that the various DB packages
(mxODBC notwithstanding :-) are of, eh, varying quality.

> BTW, I'm still waiting for one of the PSF sponsors to create 
> the mythical Sumo distribution of Python. Any news in that
> direction ?

I don't recall either of the sponsors promising, although I would say
that ActivePython might be a good start.

I remember one of the regular members promising this, but apparently
his fork was larger than his mouth (as we say in Holland).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 12:56:15 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 07:56:15 -0500
Subject: [Python-Dev] Re: patchlevel.h
In-Reply-To: Your message of "Mon, 29 Oct 2001 11:02:04 +0100."
 <20011029100204.E0766303181@snelboot.oratrix.nl>
References: <20011029100204.E0766303181@snelboot.oratrix.nl>
Message-ID: <200110291256.HAA12835@cj20424-a.reston1.va.home.com>

> There was a moderately simple fix to import.c, to test that
> submodules of frozen modules were also frozen, and not picked up
> from the filesystem.  Unfortunately whoever did the patch wasn't
> aware that MacPython frozen modules work different, and hence broke
> it.

Guilty as charged.  I didn't write it but approved it and checked it
in.  But that's what beta testing is for...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 13:06:28 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 08:06:28 -0500
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Mon, 29 Oct 2001 13:36:40 +0100."
 <20011029133640.A1804@lilith.hqd-internal>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com>
 <20011029133640.A1804@lilith.hqd-internal>
Message-ID: <200110291306.IAA12932@cj20424-a.reston1.va.home.com>

> True. But there's not much you can do about stupid governments, except
> not voting for them next time.

Doesn't seem to help in the US. :-(

> > There's a good reason why e.g. mxCrypto was developed outside the US,
> > in fact I wrote that package because I couldn't legally export
> > Andrew's code from the US. OpenSSL itself was developed in large parts
> > in Australia for much the same reason.
> 
> The problem is that there needs to be some crypto support so that HTTPS,
> SMTP over TLS, etc. works if the crypto is available. So, at least the
> crypto hooks (SSL API) must be there, right?
> 
> What would be your suggestions? Would you prefer to go in the direction
> of my original proposal - only providing a SSL API, but not the
> implementation?

Yes, that's how the current SSL support works -- you need to link in
openssl.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gerhard@bigfoot.de  Mon Oct 29 13:18:45 2001
From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=)
Date: Mon, 29 Oct 2001 14:18:45 +0100
Subject: [Python-Dev] Future of SSL
In-Reply-To: <200110291306.IAA12932@cj20424-a.reston1.va.home.com>; from guido@python.org on Mon, Oct 29, 2001 at 08:06:28AM -0500
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com>
Message-ID: <20011029141844.A1967@lilith.hqd-internal>

On Mon, Oct 29, 2001 at 08:06:28AM -0500, Guido van Rossum wrote:
> > [me, in response to a remark from Marc-André]
> > What would be your suggestions? Would you prefer to go in the direction
> > of my original proposal - only providing a SSL API, but not the
> > implementation?
> 
> Yes, that's how the current SSL support works -- you need to link in
> openssl.

So, as long as there are no actual cryptographic algorithms in the
Python source tree, but only hooks for OpenSSL, there's no problem?

If this is the case, then there would only be an issue for binary
distributions of Python, which can be built easily in free countries :-)

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))


From jack@oratrix.nl  Mon Oct 29 14:07:14 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Mon, 29 Oct 2001 15:07:14 +0100
Subject: [Python-Dev] PEP 273 - inporting from zipfiles
Message-ID: <20011029140714.81E46303181@snelboot.oratrix.nl>

Jim,
for your implementation you might want to have a look at the 
Macintosh-specific code in import.c (and possibly at related code in 
Mac/macglue.c, but that's probably less relevant).

MacPython can already have files on sys.path, and there's code to look through 
these files for PYC resources. Don't worry what they are or how it's done, but 
they contain compiled modules.

I got a pretty significant increase of import speed is when I added a cache of 
which sys.path entries were files and which were directories, so for each 
sys.path entry you know that you can skip one leg of the find_module() code. 
This is all in import.c, so you could probably enable it verbatim for zipfile 
imports.

--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | ++++ see http://www.xs4all.nl/~tank/ ++++




From jim@interet.com  Mon Oct 29 14:19:52 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 09:19:52 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
Message-ID: <3BDD6588.7CFEA7D6@interet.com>

Guido van Rossum wrote:

> Hm, I wonder if we couldn't just link with the libz.a C library and
> use the C interface, if you're implementing this in C anyway.

Since zlib is often in a DLL or shared library (.so), I am afraid of
portability problems if we use the C interface.  For example, on Windows,
linking python.dll with zlib.dll would cause python.dll to fail if zlib.dll
were unavailable, even if zlib.dll were never accessed.  It also happens
that use of the zlib module is easier.

> I wonder if we need to make an additional rule that allows a .pyc file
> to satisfy a module request even if we're in optimized mode (where
> normally only .pyo files are searched).  Otherwise, if someone ships a
> zipfile with only .pyc files, their modules can't be imported at all
> when python -O is used.

Yes, I agree.  How about if we look for the correct .py[co] first,
and if that fails, look for the other?  Either will satisfy the
import, right?  If .pyc is wanted, .pyo is OK too.
 
> |     The only way to find files in a zip archive is linear search.
> 
> But there's an index record at the end that provides quick access.

It is this index which is searched linearly.

> Do we get an "module not found" error or something better, like
> "compressed module found as <filename> but zlib unavailable"?

We get "module not found".  The second is awkward, because the
module may be found later in sys.path.
 
> Well, this is the domain of getpath.c, and that's got a different
> implementation for Unix and Windows anyway (Windows has PC/getpathp.c).

I would like discussion on what the additional sys.path names are.

> |     A C implementation exists which works, but which can be made better.

I can upload it for inspection, but I don't want it to be a patch
because it is not done yet.

JimA


From mal@lemburg.com  Mon Oct 29 14:20:25 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 29 Oct 2001 15:20:25 +0100
Subject: [Python-Dev] Future of SSL
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal>
Message-ID: <3BDD65A9.BB2988A3@lemburg.com>

Gerhard H=E4ring wrote:
>=20
> On Mon, Oct 29, 2001 at 09:36:41AM +0100, M.-A. Lemburg wrote:
> > [Integrating more crypto code into Python]
> >
> > Please don't proceed in this direction. Crypto code has a long
> > tradition of making world-wide distribution hard and painful, either
> > to export, to import or to usage restrictions.
>=20
> True. But there's not much you can do about stupid governments, except
> not voting for them next time.

I think it's more difficult than that... this is about politics
and can be a very touchy subject depending on at which angle you look=20
at it.

> > There's a good reason why e.g. mxCrypto was developed outside the US,
> > in fact I wrote that package because I couldn't legally export
> > Andrew's code from the US. OpenSSL itself was developed in large part=
s
> > in Australia for much the same reason.
>=20
> The problem is that there needs to be some crypto support so that HTTPS=
,
> SMTP over TLS, etc. works if the crypto is available. So, at least the
> crypto hooks (SSL API) must be there, right?
>=20
> What would be your suggestions? Would you prefer to go in the direction
> of my original proposal - only providing a SSL API, but not the
> implementation?

Note that even the hooks can cause legal problems, because some
(countries) consider the hooks as guns without amunition.

I'd suggest to leave serious crypto completely out of Python and
to use one of the available third-party toolkits in case it
should become necessary, e.g. M2Crypto has been around for a while
and probably provides all that's necessary to do HTTPS or other
SSL/TLS-based protocol and amkCrypto provides all the low-level
tools to write your own secure procotols.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From gmcm@hypernet.com  Mon Oct 29 14:30:30 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 29 Oct 2001 09:30:30 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BD9B5EC.119319A5@interet.com>
Message-ID: <3BDD21B6.12124.38E143B8@localhost>

Jim Ahlstrom wrote:
 [From PEP 273]
> Currently, sys.path is a list of directory names as strings. 

A nit: that's in practice, but not by definition (a couple std 
modules have been patched to allow other things on 
sys.path). After extensive use of imputil (which puts objects 
on sys.path), I think we might as well make it official that 
sys.path is a list of strings.

[Subdirectory Equivalence]

This is a bit misleading - it can be read as implying that the 
importable modules are in a flat namespace. They're not. To 
get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must 
be imported. The __init__ modules have the opportunity to 
play with __path__ to break the equivalence of R.Q.modfoo to 
R/Q/modfoo.py. Or (more likely), play games with attributes 
so that Q is, say, an instance, or maybe a module imported 
from someplace else.

Question: if Archive.zip/Q/__init__.pyc does 
"__path__.append('some/real/directory')", will it work?

[dynamic libs]
> It might be possible to extract the dynamic module from the
> zip file, write it to a plain file and load it. 

I think you can nail the door shut on this one. On many OSes, 
making  dynamic libs available to a process requires settings 
that can only (sanely) be made before the process starts.

OTOH, it's common practice these days to put dynamic libs 
inside packages. That needs to be dealt with at runtime, and 
at build time (since it breaks the expectation that you can just 
"zip up" the package).

[no *.py files]
> You also can't import source files *.py from a zip archive. 

Apparently some Linuxes / RPM distributions don't deliver 
.pyc's or .pyo's. Since they're installed as root and run as 
some-poor-user, I'm afraid there are quite a few installations 
running off .py's all the time. So while it's definitely sub-
optimal, I'm not sure it should be outlawed.

[Guido]
> But it would still be good if the .py files were also in the
> zip file, so the source can be used in tracebacks etc. 

As you probably remember me saying before, I think this is a 
very minor nice-to-have. For one thing, you've probably done 
enough testing before stuffing your code into an archive that 
you're past the point of looking at a traceback and going 
"doh!". Most of the time you'll need to look at the code 
anyway.

OTOH, this [more from Guido]:
> A C API to get a source line from a filename might be a good
> idea (plus a Python API). 

points the way towards something I'm very much in favor of: 
deferring things to that-which-did-the-importing.

[Efficiency]
> The key is the archive name from sys.path joined with the
> file name (including any subdirectories) within the archive. 

DIfferent spellings of the same path are possible in a 
filesystem, but not in a dictionary. A bit of "harmless" 
tweaking of sys.path could render an archive unreachable.

[zlib must be available at start]
I'll agree, and agree with Guido that the coolest thing would be 
to make zlib standard.

[Booting - python.zip should be part of the generated sys.path]
Agree. Nice and straightforward.

Now, from the discussion:

[restate Jim]
sys.path contains /C/D/E/Archive.zip
and Archive.zip contains "Q/R/modfoo.pyc"
so "import Q.R.modfoo" is satisfied by
/C/D/E/Archive.zip/Q/R/modfoo.pyc

[restate Finn]
Jython has sys.path /C/D/E/Archive.zip!Lib
and Archive.zip has "Lib/Q/R/modfoo.pyc"
so "import Q.R.modfoo" is satisfied by
/C/D/E/Archive.zip/Lib/Q/R/modfoo.pyc

[restate Guido]
Why not use /C/D/E/Archive.zip/Lib on sys.path?

I use embedded archives. So sys.path will have an entry like:
"/path/to/executable?84758" which says that the archive 
starts at position 84758 in the file "executable". 

Anything beyond the simple approach Jim takes gets into 
some very URL-ish territory. That's fine by me :-).

I don't really like the idea of hacking special knowledge of zip 
files into import.c (which is already a specialist in 
filesystems). Like Finn said, if this is a deployment issue (we 
want zip files now, and are willing to live with strict limitations / 
rules to get it), then OK (as long as it supports __path__ and 
some way of dealing with dynamic libs in packages).

Personally, I think package support stretched import.c to it's 
monolithic limits and it's high time the code was refactored to 
make it sanely extensible.



- Gordon


From jim@interet.com  Mon Oct 29 14:35:07 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 09:35:07 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk>
Message-ID: <3BDD691B.2599AAF4@interet.com>

Finn Bock wrote:

> Why can't .py be allowed? If a more recent .py[co] (or $py.class)
> exists, it is used. Otherwise the .py file is compiled and discarded
> when the process ends. Sure, it is slower, but a zip files with only
> .py[co] entries would be of little use with jython.

The *.py are allowed to be in the file, and jpython can use them.  In
fact, any files at all can be in the archive.

It is just that C-Python ignores them.  My reason was that if *.py[co] are
missing or out of date, zip importing will be slow and users won't figure
out what is wrong.  But I am open to changing this.

> >    Just as sys.path currently has default directory names, default
> A standard for this would be really cool.

Yes, lets make a standard now.

> So python packages and modules can exists *only* at the top level? That
> would conflict somewhat with jython where at would be common to put
> python modules into the same .zip file as java classes and java classes
> also wants to own the root of a zip file.
> 
> In the current implementaion in jython, we can put python modules under
> any path and add the zipfile!path to sys.path:
> 
>    sys.path.append("/path/to/zipfile.zip!Lib")
> 
> which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in
> the archive.

I am confused.  Zip archives are equivalant to subdirectories, so there
is no requirement to have anything at the top level.  Your example seems
to imply a second search path besides sys.path.

BTW, the code uses ".zip" as the archive flag,
not a special character '!'.
 
> [I found efficiency hard to achieve in jython because opening a zipfile
> in java also cause the zip index to be read into a dictionary. So we
> did not want to reopen a zipfile if it can be avoided. Instead we hide a
> reference to the opened file in sys.path so that when removing a zipfile
> name from sys.path, the file is eventually closed.]
> 
> Would entries in the static python dict be removed when a zipfile is
> removed from sys.path?

No, entries would not be removed.  But they would not be found either,
because their names would not be generated from the new sys.path.

> What is the __path__ vrbl set to in a module imported from a zipfile?
> Can the module make changes to __path__ and will be changes to used when
> importing submodules?
> 
> What value should __file__ have?

The __file__ is /A/B/archive.zip/name.py.  There is no special code
for __file__ nor __path__, the path name just has a ".zip" in it.

JimA


From guido@python.org  Mon Oct 29 14:34:58 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 09:34:58 -0500
Subject: [Python-Dev] Future of SSL
In-Reply-To: Your message of "Mon, 29 Oct 2001 14:18:45 +0100."
 <20011029141844.A1967@lilith.hqd-internal>
References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com>
 <20011029141844.A1967@lilith.hqd-internal>
Message-ID: <200110291434.JAA13268@cj20424-a.reston1.va.home.com>

> > > [me, in response to a remark from Marc-André]
> > > What would be your suggestions? Would you prefer to go in the direction
> > > of my original proposal - only providing a SSL API, but not the
> > > implementation?
> > 
> > Yes, that's how the current SSL support works -- you need to link in
> > openssl.
> 
> So, as long as there are no actual cryptographic algorithms in the
> Python source tree, but only hooks for OpenSSL, there's no problem?

That's what I believe the situation currently, for open source
software.  It wasn't always like this -- at some point, hooks were
enough to need export permission.  It may again be like this, if some
uninformed US senators get their way...

> If this is the case, then there would only be an issue for binary
> distributions of Python, which can be built easily in free countries :-)

Hm, but that's dangerous too -- someone could easily build an RPM that
includes openssl without realizing it.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Mon Oct 29 14:45:08 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 09:45:08 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com>   <3bd9d419.10386565@mail.wanadoo.dk>  <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk>
Message-ID: <3BDD6B74.D4E19953@interet.com>

Finn Bock wrote:

> >> Would entries in the static python dict be removed when a zipfile is
> >> removed from sys.path?
> >
> >It can be arranged that they are removed at some later point
> >(e.g. when sys.path is next searched).
> 
> An API to do this could be usefull for jython. Right now we depend on
> the GC thread to close the file. Since files are a limited resource it
> would be a good thing to have an explicit way to clean up the cached
> resources.

The static Python dictionary is a memory object which uses no open
file descriptors.  If an element of sys.path contains ".zip" and hasn't
been seen before, the zip archive is opened, searched, and closed.
The key is the name, the value includes the archive name and offset.
Thereafter, the zip archive is never opened unless it contains a needed file.
So I don't think there is a problem.

> myself), I just didn't dare to put such a limit on my jython
> implementation.

I don't understand the jpython implementation, so please point out
all problems so we can fix them now.

JimA


From gmcm@hypernet.com  Mon Oct 29 14:50:30 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 29 Oct 2001 09:50:30 -0500
Subject: [Python-Dev] Future of SSL
In-Reply-To: <200110291434.JAA13268@cj20424-a.reston1.va.home.com>
References: Your message of "Mon, 29 Oct 2001 14:18:45 +0100."             <20011029141844.A1967@lilith.hqd-internal>
Message-ID: <3BDD2666.4196.38F3926B@localhost>

I wrote some proprietary code for a client, using (hacked) 
M2Crypto and openSSL, but using only fairly lightweight DES 
(not even DES3). In order to get an export license, I had to 
talk to the Commerce Dept and the NSA. That got the export 
license, but to get it into France, the client had to talk 
extensively to the French government.

My guess is that anything beyond enabling https in core 
Python will make negotiating with the FSF look like a Mardi 
Gras party.

- Gordon


From mmuller@enduden.com  Mon Oct 29 14:35:40 2001
From: mmuller@enduden.com (Michael Muller)
Date: Mon, 29 Oct 2001 09:35:40 -0500
Subject: [Python-Dev] status of OS/2 ports of Python
Message-ID: <193.1004366140.804197.425672245@lilbastard>

Guido van Rossum wrote:
> > I would also propose to look after patches to the VAC++ build (such as
> > Michael Muller's), on the basis that although I can't test the patches
> > directly, I can test that they don't affect the rest of the system, and I
> > can work with the submitter to resolve issues that affect OS/2 specific
> > functionality.
> 
> That sounds like a very good idea.  Right now I don't really know what
> to do with the two sets of OS/2 patches...  Since Michael is cc'ed
> here, Michael, what do you think of this?

This works for me.  As I've said in the notes on my patches, there is some
"textual" conflict between Andrews patches and mine (meaning that my patches
won't work after his have been applied).  However, I see no functional
conflicts, and as I've said I'd be happy to create new patches that will
work with his.

Also, Andrew: I have a lot of experience with CVS and I'm willing to offer
any help you need.  I can also test changes on Linux.

My only wish is that this be resolved quickly so I don't have to keep
revising my patches to keep up with the (rapidly evolving) baseline ;-).

=============================================================================
michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus
-----------------------------------------------------------------------------
In this book it is spoken of the Sephiroth, and the Paths, of Spirits and
Conjurations; of Gods, Spheres, Planes and many other things which may or
may not exist.  It is immaterial whether they exist or not.  By doing
certain things certain results follow. - Aleister Crowley
=============================================================================


From jim@interet.com  Mon Oct 29 14:53:26 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 09:53:26 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com>
Message-ID: <3BDD6D66.675A4F3C@interet.com>

"M.-A. Lemburg" wrote:

> 1. Why are .py files being outlawed ?

My reason was that if *.py[co] are missing or out of date,
zip importing will be slow and users won't figure
out what is wrong.  Generally I favor user-proof features
over expert features.  I prefer things which either "Just Work"
or fail spectacularly.  But I am open to changing this.

> 2. Where's the C implementation you mention in the PEP ?

Software is like pancakes, you should always
throw the first one away.  I will post it if you want,
but it is not done.

> 3. Would it be possible to ship zlib together with Python ?
>    (the zlib license should allow this and I don't think that
>    the code size is too big)

OK by me.  But uncompressed zip archives work, and may even
be faster than conpressed archives.

JimA


From guido@python.org  Mon Oct 29 14:58:39 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 09:58:39 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Mon, 29 Oct 2001 09:19:52 EST."
 <3BDD6588.7CFEA7D6@interet.com>
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com>
Message-ID: <200110291458.JAA13956@cj20424-a.reston1.va.home.com>

> Guido van Rossum wrote:
> 
> > Hm, I wonder if we couldn't just link with the libz.a C library and
> > use the C interface, if you're implementing this in C anyway.
> 
> Since zlib is often in a DLL or shared library (.so), I am afraid of
> portability problems if we use the C interface.  For example, on Windows,
> linking python.dll with zlib.dll would cause python.dll to fail if zlib.dll
> were unavailable, even if zlib.dll were never accessed.  It also happens
> that use of the zlib module is easier.

OK, good enough.  (I'm getting more and more curious about your
implementation. ;-)

> > I wonder if we need to make an additional rule that allows a .pyc file
> > to satisfy a module request even if we're in optimized mode (where
> > normally only .pyo files are searched).  Otherwise, if someone ships a
> > zipfile with only .pyc files, their modules can't be imported at all
> > when python -O is used.
> 
> Yes, I agree.  How about if we look for the correct .py[co] first,
> and if that fails, look for the other?  Either will satisfy the
> import, right?  If .pyc is wanted, .pyo is OK too.

Sounds good to me -- this might be a good rule anyhoo.

> > |     The only way to find files in a zip archive is linear search.
> > 
> > But there's an index record at the end that provides quick access.
> 
> It is this index which is searched linearly.

That can't possibly take any time compared to other stuff -- it's a
relatively short in-memory table.

> > Do we get an "module not found" error or something better, like
> > "compressed module found as <filename> but zlib unavailable"?
> 
> We get "module not found".  The second is awkward, because the
> module may be found later in sys.path.

Hm.  Does it at least spit out a warning when zlib is needed but not
available?  It can be awkward to debug the situation where you get a
module from the filesystem rather than the version from the zipfile,
even though it is in the zipfile.  While the compression method is
listed in the zip info, it's not the first thing someone would look
for unless they were aware that this failure mode existed.

Since obviously the intention of putting the module in the zipfile was
that it should be found, I think that failure to decompress should be
turned into an immediate error -- the same way as a syntax error gets
reported and not turned into a "skip to the next directory in
sys.path" effect.

> > Well, this is the domain of getpath.c, and that's got a different
> > implementation for Unix and Windows anyway (Windows has PC/getpathp.c).
> 
> I would like discussion on what the additional sys.path names are.

Well, propose some.

> > |     A C implementation exists which works, but which can be made better.
> 
> I can upload it for inspection, but I don't want it to be a patch
> because it is not done yet.

Just say so in the patch.  You can upoad anything you want. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 15:06:59 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 10:06:59 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Mon, 29 Oct 2001 09:30:30 EST."
 <3BDD21B6.12124.38E143B8@localhost>
References: <3BDD21B6.12124.38E143B8@localhost>
Message-ID: <200110291506.KAA14000@cj20424-a.reston1.va.home.com>

> Jim Ahlstrom wrote:
>  [From PEP 273]
> > Currently, sys.path is a list of directory names as strings. 

[GMcM]
> A nit: that's in practice, but not by definition (a couple std 
> modules have been patched to allow other things on 
> sys.path). After extensive use of imputil (which puts objects 
> on sys.path), I think we might as well make it official that 
> sys.path is a list of strings.

Interesting.  So you think imputil is wrong to put objects there?
Why?  (Not arguing, just interested in your experience.)

> [Subdirectory Equivalence]
> 
> This is a bit misleading - it can be read as implying that the 
> importable modules are in a flat namespace. They're not. To 
> get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must 
> be imported. The __init__ modules have the opportunity to 
> play with __path__ to break the equivalence of R.Q.modfoo to 
> R/Q/modfoo.py. Or (more likely), play games with attributes 
> so that Q is, say, an instance, or maybe a module imported 
> from someplace else.
> 
> Question: if Archive.zip/Q/__init__.pyc does 
> "__path__.append('some/real/directory')", will it work?

It should.

> [dynamic libs]
> > It might be possible to extract the dynamic module from the
> > zip file, write it to a plain file and load it. 
> 
> I think you can nail the door shut on this one. On many OSes, 
> making  dynamic libs available to a process requires settings 
> that can only (sanely) be made before the process starts.

ANd I believe some systems require shared libraries to be owned by
root.

> OTOH, it's common practice these days to put dynamic libs 
> inside packages. That needs to be dealt with at runtime, and 
> at build time (since it breaks the expectation that you can just 
> "zip up" the package).

Yes.  This is important.

> [no *.py files]
> > You also can't import source files *.py from a zip archive. 
> 
> Apparently some Linuxes / RPM distributions don't deliver 
> .pyc's or .pyo's. Since they're installed as root and run as 
> some-poor-user, I'm afraid there are quite a few installations 
> running off .py's all the time. So while it's definitely sub-
> optimal, I'm not sure it should be outlawed.

Heh, this is an argument for Jim's position -- it would have come out
during testing that way, since the imports would fail. ;-)

> [Guido]
> > But it would still be good if the .py files were also in the
> > zip file, so the source can be used in tracebacks etc. 
> 
> As you probably remember me saying before, I think this is a 
> very minor nice-to-have. For one thing, you've probably done 
> enough testing before stuffing your code into an archive that 
> you're past the point of looking at a traceback and going 
> "doh!". Most of the time you'll need to look at the code 
> anyway.

Testing is for wimps. :-)

> OTOH, this [more from Guido]:
> > A C API to get a source line from a filename might be a good
> > idea (plus a Python API). 
> 
> points the way towards something I'm very much in favor of: 
> deferring things to that-which-did-the-importing.

Yup.

> [Efficiency]
> > The key is the archive name from sys.path joined with the
> > file name (including any subdirectories) within the archive. 
> 
> DIfferent spellings of the same path are possible in a 
> filesystem, but not in a dictionary. A bit of "harmless" 
> tweaking of sys.path could render an archive unreachable.

Hm, wouldn't the archive just be opened a second time?  Or do I
misundestand you?

> [zlib must be available at start]
> I'll agree, and agree with Guido that the coolest thing would be 
> to make zlib standard.

But we'd have to make sure it's statically linked.  (Fortunately, we
already link it statically on Windows.)

> [Booting - python.zip should be part of the generated sys.path]
> Agree. Nice and straightforward.
> 
> Now, from the discussion:
> 
> [restate Jim]
> sys.path contains /C/D/E/Archive.zip
> and Archive.zip contains "Q/R/modfoo.pyc"
> so "import Q.R.modfoo" is satisfied by
> /C/D/E/Archive.zip/Q/R/modfoo.pyc
> 
> [restate Finn]
> Jython has sys.path /C/D/E/Archive.zip!Lib
> and Archive.zip has "Lib/Q/R/modfoo.pyc"
> so "import Q.R.modfoo" is satisfied by
> /C/D/E/Archive.zip/Lib/Q/R/modfoo.pyc
> 
> [restate Guido]
> Why not use /C/D/E/Archive.zip/Lib on sys.path?
> 
> I use embedded archives. So sys.path will have an entry like:
> "/path/to/executable?84758" which says that the archive 
> starts at position 84758 in the file "executable". 
> 
> Anything beyond the simple approach Jim takes gets into 
> some very URL-ish territory. That's fine by me :-).
> 
> I don't really like the idea of hacking special knowledge of zip 
> files into import.c (which is already a specialist in 
> filesystems). Like Finn said, if this is a deployment issue (we 
> want zip files now, and are willing to live with strict limitations / 
> rules to get it), then OK (as long as it supports __path__ and 
> some way of dealing with dynamic libs in packages).
> 
> Personally, I think package support stretched import.c to it's 
> monolithic limits and it's high time the code was refactored to 
> make it sanely extensible.

Yes -- this has been on my TODO list for ages.  But who's gonna DO it?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 15:13:14 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 10:13:14 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Mon, 29 Oct 2001 09:35:07 EST."
 <3BDD691B.2599AAF4@interet.com>
References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk>
 <3BDD691B.2599AAF4@interet.com>
Message-ID: <200110291513.KAA14082@cj20424-a.reston1.va.home.com>

[Finn]
> > So python packages and modules can exists *only* at the top level? That
> > would conflict somewhat with jython where at would be common to put
> > python modules into the same .zip file as java classes and java classes
> > also wants to own the root of a zip file.
> > 
> > In the current implementaion in jython, we can put python modules under
> > any path and add the zipfile!path to sys.path:
> > 
> >    sys.path.append("/path/to/zipfile.zip!Lib")
> > 
> > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in
> > the archive.

[JimA]
> I am confused.  Zip archives are equivalant to subdirectories, so there
> is no requirement to have anything at the top level.  Your example seems
> to imply a second search path besides sys.path.

I think Finn simply means that the zipfile may have some redundant
initial suffix to all filenames (e.g. "Lib/") which could be an
artefact of how the zip file is created (zip up this directory) or
intended as an aid for other uses of the zipfile, like unpacking.  (In
the tar world, it's considered impolite not to have a common prefix;
without that, untarring can too easily populate an innocent user's
home directory with thousands of untarred files.)

> BTW, the code uses ".zip" as the archive flag,
> not a special character '!'.

That's cool.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 15:16:29 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 10:16:29 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Mon, 29 Oct 2001 09:53:26 EST."
 <3BDD6D66.675A4F3C@interet.com>
References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com>
 <3BDD6D66.675A4F3C@interet.com>
Message-ID: <200110291516.KAA14105@cj20424-a.reston1.va.home.com>

> > 3. Would it be possible to ship zlib together with Python ?
> >    (the zlib license should allow this and I don't think that
> >    the code size is too big)
> 
> OK by me.  But uncompressed zip archives work, and may even
> be faster than conpressed archives.

Good point.  Uncompressed archives will definitely be faster.  I'd be
happy to legislate uncompressed archives, except I'm worried that not
all tools commonly used to create zip archives make it easy to turn
off compression.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Mon Oct 29 15:34:30 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 10:34:30 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BDD21B6.12124.38E143B8@localhost>
Message-ID: <3BDD7706.A5BBE307@interet.com>

Gordon McMillan wrote:
>
> > Currently, sys.path is a list of directory names as strings.
> 
> A nit: that's in practice, but not by definition (a couple std
> modules have been patched to allow other things on
> sys.path). After extensive use of imputil (which puts objects
> on sys.path), I think we might as well make it official that
> sys.path is a list of strings.

A nit on a nit: Non-strings can still be allowed,
because I can ignore them.

> [Subdirectory Equivalence]
> 
> This is a bit misleading - it can be read as implying that the
> importable modules are in a flat namespace. They're not. To
> get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must
> be imported. The __init__ modules have the opportunity to
> play with __path__ to break the equivalence of R.Q.modfoo to
> R/Q/modfoo.py. Or (more likely), play games with attributes
> so that Q is, say, an instance, or maybe a module imported
> from someplace else.

I feel that this is an excellent point, but I don't understand
package import well enough to comment.
 
> Question: if Archive.zip/Q/__init__.pyc does
> "__path__.append('some/real/directory')", will it work?

My guess (and it really is a guess) is Yes.  The import.c code
which generates directory names is untouched.  My code simply
looks for ".zip" in the generated name and branches into the
zip archive at that point.  So if the directory version works,
the zip archive version should work too.
 
> [no *.py files]
> > You also can't import source files *.py from a zip archive.
> 
> Apparently some Linuxes / RPM distributions don't deliver
> .pyc's or .pyo's. Since they're installed as root and run as
> some-poor-user, I'm afraid there are quite a few installations
> running off .py's all the time. So while it's definitely sub-
> optimal, I'm not sure it should be outlawed.

For Linux/RPM's I think shipping a library directory is better
than a zip archive.  It is easier to hack on a directory.  I
think of zip archives as a way to distribute packages, and
as a replacement for freeze.

OTOH, maybe we should allow *.py to satisfy imports even if
it is slow and invisible.

> [Efficiency]
> > The key is the archive name from sys.path joined with the
> > file name (including any subdirectories) within the archive.
> 
> DIfferent spellings of the same path are possible in a
> filesystem, but not in a dictionary. A bit of "harmless"
> tweaking of sys.path could render an archive unreachable.

True, but I have little sympathy for case-insensitive file
systems.  Tweaking of sys.path will have to be done with care.
It helps that the '/' character is always used in zip, and
both backslash and colon ":" are illegal in zip archive names.
 
JimA


From jim@interet.com  Mon Oct 29 15:38:33 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 10:38:33 -0500
Subject: [Python-Dev] PEP 273 - inporting from zipfiles
References: <20011029140714.81E46303181@snelboot.oratrix.nl>
Message-ID: <3BDD77F9.949DA69D@interet.com>

Jack Jansen wrote:
> 
> Jim,
> for your implementation you might want to have a look at the
> Macintosh-specific code in import.c (and possibly at related code in
> Mac/macglue.c, but that's probably less relevant).

OK, will do.

JimA


From gmcm@hypernet.com  Mon Oct 29 15:41:04 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 29 Oct 2001 10:41:04 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <200110291506.KAA14000@cj20424-a.reston1.va.home.com>
References: Your message of "Mon, 29 Oct 2001 09:30:30 EST."             <3BDD21B6.12124.38E143B8@localhost>
Message-ID: <3BDD3240.31592.3921E147@localhost>

[Gordon]
> > After extensive use of imputil (which puts objects on
> > sys.path), I think we might as well make it official that
> > sys.path is a list of strings.
[Guido]
> Interesting.  So you think imputil is wrong to put objects there?
> Why?  (Not arguing, just interested in your experience.)

Too much code (some left in the std lib) says:
for d in sys.path:
  os.path.join(d, ...) 
and that's the line that barfs. It doesn't barf if d is a string.

[Gordon]
> > DIfferent spellings of the same path are possible in a 
> > filesystem, but not in a dictionary. A bit of "harmless" 
> > tweaking of sys.path could render an archive unreachable.
> 
> Hm, wouldn't the archive just be opened a second time?  Or do I
> misundestand you?

Without seeing the C code, I don't know if it will open it a 2nd 
time. I keep an open file for each archive, but I'm careful that 
however it's spelled, I only open the archive once. Jim is 
apparently closing the file, so opening a 2nd is probably not 
as painful. OTOH an open / seek for each satisfied import is 
relatively expensive (though still cheaper than a fs import).
 
> > [zlib must be available at start]
> > I'll agree, and agree with Guido that the coolest thing would
> > be to make zlib standard.
> 
> But we'd have to make sure it's statically linked.  (Fortunately,
> we already link it statically on Windows.)

So to avoid statically linking it twice, I assume zlib would be a 
mandatory builtin.
 
[Gordon]
> > Personally, I think package support stretched import.c to it's
> > monolithic limits and it's high time the code was refactored to
> > make it sanely extensible.
> 
> Yes -- this has been on my TODO list for ages.  But who's gonna
> DO it?

I published iu4 which I think comes very close to the "right" 
model. If it's close enough, I'll download 2.2 and see if I can 
make those into new style object subclasses[1].

- Gordon

[1] As an app developer, I usually have to look backwards, not 
forwards. I've got 6 machines, each with at least 2 Pythons, 
so I no longer deal with betas :-(. Even when they have cool 
things like generators and subclassable C types.


From barry@zope.com  Mon Oct 29 15:52:27 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 29 Oct 2001 10:52:27 -0500
Subject: [Python-Dev] Re: patchlevel.h
References: <20011026102540.3DBEB303181@snelboot.oratrix.nl>
 <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de>
 <200110261856.OAA20424@cj20424-a.reston1.va.home.com>
Message-ID: <15325.31547.431457.787791@anthem.wooz.org>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> I think that for the final release we should definitely
    GvR> attempt this.  Hopefully the speed and quality of change will
    GvR> reduce closer to the final release.  Also, the release
    GvR> candidate should help.

    GvR> Jack, We'll try to work with you.  Barry, please add
    GvR> something to this effect in the release procedure PEP (hold
    GvR> up the *final* release until Jack approves or until we lose
    GvR> patience).

Absolutely.  And done.

-Barry


From gmcm@hypernet.com  Mon Oct 29 15:54:50 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 29 Oct 2001 10:54:50 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <200110291516.KAA14105@cj20424-a.reston1.va.home.com>
References: Your message of "Mon, 29 Oct 2001 09:53:26 EST."             <3BDD6D66.675A4F3C@interet.com>
Message-ID: <3BDD357A.126.392E7B98@localhost>

[Guido]
> Good point.  Uncompressed archives will definitely be faster. 
> I'd be happy to legislate uncompressed archives, except I'm
> worried that not all tools commonly used to create zip archives
> make it easy to turn off compression.

Don't be so sure about speed. If the archive is big enough and 
the OS bad enough <wink>, it may be faster to uncompress 
than swap. I always compress at 9, and have only seen speed 
ups of 1 or 2% with compression turned off.

BTW, .pyc's compress quite well.

- Gordon


From jim@interet.com  Mon Oct 29 15:58:10 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 10:58:10 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
Message-ID: <3BDD7C92.F130E708@interet.com>

Guido van Rossum wrote:

> Since obviously the intention of putting the module in the zipfile was
> that it should be found, I think that failure to decompress should be
> turned into an immediate error -- the same way as a syntax error gets
> reported and not turned into a "skip to the next directory in
> sys.path" effect.

Hmmm, I'm not sure.  But it is easy to do.
 
> > I can upload it for inspection, but I don't want it to be a patch
> > because it is not done yet.
> 
> Just say so in the patch.  You can upoad anything you want. :-)

OK, I will post the implementation.

JimA


From gmcm@hypernet.com  Mon Oct 29 16:17:50 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 29 Oct 2001 11:17:50 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BDD7706.A5BBE307@interet.com>
Message-ID: <3BDD3ADE.19659.39438820@localhost>

> Gordon McMillan wrote:

> > [no *.py files]
> > > You also can't import source files *.py from a zip archive.
> > 
> > Apparently some Linuxes / RPM distributions don't deliver
> > .pyc's or .pyo's. Since they're installed as root and run as
> > some-poor-user, I'm afraid there are quite a few installations
> > running off .py's all the time. So while it's definitely sub-
> > optimal, I'm not sure it should be outlawed.
> 
> For Linux/RPM's I think shipping a library directory is better
> than a zip archive.  It is easier to hack on a directory.  I
> think of zip archives as a way to distribute packages, and as a
> replacement for freeze.

There's no reason to believe that telling a Linux distribution 
maintainer that they "shouldn't" do it that way will be 
successful. Heck, he might be a Perl monkey on KP <wink>.
 
> OTOH, maybe we should allow *.py to satisfy imports even if
> it is slow and invisible.
> 
> > [Efficiency]
> > > The key is the archive name from sys.path joined with the
> > > file name (including any subdirectories) within the archive.
> > 
> > DIfferent spellings of the same path are possible in a
> > filesystem, but not in a dictionary. A bit of "harmless"
> > tweaking of sys.path could render an archive unreachable.
> 
> True, but I have little sympathy for case-insensitive file
> systems.  Tweaking of sys.path will have to be done with care. It
> helps that the '/' character is always used in zip, and both
> backslash and colon ":" are illegal in zip archive names.

You don't need case-insensitive for this to come up. Relative 
paths vs. absolute paths; or paths that need norm-ing.

And then you've got the mac which uses a different path 
syntax, and even slightly different path semantics.


- Gordon


From jim@interet.com  Mon Oct 29 16:31:42 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 11:31:42 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
Message-ID: <3BDD846E.7D372942@interet.com>

Guido van Rossum wrote:

> Just say so in the patch.  You can upoad anything you want. :-)

The implementation is in patch 476047.

 http://sourceforge.net/tracker/index.php?func=detail&aid=476047&group_id=5470&atid=305470

JimA


From jim@interet.com  Mon Oct 29 17:28:44 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 12:28:44 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
Message-ID: <3BDD91CC.9F0ECB07@interet.com>

Guido van Rossum wrote:
[Me]
> > I would like discussion on what the additional sys.path names are.
> 
> Well, propose some.

OK.  I propose that there is one name added to sys.path, and the
file name is "python%s%s.zip" % (sys.version[0], sys.version[2]).
For example, python22.zip.  This is the same on all platforms.

On Unix, the directory is sys.prefix + "/lib".  So for prefix
/usr/local, the path /usr/local/lib/python2.2/ is already on
sys.path, and /usr/local/lib/python22.zip would be added.

On Windows, the directory is the directory of sys.executable.

The zip archive name is always inserted as the second item
in sys.path.  The first always seems to be ''.

JimA


From guido@python.org  Mon Oct 29 17:39:37 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 12:39:37 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Mon, 29 Oct 2001 12:28:44 EST."
 <3BDD91CC.9F0ECB07@interet.com>
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
 <3BDD91CC.9F0ECB07@interet.com>
Message-ID: <200110291739.MAA15773@cj20424-a.reston1.va.home.com>

> OK.  I propose that there is one name added to sys.path, and the
> file name is "python%s%s.zip" % (sys.version[0], sys.version[2]).
> For example, python22.zip.  This is the same on all platforms.
> 
> On Unix, the directory is sys.prefix + "/lib".  So for prefix
> /usr/local, the path /usr/local/lib/python2.2/ is already on
> sys.path, and /usr/local/lib/python22.zip would be added.
> 
> On Windows, the directory is the directory of sys.executable.
> 
> The zip archive name is always inserted as the second item
> in sys.path.  The first always seems to be ''.

Sounds good to me.  Somebody please update the PEP.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From ngps@post1.com  Mon Oct 29 17:52:01 2001
From: ngps@post1.com (Ng Pheng Siong)
Date: Tue, 30 Oct 2001 01:52:01 +0800
Subject: [Python-Dev] Re: Python's SSL: Would you like to help?
In-Reply-To: <20011029135307.C1844@lilith.hqd-internal>; from gerhard@bigfoot.de on Mon, Oct 29, 2001 at 01:53:07PM +0100
References: <20011029135307.C1844@lilith.hqd-internal>
Message-ID: <20011030015201.B1007@madcap.dyndns.org>

On Mon, Oct 29, 2001 at 01:53:07PM +0100, Gerhard Häring wrote:
> I've meanwhile started a discussion about the future of SSL support in
> Python itself on the python-dev mailinglist.
> 
> The talk is about several issues, like redesign a whole new API
> (yes/no), integrate an existing module (which one?), support much crypto
> at all.
> 
> As you seem to have a lot of experience in this field, perhaps you'd
> like to share some of your opinions there?

[ Not subscribed, hopefully this gets thru. ]

Hello,

Thanks for asking. 

I've just read the discussion in the archives. My own archives contain a
similar (but much shorter) discussion that took place on the old
python-crypto@egroups list in Apr 2000.

I've also just downloaded 2.2b1, POW (Python OpenSSL wrappers) 0.5, and
pyOpenSSL 0.4 to have a quick look-see. (Haven't been keeping track of the
competition. ;-)

I noticed there are several SSL patches on SF, mostly from Gerhard. I
didn't read the patches, though.


Here are some random thoughts based on my experience with M2Crypto:

- To do simple multi-threading, you need to bracket calls to SSL_read(),
  SSL_write(), etc. with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.

  You also (apparently) need to implement the locking required by
  OpenSSL; see M2Crypto's _thread.i. IIRC, OpenSSL's docu warns of Bad
  Things should one fail to do so. (None of the others seem to do this.)

  If you want to do callbacks for certificate verification, it's a little
  more complicated.

  First, you need to muck around with Python's thread states in OpenSSL's
  structures. M2Crypto and pyOpenSSL do these mostly the same way. POW
  doesn't do this. 

  Next, consider the Python "SSL Context" object that mirrors SSL_CTX: In
  M2Crypto, the "SSL Context" is implemented in Python; conveniently, I use
  it to keep additional Python-level policy stuff. Thus, when OpenSSL calls
  back into M2Crypto, I need a means to get at the Python SSL Context
  object, before calling back into Python. Implementing the SSL Context
  object in C may make this simpler. 
  
  (I see 2.2b1 does only SSL_VERIFY_NONE. Ditto POW and pyOpenSSL, it
  seems.)


- Even if you only want to do certificates within SSL, you need to wrap
  quite a bit of the X.509 API. See M2Crypto's _x509.i and X509.py. POW
  seems to have done so, as well.

  If you also want to do X.509 independent of SSL (say for implementing a
  CA), you'll need to do more housekeeping, like whether to free a given
  structure at the Python side or the C side. Grep for "_pyfree" in
  M2Crypto's Python code. 


- Asynchronous SSL: I think M2Crypto has this mostly right, since
  ZServerSSL works (2nd birthday came and went it seems ;-), and I've
  recently implemented Medusa-based FTP-over-TLS which interoperates with
  other people's clients.

  Based on my light reading of the code, none of 2.2b1, POW and pyOpenSSL
  do async-SSL properly at all.


- FakeSocket, SSLFakeSocket and SSLFakeFile are ugly! 

  M2Crypto wraps OpenSSL's "stackable BIOs". makefile() on an SSL
  connection pushes a buffering BIO on top of the SSL BIO and returns that,
  which then behaves like a Python file-like object. Even does readline()!
  See M2Crypto's BIO.py.

  2.2b1 has a BIO member in PySSLObject but doesn't seem to use it. Nor do
  POW and pyOpenSSL, it appears.


Well, that's only the SSL-related stuff that came to my mind readily. 
The Python development team has to decide how much of the above you want
within Python, e.g., how much of the BIO and X.509 APIs to wrap.

Not to mention PRNGs, secure memory handling and crypto-in-general.

(Lastly, I'm pleased to note that M2Crypto is still ahead of the
competition. Gotta keep moving, though! ;-)

Cheers.

-- 
Ng Pheng Siong <ngps@post1.com> * http://www.post1.com/home/ngps



From barry@zope.com  Mon Oct 29 17:53:23 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 29 Oct 2001 12:53:23 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com>
 <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com>
 <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
 <3BDD91CC.9F0ECB07@interet.com>
 <200110291739.MAA15773@cj20424-a.reston1.va.home.com>
Message-ID: <15325.38803.700797.448117@anthem.wooz.org>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Sounds good to me.  Somebody please update the PEP.

Jim, your last context diff was fine.  Feel free to send me updates in
the same format, and I'll apply them.

-Barry


From tim.one@home.com  Mon Oct 29 18:14:56 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 13:14:56 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BDD91CC.9F0ECB07@interet.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEMKMDAA.tim.one@home.com>

[James C. Ahlstrom]
> The zip archive name is always inserted as the second item
> in sys.path.  The first always seems to be ''.

Running p.py:

import sys
print `sys.path[0]`


on Windows gives:


C:\Code\python\PCbuild>python p.py
''

C:\Code\python\PCbuild>cd ..

C:\Code\python>pcbuild\python pcbuild\p.py
'pcbuild'

C:\Code\python>pcbuild\python \code\python\pcbuild\p.py
'\\code\\python\\pcbuild'

C:\Code\python>cd pcbuild

C:\Code\python\PCbuild>python .\p.py
'.'

C:\Code\python\PCbuild>move p.py ..\lib
C:\CODE\PYTHON\PCBUILD\p.py => C:\CODE\PYTHON\lib\p.py [ok]

C:\Code\python\PCbuild>python ..\lib\p.py
'..\\lib'

C:\Code\python\PCbuild>

That is, the first entry is the path (relative or absolute!) to the
directory in which the script being executed lives.



From tim.one@home.com  Mon Oct 29 18:22:10 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 13:22:10 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <200110291516.KAA14105@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMKMDAA.tim.one@home.com>

[Guido]
> I'd be happy to legislate uncompressed archives, except I'm worried that
> not all tools commonly used to create zip archives make it easy to turn
> off compression.

FWIW, WinZip has an easy-to-get-at "compression: none" setting.  I don't
know whether that creates "an uncompress archive" from Jim's POV, but it
sure doesn't save any space <wink>.



From barry@zope.com  Mon Oct 29 18:53:00 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 29 Oct 2001 13:53:00 -0500
Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions
Message-ID: <15325.42380.21091.468633@anthem.wooz.org>

Attached is the latest version of PEP 274 for dictionary
comprehensions, similar to list comprehensions.  The first version of
this PEP wasn't posted here, but several people saw the cvs checkin
and sent in comments, which have now been incorporated.

Enjoy,
-Barry

-------------------- snip snip --------------------
PEP: 274
Title: Dict Comprehensions
Version: $Revision: 1.2 $
Last-Modified: $Date: 2001/10/29 18:46:59 $
Author: barry@zope.com (Barry A. Warsaw)
Status: Draft
Type: Standards Track
Created: 25-Oct-2001
Python-Version: 2.3
Post-History: 29-Oct-2001


Abstract

    PEP 202 introduces a syntactical extension to Python called the
    "list comprehension"[1].  This PEP proposes a similar syntactical
    extension called the "dictionary comprehension" or "dict
    comprehension" for short.  You can use dict comprehensions in ways
    very similar to list comprehensions, except that they produce
    Python dictionary objects instead of list objects.


Proposed Solution

    Dict comprehensions are just like list comprehensions, except that
    you group the expression using curly braces instead of square
    braces.  Also, the left part before the `for' keyword expresses
    both a key and a value, separated by a colon.  (There is an
    optional part of this PEP that allows you to use a shortcut to
    express just the value.)  The notation is specifically designed to
    remind you of list comprehensions as applied to dictionaries.


Rationale

    There are times when you have some data arranged as a sequences of
    length-2 sequences, and you want to turn that into a dictionary.
    In Python 2.2, the dictionary() constructor will take an optional
    keyword argument that indicates specifically to interpret a
    sequences of length-2 sequences as key/value pairs, and turn them
    into a dictionary.

    However, the act of turning some data into a sequence of length-2
    sequences can be inconvenient or inefficient from a memory or
    performance standpoint.  Also, for some common operations, such as
    turning a list of things into a set of things for quick duplicate
    removal or set inclusion tests, a better syntax can help code
    clarity.

    As with list comprehensions, an explicit for loop can always be
    used (and in fact was the only way to do it in earlier versions of
    Python).  But as with list comprehensions, dict comprehensions can
    provide a more syntactically succinct idiom that the traditional
    for loop.


Examples

    >>> print {i : chr(65+i) for i in range(4)}
    {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'}

    >>> print {k : v for k, v in someDict.iteritems()} == someDict.copy()
    1

    >>> print {x.lower() : 1 for x in list_of_email_addrs}
    {'barry@zope.com'   : 1, 'barry@python.org' : 1, 'guido@python.org' : 1}

    >>> def invert(d):
    ...     return {v : k for k, v in d.iteritems()}
    ...
    >>> d = {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'}
    >>> print invert(d)
    {'A' : 0, 'B' : 1, 'C' : 2, 'D' : 3}


Open Issues

    - There is one further shortcut we could adopt.  Suppose we wanted
      to create a set of items, such as in the "list_of_email_addrs"
      example above.  Here, we're simply taking the target of the for
      loop and turning that into the key for the dict comprehension.
      The assertion is that this would be a common idiom, so the
      shortcut below allows for an easy spelling of it, by allow us to
      omit the "key :" part of the left hand clause:

      >>> print {1 for x in list_of_email_addrs}
      {'barry@zope.com'   : 1, 'barry@python.org' : 1, 'guido@python.org' : 1}

      Or say we wanted to map email addresses to the MX record handling
      their mail:

      >>> print {mx_for_addr(x) for x in list_of_email_addrs}
      {'barry@zope.com'   : 'mail.zope.com',
       'barry@python.org' : 'mail.python.org,
       'guido@python.org' : 'mail.python.org,
       }

      Questions: what about nested loops?  Where does the key come
      from?  The shortcut probably doesn't save much typing, and comes
      at the expense of legibility, so it's of dubious value.

    - Should nested for loops be allowed?  The following example,
      taken from an earlier revision of this PEP illustrates the
      problem:

      >>> print {k, v for k in range(4) for v in range(-4, 0, 1)}

      The intent of this example was to produce a mapping from a
      number to its negative, but this code doesn't work because -- as
      in list comprehensions -- the for loops are nested, not in
      parallel!  So the value of this expression is actually

      {0: -1, 1: -1, 2: -1, 3: -1}

      which seems of dubious value.  For symmetry with list
      comprehensions, perhaps this should be allowed, but it might be
      better to disallow this syntax.


Implementation

    The semantics of dictionary comprehensions can actually be modeled
    in stock Python 2.2, by passing a list comprehension to the
    builtin dictionary constructor:

    >>> dictionary([(i, chr(65+i)) for i in range(4)])

    This has two dictinct disadvantages from the proposed syntax
    though.  First, it's isn't as legible as a dict comprehension.
    Second, it forces the programmer to create an in-core list object
    first, which could be expensive.


References

    [1] PEP 202, List Comprehensions
        http://www.python.org/peps/pep-0202.html


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
fill-column: 70
End:


From guido@python.org  Mon Oct 29 19:08:17 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 14:08:17 -0500
Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions
In-Reply-To: Your message of "Mon, 29 Oct 2001 13:53:00 EST."
 <15325.42380.21091.468633@anthem.wooz.org>
References: <15325.42380.21091.468633@anthem.wooz.org>
Message-ID: <200110291908.OAA16263@cj20424-a.reston1.va.home.com>

>     - There is one further shortcut we could adopt.  Suppose we wanted
>       to create a set of items, such as in the "list_of_email_addrs"
>       example above.  Here, we're simply taking the target of the for
>       loop and turning that into the key for the dict comprehension.

Hm, I don't like this.  I think it's confusing: you really have to
think about whether

    {x for x in <whatever>}

produces a dictionary whose keys are in <whatever> or one whose values
are in <whatever>.

>     - Should nested for loops be allowed?  The following example,
>       taken from an earlier revision of this PEP illustrates the
>       problem:
> 
>       >>> print {k, v for k in range(4) for v in range(-4, 0, 1)}

Don't you mean {k: v for ...}?  The second range() also seems like you
meant range(0, -4, -1).

>       The intent of this example was to produce a mapping from a
>       number to its negative, but this code doesn't work because -- as
>       in list comprehensions -- the for loops are nested, not in
>       parallel!  So the value of this expression is actually
> 
>       {0: -1, 1: -1, 2: -1, 3: -1}
> 
>       which seems of dubious value.  For symmetry with list
>       comprehensions, perhaps this should be allowed, but it might be
>       better to disallow this syntax.

Nested for loops can be useful when used properly:

    {(k, v): k+v for k in range(4) for v in range(4)}

Your example should have been expressed using:

    {k: v for k, v in zip(range(4), range(0, -4, -1))}

--Guido van Rossum (home page: http://www.python.org/~guido/)


From bckfnn@worldonline.dk  Mon Oct 29 19:15:00 2001
From: bckfnn@worldonline.dk (Finn Bock)
Date: Mon, 29 Oct 2001 19:15:00 GMT
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <200110291513.KAA14082@cj20424-a.reston1.va.home.com>
References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk>   <3BDD691B.2599AAF4@interet.com>  <200110291513.KAA14082@cj20424-a.reston1.va.home.com>
Message-ID: <3bddaa12.11250086@mail.wanadoo.dk>

[JimA]

> I am confused.  Zip archives are equivalant to subdirectories, so there
> is no requirement to have anything at the top level.  Your example seems
> to imply a second search path besides sys.path.

[GvR]

>I think Finn simply means that the zipfile may have some redundant
>initial suffix to all filenames (e.g. "Lib/") ...

Yes, that was it!

regards,
finn


From bckfnn@worldonline.dk  Mon Oct 29 19:16:21 2001
From: bckfnn@worldonline.dk (Finn Bock)
Date: Mon, 29 Oct 2001 19:16:21 GMT
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BDD6B74.D4E19953@interet.com>
References: <3BD9B5EC.119319A5@interet.com>   <3bd9d419.10386565@mail.wanadoo.dk>  <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> <3BDD6B74.D4E19953@interet.com>
Message-ID: <3bdda1df.9151318@mail.wanadoo.dk>

[me]

> myself), I just didn't dare to put such a limit on my jython
> implementation.

[James C. Ahlstrom]

>I don't understand the jpython implementation, so please point out
>all problems so we can fix them now.

A naive implementation (like the first one I made for Jython) does not
try to handle the cleanup issues.

For example, how much memory have been consumed and not freed after the
last statement in this little silly program:

import sys, zipfile, time

zip = zipfile.ZipFile("archive.zip", "w")
for i in range(10000):
    entry = zipfile.ZipInfo()
    entry.filename = "module%06d.py" % i
    entry.date_time = time.gmtime(time.time())
    zip.writestr(entry, "# Not used\n")
zip.close()

sys.path.append("archive.zip")
try:
   import notfound
except ImportError:
   pass
sys.path.pop()


If I read your preliminary patch correctly, the 10000 entries will
remain in the ArchiveMembers dict forever and that is perfectly fine
after Guido comments on sys.path being mostly a static feature.

Jython manage to clean the member-cache when an archive is removed from
sys.path but is was quite tricky to achieve. We do it by replacing zip
entries in sys.path (like "archive.zip") with a subclass of string
(SyspathArchive) that also holds a reference to the member cache. From
pythons POV the sys.path entry is still a string with the same value
(but with a different id).

regards,
finn


From tim.one@home.com  Mon Oct 29 19:14:39 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 14:14:39 -0500
Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions
In-Reply-To: <15325.42380.21091.468633@anthem.wooz.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com>

[Barry]
> Rationale
>
>     There are times when you have some data arranged as a sequences of
>     length-2 sequences, and you want to turn that into a dictionary.
>     In Python 2.2, the dictionary() constructor will take an optional
>     keyword argument that indicates specifically to interpret a
>     sequences of length-2 sequences as key/value pairs, and turn them
>     into a dictionary.

This is implemented now, but in a different way.  Suggested rewording:

    In Python 2.2, the dictionary() constructor accepts an argument
    that is a sequence of length-2 sequences, used as (key, value)
    pairs to initialize a new dictionary object.


BTW, and not meaning to hijack your PEP <wink>, should dict.update() accept
such an argument too?  I didn't add it because

    d.update(dictionary(such_an_argument))

seemed "almost good enough".

BTW2, are we going to rename "dictionary" to "dict" before 2.2b2?  Before
2.2, "dict" was universally used on c.l.py to mean dictionary, and I'm at
least +0 on adopting that for official 2.2 use.



From barry@zope.com  Mon Oct 29 19:22:58 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 29 Oct 2001 14:22:58 -0500
Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions
References: <15325.42380.21091.468633@anthem.wooz.org>
 <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com>
Message-ID: <15325.44178.193256.554626@anthem.wooz.org>

>>>>> "TP" == Tim Peters <tim.one@home.com> writes:

    TP> BTW, and not meaning to hijack your PEP <wink>, should
    TP> dict.update() accept such an argument too?  I didn't add it
    TP> because

    TP>     d.update(dictionary(such_an_argument))

    TP> seemed "almost good enough".

Agreed.  But either way, I still there there is utility in a dict
comprehension.

    TP> BTW2, are we going to rename "dictionary" to "dict" before
    TP> 2.2b2?  Before 2.2, "dict" was universally used on c.l.py to
    TP> mean dictionary, and I'm at least +0 on adopting that for
    TP> official 2.2 use.

I wouldn't keep them both though.  Use one or the other.

-Barry


From guido@python.org  Mon Oct 29 19:27:32 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 14:27:32 -0500
Subject: [Python-Dev] Dict API refinements (was PEP 274)
In-Reply-To: Your message of "Mon, 29 Oct 2001 14:14:39 EST."
 <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com>
Message-ID: <200110291927.OAA16469@cj20424-a.reston1.va.home.com>

> BTW, and not meaning to hijack your PEP <wink>, should dict.update() accept
> such an argument too?  I didn't add it because
> 
>     d.update(dictionary(such_an_argument))
> 
> seemed "almost good enough".

Agreed; no need to add this to update().

> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2?
> Before 2.2, "dict" was universally used on c.l.py to mean
> dictionary, and I'm at least +0 on adopting that for official 2.2
> use.

Sounds good to me.  Should we adopt "dict" as an alias and keep
"dictionary" as the official name, or vice versa, or simply eradicate
dictionary and introduce dict in 2.2b2?

It's up to you to implement this, I have some other things I need to
get done first.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@home.com  Mon Oct 29 19:45:47 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 14:45:47 -0500
Subject: [Python-Dev] test_asynchat.py broken on Windows
Message-ID: <LNBBLJKPBEHFEDALKOLCGENGMDAA.tim.one@home.com>

Here on Windows, current CVS:

C:\Code\python\PCbuild>python ../lib/test/test_asynchat.py
Traceback (most recent call last):
  File "../lib/test/test_asynchat.py", line 58, in ?
    main()
  File "../lib/test/test_asynchat.py", line 53, in main
    c = echo_client()
  File "../lib/test/test_asynchat.py", line 34, in __init__
    self.connect((HOST, PORT))
  File "C:\CODE\PYTHON\lib\asyncore.py", line 311, in connect
    raise socket.error, err
socket.error: 2

The DOS box is frozen at this point, but not accumulating any cycles.
Worked 12 hours ago.  What about Linux?



From thomas.heller@ion-tof.com  Mon Oct 29 19:54:31 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Mon, 29 Oct 2001 20:54:31 +0100
Subject: [Python-Dev] class dict of subtyped types
Message-ID: <046c01c160b3$875c7340$e000a8c0@thomasnotebook>

C:\>python
Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class INT(int): pass
...
>>> INT.__dict__
<dict-proxy object at 0x00769680>
>>> INT.__dict__.update
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'dict-proxy' object has no attribute 'update'
>>> dir(INT.__dict__)
['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__'
, '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', '
__str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values']
>>>

Other dict attributes are missing as well in the dict-proxy.
Is this intentional?

Thomas



From tim.one@home.com  Mon Oct 29 19:55:36 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 14:55:36 -0500
Subject: [Python-Dev] RE: Dict API refinements (was PEP 274)
In-Reply-To: <200110291927.OAA16469@cj20424-a.reston1.va.home.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com>

[Tim]
>> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2?
>> Before 2.2, "dict" was universally used on c.l.py to mean
>> dictionary, and I'm at least +0 on adopting that for official 2.2
>> use.

[Guido]
> Sounds good to me.  Should we adopt "dict" as an alias and keep
> "dictionary" as the official name, or vice versa, or simply eradicate
> dictionary and introduce dict in 2.2b2?

s/dictionary/dict/g is my preference:  TOOWTDI.

> It's up to you to implement this, I have some other things I need to
> get done first.

That's fine -- I can't think of anything needed I don't know how to do with
ease (heck, I even know where the dictionary() docstring lives <wink>),
except for changing the descr tutorial accordingly on python.org.



From fdrake@acm.org  Mon Oct 29 19:53:49 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Mon, 29 Oct 2001 14:53:49 -0500
Subject: [Python-Dev] RE: Dict API refinements (was PEP 274)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com>
References: <200110291927.OAA16469@cj20424-a.reston1.va.home.com>
 <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com>
Message-ID: <15325.46029.119091.651467@grendel.zope.com>

Tim Peters writes:
 > That's fine -- I can't think of anything needed I don't know how to do with
 > ease (heck, I even know where the dictionary() docstring lives <wink>),
 > except for changing the descr tutorial accordingly on python.org.

  I'll volunteer to take care of that one.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From guido@python.org  Mon Oct 29 19:59:51 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 14:59:51 -0500
Subject: [Python-Dev] RE: Dict API refinements (was PEP 274)
In-Reply-To: Your message of "Mon, 29 Oct 2001 14:55:36 EST."
 <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com>
References: <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com>
Message-ID: <200110291959.OAA16703@cj20424-a.reston1.va.home.com>

> [Tim]
> >> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2?
> >> Before 2.2, "dict" was universally used on c.l.py to mean
> >> dictionary, and I'm at least +0 on adopting that for official 2.2
> >> use.
> 
> [Guido]
> > Sounds good to me.  Should we adopt "dict" as an alias and keep
> > "dictionary" as the official name, or vice versa, or simply eradicate
> > dictionary and introduce dict in 2.2b2?

[Tim]
> s/dictionary/dict/g is my preference:  TOOWTDI.

[Guido]
OK, let's do that.

> > It's up to you to implement this, I have some other things I need to
> > get done first.
> 
> That's fine -- I can't think of anything needed I don't know how to do with
> ease (heck, I even know where the dictionary() docstring lives <wink>),
> except for changing the descr tutorial accordingly on python.org.

OK, I'll fix that one.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Oct 29 20:04:12 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 15:04:12 -0500
Subject: [Python-Dev] class dict of subtyped types
In-Reply-To: Your message of "Mon, 29 Oct 2001 20:54:31 +0100."
 <046c01c160b3$875c7340$e000a8c0@thomasnotebook>
References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook>
Message-ID: <200110292004.PAA16731@cj20424-a.reston1.va.home.com>

> C:\>python
> Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> class INT(int): pass
> ...
> >>> INT.__dict__
> <dict-proxy object at 0x00769680>
> >>> INT.__dict__.update
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> AttributeError: 'dict-proxy' object has no attribute 'update'
> >>> dir(INT.__dict__)
> ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__'
> , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', '
> __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values']
> >>>
> 
> Other dict attributes are missing as well in the dict-proxy.
> Is this intentional?

The dict-proxy type is intended to provide a read-only proxy, so the
dict-proxy is consciously lacking the update, clear, popitem and
setdefault methods.  It also seems to be missing the comparison
operations; that's an oversight.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Mon Oct 29 20:09:48 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Mon, 29 Oct 2001 21:09:48 +0100
Subject: [Python-Dev] class dict of subtyped types
References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook>  <200110292004.PAA16731@cj20424-a.reston1.va.home.com>
Message-ID: <055601c160b5$a9473dd0$e000a8c0@thomasnotebook>

From: "Guido van Rossum" <guido@python.org>
> > C:\>python
> > Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> class INT(int): pass
> > ...
> > >>> INT.__dict__
> > <dict-proxy object at 0x00769680>
> > >>> INT.__dict__.update
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> > AttributeError: 'dict-proxy' object has no attribute 'update'
> > >>> dir(INT.__dict__)
> > ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__'
> > , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', '
> > __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values']
> > >>>
> > 
> > Other dict attributes are missing as well in the dict-proxy.
> > Is this intentional?
> 
> The dict-proxy type is intended to provide a read-only proxy, so the
> dict-proxy is consciously lacking the update, clear, popitem and
> setdefault methods.  It also seems to be missing the comparison
> operations; that's an oversight.
> 
But adding attributes to the INT class above after creation
is allowed, I hope? At least it currently works.

Thomas



From guido@python.org  Mon Oct 29 20:12:19 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 15:12:19 -0500
Subject: [Python-Dev] class dict of subtyped types
In-Reply-To: Your message of "Mon, 29 Oct 2001 21:09:48 +0100."
 <055601c160b5$a9473dd0$e000a8c0@thomasnotebook>
References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> <200110292004.PAA16731@cj20424-a.reston1.va.home.com>
 <055601c160b5$a9473dd0$e000a8c0@thomasnotebook>
Message-ID: <200110292012.PAA16819@cj20424-a.reston1.va.home.com>

> > The dict-proxy type is intended to provide a read-only proxy, so the
> > dict-proxy is consciously lacking the update, clear, popitem and
> > setdefault methods.  It also seems to be missing the comparison
> > operations; that's an oversight.
> > 
> But adding attributes to the INT class above after creation
> is allowed, I hope? At least it currently works.

Yes, but you have to use the attribute notation:

    INT.foo = ...

this is trapped so that certain magic happens automatically when you
set attributes that override operators, e.g.

    INT.__add__ = lambda ...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Mon Oct 29 20:57:38 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 29 Oct 2001 15:57:38 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <LNBBLJKPBEHFEDALKOLCEEMKMDAA.tim.one@home.com>
Message-ID: <3BDDC2C2.E26289D5@interet.com>

Tim Peters wrote:
>
> That is, the first entry is the path (relative or absolute!) to the
> directory in which the script being executed lives.

Right you are!

JimA


From jeremy@alum.mit.edu  Mon Oct 29 23:20:14 2001
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 29 Oct 2001 18:20:14 -0500 (EST)
Subject: [Python-Dev] Re: test_asynchat.py broken on Windows
Message-ID: <200110292320.SAA11414@newman.concentric.net>

I think there are at least three things wrong, depending on what you
count as wrong.

The test is hanging because it spawns a thread, which Python will
prevent Python from exiting until the thread exits.  The thread
doesn't exit because it's waiting for input from the client.  The
client doesn't send it any input because of the exception.

thing wrong #1: It would probably make sense to change
test_asynchat.py to use a daemon thread so that it can't prevent the
test suite from exiting.

thing wrong #2: I don't think the ENONET being returned by
connect_ex() makes any sense.  My guess is that connect_ex() is (and
always has been) broken on Windows.  Could you apply this patch and
see what happens:

Index: socketmodule.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Modules/socketmodule.c,v
retrieving revision 1.190
diff -c -c -r1.190 socketmodule.c
*** socketmodule.c	2001/10/28 12:31:33	1.190
--- socketmodule.c	2001/10/29 23:14:18
***************
*** 1268,1274 ****
--- 1268,1278 ----
  	res = connect(s->sock_fd, addr, addrlen);
  	Py_END_ALLOW_THREADS
  	if (res != 0)
+ #ifdef MS_WINDOWS
+ 		res = WSAGetLastError();
+ #else
  		res = errno;
+ #endif
  	return PyInt_FromLong((long) res);
  }
  
thing wrong #3: From the winsock documentation for connect(), it looks
like there are more errors we should be catching in the test that
includes EWOULDBLOCK and EALREADY.  I don't think this would affect
your test, though.

thing wrong #4: It doesn't look to me like the connect() call in
test_asynchat.py is guaranteed to succeed.  It should succeed almost
every time, but there's some small chance it will fail.  Perhaps the
test should be more robust about that.

That's four things, but I don't think the last thing is all that wrong.

Jeremy



From Anthony Baxter <anthony@interlink.com.au>  Mon Oct 29 23:53:41 2001
From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter)
Date: Tue, 30 Oct 2001 10:53:41 +1100
Subject: [Python-Dev] Fixing send()
In-Reply-To: Message from Anthony Baxter <anthony@interlink.com.au>
 of "Thu, 25 Oct 2001 16:24:20 +1000."
Message-ID: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au>

> Will do - I plan to check in a bunch of fixes tomorrow morning. (continuing
> gdb hell today).

Just an update to this - my plans were wrecked by a cablemodem outage,
however I'm spending tomorrow on branch mgmt - work is keen for me to 
take some of the 6 weeks leave I've built up, so I'm taking a few days 
off to work on this. 

As far as trying to delegate checkins of code - would people prefer I
post a big list here, mail the original authors of the patches, or just
do it myself?

Anthony


From greg@cosc.canterbury.ac.nz  Tue Oct 30 00:39:41 2001
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 Oct 2001 13:39:41 +1300 (NZDT)
Subject: [Python-Dev] Dict API refinements (was PEP 274)
In-Reply-To: <200110291927.OAA16469@cj20424-a.reston1.va.home.com>
Message-ID: <200110300039.NAA09256@s454.cosc.canterbury.ac.nz>

Guido van Rossum <guido@python.org>:

> Should we adopt "dict" as an alias and keep
> "dictionary" as the official name, or vice versa, or simply eradicate
> dictionary and introduce dict in 2.2b2?

I'd say eradicate "dictionary". There Should Only Be One
Way To Spell It.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From tim.one@home.com  Tue Oct 30 01:37:49 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 20:37:49 -0500
Subject: [Python-Dev] RE: test_asynchat.py broken on Windows
In-Reply-To: <200110292320.SAA11414@newman.concentric.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEPEMDAA.tim.one@home.com>

[Jeremy Hylton]
> I think there are at least three things wrong, depending on what you
> count as wrong.

I would have guessed five, if you were counting Windows too <wink>.

> The test is hanging because it spawns a thread, which Python will
> prevent Python from exiting until the thread exits.  The thread
> doesn't exit because it's waiting for input from the client.  The
> client doesn't send it any input because of the exception.
>
> thing wrong #1: It would probably make sense to change
> test_asynchat.py to use a daemon thread so that it can't prevent the
> test suite from exiting.

I don't much care about this.

> thing wrong #2: I don't think the ENONET being returned by
> connect_ex() makes any sense.  My guess is that connect_ex() is (and
> always has been) broken on Windows.  Could you apply this patch and
> see what happens:
> ...

Fixed the problem, so I checked it in.  Thanks!

> thing wrong #3: From the winsock documentation for connect(), it looks
> like there are more errors we should be catching in the test that
> includes EWOULDBLOCK and EALREADY.  I don't think this would affect
> your test, though.

I'm not sure I parsed that as intended; asyncore.dispatcher.connect (in the
traceback when test_asynchat failed) is already checking for EINPROGRESS,
EALREADY, EWOULDBLOCK, 0 and EISCONN returns from connect_ex.  But, for
posterity, I will inscribe my socket programming knowledge in the period at
the end of this sentence.

> thing wrong #4: It doesn't look to me like the connect() call in
> test_asynchat.py is guaranteed to succeed.  It should succeed almost
> every time, but there's some small chance it will fail.  Perhaps the
> test should be more robust about that.

FWIW, I've never seen it fail before.

> That's four things, but I don't think the last thing is all that wrong.

Here's a fifth:

def main():
    s = echo_server()
    s.start()
    time.sleep(1) # Give server time to initialize
    c = echo_client()
    ...

time.sleep()-is-not-a-synchronization-gimmick-ly y'rs  - tim



From tim.one@home.com  Tue Oct 30 02:01:40 2001
From: tim.one@home.com (Tim Peters)
Date: Mon, 29 Oct 2001 21:01:40 -0500
Subject: [Python-Dev] Fixing send()
In-Reply-To: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEPFMDAA.tim.one@home.com>

[Anthony Baxter]
> Just an update to this - my plans were wrecked by a cablemodem outage,
> however I'm spending tomorrow on branch mgmt - work is keen for me to
> take some of the 6 weeks leave I've built up, so I'm taking a few days
> off to work on this.

We have similar views on what constitutes "a vacation" <wink>.

> As far as trying to delegate checkins of code - would people prefer I
> post a big list here, mail the original authors of the patches, or just
> do it myself?

Do it yourself so far as is humanly possible.  That way (a) it will get
done; and (b) the developers won't come to resent bugfix releases (if we had
time to spare for this, we wouldn't ask for volunteers).

it's-a-job-as-well-as-an-easy-way-to-burn-excess-vacation-days-ly
    y'rs  - tim



From guido@python.org  Tue Oct 30 02:03:33 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 21:03:33 -0500
Subject: [Python-Dev] Fixing send()
In-Reply-To: Your message of "Tue, 30 Oct 2001 10:53:41 +1100."
 <200110292353.f9TNrfo27736@mbuna.arbhome.com.au>
References: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au>
Message-ID: <200110300203.VAA19633@cj20424-a.reston1.va.home.com>

> As far as trying to delegate checkins of code - would people prefer I
> post a big list here, mail the original authors of the patches, or just
> do it myself?

In general, trying to reach the original authors will just add
delays.  Do it yourself, or if you're not sure, post here.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From greg@cosc.canterbury.ac.nz  Tue Oct 30 02:25:09 2001
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 Oct 2001 15:25:09 +1300 (NZDT)
Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions
In-Reply-To: <15325.42380.21091.468633@anthem.wooz.org>
Message-ID: <200110300225.PAA09272@s454.cosc.canterbury.ac.nz>

+1 generally on dict comprehensions, but:

> >>> print {1 for x in list_of_email_addrs}
>      {'barry@zope.com'   : 1, 'barry@python.org' : 1, 'guido@python.org' : 1}

-1 on this bit. It's not at all clear what it should mean,
and the saving over writing it out explicitly, i.e.

  {x:1 for x in list_of_email_addrs}

is vanishingly small.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From guido@python.org  Tue Oct 30 02:28:47 2001
From: guido@python.org (Guido van Rossum)
Date: Mon, 29 Oct 2001 21:28:47 -0500
Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions
In-Reply-To: Your message of "Tue, 30 Oct 2001 15:25:09 +1300."
 <200110300225.PAA09272@s454.cosc.canterbury.ac.nz>
References: <200110300225.PAA09272@s454.cosc.canterbury.ac.nz>
Message-ID: <200110300228.VAA21968@cj20424-a.reston1.va.home.com>

> +1 generally on dict comprehensions, but:
> 
> > >>> print {1 for x in list_of_email_addrs}
> >      {'barry@zope.com'   : 1, 'barry@python.org' : 1, 'guido@python.org' : 1}
> 
> -1 on this bit. It's not at all clear what it should mean,
> and the saving over writing it out explicitly, i.e.
> 
>   {x:1 for x in list_of_email_addrs}
> 
> is vanishingly small.

You're getting better at channeling me. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fdrake@acm.org  Tue Oct 30 06:23:32 2001
From: fdrake@acm.org (Fred L. Drake)
Date: Tue, 30 Oct 2001 01:23:32 -0500 (EST)
Subject: [Python-Dev] [development doc updates]
Message-ID: <20011030062332.77F4928697@cj42289-a.reston1.va.home.com>

The development version of the documentation has been updated:

    http://python.sourceforge.net/devel-docs/

Re-arranged the material in Chapter 2 (built-in things) to make it easier to
start off with.  Functions come before Types and Exceptions, and file objects
are promoted one level in the outline, making them easier to find (they now
appear in the table of contents instead of being hidden in the "Other
Objects" category).



From mal@lemburg.com  Tue Oct 30 08:25:28 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 30 Oct 2001 09:25:28 +0100
Subject: [Python-Dev] Slices and "==" optimization
Message-ID: <3BDE63F8.541E04B2@lemburg.com>

While writing yet another XML parser I have come across two
things which would seem nice to have:

1. Even though you can construct slice objects in Python
   using slice(left,right), you can't really do anything
   with them; at least not on the standard types (Numeric's
   arrays work with them just fine).

   Couldn't we add some functionality which makes them
   compatible to lists and tuples (and perhaps even in a
   generic way for other sequences using PySequence_GetItem())
   too ?

   I thinking of extending the __getitem__ hook to accept
   slice objects. You could then maintain slices to an
   object in say a list and apply them to the underlying text
   as needed, e.g.

   s = slice(1,4)
   l = range(10)
   l[s] == [1, 2, 3]

2. Looking through the code for '==' the optimization for
   'a == a' seems too far down the call stack. Since interning
   strings makes this case rather common, I'd suggest to
   lift the optimization into the ceval loop (right along side
   with the INT op INT optimization).

Thoughts ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From martin@v.loewis.de  Tue Oct 30 08:42:12 2001
From: martin@v.loewis.de (Martin v. Loewis)
Date: Tue, 30 Oct 2001 09:42:12 +0100
Subject: [Python-Dev] Slices and "==" optimization
Message-ID: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de>

> I thinking of extending the __getitem__ hook to accept  slice objects

Please have a look at

http://sourceforge.net/tracker/?func=detail&aid=459235&group_id=5470&atid=105470

It's a known problem, but Guido won't fix it anytime soon. So I guess
contributions are welcome.

> Looking through the code for '==' the optimization for 'a == a'
> seems too far down the call stack. Since interning strings makes
> this case rather common, I'd suggest to lift the optimization into
> the ceval loop

As with all optimizations: Patches to just improve the performance are
unacceptable unless accompanied by some hard numbers to show that they
really do have the desired effect. 

History shows that patches to just improve the performance may be
unacceptable even if accompanied by such numbers. Normally, Tim will
construct a scenario where the patch causes a slow-down, and argue
that this is a common scenario :-)

Regards,
Martin


From fredrik@pythonware.com  Tue Oct 30 08:56:07 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 30 Oct 2001 09:56:07 +0100
Subject: [Python-Dev] RE: test_asynchat.py broken on Windows
References: <LNBBLJKPBEHFEDALKOLCGEPEMDAA.tim.one@home.com>
Message-ID: <00c901c16120$b954e180$ced241d5@hagrid>

> [Jeremy Hylton]
> > I think there are at least three things wrong, depending on what you
> > count as wrong.

not sure jeremy gets mail sent to his sourceforge address,
but maybe someone else can explain how his recent patch
preserves the original behaviour.

after the patch, connect will always raise an exception unless
the error code is one of the three listed code.  before the patch,
it will not raise an exception if the connection succeeds during
the connect itself (i.e. if people use asyncore with blocking
sockets, or if they're using a hypothetical transport that can
do an immediate connect).

how about adding an "else:" before that last raise?

     def connect (self, address):
          self.connected = 0
!         # XXX why not use connect_ex?
!         try:
!             self.socket.connect (address)
!         except socket.error, why:
!             if why[0] in (EINPROGRESS, EALREADY, EWOULDBLOCK):
!                 return
!             else:
!                 raise socket.error, why
!         self.addr = address
!         self.connected = 1
!         self.handle_connect()
  
      def accept (self):
--- 302,313 ----
      def connect (self, address):
          self.connected = 0
!         err = self.socket.connect_ex(address)
!         if err in (EINPROGRESS, EALREADY, EWOULDBLOCK):
!             return
!         if err in (0, EISCONN):
!             self.addr = address
!             self.connected = 1
!             self.handle_connect()
!         raise socket.error, err
  
      def accept (self):




From mal@lemburg.com  Tue Oct 30 08:55:09 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 30 Oct 2001 09:55:09 +0100
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> <3BDD6D66.675A4F3C@interet.com>
Message-ID: <3BDE6AED.25D67499@lemburg.com>

"James C. Ahlstrom" wrote:
> 
> "M.-A. Lemburg" wrote:
> 
> > 1. Why are .py files being outlawed ?
> 
> My reason was that if *.py[co] are missing or out of date,
> zip importing will be slow and users won't figure
> out what is wrong.  Generally I favor user-proof features
> over expert features.  I prefer things which either "Just Work"
> or fail spectacularly.  But I am open to changing this.

If you don't include the *.py file in the archive, chances
are high that tracebacks will no longer print out as they
do today (another problem here is that the filename being
used in the code object will probably not be found... not sure
whether we can fix this one).

By only looking at the .pyc or .pyo you'll also introduce
a Python version problem into the ZIP-archive: the magic in
these files changs rather frequently and e.g. a more
recent release of Python won't be able to load these from
a ZIP-archive which only contains .pycs from a compile
by a less recent version.
 
> > 2. Where's the C implementation you mention in the PEP ?
> 
> Software is like pancakes, you should always
> throw the first one away.  I will post it if you want,
> but it is not done.

That's true :-) Still, I'd like a chance to at least look
at the impact this has on import.c and of course play
with it so that I can test it in everyday situations.
 
> > 3. Would it be possible to ship zlib together with Python ?
> >    (the zlib license should allow this and I don't think that
> >    the code size is too big)
> 
> OK by me.  But uncompressed zip archives work, and may even
> be faster than conpressed archives.

Even if uncompressed archive would work faster, the compiled 
libz is only 70k on Linux and I think we could solve
a great number of (zlib version) problems by including zlib in 
Python. It's one of those basic tools you need frequently, but
is even more frequently not configured as Python module :-(

Since it is already included in the Windows builds, I guess
adding it to core for Unix and Macs too wouldn't hurt all
that much. 

It would also save you the trouble of maintaining the code for 
scanning uncompressed zip-archives in your ZIP import code, so
we win on two counts :-)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From mal@lemburg.com  Tue Oct 30 10:12:43 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 30 Oct 2001 11:12:43 +0100
Subject: [Python-Dev] Slices and "==" optimization
References: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de>
Message-ID: <3BDE7D1B.4850329C@lemburg.com>

"Martin v. Loewis" wrote:
> 
> > I thinking of extending the __getitem__ hook to accept  slice objects
> 
> Please have a look at
> 
> http://sourceforge.net/tracker/?func=detail&aid=459235&group_id=5470&atid=105470
> 
> It's a known problem, but Guido won't fix it anytime soon. So I guess
> contributions are welcome.

Good to know, thanks.
 
> > Looking through the code for '==' the optimization for 'a == a'
> > seems too far down the call stack. Since interning strings makes
> > this case rather common, I'd suggest to lift the optimization into
> > the ceval loop
> 
> As with all optimizations: Patches to just improve the performance are
> unacceptable unless accompanied by some hard numbers to show that they
> really do have the desired effect.
> 
> History shows that patches to just improve the performance may be
> unacceptable even if accompanied by such numbers. Normally, Tim will
> construct a scenario where the patch causes a slow-down, and argue
> that this is a common scenario :-)

Sounds like you have experience ;-) 

As with all micro-optimizations, the win usually only shows
up in some applications. The one I'm writing uses lot's of

	if tagtype == 'starttag':
	if tagtype == 'endtag':
	...

Since the 'starttag' and 'endtag' strings are interned and
I'm also interning the strings which are stored in tagtype,
I'd like to benefit from the fact that the compare will actually
work as 'is'-compare. However, I don't want to use 'is' because
I consider interning only an optimization and not a feature
of the language.

That's why I would like the simple

	if (v == w) return 0;

integrated into the ceval loop right along the INT-compare
optimization. 

This may also help in some other situations where objects are
shared.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From guido@python.org  Tue Oct 30 13:50:47 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 30 Oct 2001 08:50:47 -0500
Subject: [Python-Dev] Slices and "==" optimization
In-Reply-To: Your message of "Tue, 30 Oct 2001 09:25:28 +0100."
 <3BDE63F8.541E04B2@lemburg.com>
References: <3BDE63F8.541E04B2@lemburg.com>
Message-ID: <200110301350.IAA24747@cj20424-a.reston1.va.home.com>

> 1. Even though you can construct slice objects in Python
>    using slice(left,right), you can't really do anything
>    with them; at least not on the standard types (Numeric's
>    arrays work with them just fine).
> 
>    Couldn't we add some functionality which makes them
>    compatible to lists and tuples (and perhaps even in a
>    generic way for other sequences using PySequence_GetItem())
>    too ?
> 
>    I thinking of extending the __getitem__ hook to accept
>    slice objects. You could then maintain slices to an
>    object in say a list and apply them to the underlying text
>    as needed, e.g.
> 
>    s = slice(1,4)
>    l = range(10)
>    l[s] == [1, 2, 3]

I agree that this would be nice to have.  I won't have time for it
before 2.2 is out, but if you want to give it a crack, I might accept
it.

> 2. Looking through the code for '==' the optimization for
>    'a == a' seems too far down the call stack. Since interning
>    strings makes this case rather common, I'd suggest to
>    lift the optimization into the ceval loop (right along side
>    with the INT op INT optimization).

I would agree, except there's also the point of view that since types
can override __eq__ to always return false, this optimization should
not be done in a way that prevents __eq__ from overriding it.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Tue Oct 30 14:00:13 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 30 Oct 2001 09:00:13 -0500
Subject: [Python-Dev] Slices and "==" optimization
In-Reply-To: Your message of "Tue, 30 Oct 2001 11:12:43 +0100."
 <3BDE7D1B.4850329C@lemburg.com>
References: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de>
 <3BDE7D1B.4850329C@lemburg.com>
Message-ID: <200110301400.JAA24868@cj20424-a.reston1.va.home.com>

> As with all micro-optimizations, the win usually only shows
> up in some applications. The one I'm writing uses lot's of
> 
> 	if tagtype == 'starttag':
> 	if tagtype == 'endtag':
> 	...
> 
> Since the 'starttag' and 'endtag' strings are interned and
> I'm also interning the strings which are stored in tagtype,
> I'd like to benefit from the fact that the compare will actually
> work as 'is'-compare. However, I don't want to use 'is' because
> I consider interning only an optimization and not a feature
> of the language.

Agreed.

> That's why I would like the simple
> 
> 	if (v == w) return 0;
> 
> integrated into the ceval loop right along the INT-compare
> optimization. 

Maybe this could be done as follows:

    if (v == w && PyString_CheckExact(v)) return 0;

> This may also help in some other situations where objects are
> shared.

Now *that* is the speculation for which Tim will chop off your
head. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From akuchlin@mems-exchange.org  Tue Oct 30 14:48:34 2001
From: akuchlin@mems-exchange.org (Andrew Kuchling)
Date: Tue, 30 Oct 2001 09:48:34 -0500
Subject: [Python-Dev] 2.2 article updated
Message-ID: <E15yaBu-0003xq-00@ute.cnri.reston.va.us>

I've updated the 2.2 article with a section that gives an overview of
the type/class changes.  At this point the text should be essentially
done, except for whatever small changes get into CVS between now and
2.2final.  Please take a look and send me any comments:

http://www.amk.ca/python/2.2/

(Oops!  Michael McLay just sent me a note pointing out that I refer to
.__get__() as .get() in a few places.  So don't bother reporting that.)

If your favorite patch or improvement isn't mentioned, please let me
know.  Also inform me if you aren't mentioned as a contributor to a
patch.

BTW, I call PEP 261, "Support for "wide" Unicode characters", not
finally accepted and not implemented.  Is that correct?  Is it going
to be finished before 2.2final, or should I simply describe it as
partially implemented in the final version of the text?

--amk


From martin@v.loewis.de  Tue Oct 30 17:54:55 2001
From: martin@v.loewis.de (Martin v. Loewis)
Date: Tue, 30 Oct 2001 18:54:55 +0100
Subject: [Python-Dev] Slices and "==" optimization
In-Reply-To: message from Guido van Rossum on Tue, 30 Oct 2001 09:00:13 -0500
Message-ID: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de>

> > That's why I would like the simple
> > 
> > 	if (v == w) return 0;
> > 
> > integrated into the ceval loop right along the INT-compare
> > optimization. 
> 
> Maybe this could be done as follows:
> 
>     if (v == w && PyString_CheckExact(v)) return 0;

Maybe I'm missing some context here: where is this fragment supposed
to go to? into ceval.c:COMPARE_OP? What is the "return 0;" doing then?

In any case, I think measurements should show how much speed
improvement is gained by taking this short-cut. It sounds nice in
theory, but ...

I just added the block

		else if ((v == w) && (oparg == EQ) && PyString_CheckExact(v)) {
			x = Py_True;
			Py_INCREF(x);
		}

into the code. In an application that almost exclusively does
COMPARE_OPs on identical strings, I got a 30% speed-up. OTOH, this
same code caused a 10% slowdown if I converted the "==" into "<>".

In a real application, total speed-up will depend on two things:
- how many COMPARE_OPs are done in the code?
- how many of those compare identical strings for equality?

Running the PyXML test suite, I counted 120000 cases where
slow_compare was done, and only 700 cases identical strings were
compared for equality.

Regards,
Martin


From andymac@bullseye.apana.org.au  Tue Oct 30 12:15:28 2001
From: andymac@bullseye.apana.org.au (Andrew MacIntyre)
Date: Tue, 30 Oct 2001 23:15:28 +1100 (EDT)
Subject: [Python-Dev] OS/2 VAC++ patches
Message-ID: <Pine.OS2.4.32.0110302258080.643-100000@central>

I have looked at Michael Muller's VAC++ patches, and added comments as
followups.

In summary:
473749 - has several (minor IMO) stylistic issues (addition of OS/2
         #ifdefs).  The intent of the changes looks OK, as do most of
         the actual changes (those in OS/2 VAC specific files or
         existing OS/2 #ifdefs).
474169 - looks good to go (changes inside existing OS/2 #ifdef)
474500 - looks good to go (isolated to OS/2 specific file)

Comments on the stylistic issues?

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
        andymac@pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia



From fdrake@acm.org  Tue Oct 30 18:05:11 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 30 Oct 2001 13:05:11 -0500
Subject: [Python-Dev] Slices and "==" optimization
In-Reply-To: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de>
References: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de>
Message-ID: <15326.60375.319250.547944@grendel.zope.com>

Martin v. Loewis writes:
 > into the code. In an application that almost exclusively does
 > COMPARE_OPs on identical strings, I got a 30% speed-up. OTOH, this
 > same code caused a 10% slowdown if I converted the "==" into "<>".

  It's very questionable whether we'll find an optimization that works
well for any serious variety of applications.  Perhaps there's some
way to build up a lookup table of fast paths, and have the slow path
try that first, but I'm sceptical.

 > Running the PyXML test suite, I counted 120000 cases where
 > slow_compare was done, and only 700 cases identical strings were
 > compared for equality.

  I wouldn't expect XML applications to gain from such an
optimization, since many compares will involve Unicode objects, either
to each other or to 8-bit strings.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation



From martin@v.loewis.de  Tue Oct 30 18:41:10 2001
From: martin@v.loewis.de (Martin v. Loewis)
Date: Tue, 30 Oct 2001 19:41:10 +0100
Subject: [Python-Dev] 2.2 article updated
Message-ID: <200110301841.f9UIfAA04832@mira.informatik.hu-berlin.de>

> BTW, I call PEP 261, "Support for "wide" Unicode characters", not
> finally accepted and not implemented.  Is that correct?  Is it going
> to be finished before 2.2final, or should I simply describe it as
> partially implemented in the final version of the text?

To my knowledge, it is completely implemented. You may mention that
--disable-unicode is now also supported.

There is atleast one error in the PEP, namely that the define for wide
Unicode Py_UNICODE_WIDE (Py_UNICODE_SIZE also exists, but isn't used
except to define Py_UNICODE_WIDE).

I'm also not sure certain the all codecs handle all surrogate cases
correctly in all circumstances, but that is in the realm of bug
reports.

One further Unicode change: The bullet item "When presented with a
Unicode filename on Windows ..." should be extended with "On Unix, the
locale's character set is used if nl_langinfo(CODESET) is available."

Two new builtins are missing in your text, OverflowWarning and super.

HTH,
Martin


From mmuller@enduden.com  Tue Oct 30 18:42:45 2001
From: mmuller@enduden.com (Michael Muller)
Date: Tue, 30 Oct 2001 13:42:45 -0500
Subject: [Python-Dev] Re: OS/2 VAC++ patches
Message-ID: <222.1004467365.385140.1552565879@lilbastard>

Andrew,

Thanks for reviewing my patches.  I am posting a new patch in response to
your comments on *749.

Andrew MacIntyre wrote:
> I have looked at Michael Muller's VAC++ patches, and added comments as
> followups.
> 
> In summary:
> 473749 - has several (minor IMO) stylistic issues (addition of OS/2
>          #ifdefs).  The intent of the changes looks OK, as do most of
>          the actual changes (those in OS/2 VAC specific files or
>          existing OS/2 #ifdefs).
> 474169 - looks good to go (changes inside existing OS/2 #ifdef)
> 474500 - looks good to go (isolated to OS/2 specific file)
> 
> Comments on the stylistic issues?
> 
> --
> Andrew I MacIntyre                     "These thoughts are mine alone..."
> E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
>         andymac@pcug.org.au            |        Belconnen  ACT  2616
> Web:    http://www.andymac.org/        |        Australia


=============================================================================
michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus
-----------------------------------------------------------------------------
There is no concept that is more demeaning to the human spirit than the
notion that our freedom must be limited in the interests of our own
protection.
=============================================================================


From mal@lemburg.com  Tue Oct 30 20:07:21 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 30 Oct 2001 21:07:21 +0100
Subject: [Python-Dev] Slices and "==" optimization
References: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de>
Message-ID: <3BDF0879.D758A1A8@lemburg.com>

"Martin v. Loewis" wrote:
> 
> > > That's why I would like the simple
> > >
> > >     if (v == w) return 0;
> > >
> > > integrated into the ceval loop right along the INT-compare
> > > optimization.
> >
> > Maybe this could be done as follows:
> >
> >     if (v == w && PyString_CheckExact(v)) return 0;
> 
> Maybe I'm missing some context here: where is this fragment supposed
> to go to? into ceval.c:COMPARE_OP? What is the "return 0;" doing then?

It's pseudo code I made up. The real thing will look very much
like it though.
 
> In any case, I think measurements should show how much speed
> improvement is gained by taking this short-cut. It sounds nice in
> theory, but ...
> 
> I just added the block
> 
>                 else if ((v == w) && (oparg == EQ) && PyString_CheckExact(v)) {
>                         x = Py_True;
>                         Py_INCREF(x);
>                 }

You should first test for oparg == EQ, then for v == w and
PyString_CheckExact().
 
> into the code. In an application that almost exclusively does
> COMPARE_OPs on identical strings, I got a 30% speed-up. 

Nice :-)

> OTOH, this
> same code caused a 10% slowdown if I converted the "==" into "<>".

This sounds like an alignment problem caused by the compiler.
10% speedups/slowdowns can easily be produced by randomly moving
about a few cases in the ceval loop. All depends on the platform
and compiler though.
 
> In a real application, total speed-up will depend on two things:
> - how many COMPARE_OPs are done in the code?
> - how many of those compare identical strings for equality?

As I explained in my other reply, I have code which does:

	if variable == 'constant string': ...

Since the compiler interns the 'constant string' and I can
make sure that variable is interned too (by calling intern()),
I can easily take advantage of the optimization without
giving away any semantics.
 
If variable happens to be some other constant which is
used in a Python module (e.g. some kind of flag or parameter),
then it will be interned per-se too.

Note that the dictionary implementation relies heavily
on the same optimization (interning was added to enhance
string compare performance in dict lookups).

> Running the PyXML test suite, I counted 120000 cases where
> slow_compare was done, and only 700 cases identical strings were
> compared for equality.

As Fred mentioned, this is probably due to the fact that
Unicode objects are not interned. Neither are typical
XML tag names; could make a difference... don't know. 

I believe that PyXML spends most of the time in Python 
calls and not so much in string compares (and that's what I'm 
trying to avoid in my XML parser approach ;-).

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/



From jeremy@zope.com  Tue Oct 30 20:07:07 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Tue, 30 Oct 2001 15:07:07 -0500 (EST)
Subject: [Python-Dev] Future of SSL
In-Reply-To: <20011029141844.A1967@lilith.hqd-internal>
References: <20011027014211.A11092@lilith.hqd-internal>
 <200110270226.WAA23767@cj20424-a.reston1.va.home.com>
 <20011026235527.A2919@trump.amber.org>
 <200110271556.LAA25312@cj20424-a.reston1.va.home.com>
 <20011027214357.B6993@trump.amber.org>
 <3BDB6742.E143BF8B@zope.com>
 <20011028184024.A1066@lilith.hqd-internal>
 <3BDD1519.31496FB@lemburg.com>
 <20011029133640.A1804@lilith.hqd-internal>
 <200110291306.IAA12932@cj20424-a.reston1.va.home.com>
 <20011029141844.A1967@lilith.hqd-internal>
Message-ID: <15327.2155.83315.318149@slothrop.digicool.com>

>>>>> "GH" =3D=3D haering python <Gerhard> writes:

  GH> On Mon, Oct 29, 2001 at 08:06:28AM -0500, Guido van Rossum
  GH> wrote:
  >> > [me, in response to a remark from Marc-Andr=E9] What would be
  >> > your suggestions? Would you prefer to go in the direction of my
  >> > original proposal - only providing a SSL API, but not the
  >> > implementation?
  >>
  >> Yes, that's how the current SSL support works -- you need to link
  >> in openssl.

  GH> So, as long as there are no actual cryptographic algorithms in
  GH> the Python source tree, but only hooks for OpenSSL, there's no
  GH> problem?

I don't think the export control regs work that way.
Crypto-with-a-hole (that is, an API without the actual crypto
implementation) is still considered crypto as far as I know.

The regulations are pretty simple these days for Open Source projects.
Just send a note to the BXA (Commerce Dept.) with the URL to the
source.

I wonder if it's wise to do this for Python, since it has an SSL
interface (albeit a clunky one).

Jeremy




From gerhard@bigfoot.de  Tue Oct 30 20:28:23 2001
From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=)
Date: Tue, 30 Oct 2001 21:28:23 +0100
Subject: [Python-Dev] Future of SSL
In-Reply-To: <15327.2155.83315.318149@slothrop.digicool.com>; from jeremy@zope.com on Tue, Oct 30, 2001 at 03:07:07PM -0500
References: <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> <15327.2155.83315.318149@slothrop.digicool.com>
Message-ID: <20011030212822.A800@lilith.hqd-internal>

On Tue, Oct 30, 2001 at 03:07:07PM -0500, Jeremy Hylton wrote:
>   GH> So, as long as there are no actual cryptographic algorithms in
>   GH> the Python source tree, but only hooks for OpenSSL, there's no
>   GH> problem?
> 
> I don't think the export control regs work that way.
> Crypto-with-a-hole (that is, an API without the actual crypto
> implementation) is still considered crypto as far as I know.
> 
> The regulations are pretty simple these days for Open Source projects.
> Just send a note to the BXA (Commerce Dept.) with the URL to the
> source.
> 
> I wonder if it's wise to do this for Python, since it has an SSL
> interface (albeit a clunky one).

Looks like it would (formally) be required, looking thru this text:

http://www.bxa.doc.gov/Encryption/EncryptionRuleOct2K.html

I better stop reading crypo law texts now >:-<

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))


From barry@zope.com  Tue Oct 30 21:11:50 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Tue, 30 Oct 2001 16:11:50 -0500
Subject: [Python-Dev] Future of SSL
References: <20011027014211.A11092@lilith.hqd-internal>
 <200110270226.WAA23767@cj20424-a.reston1.va.home.com>
 <20011026235527.A2919@trump.amber.org>
 <200110271556.LAA25312@cj20424-a.reston1.va.home.com>
 <20011027214357.B6993@trump.amber.org>
 <3BDB6742.E143BF8B@zope.com>
 <20011028184024.A1066@lilith.hqd-internal>
 <3BDD1519.31496FB@lemburg.com>
 <20011029133640.A1804@lilith.hqd-internal>
 <200110291306.IAA12932@cj20424-a.reston1.va.home.com>
 <20011029141844.A1967@lilith.hqd-internal>
 <15327.2155.83315.318149@slothrop.digicool.com>
Message-ID: <15327.6038.463871.492896@anthem.wooz.org>

>>>>> "JH" == Jeremy Hylton <jeremy@zope.com> writes:

    JH> I don't think the export control regs work that way.
    JH> Crypto-with-a-hole (that is, an API without the actual crypto
    JH> implementation) is still considered crypto as far as I know.

    JH> The regulations are pretty simple these days for Open Source
    JH> projects.  Just send a note to the BXA (Commerce Dept.) with
    JH> the URL to the source.

    JH> I wonder if it's wise to do this for Python, since it has an
    JH> SSL interface (albeit a clunky one).

Another task for the PSF, eh?

-Barry


From barry@zope.com  Tue Oct 30 21:13:48 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Tue, 30 Oct 2001 16:13:48 -0500
Subject: [Python-Dev] Slices and "==" optimization
References: <3BDE63F8.541E04B2@lemburg.com>
 <200110301350.IAA24747@cj20424-a.reston1.va.home.com>
Message-ID: <15327.6156.273422.675001@anthem.wooz.org>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> I agree that this would be nice to have.  I won't have time
    GvR> for it before 2.2 is out, but if you want to give it a crack,
    GvR> I might accept it.

It's also a new feature, so you'd have to override the feature freeze
to add it for 2.2.

-Barry


From hamish_lawson@yahoo.co.uk  Tue Oct 30 21:17:18 2001
From: hamish_lawson@yahoo.co.uk (=?iso-8859-1?q?Hamish=20Lawson?=)
Date: Tue, 30 Oct 2001 21:17:18 +0000 (GMT)
Subject: [Python-Dev] Misleading error message when multiple arguments for write()
Message-ID: <20011030211718.10036.qmail@web11001.mail.yahoo.com>

The reference to 'tuple' in the error message below seems misleading.
Too many arguments have been passed rather than a tuple.

Traceback (most recent call last):
  File "C:\Temp\demo.py", line 3, in ?
    sys.stdout.write("a", "b")
TypeError: argument must be string or read-only character buffer, not
tuple


Hamish Lawson


____________________________________________________________
Nokia Game is on again. 
Go to http://uk.yahoo.com/nokiagame/ and join the new
all media adventure before November 3rd.


From guido@python.org  Tue Oct 30 22:18:52 2001
From: guido@python.org (Guido van Rossum)
Date: Tue, 30 Oct 2001 17:18:52 -0500
Subject: [Python-Dev] Misleading error message when multiple arguments for write()
In-Reply-To: Your message of "Tue, 30 Oct 2001 21:17:18 GMT."
 <20011030211718.10036.qmail@web11001.mail.yahoo.com>
References: <20011030211718.10036.qmail@web11001.mail.yahoo.com>
Message-ID: <200110302218.f9UMIsf09646@odiug.zope.com>

> The reference to 'tuple' in the error message below seems misleading.
> Too many arguments have been passed rather than a tuple.
> 
> Traceback (most recent call last):
>   File "C:\Temp\demo.py", line 3, in ?
>     sys.stdout.write("a", "b")
> TypeError: argument must be string or read-only character buffer, not
> tuple

While you've reached the right audience, and this is indeed a
bug(let), python-dev is not the right medium for bug reports.
Please submit this to the SourceForge bug tracker, which acts as our
collective memory for bugs.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jack@oratrix.nl  Tue Oct 30 22:25:44 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Tue, 30 Oct 2001 23:25:44 +0100
Subject: [Python-Dev] slicing of structseq objects fails
In-Reply-To: Message by Russell E Owen <owen@astro.washington.edu> ,
 Tue, 30 Oct 2001 10:02:14 -0800 , <p05100302b8049897fbe0@[128.95.99.44]>
Message-ID: <20011030222549.4C74AFB04A@oratrix.oratrix.nl>

Python-devvers,
Russel Owen reported the following bug, and it turns out to be a
general bug, not a MacPython one: slicing the structseq-style tuples
returned by time.gmtime() returns a sequence with NULLs in it. Shortly
afterwards Python crashes on my SGI.

Any takers?

Recently, Russell E Owen <owen@astro.washington.edu> said:
> The following code fails (G4, MacOS 9.2.1) in Classic MacPython 2.2b1:
> 
> import time
> 
> currUTCTuple= time.gmtime(time.time())
> print "currUTCTuple=%r" % (currUTCTuple,)
> print "currUTCTuple[3:6]=%r" % (currUTCTuple[3:6],)
> fmtTime = "%s:%02i:%02i" % currUTCTuple[3:6]
> print "formatted time = ", fmtTime
> 
> It displays (<NULL>, <NULL>, <NULL>) for currUTCTuple[3:6],
> ...
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 


From fredrik@pythonware.com  Tue Oct 30 22:57:27 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 30 Oct 2001 23:57:27 +0100
Subject: [Python-Dev] slicing of structseq objects fails
References: <20011030222549.4C74AFB04A@oratrix.oratrix.nl>
Message-ID: <001b01c16196$7855eb90$ced241d5@hagrid>

jack wrote:
> > The following code fails (G4, MacOS 9.2.1) in Classic MacPython 2.2b1:
> > 
> > import time
> > 
> > currUTCTuple= time.gmtime(time.time())
> > print "currUTCTuple=%r" % (currUTCTuple,)
> > print "currUTCTuple[3:6]=%r" % (currUTCTuple[3:6],)
> > fmtTime = "%s:%02i:%02i" % currUTCTuple[3:6]
> > print "formatted time = ", fmtTime
> > 
> > It displays (<NULL>, <NULL>, <NULL>) for currUTCTuple[3:6],

also note:

>>> import time
>>> tm = time.gmtime()
>>> tm
(2001, 10, 30, 22, 54, 16, 1, 303, 0)
>>> tm.tm_year
2001
>>> tm[:6]
(2001, 10, 30, 22, 54, 16)
>>> tm[3:6]
(<nil>, <nil>, <nil>)

in structseq_slice, 

    for(i = low; i < high; ++i) {
        PyObject *v = obj->ob_item[i];
        Py_INCREF(v);
        PyTuple_SET_ITEM(np, i, v);
    }

should be:

    for(i = low; i < high; ++i) {
        PyObject *v = obj->ob_item[i];
        Py_INCREF(v);
        PyTuple_SET_ITEM(np, i-low, v);
    }

</F>



From tim@zope.com  Tue Oct 30 22:55:49 2001
From: tim@zope.com (Tim Peters)
Date: Tue, 30 Oct 2001 17:55:49 -0500
Subject: [Python-Dev] Slices and "==" optimization
In-Reply-To: <3BDF0879.D758A1A8@lemburg.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHMEKCCJAA.tim@zope.com>

[MvL]
>> OTOH, this same code caused a 10% slowdown if I converted the "==" into
>> "<>".

[MAL]
> This sounds like an alignment problem caused by the compiler.
> 10% speedups/slowdowns can easily be produced by randomly moving
> about a few cases in the ceval loop. All depends on the platform
> and compiler though.

I expect a more obvious cause is at work:  every one of Martin's "<>"
compares was slowed by the new batch of failing "is it an EQ and are these
strings and are they identical?" tests and branches.  Fast paths lose when
when they don't win -- they aren't neutral then.  Indeed, we could speed
your particular app by getting rid of the int-compare fast path already
there (but I expect that one is a *net* win across apps, so don't want to
get rid of it <wink>).

>> Running the PyXML test suite, I counted 120000 cases where
>> slow_compare was done, and only 700 cases identical strings were
>> compared for equality.

> As Fred mentioned, this is probably due to the fact that
> Unicode objects are not interned. Neither are typical
> XML tag names; could make a difference... don't know.

Whatever it's due to, it means the fast-path code cost extra cycles in the
119300 (of 120000) cases it didn't pay off.  If it costs F extra cycles when
it fails, and saves S cycles when it succeeds, the question for this test is
whether 119300*F < 700*S.  S has to about about 170X larger than F for a net
win in this case, else there's a net loss.



From tim@zope.com  Tue Oct 30 23:51:11 2001
From: tim@zope.com (Tim Peters)
Date: Tue, 30 Oct 2001 18:51:11 -0500
Subject: [Python-Dev] slicing of structseq objects fails
In-Reply-To: <20011030222549.4C74AFB04A@oratrix.oratrix.nl>
Message-ID: <BIEJKCLHCIOIHAGOKOLHIEKECJAA.tim@zope.com>

[Jack Jansen]
> Python-devvers,
> Russel Owen reported the following bug, and it turns out to be a
> general bug, not a MacPython one: slicing the structseq-style tuples
> returned by time.gmtime() returns a sequence with NULLs in it. Shortly
> afterwards Python crashes on my SGI.
>
> Any takers?

Sure, it was a shallow bug in the new structseq code and has been fixed in
current CVS.  Next time, how about a bug report on SourceForge instead?



From tim@zope.com  Tue Oct 30 23:54:07 2001
From: tim@zope.com (Tim Peters)
Date: Tue, 30 Oct 2001 18:54:07 -0500
Subject: [Python-Dev] slicing of structseq objects fails
In-Reply-To: <001b01c16196$7855eb90$ced241d5@hagrid>
Message-ID: <BIEJKCLHCIOIHAGOKOLHCEKFCJAA.tim@zope.com>

[/F]
> in structseq_slice,
>
>     for(i = low; i < high; ++i) {
>         PyObject *v = obj->ob_item[i];
>         Py_INCREF(v);
>         PyTuple_SET_ITEM(np, i, v);
>     }
>
> should be:
>
>     for(i = low; i < high; ++i) {
>         PyObject *v = obj->ob_item[i];
>         Py_INCREF(v);
>         PyTuple_SET_ITEM(np, i-low, v);
>     }

Bingo.  First Jack reports a bug here, then /F goes to the trouble of
copying in before-and-after code instead of just checking in the fix.  Does
everyone really hate SourceForge that much <wink>?



From fredrik@pythonware.com  Wed Oct 31 08:04:05 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 31 Oct 2001 09:04:05 +0100
Subject: [Python-Dev] slicing of structseq objects fails
References: <BIEJKCLHCIOIHAGOKOLHCEKFCJAA.tim@zope.com>
Message-ID: <00c401c161e2$ba34f320$ced241d5@hagrid>

tim wrote:

> Bingo.  First Jack reports a bug here, then /F goes to the trouble of
> copying in before-and-after code instead of just checking in the fix.

I didn't have enough time to rebuild and validate the patch, and
I didn't want to risk checking anything in that didn't compile ;-)

</F>



From mal@lemburg.com  Wed Oct 31 08:08:26 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 31 Oct 2001 09:08:26 +0100
Subject: [Python-Dev] Slices and "==" optimization
References: <BIEJKCLHCIOIHAGOKOLHMEKCCJAA.tim@zope.com>
Message-ID: <3BDFB17A.78584E7C@lemburg.com>

Tim Peters wrote:
> 
> >> Running the PyXML test suite, I counted 120000 cases where
> >> slow_compare was done, and only 700 cases identical strings were
> >> compared for equality.
> 
> > As Fred mentioned, this is probably due to the fact that
> > Unicode objects are not interned. Neither are typical
> > XML tag names; could make a difference... don't know.
> 
> Whatever it's due to, it means the fast-path code cost extra cycles in the
> 119300 (of 120000) cases it didn't pay off.  If it costs F extra cycles when
> it fails, and saves S cycles when it succeeds, the question for this test is
> whether 119300*F < 700*S.  S has to about about 170X larger than F for a net
> win in this case, else there's a net loss.

I'd be interested in the overall performance change in the PyXML
test suite run. If you look at the code that gets executed for
the checking the fast path criteria down in slow_compare, then
your equation doesn't look that bad anymore.

I'll run pybench on this and post the results here.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From jack@oratrix.nl  Wed Oct 31 10:42:47 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Wed, 31 Oct 2001 11:42:47 +0100
Subject: [Python-Dev] slicing of structseq objects fails
In-Reply-To: Message by "Tim Peters" <tim@zope.com> ,
 Tue, 30 Oct 2001 18:54:07 -0500 , <BIEJKCLHCIOIHAGOKOLHCEKFCJAA.tim@zope.com>
Message-ID: <20011031104248.F342E303181@snelboot.oratrix.nl>

> Bingo.  First Jack reports a bug here, [...]

Ok, Tim, sorry!

You're right, of course, that I should have submitted the bug, it was just 
that I was really busy with other things, and cc-ing python-dev was a lot 
quicker than first replying to the message to the poster and pythonmac-sig, 
finding out who owns structseq, filing the bug report and telling the poster 
about the id (I've found that pointing people to the sourceforge bug reporter 
to report the bug themselves means the bug is lost in >50% of the cases).

But saving my time shouldn't be an excuse for wasting everyone else's. Sorry,
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 




From jim@interet.com  Wed Oct 31 16:44:34 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Wed, 31 Oct 2001 11:44:34 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
 <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com>
Message-ID: <3BE02A72.372AE487@interet.com>

Guido van Rossum wrote:
> 
> > OK.  I propose that there is one name added to sys.path, and the
> > file name is "python%s%s.zip" % (sys.version[0], sys.version[2]).

> Sounds good to me.  Somebody please update the PEP.

The PEP is updated.

Did we decide to allow import of *.py?

JimA


From jack@oratrix.nl  Wed Oct 31 16:55:23 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Wed, 31 Oct 2001 17:55:23 +0100
Subject: [Python-Dev] Patch submitted for cross-platform newline support
Message-ID: <20011031165523.85D36303181@snelboot.oratrix.nl>

Hi folks,
I've just submitted a patch that allows Python to do cross-platform newline 
handling, #476814. This sort-of happened after the promised PEP didn't 
materialise, and there was discussion about this among myself, Guido and the 
pythonmac-sig, and at some point Guido said "Ok, code it!". The patch has been 
tested on SGI Irix and in MacPython only.

I'd like people to have a look at this. There's one thing I don't like, and 
that's the changes I had to make to the parser sourcefiles. They're needed, 
but I'm open to suggestions as to a better way to do this. And of course I'm 
open to any suggestions.

I'm also interested in discussing whether a patch like this is appropriate 
while we're in beta. On the one hand I would say it is, because the feature is 
disabled by default. On the other hand there are changes (albeit mainly 
cosmetic ones) in a large number of places. Another argument for allowing this 
even while in beta is that I really really want it for Mac OS X (but this 
might not be a very strong argument, I guess:-).

Here's the summary of the patch:

This patch enables Python to interpret all known newline conventions,
CR, LF or CRLF, on all platforms.

This support is enabled by configuring with --with-universal-newlines
(so by default it is off, and everything should behave as usual).

With universal newline support enabled two things happen:
- When importing or otherwise parsing .py files any newline convention
  is accepted.
- Python code can pass a new "t" mode parameter to open() which reads
  files with any newline convention. "t" cannot be combined with any
  other mode flags like "w" or "+", for obvious reasons.

File objects have a new attribute "newlines" which contains the type
of newlines encountered in the file (or None when no newline has been
seen, or "mixed" if there were various types of newlines).

Also included is a test script which tests both file I/O and parsing.

--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | ++++ see http://www.xs4all.nl/~tank/ ++++




From fredrik@pythonware.com  Wed Oct 31 18:19:26 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 31 Oct 2001 19:19:26 +0100
Subject: [Python-Dev] Patch submitted for cross-platform newline support
References: <20011031165523.85D36303181@snelboot.oratrix.nl>
Message-ID: <007601c16238$9e662730$ced241d5@hagrid>

Jack wrote:
> - Python code can pass a new "t" mode parameter to open() which reads
>   files with any newline convention. "t" cannot be combined with any
>   other mode flags like "w" or "+", for obvious reasons.

so "t" can be both a mode (like "r" and "w") and a mode modifier
(like "t" and "b")?  could be confusing.

</F>



From gward@python.net  Wed Oct 31 19:14:10 2001
From: gward@python.net (Greg Ward)
Date: Wed, 31 Oct 2001 14:14:10 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BE02A72.372AE487@interet.com>
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com>
Message-ID: <20011031141410.A12872@gerg.ca>

On 31 October 2001, James C. Ahlstrom said:
> Did we decide to allow import of *.py?

Here's an idea that would apply to importing *.py whether inside ZIP
files or not: if unable to write the .pyc, write a warning to stderr.
That'll be useful to people who inadvertently put .py files without .pyc
in a ZIP file, and to people who try to import .py files from a
directory they can't write to, etc.  Now that Python has a warning
framework, warnings like
  unable to write /usr/lib/python2.2/site-packages/foo.pyc: permission denied
or
  unable to write /usr/lib/python2.2/site-packages/foo.zip/foo.pyc:
    can't write to ZIP file

sound like useful warnings.

        Greg
-- 
Greg Ward - programmer-at-large                         gward@python.net
http://starship.python.net/~gward/
Save energy: be apathetic.


From skip@pobox.com (Skip Montanaro)  Wed Oct 31 19:15:34 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 31 Oct 2001 13:15:34 -0600
Subject: [Python-Dev] statcache.lstat?
Message-ID: <15328.19926.368998.631005@beluga.mojam.com>

Does this (untested) code make sense as a possible addition to statcache?
One obvious problem I see is that if stat(path) had been called before
lstat(path), you'd pull the wrong info out of the cache.  You could get
around this by maintaining separate stat and lstat caches or caching a
different key (e.g. (path, "l")) for the lstat variety.

If this is deemed worthwhile I'll make it work and submit a patch to SF.  I
wanted to check first to make sure I wasn't missing something obvious that
would keep it from ever working.

Skip

*** /tmp/skip/statcache.py.~1.11~fVASKs Wed Oct 31 13:10:22 2001
--- /tmp/skip/statcache.pyfVAfUy        Wed Oct 31 13:10:22 2001
***************
*** 6,12 ****
  import os as _os
  from stat import *
  
! __all__ = ["stat","reset","forget","forget_prefix","forget_dir",
             "forget_except_prefix","isdir"]
  
  # The cache.  Keys are pathnames, values are os.stat outcomes.
--- 6,12 ----
  import os as _os
  from stat import *
  
! __all__ = ["stat","lstat","reset","forget","forget_prefix","forget_dir",
             "forget_except_prefix","isdir"]
  
  # The cache.  Keys are pathnames, values are os.stat outcomes.
***************
*** 22,27 ****
--- 22,37 ----
      if ret is None:
          cache[path] = ret = _os.stat(path)
      return ret
+ 
+ if hasattr(_os, "lstat"):
+     def lstat(path):
+         """Lstat a file, possibly out of the cache."""
+         ret = cache.get(path, None)
+         if ret is None:
+             cache[path] = ret = _os.lstat(path)
+         return ret
+ else:
+     lstat = stat
  
  def reset():
      """Clear the cache."""



From fredrik@pythonware.com  Wed Oct 31 19:37:18 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 31 Oct 2001 20:37:18 +0100
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca>
Message-ID: <004701c16244$27bd9a80$ced241d5@hagrid>

Greg wrote:    
>
> > Did we decide to allow import of *.py?
> 
> Here's an idea that would apply to importing *.py whether inside ZIP
> files or not: if unable to write the .pyc, write a warning to stderr.
> That'll be useful to people who inadvertently put .py files without .pyc
> in a ZIP file

can anyone explain why I shouldn't be allowed to ship, say,
PIL's Python code in a version-independent ZIP file?

</F>



From jeremy@zope.com  Wed Oct 31 19:43:37 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Wed, 31 Oct 2001 14:43:37 -0500 (EST)
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <20011031141410.A12872@gerg.ca>
References: <3BD9B5EC.119319A5@interet.com>
 <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com>
 <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
 <3BDD91CC.9F0ECB07@interet.com>
 <200110291739.MAA15773@cj20424-a.reston1.va.home.com>
 <3BE02A72.372AE487@interet.com>
 <20011031141410.A12872@gerg.ca>
Message-ID: <15328.21609.836727.605337@slothrop.digicool.com>

>>>>> "GW" == Greg Ward <gward@python.net> writes:

  GW> Here's an idea that would apply to importing *.py whether inside
  GW> ZIP files or not: if unable to write the .pyc, write a warning
  GW> to stderr.

This is exactly what the interpreter does when run with -v.  If you
load a .py file and can't write the .pyc -- say you don't have write
permission in the directory -- it prints a warning to stderr.

+1 on ZIP import doing the same thing

Jeremy



From jeremy@zope.com  Wed Oct 31 19:48:29 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Wed, 31 Oct 2001 14:48:29 -0500 (EST)
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <004701c16244$27bd9a80$ced241d5@hagrid>
References: <3BD9B5EC.119319A5@interet.com>
 <200110262034.QAA20802@cj20424-a.reston1.va.home.com>
 <3BDD6588.7CFEA7D6@interet.com>
 <200110291458.JAA13956@cj20424-a.reston1.va.home.com>
 <3BDD91CC.9F0ECB07@interet.com>
 <200110291739.MAA15773@cj20424-a.reston1.va.home.com>
 <3BE02A72.372AE487@interet.com>
 <20011031141410.A12872@gerg.ca>
 <004701c16244$27bd9a80$ced241d5@hagrid>
Message-ID: <15328.21901.6102.308855@slothrop.digicool.com>

>>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:

  FL> can anyone explain why I shouldn't be allowed to ship, say,
  FL> PIL's Python code in a version-independent ZIP file?

Excellent point!

We ought to collect some requirements/goals for the ZIP import stuff.
The current draft of the PEP is a little thin on motivation.

Requirement: A ZIP archive be usable any version of Python that
supports ZIP-based imports.

It might also be nice to have a imputil- or ihook-based importer that
works with older versions of Python.  

Jeremy



From tim@zope.com  Wed Oct 31 20:14:33 2001
From: tim@zope.com (Tim Peters)
Date: Wed, 31 Oct 2001 15:14:33 -0500
Subject: [Python-Dev] Patch submitted for cross-platform newline support
In-Reply-To: <007601c16238$9e662730$ced241d5@hagrid>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEMLCJAA.tim@zope.com>

[Jack Jansen]
> - Python code can pass a new "t" mode parameter to open() which reads
>   files with any newline convention. "t" cannot be combined with any
>   other mode flags like "w" or "+", for obvious reasons.

[/F]
> so "t" can be both a mode (like "r" and "w") and a mode modifier
> (like "t" and "b")?  could be confusing.

Well, the "t" modifier isn't std; does anyone other than Microsoft support
it?  MS also gives meaning to non-std "c" and "n" modifiers.  Luckily,
Python doesn't define which modes or modifiers it accepts <wink>.



From gmcm@hypernet.com  Wed Oct 31 21:04:31 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Wed, 31 Oct 2001 16:04:31 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <15328.21901.6102.308855@slothrop.digicool.com>
References: <004701c16244$27bd9a80$ced241d5@hagrid>
Message-ID: <3BE0210F.12966.4496B695@localhost>

> >>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:
> 
>   FL> can anyone explain why I shouldn't be allowed to ship, say,
>   FL> PIL's Python code in a version-independent ZIP file?

Jeremy Hylton: 
> Excellent point!

Being the permissive sort, I'd say you should be allowed. But I 
think it would be foolish.
 
> We ought to collect some requirements/goals for the ZIP import
> stuff. The current draft of the PEP is a little thin on
> motivation.

Speed, ease of deployment, minimal disk space. MacPython 
does it, Jython does it, Installer's been doing it since '98 
(although it doesn't use zip format archives).
 
> Requirement: A ZIP archive be usable any version of Python that
> supports ZIP-based imports.

Hmm. You want .pyc's with every known magic number (there 
goes disk space)? Or just source and throw away the .pyc? 
Or maybe insert into the zip file? Either way, there goes 
speed.

Besides, as I noted, and Guido concurred, we need some way 
to handle extension modules "inside" packages, since they 
can't go inside the zip. On Windows, extensions are only 
usable by one version of Python.

This is silliness. I have clients with versions 1.52 to 2.1. I have 
a fair amount of code I reuse, and I keep it version-
independent. I have had to change something in it for every 
Python version to keep it version independent.

> It might also be nice to have a imputil- or ihook-based importer
> that works with older versions of Python.  

There's at least one imputil-based one that works with zipfile.

- Gordon


From skip@pobox.com (Skip Montanaro)  Wed Oct 31 21:11:12 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 31 Oct 2001 15:11:12 -0600
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <3BE0210F.12966.4496B695@localhost>
References: <004701c16244$27bd9a80$ced241d5@hagrid>
 <3BE0210F.12966.4496B695@localhost>
Message-ID: <15328.26864.350472.491216@beluga.mojam.com>

    >> Requirement: A ZIP archive be usable any version of Python that
    >> supports ZIP-based imports.

    Gordon> Hmm. You want .pyc's with every known magic number (there goes
    Gordon> disk space)? Or just source and throw away the .pyc?  Or maybe
    Gordon> insert into the zip file? Either way, there goes speed.

Can't zip files be treated conceptually as directories?  If I import a
py-based module from a zip archive I see no particular reason the
byte-compiled file can't be added to the archive (conceptually, written to
the directory), speeding up the import the next time.  Is this not possible?

Skip



From mal@lemburg.com  Wed Oct 31 21:12:02 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 31 Oct 2001 22:12:02 +0100
Subject: [Python-Dev] pickle faster in 2.2 ?
Message-ID: <3BE06922.5C43E09F@lemburg.com>

While hacking on an XML pickler, I found that pickle.py got nearly
twice as fast in 2.2 comparing to 2.1 and 2.0.

The code in pickle.py doesn't seem to have changed much. Anybody
know where that speedup came from ? Can somebody on another 
(non-Linux) system please verify this.

Thanks,
-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From guido@python.org  Wed Oct 31 21:20:23 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 31 Oct 2001 16:20:23 -0500
Subject: [Python-Dev] statcache.lstat?
In-Reply-To: Your message of "Wed, 31 Oct 2001 13:15:34 CST."
 <15328.19926.368998.631005@beluga.mojam.com>
References: <15328.19926.368998.631005@beluga.mojam.com>
Message-ID: <200110312120.QAA32561@cj20424-a.reston1.va.home.com>

> Does this (untested) code make sense as a possible addition to statcache?

For me, statcache was a failed experiment.  What's the use you have in
mind?  Why can't you use os.lstat() directly?

> One obvious problem I see is that if stat(path) had been called before
> lstat(path), you'd pull the wrong info out of the cache.  You could get
> around this by maintaining separate stat and lstat caches or caching a
> different key (e.g. (path, "l")) for the lstat variety.

Definitely use two caches or a separate key.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Wed Oct 31 21:22:46 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 31 Oct 2001 22:22:46 +0100
Subject: [Python-Dev] pickle faster in 2.2 ?
References: <3BE06922.5C43E09F@lemburg.com>
Message-ID: <3BE06BA6.59EE23E@lemburg.com>

"M.-A. Lemburg" wrote:
> 
> While hacking on an XML pickler, I found that pickle.py got nearly
> twice as fast in 2.2 comparing to 2.1 and 2.0.
> 
> The code in pickle.py doesn't seem to have changed much. Anybody
> know where that speedup came from ? Can somebody on another
> (non-Linux) system please verify this.

Nevermind; it was the change in pickle.py to use cStringIO instead 
of StringIO that caused the difference.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From guido@python.org  Wed Oct 31 21:31:53 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 31 Oct 2001 16:31:53 -0500
Subject: [Python-Dev] pickle faster in 2.2 ?
In-Reply-To: Your message of "Wed, 31 Oct 2001 22:12:02 +0100."
 <3BE06922.5C43E09F@lemburg.com>
References: <3BE06922.5C43E09F@lemburg.com>
Message-ID: <200110312131.QAA32734@cj20424-a.reston1.va.home.com>

> While hacking on an XML pickler, I found that pickle.py got nearly
> twice as fast in 2.2 comparing to 2.1 and 2.0.

What benchmark?

> The code in pickle.py doesn't seem to have changed much. Anybody
> know where that speedup came from ? Can somebody on another 
> (non-Linux) system please verify this.

I believe DOM nodes are now new-style classes, for better or for worse
(it might create problems when combining with classic mixins).  Could
that explain it?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gmcm@hypernet.com  Wed Oct 31 22:31:02 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Wed, 31 Oct 2001 17:31:02 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <15328.26864.350472.491216@beluga.mojam.com>
References: <3BE0210F.12966.4496B695@localhost>
Message-ID: <3BE03556.11078.44E5ECD7@localhost>

Jeremy: 
>     >> Requirement: A ZIP archive be usable any version of Python
>     that >> supports ZIP-based imports.
> 
>     Gordon> Hmm. You want .pyc's with every known magic number
>     (there goes Gordon> disk space)? Or just source and throw
>     away the .pyc?  Or maybe Gordon> insert into the zip file?
>     Either way, there goes speed.
Skip:
> Can't zip files be treated conceptually as directories?  If I
> import a py-based module from a zip archive I see no particular
> reason the byte-compiled file can't be added to the archive
> (conceptually, written to the directory), speeding up the import
> the next time.  Is this not possible?

Zip files are normally unpacked to the filesystem and rebuilt 
when modified. If this feature is desired, I think MetaKit would 
be a much better file format (I have used MetaKit as a pyc 
archive so I could replace individual modules).

But unless you want to big-boy SQL servers as "archives", 
you're going to have trouble with multi-user.

To combine this with Jeremy's requirement, we can't put the 
.pyc's alongside the py's anyway, we need some 
(disappearing) node in the hierarchy to hold the byte code 
version (magic number). A bit more trickery for import.c.

We can meet a real need as long as we don't put tail fins and 
heavy chrome all over a Yugo (zip files). They're easy to 
produce & ship. I don't think we should disallow .py files, but I 
think 99% of their use will be version specific .pyc's only (and 
many of those to users who think "traceback, innermost last" 
is Aramaic). It's nothing compared to the effort of dealing with 
the binaries across versions (or even the std lib across 
versions, sometimes).

OTOH, I'm worried that  the extension-modules-inside-
packages requirement is going to break import.c's back. There 
are many undocumented features of the implementation of 
import that are relied on (such as xml.__init__'s trick, or 
extentions modules inside packages). 

Heck, we might even have to clarify the rules.

doom-and-gloom-and-goblins-ly y'rs

- Gordon


From skip@pobox.com (Skip Montanaro)  Wed Oct 31 22:34:17 2001
From: skip@pobox.com (Skip Montanaro) (Skip Montanaro)
Date: Wed, 31 Oct 2001 16:34:17 -0600
Subject: [Python-Dev] statcache.lstat?
In-Reply-To: <200110312120.QAA32561@cj20424-a.reston1.va.home.com>
References: <15328.19926.368998.631005@beluga.mojam.com>
 <200110312120.QAA32561@cj20424-a.reston1.va.home.com>
Message-ID: <15328.31849.14360.58802@beluga.mojam.com>

    >> Does this (untested) code make sense as a possible addition to
    >> statcache? 

    Guido> For me, statcache was a failed experiment.  What's the use you
    Guido> have in mind?  Why can't you use os.lstat() directly?

I can use os.lstat (or os.stat) directly.  I'm working on a file selector
widget written in Python (and PyGtk).  As people traverse the directory
tree, it seems to make sense to cache the stat results.

If statcache is indeed a failed experiment, perhaps it should be
deprecated. 

Skip


From guido@python.org  Wed Oct 31 22:38:03 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 31 Oct 2001 17:38:03 -0500
Subject: [Python-Dev] statcache.lstat?
In-Reply-To: Your message of "Wed, 31 Oct 2001 16:34:17 CST."
 <15328.31849.14360.58802@beluga.mojam.com>
References: <15328.19926.368998.631005@beluga.mojam.com> <200110312120.QAA32561@cj20424-a.reston1.va.home.com>
 <15328.31849.14360.58802@beluga.mojam.com>
Message-ID: <200110312238.RAA01093@cj20424-a.reston1.va.home.com>

>     Guido> For me, statcache was a failed experiment.  What's the use you
>     Guido> have in mind?  Why can't you use os.lstat() directly?
> 
> I can use os.lstat (or os.stat) directly.  I'm working on a file selector
> widget written in Python (and PyGtk).  As people traverse the directory
> tree, it seems to make sense to cache the stat results.

But why bother?  And why not let them see changes in the filesystem?

> If statcache is indeed a failed experiment, perhaps it should be
> deprecated. 

Fine with me.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Wed Oct 31 22:41:44 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 31 Oct 2001 17:41:44 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Wed, 31 Oct 2001 17:31:02 EST."
 <3BE03556.11078.44E5ECD7@localhost>
References: <3BE0210F.12966.4496B695@localhost>
 <3BE03556.11078.44E5ECD7@localhost>
Message-ID: <200110312241.RAA01122@cj20424-a.reston1.va.home.com>

- Writing stuff back to .zip files is totally the wrong approach.

- I don't care about having .so files inside packages *in zipfiles*.

- I'm not sure I care about having .so files inside packages on the
  filesystem; they are useful in Zope, but for very hackish reasons.

- If the zip file has the .py file but no .pyc or the wrong .pyc, tant
  pis.  Let it be slower.  (But if it has the .pyo, use that.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Wed Oct 31 23:07:35 2001
From: just@letterror.com (Just van Rossum)
Date: Thu,  1 Nov 2001 00:07:35 +0100
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: <200110312241.RAA01122@cj20424-a.reston1.va.home.com>
Message-ID: <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23>

Guido van Rossum wrote:

> - I'm not sure I care about having .so files inside packages on the
>   filesystem; they are useful in Zope, but for very hackish reasons.

Why? If I write a package which is mostly in Python, it feels very natural to
put the C extensions also in the package.

Just


From jack@oratrix.nl  Wed Oct 31 23:14:58 2001
From: jack@oratrix.nl (Jack Jansen)
Date: Thu, 01 Nov 2001 00:14:58 +0100
Subject: [Python-Dev] Patch submitted for cross-platform newline support
In-Reply-To: Message by "Tim Peters" <tim@zope.com> ,
 Wed, 31 Oct 2001 15:14:33 -0500 , <BIEJKCLHCIOIHAGOKOLHEEMLCJAA.tim@zope.com>
Message-ID: <20011031231503.092DB151ADB@oratrix.oratrix.nl>

Recently, "Tim Peters" <tim@zope.com> said:
> Well, the "t" modifier isn't std; does anyone other than Microsoft support
> it?  MS also gives meaning to non-std "c" and "n" modifiers.  Luckily,
> Python doesn't define which modes or modifiers it accepts <wink>.

Oops, big oops! I picked the "t" because it was intended _not_ to be
supported on any platform: my code eats it, and turns on the universal
newline feature on the file object. What does "t" do on Windows? What
would be a better choice?
--
Jack Jansen             | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.cwi.nl/~jack        | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm 


From guido@python.org  Wed Oct 31 23:38:55 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 31 Oct 2001 18:38:55 -0500
Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives
In-Reply-To: Your message of "Thu, 01 Nov 2001 00:07:35 +0100."
 <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23>
References: <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23>
Message-ID: <200110312338.SAA01368@cj20424-a.reston1.va.home.com>

> Guido van Rossum wrote:
> 
> > - I'm not sure I care about having .so files inside packages on the
> >   filesystem; they are useful in Zope, but for very hackish reasons.
> 
> Why? If I write a package which is mostly in Python, it feels very
> natural to put the C extensions also in the package.
> 
> Just

Yes, it does, and as long as it works, I have no problem with that.
Distutils supports this too, AFAIK.

But if there are mechanical problems with making it work (zip files
are a good example of that) I don't see why we should torture
ourselves to get it to work when simpler solutions exist (such as
putting the extension at the top level under a private name with the
package name as a prefix).

BTW, the hackish reasons I referred to are this: Zope often wants to
*replace* existing extensions with its own versions, and places
e.g. its own cPickle.so insize a package to force the import even if
cPickle is built-in statically.  But I'm not sure that that works if
there's a toplevel cPickle.so which has already been imported; it may
work on some systems but fail on others, depending on the shared
library architecture (often one of the most hackish parts of
user-space OS support).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim@zope.com  Wed Oct 31 23:44:44 2001
From: tim@zope.com (Tim Peters)
Date: Wed, 31 Oct 2001 18:44:44 -0500
Subject: [Python-Dev] Patch submitted for cross-platform newline support
In-Reply-To: <20011031231503.092DB151ADB@oratrix.oratrix.nl>
Message-ID: <BIEJKCLHCIOIHAGOKOLHKENJCJAA.tim@zope.com>

[Jack Jansen]
> Oops, big oops! I picked the "t" because it was intended _not_ to be
> supported on any platform: my code eats it, and turns on the universal
> newline feature on the file object. What does "t" do on Windows?

MS treats "t" as an explicit way of saying "not b" <wink>, i.e. text mode.
The MS libraries can be configured to use "b" as the default, and then you
need "t" to force text mode (when needed).

> What would be a better choice?

I don't know.  MS adds t, c and n as extensions to C's fopen() gimmicks, and
there's really no way to guess what the union of all other platforms may do.
Python (before your patch) passes on *whatever* mode string the user
supplies, without even peeking at it.  That's a mixed blessing:  I've been
very happy to be able to pass "c" on MS (which triggers a "commit" mode that
vastly increases the chances file output winds up on disk after a crash);
OTOH, across platforms the kinds of error msgs we get for malformed mode
strings aren't very good:

>>> f = file('oops', 'br')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
IOError: [Errno 0] Error: 'oops'
>>>

Radical idea:  don't do anything to turn on "universal newlines" -- say it's
just what "text mode" means in Python.  Then you only have to worry about
picking a letter to turn it off <wink>.



From guido@python.org  Wed Oct 31 23:48:20 2001
From: guido@python.org (Guido van Rossum)
Date: Wed, 31 Oct 2001 18:48:20 -0500
Subject: [Python-Dev] Patch submitted for cross-platform newline support
In-Reply-To: Your message of "Wed, 31 Oct 2001 18:44:44 EST."
 <BIEJKCLHCIOIHAGOKOLHKENJCJAA.tim@zope.com>
References: <BIEJKCLHCIOIHAGOKOLHKENJCJAA.tim@zope.com>
Message-ID: <200110312348.SAA01457@cj20424-a.reston1.va.home.com>

> Radical idea:  don't do anything to turn on "universal newlines" -- say it's
> just what "text mode" means in Python.  Then you only have to worry about
> picking a letter to turn it off <wink>.

That would be my choice.  KISS.

--Guido van Rossum (home page: http://www.python.org/~guido/)