From skip@manatee.mojam.com Sun Dec 1 13:00:16 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 1 Dec 2002 07:00:16 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200212011300.gB1D0GMS011844@manatee.mojam.com> Bug/Patch Summary ----------------- 313 open / 3091 total bugs (+4) 96 open / 1810 total patches (+2) New Bugs -------- __all__ not in RefMan index (2002-11-24) http://python.org/sf/643227 typo in Object/abstract.c : ternary_op (2002-11-24) http://python.org/sf/643260 Pythonwin : close() on a stdout (2002-11-25) http://python.org/sf/643571 New class special method lookup change (2002-11-25) http://python.org/sf/643841 ic module "path too long" error (2002-11-26) http://python.org/sf/644243 file URLs mis-handled by webbrowser (2002-11-26) http://python.org/sf/644246 Poor error message for augmented assign (2002-11-26) http://python.org/sf/644345 bdist_rpm fails when installin man pages (2002-11-27) http://python.org/sf/644744 for lin in file: file.tell() tells wrong (2002-11-29) http://python.org/sf/645594 SequenceMatcher finds suboptimal sequenc (2002-11-29) http://python.org/sf/645629 .extend() doc, .__doc__ too restrictive (2002-11-29) http://python.org/sf/645777 old UnicodeData.txt (2002-12-01) http://python.org/sf/646408 New Patches ----------- Allow keyword args in dict.update() (2002-11-25) http://python.org/sf/643565 refactoring and documenting ModuleFinder (2002-11-25) http://python.org/sf/643711 Set Next Statement for Python debuggers (2002-11-25) http://python.org/sf/643835 Fix for os.path.expanduser for Win2000 (2002-11-25) http://python.org/sf/643943 Build _tkinter in setup.jaguar.py (2002-11-27) http://python.org/sf/644920 Allow device name in linuxaudiodev.open (2002-11-27) http://python.org/sf/644977 Don't FPE on FreeBSD... (2002-11-28) http://python.org/sf/645382 Give some clue why modules fail... (2002-11-28) http://python.org/sf/645383 fix buffer overrun in pmerge (2002-11-28) http://python.org/sf/645404 Import from Zip Archive (2002-11-29) http://python.org/sf/645650 better timer resolution for profile.py (2002-11-29) http://python.org/sf/645894 Closed Bugs ----------- bdist builds bogus .zips (2001-03-22) http://python.org/sf/410541 webbrowser.py: selection of browser (2001-12-20) http://python.org/sf/495695 PyMapping_Keys unexported in dll (2002-08-02) http://python.org/sf/590207 Can't assign to __name__ or __bases__ of new class (2002-08-09) http://python.org/sf/593154 __getstate__() documentation is vague (2002-11-13) http://python.org/sf/637941 better error reporting for MRO conflict (2002-11-22) http://python.org/sf/642489 Webbrowser (and ic) on MacOSX failure (2002-11-23) http://python.org/sf/642720 Closed Patches -------------- Import from Zip archive (2001-12-12) http://python.org/sf/492105 Return objects in Tkinter (2002-02-16) http://python.org/sf/518625 _getdefaultlocale for OS X (2002-11-03) http://python.org/sf/632973 Cleanup of test_strptime.py (2002-11-04) http://python.org/sf/633633 SimpleSets (2002-11-13) http://python.org/sf/638095 _strptime fixes for None locale and tz (2002-11-15) http://python.org/sf/639112 reST version of Lib/test/README (2002-11-20) http://python.org/sf/641170 time and timetz for the datetime module (2002-11-21) http://python.org/sf/641958 Expose PyImport_FrozenModules in imp (2002-11-22) http://python.org/sf/642578 From skip@pobox.com Sun Dec 1 13:37:03 2002 From: skip@pobox.com (Skip Montanaro) Date: Sun, 1 Dec 2002 07:37:03 -0600 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: References: <15845.608.895210.155241@montanaro.dyndns.org> Message-ID: <15850.4223.434280.797090@montanaro.dyndns.org> Regarding how to kill a process on Windows which was started with os.spawn*, on the Spambayes list I asked: >> How would I kill that process on Windows, Mark Hammond replied: This worked for me just then: >>> os.spawnl(os.P_NOWAIT, "f:\\windows\\notepad.exe") 548 >>> import win32api >>> win32api.TerminateProcess(548,0) Tim also replied: Tim> For 2.3, I implemented something or other for the Python core; Tim> probably killpid(); Shouldn't one of these solutions (or some variant) appear under the veneer of os.kill()? I realize the semantics might not be exactly the same (perhaps Windows solutions don't support different signals or a different set of signals), but it seems to me that if an attempt is made to provide the os.spawn* functions across platforms we should make a similar attempt to provide a way to kill spawned processes as well. Corollary question: Is there some reason the win32api stuff isn't in the core? Skip From tim_one@email.msn.com Sun Dec 1 15:43:41 2002 From: tim_one@email.msn.com (Tim Peters) Date: Sun, 1 Dec 2002 10:43:41 -0500 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: <15850.4223.434280.797090@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Regarding how to kill a process on Windows which was started with > os.spawn*, on the Spambayes list I asked: > > >> How would I kill that process on Windows, > > Mark Hammond replied: > > This worked for me just then: > > >>> os.spawnl(os.P_NOWAIT, "f:\\windows\\notepad.exe") > 548 > >>> import win32api > >>> win32api.TerminateProcess(548,0) > > Tim also replied: > > Tim> For 2.3, I implemented something or other for the Python core; > Tim> probably killpid(); I didn't remember correctly -- I implemented waitpid() for Windows. Processes on Windows aren't meant to be externally killed, and the docs have dire warnings about use of TerminateProcess. Not that people don't use it, but it's dangerous. > Shouldn't one of these solutions (or some variant) appear under the > veneer of os.kill()? Signals make little sense on Windows. > I realize the semantics might not be exactly the same > (perhaps Windows solutions don't support different signals Mone -- the second argument to TerminateProcess isn't a signal number, it's the exit code to be returned by the terminated process. > or a different set of signals), Nope, none. > but it seems to me that if an attempt is made to provide > the os.spawn* functions across platforms we should make a similar > attempt to provide a way to kill spawned processes as well. Be my guest . > Corollary question: Is there some reason the win32api stuff isn't in > the core? I *expect* Guido would like that, provided the code fit the Python style guide and the docs were bashed into LaTeX. That's a massive amount of no-fun work, and I don't expect Mark to volunteer for either. From gward@python.net Sun Dec 1 17:07:13 2002 From: gward@python.net (Greg Ward) Date: Sun, 1 Dec 2002 12:07:13 -0500 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: References: <15850.4223.434280.797090@montanaro.dyndns.org> Message-ID: <20021201170713.GA26134@cthulhu.gerg.ca> On 01 December 2002, Tim Peters said: > > Shouldn't one of these solutions (or some variant) appear under the > > veneer of os.kill()? > > Signals make little sense on Windows. And that's different from Unix somehow? >smirk< Greg From mal@lemburg.com Sun Dec 1 20:58:16 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 01 Dec 2002 21:58:16 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211 References: Message-ID: <3DEA77E8.9030007@lemburg.com> Jack Jansen wrote: > > On vrijdag, nov 29, 2002, at 21:47 Europe/Amsterdam, > jvr@users.sourceforge.net wrote: > >> Update of /cvsroot/python/python/dist/src/Python >> In directory sc8-pr-cvs1:/tmp/cvs-serv20813/Python >> >> Modified Files: >> import.c >> Log Message: >> Slightly improved version of patch #642578: "Expose >> PyImport_FrozenModules >> in imp". This adds two functions to the imp module: get_frozenmodules() >> and set_frozenmodules(). > > > Something that's been bothering me about frozen modules in the classical > sense (i.e. those that are stored in C static data structures) is that > the memory used by them is gone without any chance at recovery. For big > frozen Python programs that are to be run on small machines this is a > waste of precious memory. > > With modules "frozen" with set_frozenmodules you could conceivably free > the data again after it has been imported (similar to what MacPython-OS9 > does with modules "frozen" in "PYC " resources). > > Would that be worth the added complexity? Allocating the data on the heap would prevent sharing the static data between processes (provided you have multiple of the running), so if you want to add that feature, please make it an option and not the default. mxCGIPython heavily relies on the fact that frozen modules are shared between processes. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From just@letterror.com Sun Dec 1 21:39:29 2002 From: just@letterror.com (Just van Rossum) Date: Sun, 1 Dec 2002 22:39:29 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211 In-Reply-To: <3DEA77E8.9030007@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Allocating the data on the heap would prevent sharing the > static data between processes (provided you have multiple > of the running), so if you want to add that feature, please > make it an option and not the default. Don't worry: I think extending the freeze mechanism even further is a waste of time; I think Jack's concerns could also be addressed with an import hook. Also, about the checkin that that triggered Jack's remark: I'm very likely to back out the imp.[sg]et_frozenmodules patch as what I wanted to achieve would be a much better candidate for the zip import mechanism, which is sadly still pending. I'll have a closer look at that and see whether I can help getting this in before 2.3.a1. Just From mhammond@skippinet.com.au Sun Dec 1 22:31:54 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Mon, 2 Dec 2002 09:31:54 +1100 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: <15850.4223.434280.797090@montanaro.dyndns.org> Message-ID: > Mark Hammond replied: > > This worked for me just then: > > >>> os.spawnl(os.P_NOWAIT, "f:\\windows\\notepad.exe") > 548 > >>> import win32api > >>> win32api.TerminateProcess(548,0) > > Tim also replied: > > Tim> For 2.3, I implemented something or other for the Python core; > Tim> probably killpid(); > > Shouldn't one of these solutions (or some variant) appear under the veneer > of os.kill()? I realize the semantics might not be exactly the same The key difference is that kill() wants a PID, while TerminateProcess wants a handle. Now, we *could* add a kill() (assuming it is not already there) but that would not solve your specific problem, as you have a handle and you would then need a platform specific way to convert it to a PID. It seems too hard to win. > (perhaps Windows solutions don't support different signals or a different > set of signals), but it seems to me that if an attempt is made to provide > the os.spawn* functions across platforms we should make a similar > attempt to > provide a way to kill spawned processes as well. Yes, a higher-level execution API would be worthwhile, especially if that could be expressed using posix semantics as a fallback. I am thinking of what threading is to thread, this new API could be to the various spawn/exec/popen functions. > Corollary question: Is there some reason the win32api stuff isn't in the > core? Even if we ignore Tim's requirements ('cos I know I will ) there is still alot of work, and a *huge* amount of code. Do we want win32api, all of the "win32*" modules in win32all, or all of win32all. The win32api modules seems very useful, and it is for trivial things - but once you start doing real things, the other win32 modules start becoming useful, and so it goes. On the other hand, if these extensions were in the core, I wouldn't personally be worrying about those damn hex literal warnings . Mark. From skip@pobox.com Sun Dec 1 22:45:38 2002 From: skip@pobox.com (Skip Montanaro) Date: Sun, 1 Dec 2002 16:45:38 -0600 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: References: <15850.4223.434280.797090@montanaro.dyndns.org> Message-ID: <15850.37138.572612.258716@montanaro.dyndns.org> >> Shouldn't one of these solutions (or some variant) appear under the >> veneer of os.kill()? I realize the semantics might not be exactly >> the same Mark> The key difference is that kill() wants a PID, while Mark> TerminateProcess wants a handle. Posix.kill() wants a pid. Can't os.kill() want anything it wants? ;-) In particular, can't it test its arguments to see if it got a handle or a pid then "Do The Right Thing" (tm Spike Lee I think)? I guess I'm lobbying for os.kill() or something like it (os.retire()?) to be the terminator for os.spawn*, regardless what those functions return. It seems weird to be able to create processes on various platforms but not be able to kill them as well. For the python-dev types not on the spambayes list, I do have an application in mind. In my environment I need to tunnel SB's pop3proxy app through an ssh connection. Starting and stopping it works okay on my MacOSX system, and I presume it will under other Unix-based systems. I began thinking about whether this would port to other platforms and noticed that os.kill() is only supported under Unix. It may turn out that my desire is moot if ssh can't be run under Windows anyway. Skip From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_002D_01C298DF.B68FC6C0 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable I think Python should have a more universal import mechanism , which = could be done by adding a new C_CUSTOM type to the definitions for dynamic loading = (importdl.h) , a new struct _customtab to import.h and a new function to Python API = (import.c). Then any archive,database,internet import architecture could be = implemented=20 in C extensions without messing with Python core. Wiktor Sadowski www.wiktorsadowski.com IMPLEMENTATION (tested on win_98) import.h extern DL_IMPORT(int) PyImport_AppendCustomtab(int = (*get_name)(char*),void (*initfunc)(char*)); struct _customtab { int (*get_name)(char*); void (*initfunc)(char*); struct _customtab *next; }; importdl.h enum filetype { SEARCH_ERROR, PY_SOURCE, PY_COMPILED, C_EXTENSION, PY_RESOURCE, /* Mac only */ PKG_DIRECTORY, C_BUILTIN, PY_FROZEN, C_CUSTOM, PY_CODERESOURCE /* Mac only */ }; import.c struct _customtab *PyImport_Customtab; int=20 PyImport_AppendCustomtab(int (*get_name)(char*),void = (*initfunc)(char*)) { struct _customtab *pycustomtab; struct _customtab *newtab=3DPyMem_NEW(struct _customtab,1); newtab->get_name=3Dget_name; newtab->initfunc=3Dinitfunc; newtab->next=3DNULL; pycustomtab=3DPyImport_Customtab; if (pycustomtab) { while(pycustomtab->next) pycustomtab=3Dpycustomtab->next;=20 pycustomtab->next=3Dnewtab; } else PyImport_Customtab=3Dnewtab; return 1;=20 } static struct filedescr * find_module(char *realname, PyObject *path, char *buf, size_t buflen, FILE **p_fp) .........................................................................= .......................... static struct filedescr fd_custom =3D {"", "", C_CUSTOM}; .........................................................................= .......................... before find_frozen !: if (is_custom(name)) { strcpy(buf, name); return &fd_custom; } if (find_frozen(name) !=3D NULL) { strcpy(buf, name); return &fd_frozen; .........................................................................= .......................... and again: if (path =3D=3D NULL) { if (is_custom(name)) { strcpy(buf, name); return &fd_custom; } LOAD_MODULE static PyObject * load_module(char *name, FILE *fp, char *buf, int type) .........................................................................= .......................... switch (type) { ............................................. case C_CUSTOM: if (buf !=3D NULL && buf[0] !=3D '\0') name =3D buf; =20 err =3D init_custom(name); =20 if (err < 0) return NULL; if (err =3D=3D 0) { PyErr_Format(PyExc_ImportError, "Purported %s module %.200s not found", type =3D=3D C_BUILTIN ? "builtin" : type =3D=3D C_CUSTOM ? "custom":"frozen", name); return NULL; } modules =3D PyImport_GetModuleDict(); m =3D PyDict_GetItemString(modules, name); if (m =3D=3D NULL) { PyErr_Format( PyExc_ImportError, "%s module %.200s not properly initialized", type =3D=3D C_BUILTIN ? "builtin" : type =3D=3D C_CUSTOM ? "custom":"frozen", name); return NULL; } Py_INCREF(m); break; IS_CUSTOM static int is_custom(char *name) { struct _customtab *p; if (!PyImport_Customtab) return 0; p=3DPyImport_Customtab; while(p){ if (p->get_name) if ((*p->get_name)(name)) return 1; p=3Dp->next;=20 } return 0; } INIT_CUSTOM static int init_custom(char *name) { struct _customtab *p =3D PyImport_Customtab; if (_PyImport_FindExtension(name, name) !=3D NULL) return 1; while(p) { if (p->get_name){ if ((*p->get_name)(name)) { if (p->initfunc =3D=3D NULL) { PyErr_Format(PyExc_ImportError, "Cannot re-init internal module %.200s", name); return -1; } if (Py_VerboseFlag) PySys_WriteStderr("import %s # builtin\n", name); (*p->initfunc)(name); if (PyErr_Occurred()) return -1; if (_PyImport_FixupExtension(name, name) =3D=3D NULL) return -1; return 1; } } p=3Dp->next; } return 0; } ------=_NextPart_000_002D_01C298DF.B68FC6C0 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
I think Python should have a more universal import mechanism , = which could=20 be done
by adding a new C_CUSTOM type  to the definitions for dynamic = loading=20 (importdl.h) ,
a new struct _customtab to import.h and a new function = to Python API=20 (import.c).
Then any archive,database,internet import architecture = could be=20 implemented
in C extensions without messing with Python core.
 
Wiktor Sadowski
 
 
IMPLEMENTATION
(tested on win_98)
 
import.h
extern DL_IMPORT(int) PyImport_AppendCustomtab(int =20 (*get_name)(char*),void (*initfunc)(char*));
 
struct _customtab {
 int =20 (*get_name)(char*);
    void=20 (*initfunc)(char*);
 struct _customtab *next;
};
 
 
 
importdl.h
enum filetype=20 {
 SEARCH_ERROR,
 PY_SOURCE,
 PY_COMPILED,
&nb= sp;C_EXTENSION,
 PY_RESOURCE,=20 /* Mac only=20 */
 PKG_DIRECTORY,
 C_BUILTIN,
 PY_FROZEN,
&nb= sp;C_CUSTOM,
 PY_CODERESOURCE=20 /* Mac only */
};
 
import.c
struct _customtab *PyImport_Customtab;
 
 
int
PyImport_AppendCustomtab(int  (*get_name)(char*),void=20 (*initfunc)(char*))
{
  struct _customtab = *pycustomtab;
 =20 struct _customtab *newtab=3DPyMem_NEW(struct _customtab,1);
 =20 newtab->get_name=3Dget_name;
  = newtab->initfunc=3Dinitfunc;
 =20 newtab->next=3DNULL;
 
  pycustomtab=3DPyImport_Customtab;
  if (pycustomtab)
  {
 =20 while(pycustomtab->next)
    =20 pycustomtab=3Dpycustomtab->next;
 =20 pycustomtab->next=3Dnewtab;
  }
 =20 else
     PyImport_Customtab=3Dnewtab;
  return 1;
 }
 
 
static struct filedescr *
find_module(char *realname, PyObject = *path,=20 char *buf, size_t buflen,
     FILE **p_fp)
....................................................................= ...............................
static struct filedescr fd_custom =3D {"", "", C_CUSTOM};
....................................................................= ...............................
before find_frozen !:
 
if (is_custom(name)) {
   strcpy(buf,=20 name);
   return = &fd_custom;
  }
 
  if (find_frozen(name) !=3D NULL)=20 {
   strcpy(buf, name);
   return=20 &fd_frozen;
....................................................................= ...............................
and again:
if (path =3D=3D NULL) {
  if (is_custom(name)) {
   strcpy(buf,=20 name);
   return = &fd_custom;
  }
 
LOAD_MODULE
static PyObject *
load_module(char *name, FILE *fp, char *buf, = int=20 type)
....................................................................= ...............................
switch (type) {
.............................................
case C_CUSTOM:
 
  if (buf !=3D NULL && buf[0] !=3D=20 '\0')
   name =3D = buf;
  
   err=20 =3D init_custom(name);
  
  if (err <=20 0)
   return NULL;
  if (err =3D=3D 0)=20 {
   PyErr_Format(PyExc_ImportError,
  &nb= sp;     =20 "Purported %s module %.200s not=20 found",
         type =3D=3D = C_BUILTIN=20 ?
      "builtin" : type =3D=3D = C_CUSTOM ?=20 "custom":"frozen",
        =20 name);
   return = NULL;
  }
  modules=20 =3D PyImport_GetModuleDict();
  m =3D = PyDict_GetItemString(modules,=20 name);
  if (m =3D=3D NULL)=20 {
   PyErr_Format(
    PyExc_Imp= ortError,
    "%s=20 module %.200s not properly initialized",
    type = =3D=3D=20 C_BUILTIN ?
      "builtin" : type = =3D=3D C_CUSTOM=20 ?=20 "custom":"frozen",
    name);
   = ;return=20 NULL;
  }
  Py_INCREF(m);
  break;=
 
IS_CUSTOM
 
static int
is_custom(char *name)
{
  struct = _customtab=20 *p;
 
if (!PyImport_Customtab)
  return 0;
 
p=3DPyImport_Customtab;
 
while(p){
     if=20 (p->get_name)
         if=20 ((*p->get_name)(name))
      return=20 1;
     p=3Dp->next;
}
 return = 0;
}
 
INIT_CUSTOM
static int
init_custom(char *name)
{
 struct = _customtab *p =3D=20 PyImport_Customtab;
 
 if (_PyImport_FindExtension(name, name) !=3D=20 NULL)
  return 1;
 
 while(p) {
     if=20 (p->get_name){
   if ((*p->get_name)(name))=20 {
    if (p->initfunc =3D=3D NULL)=20 {
     PyErr_Format(PyExc_ImportError,
&nb= sp;     "Cannot=20 re-init internal module=20 %.200s",
      name);
  &nb= sp;  return=20 -1;
    }
    if=20 (Py_VerboseFlag)
     PySys_WriteStderr("impo= rt %s #=20 builtin\n",=20 name);
    (*p->initfunc)(name);
  = ;  if=20 (PyErr_Occurred())
     return=20 -1;
    if (_PyImport_FixupExtension(name, name) = =3D=3D=20 NULL)
     return=20 -1;
    return=20 1;
   }
  }
 =20 p=3Dp->next;
 }
 return 0;
}
 
 
 
 
------=_NextPart_000_002D_01C298DF.B68FC6C0-- From greg@cosc.canterbury.ac.nz Mon Dec 2 03:23:21 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 02 Dec 2002 16:23:21 +1300 (NZDT) Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211 In-Reply-To: Message-ID: <200212020323.gB23NLC21006@kuku.cosc.canterbury.ac.nz> Jack Jansen : > Something that's been bothering me about frozen modules in the > classical sense (i.e. those that are stored in C static data > structures) is that the memory used by them is gone without any > chance at recovery. With virtual memory, that's probably not a problem, since it'll get paged out after it's been used. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From barry@python.org Mon Dec 2 04:39:13 2002 From: barry@python.org (Barry A. Warsaw) Date: Sun, 1 Dec 2002 23:39:13 -0500 Subject: [Python-Dev] Mac OSX issues References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15850.58353.356475.182137@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried GvR> other values, but the default of 512 (KB) is insufficient. I GvR> know the README file explains this, but now that I've GvR> experienced this myself, I wonder if we shouldn't hack main() GvR> to increase the stack size to 2 MB, inside an #ifdef darwin GvR> or something like that. +1, but I'll defer to Jack. We discussed this a while back (for Py2.2.2 IIRC), which is when I updated the README bit. FWIW, Pipermail (in Mailman) suffers the same fate and contains the following bit of code. -Barry -------------------- snip snip -------------------- # MacOSX has a default stack size that is too small for deeply recursive # regular expressions. We see this as crashes in the Python test suite when # running test_re.py and test_sre.py. The fix is to set the stack limit to # 2048; the general recommendation is to do in the shell before running the # test suite. But that's inconvenient for a daemon like the qrunner. # # AFAIK, this problem only affects the archiver, so we're adding this work # around to this file (it'll get imported by the bundled pipermail or by the # bin/arch script. We also only do this on darwin, a.k.a. MacOSX. if sys.platform == 'darwin': try: import resource except ImportError: pass else: soft, hard = resource.getrlimit(resource.RLIMIT_STACK) newsoft = min(hard, max(soft, 1024*2048)) resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard)) From brett@python.org Mon Dec 2 04:54:52 2002 From: brett@python.org (Brett Cannon) Date: Sun, 1 Dec 2002 20:54:52 -0800 (PST) Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 Message-ID: Here is the rough draft of the Summary. Guys have about a day to point out its flaws. ++++++++++++++++++++++++++++++++++++++++++++++++++ python-dev Summary, 2002-11-16 through 2002-11-30 ++++++++++++++++++++++++++++++++++++++++++++++++++ ====================== Summary Announcements ====================== Nothing to report to speak of. ==================== `bsddb3 imported`__ ==================== __ http://mail.python.org/pipermail/python-dev/2002-November/030247.html Martin v. Loewis merged bsddb3 3.4.0 into CVS under the name ``bsddb``. The old ``bsddb`` module is now no longer compiled by default; if it does get compiled, though, it ends up with the name ``bsddb185``. Barry Warsaw also requested that the extensive testing suite be incorporated and "run it only with a regrtest -u option". Martin wasn't sure how Barry wanted them incorporated, though, since there are multiple files to the test and most testing suites in the stdlib are a single file. Barry suggested that the testing files be put in a directory with the package and that test_bsddb.py just call the tests in that directory, much like how the email package does it. They were integrated and some errors and warnings were found that are being dealt with. It was also agreed upon that development will be moved over to Python so as to keep the module in Python sync'ed up properly and to keep poor Martin from having to import the files into Python's CVS constantly. ======================= `Licensing question`__ ======================= __ http://mail.python.org/pipermail/python-dev/2002-November/030256.html David Abrahams asked about a licensing issue with Boost.Python_ (it is a free library that "enables seamless interoperability between C++" and Python) and it's modified Python.h_ file that it uses. Originally there was no license at the top of that file, but that does not work for some corporations using Boost. So David stuck his own license at the top and asked if this is the right thing to do. Guido asked him to provide the PSF_ license_ at the top of the file and to mention what changes he made. The copyright had been added to the file for Python 2.2.2. .. _Boost.Python: http://www.boost.org/libs/python/doc/ .. _Python.h: .. _PSF: http://www.python.org/psf/ .. _license: http://www.python.org/2.2.2/license.html ========================= `Re: PyNumber_Check()`__ ========================= __ http://mail.python.org/pipermail/python-dev/2002-November/030236.html M.A. Lemburg noticed that PyNumber_Check()'s semantics on what will cause it to return had changed. He asked if it should check whether one of "nb_int, nb_long, nb_float is available (in addition to the tp_as_number slot)". Guido responded that he would like to see it deprecated. We got a history lesson of how PyNumber_Check() was written "when the presence or absence of the as_number "extension" to the type object was thought to be useful". Regardless, Guido said that testing like this does not prove something is a "number" and if you wanted to test this way you could do it yourself. In response, MAL said that perhaps PyNumber_Check() should be changed so that it returned true for something that is "usable as input to float(), int() or long()". Guido said that would be fine "as long as we all agree that that's *exactly* what they check for, and as long as we agree that there may be overlapping areas" for the various Py*_Check() functions. Guido later said testing for nb_int, nb_long, and nb_float was fine. =================================================== `Plea: can modulefinder.py move to the library?`__ =================================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030240.html Just van Rossum wanted to move Freeze's modulefinder.py_ into the stdlib so that it can be distributed with binary releases. In case you don't know what modulefinder.py does, it attempts to find all pure Python module dependencies for a pure Python module. In other words, it checks what the module imports and if it is a Python file, and if it is, records that; it repeats this for all modules it finds, creating a listing of modules needed for the module to run. Guido said that the module needed some work before it could be considered; it had ``print`` statements that were unneeded outside of Freeze and it had no documentation. Just agreed that the documentation needed to be done. As for the ``print`` statements, though, they only come out when ``debug`` is set to true; by default it is false. Guido said that was fine and agreed with the removal of the Windows-specific ``print`` statements. Thomas Heller later said in `another thread`_ that `patch #643711`_ was opened primarily for him and Just to do work in but that everyone was invited to help out. .. _another thread: http://mail.python.org/pipermail/python-dev/2002-November/030445.html .. _modulefinder.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/freeze/modulefinder.py .. _patch #643711: http://www.python.org/sf/643711 ============================ `Dictionary Foolishness?`__ ============================ __ http://mail.python.org/pipermail/python-dev/2002-November/030282.html Raymond Hettinger suggested having "dictionaries support the repetition" to allow one to create a dictionary with enough space as specified by the repetition:: >>> [0] * n # allocate an n-length list >>> {} * n # allocate an n-element dictionary Aahz recalled that dictionaries are resized upon adding to a dictionary and they could theoretically grow smaller. That would seem to possibly limit the usefulness of this idea. Guido then voted -1 (practically a death wish for an idea unless people clamor for it) saying that it relied too much on "arbitrary magic by side effect". He said if people *really* wanted this a method could be proposed. ============================= `dict() enhancement idea?`__ ============================= __ http://mail.python.org/pipermail/python-dev/2002-November/030290.html Just van Rossum suggested overloading the dictionary constructor so that arguments that went to ``**kwargs`` would be used to create the dictionary (this can be seen in the "`Python Cookbook`_" as recipe 1.2 or online at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52313). This is desired because that means cleaner code for creating dicts:: >>> dict(pigs='!fly', birds='fly') Barry commented that he liked it and had something similar in his code for Mailman_ . Thomas Heller voted +1 for it and also said that he used the idiom. Raymond Hettinger and myself also voted +1 for it. .. _Python Cookbook: .. _Mailman: =========================================== `Yet another string formatting proposal`__ =========================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030304.html Oren Tirosh proposed something (read the title to figure out what). He proposed the following syntax:: >>> "\(a) + \(b) = \(a+b)\n" >>> r"\(a) + \(b) = \(a+b)\n".cook() The advantages, according to Oren, are that it would not require introducing the use of a new symbol like ``$``, nor a new string prefix nor a new method. The ``.cook()`` method would be used to evaluate raw strings at a later point; it would draw arguments from the local and global namespace. The biggest drawback was unfamiliarity for programmers. Frederik Lundh pointed out that ``\(`` is "commonly used to escape parentheses in regular expression strings" (Effbot wrote re_ , so he should know). Oren then said that curly braces could (and pretty will) be used instead. Michael Chermside likes this design idea, but thinks the name for ``.cook()`` is not that great. Oren was going for a name that tied into "raw". Michael suggested the name ``.sub()`` to build off of the two PEPs already in existence covering string formatting (`PEP 215`_ and `PEP 292`_ ). .. _re: http://www.python.org/doc/current/lib/module-re.html .. _PEP 215: http://www.python.org/peps/pep-0215.html .. _PEP 292: http://www.python.org/peps/pep-0292.html ===================== `Expect in python`__ ===================== __ http://mail.python.org/pipermail/python-dev/2002-November/030313.html Eric Raymond proposed adding pexpect_ to the stdlib when it reaches version 1 (it is currently at 0.94). His thought was that having functionality like Expect_ would be a boon for Python and use for system administration. Eric said he had been using the module and had no problems with it (Prabhu Ramachandran also said it had worked for him). David Ascher said that he would like to see a more abstract API to allow it work for things other than character streams. He also would like to see something work better on Windows. Eric said that he would not want to hold up this for hopes of getting something better since it already works well for what it does. But it appears that the creator of pexpect is more than willing to help maintain the module if it makes it into the stdlib. Zach Weinberg said that he would be willing to put some work into making the pty_ module more portable since pexpect does its thing using pty. .. _pexpect: http://pexpect.sf.net/ .. _Expect: http://expect.nist.gov/ .. _pty: http://www.python.org/doc/current/lib/module-pty.html =================================== `PEP 288: Generator Attributes`__ =================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030321.html Raymond Hettinger has revised `PEP 288`_ with a new proposal on how to pass things into a generator that has already started. He has asked for comments on the changes, so let him know what you think. .. _PEP 288: http://www.python.org/peps/pep-0288.html =============================================== `PyMem_MALLOC (was [Python-Dev] Snake farm)`__ =============================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030333.html Continuation of http://mail.python.org/pipermail/python-dev/2002-November/029853.html There was a possible issue with ``PyMem_MALLOC()`` that Marc Recht had discovered on FreeBSD. It eventually was tracked down to FreeBSD's implementation of ``malloc()``: ``malloc(0)`` always return 0x800. M.A. Lemburg suggested changing a test in the configure script to try to catch when a platform returned an address for ``malloc(0)`` and treat it just like when it would return ``NULL`` (``NULL`` can't be blindly returned since that would signal a memory error; returning ``NULL`` in a C extension signals an error). Marc came back with news that C99 says that this is legitimate behavior for ``malloc()`` so this could possibly affect other platforms. Marc suggested that ``PyMem_MALLOC()`` just be redefined to ``n ? malloc(n) : NULL``. Problem is that the ``NULL`` issue mentioned above comes into play with this solution. Tim Peters suggested either ``malloc(n || 1)`` or ``malloc(n ? n : 1)`` (the former being a Python idiom that doesn't cut it in C). he does not want to mess with the configure scripts since they have "proven itself too brittle too many times". Tim wanted a way to prevent ever calling the function with 0, but Guido couldn't see any way of doing that without an extra jump. The committed solution is ``malloc((n) ? (n) : 1)``. Easier to just waste one byte then have to deal with the special casing of passing 0. The extra test was not really a worry since no measurable performance reported by Tim. Besides, Tim pointed out "this is ideal for a conditional-move instruction, and more architectures are growing that". ==================================================== `Half-baked proposal: * (and **?) in assignments`__ ==================================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030349.html Gareth McCaughan suggested cutting down one of the separations between parameter passing and assignment by allowing assignment to use arbitrary argument lists:: >>> a,b,*c = 1,2,3,4,5 # c == (3, 4, 5) >>> year, month, day, *dummy = time.localtime() I argued that I didn't like the slightly cluttered look on the left-hand side (LHS) of the assignment. Martin v. Loewis and I basically ended up saying we wanted to keep assignments clear and concise and that this would not help to keep that. Steve Holden basically ended up agreeing. Brian Quinlin, Patrick O'Brien, Nathan Clegg and Timothy Delany liked the idea. The biggest argument in support was that it would allow for a more functional programming style (and that obviously can be good or bad depending on your P.O.V.; I say bad =):: >>> car,*cdr = [head, t1, t2, t3] # car == head, cdr == (t1, t2, t3) In case you don't have functional programming (especially Lisp/Scheme) experience, the basic data structure in Lisp-like language is a list and the most common way to manipulate those lists is with the functions ``car`` and ``cdr``. ``car`` returns the "head", or front, of the list; ``cdr`` returns the "tail", or everything but the head, of the list. This allows for simple recursion since you just pass the ``cdr`` of a list on the recursive call after having dealt with the head of the list. There was also the suggestion of allowing the arbitrary assignment variable to be anywhere in the list of assignment variables:: >>> a,*b,c = 1,2,3,4,5 # a == 1, b == (2, 3, 4), c == 5 To prove that this was not really needed I wrote a function that took in an iterable and the number of variables to assign to and then returned the proper number iterations on the iterator and then the iterator as the last thing returned. Alex Martelli of course improved upon it (and also continued to correct my slightly incorrect statements):: def peel(iterable, arg_cnt=1): """Return ``arg_cnt`` values from iterator of ``iterable`` and then the iterator itself.""" iterator = iter(iterable) for num in xrange(arg_cnt): yield iterator.next() yield iterator The idea of a module for the stdlib containing iterator helper functions was suggested by Alex. One is in progress by Raymond Hettinger. Armin Rigo suggested having iterators become a type. That was quickly shot down, although having the suggested iterator helper module contain a class that could be subclassed by iterators was received with positive comments. The thread ended very quickly after Guido said that he didn't think "that there's a sufficient need to add new syntax". =================================== `from tuples to immutable dicts`__ =================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030358.html Armin Rigo said that he would like to have an immutable type that acted like a dictionary; basically like a struct from C. Martin v. Loewis agreed on the need, but opposed the idea of adding another built-in type or syntax for such a type; that left something for the stdlib. Martin suggested something like:: >>> struct_seq(name, doc, n_in_sequence, (fields)) where ``field`` is a bunch of (name, doc) tuples. What would be returned would be a "thing [that] would be similar to os.stat_result: you [can] call it with the mandatory fields in sequence, and can call it with the optional fields by keyword argument". Armin didn't like it since it went against his initial proposal "which was to have a lightweight and declaration-less way to build structures". He basically ended up suggesting something along the lines of tuples with keyword arguments. Martin didn't like it since he didn't see a great use for it. In the end Armin said to just drop the idea. ============================================ `urllib performance issue on FreeBSD 4.x`__ ============================================ __ http://mail.python.org/pipermail/python-dev/2002-November/030391.html Andrew MacIntyre brought a thread on python-list to python-dev's attention about urllib performance compared to wget (wget is used to download web sites and files). Apparently a patch was applied to Python 2.2.2 that made the used socket be unbuffered instead of using the system default. The question became why this was done. The answer (thanks to Martin v. Loewis) was to prevent deadlock. Apparently under HTTP 1.1 a server can keep a connection open while waiting for the next command. If the connection was buffered it would block until it read enough to fill the buffer which may never come. Frederik Lundh suggested that a subclass or option be available that allowed the choosing of unbuffered or not. Andrew said he would put it on his todo list. ===================================== `test failures on Debian unstable`__ ===================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030384.html Failures on the build of Debian's unstable version of Python led to a discussion about how modules are skipped in the testing suite. `Lib/test/regrtest.py`_ keeps a list of tests that are expected to be skipped on various platforms. Martin v. Loewis doesn't like it because tests such as for the bz2 module are attempted regardless of whether the bz2 library is even installed and yet it is expected to succeed on Linux. Martin summarized that "For many of the tests that are somtimes skipped, knowing the system does not tell you whether the test will should rightfully be skipped, on that system" since tests are skipped often because a module was not there that needed to be imported for the test. Tim Peters, on the other hand, likes it. Since he maintains the Windows distribution from PythonLabs he likes it since it lets him know when new things have been added to Python and might need to be excluded from the Windows distro. Neil (who pointed out the Debian problems) was able to recognize that the tests that failed were meant to pass under Linux. Tim admitted he only cared about keeping the mechanism for Windows; he could care less if it is removed for Linux. Patrick O'Brien chimed in (with Aahz supporting) that the feature is handy since you can easily find out libraries you are missing that you could potentially install. Guido stepped in and suggested setting up a mechanism that would allow an external table in a file to be used when present instead of the default list of tests to skip. Don't think anyone has stepped up to implement this. .. _Lib/test/regrtest.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/test/regrtest.py =================================================================== `Currently baking idea for dict.sequpdate(iterable, value=True)`__ =================================================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030431.html Raymond Hettinger presented "a write-up for a proposed dictionary update method". It basically took an iterable and added keys based on the values returned by the iterator with a value as passed in and used for all new keys. The rationale was to have a fast way to be able to do membership testing using dict's ``__contains__`` or removing duplicates by creating a dict and then outputting the keys using the aptly named ``.keys()``. Previous objecctions to something like this were about the dict constructor and the ``sets`` module. The ones about the constructor are dealt with by making this a method. The latter was argued against by saying that the ``sets`` module is slow. Frederik Lundh brought up that we really don't need multiple ways of doing the same thing. Just van Rossum agreed and said this killed the idea for him. Guido chimed in and said that the ``sets`` module was to help solidify the sets API so that at some point it could be coded in C. To address the speed complaint Guido suggested limiting the ``sets`` module initially to make it faster so that the type won't be held back or unutilized because of its speed. Tim Peters spoke up, though, and said that the spambayes_ project used ``sets`` and he didn't have any complaints. But when major membership testing was needed a dict was used. And Tim pointed out that in order for any C sets code to be fast it would have to directly use dict's C ``__contains__`` code. What this method should return was brought up by Just. Some thought ``None`` since ``.update()`` returns that. Others said ``True``. Guido said ``None`` since ``True`` should only be used when something is explicitly true. Making it a class method was also suggested by Just as an easy way to make it like a constructor. Raymond agreed and changed his proprosal as well as to have the method be named ``.fromseq()``. But then Walter Dorwald said ``.fromkeyseq()`` should be used since there "is another constructor that creates the dict from a sequence of items". Guido voted +1 on that idea. .. _spambayes: http://spambayes.sf.net/ ====================================== `Re: release22-maint branch broken`__ ====================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030330.html Tim Rice discovered that trying to build Python from a directory other then where the source was did not work for the Python 2.2.* CVS. It was all eventually solved and fixed in the CVS branch. I am mentioning it here in case someone reading this had a similar issue. ================================ `Dictionary evaluation order`__ ================================ __ http://mail.python.org/pipermail/python-dev/2002-November/030458.html Gustavo Niemeyer asked about how to handle code like ``{f1():f2(), f3(): f4()}`` and its execution order as pointed out by `bug #448679`_ . As it stood it evaluated in the order of f2, f1, f4, f3. Apparently Guido once upon a time considered this a bug. But Guido mentioned that left-to-right evaluation is not always wanted since ``a = {}; a[f1()] = f2()`` would want f2 to evaluate first. He asked what Jython did. Finn Bock said that Jython went f1, f2, f3, f4. In that case Guido didn't see any reason to fix it. But Tim Peters brought up the point that the bug was more about the lack of specifics on this in the documentation. Gustavo said he would make the code fix along with patches to the docs. .. _bug #448679: http://www.python.org/sf/448679 =========================== `int/long FutureWarning`__ =========================== __ http://mail.python.org/pipermail/python-dev/2002-November/030520.html Mark Hammond asked how the upcoming change in Python 2.4 of hex/oct constants will affect his C extension code and something like ``PyArg_ParseTuple()`` (this function takes arguments passed to something and breaks it up into its individual parts since all arguments are passed as tuples in C code). In case you don't know about the warnings, Python 2.3 warns you that code like ``SOMETHING = 0x80000000`` could have a different meaning in Python 2.4; most likely it will be treated as a positive long. You can currently get rid of the warnings by changing the constant into a long by tacking on a ``L`` to the end of the number. Martin v. Loewis that if Mark appended the ``L`` to his constants that it would not work for an ``i`` argument for ``PyArg_ParseTuple()``. But Guido stepped up and said that there will be no issue since Python will be changed so that Mark's code will accept the constant as a positive long. This caused Guido to wonder if the warning could be changed to some other warning that is not normally printed out. Guido then mentioned that he has "long promised a set of new format codes for PyArg_ParseTuple() to specify taking the lower N bits (for N in 8, 16, 32, 64) and throwing the rest away, without range checks". "If someone else can get to this first, that would be great". So someone be nice to Guido and do this for him. =) Either way no specific resolution has been reached. As of right now you can just live with the warnings, supress the warnings, or change your constants to longs and hope you are not passing into a C extension function that wants an int. ========================================== `assigning to new-style-class.__name__`__ ========================================== __ http://mail.python.org/pipermail/python-dev/2002-November/030475.html Michael Hudson has been working on `patch #635933`_ to allow for assignment to ``__name__`` and ``__bases__`` for new-style classes (this was all so that ``__name__`` would handle nested classes properly to allow for proper pickling; that thread was called `metaclass insanity`_ ). He ran into a slight issue with dealing with assigning to ``__name__``. To get it working, Michael wanted to treat heap and non-heap types differently. For non-heap types Michael wanted to "everything in tp_name up to the first dot is __module__, the rest is __name__". For non-heap types, he wanted to have ``__module__`` as "always __dict__['__module__'], __name__ is always tp_name (or rather ((etype*)type)->name)". And as for the issue of if someone is crazy enough to delete the dict key of ``__module__``, Michael said Python wouldn't crash but you probably would not like the outcome of running code. =) Guido responded saying that Michael's proposal was acceptable. But then there was an issue with ``.mro()`` after the bases had been rearranged. Michael worried about what to do when there was a conflict down the intheritence tree. He thought reverting back to the way things were if there was an issue was best. This would require keeping around copies of the previous states until the changes propogated all the way through. Samuele Pedroni stepped in to try to answer this question (Samuele rewrote the MRO code recently and is directly mentioned in `C3 implementation`_ ). He came up with a possible case where there could be a possible order disagreement if two of the bases of a class had the same bases but one had the order swapped compared to the other (so C has bases of (A, B) and D has bases of (A, B) as well and E had bases (C, D); if C's bases became (B, A), E now has an order disagreement). He suggested that "the mros of the subclasses should be computed lazily when needed (e.g. on the first - after the changes - dispatch), although this may produce inconsistences and errors at odd times". Michael showed that his solution would catch the problem. But he did not like the idea of lazily evaluating; he wanted a more restrictive solution since this is a new thing. Michael stated that what he wanted this for was to "to swap out one class for another -- making instances of the old class instances of the new class, which was possible and making subclasses of the old subclasses of the new, which wasn't". It also turned out neither APL nor Dylan allow this kind of thing so Michael is breaking new ground. Samuele asked about when the classes had solid bases (i.e., only a single superclass such as ``object``). Michael said it would handled with no problem. .. _C3 implementation: http://mail.python.org/pipermail/python-dev/2002-October/029230.html .. _patch #635933: http://www.python.org/sf/635933 .. _metaclass insanity: http://mail.python.org/pipermail/python-dev/2002-November/029872.html ===================== `Classmethod Help`__ ===================== __ http://mail.python.org/pipermail/python-dev/2002-November/030452.html Raymond Hettinger emailed the list because Guido said that the few people in the world who understand descriptors for C code are on the list. The main reason I am mentioning the thread here, though, is because Armin Rigo gave the answer that "There are METH_CLASS and METH_STATIC flags that you can set in the tp_methods table". You also learn thanks to Guido that you should only use ``PyErr_BadInternalCall()`` when you know that a "bad argment must have been created by a broken piece of C code". From tim.one@comcast.net Mon Dec 2 05:44:30 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 02 Dec 2002 00:44:30 -0500 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: <15850.37138.572612.258716@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > Posix.kill() wants a pid. Can't os.kill() want anything it wants? ;-) Yes. On Windows it wants a handle, because that's what spawn*() returns on Windows. But, again, Windows processes aren't intended to be killed externally. > In particular, can't it test its arguments to see if it got a handle > or a pid then "Do The Right Thing" (tm Spike Lee I think)? No, process handles and process ids are both just "little integers" on Windows. > ... > For the python-dev types not on the spambayes list, I do have an > application in mind. In my environment I need to tunnel SB's > pop3proxy app through an ssh connection. Starting and stopping > it works okay on my MacOSX system, and I presume it will under other > Unix-based systems. I began thinking about whether this would port > to other platforms and noticed that os.kill() is only supported under > Unix. Why do you need to kill a process externally? For example, why can't you make "please stop now" a part of the protocol, so that a process can terminate itself gracefully when told to? > It may turn out that my desire is moot if ssh can't be run under > Windows anyway. http://www.python.org/dev/devfaq.html#w1 From martin@v.loewis.de Mon Dec 2 09:25:22 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 02 Dec 2002 10:25:22 +0100 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: References: Message-ID: Tim Peters writes: > Yes. On Windows it wants a handle, because that's what spawn*() > returns on Windows. But, again, Windows processes aren't intended > to be killed externally. That sounds like FUD. Why is it that processes aren't intended to be killed? Because the assassin can't know what state the process is in, so the process may not complete correctly? This is a general problem with killing, not specific to Windows, and it never stopped a killer. Or can you crash the operating system or the killer process by terminating some other process? > Why do you need to kill a process externally? For example, why can't you > make "please stop now" a part of the protocol, so that a process can > terminate itself gracefully when told to? Because the process did not respond to the protocol. Should I kill it anyway and send a bug report to Microsoft? Regards, Martin From Jack.Jansen@cwi.nl Mon Dec 2 09:59:51 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 2 Dec 2002 10:59:51 +0100 Subject: [Python-Dev] Mac OSX issues In-Reply-To: <15850.58353.356475.182137@gargle.gargle.HOWL> Message-ID: On Monday, Dec 2, 2002, at 05:39 Europe/Amsterdam, Barry A. Warsaw wrote: > >>>>>> "GvR" == Guido van Rossum writes: > > GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried > GvR> other values, but the default of 512 (KB) is insufficient. I > GvR> know the README file explains this, but now that I've > GvR> experienced this myself, I wonder if we shouldn't hack main() > GvR> to increase the stack size to 2 MB, inside an #ifdef darwin > GvR> or something like that. > > +1, but I'll defer to Jack. We discussed this a while back (for > Py2.2.2 IIRC), which is when I updated the README bit. FWIW, > Pipermail (in Mailman) suffers the same fate and contains the > following bit of code. I'm +0 on this. The reason I'm not +1 is that 2MB is just as arbitrary as 1MB, and it's only picked because it'll make the current set of tests pass. And the current set of tests doesn't pass on MacOSX because another equally arbitrary number in test_re happens to create a stack >= 1MB but <= 2MB. I would be +1 on just changing the test to fit in a 1MB stack. That is, unless there are subtleties in the test that I missed, and there really is a reason for the specific choice of the number 50000. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From guido@python.org Mon Dec 2 10:01:12 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 02 Dec 2002 05:01:12 -0500 Subject: [Python-Dev] Mac OSX issues In-Reply-To: Your message of "Sun, 01 Dec 2002 23:39:13 EST." <15850.58353.356475.182137@gargle.gargle.HOWL> References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> Message-ID: <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> > GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried > GvR> other values, but the default of 512 (KB) is insufficient. I > GvR> know the README file explains this, but now that I've > GvR> experienced this myself, I wonder if we shouldn't hack main() > GvR> to increase the stack size to 2 MB, inside an #ifdef darwin > GvR> or something like that. [Barry] > +1, but I'll defer to Jack. We discussed this a while back (for > Py2.2.2 IIRC), which is when I updated the README bit. FWIW, > Pipermail (in Mailman) suffers the same fate and contains the > following bit of code. Thanks! I've added your code to regrtest.py. I'm reluctant to do this in PyMain(), because I don't know whether this adds to Python's resource usage; the re tests are pretty extreme. I'm also considering disabling test_largefile on Mac OSX unless a -u option is given. I'm still seeing errors in test_socket; today (on a different network) I got this: test test_socket failed -- Traceback (most recent call last): File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes ip = socket.gethostbyname(hostname) gaierror: (7, 'No address associated with nodename') --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@cwi.nl Mon Dec 2 10:06:44 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 2 Dec 2002 11:06:44 +0100 Subject: [Python-Dev] PyArg_Parse (was: int/long FutureWarning) Message-ID: Martin: > Jack Jansen writes: > >> How about taking a completely different angle on this matter, and >> looking at PyArg_Parse itself? If we can keep PyArg_Parse 100% >> backward compatible (which would mean that its "i" format would take >> any IntObject or LongObject between -1e31 and 1e32-1) and introduce a >> new (preferred) way to parse arguments that not only does the right >> thing by being expressive enough to make a difference between >> "currency integers" and "bitmap integers", but also cleans up the >> incredible amount of cruft that PyArg_Parse has accumulated over the >> years? > > I had a similar idea, so I'd encourage you to spell out your proposal > in more detail, or even in an implementation. > My idea was to provide a ParseTuple wrapper, [...] > Those of you needing to support older Python releases could > > #define PyArg_ParseTupleLenient PyArg_ParseTuple > > in your distribution, or provide other appropriate wrappers. Unfortunately this wouldn't work if the distributions are binary distributions. But, aside from that, I think I would want to go much further than this _if_ I put time in redesigning PyArg_Parse. I would first like to take inventory of all the problems there are with PyArg_Parse, and then see whether we can design something that will solve most or all of these issues, without being overly complex in everyday use. And before I embark on that journey I would first like to have a group of people willing to put effort into this, plus the go-ahead of Guido (there's little point in designing a new mechanism if there is no chance of it being adopted as the general case, especially if this new mechanism may need a new PyMethodDef flag or some such thing). As a kickoff, here are some of my gripes about PyArg_Parse. 1. The format chars are arcane and without any logic. There is no logic to signed/unsigned type specifiers, some modifiers are suffixes (s#), some are different format chars (s versus z), some are prefixes (e). Everyone knows a basic set of 5 or 6 and has to look the rest up. Some types have special shortcuts without a clear rationale (String and Unicode are the only types to have an "O!", typename shortcut in the form of "S" and "U"). 2. There is conversion information interspersed with the argument list, for instance with O!, O& or es formats. This makes it very difficult to represent or build an argument parsing format in Python (this is worsened because some of the C objects in the argument list, such as the O& routine pointers, have no Python equivalent). And representing an argument list parser in Python is something that you need if you want to do dynamic wrapping of any kind (calldll, PyObjC, etc). 3. There is no way to create new, temporary objects during argument parsing, because there is no corresponding "release" call and no way to make the caller release new objects. Having temporary objects would make conversion a lot easier. Unicode and strings are the first types that come to mind, but there are probably others. 4. PyArg_ParseTupleAndKeywords makes the situation even worse. Each argument now has *three* different "index positions", the real index in the keyword list, a modified one in the format string (ignore all non-alphabetic chars and "e") and a third one in the argument list (ignore all extraneous arguments corresponding to es or O& or what-have-you). -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From guido@python.org Mon Dec 2 10:30:41 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 02 Dec 2002 05:30:41 -0500 Subject: [Python-Dev] Mac OSX issues In-Reply-To: Your message of "Mon, 02 Dec 2002 10:59:51 +0100." References: Message-ID: <200212021030.gB2AUfR24830@pcp02138704pcs.reston01.va.comcast.net> [Jack] (BTW all your mail comes poorly folded like shown here. What's up with that?) > I'm +0 on this. The reason I'm not +1 is that 2MB is just as arbitrary > as 1MB, and > it's only picked because it'll make the current set of tests pass. And > the > current set of tests doesn't pass on MacOSX because another equally > arbitrary > number in test_re happens to create a stack >= 1MB but <= 2MB. I would > be +1 on just changing the test to fit in a 1MB stack. > > That is, unless there are subtleties in the test that I missed, and > there really > is a reason for the specific choice of the number 50000. I vaguely recall that the number 50000 is somewhat historically significant, but I don't remember why. But it works on all other platforms I know. Do you know if setting the stack limit actually allocates that much memory for the stack in the process in Mac OSX, or does it only reserve VM space (like on traditional Unix)? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 2 10:33:07 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 02 Dec 2002 05:33:07 -0500 Subject: [Python-Dev] PyArg_Parse (was: int/long FutureWarning) In-Reply-To: Your message of "Mon, 02 Dec 2002 11:06:44 +0100." References: Message-ID: <200212021033.gB2AX7E24846@pcp02138704pcs.reston01.va.comcast.net> > But, aside from that, I think I would want to go much further than this > _if_ I put time in redesigning PyArg_Parse. I would first like to take > inventory of all the problems there are with PyArg_Parse, and then see > whether we can design something that will solve most or all of these > issues, without being overly complex in everyday use. > > And before I embark on that journey I would first like to have a group > of people willing to put effort into this, plus the go-ahead of Guido > (there's little point in designing a new mechanism if there is no > chance of it being adopted as the general case, especially if this new > mechanism may need a new PyMethodDef flag or some such thing). +1 on a redesign of PyArg_Parse. Though I don't want to hold up the 2.3a1 release. > As a kickoff, here are some of my gripes about PyArg_Parse. (I can't spend too much time not paying attention to the Zope3 sprintathon, so I'll try to read these later. Chances are I agree. Surely the format characters are a big mess.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 2 10:37:23 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 02 Dec 2002 05:37:23 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: Your message of "Sun, 01 Dec 2002 02:16:52 +0100." <003001c298d7$550e8240$0201010a@wiktor> References: <003001c298d7$550e8240$0201010a@wiktor> Message-ID: <200212021037.gB2AbNh24921@pcp02138704pcs.reston01.va.comcast.net> > I think Python should have a more universal import mechanism , which > could be done by adding a new C_CUSTOM type to the definitions for > dynamic loading (importdl.h) , a new struct _customtab to import.h > and a new function to Python API (import.c). Then any > archive,database,internet import architecture could be implemented > in C extensions without messing with Python core. > > Wiktor Sadowski No time to review in detail, but +1 on the concept. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Mon Dec 2 11:21:41 2002 From: mwh@python.net (Michael Hudson) Date: 02 Dec 2002 11:21:41 +0000 Subject: [Python-Dev] Mac OSX issues In-Reply-To: Jack Jansen's message of "Mon, 2 Dec 2002 10:59:51 +0100" References: Message-ID: <2mr8d0k6nu.fsf@starship.python.net> Jack Jansen writes: > I'm +0 on this. The reason I'm not +1 is that 2MB is just as > arbitrary as 1MB, Not quite; it's also the stack size passed to pthreads IIRC. So after ulimit -s 2048 all threads get the same stack size (I think). Cheers, M. -- 42. You can measure a programmer's perspective by noting his attitude on the continuing vitality of FORTRAN. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From just@letterror.com Mon Dec 2 11:37:37 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 2 Dec 2002 12:37:37 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <003001c298d7$550e8240$0201010a@wiktor> Message-ID: Wiktor Sadowski wrote: > I think Python should have a more universal import mechanism , which > could be done by adding a new C_CUSTOM type to the definitions for > dynamic loading (importdl.h) , a new struct _customtab to import.h > and a new function to Python API (import.c). Then any > archive,database,internet import architecture could be implemented in > C extensions without messing with Python core. (Could you please post a normal diff of your patch? It looks very interesting.) I think I agree with this concept as well. Yesterday I tried to review the latest incarnation of the "import from zip file" patch and ran away screaming: it's way too intrusive on import.c, touches too many files and is otherwise simply incomprehensible to me. I've been doing some thinking about the import mechanism, too, and my current gripe is about the __import__ hook: I find it utterly useless that the implementer of the hook is supposed to deal with sys.modules. What I wish for is a hook that only gets invoked if the module isn't already in sys.modules. Here's a quick idea. Say we add a new dynamic variable to sys named "import_hooks". It's a list that is empty by default, but may contain import hooks. If an import occurs, sys.modules is checked first, and *only* if the module isn't yet loaded, something like this will be done: for hook in sys.import_hooks: mod = hook(absolutemodulename, parentpackage_or_None) if mod is not None: return mod return builtin_import_hook(absolutemodulename, parentpackage_or_None) (Hm, instead of calling builtin_import_hook() explicitly, it could also be part of the default sys.import_hooks list. Yeah, that's better, as it allows one to completely disable builtin import.) For an import from within a package, the mechanism is possibly invoked twice. For example if a submodule of Foo imports Baz, the mechanism is called once with "Foo.Baz" and if that fails (returns None), it's called again with "Baz". I think this will make writing import hooks a) more attractive and b) far less brittle, since there's so much less to implement. A new-style import hook could use sys.path if that makes sense for it, but for (say) a zip file importer it makes more sense (to me at least) to just add a hook per zip file. No need to allow non-string types on sys.path. I'm not entirely sure yet how my idea and yours could be integrated, but I think it's work a look. Just From Jack.Jansen@cwi.nl Mon Dec 2 12:38:23 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 2 Dec 2002 13:38:23 +0100 Subject: [Python-Dev] Mac OSX issues In-Reply-To: <200212021030.gB2AUfR24830@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Monday, Dec 2, 2002, at 11:30 Europe/Amsterdam, Guido van Rossum wrote: > I vaguely recall that the number 50000 is somewhat historically > significant, but I don't remember why. But it works on all other > platforms I know. Except all other platforms with small stacks (MacOS9, WinCE, probably more). > Do you know if setting the stack limit actually allocates that much > memory for the stack in the process in Mac OSX, or does it only > reserve VM space (like on traditional Unix)? I don't know, but I can't imagine it does anything more than reserve VM space. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From Jack.Jansen@cwi.nl Mon Dec 2 12:45:38 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 2 Dec 2002 13:45:38 +0100 Subject: [Python-Dev] Mac OSX issues In-Reply-To: <2mr8d0k6nu.fsf@starship.python.net> Message-ID: On Monday, Dec 2, 2002, at 12:21 Europe/Amsterdam, Michael Hudson wrote: > Jack Jansen writes: > >> I'm +0 on this. The reason I'm not +1 is that 2MB is just as >> arbitrary as 1MB, > > Not quite; it's also the stack size passed to pthreads IIRC. So after > ulimit -s 2048 all threads get the same stack size (I think). On MacOSX the pthread default stack size is 512K, which is the same as the default stack ulimit. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From barry@python.org Mon Dec 2 13:15:00 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 2 Dec 2002 08:15:00 -0500 Subject: [Python-Dev] Mac OSX issues References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15851.23764.693884.646123@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> [Barry] >> +1, but I'll defer to Jack. We discussed this a while back >> (for Py2.2.2 IIRC), which is when I updated the README bit. >> FWIW, Pipermail (in Mailman) suffers the same fate and contains >> the following bit of code. GvR> Thanks! I've added your code to regrtest.py. I'm reluctant GvR> to do this in PyMain(), because I don't know whether this GvR> adds to Python's resource usage; the re tests are pretty GvR> extreme. Cool, works for me. GvR> I'm also considering disabling test_largefile on Mac OSX GvR> unless a -u option is given. GvR> I'm still seeing errors in test_socket; today (on a different GvR> network) I got this: GvR> test test_socket failed -- Traceback (most recent call last): GvR> File "/Users/guido/python/src/Lib/test/test_socket.py", GvR> line 223, in testHostnameRes ip = GvR> socket.gethostbyname(hostname) GvR> gaierror: (7, 'No address associated with nodename') I'll be booting up OSX tonight, so I'll try doing a cvsup and running the tests locally. -Barry From barry@python.org Mon Dec 2 13:32:04 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 2 Dec 2002 08:32:04 -0500 Subject: [Python-Dev] PyArg_Parse (was: int/long FutureWarning) References: Message-ID: <15851.24788.810331.792541@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: JJ> And before I embark on that journey I would first like to have JJ> a group of people willing to put effort into this, plus the JJ> go-ahead of Guido (there's little point in designing a new JJ> mechanism if there is no chance of it being adopted as the JJ> general case, especially if this new mechanism may need a new JJ> PyMethodDef flag or some such thing). You might not want to wait that long, but it would be a good topic for a Pycon sprint. -Barry From fredrik@pythonware.com Mon Dec 2 15:01:14 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 2 Dec 2002 16:01:14 +0100 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? References: Message-ID: <077101c29a13$a9b78cf0$0900a8c0@spiff> brett wrote: > those programs where deadlock would occur do not lose or gain anything > since they couldn't run on a platform without ``thread`` anyway. try: import threading except ImportError: threading =3D None # use non-threaded behaviour From guido@python.org Mon Dec 2 15:36:53 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 02 Dec 2002 10:36:53 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Your message of "Mon, 02 Dec 2002 16:01:14 +0100." <077101c29a13$a9b78cf0$0900a8c0@spiff> References: <077101c29a13$a9b78cf0$0900a8c0@spiff> Message-ID: <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> > try: > import threading > except ImportError: > threading = None # use non-threaded behaviour But then you'd have to sprinkle the rest of the code with "if threading is not None" tests. Brett's dummy_thread is an attempt to reduce this clutter. I've been known to write a mini-version of dummy_thread.py in code to minimize the clutter; it seems useful to generalize this. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 2 16:23:06 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 02 Dec 2002 11:23:06 -0500 Subject: [Python-Dev] Sleepycat Berkeley DB on OSX Message-ID: <200212021623.gB2GN6g21364@pcp02138704pcs.reston01.va.comcast.net> When running test_bsddb3 on Mac OSX, using BSDDB 4.0.14, I get tracebacks like these: Exception in thread writer 0: Traceback (most recent call last): File "/Users/guido/python/src/Lib/threading.py", line 416, in __bootstrap self.run() File "/Users/guido/python/src/Lib/threading.py", line 404, in run apply(self.__target, self.__args, self.__kwargs) File "/Users/guido/python/src/Lib/bsddb/test/test_thread.py", line 115, in writerThread dbutils.DeadlockWrap(d.put, key, self.makeData(key), max_retries=12) File "/Users/guido/python/src/Lib/bsddb/dbutils.py", line 61, in DeadlockWrap except _db.DBLockDeadlockError: NameError: global name '_db' is not defined and these: Exception in thread reader 8: Traceback (most recent call last): File "/Users/guido/python/src/Lib/threading.py", line 416, in __bootstrap self.run() File "/Users/guido/python/src/Lib/threading.py", line 404, in run apply(self.__target, self.__args, self.__kwargs) File "/Users/guido/python/src/Lib/bsddb/test/test_thread.py", line 140, in readerThread c = d.cursor() DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of available locks') Any clues??? I don't get these on Linux, although there I get a failure too: test test_bsddb3 failed -- errors occurred; run in verbose mode for details Frankly, verbose mode waas too verbose to bother. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Mon Dec 2 16:37:07 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 2 Dec 2002 11:37:07 -0500 Subject: [Python-Dev] Sleepycat Berkeley DB on OSX References: <200212021623.gB2GN6g21364@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15851.35891.468176.894108@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> Any clues??? GvR> I don't get these on Linux, although there I get a failure GvR> too: GvR> test test_bsddb3 failed -- errors occurred; run in verbose GvR> mode for details GvR> Frankly, verbose mode waas too verbose to bother. :-( None off the top of my head, but I'm swapping in my BerkeleyDB pages to try to finish off support for 4.1.24. I'll look into these when I boot OSX again tonight. -Barry From gward@python.net Mon Dec 2 17:06:09 2002 From: gward@python.net (Greg Ward) Date: Mon, 2 Dec 2002 12:06:09 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: References: <003001c298d7$550e8240$0201010a@wiktor> Message-ID: <20021202170609.GA5209@cthulhu.gerg.ca> On 02 December 2002, Just van Rossum said: > I've been doing some thinking about the import mechanism, too, and my > current gripe is about the __import__ hook: I find it utterly useless > that the implementer of the hook is supposed to deal with > sys.modules. What I wish for is a hook that only gets invoked if the > module isn't already in sys.modules. I can see where you're coming from, but one disadvantage to your scheme is that it would not be possible to override .py/.pyc/.pyo files -- they would always be imported in preference to any other scheme. This could well be a feature. ;-) Greg -- Greg Ward http://www.gerg.ca/ What happens if you touch these two wires tog-- From mwh@python.net Mon Dec 2 17:38:54 2002 From: mwh@python.net (Michael Hudson) Date: 02 Dec 2002 17:38:54 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime obj_delta.c,1.14,1.15 test_both.py,1.15,1.16 In-Reply-To: tim_one@users.sourceforge.net's message of "Mon, 02 Dec 2002 09:31:39 -0800" References: Message-ID: <2mlm38jp75.fsf@starship.python.net> tim_one@users.sourceforge.net writes: > + /* XXX A further broken attempt to get pickling to work. > + * XXX This avoids the problem above, but dies instead with > + * XXX PicklingError: Can't pickle : it's not > + * XXX found as __builtin__.tmedelta You need to arrange for a module name to appear in timedelta's tp_name: Index: obj_delta.c =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/datetime/obj_delta.c,v retrieving revision 1.15 diff -c -r1.15 obj_delta.c *** obj_delta.c 2 Dec 2002 17:31:21 -0000 1.15 --- obj_delta.c 2 Dec 2002 17:37:55 -0000 *************** *** 673,679 **** static PyTypeObject PyDateTime_DeltaType = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ ! "timedelta", /* tp_name */ sizeof(PyDateTime_Delta), /* tp_basicsize */ 0, /* tp_itemsize */ 0, /* tp_dealloc */ --- 673,679 ---- static PyTypeObject PyDateTime_DeltaType = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ ! "datetime.timedelta", /* tp_name */ sizeof(PyDateTime_Delta), /* tp_basicsize */ 0, /* tp_itemsize */ 0, /* tp_dealloc */ seems a good bet. Cheers, M. -- I'm not sure that the ability to create routing diagrams similar to pretzels with mad cow disease is actually a marketable skill. -- Steve Levin -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From just@letterror.com Mon Dec 2 17:50:32 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 2 Dec 2002 18:50:32 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <20021202170609.GA5209@cthulhu.gerg.ca> Message-ID: Greg Ward wrote: > On 02 December 2002, Just van Rossum said: > > I've been doing some thinking about the import mechanism, too, and my > > current gripe is about the __import__ hook: I find it utterly useless > > that the implementer of the hook is supposed to deal with > > sys.modules. What I wish for is a hook that only gets invoked if the > > module isn't already in sys.modules. > > I can see where you're coming from, but one disadvantage to your scheme > is that it would not be possible to override .py/.pyc/.pyo files -- they > would always be imported in preference to any other scheme. This could > well be a feature. ;-) I don't follow. In my scheme (which in the meantime I've partially implemented) any first import will be done by an import hook on sys.import_hooks, so you have full control. If you mean that you can't hook imports that are already in sys.modules: yes, that's a feature ;-). On the other hand, the existing __import__ hook will not change, so you can still do it using __import__. Look at it this way: the default __import__ implements the new scheme; it's the low level hook. For 99.9% of import hook needs you wouldn't touch it, though. Here's the protocol that my current implementation uses: - sys.import_hooks is a list of hooks - a hook is a callable object taking two arguments: [sub]modulename and path. Path is either the parents package's __path__ or None. - a hook either returns None ("module not found") or a two-tuple: (loadfunc, cookie). loadfunc is a callable object that will be called with two arguments: the fullname and "cookie". The cookie is just an arbitrary object, private to the hook. The loaderfunc must return the imported module. - (Currently it expects that the loaderfunc will insert the module in sys.path, I'm not sure I like that. Actually, I am sure I don't like that , but changing that would require more changes to import.c that I'd like. I'll have to investigate.) It plugs neatly into import_submodule(), right now it's a tiny patch. Just From tim.one@comcast.net Mon Dec 2 18:35:58 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 02 Dec 2002 13:35:58 -0500 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: Message-ID: [Tim] >> Yes. On Windows it wants a handle, because that's what spawn*() >> returns on Windows. But, again, Windows processes aren't intended >> to be killed externally. [martin@v.loewis.de] > That sounds like FUD. Argue w/ Microsoft -- you've read their docs. > Why is it that processes aren't intended to be killed? Because the > assassin can't know what state the process is in, so the process > may not complete correctly? It's easy to find long articles about the dangers on the net. Here's one I don't think has been posted before: http://tinyurl.com/35o6 > This is a general problem with killing, not specific to Windows, and > it never stopped a killer. > > Or can you crash the operating system or the killer process by > terminating some other process? It's certainly possible to crash Win9x by killing processes, and easy to hang the killer process (indeed, I have four shareware "process killers" on my home box, as no single one of them is able to kill everything -- sometimes I have a hung process, and three hung process killers trying to nuke it! Win98SE generally becomes too unstable to continue running at that point.). >> Why do you need to kill a process externally? For example, why >> can't you make "please stop now" a part of the protocol, so that a >> process can terminate itself gracefully when told to? > Because the process did not respond to the protocol. Should I kill it > anyway and send a bug report to Microsoft? The questions were addressed to Skip about his specific app. It sounded to me like he wanted to kill() routinely, instead of pursuing a clean shutdown. The TerminateProcess function is used to unconditionally cause a process to exit. Use it only in extreme circumstances. The state of global data maintained by dynamic-link libraries (DLLs) may be compromised if TerminateProcess is used rather than ExitProcess. because TerminateProcess doesn't notify attached DLLs of process termination -- it's not solely the killed process's state that can get hosed. From tim.one@comcast.net Mon Dec 2 18:42:35 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 02 Dec 2002 13:42:35 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime obj_delta.c,1.14,1.15 test_both.py,1.15,1.16 In-Reply-To: <2mlm38jp75.fsf@starship.python.net> Message-ID: [Michael Hudson] > You need to arrange for a module name to appear in timedelta's > tp_name: Ya, I figured that out. The next surprise was the need to create a __safe_for_unpickling__ attr in the type's dict. This was confusing because none of it was necessary for the Python implementation of this stuff, and examples of the necessity for, and mechanics of, __reduce__, were hard to find. From gward@python.net Mon Dec 2 19:05:59 2002 From: gward@python.net (Greg Ward) Date: Mon, 2 Dec 2002 14:05:59 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: References: <20021202170609.GA5209@cthulhu.gerg.ca> Message-ID: <20021202190559.GA5342@cthulhu.gerg.ca> On 02 December 2002, Just van Rossum said: > I don't follow. In my scheme (which in the meantime I've partially > implemented) any first import will be done by an import hook on > sys.import_hooks, so you have full control. Ahh! I get it, sorry. Makes sense now. > If you mean that you can't > hook imports that are already in sys.modules: yes, that's a feature Agreed. Once something has been imported, the import hooks should not be bothered with it again. > - sys.import_hooks is a list of hooks > - a hook is a callable object taking two arguments: > [sub]modulename and path. Path is either the parents package's > __path__ or None. > - a hook either returns None ("module not found") or a two-tuple: > (loadfunc, cookie). loadfunc is a callable object that will Sounds sensible, but please s/cookie/data/ (or similar). > be called with two arguments: the fullname and "cookie". The > cookie is just an arbitrary object, private to the hook. The > loaderfunc must return the imported module. > - (Currently it expects that the loaderfunc will insert the module > in sys.path, I'm not sure I like that. Actually, I am sure I ^^^^ You really meant sys.module, right? > don't like that , but changing that would require more > changes to import.c that I'd like. I'll have to investigate.) I agree: if an import hook is not responsible for checking sys.modules before doing an import, it should not be responsible for updating sys.modules after an import. Greg -- Greg Ward http://www.gerg.ca/ Clarke's Law: Any sufficiently advanced technology is indistinguishable from magic. From just@letterror.com Mon Dec 2 19:19:22 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 2 Dec 2002 20:19:22 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) Message-ID: Greg Ward wrote: > > - a hook either returns None ("module not found") or a two-tuple: > > (loadfunc, cookie). loadfunc is a callable object that will > > Sounds sensible, but please s/cookie/data/ (or similar). Ok. > > be called with two arguments: the fullname and "cookie". The > > cookie is just an arbitrary object, private to the hook. The > > loaderfunc must return the imported module. > > - (Currently it expects that the loaderfunc will insert the module > > in sys.path, I'm not sure I like that. Actually, I am sure I > ^^^^ > > You really meant sys.module, right? sys.modules actually ;-) > > don't like that , but changing that would require more > > changes to import.c that I'd like. I'll have to investigate.) > > I agree: if an import hook is not responsible for checking sys.modules > before doing an import, it should not be responsible for updating > sys.modules after an import. I hadn't even looked at it that way, but yeah, that's an excellent rationale... But: they _may_ do so if they desire; a module is even allowed to stuff a whole different object into sys.modules instead of itself, so I'll have to be careful. Just From jeremy@alum.mit.edu Mon Dec 2 21:41:21 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Mon, 2 Dec 2002 16:41:21 -0500 Subject: [Python-Dev] RE: [Python-checkins] python/nondist/sandbox/datetime obj_delta.c,1.14,1.15 test_both.py,1.15,1.16 In-Reply-To: References: <15851.51138.218721.353223@grendel.zope.com> Message-ID: <15851.54145.324810.106619@slothrop.zope.com> >>>>> "TP" == Tim Peters writes: TP> I'd rather find a way to avoid __reduce__ entirely, though! Me, too! TP> The Python implementation of these things didn't need it, and in TP> the date and datetime cases it's creating bigger pickles than it TP> should -- __getstate__ and __setstate__ already did all that was TP> necessary, and no more than that. Supplying an argument tuple TP> for __reduce__'s benefit loses either way: I either put the real TP> date/datetime arguments there, but then the pickle is of a big TP> tuple rather than of a tiny string. Or I put a dummy argument TP> tuple there and also include the tiny string for __setstate__, TP> but these constructors require at least 3 arguments so that the TP> "dummy argument tuple" consumes substantial space of its own. TP> So, as it stands, ignoring the new-style-class administrative TP> pickle bloat in the Python implementation, the *guts* of the TP> pickles produced by the Python implementation are substantially TP> smaller than those produced by the C implementation. The __reduce__() approach seems to be (either or both) overly complex and underdocumented. It's a real shame that something simple like __getstate__() and __setstate__() can't be made to work for new-style classes. If I recall correctly, the problem is that there is no way to distinguish a user-defined new-style class from a builtin type from a user-defined subclass of a builtin type. As a result, there's no way for pickle to decide if it should be looking for __getstate__() or invoking the complicated machinery that allows subclasses of builtin types to be pickleable. Another victim of unification. > I figured timedelta *was* "a class", although now I guess that, in > this context, it's not "a class" but "a type". That's what you get > when you unify the concepts . Jerem From dave@boost-consulting.com Mon Dec 2 22:33:25 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 02 Dec 2002 17:33:25 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <003001c298d7$550e8240$0201010a@wiktor> ("Wiktor Sadowski"'s message of "Sun, 1 Dec 2002 02:16:52 +0100") References: <003001c298d7$550e8240$0201010a@wiktor> Message-ID: "Wiktor Sadowski" writes: > I think Python should have a more universal import mechanism , which could be done > by adding a new C_CUSTOM type to the definitions for dynamic loading (importdl.h) , > a new struct _customtab to import.h and a new function to Python API (import.c). > Then any archive,database,internet import architecture could be implemented > in C extensions without messing with Python core. As long as we're discussing changes to the import mechanism, I'll mention a facility that some of my users want. I apologize in advance for not knowing whether the proposed patch covers this ability or not. My users want extension modules to be able to control how their attributes are imported when using "import *". In particular, Boost.Python includes a kind of multimethod facility, and some of my users would like it if from bpl_extension import * caused multimethod definitions in bpl_extension to be /merged/ into the current module, rather than replacing any multimethod bindings with the same name in the current module. -Dave -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From martin@v.loewis.de Mon Dec 2 22:47:09 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 02 Dec 2002 23:47:09 +0100 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: References: Message-ID: Tim Peters writes: > > That sounds like FUD. > > Argue w/ Microsoft -- you've read their docs. Apparently not the ones you've read. The documentation of TerminateProcess does not include any warning, except for the recommendation to only use it extreme circumstances, and that the global state maintained by DLLs may be compromised. Please correct me if I'm wrong, but on NT+, there isn't any global state maintained by DLLs that could get corrupted. I interpret the warning as suggesting that "normal" shutdown procedures should be used if possible. > It's easy to find long articles about the dangers on the net. Here's one I > don't think has been posted before: > > http://tinyurl.com/35o6 It mentions a number of methods to find the window, then explains that sending WM_CLOSE may fail, to finally explain that TerminateProcess should always work, and that you may lose data by using it. This is exactly the same as on Unix: If you kill a process that was just saving a file, the file may get corrupted. If there were unsaved data, they are lost. This is no reason not to use TerminateProcess. > It's certainly possible to crash Win9x by killing processes, and > easy to hang the killer process (indeed, I have four shareware > "process killers" on my home box, as no single one of them is able > to kill everything -- sometimes I have a hung process, and three > hung process killers trying to nuke it! That sounds like a bug in the killers. TerminateProcess returns immediately, according to the documentation. Now, the process may not terminate if it is blocked in the kernel (the same issue exists on Unix, leading to zombie processes). If the killer then invokes WaitForProcessClose, it is not surprising if it hangs. Your observations suggest "Don't use TerminateProcess on Win9x". I would extend this warning to "Don't use Win9x" :-) > because TerminateProcess doesn't notify attached DLLs of process > termination -- it's not solely the killed process's state that can get > hosed. Can you explain what the "global state" of a DLL is? It may have global state on Win9x, and be "attached" to a process - but I understand there is nothing "global" on NT+; the code gets mapped into the process (which gets cleaned up when the process is terminated), and there are no shared data. Regards, Martin From greg@cosc.canterbury.ac.nz Mon Dec 2 22:58:36 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 03 Dec 2002 11:58:36 +1300 (NZDT) Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: Message-ID: <200212022258.gB2Mwao28701@kuku.cosc.canterbury.ac.nz> > Because the assassin can't know what state the process is in, > so the process may not complete correctly? > > This is a general problem with killing, not specific to Windows, and > it never stopped a killer. Psychotic Killer: [wielding knife and pulling back shower curtain] HA! Prepare to die! Female Programmer: No, wait! You can't kill me now, you don't know what state I'm in! From tim.one@comcast.net Mon Dec 2 22:56:05 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 02 Dec 2002 17:56:05 -0500 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: Message-ID: Martin, I'm not interested in arguing about this. If you want to implement os.kill() for Windows, go ahead. I only ask that the docs repeat the vendor's warnings against using it. It's fine for adult use, where "adult" means you know you're courting trouble and are prepared to accept the consequences. From greg@cosc.canterbury.ac.nz Mon Dec 2 23:04:02 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 03 Dec 2002 12:04:02 +1300 (NZDT) Subject: [Python-Dev] Mac OSX issues In-Reply-To: <200212021030.gB2AUfR24830@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212022304.gB2N42Z28704@kuku.cosc.canterbury.ac.nz> Guido: > Do you know if setting the stack limit actually allocates that much > memory for the stack in the process in Mac OSX, or does it only > reserve VM space (like on traditional Unix)? Strictly speaking, in traditional Unix it doesn't do either, it just sets a limit on how big the stack can grow. Actually allocating stack pages is done on demand. I expect this is what MacOSX does. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From mhammond@skippinet.com.au Mon Dec 2 23:47:35 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 3 Dec 2002 10:47:35 +1100 Subject: [Python-Dev] ssl for Windows Message-ID: I would like to check in a patch for http://www.python.org/sf/576711, which adds the _ssl module to Windows builds. The only slightly controversial thing is how the build process operates. I have quoted the relevant part of PCbuild/readme.txt below. test_socket_ssl passes once this is built, and at least 2 other people have reported build success. Are there any objections to the scheme? Should I check it in? Mark. _ssl Python wrapper for the secure sockets library. Get the latest source code for OpenSSL from http://www.openssl.org Unpack into the "dist" directory, retaining the folder name from the archive - for example, the latest stable OpenSSL will install as dist/openssl-0.9.6g You can (theoretically) use any version of OpenSSL you like - the build process will automatically select the latest version. You must also install ActivePerl from http://www.activestate.com/Products/ActivePerl/ as this is used by the OpenSSL build process. Complain to them The MSVC project simply invokes PCBuild/build_ssl.py to perform the build. This Python script locates and builds your OpenSSL installation, then invokes a simple makefile to build the final .pyd. build_ssl.py attempts to catch the most common errors (such as not being able to find OpenSSL sources, or not being able to find a Perl that works with OpenSSL) and give a reasonable error message. If you have a problem that doesn't seem to be handled correctly (eg, you know you have ActivePerl but we can't find it), please take a peek at build_ssl.py and suggest patches. Note that build_ssl.py should be able to be run directly from the command-line. build_ssl.py/MSVC isn't clever enough to clean OpenSSL - you must do this by hand. From just@letterror.com Tue Dec 3 00:29:28 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 01:29:28 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Message-ID: --41333375-49397760-3247867773=:9006 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit Folks, I've attached a proof-of-concept patch to import.c that implements the sys.import_hooks idea I outlined earlier. Here's an updated description. - sys.import_hooks is a list of hooks. - A hook is a callable object taking two arguments: 'modulename' and 'path'. 'path' is either the parents package's __path__ or None, 'modulename' is *always* a "simple" name: no dots. - Hooks are never called for modules already in sys.modules. - A hook either returns None ("module not found") or a two-tuple: (loadfunc, data). loadfunc is a callable object that will be called with two arguments: the 'fullname' and 'data'. 'data' is just an arbitrary object, private to the hook. The loaderfunc must return the imported module. It _may_ add the module to sys.modules, but it doesn't need to. - A hook is not supposed to fail: it should return None or a (loader, data) tuple. If a hook _does_ fail with an exception, the import is aborted and subsequent hooks won't get called. The exception will be propagated to the caller. (XXX this is perhaps brittle; maybe errors need to be ignored by default, or a warning should be issued instead.) sys.import_hooks currently gets filled with a default hook: imp.builtin_import_hook. This is simply a wrapper around find_module() and friends. Perhaps it's an idea to split it into discrete hooks for builtin, frozen, and file system imports. This can be done later. An alternative to the current default hook would be to keep sys.import_hooks empty by default and fall back to find_module/load_module if no hook managed to find/load the module. Pro: this would allow for some optimizations, as the hook mechanism would be bypassed (and therefore less code in import.c). Con: it would then be awkward to completely disable the builtin import mechanism, it's less transparent, less elegant. (Ok, I thought a bit more about this: I don't like it at all. All imports should be done by a hook on sys.import_hooks. If sys.import_hooks is empty, _any_ new import will fail.) Rationale: - Make writing import hooks simpler; this new scheme avoids much of the complexity involved in writing a traditional __import__ hook. Real rationale: - Allow the "Import from Zip archive" patch to be rewritten as an external module, avoiding the huge mess it is right now, as it touches more or less all of import.c and then some. Paul Moore is doing an admirable job to get it working and cleaned up, but I honestly don't see this getting into the core: it's such a huge and complex patch that it's almost impossible to properly review. It's a maintainance nightmare waiting to happen. And let's face it, import.c isn't all that maintainable as it is... Longer term rationale: - It would allow us to refactor the builtin import mechanism(s) by writing specialized hooks, replacing the current monolithic find_module/load_module code. Con: It is slower for first-time imports. If you care, go ahead and measure ;-) Comments? Just --41333375-49397760-3247867773=:9006 Content-Type: application/octet-stream; Name="newimporthook.patch"; X-Mac-Type="54455854"; X-Mac-Creator="522A6368" Content-Transfer-Encoding: base64 Content-Disposition: attachment; Filename="newimporthook.patch" SW5kZXg6IFB5dGhvbi9pbXBvcnQuYwo9PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09ClJDUyBm aWxlOiAvY3Zzcm9vdC9weXRob24vcHl0aG9uL2Rpc3Qvc3JjL1B5dGhvbi9pbXBv cnQuYyx2CnJldHJpZXZpbmcgcmV2aXNpb24gMi4yMTEKZGlmZiAtYyAtcjIuMjEx IGltcG9ydC5jCioqKiBQeXRob24vaW1wb3J0LmMJMjkgTm92IDIwMDIgMjA6NDc6 NDAgLTAwMDAJMi4yMTEKLS0tIFB5dGhvbi9pbXBvcnQuYwkyIERlYyAyMDAyIDIz OjQwOjUwIC0wMDAwCioqKioqKioqKioqKioqKgoqKiogMTA1LDExMCAqKioqCi0t LSAxMDUsMTI4IC0tLS0KICB9OwogICNlbmRpZgogIAorIC8qIGltcG9ydF9ob29r cyBzdHVmZiwgcHJvYmFibHkgbmVlZHMgcmVvcmdhbml6YXRpb24gKi8KKyAvKiBm b3J3YXJkIGRlY2xzICovCisgc3RhdGljIEZJTEUgKmdldF9maWxlKGNoYXIgKnBh dGhuYW1lLCBQeU9iamVjdCAqZm9iLCBjaGFyICptb2RlKTsKKyBzdGF0aWMgUHlP YmplY3QgKmNhbGxfZmluZF9tb2R1bGUoY2hhciAqbmFtZSwgUHlPYmplY3QgKnBh dGgpOworIAorIC8qICJtYW51YWwiIGNvbnN0cnVjdGlvbiBvZiB0d28gQ0Z1bmN0 aW9uIG9iamVjdHM7IGNhbid0IHVzZSB0aGUKKyAgICByZWd1bGFyIG1lY2hhbmlz bSBhcyB3ZSBuZWVkIHRoZXNlIGZ1bmN0aW9uIG9iamVjdHMgZXZlbiBpZiB0aGUK KyAgICBpbXAgbW9kdWxlIGlzbid0IGltcG9ydGVkLiAqLworIHN0YXRpYyBQeU9i amVjdCAqaW1wX2J1aWx0aW5faW1wb3J0X2hvb2soUHlPYmplY3QgKnNlbGYsIFB5 T2JqZWN0ICphcmdzKTsKKyBzdGF0aWMgUHlNZXRob2REZWYgYnVpbHRpbl9pbXBv cnRfaG9va19kZWYgPSAKKyAJeyJidWlsdGluX2ltcG9ydF9ob29rIiwgaW1wX2J1 aWx0aW5faW1wb3J0X2hvb2ssIE1FVEhfVkFSQVJHUywgIlhYWCJ9OworIHN0YXRp YyBQeU9iamVjdCAqYnVpbHRpbl9pbXBvcnRfaG9va19mdW5jID0gTlVMTDsKKyAK KyBzdGF0aWMgUHlPYmplY3QgKmltcF9idWlsdGluX2xvYWRlcihQeU9iamVjdCAq c2VsZiwgUHlPYmplY3QgKmFyZ3MpOworIHN0YXRpYyBQeU1ldGhvZERlZiBidWls dGluX2xvYWRlcl9kZWYgPSAKKyAJeyJidWlsdGluX2xvYWRlciIsIGltcF9idWls dGluX2xvYWRlciwgTUVUSF9WQVJBUkdTLCAiWFhYIn07Cisgc3RhdGljIFB5T2Jq ZWN0ICpidWlsdGluX2xvYWRlcl9mdW5jID0gTlVMTDsKKyAKICAvKiBJbml0aWFs aXplIHRoaW5ncyAqLwogIAogIHZvaWQKKioqKioqKioqKioqKioqCioqKiAxMTQs MTE5ICoqKioKLS0tIDEzMiwxMzggLS0tLQogIAlzdHJ1Y3QgZmlsZWRlc2NyICpm aWxldGFiOwogIAlpbnQgY291bnREID0gMDsKICAJaW50IGNvdW50UyA9IDA7Cisg CVB5T2JqZWN0ICppbXBvcnRfaG9va3M7CiAgCiAgCS8qIHByZXBhcmUgX1B5SW1w b3J0X0ZpbGV0YWI6IGNvcHkgZW50cmllcyBmcm9tCiAgCSAgIF9QeUltcG9ydF9E eW5Mb2FkRmlsZXRhYiBhbmQgX1B5SW1wb3J0X1N0YW5kYXJkRmlsZXRhYi4KKioq KioqKioqKioqKioqCioqKiAxNTAsMTU1ICoqKioKLS0tIDE2OSwxODUgLS0tLQog IAkJICAgY29kZSBjcmVhdGVkIGluIG5vcm1hbCBvcGVyYXRpb24gbW9kZS4gKi8K ICAJCXB5Y19tYWdpYyA9IE1BR0lDICsgMTsKICAJfQorIAorIAkvKiBTZXR1cCBz eXMuaW1wb3J0X2hvb2tzOyBjcmVhdGUgZnVuY3Rpb24gb2JqZWN0cyBmb3IgdGhl CisgCSAgIGRlZmF1bHQgaG9vayBhbmQgbWF0Y2hpbmcgbG9hZGVyIGZ1bmN0aW9u cy4gKi8KKyAJYnVpbHRpbl9pbXBvcnRfaG9va19mdW5jID0gUHlDRnVuY3Rpb25f TmV3KCZidWlsdGluX2ltcG9ydF9ob29rX2RlZiwKKyAJCQkJCQkgICBOVUxMKTsK KyAJYnVpbHRpbl9sb2FkZXJfZnVuYyA9IFB5Q0Z1bmN0aW9uX05ldygmYnVpbHRp bl9sb2FkZXJfZGVmLCBOVUxMKTsKKyAJaW1wb3J0X2hvb2tzID0gUHlfQnVpbGRW YWx1ZSgiW09dIiwgYnVpbHRpbl9pbXBvcnRfaG9va19mdW5jKTsKKyAJaWYgKGlt cG9ydF9ob29rcyAhPSBOVUxMKSB7CisgCQlQeVN5c19TZXRPYmplY3QoImltcG9y dF9ob29rcyIsIGltcG9ydF9ob29rcyk7CisgCQlQeV9ERUNSRUYoaW1wb3J0X2hv b2tzKTsKKyAJfQogIH0KICAKICB2b2lkCioqKioqKioqKioqKioqKgoqKiogMjQ2 LDI1MiAqKioqCiAgCSJwYXRoIiwgImFyZ3YiLCAicHMxIiwgInBzMiIsICJleGl0 ZnVuYyIsCiAgCSJleGNfdHlwZSIsICJleGNfdmFsdWUiLCAiZXhjX3RyYWNlYmFj ayIsCiAgCSJsYXN0X3R5cGUiLCAibGFzdF92YWx1ZSIsICJsYXN0X3RyYWNlYmFj ayIsCiEgCU5VTEwKICB9OwogIAogIHN0YXRpYyBjaGFyKiBzeXNfZmlsZXNbXSA9 IHsKLS0tIDI3NiwyODIgLS0tLQogIAkicGF0aCIsICJhcmd2IiwgInBzMSIsICJw czIiLCAiZXhpdGZ1bmMiLAogIAkiZXhjX3R5cGUiLCAiZXhjX3ZhbHVlIiwgImV4 Y190cmFjZWJhY2siLAogIAkibGFzdF90eXBlIiwgImxhc3RfdmFsdWUiLCAibGFz dF90cmFjZWJhY2siLAohIAkiaW1wb3J0X2hvb2tzIiwgTlVMTAogIH07CiAgCiAg c3RhdGljIGNoYXIqIHN5c19maWxlc1tdID0gewoqKioqKioqKioqKioqKioKKioq IDE5NjEsMTk2NiAqKioqCi0tLSAxOTkxLDIxMTAgLS0tLQogIH0KICAKICBzdGF0 aWMgUHlPYmplY3QgKgorIGltcF9idWlsdGluX2ltcG9ydF9ob29rKFB5T2JqZWN0 ICpzZWxmLCBQeU9iamVjdCAqYXJncykgeworIAljaGFyICpuYW1lOworIAlQeU9i amVjdCAqcGF0aCwgKnJlczsKKyAJCisgCWlmICghUHlBcmdfUGFyc2VUdXBsZShh cmdzLCAic08iLCAmbmFtZSwgJnBhdGgpKQorIAkJcmV0dXJuIE5VTEw7CisgCWlm IChwYXRoID09IFB5X05vbmUpCisgCQlwYXRoID0gTlVMTDsKKyAJcmVzID0gY2Fs bF9maW5kX21vZHVsZShuYW1lLCBwYXRoKTsKKyAJaWYgKHJlcyA9PSBOVUxMKSB7 CisgCQlpZiAoIVB5RXJyX0V4Y2VwdGlvbk1hdGNoZXMoUHlFeGNfSW1wb3J0RXJy b3IpKQorIAkJCXJldHVybiBOVUxMOworIAkJUHlFcnJfQ2xlYXIoKTsKKyAJCVB5 X0lOQ1JFRihQeV9Ob25lKTsKKyAJCXJldHVybiBQeV9Ob25lOworIAl9CisgCXJl dHVybiBQeV9CdWlsZFZhbHVlKCIoT08pIiwgYnVpbHRpbl9sb2FkZXJfZnVuYywg cmVzKTsKKyB9CisgCisgc3RhdGljIFB5T2JqZWN0ICoKKyBpbXBfYnVpbHRpbl9s b2FkZXIoUHlPYmplY3QgKnNlbGYsIFB5T2JqZWN0ICphcmdzKSB7CisgCWNoYXIg KmZ1bGxuYW1lLCAqcGF0aG5hbWUsICpzdWZmaXgsICptb2RlOworIAlpbnQgdHlw ZTsKKyAJUHlPYmplY3QgKmRhdGEsICpmb2IsICptb2Q7CisgCUZJTEUgKmZwOwor IAorIAlpZiAoIVB5QXJnX1BhcnNlVHVwbGUoYXJncywgInNPOmJ1aWx0aW5fbG9h ZGVyIiwKKyAJCQkgICAgICAmZnVsbG5hbWUsICZkYXRhKSkKKyAJCXJldHVybiBO VUxMOworIAlpZiAoIVB5QXJnX1BhcnNlVHVwbGUoZGF0YSwgIk9zKHNzaSk6YnVp bHRpbl9sb2FkZXIiLAorIAkJCSAgICAgICZmb2IsICZwYXRobmFtZSwgJnN1ZmZp eCwKKyAJCQkgICAgICAmbW9kZSwgJnR5cGUpKQorIAkJcmV0dXJuIE5VTEw7Cisg CS8qIFhYWCBjaGVjayBtb2RlLCBhcyBpbiBsb2FkX21vZHVsZSwgYnV0IHRoZSBj b2RlIG5lZWRzIHJlZmFjdG9yaW5nICovCisgCWlmIChmb2IgPT0gUHlfTm9uZSkK KyAJCWZwID0gTlVMTDsKKyAJZWxzZSB7CisgCQlpZiAoIVB5RmlsZV9DaGVjayhm b2IpKSB7CisgCQkJUHlFcnJfU2V0U3RyaW5nKFB5RXhjX1ZhbHVlRXJyb3IsCisg CQkJCSJsb2FkX21vZHVsZSBhcmcjMiBzaG91bGQgYmUgYSBmaWxlIG9yIE5vbmUi KTsKKyAJCQlyZXR1cm4gTlVMTDsKKyAJCX0KKyAJCWZwID0gZ2V0X2ZpbGUocGF0 aG5hbWUsIGZvYiwgbW9kZSk7CisgCQlpZiAoZnAgPT0gTlVMTCkKKyAJCQlyZXR1 cm4gTlVMTDsKKyAJfQorIAltb2QgPSBsb2FkX21vZHVsZShmdWxsbmFtZSwgZnAs IHBhdGhuYW1lLCB0eXBlKTsKKyAJaWYgKGZwKQorIAkJZmNsb3NlKGZwKTsKKyAJ cmV0dXJuIG1vZDsKKyB9CisgCisgc3RhdGljIFB5T2JqZWN0ICoKKyBmaW5kX21v ZHVsZTIoY2hhciAqc3VibmFtZSwgUHlPYmplY3QgKnBhdGgpCisgeworIAlpbnQg aSwgbmhvb2tzOworIAlQeU9iamVjdCAqYXJncywgKnJlcyA9IE5VTEw7CisgCVB5 T2JqZWN0ICppbXBvcnRfaG9va3MgPSBQeVN5c19HZXRPYmplY3QoImltcG9ydF9o b29rcyIpOworIAorIAlpZiAoaW1wb3J0X2hvb2tzID09IE5VTEwgfHwgIVB5TGlz dF9DaGVjayhpbXBvcnRfaG9va3MpKSB7CisgCQlQeUVycl9TZXRTdHJpbmcoUHlF eGNfSW1wb3J0RXJyb3IsCisgCQkJCSJzeXMuaW1wb3J0X2hvb2tzIG11c3QgYmUg YSBsaXN0IG9mIGltcG9ydCBob29rcyIpOworIAkJcmV0dXJuIE5VTEw7CisgCX0K KyAJaWYgKHBhdGggPT0gTlVMTCkKKyAJCXBhdGggPSBQeV9Ob25lOworIAlhcmdz ID0gUHlfQnVpbGRWYWx1ZSgiKHNPKSIsIHN1Ym5hbWUsIHBhdGgpOworIAlpZiAo YXJncyA9PSBOVUxMKQorIAkJcmV0dXJuIE5VTEw7CisgCW5ob29rcyA9IFB5TGlz dF9TaXplKGltcG9ydF9ob29rcyk7CisgCWZvciAoaSA9IDA7IGkgPCBuaG9va3M7 IGkrKykgeworIAkJUHlPYmplY3QgKmhvb2sgPSBQeUxpc3RfR2V0SXRlbShpbXBv cnRfaG9va3MsIGkpOworIAkJaWYgKGhvb2sgPT0gTlVMTCkKKyAJCQlnb3RvIGVy cm9yOworIAkJcmVzID0gUHlPYmplY3RfQ2FsbChob29rLCBhcmdzLCBOVUxMKTsK KyAJCWlmIChyZXMgPT0gTlVMTCkKKyAJCQlnb3RvIGVycm9yOworIAkJaWYgKHJl cyAhPSBQeV9Ob25lKQorIAkJCWJyZWFrOworIAkJUHlfREVDUkVGKHJlcyk7Cisg CQlyZXMgPSBOVUxMOworIAkJY29udGludWU7CisgCX0KKyAJaWYgKHJlcyA9PSBO VUxMKSB7CisgCQlQeV9JTkNSRUYoUHlfTm9uZSk7CisgCQlyZXMgPSBQeV9Ob25l OworIAl9CisgZXJyb3I6CisgCVB5X0RFQ1JFRihhcmdzKTsKKyAJcmV0dXJuIHJl czsKKyB9CisgCisgc3RhdGljIFB5T2JqZWN0ICoKKyBsb2FkX21vZHVsZTIoY2hh ciAqZnVsbG5hbWUsIFB5T2JqZWN0ICpmaW5kZXJfcmVzdWx0KQorIHsKKyAJUHlP YmplY3QgKmxvYWRlcmZ1bmMsICpkYXRhLCAqYXJncywgKm1vZDsKKyAJCisgCWlm ICghUHlBcmdfUGFyc2VUdXBsZShmaW5kZXJfcmVzdWx0LCAiT08iLCAmbG9hZGVy ZnVuYywgJmRhdGEpKQorIAkJcmV0dXJuIE5VTEw7CisgCWFyZ3MgPSBQeV9CdWls ZFZhbHVlKCIoc08pIiwgZnVsbG5hbWUsIGRhdGEpOworIAlpZiAoYXJncyA9PSBO VUxMKQorIAkJcmV0dXJuIE5VTEw7CisgCW1vZCA9IFB5T2JqZWN0X0NhbGwobG9h ZGVyZnVuYywgYXJncywgTlVMTCk7CisgCVB5X0RFQ1JFRihhcmdzKTsKKyAJaWYg KG1vZCAhPSBOVUxMICYmIG1vZCAhPSBQeV9Ob25lKSB7CisgCQkvKiBpbnNlcnQg bW9kdWxlIGluIHN5cy5tb2R1bGVzLCBpZiBpdCBpc24ndCB0aGVyZSB5ZXQgKi8K KyAJCVB5T2JqZWN0ICptb2R1bGVzID0gUHlJbXBvcnRfR2V0TW9kdWxlRGljdCgp OworIAkJaWYgKFB5RGljdF9HZXRJdGVtU3RyaW5nKG1vZHVsZXMsIGZ1bGxuYW1l KSA9PSBOVUxMKQorIAkJCVB5RGljdF9TZXRJdGVtU3RyaW5nKG1vZHVsZXMsIGZ1 bGxuYW1lLCBtb2QpOworIAl9CisgCXJldHVybiBtb2Q7CisgfQorIAorIHN0YXRp YyBQeU9iamVjdCAqCiAgaW1wb3J0X3N1Ym1vZHVsZShQeU9iamVjdCAqbW9kLCBj aGFyICpzdWJuYW1lLCBjaGFyICpmdWxsbmFtZSkKICB7CiAgCVB5T2JqZWN0ICpt b2R1bGVzID0gUHlJbXBvcnRfR2V0TW9kdWxlRGljdCgpOwoqKioqKioqKioqKioq KioKKioqIDE5NzYsMTk4NCAqKioqCiAgCX0KICAJZWxzZSB7CiAgCQlQeU9iamVj dCAqcGF0aDsKLSAJCWNoYXIgYnVmW01BWFBBVEhMRU4rMV07Ci0gCQlzdHJ1Y3Qg ZmlsZWRlc2NyICpmZHA7Ci0gCQlGSUxFICpmcCA9IE5VTEw7CiAgCiAgCQlpZiAo bW9kID09IFB5X05vbmUpCiAgCQkJcGF0aCA9IE5VTEw7Ci0tLSAyMTIwLDIxMjUg LS0tLQoqKioqKioqKioqKioqKioKKioqIDE5OTEsMjAwOSAqKioqCiAgCQkJfQog IAkJfQogIAohIAkJYnVmWzBdID0gJ1wwJzsKISAJCWZkcCA9IGZpbmRfbW9kdWxl KHN1Ym5hbWUsIHBhdGgsIGJ1ZiwgTUFYUEFUSExFTisxLCAmZnApOwohIAkJUHlf WERFQ1JFRihwYXRoKTsKISAJCWlmIChmZHAgPT0gTlVMTCkgewohIAkJCWlmICgh UHlFcnJfRXhjZXB0aW9uTWF0Y2hlcyhQeUV4Y19JbXBvcnRFcnJvcikpCiEgCQkJ CXJldHVybiBOVUxMOwohIAkJCVB5RXJyX0NsZWFyKCk7CiAgCQkJUHlfSU5DUkVG KFB5X05vbmUpOwogIAkJCXJldHVybiBQeV9Ob25lOwogIAkJfQohIAkJbSA9IGxv YWRfbW9kdWxlKGZ1bGxuYW1lLCBmcCwgYnVmLCBmZHAtPnR5cGUpOwohIAkJaWYg KGZwKQohIAkJCWZjbG9zZShmcCk7CiAgCQlpZiAobW9kICE9IFB5X05vbmUpIHsK ICAJCQkvKiBJcnJlc3BlY3RpdmUgb2YgdGhlIHN1Y2Nlc3Mgb2YgdGhpcyBsb2Fk LCBtYWtlIGEKICAJCQkgICByZWZlcmVuY2UgdG8gaXQgaW4gdGhlIHBhcmVudCBw YWNrYWdlIG1vZHVsZS4KLS0tIDIxMzIsMjE0OCAtLS0tCiAgCQkJfQogIAkJfQog IAohIAkJcmVzID0gZmluZF9tb2R1bGUyKHN1Ym5hbWUsIHBhdGgpOwohIAkJaWYg KHJlcyA9PSBOVUxMKQohIAkJCS8qIGZpbmRpbmcgcmFpc2VkIGFuIGVycm9yICov CiEgCQkJcmV0dXJuIE5VTEw7CiEgCQlpZiAocmVzID09IFB5X05vbmUpIHsKISAJ CQkvKiB0aGUgbW9kdWxlIHdhc24ndCBmb3VuZCAqLwogIAkJCVB5X0lOQ1JFRihQ eV9Ob25lKTsKICAJCQlyZXR1cm4gUHlfTm9uZTsKICAJCX0KISAJCW0gPSBsb2Fk X21vZHVsZTIoZnVsbG5hbWUsIHJlcyk7CiEgCiAgCQlpZiAobW9kICE9IFB5X05v bmUpIHsKICAJCQkvKiBJcnJlc3BlY3RpdmUgb2YgdGhlIHN1Y2Nlc3Mgb2YgdGhp cyBsb2FkLCBtYWtlIGEKICAJCQkgICByZWZlcmVuY2UgdG8gaXQgaW4gdGhlIHBh cmVudCBwYWNrYWdlIG1vZHVsZS4KKioqKioqKioqKioqKioqCioqKiAyNzI0LDI3 MjkgKioqKgotLS0gMjg2MywyODcyIC0tLS0KICAJaWYgKHNldGludChkLCAiUFlf RlJPWkVOIiwgUFlfRlJPWkVOKSA8IDApIGdvdG8gZmFpbHVyZTsKICAJaWYgKHNl dGludChkLCAiUFlfQ09ERVJFU09VUkNFIiwgUFlfQ09ERVJFU09VUkNFKSA8IDAp IGdvdG8gZmFpbHVyZTsKICAKKyAJaWYgKFB5RGljdF9TZXRJdGVtU3RyaW5nKGQs ICJidWlsdGluX2ltcG9ydF9ob29rIiwgYnVpbHRpbl9pbXBvcnRfaG9va19mdW5j KSA8IDApCisgCQlnb3RvIGZhaWx1cmU7CisgCWlmIChQeURpY3RfU2V0SXRlbVN0 cmluZyhkLCAiYnVpbHRpbl9sb2FkZXIiLCBidWlsdGluX2xvYWRlcl9mdW5jKSA8 IDApCisgCQlnb3RvIGZhaWx1cmU7CiAgICBmYWlsdXJlOgogIAk7CiAgfQo= --41333375-49397760-3247867773=:9006-- From skip@pobox.com Mon Dec 2 18:43:01 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 2 Dec 2002 12:43:01 -0600 Subject: [Python-Dev] Mac OSX issues In-Reply-To: <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15851.43445.443010.182362@montanaro.dyndns.org> Guido> I'm still seeing errors in test_socket; today (on a different Guido> network) I got this: Guido> test test_socket failed -- Traceback (most recent call last): Guido> File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes Guido> ip = socket.gethostbyname(hostname) Guido> gaierror: (7, 'No address associated with nodename') What does the hostname(1) command return? By default I think it generates some oddball thing like "Montanaro.local.". Apple calls it a Rendevous name. (I've no idea what that is.) The fix for me was recommended by another group: get a dyndns.org account and change the HOSTNAME line in /etc/hostconfig from HOSTNAME=-AUTOMATIC- to HOSTNAME=montanaro.dyndns.org Skip From skip@pobox.com Mon Dec 2 18:36:47 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 2 Dec 2002 12:36:47 -0600 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: References: <15850.37138.572612.258716@montanaro.dyndns.org> Message-ID: <15851.43071.25770.622477@montanaro.dyndns.org> >> In my environment I need to tunnel SB's pop3proxy app through an ssh >> connection. Tim> Why do you need to kill a process externally? For example, why Tim> can't you make "please stop now" a part of the protocol, so that a Tim> process can terminate itself gracefully when told to? I start the background ssh process from within pop3proxy, so it seems to me I ought to be the one to terminate it. As for protocol, I'm not aware of another way to terminate a running ssh other than to call kill(2) with a specified signal. That's more-or-less the Unix way. I don't have the luxury of modifying ssh to understand some application-specific protocol. Skip From bkline@rksystems.com Tue Dec 3 02:47:12 2002 From: bkline@rksystems.com (Bob Kline) Date: Mon, 2 Dec 2002 21:47:12 -0500 (EST) Subject: [Python-Dev] Getting python-bz2 into 2.3 In-Reply-To: <200211301531.gAUFV4T12368@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sat, 30 Nov 2002, Guido van Rossum wrote: > More testing is always welcome; if you could test-drive the > instructions in PCbuild/readme.txt for Tcl/Tk and Sleepycat Berkeley > DB on multiple platforms that would be super. The instructions for building Tcl and Tk worked as written with no problems (though I would have no idea where to begin with the XXX questions, such as "Should we compile with OPTS=threads?"). The project file for the _tkinter module also worked without any hitches, but I don't see a test suite (no surprise for a GUI-based module) and I have no clue about how to test it or even get some of my simple _tkinter program to run. I'm so far out of my depth here with respect to Tcl/Tk that I won't waste list bandwidth to pursue this any further. As for the Berkeley DB package, that too built without any difficulties. The instructions conclude: ============================== snip ============================== XXX We're actually linking against Release_static\libdb40s.lib. XXX This yields the following warnings: """ Compiling... _bsddb.c Linking... Creating library ./_bsddb.lib and object ./_bsddb.exp LINK : warning LNK4049: locally defined symbol "_malloc" imported LINK : warning LNK4049: locally defined symbol "_free" imported LINK : warning LNK4049: locally defined symbol "_fclose" imported LINK : warning LNK4049: locally defined symbol "_fopen" imported _bsddb.pyd - 0 error(s), 4 warning(s) """ XXX This isn't encouraging, but I don't know what to do about it. ============================== snip ============================== I did a little poking around, and found that the problem is that the static library against which the _bsddb package is built is compiled with the -MT option, which causes the multi-threaded runtime libraries to be statically linked into libdb40s.lib, hence the warnings. The warnings should probably be taken seriously, because I think they mean that we would end up with (for example) two sets of static heap control information. The instructions in PCBuild/readme.txt were actually a little confusing, because right before the part I just quoted above tells us to "[c]hoose configuration "db_buildall - Win32 Release", and build db_buildall.exe." That version uses the -MD compilation option, which uses the DLL version of the runtime libraries (which is what we want), but it creates a DLL for libdb40 instead of a library which can be statically linked into _bsddb.pyd. So it looks as there are two ways to eliminate the warnings above: 1. Use the DLL version of libdb40 and link to its import library. 2. Create a third configuration for the underlying Berkeley DB library, using the -MD compilation option, but building a statically linkable library instead of a DLL. I would have assumed that the second option was needed, because I assumed that Python wouldn't want to drag around lots of DLLs for the various packages that get installed, but then I noticed that the _tkinter package is doing just that, so I guess I'm not sure. If the second option is used, it should presumably be done with the cooperation of the folks behind the Berkeley DB library, so it wouldn't be necessary to recreate the third configuration by hand each time a new release of the library came out. Finally, I was reluctant to dig any further into this package because I read at the top of the instructions: "XXX The Sleepycat release we use will probably change before XXX 2.3a1." and I didn't know whether that referred to a version upgrade or a complete replacement of the underlying DB engine with some other library, which would make any further work throwaway. -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com From bac@OCF.Berkeley.EDU Tue Dec 3 03:51:11 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 2 Dec 2002 19:51:11 -0800 (PST) Subject: [Python-Dev] Mac OSX issues In-Reply-To: <15851.43445.443010.182362@montanaro.dyndns.org> References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> <15851.43445.443010.182362@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > What does the hostname(1) command return? By default I think it generates > some oddball thing like "Montanaro.local.". Apple calls it a Rendevous > name. (I've no idea what that is.) The fix for me was recommended by Rendevous is Apple's implementation of IEEE's zero-conf. The "oddball" name is what is used for the local DNS entries used by Rendevous since it stores info by name instead of IP. -Brett From tim.one@comcast.net Tue Dec 3 04:17:30 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 02 Dec 2002 23:17:30 -0500 Subject: [Python-Dev] ssl for Windows In-Reply-To: Message-ID: [Mark Hammond] > I would like to check in a patch for > http://www.python.org/sf/576711, which > adds the _ssl module to Windows builds. Good work, Mark. Don't wait for me -- it appears that most of my Windows Python work has become "spare time" activity. > The only slightly controversial thing is how the build process > operates. It's just a bit of a mess. I don't mind installing a current Perl, if that's what it takes. > I have quoted the relevant part of PCbuild/readme.txt below. > test_socket_ssl passes once this is built, and at least 2 other > people have reported build success. > > Are there any objections to the scheme? Should I check it in? Please do; that will force the issues in a helpful way. From skip@pobox.com Tue Dec 3 04:29:09 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 2 Dec 2002 22:29:09 -0600 Subject: [Python-Dev] extension module .o files wind up in the wrong place Message-ID: <15852.13077.577684.9372@montanaro.dyndns.org> I notice that when building Python in a subdirectory of the src dir that the .o files for extension modules are written in the wrong directory. Graphically, if my source tree looks like head/dist/src Makefile.pre.in /Modules /Python ... build Makefile /Modules /Python ... If I run cd build ../configure --prefix=/Users/skip/local make extension module .o files wind up in .../src/Modules instead of .../src/build/Modules. I don't have time to look into this right now. I only just noticed it while I was trying to do parallel builds in build.dbg and build.opt directories. The debug .o files got written first, so when the optimized make came along, there were already .o files, which had been compiled with --with-pydebug, got linked improperly. Skip From greg@cosc.canterbury.ac.nz Tue Dec 3 04:48:55 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 03 Dec 2002 17:48:55 +1300 (NZDT) Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: <15852.13077.577684.9372@montanaro.dyndns.org> Message-ID: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> Skip Montanaro : > extension module .o files wind up in .../src/Modules instead of > .../src/build/Modules. If this involves VPATH, I think I might know what's happening -- something similar happened to me recently. Have you by any chance previously done a non-VPATH build in the source tree? It seems that, when using a VPATH, any pathname referencing something in the build directory for which a corresponding file exists in the VPATH directory gets changed to refer to the VPATH directory. So you need to make sure there are *only* source files in the source tree before doing a build using a VPATH. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From skip@pobox.com Tue Dec 3 05:02:12 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 2 Dec 2002 23:02:12 -0600 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> References: <15852.13077.577684.9372@montanaro.dyndns.org> <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> Message-ID: <15852.15060.656223.659613@montanaro.dyndns.org> Greg> So you need to make sure there are *only* source files in the Greg> source tree before doing a build using a VPATH. Yup, "cvs up" followed by "make distclean", then mkdir build cd build ../configure --prefix=/Users/skip/local make Skip From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_0696_01C299CB.38DFD840 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable Just van Rossum wrote: >>Could you please post a normal diff of your patch? It looks very = interesting. Sorry, I haven't got time for cleaning up my modified Python C code and = building the diffs. >.I'm not entirely sure yet how my idea and yours could be integrated, = but I think > it's work a look. The concept is the same , there are differences in implementations. The implementation you have posted is much more pure Python oriented. I am using C structures and pointers to C functions (wherever possible) = instead of Python objects=20 (well PyObjects are still C structures) , so maybe it's a little faster = (maybe not)=20 Looks like I was wasting my time.;) Wiktor Sadowski art@wiktorsadowski.com ------=_NextPart_000_0696_01C299CB.38DFD840 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
Just van=20 Rossum  wrote:
>>Could you please post a normal diff of your patch? It looks = very=20 interesting.
Sorry, I haven't got time for cleaning up my modified = Python C=20 code and  building the diffs.
 
>.I'm not=20 entirely sure yet how my idea and yours could be integrated, but I = think
>=20 it's work a look.
The concept is the same , there are differences in = implementations.
The implementation you have posted is much more pure Python=20 oriented.
I am using C structures and pointers to C functions (wherever = possible)=20 instead of Python objects
(well PyObjects are still C structures) , so maybe = it's a=20 little faster (maybe not)
 
Looks like I was wasting my time.;)
 
Wiktor Sadowski
art@wiktorsadowski.com<= /DIV>
 
 
 
 
 ------=_NextPart_000_0696_01C299CB.38DFD840-- From tim.one@comcast.net Tue Dec 3 05:02:06 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 03 Dec 2002 00:02:06 -0500 Subject: [Python-Dev] Getting python-bz2 into 2.3 In-Reply-To: Message-ID: [Bob Kline, tries building a more-or-less complete Windows Python] > The instructions for building Tcl and Tk worked as written with no > problems (though I would have no idea where to begin with the XXX > questions, such as "Should we compile with OPTS=threads?"). That's OK, neither do I . We didn't before (8.3.2), and changing anything is risky. > The project file for the _tkinter module also worked without any > hitches, but I don't see a test suite (no surprise for a GUI-based > module) and I have no clue about how to test it or even get some of > my simple _tkinter program to run. I'm so far out of my depth here > with respect to Tcl/Tk that I won't waste list bandwidth to pursue > this any further. I don't know how to test it either. I install Python after building it, then try Tkinter._test() from a DOS-box Python shell, and try a bunch of stuff in IDLE. That all looked fine. It was more worrisome to me that some of Tcl and Tk's own tests didn't pass, but I can't dig into that. > As for the Berkeley DB package, that too built without any difficulties. > The instructions conclude: > > ============================== snip ============================== > XXX We're actually linking against Release_static\libdb40s.lib. > XXX This yields the following warnings: > """ > Compiling... > _bsddb.c > Linking... > Creating library ./_bsddb.lib and object ./_bsddb.exp > LINK : warning LNK4049: locally defined symbol "_malloc" imported > LINK : warning LNK4049: locally defined symbol "_free" imported > LINK : warning LNK4049: locally defined symbol "_fclose" imported > LINK : warning LNK4049: locally defined symbol "_fopen" imported > _bsddb.pyd - 0 error(s), 4 warning(s) > """ > XXX This isn't encouraging, but I don't know what to do about it. > ============================== snip ============================== > > I did a little poking around, and found that the problem is that the > static library against which the _bsddb package is built is compiled > with the -MT option, which causes the multi-threaded runtime libraries > to be statically linked into libdb40s.lib, hence the warnings. The > warnings should probably be taken seriously, because I think they mean > that we would end up with (for example) two sets of static heap control > information. Right, but it's not clear that it matters. For example, it's unlikely there's a case where memory is allocated by something in libdb40s but freed outside it, or vice versa. Then two heaps are just two distinct heaps. I'm inclined to ignore it for now, since we're probably going to move to a current Sleepycat release anyway (but I don't know more about that -- Barry is straightening that out). It's quite unclear to me what good libdb40s.lib *is* if anyone using, e.g., malloc, can't link it in safely! > The instructions in PCBuild/readme.txt were actually a little > confusing, because right before the part I just quoted above tells us > to "[c]hoose configuration "db_buildall - Win32 Release", and build > db_buildall.exe." It just echoes the Sleepycat instructions here. > That version uses the -MD compilation option, In part. It also triggers the building of subprojects that end up creating libdb40s.lib too. > which uses the DLL version of the runtime libraries (which is what we > want), but it creates a DLL for libdb40 instead of a library which can > be statically linked into _bsddb.pyd. It creates both the DLL and the static library. > So it looks as there are two ways to eliminate the warnings > above: > > 1. Use the DLL version of libdb40 and link to its import library. This works but is clumsier: you can't run the Python tests then from the build directory, without first physically copying the Sleepycat DLL into PCbuild, and you *also* need to build-and-copy a distinct debug-mode Sleepycat DLL else you can't run the debug-build Python tests. In addition, the installer needs to be fiddled then to copy more files. Like the old 1.85 bsddb Windows port from Sam Rushing, and like the current bz2 and zlib modules, life is easier all around if 3rd-party code can be sucked directly into the Python wrapper .pyd file. > 2. Create a third configuration for the underlying Berkeley DB > library, using the -MD compilation option, but building a > statically linkable library instead of a DLL. I'm not sure how to do that, but don't want to muck with Sleepycat's build process regardless. > I would have assumed that the second option was needed, because I > assumed that Python wouldn't want to drag around lots of DLLs for the > various packages that get installed, but then I noticed that the > _tkinter package is doing just that, so I guess I'm not sure. It doesn't want to, but Tcl/Tk are big and do their own kinds of linking tricks. > If the second option is used, it should presumably be done with the > cooperation of the folks behind the Berkeley DB library, so it wouldn't > be necessary to recreate the third configuration by hand each time a > new release of the library came out. > > Finally, I was reluctant to dig any further into this package because I > read at the top of the instructions: > > "XXX The Sleepycat release we use will probably change before > XXX 2.3a1." > > and I didn't know whether that referred to a version upgrade or a > complete replacement of the underlying DB engine with some other > library, which would make any further work throwaway. It refers to the current Sleepycat release; if you followed the build instructions closely , they started by pointing you to a page of obsolete releases. For some reason Barry understands, we can't yet use the current release (I think the bsddb C API changed, and our wrapper needs to change accordingly). Since I'm having a hard time (as above) imagining what good libdb40s.lib is in its current state, I'm simply hoping that Sleepycat fixed the linking glitches in their current release. Thanks for the effort, Bob! It helps. From tim.one@comcast.net Tue Dec 3 05:08:49 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 03 Dec 2002 00:08:49 -0500 Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn? In-Reply-To: <15851.43071.25770.622477@montanaro.dyndns.org> Message-ID: [Tim] > Why do you need to kill a process externally? For example, why > can't you make "please stop now" a part of the protocol, > so that a process can terminate itself gracefully when told to? [Skip Montanaro] > I start the background ssh process from within pop3proxy, so it > seems to me I ought to be the one to terminate it. OK, that wasn't clear to me before. > As for protocol, I'm not aware of another way to terminate a running > ssh other than to call kill(2) with a specified signal. That's > more-or-less the Unix way. I don't have the luxury of modifying > ssh to understand some application-specific protocol. No, I'm not asking you to be silly . The Unix signal model isn't going to fly for you on Windows regardless. From martin@v.loewis.de Tue Dec 3 07:41:23 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 08:41:23 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: Message-ID: Just van Rossum writes: > - Allow the "Import from Zip archive" patch to be rewritten as an > external module, avoiding the huge mess it is right now, as it > touches more or less all of import.c and then some. Paul Moore is > doing an admirable job to get it working and cleaned up, but I > honestly don't see this getting into the core: Actually, I do see it getting into the core, nearly as-is. In what time frame would you expect to implement the zipfile import patch using your framework? Regards, Martin From martin@v.loewis.de Tue Dec 3 08:17:52 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 09:17:52 +0100 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> References: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> Message-ID: Greg Ewing writes: > If this involves VPATH, I think I might know what's happening -- > something similar happened to me recently. > > Have you by any chance previously done a non-VPATH build in the source > tree? I get the same problem when building in the source directory (/usr/src/python). VPATH is not set in the Makefile, and the compilation commands read like this gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I. -I/usr/src/python/./Include -I/usr/local/include -I/usr/src/python/Include -I/usr/src/python -c /usr/src/python/Modules/regexmodule.c -o /usr/src/python/Modules/regexmodule.o I believe this is triggered by the change ---------------------------- revision 1.45 date: 2001/08/05 22:31:19; author: jackjansen; state: Exp; lines: +63 -6 Replace moddir and incdir by moddirlist and incdirlist, lists of source and include directories that are searched for modules. This is needed because the Mac modules and include files live in the Mac subtree. In addition (and that's actually what the mod is all about) on OSX we build all the Mac extension modules. ---------------------------- Here, moddirlist is ['/usr/src/python/Modules'], so the name of the source module will be /usr/src/python/Modules/regexmodule.c, instead of Modules/regexmodule.c. In turn, build_ext.build_temp is ignored, and the object files end up in the wrong place. Jack, can you please correct this problem? Regards, Martin From guido@python.org Tue Dec 3 08:25:46 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 03:25:46 -0500 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: Your message of "Tue, 03 Dec 2002 17:48:55 +1300." <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> References: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz> Message-ID: <200212030825.gB38Pk101300@pcp02138704pcs.reston01.va.comcast.net> > If this involves VPATH, I think I might know what's happening -- > something similar happened to me recently. > > Have you by any chance previously done a non-VPATH build in the source > tree? > > It seems that, when using a VPATH, any pathname referencing something > in the build directory for which a corresponding file exists in the > VPATH directory gets changed to refer to the VPATH directory. So you > need to make sure there are *only* source files in the source tree > before doing a build using a VPATH. This is indeed a common pitfall (== I've wasted an hour debugging this once :-). Can someone add a note to README? --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Tue Dec 3 08:36:32 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 09:36:32 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Message-ID: Martin v. L=F6wis wrote: > Actually, I do see it getting into the core, nearly as-is. Ok, let's then say I'm opposed to the patch as it stand, as it messes up import.c too much. The zip import stuff is _only_ in import.c because it'= s currently not possible to do it differently. This *must* be factored out,= and now is the time. Don't you at least agree it's a hairy patch? > In what time frame would you expect to implement the zipfile import > patch using your framework? Don't know, I can't do it by myself. Shouldn't be too hard, though. Just From just@letterror.com Tue Dec 3 08:41:26 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 09:41:26 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <069901c299c2$d7988f80$0201010a@wiktor> Message-ID: Wiktor Sadowski wrote: > Just van Rossum wrote: > > >>Could you please post a normal diff of your patch? It looks very > >>interesting. > Sorry, I haven't got time for cleaning up my modified Python C code > and building the diffs. > > >.I'm not entirely sure yet how my idea and yours could be > >integrated, but I think > > it's work a look. > > The concept is the same , there are differences in implementations. > The implementation you have posted is much more pure Python oriented. > I am using C structures and pointers to C functions (wherever > possible) instead of Python objects (well PyObjects are still C > structures) , so maybe it's a little faster (maybe not) I'm sure yours will be faster. However, the _same_ stuff is needed on the Python end, and I think it can still be made fairly easy to do what you want with sys.import_hooks. It can probably use a new C API to make things more straightforward. > Looks like I was wasting my time.;) Not at all! If it wan't for your post I might not have implemented the import_hooks stuff nearly as quickly, if at all ;-) import.c is a mess, and I think my patch opens doors to actually clean it up (perhaps for 2.4). Just From guido@python.org Tue Dec 3 08:44:14 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 03:44:14 -0500 Subject: [Python-Dev] Mac OSX issues In-Reply-To: Your message of "Mon, 02 Dec 2002 12:43:01 CST." <15851.43445.443010.182362@montanaro.dyndns.org> References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> <15851.43445.443010.182362@montanaro.dyndns.org> Message-ID: <200212030844.gB38iEJ01481@pcp02138704pcs.reston01.va.comcast.net> > Guido> I'm still seeing errors in test_socket; today (on a different > Guido> network) I got this: > > Guido> test test_socket failed -- Traceback (most recent call last): > Guido> File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes > Guido> ip = socket.gethostbyname(hostname) > Guido> gaierror: (7, 'No address associated with nodename') > > What does the hostname(1) command return? By default I think it generates > some oddball thing like "Montanaro.local.". Apple calls it a Rendevous > name. (I've no idea what that is.) The fix for me was recommended by > another group: get a dyndns.org account and change the HOSTNAME line in > /etc/hostconfig from > > HOSTNAME=-AUTOMATIC- > > to > > HOSTNAME=montanaro.dyndns.org Thanks, Skip, but I'm more interested in fixing test_socket so it doesn 't rely on this. Maybe testHostnameRes() should be disabled when gethostbyaddr() gives an error? It makes unwarranted assumptions IMO about how DNS is set up. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Dec 3 08:59:05 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 03:59:05 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: Your message of "Mon, 02 Dec 2002 17:33:25 EST." References: <003001c298d7$550e8240$0201010a@wiktor> Message-ID: <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> > My users want extension modules to be able to control how their > attributes are imported when using "import *". In particular, > Boost.Python includes a kind of multimethod facility, and some of my > users would like it if > > from bpl_extension import * > > caused multimethod definitions in bpl_extension to be /merged/ into > the current module, rather than replacing any multimethod bindings > with the same name in the current module. Alas, the current import hooking mechanisms don't allow control over this (the interpretation of * is hardcoded). Feel free to suggest an API and/or implementation of a hook for that, after reading how it's done now. The crucial code is import_all_from() in ceval.c. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Tue Dec 3 08:57:53 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 09:57:53 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: Message-ID: Just van Rossum writes: > Don't you at least agree it's a hairy patch? It is very hard to read, as a patch, yes, and it does a lot of things that aren't clearly needed. However, in all cases I've investigated, the resulting import.c works just as fine as the original one. Don't read the patch, read the resulting code instead. I don't like to reject the patch on grounds of "it is too complicated". Perhaps the complexity is inherent, and it can't be done trivially. In that case, it would be a pity since it means that we can never get zipfile imports. The feature that this patch provides is well-defined. If somebody volunteers to cleanup the implementation later - this would be definitely appreciated. > > In what time frame would you expect to implement the zipfile import > > patch using your framework? > > Don't know, I can't do it by myself. Shouldn't be too hard, though. That, of course, speaks against the framework, or atleast against rejecting the zipfile patch. It seems that the likeliness is high that we get a new framework, and no application for it. If we require zipfile importing to use the new framework, we will never get zipfile importing. This doesn't sound very promising, so I'd mandate that any new framework demonstrates its usefulness in a non-trivial case, which is best done by implementing (or re-implementing) zipfile import. Regards, Martin From just@letterror.com Tue Dec 3 09:16:31 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 10:16:31 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Message-ID: Martin v. L=F6wis wrote: > It is very hard to read, as a patch, yes, and it does a lot of things > that aren't clearly needed. >=20 > However, in all cases I've investigated, the resulting import.c works > just as fine as the original one. Don't read the patch, read the > resulting code instead. And then it still does many things that "aren't clearly needed". > I don't like to reject the patch on grounds of "it is too > complicated". Perhaps the complexity is inherent, and it can't be done > trivially. The complexity is inherent *because* there's no import hook mechanism tha= t's even remotely usable from C. My patch changes that. > In that case, it would be a pity since it means that we can > never get zipfile imports. No it doesn't. It just means that a little more work is needed. > The feature that this patch provides is well-defined. If somebody > volunteers to cleanup the implementation later - this would be > definitely appreciated. It doesn't *belong* in import.c. It's bloat. > > > In what time frame would you expect to implement the zipfile import > > > patch using your framework? > >=20 > > Don't know, I can't do it by myself. Shouldn't be too hard, though. >=20 > That, of course, speaks against the framework, or atleast against > rejecting the zipfile patch. I rather meant "I prefer not to do it by myself". > It seems that the likeliness is high that we get a new framework, and > no application for it. If we require zipfile importing to use the new > framework, we will never get zipfile importing. Bull. > This doesn't sound very promising, so I'd mandate that any new > framework demonstrates its usefulness in a non-trivial case, which is > best done by implementing (or re-implementing) zipfile import. ... Just From martin@v.loewis.de Tue Dec 3 09:31:53 2002 From: martin@v.loewis.de (Martin v. Löwis) Date: Tue, 3 Dec 2002 10:31:53 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <000d01c29aae$d1220590$6f21e8d9@mira> > And then it still does many things that "aren't clearly needed". Yes, but these are mostly reorganizations of stuff, eg. pieces of a function moved into functions of their own. This is, in general, a good thing, as it improves readability of the resulting code (not of the patch, though). > The complexity is inherent *because* there's no import hook mechanism that's > even remotely usable from C. My patch changes that. I would expect that the complexity is inherent in "we implement our own file system" (i.e. zipfiles). So I don't trust promises that a great new framework will make things simpler, unless proven wrong, by seeing actual things that become actually simpler. > No it doesn't. It just means that a little more work is needed. The current patch was sitting on SF for almost a year, and nobody even looked at it. How many more years do you want to wait for somebody finding time for "a little more work"? > It doesn't *belong* in import.c. It's bloat. I don't understand. import.c does the implementation of the Python import statement, and the patch extends the semantics of the import statement, by allowing .zip files to be on sys.path. You could argue that the feature itself is bloat, but if the feature is desirable, I can't see how the implementation is bloat. > I rather meant "I prefer not to do it by myself". I can understand that. Unfortunately, past experience tells me that nobody will do it, then. If more people speak in favour of rejecting the zipfile import patch, we should probably tell Paul Moore, so that won't invest more time into what is a doomed project. Regards, Martin From guido@python.org Tue Dec 3 09:52:35 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 04:52:35 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Tue, 03 Dec 2002 10:31:53 +0100." <000d01c29aae$d1220590$6f21e8d9@mira> References: <000d01c29aae$d1220590$6f21e8d9@mira> Message-ID: <200212030952.gB39qZA02315@pcp02138704pcs.reston01.va.comcast.net> > If more people speak in favour of rejecting the zipfile import patch, we > should probably tell Paul Moore, so that won't invest more time into > what is a doomed project. I agree with both sides here. :-( / :-) The zipfile feature is important, and if this patch is the only way to get the feature in, so be it. But import.c really can use a cleanup. I wouldn't be against incorporating some version of the patch now, and cleaning it up later, possibly adding a new import hooking framework and moving some of the zip specific code out of import.c again. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Tue Dec 3 10:08:52 2002 From: mwh@python.net (Michael Hudson) Date: 03 Dec 2002 10:08:52 +0000 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: Skip Montanaro's message of "Mon, 2 Dec 2002 22:29:09 -0600" References: <15852.13077.577684.9372@montanaro.dyndns.org> Message-ID: <2misybjtxn.fsf@starship.python.net> Skip Montanaro writes: > I don't have time to look into this right now. I only just noticed it while > I was trying to do parallel builds in build.dbg and build.opt directories. > The debug .o files got written first, so when the optimized make came along, > there were already .o files, which had been compiled with --with-pydebug, > got linked improperly. Ah, this would be why I couldn't build a debug python on my iBook, then. If this is Jack's fault, as Martin suggests, we should bully him until he fixes it! Cheers, M. -- > so python will fork if activestate starts polluting it? I find it more relevant to speculate on whether Python would fork if the merpeople start invading our cities riding on the backs of giant king crabs. -- Brian Quinlan, comp.lang.python From mal@lemburg.com Tue Dec 3 10:28:13 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 03 Dec 2002 11:28:13 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <000d01c29aae$d1220590$6f21e8d9@mira> Message-ID: <3DEC873D.7020406@lemburg.com> Martin v. L=F6wis wrote: > If more people speak in favour of rejecting the zipfile import patch, w= e > should probably tell Paul Moore, so that won't invest more time into > what is a doomed project. I have been waiting for such a patch to get incorporated for a long time now. ZIP file imports would be great for packaging software in a way which lets the packager have more control of what's lying around on the user machine, e.g. it allows removing cruft very easily and makes user space tampering harder. Whatever you come up with, please make sure that ZIP file based imports become possible. Thanks, --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From just@letterror.com Tue Dec 3 10:37:51 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 11:37:51 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <000d01c29aae$d1220590$6f21e8d9@mira> Message-ID: Martin v. L=F6wis wrote: > > And then it still does many things that "aren't clearly needed". >=20 > Yes, but these are mostly reorganizations of stuff, eg. pieces of a > function moved into functions of their own. This is, in general, a good > thing, as it improves readability of the resulting code (not of the > patch, though). Besides bloat, my main objection to the patch is that it refactors import= .c in a way that's not general enough and actually make import.c more complex tha= n needed. This is IMO a bad thing. > > The complexity is inherent *because* there's no import hook mechanism > > that's even remotely usable from C. My patch changes that. >=20 > I would expect that the complexity is inherent in "we implement our own > file system" (i.e. zipfiles). So I don't trust promises that a great ne= w > framework will make things simpler, unless proven wrong, by seeing > actual things that become actually simpler. True. See below. > > No it doesn't. It just means that a little more work is needed. >=20 > The current patch was sitting on SF for almost a year, and nobody even > looked at it. How many more years do you want to wait for somebody > finding time for "a little more work"? It's been sitting there for so long *because* it's such a hairy patch. I'= ve just started writing a new madule called "zipimport" which will show what I me= an. The rest of today will be very busy here, so you _might_ have to wait until t= omorrow for working code. Just From fredrik@pythonware.com Tue Dec 3 11:34:21 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 3 Dec 2002 12:34:21 +0100 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <0ad001c29abf$ed602f50$0900a8c0@spiff> guido wrote: > > try: > > import threading > > except ImportError: > > threading =3D None # use non-threaded behaviour >=20 > But then you'd have to sprinkle the rest of the code with > "if threading is not None" tests. Brett's dummy_thread is an attempt > to reduce this clutter. the problem (if there is one) is that the proposed integration may break code that uses the above pattern. deadlocks doesn't always happen when you want them to, so it may not be obvious to developers and testers that their program isn't as reliable as it was before the upgrade. From Paul.Moore@atosorigin.com Tue Dec 3 11:48:29 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 3 Dec 2002 11:48:29 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB88619969@UKDCX001.uk.int.atosorigin.com> (I'm reading python-dev via the web, so this is a combined reply to various posts) First of all, let me say that I'm interested in imports from zipfiles, as a feature. I don't have any intrinsic reason to prefer a specific implementation over any other. I picked up Jim Ahlstrom's patch and had a look at it simply because it seemed the best way to help. My view is that import.c is, indeed, very complex and difficult code to follow. I haven't made a significant attempt to do so, taking the view that as Jim's patch basically worked for him, I will just look at updating things to cope with the changes in CVS since his patch was current. (The comments in SF indicate that the patch was nearly accepted as it stood, and just needed updating to match current CVS). I agree with Just's comments that a cleanup in import.c would be useful. But I'm not in a position to do that, much as I would like it. (One example - "universal newline" support could do with being better localised - I just implemented the 4th copy of a tiny chunk of code to juggle file open modes to support it). I think that the zipfile patch is close to being acceptable. I don't have a problem if it gets rejected, but can I make a plea - if it does, can it be "Rejected because patch XXX does the job better" rather than "Rejected because this can be implemented better using feature YYY". In other words, as Martin suggests, can any competing proposal actually *implement* zipfile imports, so that we don't hit a new delay. By the way, the zipfile import also implements a cacheing feature for directory lookups. PEP 273 gives more details, including some benchmarks, but the indications are that the cache speeds up normal imports as well as being crucial for zipfile imports (where directory lookups are *slow*). I can't see how the design of Just's patch would help us implement this cache - maybe he can comment. Finally, while we're looking at imports and hooks, has anyone considered Gordon McMillan's imputil replacement, as described at http://www.mcmillan-inc.com/iu.html? I've used it to implement a proof-of-concept zipfile import (in Python) before now, and it was pretty simple to use. Maybe this would be a good candidate for the standard library? Thanks to everyone for the interest and support. Paul. From guido@python.org Tue Dec 3 12:07:53 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 07:07:53 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Your message of "Tue, 03 Dec 2002 12:34:21 +0100." <0ad001c29abf$ed602f50$0900a8c0@spiff> References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> Message-ID: <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> > > > try: > > > import threading > > > except ImportError: > > > threading = None # use non-threaded behaviour > > > > But then you'd have to sprinkle the rest of the code with > > "if threading is not None" tests. Brett's dummy_thread is an attempt > > to reduce this clutter. > > the problem (if there is one) is that the proposed integration may > break code that uses the above pattern. I don't propose to automate this. I propose this: try: import threading except ImportError: import dummy_threading as threading > deadlocks doesn't always happen when you want them to, so it > may not be obvious to developers and testers that their program > isn't as reliable as it was before the upgrade. So require testing on a system that has threads. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Tue Dec 3 13:20:07 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 3 Dec 2002 08:20:07 -0500 Subject: [Python-Dev] Mac OSX issues References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> <15851.43445.443010.182362@montanaro.dyndns.org> <200212030844.gB38iEJ01481@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15852.44935.756518.645276@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> Thanks, Skip, but I'm more interested in fixing test_socket GvR> so it doesn 't rely on this. Maybe testHostnameRes() should GvR> be disabled when gethostbyaddr() gives an error? It makes GvR> unwarranted assumptions IMO about how DNS is set up. BTW, test_socket.py passes just fine for me. hostname(1) returns "freewill.local." (yes, trailing dot). The difference may be that I used Netinfo Manager to add hostname entries for all my local machines, including the MacOSX machine I ran the tests on. My Netinfo-fu is pretty meager so I don't know if I did it completely right, but it seems to work for me. -Barry From sholden@holdenweb.com Tue Dec 3 13:28:46 2002 From: sholden@holdenweb.com (Steve Holden) Date: Tue, 3 Dec 2002 08:28:46 -0500 Subject: [Python-Dev] New and Improved Import Hooks References: <000d01c29aae$d1220590$6f21e8d9@mira> <200212030952.gB39qZA02315@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <009d01c29acf$eab0d380$6300000a@holdenweb.com> [MvL] > > If more people speak in favour of rejecting the zipfile import patch, we > > should probably tell Paul Moore, so that won't invest more time into > > what is a doomed project. [Guido] > > I agree with both sides here. :-( / :-) > > The zipfile feature is important, and if this patch is the only way to > get the feature in, so be it. > > But import.c really can use a cleanup. > > I wouldn't be against incorporating some version of the patch now, and > cleaning it up later, possibly adding a new import hooking framework > and moving some of the zip specific code out of import.c again. > Agreed. Like MAL I feel this is an important patch to get in to the distribution, even if it leaves a refactoring job behind it. Safe for me to say when I no longer do C, I suppose :-) but I'm speaking from a user perspective here. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From jepler@unpythonic.net Tue Dec 3 14:24:20 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 3 Dec 2002 08:24:20 -0600 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> References: <003001c298d7$550e8240$0201010a@wiktor> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021203142419.GC2507@unpythonic.net> On Tue, Dec 03, 2002 at 03:59:05AM -0500, Guido van Rossum wrote: > Alas, the current import hooking mechanisms don't allow control over > this (the interpretation of * is hardcoded). Feel free to suggest an > API and/or implementation of a hook for that, after reading how it's > done now. The crucial code is import_all_from() in ceval.c. Currently __all__ is a list of all the attributes to be exported by the module. What if we let __all__ be a list (with its current interpretation) or a callable. If __all__ is callable, then it is called with two parameters: the importing module, and the imported module. It can perform the desired operation. Any returned value is ignored, any raised exception is propagated. In David Abraham's case, the hook might look something like this: def __all__(exporter, importer): for name, attr in vars(exporter).items(): if name.startswith('_'): continue if isinstance(attr, Multimethod): merge_one_multimethod(importer, name, attr) else: setattr(importer, name, method) (David, apologies if I've simplified this too much. All I know about your desired functionality is that you want to "merge multimethods", and I have some idea what a multimethod is..) Jeff From neal@metaslash.com Tue Dec 3 14:28:05 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 3 Dec 2002 09:28:05 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619969@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619969@UKDCX001.uk.int.atosorigin.com> Message-ID: <20021203142805.GC16807@epoch.metaslash.com> On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote: > > My view is that import.c is, indeed, very complex and difficult > code to follow. > I agree with Just's comments that a cleanup in import.c would be > useful. I agree with both of these statements. I have reviewed the patch. While it's quite involved, I don't think it's worth rejecting. I would really like to see a cleanup, though. Rejecting the patch would be worthwhile if the interface would need to change after incorporating this patch. I don't believe this is the case. Whether this patch is accepted or not wouldn't change future patches that Just wants to incorporate. > I think that the zipfile patch is close to being acceptable. I > don't have a problem if it gets rejected, but can I make a plea > - if it does, can it be "Rejected because patch XXX does the job > better" rather than "Rejected because this can be implemented > better using feature YYY". Agreed. I don't recall if there is a lot of code duplication for file handling in import.c, there's certainly some. But in reviewing bz2, there was a lot of duplication. Gustavo and I discussed reusing the file code. Perhaps, PyFileObject could be refactored to make reuse easier in bz2 and import. If Just incorporates his import patch, perhaps the zipfile code could be refactored to use the more generic framework. I don't view these two patches as mutually exclusive. If all of these issues are addressed, the resulting code could be clearer, even if intermediate versions are not. Neal From mwh@python.net Tue Dec 3 14:31:15 2002 From: mwh@python.net (Michael Hudson) Date: 03 Dec 2002 14:31:15 +0000 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: David Abrahams's message of "Mon, 02 Dec 2002 17:33:25 -0500" References: <003001c298d7$550e8240$0201010a@wiktor> Message-ID: <2mfztfjhsc.fsf@starship.python.net> David Abrahams writes: > My users want extension modules to be able to control how their > attributes are imported when using "import *". In particular, > Boost.Python includes a kind of multimethod facility, and some of my > users would like it if > > from bpl_extension import * > > caused multimethod definitions in bpl_extension to be /merged/ into > the current module, rather than replacing any multimethod bindings > with the same name in the current module. Eww. I can see where the idea is coming from, but allowing arbitrary magic somewhere only name binding is allowed today sounds horrible to me. Cheers, M. -- please realize that the Common Lisp community is more than 40 years old. collectively, the community has already been where every clueless newbie will be going for the next three years. so relax, please. -- Erik Naggum, comp.lang.lisp From pedronis@bluewin.ch Tue Dec 3 14:24:59 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 15:24:59 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) References: <003001c298d7$550e8240$0201010a@wiktor> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> Message-ID: <00a701c29ad7$c33b8860$6d94fea9@newmexico> ----- Original Message ----- From: "Jeff Epler" To: Sent: Tuesday, December 03, 2002 3:24 PM Subject: Re: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) > On Tue, Dec 03, 2002 at 03:59:05AM -0500, Guido van Rossum wrote: > > Alas, the current import hooking mechanisms don't allow control over > > this (the interpretation of * is hardcoded). Feel free to suggest an > > API and/or implementation of a hook for that, after reading how it's > > done now. The crucial code is import_all_from() in ceval.c. > > Currently __all__ is a list of all the attributes to be exported by the > module. What if we let __all__ be a list (with its current > interpretation) or a callable. > > If __all__ is callable, then it is called with two parameters: the > importing module, and the imported module. It can perform the desired > operation. Any returned value is ignored, any raised exception is > propagated. > > In David Abraham's case, the hook might look something like this: > > def __all__(exporter, importer): > for name, attr in vars(exporter).items(): > if name.startswith('_'): continue > if isinstance(attr, Multimethod): > merge_one_multimethod(importer, name, attr) > else: > setattr(importer, name, method) > > (David, apologies if I've simplified this too much. All I know about > your desired functionality is that you want to "merge multimethods", and > I have some idea what a multimethod is..) > I find this idea of a semantics change rather unpythonic; ( 2nd in general should not import * be avoided ?) Why then only import * should have a user-defined behavior? What about from module import x if the importing module has already a defined x then maybe that should be also under user control? I'm surely missing something. regards. From dave@boost-consulting.com Tue Dec 3 14:40:43 2002 From: dave@boost-consulting.com (David Abrahams) Date: Tue, 03 Dec 2002 09:40:43 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <20021203142419.GC2507@unpythonic.net> (Jeff Epler's message of "Tue, 3 Dec 2002 08:24:20 -0600") References: <003001c298d7$550e8240$0201010a@wiktor> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> Message-ID: Jeff Epler writes: > On Tue, Dec 03, 2002 at 03:59:05AM -0500, Guido van Rossum wrote: >> Alas, the current import hooking mechanisms don't allow control over >> this (the interpretation of * is hardcoded). Feel free to suggest an >> API and/or implementation of a hook for that, after reading how it's >> done now. The crucial code is import_all_from() in ceval.c. > > Currently __all__ is a list of all the attributes to be exported by the > module. What if we let __all__ be a list (with its current > interpretation) or a callable. > > If __all__ is callable, then it is called with two parameters: the > importing module, and the imported module. It can perform the desired > operation. Any returned value is ignored, any raised exception is > propagated. > > In David Abraham's case, the hook might look something like this: > > def __all__(exporter, importer): > for name, attr in vars(exporter).items(): > if name.startswith('_'): continue > if isinstance(attr, Multimethod): > merge_one_multimethod(importer, name, attr) > else: > setattr(importer, name, method) > > (David, apologies if I've simplified this too much. All I know about > your desired functionality is that you want to "merge multimethods", and > I have some idea what a multimethod is..) Nice; I think this pretty much captures what my users are after. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From just@letterror.com Tue Dec 3 14:52:18 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 15:52:18 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <20021203142805.GC16807@epoch.metaslash.com> Message-ID: Neal Norwitz wrote: > On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote: > > > > My view is that import.c is, indeed, very complex and difficult > > code to follow. I agree with Just's comments that a cleanup in > > import.c would be useful. > > I agree with both of these statements. I have reviewed the patch. > While it's quite involved, I don't think it's worth rejecting. > I would really like to see a cleanup, though. I'm working hard on a zipimport module that uses sys.import_hooks. It's going to be SO much cleaner than the existing zip import patch. > Rejecting the patch would be worthwhile if the interface would need to > change after incorporating this patch. I don't believe this is the > case. Whether this patch is accepted or not wouldn't change future > patches that Just wants to incorporate. I personally reject the patch because it modifies import.c beyond repair. Once this stuff is in, there's NO way we can go back without a huge effort, meaning it won't happen. It adds a whole directory listing caching scheme that is a) totally unneeded and b) has (IMO) _nothing_ to do with zip importing per se. That ExternalNames/External names stuff? I don't understand why it is needed just to enable zip imports. It's a lot of stuff! > > I think that the zipfile patch is close to being acceptable. I > > don't have a problem if it gets rejected, but can I make a plea > > - if it does, can it be "Rejected because patch XXX does the job > > better" rather than "Rejected because this can be implemented > > better using feature YYY". > > Agreed. And that's ecxactly what I intend to say in a couple of days if not sooner. [ ... ] > If Just incorporates his import patch, perhaps the zipfile code could > be refactored to use the more generic framework. I don't view these > two patches as mutually exclusive. I *totally* do. > If all of these issues are > addressed, the resulting code could be clearer, even if intermediate > versions are not. A rewrite now is MUCH easier than cleaning it up later. It's going to be much less code. Wait and see. Just From dave@boost-consulting.com Tue Dec 3 14:45:24 2002 From: dave@boost-consulting.com (David Abrahams) Date: Tue, 03 Dec 2002 09:45:24 -0500 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <00a701c29ad7$c33b8860$6d94fea9@newmexico> ("Samuele Pedroni"'s message of "Tue, 3 Dec 2002 15:24:59 +0100") References: <003001c298d7$550e8240$0201010a@wiktor> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> <00a701c29ad7$c33b8860$6d94fea9@newmexico> Message-ID: "Samuele Pedroni" writes: > From: "Jeff Epler" > >> Currently __all__ is a list of all the attributes to be exported by the >> module. What if we let __all__ be a list (with its current >> interpretation) or a callable. > I find this idea of a semantics change rather unpythonic; Of course you do; look at Jeff's email address! Wish I'd thought of establishing that one myself . Calling something "un-pythonic" is a very convenient form of derision. But seriously: it's not a semantics change, but an extension... unless you think people are making __all__ callable today, and counting on an exception from the import mechanism. > ( 2nd in general should not import * be avoided ?) > > Why then only import * should have a user-defined behavior? > > What about > > from module import x I had the same thoughts. I am not entirely convinced that what my user has asked for is the best approach. > if the importing module has already a defined x then maybe that > should be also under user control? I don't understand that suggestion. Care to elaborate? > I'm surely missing something. it ain't neccessarily so... -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From pedronis@bluewin.ch Tue Dec 3 14:41:14 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 15:41:14 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) References: <003001c298d7$550e8240$0201010a@wiktor> <2mfztfjhsc.fsf@starship.python.net> Message-ID: <00ef01c29ada$083da180$6d94fea9@newmexico> From: "Michael Hudson" > > Eww. I can see where the idea is coming from, but allowing arbitrary > magic somewhere only name binding is allowed today sounds horrible to > me. > Indeed, that's exactly my concern too, and that then it should consider all from import forms. From pedronis@bluewin.ch Tue Dec 3 14:48:12 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 15:48:12 +0100 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) References: <003001c298d7$550e8240$0201010a@wiktor><200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net><20021203142419.GC2507@unpythonic.net><00a701c29ad7$c33b8860$6d94fea9@newmexico> Message-ID: <011f01c29adb$01033d20$6d94fea9@newmexico> From: "David Abrahams" > "Samuele Pedroni" writes: ... > > What about > > > > from module import x > > I had the same thoughts. I am not entirely convinced that what my user > has asked for is the best approach. > > > if the importing module has already a defined x then maybe that > > should be also under user control? > > I don't understand that suggestion. Care to elaborate? > my point is that if you offer a way to hook into the behavior of from import * then you need for completeness and to establish a coherent whole to offer a hook also in from import foo,bar regards. From Paul.Moore@atosorigin.com Tue Dec 3 15:47:42 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 3 Dec 2002 15:47:42 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com> From: Just van Rossum [mailto:just@letterror.com] > Neal Norwitz wrote: > > > On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote: > > > > > My view is that import.c is, indeed, very complex and > > > difficult code to follow. I agree with Just's comments that > > > a cleanup in import.c would be useful. > > > > I agree with both of these statements. I have reviewed the > > patch. While it's quite involved, I don't think it's worth > > rejecting I would really like to see a cleanup, though > > I'm working hard on a zipimport module that uses > sys.import_hooks. It's going to be SO much cleaner than the > existing zip import patch. That's great news, as I've said before. Can I ask that you consider a few issues, though. I don't have particular use cases in mind, but I believe that PEP 273 had a number of implicit assumptions, which your proposal does not incorporate. I'd like to make the differences explicit, just so that people get a chance to comment on whether they care. 1. Your import hooks seem to act at a level "above" sys.path, in the sense that the normal sys.path search becomes just another hook. With that in mind, I don't see a way for a user to add a zipfile in the *middle* of sys.path. 2. The syntax needed to add a zipfile search is different from the familiar one for sys.path additions. This isn't an issue as such, but may make the feature a little less accessible. 3. By using a non-sys.path approach, you make it impossible for users to set PYTHONPATH to include zipfiles. Can I again ask you to review Gordon McMillan's import manager. I really feel that you're reinventing the wheel here, and Gordon's code is well tested, as it's part of his Installer package. Maybe Gordon's code (which is pure Python) isn't appropriate to use, but I'm sure the design is worth looking at. > > Rejecting the patch would be worthwhile if the interface > > would need to change after incorporating this patch. I don't > > believe this is the case. Whether this patch is accepted > > or not wouldn't change future patches that Just wants to > > incorporate. > > I personally reject the patch because it modifies import.c > beyond repair. Once this stuff is in, there's NO way we can go > back without a huge effort, meaning it won't happen. I think you're exaggerating. It makes things worse, certainly, but I'd strongly dispute your claim of "beyond repair". I simply don't think the change is that major. Either it's already beyond repair, or the patched version is still fixable. And nobody is proposing "going back". A refactoring is "going forward". Whether it's a refactoring of code that *already* supports zipfiles or a refactoring which then has to be changed to add zipfile support shouldn't matter. > It adds > a whole directory listing caching scheme that is a) totally > unneeded and b) has (IMO) _nothing_ to do with zip importing > per se. That ExternalNames/External names stuff? I don't > understand why it is needed just to enable zip imports. It's a > lot of stuff! I've explained this a couple of times now. Please read the PEP for details, but Jim claims that zipfile imports need this if they are not to be too slow. A directory lookup in a zipfile is basically a linear search through the list of all files in the zipfile. For a large zipfile (such as the Python library) this is going to be *slow*. Please benchmark your implementation on a zipped version of the full Python library, replacing the directory version, and compare with Jim's benchmarks. While speed isn't the be-all and end-all of this patch, I'd hate to lose a significant speed gain... And the cache has a nice side-effect in that it speeds up general imports, particularly on networked drives. > > > I think that the zipfile patch is close to being > > > acceptable. I don't have a problem if it gets rejected, but > > > can I make a plea - if it does, can it be "Rejected because > > > patch XXX does the job better" rather than "Rejected > > > because this can be implemented better using feature YYY". > > > > Agreed. > > And that's ecxactly what I intend to say in a couple of days if > not sooner. Great! Will your code address all of the above points, and the issues raised on python-dev in Oct/Nov 2001 (see below for references)? > [ ... ] > > If Just incorporates his import patch, perhaps the zipfile > > code could be refactored to use the more generic framework. I > > don't view these two patches as mutually exclusive. > > I *totally* do. > > > If all of these issues are addressed, the resulting code > > could be clearer, even if intermediate versions are not. > > A rewrite now is MUCH easier than cleaning it up later. It's > going to be much less code. Wait and see. Here are some references that anyone looking at this stuff should read. There was a *lot* of discussion on this patch in Oct/Nov 2001. Start of the PEP 273 thread: http://mail.python.org/pipermail/python-dev/2001-October/018158.html Thread on caching of lookups http://mail.python.org/pipermail/python-dev/2001-November/018346.html Thomas Heller proposing Gordon McMillan's code for the standard library: http://mail.python.org/pipermail/python-dev/2001-October/017750.html As I've said, I'm not against rejecting the current patch - but I'd be concerned if all of the work that went into it (ie, Jim's design and discussions on python-dev, *not* the specific code implementing it) was just lost. Paul. From jepler@unpythonic.net Tue Dec 3 16:03:09 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 3 Dec 2002 10:03:09 -0600 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <00a701c29ad7$c33b8860$6d94fea9@newmexico> References: <003001c298d7$550e8240$0201010a@wiktor> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> <00a701c29ad7$c33b8860$6d94fea9@newmexico> Message-ID: <20021203160258.GD2507@unpythonic.net> On Tue, Dec 03, 2002 at 03:24:59PM +0100, Samuele Pedroni wrote: > Why then only import * should have a user-defined behavior? > > What about > > from module import x > > if the importing module has already a defined x then maybe that should be also > under user control? > > I'm surely missing something. I don't have a good sense of why this feature is desirable, I was just suggesting how David's request might be implemented without introducing new __special__ names. If it was desired that 'from module import x' be able to use the special behavior, then I'd suggest leaving __all__ as it is now, and having __export__, with the default behaving about like this: def __export__(exporter, importer, names): for name in names: setattr(importer, name, getattr(exporter, name)) and David's __export__ being about like this: def __export__(exporter, importer, names): for name in names: attr = getattr(exporter, name) if isinstance(attr, Multimethod): merge_one_multimethod(exporter, name, attr) else: setattr(exporter, name, attr) maybe __export__ should be called once per name, or maybe a special value should be passed for * (instead of __all__) Of course, 'from m import *' has that "do with me what you will" feeling to it, while 'from m import x' doesn't. Jeff From jepler@unpythonic.net Tue Dec 3 16:08:28 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 3 Dec 2002 10:08:28 -0600 Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211) In-Reply-To: <20021203160258.GD2507@unpythonic.net> References: <003001c298d7$550e8240$0201010a@wiktor> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> <00a701c29ad7$c33b8860$6d94fea9@newmexico> <20021203160258.GD2507@unpythonic.net> Message-ID: <20021203160827.GE2507@unpythonic.net> small change to example code On Tue, Dec 03, 2002 at 10:03:09AM -0600, Jeff Epler wrote: > and David's __export__ being about like this: > > def __export__(exporter, importer, names): > for name in names: > attr = getattr(exporter, name) > if isinstance(attr, Multimethod): - merge_one_multimethod(exporter, name, attr) + merge_one_multimethod(importer, name, attr) > else: > setattr(exporter, name, attr) From pedronis@bluewin.ch Tue Dec 3 16:05:24 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 17:05:24 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com> Message-ID: <016e01c29ae5$c9ec73a0$6d94fea9@newmexico> From: "Moore, Paul" >That's great news, as I've said before. Can I ask that you >consider a few issues, though. I don't have particular use cases >in mind, but I believe that PEP 273 had a number of implicit >assumptions, which your proposal does not incorporate. I'd like >to make the differences explicit, just so that people get a >chance to comment on whether they care. > >1. Your import hooks seem to act at a level "above" sys.path, > in the sense that the normal sys.path search becomes just > another hook. With that in mind, I don't see a way for a user > to add a zipfile in the *middle* of sys.path. It is worth to notice that Jython 2.1 already implement more or less PEP 273 approach, allowing to add zipfiles to sys.path. We can live with a different/new approach and then plan a migration path with warning etc, but if that could be avoided it would be nice. regards. From just@letterror.com Tue Dec 3 16:39:36 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 17:39:36 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > That's great news, as I've said before. Can I ask that you > consider a few issues, though. I don't have particular use cases > in mind, but I believe that PEP 273 had a number of implicit > assumptions, which your proposal does not incorporate. I'd like > to make the differences explicit, just so that people get a > chance to comment on whether they care. > > 1. Your import hooks seem to act at a level "above" sys.path, > in the sense that the normal sys.path search becomes just > another hook. With that in mind, I don't see a way for a user > to add a zipfile in the *middle* of sys.path. True. There would be ways around that, but it's currently not easy. I don't feel this is a particularly important feature, though. > 2. The syntax needed to add a zipfile search is different from > the familiar one for sys.path additions. This isn't an issue > as such, but may make the feature a little less accessible. The way I see it: sys.path is for file system imports. It's a list of directories, nothing else. > 3. By using a non-sys.path approach, you make it impossible for > users to set PYTHONPATH to include zipfiles. This could be fixed by site.py: it could remove any .zip files from sys.path and add hooks for them in sys.import_hooks. Or it could be fixed in the C code that adds the script's directory to sys.path. Or we could add a tiny change to import.c that invokes the zipimport module if a zip file is encountered. [ snip ] > > It adds > > a whole directory listing caching scheme that is a) totally > > unneeded and b) has (IMO) _nothing_ to do with zip importing > > per se. That ExternalNames/External names stuff? I don't > > understand why it is needed just to enable zip imports. It's a > > lot of stuff! > > I've explained this a couple of times now. Please read the PEP > for details, but Jim claims that zipfile imports need this if > they are not to be too slow. Well, I am not going to need it: upon reading the zip contents table I build a dict, mapping (full) module names to toc info. One dict lookup will be enough to get at the right data. Let's discuss the rest when my implementation actually works. Just From Paul.Moore@atosorigin.com Tue Dec 3 16:48:14 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 3 Dec 2002 16:48:14 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E4A@UKDCX001.uk.int.atosorigin.com> From: Just van Rossum [mailto:just@letterror.com] > > I've explained this a couple of times now. Please read > > the PEP for details, but Jim claims that zipfile imports > > need this if they are not to be too slow. > Well, I am not going to need it: upon reading the zip > contents table I build a dict, mapping (full) module > names to toc info. One dict lookup will be enough to > get at the right data. But that's exactly what Jim's cache is doing. His code is general, and offers the service to *all* import methods, rather than restricting it to only zipfiles. His benchmarks showed that this was a (small) win. Perhaps you could refactor your implementation so that the cache isn't just for zipfiles? (I can easily imagine other extensions that would benefit - tarfiles, metakit databases, imports over the web, ...) Paul. From skip@pobox.com Tue Dec 3 16:52:26 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 3 Dec 2002 10:52:26 -0600 Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS Message-ID: <15852.57674.16727.930806@montanaro.dyndns.org> Do we really need both Doc/whatsnew/* and Misc/NEWS? It seems that people frequently make nearly the same edits to both files, or add to Misc/NEWS only and someone later (usually Andrew) goes back and twiddles the necessary bits in Doc/whatsnew. Skip From just@letterror.com Tue Dec 3 17:10:09 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 18:10:09 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E4A@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > Perhaps you could refactor your implementation so that the > cache isn't just for zipfiles? (I can easily imagine other > extensions that would benefit - tarfiles, metakit databases, > imports over the web, ...) The directory listing is the hardest part and can't be general; caching is as simple as using a dict. I think it's the responsibility of the import hook. Btw. I could provide a zip import hook that's not specific to one archive, but rather searches sys.path for zip files. I'll play with that. Just From mwh@python.net Tue Dec 3 17:30:28 2002 From: mwh@python.net (Michael Hudson) Date: 03 Dec 2002 17:30:28 +0000 Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS In-Reply-To: Skip Montanaro's message of "Tue, 3 Dec 2002 10:52:26 -0600" References: <15852.57674.16727.930806@montanaro.dyndns.org> Message-ID: <2m7ker57t7.fsf@starship.python.net> Skip Montanaro writes: > Do we really need both Doc/whatsnew/* and Misc/NEWS? It seems that people > frequently make nearly the same edits to both files, or add to Misc/NEWS > only and someone later (usually Andrew) goes back and twiddles the necessary > bits in Doc/whatsnew. I'd have said yes: Misc/NEWS is for "developers" and whatsnew is for "users", although this is blatantly a bogus categorization. NEWS items are brief to the point of curtness, whatsnew has more explanation and some examples. Also I'd hope/expect NEWS to be more comprehensive -- whatsnew shouldn't need to have every little thing in it. Cheers, M. -- at any rate, I'm satisfied that not only do they know which end of the pointy thing to hold, but where to poke it for maximum effect. -- Eric The Read, asr, on google.com From amk@amk.ca Tue Dec 3 17:23:08 2002 From: amk@amk.ca (A.M. Kuchling) Date: Tue, 03 Dec 2002 12:23:08 -0500 Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS In-Reply-To: <15852.57674.16727.930806@montanaro.dyndns.org> References: <15852.57674.16727.930806@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > Do we really need both Doc/whatsnew/* and Misc/NEWS? Yes, we do. Misc/NEWS is complete, listing everything that's changed since the previous version. whatsnew is friendly, and there's a cutoff point; some changes are too trivial or esoteric to be worth mentioning. Feel free to only edit Misc/NEWS; before releases I'll go through it to find things I've missed, so there's no need to update whatsnew23.tex. I don't mind if people edit whatsnew23.tex to add items, but it's certainly not necessary. --amk From martin@v.loewis.de Tue Dec 3 17:42:19 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 18:42:19 +0100 Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS In-Reply-To: <2m7ker57t7.fsf@starship.python.net> References: <15852.57674.16727.930806@montanaro.dyndns.org> <2m7ker57t7.fsf@starship.python.net> Message-ID: Michael Hudson writes: > I'd have said yes: Misc/NEWS is for "developers" and whatsnew is for > "users", although this is blatantly a bogus categorization. > > NEWS items are brief to the point of curtness, whatsnew has more > explanation and some examples. Also I'd hope/expect NEWS to be more > comprehensive -- whatsnew shouldn't need to have every little thing in > it. +1. NEWS ought to be complete, with respect to changes that could remotely effect existing code. whatsnew.tex should focus on things that are both new and useful. Regards, Martin From martin@v.loewis.de Tue Dec 3 17:47:59 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 18:47:59 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: Message-ID: Just van Rossum writes: > I'm working hard on a zipimport module that uses > sys.import_hooks. It's going to be SO much cleaner than the existing > zip import patch. Ok. I then won't apply the existing patch until a) your zipfile patch comes along to review, or b) significant time has passed after 2.3a1 (for some reasonable value for "significant") IOW, it would be fine for me if 2.3a1 does not have zipfile import, but I think it must be distributed with 2.3 final (and the betas before it). Regards, Martin From mwh@python.net Tue Dec 3 18:02:12 2002 From: mwh@python.net (Michael Hudson) Date: 03 Dec 2002 18:02:12 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1 In-Reply-To: tim_one@users.sourceforge.net's message of "Tue, 03 Dec 2002 09:50:12 -0800" References: Message-ID: <2misybq8uz.fsf@starship.python.net> tim_one@users.sourceforge.net writes: > New program just to display pickle sizes. This makes clear that the > copy_reg based C implementation is much more space-efficient in the > end than the __getstate__/__setstate__ based Python implementation, > but that 4-byte date objects still suffer > 10 bytes of overhead each > no matter how many of them you pickle in one gulp. Presumably there's a possibility of an optimization for pickling homogeneous (i.e. all the same type) lists (in pickle.py, not here). Hard to say whether it would be worth it, though. Cheers, M. -- This is the fixed point problem again; since all some implementors do is implement the compiler and libraries for compiler writing, the language becomes good at writing compilers and not much else! -- Brian Rogoff, comp.lang.functional From mwh@python.net Tue Dec 3 18:27:59 2002 From: mwh@python.net (Michael Hudson) Date: 03 Dec 2002 18:27:59 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1 In-Reply-To: Michael Hudson's message of "03 Dec 2002 18:02:12 +0000" References: <2misybq8uz.fsf@starship.python.net> Message-ID: <2mfztfq7o0.fsf@starship.python.net> Michael Hudson writes: > tim_one@users.sourceforge.net writes: > > > New program just to display pickle sizes. This makes clear that the > > copy_reg based C implementation is much more space-efficient in the > > end than the __getstate__/__setstate__ based Python implementation, > > but that 4-byte date objects still suffer > 10 bytes of overhead each > > no matter how many of them you pickle in one gulp. > > Presumably there's a possibility of an optimization for pickling > homogeneous (i.e. all the same type) lists (in pickle.py, not here). > > Hard to say whether it would be worth it, though. Here's a fairly simple minded patch to the pickling side of pickle.py: it seems to save about 6 bytes per object in the good cases. with: list of 100 dates via C -- 1236 bytes, 12.36 bytes/obj without: list of 100 dates via C -- 1871 bytes, 18.71 bytes/obj I'm not going to pursue this further unless someone thinks it's a worthwhile move. Cheers, M. Index: pickle.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/pickle.py,v retrieving revision 1.72 diff -c -r1.72 pickle.py *** pickle.py 13 Nov 2002 22:01:26 -0000 1.72 --- pickle.py 3 Dec 2002 18:24:37 -0000 *************** *** 109,114 **** --- 109,115 ---- INST = 'i' LONG_BINGET = 'j' LIST = 'l' + HOM_LIST = 'k' EMPTY_LIST = ']' OBJ = 'o' PUT = 'p' *************** *** 439,445 **** --- 440,478 ---- def save_empty_tuple(self, object): self.write(EMPTY_TUPLE) + def save_hom_list(self, object): + reduce = dispatch_table[type(object[0])] + + write = self.write + save = self.save + memo = self.memo + + write(HOM_LIST) + + o = object[0] + + c, a = reduce(o) + + l = [a] + + for o in object[1:]: + l.append(reduce(o)[1]) + + save(c) + save(l) + def save_list(self, object): + t = {} + + for o in object: + t[type(o)] = 1 + if len(t) > 1: + break + else: + if t and dispatch_table.has_key(t.iterkeys().next()): + self.save_hom_list(object) + return + d = id(object) write = self.write -- Unfortunately, nigh the whole world is now duped into thinking that silly fill-in forms on web pages is the way to do user interfaces. -- Erik Naggum, comp.lang.lisp From martin@v.loewis.de Tue Dec 3 18:05:40 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 19:05:40 +0100 Subject: [Python-Dev] Make universal newlines non-optional Message-ID: At the time the universal newline code was committed, there was discussion whether or not it should be optional. I'd like to propose that it doesn't need to be optional: having a configuration option for it complicates the build documentation, and the space savings that one may gain from disabling it are minimal. So I volunteer to remove the #ifdef machinery around it. Any objections? Regards, Martin From tim.one@comcast.net Tue Dec 3 18:52:55 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 03 Dec 2002 13:52:55 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1 In-Reply-To: <2misybq8uz.fsf@starship.python.net> Message-ID: [Tim] >> New program just to display pickle sizes. This makes clear that the >> copy_reg based C implementation is much more space-efficient in the >> end than the __getstate__/__setstate__ based Python implementation, >> but that 4-byte date objects still suffer > 10 bytes of overhead each >> no matter how many of them you pickle in one gulp. [Michael Hudson] > Presumably there's a possibility of an optimization for pickling > homogeneous (i.e. all the same type) lists (in pickle.py, not here). > > Hard to say whether it would be worth it, though. I don't really care about lists of date objects. The intent was just to see how much of the administrative pickle bloat could be saved by pickle's internal memo facility when multiple date objects appear in a structure for *whatever* reason (be it a list or tuple of dates, or a dict keyed by dates, or an object with multiple data-value attributes, or ...). The administrative pickle bloat for a single date instance is severe: pickling 2000-12-13 via Python -- pickle length 80 pickling 2000-12-13 via C -- pickle length 43 The internal pickle memo saves a lot if more than one data instance appears, via "remembering" parts of the overhead scaffolding. But in the end a data object has a 4-byte state string, and there's still a lot more overhead than state stored in the pickles. > ... > Here's a fairly simple minded patch to the pickling side of pickle.py: > it seems to save about 6 bytes per object in the good cases. > > with: > list of 100 dates via C -- 1236 bytes, 12.36 bytes/obj > > without: > list of 100 dates via C -- 1871 bytes, 18.71 bytes/obj The "without" number is most curious. When I run the checked-in code, I get list of 100 dates via C -- 1533 bytes, 15.33 bytes/obj This was w/ CVS Python, Win2K and MSVC6. Ah! You must have fiddled the script to use pickle instead of cPickle. I get 1871 bytes then. I'm surprised they're so different. > I'm not going to pursue this further unless someone thinks it's a > worthwhile move. I don't personally have large pickles of homogeneous lists, so hard to say. The C implementation of pickle would also need to be fiddled. Cutting the pickle overheads for instances of new-style classes would more clearly be worthwhile. From bac@OCF.Berkeley.EDU Tue Dec 3 18:56:13 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 3 Dec 2002 10:56:13 -0800 (PST) Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > > > > try: > > > > import threading > > > > except ImportError: > > > > threading = None # use non-threaded behaviour > > > > > > But then you'd have to sprinkle the rest of the code with > > > "if threading is not None" tests. Brett's dummy_thread is an attempt > > > to reduce this clutter. > > > > the problem (if there is one) is that the proposed integration may > > break code that uses the above pattern. > But if it does it will be on platforms it couldn't run on previously. ``threading`` uses ``thread`` when possible and only falls back on ``dummy_thread`` when it isn't available. That means this could only be a problem where the code was not even runnable before. But yes, there are chances of deadlock in certain situations. It basically comes down to whether we want to protect users from themselves by not having them get into a possible deadlock situation. > I don't propose to automate this. I propose this: > > try: > import threading > except ImportError: > import dummy_threading as threading > So is this an indirect request for me to write ``dumy_threading``? If it is it won't be hard: I would do an ``import *`` on ``threading`` and then override the functions it uses from ``thread`` with ``import ... from ... as ...`` from ``dummy_thread``. Also would mean rewriting the ``threading`` testing suite. Prefectly happy to write it, just need you to say you want it. -Brett From fredrik@pythonware.com Tue Dec 3 19:24:31 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 3 Dec 2002 20:24:31 +0100 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <005b01c29b01$9ca50c10$ced241d5@hagrid> Brett Cannon wrote: > > > > > > try: > > > > > import threading > > > > > except ImportError: > > > > > threading = None # use non-threaded behaviour > But if it does it will be on platforms it couldn't run on previously. Sigh. Let's try one more time: Under 2.2 and earlier, on a platform that doesn't support threads, the above code will set threading to None, and the application can then use "if threading is None" to provide application-specific work- arounds. (catching import errors is a very common pattern when you're writing portable Python code). If you change Python so that "import threading" succeeds also on plat- forms that don't support threads, the application-specific workarounds will no longer be used, and the application may no longer work properly. And since we're talking deadlocks, chances are that nobody will notice before it's too late. From guido@python.org Tue Dec 3 19:29:43 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 14:29:43 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Tue, 03 Dec 2002 17:05:24 +0100." <016e01c29ae5$c9ec73a0$6d94fea9@newmexico> References: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com> <016e01c29ae5$c9ec73a0$6d94fea9@newmexico> Message-ID: <200212031929.gB3JThr04042@pcp02138704pcs.reston01.va.comcast.net> > >1. Your import hooks seem to act at a level "above" sys.path, > > in the sense that the normal sys.path search becomes just > > another hook. With that in mind, I don't see a way for a user > > to add a zipfile in the *middle* of sys.path. > > It is worth to notice that Jython 2.1 already implement more or less > PEP 273 approach, allowing to add zipfiles to sys.path. I'd be disappointed if adding zipfiles to sys.path wasn't possible. After all that's what PEP 273 says. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Dec 3 19:34:28 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 14:34:28 -0500 Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS In-Reply-To: Your message of "Tue, 03 Dec 2002 10:52:26 CST." <15852.57674.16727.930806@montanaro.dyndns.org> References: <15852.57674.16727.930806@montanaro.dyndns.org> Message-ID: <200212031934.gB3JYT004084@pcp02138704pcs.reston01.va.comcast.net> > Do we really need both Doc/whatsnew/* and Misc/NEWS? It seems that people > frequently make nearly the same edits to both files, or add to Misc/NEWS > only and someone later (usually Andrew) goes back and twiddles the necessary > bits in Doc/whatsnew. They serve different purposes. Misc/NEWS is supposed to be exhaustive, but terse. whatsnew23.tex is supposed to give the highlights, with examples, etc. --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Tue Dec 3 19:36:44 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 3 Dec 2002 11:36:44 -0800 (PST) Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: <005b01c29b01$9ca50c10$ced241d5@hagrid> References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <005b01c29b01$9ca50c10$ced241d5@hagrid> Message-ID: [Fredrik Lundh] > Brett Cannon wrote: > > > > > > > > try: > > > > > > import threading > > > > > > except ImportError: > > > > > > threading = None # use non-threaded behaviour > > > But if it does it will be on platforms it couldn't run on previously. > > Sigh. Let's try one more time: > > Under 2.2 and earlier, on a platform that doesn't support threads, > the above code will set threading to None, and the application can > then use "if threading is None" to provide application-specific work- > arounds. > I get this and always have. I just have not thought of it as huge of an issue as you do. If it is that big of a worry, add a ``import thread; del thread`` statement to make sure ``thread`` is there and warn users not to use the code. But I know you are going to be against the idea of modifying existing code for this, so I am going to let it go. Your point is valid, we just differ on its importance. Is Guido's suggested ``dummy_threading`` okay with you? > (catching import errors is a very common pattern when you're writing > portable Python code). > I know, I use it myself. > If you change Python so that "import threading" succeeds also on plat- > forms that don't support threads, the application-specific workarounds > will no longer be used, and the application may no longer work properly. > > And since we're talking deadlocks, chances are that nobody will notice > before it's too late. > OK, I am going to concede on this and stop pushing for it. Let me know what you think of Guido's ``dummy_threading`` idea. I assume you don't object to changing ``Queue``, though, right? -Brett From pedronis@bluewin.ch Tue Dec 3 19:28:41 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 20:28:41 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com> <016e01c29ae5$c9ec73a0$6d94fea9@newmexico> <200212031929.gB3JThr04042@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <003d01c29b02$30471580$6d94fea9@newmexico> From: "Guido van Rossum" > > >1. Your import hooks seem to act at a level "above" sys.path, > > > in the sense that the normal sys.path search becomes just > > > another hook. With that in mind, I don't see a way for a user > > > to add a zipfile in the *middle* of sys.path. > > > > It is worth to notice that Jython 2.1 already implement more or less > > PEP 273 approach, allowing to add zipfiles to sys.path. > > I'd be disappointed if adding zipfiles to sys.path wasn't possible. > After all that's what PEP 273 says. but - if I understand correctly - given the approach taken by the import_hooks patch there's the issue: consider all directories then all zipfiles versus consider both kind of import sources according to the order in sys.path, which is the most natural extension. regards. From martin@v.loewis.de Tue Dec 3 19:55:39 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 20:55:39 +0100 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Brett Cannon writes: > > I don't propose to automate this. I propose this: > > > > try: > > import threading > > except ImportError: > > import dummy_threading as threading > > > > So is this an indirect request for me to write ``dumy_threading``? I would suggest to make this import dummy_thread as threading i.e. have dummy_thread expose the union of the thread and threading APIs. > If it is it won't be hard: I would do an ``import *`` on > ``threading`` That wont't work: "from threading import *" will raise an ImportError. I suggest to add dummy_thread to sys.modules, then execfile threading.py in dummy_thread, then remove it from sys.modules. Regards, Martin From guido@python.org Tue Dec 3 19:58:31 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 14:58:31 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Your message of "Tue, 03 Dec 2002 10:56:13 PST." References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net> > > I don't propose to automate this. I propose this: > > > > try: > > import threading > > except ImportError: > > import dummy_threading as threading > > > > So is this an indirect request for me to write ``dumy_threading``? If it > is it won't be hard: I would do an ``import *`` on ``threading`` and then > override the functions it uses from ``thread`` with ``import ... from ... > as ...`` from ``dummy_thread``. Also would mean rewriting the > ``threading`` testing suite. Prefectly happy to write it, just need you > to say you want it. Yes. I don't want threading to automatically substitute dummy_thread for thread. Multiple imports of threading must all fail when thread cannot be imported. --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Tue Dec 3 20:05:52 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 21:05:52 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <003d01c29b02$30471580$6d94fea9@newmexico> Message-ID: [Guido] > > I'd be disappointed if adding zipfiles to sys.path wasn't possible. > > After all that's what PEP 273 says. [Samuele Pedroni] > but - if I understand correctly - given the approach taken by the > import_hooks patch there's the issue: > > consider all directories then all zipfiles (or vice versa ;-) > versus consider both kind of import sources according to the order in > sys.path, which is the most natural extension. Right. I admit I dismissed sys.path a wee bit too quickly when proposing an alternative zipimport mechanism. Regarding sys.import_hooks, I see three possibilities: #1 for p in sys.path: for hook in sys.import_hooks: hook(p, ...) #2 for hook in sys.import_hooks: for p in sys.path: hook(p, ...) #3 for hook in sys.import_hooks: # the hook can traversing sys.path if needed. hook(...) My current sys.import_hooks patch implements #3 and I think that's the most logical thing to do; after all sys.path may not be meaningful for a hook. #2 makes little sense I guess. #1 could be an option, but, given the current state of import.c, would be a lot harder to implement. I also think that #3 is the most efficient scheme, as the hook can save some local state while traversing sys.path. I will change my zipimport hook (which _still_ isn't finished ;-) to also traverse sys.path, and check for *.zip path endings. Just From pedronis@bluewin.ch Tue Dec 3 20:03:13 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 21:03:13 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <007301c29b07$02de61c0$6d94fea9@newmexico> From: "Just van Rossum" > I will change my zipimport hook (which _still_ isn't finished ;-) to also > traverse sys.path, and check for *.zip path endings. > my point was that, given dir1;zip0;dir2 dir1 should take over zip0 that should take over dir2. [This is what Jython 2.1 does btw] If the resulting import precedence order is that both dir1 and dir2 take over zip0, then maybe allowing zip0 in sys.path is just confusing, and zipfiles should be specified elsewhere. In general I clearly see that there can be import hooks that can leverage sys.path and its order semantics and some that can't. regards. From guido@python.org Tue Dec 3 20:09:58 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 03 Dec 2002 15:09:58 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Your message of "Tue, 03 Dec 2002 11:36:44 PST." References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <005b01c29b01$9ca50c10$ced241d5@hagrid> Message-ID: <200212032009.gB3K9wP04418@pcp02138704pcs.reston01.va.comcast.net> > > Under 2.2 and earlier, on a platform that doesn't support threads, > > the above code will set threading to None, and the application can > > then use "if threading is None" to provide application-specific work- > > arounds. > > I get this and always have. I just have not thought of it as huge of an > issue as you do. It is a big deal, Brett. --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Tue Dec 3 20:45:35 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 21:45:35 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <007301c29b07$02de61c0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > From: "Just van Rossum" > > I will change my zipimport hook (which _still_ isn't finished ;-) to also > > traverse sys.path, and check for *.zip path endings. > > > > my point was that, given > > dir1;zip0;dir2 > > dir1 should take over zip0 that should take over dir2. > > [This is what Jython 2.1 does btw] > > If the resulting import precedence order is that both dir1 and dir2 take over > zip0, then maybe allowing zip0 in sys.path is just confusing, > and zipfiles should be specified elsewhere. True, true. Does Jython implement the (old) idea that items on sys.path can be import handler *objects*? (If yes, can you briefly describe the interface of these objects?) After seeing what you mean I think that may be the only sensible solution. Say: for p in sys.path: if hasattr(p, "find_module"): x = p.find_module(...) elif not isinstance(p, basestring): continue else: x = builtin_find_module(...) ...load module... Just From martin@v.loewis.de Tue Dec 3 20:49:28 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 03 Dec 2002 21:49:28 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: Message-ID: Just van Rossum writes: > Does Jython implement the (old) idea that items on sys.path can be import > handler *objects*? (If yes, can you briefly describe the interface of these > objects?) > > After seeing what you mean I think that may be the only sensible solution. I think one requirement is that you can put zipfiles in PYTHONPATH and have that work. Regards, Martin From pedronis@bluewin.ch Tue Dec 3 20:43:11 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 21:43:11 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <00f501c29b0c$989af480$6d94fea9@newmexico> From: "Just van Rossum" > Samuele Pedroni wrote: > > > From: "Just van Rossum" > > > I will change my zipimport hook (which _still_ isn't finished ;-) to also > > > traverse sys.path, and check for *.zip path endings. > > > > > > > my point was that, given > > > > dir1;zip0;dir2 > > > > dir1 should take over zip0 that should take over dir2. > > > > [This is what Jython 2.1 does btw] > > > > If the resulting import precedence order is that both dir1 and dir2 take over > > zip0, then maybe allowing zip0 in sys.path is just confusing, > > and zipfiles should be specified elsewhere. > > True, true. > > Does Jython implement the (old) idea that items on sys.path can be import > handler *objects*? (If yes, can you briefly describe the interface of these > objects?) > No jython accepts strings of the form *.zip in sys.path, that are converted upon scanning on import to string subclasses' instances with the same value as string, through which import happens using an internal interface implemented by these string subclasses. We also set __path__s to contain things of the form *.zip/rel/path/to/package. regards. From mchermside@ingdirect.com Tue Dec 3 21:11:35 2002 From: mchermside@ingdirect.com (Chermside, Michael) Date: Tue, 3 Dec 2002 16:11:35 -0500 Subject: [Python-Dev] Re: New and Improved Import Hooks Message-ID: <902A1E710FEAB740966EC991C3A38A8903C35B21@INGDEXCHANGEC1.ingdirect.com> > > 1. Your import hooks seem to act at a level "above" sys.path, > > in the sense that the normal sys.path search becomes just > > another hook. With that in mind, I don't see a way for a user > > to add a zipfile in the *middle* of sys.path. >=20 > True. There would be ways around that, but it's currently not easy. I = don't feel > this is a particularly important feature, though. Hmm... Not sure I agree. I think this is one area where we'd be wise to follow Java's example. Normally I think it's wise for Python to do its own design instead of imitating the errors of other languages, but this is the exception. (1) Java (AFAIK) originated the idea of mixing .zip files as part of the path. (2) Their approach makes sense (in my mind, it would make sense to say "zip archives appear in path and are treated as if they=20 were subdirectories", and it would ALSO make sense to say "the list of zip archives is separate from the path". But some mix like "zip archives appear in the path, but we treat them as if they were at the end even though they aren't" doesn't make sense). (3) It makes sense for Jython (for obvious reasons!). Because of (1), along with the popularity of java, putting .zip's on the path will seem "natural and familiar" to lots of Python programmers. Because of (2) it isn't a BAD idea. And (3) is just another good motivation... let's keep c-python and jython as similar as possible! I sympathize about this being "not currently easy", but... -- Michael Chermside PS: I like the import-hooks idea, and I think the zip importer is an excellent test case to really see how well the concept plays in practice. From just@letterror.com Tue Dec 3 21:20:57 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 3 Dec 2002 22:20:57 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <00f501c29b0c$989af480$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > No jython accepts strings of the form *.zip in sys.path, that are > converted upon scanning on import to string subclasses' instances > with the same value as string, through which import happens using an > internal interface implemented by these string subclasses. > > We also set __path__s to contain things of the form *.zip/rel/path/to/package. And these package __path__s are _also_ "import hook" string subclasses, right? That makes an awful lot of sense. Just From pedronis@bluewin.ch Tue Dec 3 21:44:48 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 3 Dec 2002 22:44:48 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <01bf01c29b15$34474660$6d94fea9@newmexico> From: "Just van Rossum" > Samuele Pedroni wrote: > > > No jython accepts strings of the form *.zip in sys.path, that are > > converted upon scanning on import to string subclasses' instances > > with the same value as string, through which import happens using an > > internal interface implemented by these string subclasses. > > > > We also set __path__s to contain things of the form *.zip/rel/path/to/package. > > And these package __path__s are _also_ "import hook" string subclasses, right? Yes The interface is internal because here was no CPython prior art, and because we also support importing java classes from sys.path (both from dirs and zipfiles, although there are some bugs wrt to the latter), and our import code also begs for some refactoring... so things are in flux. [In the future (2.2) we were planning to add support for Java ClassLoaders too, but it is still unclear whether to put them in sys.path makes sense or not. I'm reporting this not because Java ClassLoaders are relevant to CPython but to point out that probably not all import hooks can be sys.path centered, and then a question is whether they should be considered before or after sys.path ] regards. From tdelaney@avaya.com Tue Dec 3 23:25:18 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Wed, 4 Dec 2002 10:25:18 +1100 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: > From: Just van Rossum [mailto:just@letterror.com] > > Regarding sys.import_hooks, I see three possibilities: > > #1 > for p in sys.path: > for hook in sys.import_hooks: > hook(p, ...) > > #2 > for hook in sys.import_hooks: > for p in sys.path: > hook(p, ...) > > #3 > for hook in sys.import_hooks: > # the hook can traversing sys.path if needed. > hook(...) It seems to me there needs to be two separate sets of import hooks. 1. Module import hooks. 2. Source import hooks. Module import hooks are the type being discussed for sys.import_hooks - a sequence of callables which are called one after the other until a module is successfully imported. Source import hooks provide the exact mechanism for doing an import from a particular data source. Doing this, in effect we would have the following algorithm ... for hook in sys.import_hooks: hook(...) and the algorithm for the builtin import hook would be ... for p in sys.path: for hook in sys.source_import_hooks: hook(p, ...) which would end up being the default behaviour. Individual module import hooks could (and probably should) use the appropriate source import hooks. I think this would suit all requirements, whilst being both general and clean. It would provide a simple mechanism for allowing .tar files in sys.path, internet imports, etc. Tim Delaney From bac@OCF.Berkeley.EDU Wed Dec 4 01:49:12 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 3 Dec 2002 17:49:12 -0800 (PST) Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net> References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > > > I don't propose to automate this. I propose this: > > > > > > try: > > > import threading > > > except ImportError: > > > import dummy_threading as threading > > > > > > > So is this an indirect request for me to write ``dumy_threading``? If it > > is it won't be hard: I would do an ``import *`` on ``threading`` and then > > override the functions it uses from ``thread`` with ``import ... from ... > > as ...`` from ``dummy_thread``. Also would mean rewriting the > > ``threading`` testing suite. Prefectly happy to write it, just need you > > to say you want it. > > Yes. I don't want threading to automatically substitute dummy_thread > for thread. Multiple imports of threading must all fail when thread > cannot be imported. > Do you prefer I do a separate ``dummy_threading`` module or follow Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` substitute itself? -Brett From shane.holloway@ieee.org Wed Dec 4 02:28:51 2002 From: shane.holloway@ieee.org (Shane Holloway (IEEE)) Date: Tue, 03 Dec 2002 19:28:51 -0700 Subject: [Python-Dev] Re: New and Improved Import Hooks Message-ID: <3DED6863.6000100@ieee.org> As it seems I am supposed to introduce myself on my first post, I am a professional software engineer in Colorado, USA, blessed with the chance to implement my solutions in Python. I've a fairly diverse background in languages, but the most experience with C++. I keep working on the RuneBlade project , which helps me deliver my Python solutions just that much faster. Seaching for a better language, Python has satisfied my thirst, seeming to combine my best experiences from Lisp, Pascal, C++ and others. Thanks for all of your hard work! Now it's time for me to try and help! IMO, Just's approach seems to be flexable enough to solve more than the problem at hand. Not only does the framework help to implement PEP 273, allowing for import from zip files, but it provides a very logical place for other import mechanisms to be implemented. Expanding upon Tim Delaney's desires to include importing from tar files and internet references, consider being able to extend this mechanism to importing from an XML document, database of modules, bz2 files, and even encrypted collections, The key, as I see it, is the flexability of the approach. One can even add and remove import stratigies on the fly without prior knowledge of all the import hooks! :-) As for keeping the import order of the sys.path, Tim seems to have hit the mark with the default importer supporting "sub-hooks" for the different path entry types. The best part about the import_hooks approach is that it leaves enough flexability with developer to implement creative approaches, while maintaining a commonly recognized standard for import hook interfaces. Even module-caching import hooks can use this model. One could write a simple hook that would log all imports, or all imports that could not be found, regardless of others import hooks. (Well, at least, if they behave properly ;) -Shane Holloway From tdelaney@avaya.com Wed Dec 4 02:36:52 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Wed, 4 Dec 2002 13:36:52 +1100 Subject: [Python-Dev] Re: New and Improved Import Hooks Message-ID: > From: Shane Holloway (IEEE) [mailto:shane.holloway@ieee.org] > > Expanding upon Tim > Delaney's desires to include importing from tar files and internet > references, Hey - I didn't say *I* wanted this (they came from other posts ;) but it's an obvious extension ... > consider being able to extend this mechanism to importing > from an XML document, database of modules, bz2 files, and > even encrypted > collections, As are all of these. I think if the import mechanism is going to be changed, it should be changed so as to be flexible. The fact that such a change is also (IMO) a much cleaner approach is icing. Tim Delaney From tim.one@comcast.net Wed Dec 4 03:12:29 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 03 Dec 2002 22:12:29 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Message-ID: [Guido] >>>> I don't propose to automate this. I propose this: >>>> >>>> try: >>>> import threading >>>> except ImportError: >>>> import dummy_threading as threading ... [Brett Cannon] > Do you prefer I do a separate ``dummy_threading`` module or follow > Martin's suggestions and just turn ``dummy_thread`` into > a ``threading`` substitute itself? Looking at his code, I think it's clear he wants a separate dummy_threading.py and nothing fancier than that. The idiomatic use would be as shown, where a user who knows what they're doing can import dummy_threading (if desired) and *name* it threading locally themself (if desired). No existing code could break, except for users who already have a file named dummy_threading.py on their PYTHONPATH somewhere after the std library. From bac@OCF.Berkeley.EDU Wed Dec 4 04:27:20 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 3 Dec 2002 20:27:20 -0800 (PST) Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: References: Message-ID: [Tim Peters] > [Guido] > >>>> I don't propose to automate this. I propose this: > >>>> > >>>> try: > >>>> import threading > >>>> except ImportError: > >>>> import dummy_threading as threading > > ... > > [Brett Cannon] > > Do you prefer I do a separate ``dummy_threading`` module or follow > > Martin's suggestions and just turn ``dummy_thread`` into > > a ``threading`` substitute itself? > > Looking at his code, I think it's clear he wants a separate > > dummy_threading.py > > and nothing fancier than that. The idiomatic use would be as shown, where a > user who knows what they're doing can import dummy_threading (if desired) > and *name* it threading locally themself (if desired). No existing code > could break, except for users who already have a file named > dummy_threading.py on their PYTHONPATH somewhere after the std library. > OK. Then I will just whip together the module (most likely following Martin's correction of my previous from-the-cuff idea on how to do it) and add it to the patch. So if anyone was planning on reviewing the patch, please hold off. I will email the list again once I have the module and the patches regenerated (although someone can go ahead and delete all the files in the patch; I will just put all the files up anew since practically all of them will change). -Brett From rjones@ekit-inc.com Wed Dec 4 04:47:14 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Wed, 4 Dec 2002 15:47:14 +1100 Subject: [Python-Dev] PEP 301: Package Index and Metadata for Distutils Message-ID: <200212041547.14551.rjones@ekit-inc.com> I've implemented all I believe is necessary for the package index to be pronounced accepted or not. This includes the last few missing pieces: - library reference patch against latest CVS - distutils/dist.py patch against latest CVS - use of distutils.version.LooseVersion sorting in the index database Where to now? Is there more to do? Should I stop now because it's not worth continuing? Richard From guido@python.org Wed Dec 4 08:44:21 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 04 Dec 2002 03:44:21 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Your message of "Tue, 03 Dec 2002 17:49:12 PST." References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212040844.gB48iLx06213@pcp02138704pcs.reston01.va.comcast.net> > Do you prefer I do a separate ``dummy_threading`` module or follow > Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` > substitute itself? Two separate modules. They have different APIs and it wouldn't be wise to export a different API in the dummy case than in the real case. (I'm sure you can figure out why.) --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Dec 4 09:59:56 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 04 Dec 2002 10:59:56 +0100 Subject: [Python-Dev] PEP 301: Package Index and Metadata for Distutils References: <200212041547.14551.rjones@ekit-inc.com> Message-ID: <3DEDD21C.9030909@lemburg.com> Richard Jones wrote: > I've implemented all I believe is necessary for the package index to be > pronounced accepted or not. This includes the last few missing pieces: > > - library reference patch against latest CVS > - distutils/dist.py patch against latest CVS > - use of distutils.version.LooseVersion sorting in the index database > > Where to now? Is there more to do? Should I stop now because it's not worth > continuing? I like the idea. One nit: in order to make the setup.py script work for Python 1.5.2 and up, you should provide a way which let's the distutils versions of those platforms ignore the classification keyword argument for setup(). It should be fairly easy, I believe (untested, but you get the picture...): # Add 'classifiers' support if sys.version < '2.3: from distutils.dist import Distribution, translate_longopt Distribution.display_options.append( ('classifiers', None, "print the list of classifiers (needs Python 2.3)")) Distribution.display_option_names = map(lambda x: translate_longopt(x[0]), Distribution.display_options) setup(...) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Jack.Jansen@cwi.nl Wed Dec 4 10:15:56 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Wed, 4 Dec 2002 11:15:56 +0100 Subject: [Python-Dev] Make universal newlines non-optional In-Reply-To: Message-ID: <6089D968-0771-11D7-93BC-0030655234CE@cwi.nl> On Tuesday, Dec 3, 2002, at 19:05 Europe/Amsterdam, Martin v. L=F6wis=20 wrote: > At the time the universal newline code was committed, there was > discussion whether or not it should be optional. > > I'd like to propose that it doesn't need to be optional: having a > configuration option for it complicates the build documentation, and > the space savings that one may gain from disabling it are minimal. > > So I volunteer to remove the #ifdef machinery around it. Any > objections? Well, we don't know yet whether it will work on non-mainstream=20 platforms. It'll probably be okay on BeOS or Amiga or such (and maybe the=20 maintainers of those platforms track CVS anyway?), but I wouldn't vouch for Windows=20= CE or so. -- - Jack Jansen =20 http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma=20 Goldman - From Jack.Jansen@cwi.nl Wed Dec 4 10:24:38 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Wed, 4 Dec 2002 11:24:38 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Message-ID: <978F6A8C-0772-11D7-93BC-0030655234CE@cwi.nl> On Tuesday, Dec 3, 2002, at 21:05 Europe/Amsterdam, Just van Rossum wrote: > Regarding sys.import_hooks, I see three possibilities: > > #1 > for p in sys.path: > for hook in sys.import_hooks: > hook(p, ...) > > #2 > for hook in sys.import_hooks: > for p in sys.path: > hook(p, ...) > > #3 > for hook in sys.import_hooks: > # the hook can traversing sys.path if needed. > hook(...) Note that there is "prior art" for case 1: the PYC resource importer for MacPython, which also looks at files (in stead of directories) on sys.path. It does have a bit of hairy code so it skips sys.path entries that don't apply to it (directories, basically). This was a major speedup, so maybe this should be generalized to something like for p in sys.path: for hook in sys.import_hooks: if not hook_can_handle.has_key((hook, p)): hook_can_handle[(hook, p)] = hook(p, None) # None requests "compatibility test" if hook_can_handle[(hook, p)]: hook(p, ....) -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From fredrik@pythonware.com Wed Dec 4 10:33:05 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 4 Dec 2002 11:33:05 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <978F6A8C-0772-11D7-93BC-0030655234CE@cwi.nl> Message-ID: <11a801c29b80$89bb2e20$0900a8c0@spiff> Jack Jansen wrote: > > Regarding sys.import_hooks, I see three possibilities: > > > > #1 > > for p in sys.path: > > for hook in sys.import_hooks: > > hook(p, ...) > > > > #2 > > for hook in sys.import_hooks: > > for p in sys.path: > > hook(p, ...) > > > > #3 > > for hook in sys.import_hooks: > > # the hook can traversing sys.path if needed. > > hook(...) what's wrong with: #4 for path in sys.path: if isinstance(path, types.StringTypes): do_standard_import(...) else: path.do_import(...) after all, everyone knows how the path works, and everyone knows how to manipulate sys.path inside a Python program. (for PIL and other plug-in architectures, a "path.get_list_of_modules" method would also be nice). From martin@v.loewis.de Wed Dec 4 10:41:01 2002 From: martin@v.loewis.de (Martin v. Löwis) Date: Wed, 4 Dec 2002 11:41:01 +0100 Subject: [Python-Dev] Make universal newlines non-optional References: <6089D968-0771-11D7-93BC-0030655234CE@cwi.nl> Message-ID: <002b01c29b81$a4689900$df3fe8d9@mira> > Well, we don't know yet whether it will work on non-mainstream > platforms. It'll probably be okay on BeOS or Amiga or such > (and maybe the maintainers of those platforms track CVS > anyway?), but I wouldn't vouch for Windows CE or so. What would be potential consequences of it not working? If it means that strange things may happen if you pass "U" to open, the feature could be built, anyway. Also, do you doubt that it *could* work (with sufficient modifications) on all systems we care about? If so, users of these systems would need to provide the necessary changes. Regards, Martin From Jack.Jansen@cwi.nl Wed Dec 4 11:02:07 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Wed, 4 Dec 2002 12:02:07 +0100 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: Message-ID: On Tuesday, Dec 3, 2002, at 09:17 Europe/Amsterdam, Martin v. L=F6wis=20 wrote: > Greg Ewing writes: > >> If this involves VPATH, I think I might know what's happening -- >> something similar happened to me recently. >> >> Have you by any chance previously done a non-VPATH build in the = source >> tree? > > I get the same problem when building in the source directory > (/usr/src/python). VPATH is not set in the Makefile, and the > compilation commands read like this > > gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I.=20 > -I/usr/src/python/./Include -I/usr/local/include=20 > -I/usr/src/python/Include -I/usr/src/python -c=20 > /usr/src/python/Modules/regexmodule.c -o=20 > /usr/src/python/Modules/regexmodule.o > > I believe this is triggered by the change > > ---------------------------- > revision 1.45 > date: 2001/08/05 22:31:19; author: jackjansen; state: Exp; lines:=20= > +63 -6 > Replace moddir and incdir by > moddirlist and incdirlist, lists of source and include > directories that are searched for modules. > This is needed because the Mac modules and include files > live in the Mac subtree. I backed up to 1.44, but I still see the same problem. And it stands to=20= reason: my find_module_file() call was a replacement for os.path.join(moddir, filename). And with=20 moddir being an absolute path this will also return an absolute path. I am tempted to think that this problem has always existed, but has=20 gone unnoticed until now. Now that I know about the problem it clears up some strange behaviour=20 I've seen in the past. I would love to see this fixed, but I'm afraid I'm not the person to do=20= it. Distutils never ceases to amaze me, and it's path-mangling tricks are beyond me.=20= I'm pretty sure that if we somehow stuff "../Modules/regexmodule.c" (assuming we're building=20= in a subdir of the main dir) into the list of module sources we'll get something like=20 build/tmp.blabla/../Modules/regexmodule.o for the object filename. Does anyone else feel up to fixing this? -- - Jack Jansen =20 http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma=20 Goldman - From Jack.Jansen@cwi.nl Wed Dec 4 11:15:37 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Wed, 4 Dec 2002 12:15:37 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <11a801c29b80$89bb2e20$0900a8c0@spiff> Message-ID: On Wednesday, Dec 4, 2002, at 11:33 Europe/Amsterdam, Fredrik Lundh wrote: > what's wrong with: > > #4 > for path in sys.path: > if isinstance(path, types.StringTypes): > do_standard_import(...) > else: > path.do_import(...) > After reading Just's later message and the explanation about how Jython does it I think that my preference is now something like the "if hasattr(path, 'find_module')" fix. This will allow you to use a subtype of strings on sys.path. Make one pass over sys.path replacing the zipfiles and other funnies by a specialised subtype of string. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From Jack.Jansen@cwi.nl Wed Dec 4 11:18:58 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Wed, 4 Dec 2002 12:18:58 +0100 Subject: [Python-Dev] Make universal newlines non-optional In-Reply-To: <002b01c29b81$a4689900$df3fe8d9@mira> Message-ID: <2F08AD4B-077A-11D7-93BC-0030655234CE@cwi.nl> On Wednesday, Dec 4, 2002, at 11:41 Europe/Amsterdam, Martin v. L=F6wis=20= wrote: >> Well, we don't know yet whether it will work on non-mainstream >> platforms. It'll probably be okay on BeOS or Amiga or such >> (and maybe the maintainers of those platforms track CVS >> anyway?), but I wouldn't vouch for Windows CE or so. > > What would be potential consequences of it not working? Currently you could simply disable it. If the ifdefs are gone then the=20= platform maintainers will have to actually fix it. But maybe I'm too defensive, -- - Jack Jansen =20 http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma=20 Goldman - From mwh@python.net Wed Dec 4 11:29:10 2002 From: mwh@python.net (Michael Hudson) Date: 04 Dec 2002 11:29:10 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1 In-Reply-To: Tim Peters's message of "Tue, 03 Dec 2002 13:52:55 -0500" References: Message-ID: <2md6oiqayh.fsf@starship.python.net> Tim Peters writes: > [Michael Hudson] > > Presumably there's a possibility of an optimization for pickling > > homogeneous (i.e. all the same type) lists (in pickle.py, not here). > > > > Hard to say whether it would be worth it, though. > > I don't really care about lists of date objects. The intent was just to see > how much of the administrative pickle bloat could be saved by pickle's > internal memo facility when multiple date objects appear in a structure for > *whatever* reason (be it a list or tuple of dates, or a dict keyed by dates, > or an object with multiple data-value attributes, or ...). OK. > The administrative pickle bloat for a single date instance is severe: > > pickling 2000-12-13 via Python -- pickle length 80 > pickling 2000-12-13 via C -- pickle length 43 Yow. > The internal pickle memo saves a lot if more than one data instance appears, > via "remembering" parts of the overhead scaffolding. But in the end a data > object has a 4-byte state string, and there's still a lot more overhead than > state stored in the pickles. Seems so. > > ... > > Here's a fairly simple minded patch to the pickling side of pickle.py: > > it seems to save about 6 bytes per object in the good cases. > > > > with: > > list of 100 dates via C -- 1236 bytes, 12.36 bytes/obj > > > > without: > > list of 100 dates via C -- 1871 bytes, 18.71 bytes/obj > > The "without" number is most curious. When I run the checked-in code, I get > > list of 100 dates via C -- 1533 bytes, 15.33 bytes/obj > > This was w/ CVS Python, Win2K and MSVC6. Ah! You must have fiddled the > script to use pickle instead of cPickle. Yep. > I get 1871 bytes then. I'm surprised they're so different. I know cPickle does some tricks if refcounts are 1 that aren't duplicated in pickle.py. > > I'm not going to pursue this further unless someone thinks it's a > > worthwhile move. > > I don't personally have large pickles of homogeneous lists, so hard to say. Me neither. > The C implementation of pickle would also need to be fiddled. That was part of "pursuing this further" (as is, e.g., unpickling support). > Cutting the pickle overheads for instances of new-style classes > would more clearly be worthwhile. Hard to see how to do that. You could reduce the overhead by giving _datetime_unpickler a shorter name :-) In general, though, any improvements we can add here are likely to be blown away by ramming the pickle through zlib, so I'm not sure this is worth worrying about too much. Cheers, M. -- And then the character-only displays went away (leading to increasingly silly graphical effects and finally to ads on web pages). -- John W. Baxter, comp.lang.python From just@letterror.com Wed Dec 4 11:52:39 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 12:52:39 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <11a801c29b80$89bb2e20$0900a8c0@spiff> Message-ID: Fredrik Lundh wrote: > what's wrong with: > > #4 > for path in sys.path: > if isinstance(path, types.StringTypes): > do_standard_import(...) > else: > path.do_import(...) That, or what Jack said, which I think is even better: #5 for path in sys.path: if hasattr(path, "find_module"): loader = path.find_module(name) if isinstance(path, types.StringTypes): do_standard_import(...) else: pass (I originally thought it might be important that path elements be str subclasses, but I'm not so sure anymore. Especially after I wrote a str subclass in C: it's a HUGE pain ;-) Just From mwh@python.net Wed Dec 4 13:00:40 2002 From: mwh@python.net (Michael Hudson) Date: 04 Dec 2002 13:00:40 +0000 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: Jack Jansen's message of "Wed, 4 Dec 2002 12:02:07 +0100" References: Message-ID: <2m7keqq6pz.fsf@starship.python.net> Jack Jansen writes: > I am tempted to think that this problem has always existed, but has=20 > gone unnoticed until now. It's definitely gotten worse recently. > Now that I know about the problem it clears up some strange behaviour=20 > I've seen in the past. > I would love to see this fixed, but I'm afraid I'm not the person to do=20= > > it. Distutils > never ceases to amaze me, and it's path-mangling tricks are beyond me.=20= > > I'm pretty sure that > if we somehow stuff "../Modules/regexmodule.c" (assuming we're building=20= > > in a subdir of the main dir) > into the list of module sources we'll get something like=20 > build/tmp.blabla/../Modules/regexmodule.o > for the object filename. > > Does anyone else feel up to fixing this? I'll have a look, but so far I'm only confusing myself. The first out-of-tree build I did today put one .o file in the wrong place (bsddbmodule.o, I think). Now all of them are ending up in the wrong place. Cheers, M. -- Very clever implementation techniques are required to implement this insanity correctly and usefully, not to mention that code written with this feature used and abused east and west is exceptionally exciting to debug. -- Erik Naggum on Algol-style "call-by-name" From mwh@python.net Wed Dec 4 13:26:55 2002 From: mwh@python.net (Michael Hudson) Date: 04 Dec 2002 13:26:55 +0000 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: Michael Hudson's message of "04 Dec 2002 13:00:40 +0000" References: <2m7keqq6pz.fsf@starship.python.net> Message-ID: <2m4r9trk2o.fsf@starship.python.net> Michael Hudson writes: > The first out-of-tree build I did today put one .o file in the wrong > place (bsddbmodule.o, I think). Now all of them are ending up in the > wrong place. Not sure how I managed that one, but I've found the offending checkin: ---------------------------- revision 1.52 date: 2002/11/14 01:29:00; author: akuchling; state: Exp; lines: +3 -3 [Bug #599248] ext module generation problem If you have source files srcdir1/foo.c and srcdir2/foo.c, the temporary .o for both files is written to build/temp./foo.o. This patch sets strip_dir to false for both calls to object_filename, so now the object files are written to temp./srcdir1/foo.o and .../srcdir2/foo.o. 2.2 bugfix candidate ---------------------------- Now to see if patching it up is easy... Cheers, M. -- ARTHUR: Ford, you're turning into a penguin, stop it. -- The Hitch-Hikers Guide to the Galaxy, Episode 2 From Paul.Moore@atosorigin.com Wed Dec 4 13:27:38 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 4 Dec 2002 13:27:38 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E4E@UKDCX001.uk.int.atosorigin.com> From: Martin v. L=F6wis [mailto:martin@v.loewis.de] > Just van Rossum writes: >> I'm working hard on a zipimport module that uses >> sys.import_hooks. It's going to be SO much cleaner than >> the existing zip import patch. > Ok. I then won't apply the existing patch until > a) your zipfile patch comes along to review, or > b) significant time has passed after 2.3a1 (for some > reasonable value for "significant") The only reservation I have about this is if the existing patch starts going "out of date" again. I can't get CVS to work on my slow dial-up connection, for anything other than a straight cvs checkout, so I'm reduced to full downloads and manually updating the patch. This is both tedious and error-prone. Just promised his patch in "a couple of days". If that's the timescale, I'm fine with this plan. But if it's a couple of weeks, then I'm going to be annoyed if the reintegration turns out to be complicated. Paul. From Paul.Moore@atosorigin.com Wed Dec 4 14:06:28 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 4 Dec 2002 14:06:28 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB8861996C@UKDCX001.uk.int.atosorigin.com> > My current sys.import_hooks patch implements #3 and I think > that's the most logical thing to do; after all sys.path may not > be meaningful for a hook. #2 makes little sense I guess. #1 > could be an option, but, given the current state of import.c, > would be a lot harder to implement. I also think that #3 is the > most efficient scheme, as the hook can save some local state > while traversing sys.path. Have you looked at Gordon McMillan's import utilities? He handles this by having a "shadowpath" in the manager which maps sys.path entries to "Owners", where each Owner is responsible for loading from one location (directory, zipfile, whatever). I'm convinced you're reinventing the wheel here, at least in part. (But reinventing it in C, rather than Python, so it's not a completely bad thing :-)) Paul. From just@letterror.com Wed Dec 4 14:51:38 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 15:51:38 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996C@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > Have you looked at Gordon McMillan's import utilities? He handles > this by having a "shadowpath" in the manager which maps sys.path > entries to "Owners", where each Owner is responsible for loading > from one location (directory, zipfile, whatever). > > I'm convinced you're reinventing the wheel here, at least in part. > (But reinventing it in C, rather than Python, so it's not a > completely bad thing :-)) In the meantime things have changed: death to sys.import_hooks, and yay to arbitrary objects on sys.path. I think this is more or less the same as iu.py, except that there's no need for a "shadowpath" (if not a string, the sys.path item _is_ the "owner"). Just From Paul.Moore@atosorigin.com Wed Dec 4 14:55:16 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 4 Dec 2002 14:55:16 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E51@UKDCX001.uk.int.atosorigin.com> From: Just van Rossum [mailto:just@letterror.com] > In the meantime things have changed: death to sys.import_hooks, > and yay to arbitrary objects on sys.path. I think this is more > or less the same as iu.py, except that there's no need for a > "shadowpath" (if not a string, the sys.path item _is_ the > "owner"). How do I put a zipfile on my PYTHONPATH environment variable? Paul. From Paul.Moore@atosorigin.com Wed Dec 4 15:06:51 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 4 Dec 2002 15:06:51 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E52@UKDCX001.uk.int.atosorigin.com> This is a multi-part message in MIME format. ------_=_NextPart_001_01C29BA6.C61AC446 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable From: Just van Rossum [mailto:just@letterror.com] > I think this is more or less the same as iu.py, > except that there's no need for a "shadowpath" Thinking further, what is the advantage of your code over iu.py, if this is true? Performance? Do you have figures? Paul. PS I attach a small test script I used to get a feel for the performance of Jim's patch. Unzip this file, then do cd test zip -0r ../test0 * zip -9r ../test9 * Create a copy of the standard library as Lib, and do cd Lib zip -0r ../stdlib0 * zip -9r ../stdlib9 * Then run test.py. Figures for me, with CVS + Jim's patch, on a 450MHz Athlon with 256M RAM running Windows 2000, were: Lib [[['Lib\\email']]]: 0.309 stdlib0.zip [[['stdlib0.zip\\email']]]: 0.866 stdlib9.zip [[['stdlib9.zip\\email']]]: 1.154 test [[test\testCR.pyc]]: 0.344 test0.zip [[test0.zip\testCR.py]]: 0.204 test9.zip [[test9.zip\testCR.py]]: 0.348 So import from zip is noticeably slower than directory imports on big zips, but *faster* on small ones. Compressed zipfiles are slower than uncompressed. ------_=_NextPart_001_01C29BA6.C61AC446 Content-Type: application/x-zip-compressed; name="test.zip" Content-Transfer-Encoding: base64 Content-Description: test.zip Content-Disposition: attachment; filename="test.zip" UEsDBAoAAAAAAF2lgy0AAAAAAAAAAAAAAAAFABEAdGVzdC9VVA0AB1IX7T0A9Os9URftPVBLAwQU AAIACAA4l4MtKBXJLB8AAAAiAAAADgARAHRlc3QvdGVzdENSLnB5VVQNAAes/uw9APTrPVEX7T1L SU1TKEktLtHQtOJVAIKCosy8EgUlzzywqHOQEi8AUEsDBBQAAgAIACaXgy2Voha5IwAAACYAAAAQ ABEAdGVzdC90ZXN0Q1JMRi5weVVUDQAHiP7sPQD06z1SF+09S0lNUyhJLS7R0LTi5VIAgoKizLwS BSXPPLCwc5CPmxIvFwBQSwMEFAACAAgAMZeDLd8DWIcfAAAAIgAAAA4AEQB0ZXN0L3Rlc3RMRi5w eVVUDQAHnv7sPQD06z1TF+09S0lNUyhJLS7R0LTiUgCCgqLMvBIFJc88sKiPmxIXAFBLAwQUAAIA CABGl4MtIqMenSAAAAAjAAAADwARAHRlc3QvdGVzdFBZQy5weVVUDQAHxP7sPQD06z1UF+09S0lN UyhJLS7R0LTiUgCCgqLMvBIFJc88sGhApLMSFwBQSwMEFAACAAgAhKSDLcEhu3RtAAAA6gAAABAA EQB0ZXN0L3Rlc3RQWUMucHljVVQNAAe4Fe09APTrPVUX7T2z/sTLdejfG9tkBihghNLFvEAihYGh hYEhCshgZAjWYAKKoKhjBqnjZABLu3sAFUPU+BVzAUnPPIWS1OKSgEhnDZB6NKKYH0iA5GOgivQK KotZoGIgs4uZwLb4aYA5MBk0Hn7jQGrtwRpAjgYAUEsDBBQAAgAIAAelgy02cU8gcAEAAEcEAAAH ABEAdGVzdC5weVVUDQAHrhbtPQD06z1pF+09tVLLasMwELwb/A9bg7ENqUnppQ3kFMgph9JbMUYo sdyIyJaRVEry9dVafgUnh0Ir8EOzu7MzK/GqkcqAPmvfI/ZNGmqOsEYgxd9slfue7xWsBMU0M5Us vgTTcbLyPbCrlApqWjHgdVvTxdMTO49JuHjp8h7WENnECGhdjMhF8H0HSdPCqTZUGf3NzTGOCImm XLgKJqYNM6zJXUqv3doYLHU+SiUrMBz1OuMHIQ+n3qFh2lghscX6dtemO35UZsnb2gEcm2YW76R0 XVhFubhCLIcZJ8hxfIrWnyx+Wi6nVhUTkhZxS5DMYGTp+7PDKAkencj7HhrFawNBqCHLQp3nKwhf 0ucygBDQ/sJJTkk7PEIWyJ9M54TP3w8KWTfvM2i3vZF1A3z72PxiqI4muYnvtsmd/HsR2/ufjsL1 tWdRcsGmZ4EubQb6jIMd3wcLCLQp7BVephfejNvXdtuPYHrNZzQYxEL8Diy4mXMMV8D3fgBQSwEC FgsKAAAAAABdpYMtAAAAAAAAAAAAAAAABQAJAAAAAAAAABAA/0EAAAAAdGVzdC9VVAUAB1IX7T1Q SwECFgsUAAIACAA4l4MtKBXJLB8AAAAiAAAADgAJAAAAAAABACAAtoE0AAAAdGVzdC90ZXN0Q1Iu cHlVVAUAB6z+7D1QSwECFgsUAAIACAAml4MtlaIWuSMAAAAmAAAAEAAJAAAAAAABACAAtoGQAAAA dGVzdC90ZXN0Q1JMRi5weVVUBQAHiP7sPVBLAQIWCxQAAgAIADGXgy3fA1iHHwAAACIAAAAOAAkA AAAAAAEAIAC2gfIAAAB0ZXN0L3Rlc3RMRi5weVVUBQAHnv7sPVBLAQIWCxQAAgAIAEaXgy0iox6d IAAAACMAAAAPAAkAAAAAAAEAIAC2gU4BAAB0ZXN0L3Rlc3RQWUMucHlVVAUAB8T+7D1QSwECFgsU AAIACACEpIMtwSG7dG0AAADqAAAAEAAJAAAAAAAAACAAtoGsAQAAdGVzdC90ZXN0UFlDLnB5Y1VU BQAHuBXtPVBLAQIWCxQAAgAIAAelgy02cU8gcAEAAEcEAAAHAAkAAAAAAAEAIAC2gVgCAAB0ZXN0 LnB5VVQFAAeuFu09UEsFBgAAAAAHAAcA2AEAAP4DAAAAAA== ------_=_NextPart_001_01C29BA6.C61AC446-- From skip@pobox.com Wed Dec 4 15:17:18 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 4 Dec 2002 09:17:18 -0600 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: References: Message-ID: <15854.7294.853169.73266@montanaro.dyndns.org> Jack> Distutils never ceases to amaze me, and it's path-mangling tricks Jack> are beyond me. :-) Jack> I'm pretty sure that if we somehow stuff Jack> "../Modules/regexmodule.c" (assuming we're building in a subdir of Jack> the main dir) into the list of module sources we'll get something Jack> like build/tmp.blabla/../Modules/regexmodule.o for the object Jack> filename. Jack> Does anyone else feel up to fixing this? I poked around a bit yesterday but didn't get too far. Deep distutils pondering is beyond my feeble brain. In PyBuildExt.build_extensions() in setup.py, you will see os.getcwd() tacked onto the front of the generated path in a couple places. That appears to be the cause of the absolute paths we see. I think removing it will be a necessary, but not sufficient, condition for a fix. Removing it will create a relative path, but I think it will still be incorrect. One other observation: in a separate build directory no Mac/Modules directory is created (though Mac/Python is). This is presumably where the Mac .o files belong. Skip From just@letterror.com Wed Dec 4 15:12:37 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 16:12:37 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E51@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > From: Just van Rossum [mailto:just@letterror.com] > > In the meantime things have changed: death to sys.import_hooks, > > and yay to arbitrary objects on sys.path. I think this is more > > or less the same as iu.py, except that there's no need for a > > "shadowpath" (if not a string, the sys.path item _is_ the > > "owner"). > > How do I put a zipfile on my PYTHONPATH environment variable? We do what Jython does: somewhere during initialisation, *.zip sys.path items will be replaced by zipimporter instances. Just From just@letterror.com Wed Dec 4 15:18:46 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 16:18:46 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E52@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > From: Just van Rossum [mailto:just@letterror.com] > > I think this is more or less the same as iu.py, > > except that there's no need for a "shadowpath" > > Thinking further, what is the advantage of your code > over iu.py, if this is true? Simplicity. I suspect that the shadowpath is only needed if sys.path items must be strings. I could be wrong, though. And it's more or less what Jython does, which is nice. It's simple and works well. (Samuele: pls correct me if I'm wrong ;-) > Performance? Do you have figures? No and no ;-) (Later, later: the new hook already works, now I need to get a proof-of-concept of zipimport going.) Just From Paul.Moore@atosorigin.com Wed Dec 4 16:28:02 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 4 Dec 2002 16:28:02 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E57@UKDCX001.uk.int.atosorigin.com> From: Just van Rossum [mailto:just@letterror.com] > We do what Jython does: somewhere during initialisation, > *.zip sys.path items will be replaced by zipimporter > instances. BTW, on Gordon's site, in the iu documentation, he mentions that "the assumption that sys.path is occupied solely by strings seems ineradicable". I don't know if he's got any specific issues in mind, but it might be worth checking out (get the screams of protest out of the way before the code goes in :-)) I've copied Gordon in case he can give any specifics... Paul. From gmcm@hypernet.com Wed Dec 4 17:07:32 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 4 Dec 2002 12:07:32 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E57@UKDCX001.uk.int.atosorigin.com> Message-ID: <3DEDF004.25379.630FD638@localhost> On 4 Dec 2002 at 16:28, Moore, Paul wrote: > BTW, on Gordon's site, in the iu documentation, he > mentions that "the assumption that sys.path is > occupied solely by strings seems ineradicable". I > don't know if he's got any specific issues in mind, > but it might be worth checking out (get the screams > of protest out of the way before the code goes in > :-)) > > I've copied Gordon in case he can give any > specifics... Code like this: for p in sys.path: x = os.path.join(p, ...) .... is very common (I patched linecache.py for this after imputil went into the std lib). Since PYTHONPATH can consist only of strings, it seems wise to tackle the issue (dealing with strings that describe non-directory collections of modules) instead of postponing it. Also seems sensible to make it so that if X works on PYTHONPATH, sys.path.append(X) should work, too. The shadowpath consists only of objects, and len(shadowpath) <= len(sys.path); it's either lazy or caching, depending on POV. [There's a metapath, too. That lets you put overrides in front. You can even override frozen and builtin modules.] -- Gordon http://www.mcmillan-inc.com/ From martin@v.loewis.de Wed Dec 4 19:56:15 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 04 Dec 2002 20:56:15 +0100 Subject: [Python-Dev] Make universal newlines non-optional In-Reply-To: <2F08AD4B-077A-11D7-93BC-0030655234CE@cwi.nl> References: <2F08AD4B-077A-11D7-93BC-0030655234CE@cwi.nl> Message-ID: Jack Jansen writes: > Currently you could simply disable it. If the ifdefs are gone then the > platform > maintainers will have to actually fix it. > > But maybe I'm too defensive, Ok. Then I'd propose to leave this for 2.3, and remove the option for 2.4. Regards, Martin From martin@v.loewis.de Wed Dec 4 19:58:58 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 04 Dec 2002 20:58:58 +0100 Subject: [Python-Dev] extension module .o files wind up in the wrong place In-Reply-To: <15854.7294.853169.73266@montanaro.dyndns.org> References: <15854.7294.853169.73266@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > I poked around a bit yesterday but didn't get too far. Deep distutils > pondering is beyond my feeble brain. In PyBuildExt.build_extensions() in > setup.py, you will see os.getcwd() tacked onto the front of the generated > path in a couple places. That appears to be the cause of the absolute paths > we see. I think removing it will be a necessary, but not sufficient, > condition for a fix. I now believe that Michael's analysis is correct: I.e. it is fine that setup.py produces absolute paths, since distutils is supposed to make them relative again. Regards, Martin From just@letterror.com Wed Dec 4 21:12:48 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 22:12:48 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEDF004.25379.630FD638@localhost> Message-ID: Gordon McMillan wrote: > Code like this: > for p in sys.path: > x = os.path.join(p, ...) > .... > is very common (I patched linecache.py for this after imputil went > into the std lib). Since PYTHONPATH can consist only of strings, it > seems wise to tackle the issue (dealing with strings that describe > non-directory collections of modules) instead of postponing it. Also > seems sensible to make it so that if X works on PYTHONPATH, > sys.path.append(X) should work, too. Erm, I'm not sure what you're saying... Are you saying that we should fix all cases where non-strings on sys.path cause problems, or are you saying that there's so much code out there assuming sys.path contains strings, and that we therefore should stick with strings? Both positions can be defended, and both have their problems. A) Stick with strings. Hooks can be implemented by subclassing str. This is great for hooks written in Python, but subclassing str in C is not straightforward. Things can still break, though: eg. os.path.basename(strsubinst) will return a regular string, not an instance of the subclass; might be an issue. B) Allow arbitrary objects on sys.path. Hooks are then easier to write (in C), but some code breakage will occur. The std library we can fix (if needed), but third-party code might break. I would very much prefer B, but if it turns out that we can't break the string assumption, I'd still be happy with A (rather that than nothing!). Regarding PYTHONPATH and sys.path.append("/path/to/my/archive.zip"): for now I'd suggest that the sys.path traversing code checks for a .zip extension, and replace the item with a zipimporter instance. This check can be very cheap. Later we could add a general extension-checking feature, where one could register an import hook for a specific extension. This might be a case of YAGNI, though... Just From skip@pobox.com Wed Dec 4 21:22:48 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 4 Dec 2002 15:22:48 -0600 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: <3DEDF004.25379.630FD638@localhost> Message-ID: <15854.29224.574966.648231@montanaro.dyndns.org> Just> Are you saying that we should fix all cases where non-strings on Just> sys.path cause problems, or are you saying that there's so much Just> code out there assuming sys.path contains strings, and that we Just> therefore should stick with strings? Just> Both positions can be defended, and both have their problems. ... I think there is a third position: make sys.path magic somehow (but have it still (appear to) be a list of strings) so that changes to it affect a behind-the-scenes list of objects which is normally used to do path-ish stuff. Skip From mal@lemburg.com Wed Dec 4 21:56:12 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 04 Dec 2002 22:56:12 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> Message-ID: <3DEE79FC.1020802@lemburg.com> Skip Montanaro wrote: > Just> Are you saying that we should fix all cases where non-strings on > Just> sys.path cause problems, or are you saying that there's so much > Just> code out there assuming sys.path contains strings, and that we > Just> therefore should stick with strings? > > Just> Both positions can be defended, and both have their problems. > > ... > > I think there is a third position: make sys.path magic somehow (but have it > still (appear to) be a list of strings) so that changes to it affect a > behind-the-scenes list of objects which is normally used to do path-ish > stuff. Instead of making imports even more magical, why not do what the subject line says: add more useful hooks to the existing C implementation of the import statement, e.g. provide hook callbacks to manage how the import mechanism looks up, finds and opens files and callbacks to manage extracting module data from the found entities. The only drawback I see for the current import hook situation is that each hook has to implement *all* the import functionality, since there's only one entry point: the top-level import statement. The existing machinery could be put to much better use (w/r to adding support for e.g. ZIP files) by adding more callbacks in appropriate places. Anyway, have fun, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From DavidA@ActiveState.com Wed Dec 4 22:02:47 2002 From: DavidA@ActiveState.com (David Ascher) Date: Wed, 04 Dec 2002 14:02:47 -0800 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEDF004.25379.630FD638@localhost> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> Message-ID: <3DEE7B87.5090602@ActiveState.com> Skip Montanaro wrote: > Just> Are you saying that we should fix all cases where non-strings on > Just> sys.path cause problems, or are you saying that there's so much > Just> code out there assuming sys.path contains strings, and that we > Just> therefore should stick with strings? > > Just> Both positions can be defended, and both have their problems. > > ... > > I think there is a third position: make sys.path magic somehow (but have it > still (appear to) be a list of strings) so that changes to it affect a > behind-the-scenes list of objects which is normally used to do path-ish > stuff. I'm not sure I understand Skip's proposal, which may be the same as the following, which strives for backwards compatibility: - Define an alternative path which can have non-strings on it, and define sys.path to be a "view" of the string elements in this superpath. IOW: assert sys.path == ['a', 'b', 'c'] assert sys.superpath == ['a', 'b', 'c'] sys.superpath.insert(0, CodeGenerator()) assert sys.superpath == [, 'a', 'b', 'c'] assert sys.path == ['a', 'b', 'c'] sys.path.insert(0, 'foo') assert sys.superpath == ['foo', , 'a', 'b', 'c'] assert sys.path == ['foo', 'a', 'b', 'c'] the superpath is used in by the import mechanism, and modifications to sys.path propagate back. It's not ideal, but I think it's backwards compatible. --da From skip@pobox.com Wed Dec 4 22:12:00 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 4 Dec 2002 16:12:00 -0600 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEE7B87.5090602@ActiveState.com> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> Message-ID: <15854.32176.422808.960868@montanaro.dyndns.org> >> I think there is a third position: make sys.path magic somehow (but >> have it still (appear to) be a list of strings) so that changes to it >> affect a behind-the-scenes list of objects which is normally used to >> do path-ish stuff. David> I'm not sure I understand Skip's proposal, which may be the same David> as the following, which strives for backwards compatibility: David> - Define an alternative path which can have non-strings on it, David> and define sys.path to be a "view" of the string elements in David> this superpath. Yeah, somethin' like that... Skip From rjones@ekit-inc.com Wed Dec 4 22:06:11 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Thu, 5 Dec 2002 09:06:11 +1100 Subject: [Python-Dev] PEP 301: Package Index and Metadata for Distutils In-Reply-To: <3DEDD21C.9030909@lemburg.com> References: <200212041547.14551.rjones@ekit-inc.com> <3DEDD21C.9030909@lemburg.com> Message-ID: <200212050906.11044.rjones@ekit-inc.com> On Wed, 4 Dec 2002 8:59 pm, M.-A. Lemburg wrote: > One nit: in order to make the setup.py script work for Python 1.5.2 > and up, you should provide a way which let's the distutils > versions of those platforms ignore the classification keyword > argument for setup(). The documentation of the classifiers keyword will include a specific warning about backwards compatibility. If there no objections, I'll include a code fragment that setup.py authors may use so their setup.py will run under python <2.2.3. Richard From just@letterror.com Wed Dec 4 22:32:26 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 23:32:26 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEE7B87.5090602@ActiveState.com> Message-ID: David Ascher wrote: > I'm not sure I understand Skip's proposal, which may be the same as the > following, which strives for backwards compatibility: > > - Define an alternative path which can have non-strings on it, and define > sys.path to be a "view" of the string elements in this superpath. > > IOW: > > assert sys.path == ['a', 'b', 'c'] > assert sys.superpath == ['a', 'b', 'c'] > sys.superpath.insert(0, CodeGenerator()) > assert sys.superpath == [, 'a', 'b', 'c'] > assert sys.path == ['a', 'b', 'c'] > sys.path.insert(0, 'foo') > assert sys.superpath == ['foo', , 'a', 'b', 'c'] > assert sys.path == ['foo', 'a', 'b', 'c'] > > the superpath is used in by the import mechanism, and modifications to > sys.path propagate back. > > It's not ideal, but I think it's backwards compatible. How is this different (besides being more complex ) from making import hooks string subclasses? (I'd still like to see more examples of code breaking if a sys.path item isn't a string. Not that I don't believe it, but I'd like to get an impression of the severeness of the damage.) Just From pedronis@bluewin.ch Wed Dec 4 22:23:09 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 4 Dec 2002 23:23:09 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> Message-ID: <037001c29be3$ba396400$6d94fea9@newmexico> From: "David Ascher" > Skip Montanaro wrote: > > > Just> Are you saying that we should fix all cases where non-strings on > > Just> sys.path cause problems, or are you saying that there's so much > > Just> code out there assuming sys.path contains strings, and that we > > Just> therefore should stick with strings? > > > > Just> Both positions can be defended, and both have their problems. > > > > ... > > > > I think there is a third position: make sys.path magic somehow (but have it > > still (appear to) be a list of strings) so that changes to it affect a > > behind-the-scenes list of objects which is normally used to do path-ish > > stuff. > > I'm not sure I understand Skip's proposal, which may be the same as the > following, which strives for backwards compatibility: > > - Define an alternative path which can have non-strings on it, and define > sys.path to be a "view" of the string elements in this superpath. > > IOW: > > assert sys.path == ['a', 'b', 'c'] > assert sys.superpath == ['a', 'b', 'c'] > sys.superpath.insert(0, CodeGenerator()) > assert sys.superpath == [, 'a', 'b', 'c'] > assert sys.path == ['a', 'b', 'c'] > sys.path.insert(0, 'foo') > assert sys.superpath == ['foo', , 'a', 'b', 'c'] > assert sys.path == ['foo', 'a', 'b', 'c'] > > the superpath is used in by the import mechanism, and modifications to > sys.path propagate back. > > It's not ideal, but I think it's backwards compatible. > I would it support this kind of code: sys.path = L L.append(...) ? regards. From fredrik@pythonware.com Wed Dec 4 22:42:14 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 4 Dec 2002 23:42:14 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <028701c29be6$665c3ee0$ced241d5@hagrid> Just van Rossum wrote: > (I'd still like to see more examples of code breaking if a sys.path item isn't a > string. Not that I don't believe it, but I'd like to get an impression of the > severeness of the damage.) >From the ImageFont module in PIL 1.1.3: def load_path(filename): "Load a font file, searching along the Python path." for dir in sys.path: try: return load(os.path.join(dir, filename)) except IOError: pass raise IOError, "cannot find font file" (this is fixed in 1.1.4) FWIW, I've done lots of stuff that scans sys.path directories for various purposes. It would probably be a good idea to use a string subclass for the zip importer, but I don't think we need to make stringness an absolute requirement. Let's make it *really* easy to use zip archives, and reasonably easy to use other custom importers. From just@letterror.com Wed Dec 4 22:49:30 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 4 Dec 2002 23:49:30 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <028701c29be6$665c3ee0$ced241d5@hagrid> Message-ID: Fredrik Lundh wrote: > FWIW, I've done lots of stuff that scans sys.path directories for > various purposes. It would probably be a good idea to use a string > subclass for the zip importer, but I don't think we need to make > stringness an absolute requirement. +1 > Let's make it *really* easy to use zip archives, and reasonably easy > to use other custom importers. +1 Just From bbum@codefab.com Wed Dec 4 23:25:53 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Wed, 4 Dec 2002 18:25:53 -0500 Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies Message-ID: I'm writing a Transport class to handle proxy servers such that the xmlrpclib module will be able to communicate with a server that is on the other side of a proxy somewhere. Most of the difficulty in doing so is the need to navigate the urllib vs. urllib2 vs. httplib maze. I believe I can do what I need entirely via urllib, but may use urllib2 because the capabilities better fit the needs of my ultimate use. However, I would like to create a Transport class/subclass that can be dropped into the existing xmlrpclib module in the core such that this problem is solved once and for all. As such, I would really like some feedback from the python-dev community as to ensure that the efforts I put forth to solve my particular needs are applicable outside of those needs... thanks! b.bum From DavidA@ActiveState.com Wed Dec 4 23:49:34 2002 From: DavidA@ActiveState.com (David Ascher) Date: Wed, 04 Dec 2002 15:49:34 -0800 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEDF004.25379.630FD638@localhost> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> Message-ID: <3DEE948E.3010802@ActiveState.com> Samuele Pedroni wrote: > I would it support this kind of code: > > sys.path = L > L.append(...) ? Sure -- sys.path wouldn't be a "real" list object, just one that behaved as one. Just: > How is this different (besides being more complex ) from making > import hooks string subclasses? It doesn't require the import hook to be a string subclass, which you said was hard to do if the import hook is to be written in C. If making it easy to write a C import hook isn't a requirement anymore, let me know! --david From just@letterror.com Thu Dec 5 01:00:23 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 02:00:23 +0100 Subject: [Python-Dev] New Import Hook & New Zip Importer Message-ID: --643623-2675965952-3248042441=:9000 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit Phew, it works. The attached tgz file contains: zipper.patch: patches setup.py, Python/import.c, Python/importdl.h zipimport.c: zipimport module, move to Modules/ Do I need to mention it's a rough patch ? import.c changes: - Allow arbitrary objects on sys.path. - If a path entry ends in ".zip", replace that entry with a zipimporter instance. (obviously, the zipimport module must be available for this to work.) - The search algo is: [handle builtin modules and frozen] for p on sys.path: if hasattr(p, "find_module"): handler = p.find_module(name) if handler is None: continue if isinstance(p, str): [do builtin import] zipimport.c implements: - The zipimporter object; explicit use example: sys.path.append(zipimport.zipimporter("myarchive.zip")) this is effectively equivalent to sys.path.append("myarchive.zip") - the zipimporter has two methods: find_module(subname) -> None or self load_module(fullname) -> module import.c todo/notes: - clean up, refactor, debug, document The patch adds two new static functions, find_module2() and load_module2(). Ideally these should _replace_ find_module and load_module, but since the signature of the new ones is different I decided to keep the originals (which are reimplemented to call the new ones) to keep the patch small for easier review. zipimport.c todo: - rework the zipimporter class as a subclass of str - doco - add a get_source(fullname) method (linecache.py could use that!) Have fun! Just --643623-2675965952-3248042441=:9000 Content-Type: application/octet-stream; Name="zipper.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000" Content-Transfer-Encoding: base64 Content-Disposition: attachment; Filename="zipper.tgz" H4sIAAAAAAAAA+w8a3fbRq79Kv2KiXJqSzbtSPIjibXJHsWWG986lo/tdNNmc3ho amRxI5EMScVxG//3C2CefEhWbl8f7uq0kTkcYDAABgNgMPo1iGOePPnuz/y027vt p3t78N1uP90X3+3urvgWn++gw9Odvc7+/u7T79qdTnf/6Xds70+lSn7maeYljH33 H/hjWT/oNh7/FQT9tZ9fhfzhK5jFUZJt+3/8GO1Ou72/u7tA/rud9tPdgvx3dp/u fMfafzwp5c//c/k/DkJ/Oh9x1ji/yyZRuD1p1E1bmiVzP5vx2TVP8m+idMTHab5t 5iXpxJvmG/1oFgdTjo31+mMACkLOzn92B++uWNtqOKSWjtUypJaubnnVP6KWrQ5g AnFkgc+CMGMp9xJ/4kbJiCfvP7AX7DeJzVFIHDngfc+Gm3ifufvrNLgGkHYPcD7Z YHod8IRF1//hfsZo9CALopB54Yil8xjfs40n9Xp2F3N4zQSbcsC/BPGJ/LuH1JY6 /Favnd8NaQj39aB/ZB7ZBs4n+Mx7jAFJsZdNQm/GWTRm2YQjEiY7IBEWWBb5AIIf AMu86ynB+FGY8TBL2SjwswJEnMDkvvQERGM8n07vtj/NvWkwDvhoO/YSAIQv/6N3 w7eRiO0GYrjvaQGc310BEyQ+a9IuNveIp+MoufWSETDSnyK0hlRU3PDMBdqb/gRW opp8q1fuOI28kQvzd2fRaD7lTWs8tpHy6dhhAgfOBMl16jX4sByLXJhTcgfotWLZ ZB9OuP+xGcUtDUQzUc0OWytOslUvE1q3JO2G/LaZY9OGP00dZov7Jvf88XaUtuqg IKX59Wzhjeaz2R20yNHFzD/eToM0E8ugIVnZcFgDJo5fQuDwVxtlWAvGrPno/K6f 3LjnsHj51Tye8n44+pHf3cJ6SpuCNCQIgIePvg6HB9bUGoK/NSZGBeac311mSRDe EGfgmWhUveST/G61oDnh2TwJ2dnb01OkB+cIlOcF2wJ2bb3MYtebTlFLkHltkB9R TxCPXhCGFi4qwrH1Uq2QFxK3agc+lNoEU0zzfV3R1TTcbgn+39umJydmNBJVGrmC nP8+CW67RLfrPizKHF+1SDVXdQ9BoC3drQ5Sen7nnpwdXgyOmzlMLdJp913uncRh yVhKrixoIVBlQjTR/VT8URgMUS7AiTOyCEYt4NOU01BF2gGulVOUtq0Zn6NglFON EReqW9IOkj7O/miwmDO5d3LkUrvhmHiOrl3cnWjdjBPOqVvLpnKBtQKLOFL21XSB vbCoydUWqrx6ATRntsguGxO9UIml5oLO5hTWok/orNjxSuqKuKtMjBG+XPbSeLC1 NVZsBy6fRSEX6qYIZth+NkeXyO2PRjlkDqNhlZIZkLKOGVujtcwaQXDGVjz1Uuid mMf53RsvjnHSr70UVnzTWo26f6tCg4nChTYOiYJ9WzCZhVEGW/gcXB/hOyg8ijk9 vQhkywpKRhv5n6lk2gGALd+xtn34G4Y1ph79wOBbVdCifqkKGhGU1VBTROokPY0f eHaS8VmlGKUttOCMTlWoOMyFpiAdGuP3sK9fmXp3GfzK7Veg8jtKXQZJ4l7yTFpR eP7i07ygPUqE+cctiX32pnOeshmET+yaswwRp+h3Tnl4k03YToOU7SbKIjYi7SD1 BaeQNQPyvFnA/sF24Gtz0x78cAp+fZOAM2KSoFnxyBJpoBdchlxRiok0ovMbhHOe 60DLHdhQ5FEFb4gnTwVZqzGl1uBTPiOHGx32FTj0VHAoxyJcgjWhqUWnV3kVxY1O a4rDMs0QRPHIMj7XCfc+WhaEhjAG8JGY4tD350nCR03S3Oppi8UnJg5RnheuZ0Sq MhvjJJrZsQpOEid3QDZE7l22O660GODzFuQNh8B0dARxlm1CZtSaCj+pXvutkdsY 2IItzWFvBlev3Z/6F/2LHy6B8Hfv3jXuHYTPrWq2wFothkcWOrUaft2j9Uw5ah6f luKlNxRNl6eDrdZ0jOd35Q5f/c/gEKLYaDxOeRaNbePnKAa30CWkmQhPET8PgaI3 IXpKUO1dPggqHQ5HgdLEaabnd0cg1surC+BSM+cGgUGrI8dAy30MCyvC7YYdmB0N jgcXF4Mjt390dDG4vGzid4u1V4s+BSdzQTbsXSdXzRLaNYFHhHOg8m2YUMOkxAoR TwqWIc+NloMwtPox8o7dACwUdqMtszniIv6PklaFV+hIGPlIIDlkcYK7VKkZXF4v y5Lyi3TRC0zFQEhffpHwuKK7l7ohuTmVr1L+ac5DvwIbvJwJt6T8buKlkwrCYOYV 85BzsLbHkCeBDztAH+bnFBgRLeRExRug8Xo+HnM1gnt1fnza/+ESDNNx/+3pFcON QDe+6l8Orn4+HzhMQI+n3k1KkEX11gOMqiSZJd5nWOQVPPNxs6sQTOBPFkrtFkw5 xmpifZbfB1mV6LA15F+yEvnSoJopyIaKjmSq7I7UUKmilZRde1U80PmpfCtP/QQx LXpVOQLisvjSxCA3TsDaFSN2rUX4IJWBbIFUtr5Yolpv9BItZHhMF3jIq+0RnzqG MgzG1J4ALYBDuAseZRgxq2jlFzCsBVnJzBhmAskhFo+jKINdE/4FQwdrBXaBpIld HLa+va78RXpd8hRlKrUngw9yGAGFP4sRALa17fiuYbutIn+6tL9fADh8ECIqQAwl RJHIXM6FQn2Z8oV9nX9pYiMwiTgj/fml7qWIAK2ccfABWUQoTHwW5OJ7zAjco8Dq sFeqZIPaeNnWS9xJ6/juB2gImUdZW2iEvywfyGHX82A6gkbUT4cpM0m+2x3TmdcW YlLhF7h0KWJCVxJ9pYCn2zRUn5lgIEgBaTYB3doi/9Jh3PMnDPch9DRvJwE+pYwH 2QSsAvrHDFjkIZ6UA8kC7IAQ46ep0s7CPWCPaUZuNHZVZm2TNR414F9qD0LVrhAw tFsJT1ONQDWgnn8E0fVYm5yAeaje8JGGHnmZ5+IW6kho2k5hKnZv6qVBxgG47VUg SwGElXAIQFoMAMF+woXF4yFqKs4vSOOPN456AujrKJqCS5XMSd0xVS9FSMKRWXSQ fKJRoBm2Po9Rv8HfColqseQcVjzMUNAtktUVDENyJ1mn6pDAaAbJleEGImWfos0C HUyNqO981wqnYGHmH80TKSY6/DDBW74OSJN5GKIKByGL4iyYAcspBAAh5EYooPQX j4BAZ1GG0/IwXRSB2QxmHAUxi5mPoRoNmFKsdICkeGwDDRAJFF9QugLxEApoiKaw zvEpJO5gb0f/5WOPNAvAB0FPn4+261WnFPXqY4r6b4UjmGLKwSQbjk9OB2xjHMOf 0wimYFaInwCYpfKWKueUFLpAE7JD4ZhwoDjRHXDFSkD5Bh9U58BhoKEi7IRBgUuZ zIGAJjpCo3tye8Fl/f5N/915/+r16eAMFvreB/UOR1n0biN21NZEhjZLYDxjKl8y A7cs1TAER2k8jW7tdANaUrKrlCQL0CpGDCfWsPNZJrGG+018J/dEk1SF3SEGoYyj 2BAGG1Jy3VCbJr63M80LIuChin61L3DARCCMqMnwo/QWUAcC4x9hKIdtdbsO67Z6 6CFcQqMwPDwc4WomnUZ/QiX86IzVvQDpnsLUj6HrMXQBRJSraH9p77f3dl/ttQXp gPKVNzpgA4HtEFXSm7KjAAxBcBN6QJRAXxv70yglPL0Vg36coJo3Zgm9B6ZMWS9K 5RzB9uee8dumldZaktCyWLVPnKrXcnpPOJdxpkcJzUtlyi0ucAzL7ogDMh3zgjVR p1pSc8npQuXG1SJOioV30evJvLAmrrAW2y1xgKuHJVGKToLl01UIJ/5MpWy77Q7J 1uRytIRt0SI4e22NtIBIWLmdtjxNqylzVCLqcgIiLlJFNnmlnmi0VusJZnAljmhL uVJvbUxX6q0t6GokW1YWAHb3gaMGw2adcsMPoFmxW2+pGHe7SozWhrGyehmSc/YZ tczmh3mFq7IWWwcVRYdbgwnHGzFtxJubuLpwm2jBZuorCjZigkNN7o9GaPhwgw/B J2bXd5k0T4UJv8jvb9ABnSlxEIgRU+4chl69UAEFEQNjockiR4B8AKAfQ5IeSz+C EaMhreyxsKSUJ0ZffOZ9BIcETSc6ERASwrqzVzfsTX6UJFwGtJbcctu5ENi3W4Hd 9o62AppGbQhOiZwlJsDWkE3W3bcJ2WlD0x+ksEiR4I1ki06J5ZQUZDnt1UqGklxw 6kyuiLDOmB5df7ROZ/dZEqrtHWiW1hsMmvCCHEtb2ZZ6vcU6LSVNcfqFY7WkRxFS wGYFXmL8fIgtkFOITebdm956d+BAzn2fc5kuqW0ImPV/t9f1iXMJPpeZpwM3Uihw UjEEQAyB8FdpkVUrZn4VQpwHlDc3YnligeNukDOz/mSdhhGLDUanIwdYkeqwAVPW sNXHTFUEiKHIJ5Tb3mJGyCni+1wtBkKIIWrlDIAuPZBHlNR7a6snyMwz0BDSocd7 olqQfcFhrY1ydVEifIvQ5YelKAVK88mfu5EfIk+UpKNDHnzOepSP3OzjIOuYqoDc fYXB/k94+tNsDIfDhqOOpsp/tDSPJFGX1UQ5zDqu+wcTJkCcHHH0ynqCsMrzFWJa ViYtDegD5AlP2YQlJiKhYdjCuETGwphCKZ20teolCvUZW/ksr5TdQZzqRKsiphJ+ GekY+tS2Dyu46f7Ek2toPJ56N+I06/Iudf+VQLh8mYFRSpqNx8z2ZD3Ygr6fjmTi hVzx79N/h+Kkl0azgwnlqmLtXZ0mWDzgKpRrVB15iblUnpojP0ZcicQdz0O/ifUl hbhTqLhw3YSTjmr9hlpRn5qt3PE4Vj2iGZTVj5ULQaJEPx966dhIgCxw1U0lgDnR xly9RImwgM/Mp7GoIgWnjWeeqIH5gBvEINIv4I0bTcUnK37GR0tX9aPQ1wLvEu+W xsG4Fe2GmBs+aTp1cHs9H+cDeQqeE4iG0FNJXXDJR7Kc9MEo89E4tsPL4yiZeVkh thQbbSHAJM+FQkzaKL9PGzmNrIi7gGzlebGNFp1Bum9ETZLxpTdpbyTSqP/DETCg geCpEKLnI2EaBAMAcdwiAsN8qFkmFxiKrFviNJFXh71eCBuY4/4Yv3EODjoDdlqF BiwZCUT09astQdjENNgqKYAFkydciKghY2rk+zFWYwFxi0X1Xo9N7s4v68LRSL1b sZ8FlFtApoDHStuaBiBDqBSaVrYsE0DHTPzZD0dU20D8MbMk+1CkD7ljsC0tODHR I4qECCYooBfV1crB0j4sMSjkiAUg3urkLrsNwF0jY6OSH490wfZSkVgpigViMSlk IrHHGrIjjkaJjM9eMMX6aat2RW1esMsYuyADjqKB1ocv3OaKYl4OnZaTNJiHsGCO AQeeBlnwWIQ0hWVubNVWZ0/uLHJjMZ1bVTuRgrS2HcH4Svs7D2Uxv+tHI94sVFUz cx4lnq2iZBwib18Rg21AbbXU1SuKNCroV9G37ETqqjpQIhMTyDPvJvCLGx62idq8 ZktpJirz+/YHtgah03hMXnfTwqCav34VOwt17+S7N033ly8Ze9aqhuoug+rsL4Da WQbV3dVQizeLvMrT9th4haU4hEUe4oPV+H67226n0tHTPm4pM4kCK8WkQprGjNAe scmeOUJcW8CUnrIEI77YQ5DFaYfQSdZdYX9TpSjVlRp71fPNVVyJ2aIDJ2+a6Aok MVtlgDxGdMkqEw25nBOyQWhw5UIZB1/caRByV2QtUqucMY3mic+tA9tP4FPEldov u0r914nI/Aownaw6x+ALH7nija5uzcRRMMRm/04gyiyLwvS34au3i2asF1K+88Ld 4FPlFG3gVjGHasWpRDLFVxufKAAM13VstNGMyUsRHUPVUYWxNZnClTHkvQx58ABa 49uIe4Xx2uu5yjhCxj6piKKpS8p1kWxuJlRpLQ2WGN3Y17xwKtVH6qzs9A121lKu vKV1ykpREHJJZS3tW13K2ki4h2IOehdeKnlT0Sp2Z4An7y4I43lWSQLYRAlbMBFF jVppvX77FaLK20N5vsvoQbIfMGPwABGV3vWUBOUCr4qwV4itNTpzueWh+w5q6WqI h4qGCwXQVmSu8gQHBRbmSqCNqq6ZSa5Zs1wz07TAchNekzNew9C/glLpMuVCRPta Sr744EEOa2dtuY+bgkPqTyiXTIroYw2TOKM/oDQELYfCejZKZBijHBltcSSmQ4lK Pg4LmAse2VLMpV1Y2EkqJBw3G4OLi+GFTGrkN0C5255jx6bOGSlCR3zszafZwUq+ dy5ovQZnBJPwVArdkKdNI/t2lilMtl1a5RQ+5FXUZa209gUHX7iPLobIgAy+WPwS q9SedtnrWDl1JCYIvMQCmYRrF0RXP5vUUVlienWK3ObaGrPLtdXxLeajXJeqbFyq AzKFJRRJWbc5oQ9WgZAKqLwEDgQbE7GczvWpbtc6vf9A+XDZTaR38dzTvnigz4iw xybrspcvSsf4q4ZjuXhM33jJn+QX0oX3drZf3rMx3JR0EaCcnGz6YDLd+RfoRHyw ssuKcQsDMfuOTOkCqAwzh0P0rPO35Jh1k8Pc0CJ2mjEXpkmpIJL8cBI3BjiUywPp ExqRJqlMGIv6sobSG327jxWTv++HH6QRQGJa0uVW60G3K6pxyEcy51KKi+0C/nIA KqlekPLEnCaj8wCMwqtSnKWcpVzq4kuwqKlylTVRWolrM5dDMNkD+/Z3Jz9r6tTL 380qXBYvZ//vvynh7BryYc6gUQeCTmMuzID/ZA2TkDhgjbdn/Z/6J6f9V6eDYv7U vioxi0t3JHJ3FKouKRRK92fxspp9aYWoYp+uPKL49NsFWWrtGSl2UbEtBph3zfLV anXEIZTGALlDWWtGTFaW8oLL6mo60MXL+viXqP6u5So+28gRU3Caf9kRL1X96r21 u5C0hKrtNo0ho/siht/yiVgnFp1IKJ//fPV6eOb2z0/cnwYXlyfDM1ScB1Y5uSbU pZBBGoNKGI4stgG52wtsqSXT/M4PoFclShpbYV2j3v3dv13x38/v/5jff4GvbbD3 /uQPH2P577909/Z29uXvv3Q7+90O/v7Lfmfvv7//8ld8TvCQ9QA2gmweg7Gsv/j9 n/rF4SUFWAfsif85TaIoexLTj8uor1GQZk/SxH+ihnU+18HEJwH/jJU/CXxRlXhn u9Pt1EfBeMy2fLaV0LOhdWNjQz/Uuk/ZWfSZdUGhWGfnYBf+22dbqGA1gWZra8v0 3mNH3Bed223svNcVnRGp/aFBus/3nZ12h1ED4hENAAIfXc1N9dzptgcrKRw12eBL xkOcRnPdnURZCv+vO+y9ftj21z+0WGsVcLzpA96jAJcPq4N/wWMoTPikhMA8iv1b Idp8EJHeSQiP9ZNRihKLmMfMSxLvTuZb0xXIpP6Emf7KEyfVVPxG0RMx7mi6PfmL 9bU0/gLF7W53nll6i4/MIhp1qoSq1t1nx/xaanDnYLdzsLMrNZjwoeKVob5Nkzv7 Trcr9Jix2vmPP7hHJxeDw6vhxc8Othy6r96enF6dnNHT+c/u8cXwlwE8PaKnw+HR 4GJwOXx7cTjAE743ns+icEqHkIyBB0k6IH/tCFlKV7TAP0PacewdtWR+39hOcXDo cvLm3H09HP64nJAqRdr2/0412vYXKlG308lpETwzTXJJibb9Wve5MYJd0IanB7tt rUKIraRDAPRtGvSsve886+wYWyga9pVg63T0eyx/84n0ovp3nGQCVmQ0HSZqKlT2 lWo2wN3cXArcLUGbbgqDHr6kDBv2T50oRBYC1EKVolSvMXvoZmq8DZvCpfi7lQNs mgGs9g0bfZF8d5xEv/JQIhcPErfM1igRnOJlfHPvCMMhdbUt0zVx8jfO/nVy9ZrR 73q8GZxdDY6qJf/8mfO8bUn+ebvjPN/ZU5IvsxN2mlIOXdaYqcNgxVKsD5nykJiO 7JAciN1xjDvLb8grg2QCs5niLXT1l7jCrNKJ0LcsjYSn9EZm5PFvfEyoaionKkO2 IHZND0fkEBaqjODWuT7KMpfbEOgL9On00gU8N+VLgzHf2yQCBfLUnmAu+wLP9/hP 9emijdWMFf9fBpBXakhwghMy0UPiUQmU3Lys2rcyIaoMRMjmXjBt4YKCdxVrarGW WWsKr54LIeKwJQ3bBB6A5tH1UYeFCI4rqSaVEzXTZFJZlXqNR7HmXtXqed5pO8+7 bWv17Had53t69dQWzXo8cvHWaBaE9Itg+Btg8L/eLO97DwArA2AB5/ZegaDqXtdm 54NZMVipUH6VW7y5GwKbnf9t78h22zaCz/oLykVrq1ZckyItKUYKFKhTCGgOVDHa 9EWQbToRHB0QHaUu4H/vnLuzPBQnMdw8RA9BTC6XO7Nz7VzEeb8DSuNmDhd7L1+/ GE9ejJOOQWwx3eQXE4dVe9Gimzear3MGCSMaXoBBzEtcyrQo8vX1ntto50gXkWjL 0ZRkqvVoKPPvVJFWu8dprztMh86+IsdysFgMReMNjnJ6Cc6LMa7/NqoGy21OuvsS 5R8u9Xm+fmtW35wgQrTXB1E+SB3tfelCPXs1JBHc2+pr0Z4B2o96Du23utsMlV+r AjQrlKX2XIusuyJcHrRrpiSjCpLuZTt5twawW8OB263Ph+9z9+kLgL4v0qjf+aOs O+wboRqD2dpFd5Mn7UtWin/OFpOnsNw/8jdghufr/EJCBhKuBvmN5fmXXRFmsiQp B22HSG6R2kC7YcW68uNg7uyUgIQXeuhEjNXCOADFMcgsjHCSig8T536oFl+R/tKO B8xWPkdkdWM43d7a0NJ/B/S4LlusR2esrIO8I6eGJ2+XyyuR9dgMF/T96Xj0/LfJ 6fMRntkcGl/enC5m5z7va9Nx6JRF+REnC4py+wu/jHU+eK5BDAwBUcO+RVTSB0Rl hhiYdkW6ed2h1S9w/Cren52/A12CR5d9WXnbbSit/OSf6fm1LJ82v4yNUk83kwG/ AQVsG1UJYr2YtbO0Q/vM4l+N2ZaabfVxy/KUmMWx44SPTuoTasLRbkitsduq2Lt2 aNg9UcY3WSn4Mv7fk0jHtpy1ogf722N3M+A2gwsnQNyMeue2Ai08pgDKTSILHRZE 9/YjHuRouUQRSAxKyVqZJaPZLgoz+TZGuGhIfT/ywWkwScanT5+O/hqP/j7BYLuI JGUXY1Ij53TqLU4QgwnKwoAjjmK4NIw9R9RMFgKCZ8kNdZGezhYFxcz5WC8o2TeA /BylmMaAtiL8+Simxiq7q117MeGLM7xID6Mxrjd7fPPf4ImULx7sOlKirtLYkhI7 P2hbADzxowlI5KTsawPIFKKjEKSJieEZkgQYjdYxWwLMLuhnedc/V+ZZ21Q8TC53 S6yjtpCSPI3ajmmNUsZDGkT/dGoT6ZaBISjmDSHDl5ZUFnWB6AlQvIPFb5u697tB dgk88dZ3s0IZSxMyV0kmdNAlGpBwZu3z5rBqJN0MU0f3y3wsKprV23x6jnWixVvi gJ/YzgduG6/y/OL96jE33iluigOyzijFg2o2qO8WbAnIrSZ2HaJaj/3x4Uckc2xd UsDJZRl9wKJQLGDA3ABMBTpbUkeX6/UNaq5rTKFZXumTVM0K9+cUtQdpRylpReeA B5DPtOXk0ojWN1q8fDc9z7EhHpsBJ68mo1cnz5wdAIjCx6ylUB5ybBT+s+m5GF38 dmGiak7tRssbsfDEybkmfQHA+D+eiJ/On25fT9RPrI7gZjGnCsNOWTZzPRyYUvbp sLDkHfa6cXzorfgH2F1hkhrTRa3zz9z/jxBAuwF9XyUZ3M1854KoByOZegGR9FBA DIw+B4oCqoqNhRtiwAZNHhILW84z8WGKYKQBGD0wS+Je34OBRgHZo5R9Il5lfD0a BuaHeV7IMoWrW5eaOJ0E02knyys+dVs/npjBgG2q4gdjgjJv55gCkRc6xX2iyjvj LKLQ6KzHU4bmW2bNtzhFPKV3xNOn4OCBgG2miriXIWj2LJ8kcCnpGWM1OHCoJ6R0 Fr/7YsODeKOTIU6RYNPMxP/jrJd24yxz0rxaaBBF1SgX87hWDwdREPQ8X9PXRJqC HVuhCR6Yl+MJW6MUrbk2mVa3Oi+Tusxo4IOWpu78Oi+/5n5W3fgGJ/tRTfCujJSa KI9HjvjpK4Xvx6Wrx+rNz9drE/WxvmM+ulCrAmwWDAdWPWgympoqFT3YLE0x2jlb F9fcmo5bzqHizneLCFt5uDrtPWqaFy3ATozoXCwyRisLCMIm2Z8lIOizxLqBjtC7 cZQa9vDN+efCjpJWHxGsJN8UyMdyQikFVre7L2wMpaUZxVEU9M4WoUFHQj9jYMxX /AbNmdduiqrrQmCT0BSC6woU7uJJJmnFy/+V7K6rxfIDnBs+oKmlef1cQsgUQhUI 3190durdT/GwD+JqOPDpFdX9IHklOcXtAPkcccJrTdEWnLA27MS3lIGMb49TleC0 34V/+lvIpHlZPtDqQmKfuoh6XMGiYvRpO1yJ8URSuM3sx2m6nKSPlzjEZsPMxfsz G6glSWrRhn7dVUf9nE63SU8359+9WJW96OrkQRICyslXSIzP2DSoElOnw6iOD4+6 SRwnxhorQVUBISnD4IPND7X0ug0CskeqOTSxrNpPf1iV7z7/wRC3ha+tGvRVG5fq bQcIHv0s6sU73MWbZls16E2pkTHexUhcQqM1GKMroFusdJNGqNSWqZC+qHAOx9V0 uX/X1ImAdX4J8hqbmSPru7ZL/P0zl0EhGWmy1XEGW90rOdK+ED9JCUGOFCySmEGr Sngr8lQiz+scs75WAgVtQc1iqcXI2bvZG1BWiBT2rAFW+BFswMKOG1duTSdS9HuI Sj5w/i0hyPBbNeUuM8H3atQtVpPVXnkAdjP0495+PaTyf2f3fvt9+23//QdKoPcA AHgAAA== --643623-2675965952-3248042441=:9000-- From bac@OCF.Berkeley.EDU Thu Dec 5 01:55:23 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 4 Dec 2002 17:55:23 -0800 (PST) Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: <200212040844.gB48iLx06213@pcp02138704pcs.reston01.va.comcast.net> References: <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net> <200212040844.gB48iLx06213@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > > Do you prefer I do a separate ``dummy_threading`` module or follow > > Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` > > substitute itself? > > Two separate modules. They have different APIs and it wouldn't be > wise to export a different API in the dummy case than in the real > case. (I'm sure you can figure out why.) > I would hope I could. =) OK, so as I said in the response to Tim's email, anyone who was actually going to look at the patch can hold off until I email the list again when I have ``dummy_threading`` done. And if someone can delete the old files from the patch I would appreciate it (almost all of them will have to be regenerated anyway). -Brett From tim.one@comcast.net Thu Dec 5 02:04:26 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 04 Dec 2002 21:04:26 -0500 Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: Message-ID: [Brett Cannon] > OK, so as I said in the response to Tim's email, anyone who was > actually going to look at the patch can hold off until I email the > list again when I have ``dummy_threading`` done. I intend to review it, but can't promise to get to it in a timely fashion. > And if someone can delete the old files from the patch I would > appreciate it (almost all of them will have to be regenerated anyway). OK, they're deleted. I would have done it the first time you asked, except the "almost all" bit (which was there the first time too) seems to imply that you don't want *all* of the deleted. So applying wisdom worthy of Solomon, I just deleted all of them . From bac@OCF.Berkeley.EDU Thu Dec 5 02:12:00 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 4 Dec 2002 18:12:00 -0800 (PST) Subject: [Python-Dev] Can someone look at dummy_thread (#622537)? In-Reply-To: References: Message-ID: [Tim Peters] > [Brett Cannon] > I intend to review it, but can't promise to get to it in a timely fashion. > Cool. Thanks, Tim. > > And if someone can delete the old files from the patch I would > > appreciate it (almost all of them will have to be regenerated anyway). > > OK, they're deleted. I would have done it the first time you asked, except > the "almost all" bit (which was there the first time too) seems to imply > that you don't want *all* of the deleted. So applying wisdom worthy of > Solomon, I just deleted all of them . > I only hesitated because I don't see ``dummy_thread`` from changing, but because I had a previous patch have a lot of issues because old files were not deleted, I am glad you just went ahead and used your wisdom. -Brett From fredrik@pythonware.com Thu Dec 5 08:28:03 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 09:28:03 +0100 Subject: [Python-Dev] New Import Hook & New Zip Importer References: Message-ID: <00d301c29c38$3cfdc1c0$ced241d5@hagrid> Just van Rossum wrote: > zipimport.c todo: > - rework the zipimporter class as a subclass of str > - doco > - add a get_source(fullname) method (linecache.py could use that!) have you tested the following code with uncompressed PY files? buf[data_size] = 'Z'; /* saw this in zipfile.py */ data_size++; raw_data = PyString_FromStringAndSize(buf, data_size); PyMem_Free(buf); if (raw_data == NULL) return NULL; if (compress == 0) /* data is not compressed */ return raw_data; From fredrik@pythonware.com Thu Dec 5 08:31:52 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 09:31:52 +0100 Subject: [Python-Dev] New Import Hook & New Zip Importer References: Message-ID: <00d901c29c38$c4eca560$ced241d5@hagrid> just wrote: > zipimport.c todo: > - rework the zipimporter class as a subclass of str > - doco > - add a get_source(fullname) method (linecache.py could use that!) unmarshal_code should check that PyString_Size(data) is at least eight (or perhaps nine) bytes. From mal@lemburg.com Thu Dec 5 09:48:14 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 10:48:14 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <3DEF20DE.7010803@lemburg.com> Just van Rossum wrote: > Fredrik Lundh wrote: > >>FWIW, I've done lots of stuff that scans sys.path directories for >>various purposes. It would probably be a good idea to use a string >>subclass for the zip importer, but I don't think we need to make >>stringness an absolute requirement. > > +1 > >>Let's make it *really* easy to use zip archives, and reasonably easy >>to use other custom importers. > > +1 Why not leave sys.path as list of ordinary strings and add all the extra support to the existing import mechanism via hooks at the right places ?! E.g. such a hook may scan sys.path for files which end in '.zip' and then redirect the lookups into their ZIP archive directory. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Paul.Moore@atosorigin.com Thu Dec 5 09:07:41 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Thu, 5 Dec 2002 09:07:41 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB8861996F@UKDCX001.uk.int.atosorigin.com> From: Just van Rossum [mailto:just@letterror.com] > Gordon McMillan wrote: > > Code like this: > > for p in sys.path: > > x =3D os.path.join(p, ...) > > .... > > is very common (I patched linecache.py for this after imputil > > went into the std lib). Since PYTHONPATH can consist only of > > strings, it seems wise to tackle the issue (dealing with strings > > that describe non-directory collections of modules) instead of > > postponing it. Also seems sensible to make it so that if X works > > on PYTHONPATH, sys.path.append(X) should work, too. > > Erm, I'm not sure what you're saying... Are you saying that we > should fix all cases where non-strings on sys.path cause problems, > or are you saying that there's so much code out there assuming > sys.path contains strings, and that we therefore should stick with > strings? I read Gordon's comments as the latter, which implies your approach (A) below. > Both positions can be defended, and both have their problems. > > A) Stick with strings. Hooks can be implemented by subclassing str. > This is great for hooks written in Python, but subclassing str > in C is not straightforward. Things can still break, though: eg. > os.path.basename(strsubinst) will return a regular string, not an > instance of the subclass; might be an issue. > > B) Allow arbitrary objects on sys.path. Hooks are then easier to > write (in C), but some code breakage will occur. The std library we > can fix (if needed), but third-party code might break. > > I would very much prefer B, but if it turns out that we can't break > the string assumption, I'd still be happy with A (rather that than > nothing!). As I say above, I prefer (A) of these two. But in practice, I don't see the problem with Gordon's metapath approach. Equivalently, every element of sys.path must be a string, and there is a dictionary mapping sys.path elements to Owner instances (if it helps, you can say that if the dictionary doesn't contain a particular element as a key, it can be treated as a normal directory). The advantage is that we stick to pure strings, not string subclasses... [BTW, at present, sys.path does not seem to support Unicode strings. This seems like a minor wart. If you allow anything more than strings on sys.path, I'd suggest that this be tidied up, too...] > Regarding PYTHONPATH and sys.path.append("/path/to/my/archive.zip"): > for now I'd suggest that the sys.path traversing code checks > for a .zip extension, and replace the item with a zipimporter > instance. This check can be very cheap. Later we could add a general > extension-checking feature, where one could register an import hook > for a specific extension. This might be a case of YAGNI, though... Is this check going to happen whenever sys.path gets changed? If so, how do you trap that? And yes, I very definitely need a way of registering a user-defined hook for path entries (and not always based on extension!!!) Imagining a hook to handle something like "http://my.repository/python/" on sys.path is not hard, or unreasonable... (Security considerations = aside). Paul. From guido@python.org Thu Dec 5 09:11:17 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 04:11:17 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Wed, 04 Dec 2002 15:49:34 PST." <3DEE948E.3010802@ActiveState.com> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> Message-ID: <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> > > I would it support this kind of code: > > > > sys.path = L > > L.append(...) ? > > Sure -- sys.path wouldn't be a "real" list object, just one that > behaved as one. Note that I've seen people write sys.path = ... where ... is a list they calculated (from the old sys.path, to be sure). E.g. sys.path = filter(os.path.exists, sys.path) # remove bogus path elements I still think I'd like sys.path to be a real list of real strings (not necessarily representing files or directories though), with a configurable set of handlers that are given a chance to deal with each string in turn. (Or was this already rejected? I admit the Zope3 sprint is taking 150% if my attention this week.) --Guido van Rossum (home page: http://www.python.org/~guido/) From Paul.Moore@atosorigin.com Thu Dec 5 09:21:29 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Thu, 5 Dec 2002 09:21:29 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB88619970@UKDCX001.uk.int.atosorigin.com> From: Guido van Rossum [mailto:guido@python.org] > I still think I'd like sys.path to be a real list of real strings > (not necessarily representing files or directories though), with > a configurable set of handlers that are given a chance to deal > with each string in turn. That's exactly what I am arguing for. Jim Ahlstrom's existing patch makes this happen, for the extremely specific case of zip files only. Gordon McMillan's iu.py code offers this in a general way, but in Python - and hence with a (fairly small!) efficiency hit, and no obvious way of writing hooks in C. I believe that taking Gordon's design and implementing it in C, along with a standardised zip import hook, satisfies both Just's requirements and the requirements of PEP 273. But if it's not possible to do this in the available timescale, we should just go with Jim's patch for now (and maybe add iu.py to the standard library, as an interim pure-python solution), and look at import hooks in the longer term. I see no obvious value in going through the design process again, from scratch. Paul. From guido@python.org Thu Dec 5 09:27:02 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 04:27:02 -0500 Subject: [Python-Dev] Mac OSX issues In-Reply-To: Your message of "Tue, 03 Dec 2002 08:20:07 EST." <15852.44935.756518.645276@gargle.gargle.HOWL> References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> <15851.43445.443010.182362@montanaro.dyndns.org> <200212030844.gB38iEJ01481@pcp02138704pcs.reston01.va.comcast.net> <15852.44935.756518.645276@gargle.gargle.HOWL> Message-ID: <200212050927.gB59R3A11368@pcp02138704pcs.reston01.va.comcast.net> > BTW, test_socket.py passes just fine for me. hostname(1) returns > "freewill.local." (yes, trailing dot). > > The difference may be that I used Netinfo Manager to add hostname > entries for all my local machines, including the MacOSX machine I ran > the tests on. My Netinfo-fu is pretty meager so I don't know if I did > it completely right, but it seems to work for me. That fixed it for me too. I still think the test should be skipped rather than fail when gethostbyname() raises an error. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Thu Dec 5 10:25:01 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 11:25:01 +0100 Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies References: Message-ID: <3DEF297D.8080603@lemburg.com> Bill Bumgarner wrote: > > I'm writing a Transport class to handle proxy servers such that the > xmlrpclib module will be able to communicate with a server that is on > the other side of a proxy somewhere. Doesn't urllib already support this out of the box ? You can use its machinery for finding out the proxy settings and then add them to Transport as needed. > Most of the difficulty in doing so is the need to navigate the urllib > vs. urllib2 vs. httplib maze. I believe I can do what I need entirely > via urllib, but may use urllib2 because the capabilities better fit the > needs of my ultimate use. > > However, I would like to create a Transport class/subclass that can be > dropped into the existing xmlrpclib module in the core such that this > problem is solved once and for all. > > As such, I would really like some feedback from the python-dev community > as to ensure that the efforts I put forth to solve my particular needs > are applicable outside of those needs... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From just@letterror.com Thu Dec 5 09:48:13 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 10:48:13 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996F@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > > A) Stick with strings. Hooks can be implemented by subclassing str. > > This is great for hooks written in Python, but subclassing str > > in C is not straightforward. Things can still break, though: eg. > > os.path.basename(strsubinst) will return a regular string, not an > > instance of the subclass; might be an issue. > > > > B) Allow arbitrary objects on sys.path. Hooks are then easier to > > write (in C), but some code breakage will occur. The std library we > > can fix (if needed), but third-party code might break. > > > > I would very much prefer B, but if it turns out that we can't break > > the string assumption, I'd still be happy with A (rather that than > > nothing!). > > As I say above, I prefer (A) of these two. But in practice, I don't > see the problem with Gordon's metapath approach. Equivalently, every > element of sys.path must be a string, and there is a dictionary mapping > sys.path elements to Owner instances (if it helps, you can say that if > the dictionary doesn't contain a particular element as a key, it can be > treated as a normal directory). The advantage is that we stick to pure > strings, not string subclasses... What's the advantage of banning string subclasses? What's so bad about lifting limitations? > > Regarding PYTHONPATH and sys.path.append("/path/to/my/archive.zip"): > > for now I'd suggest that the sys.path traversing code checks > > for a .zip extension, and replace the item with a zipimporter > > instance. This check can be very cheap. Later we could add a general > > extension-checking feature, where one could register an import hook > > for a specific extension. This might be a case of YAGNI, though... > > Is this check going to happen whenever sys.path gets changed? If so, > how do you trap that? It's trapped every single time sys.path is traversed. It's a very cheap test. > And yes, I very definitely need a way of registering a user-defined > hook for path entries (and not always based on extension!!!) Imagining > a hook to handle something like "http://my.repository/python/" on > sys.path is not hard, or unreasonable... (Security considerations aside). That would be way cool! But let's please not try to do too much *now*; let's get zipimport going and see from there. Just From just@letterror.com Thu Dec 5 09:31:13 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 10:31:13 +0100 Subject: [Python-Dev] New Import Hook & New Zip Importer In-Reply-To: <00d301c29c38$3cfdc1c0$ced241d5@hagrid> Message-ID: Fredrik Lundh wrote: > have you tested the following code with uncompressed PY files? > > buf[data_size] = 'Z'; /* saw this in zipfile.py */ > data_size++; No, whoops! The "zipped-uncompressed.zip" file that comes with the other patch is actually compressed... I'll test & fix. > unmarshal_code should check that PyString_Size(data) is at least > eight (or perhaps nine) bytes. Ok, done. Thank you! Just From just@letterror.com Thu Dec 5 09:41:20 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 10:41:20 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF20DE.7010803@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Why not leave sys.path as list of ordinary strings and add > all the extra support to the existing import mechanism via > hooks at the right places ?! > > E.g. such a hook may scan sys.path for files which end > in '.zip' and then redirect the lookups into their ZIP > archive directory. Then you get the problem that Samuele pointed out: if the *hook* scans sys.path you get a funny path resolution order: hooks[0] may find the module on sys.path[3] before hooks[2] finds it on sys.path[2]. We should document that - sys.path *may* contain arbitrary objects, and that sys.path traversing code should be careful. - the std library promises to not add objects to sys.path that are not isinstance(p, (str, unicode)). I think this is the perfect compromise. Thank you ! Just From mal@lemburg.com Thu Dec 5 10:51:52 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 11:51:52 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <3DEF2FC8.4060700@lemburg.com> Just van Rossum wrote: > M.-A. Lemburg wrote: > > >>Why not leave sys.path as list of ordinary strings and add >>all the extra support to the existing import mechanism via >>hooks at the right places ?! >> >>E.g. such a hook may scan sys.path for files which end >>in '.zip' and then redirect the lookups into their ZIP >>archive directory. > > > Then you get the problem that Samuele pointed out: if the *hook* scans sys.path > you get a funny path resolution order: hooks[0] may find the module on > sys.path[3] before hooks[2] finds it on sys.path[2]. Uhm, I was unclear here I guess... I meant that the strings in sys.path should trigger a hook which then takes care of managing the lookup and possibly the import. Much like what urllib does with URLs. Here's a sketch: 1. User programs register import hooks based on REs which are used to match the entries in sys.path, e.g. ".*\.zip" for ZIP importers (caching could help in improving the mapping performance). 2. When Python sees an import request, it scans sys.path and creates hook objects for each entry which it then calls to say "go look and check whether you have module X" until one of the hooks succeeds. 3. Python then uses the hook object to complete the import in much a similar way as e.g. SAX parsers call out to event handlers. The idea is to reuse as much of the existing import machinery as possible -- writing these hooks in C wouldn't be too hard either. > We should document that > > - sys.path *may* contain arbitrary objects, and that sys.path traversing > code should be careful. > - the std library promises to not add objects to sys.path that are not > isinstance(p, (str, unicode)). > > I think this is the perfect compromise. Thank you ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From just@letterror.com Thu Dec 5 10:18:57 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 11:18:57 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF2FC8.4060700@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Here's a sketch: > > 1. User programs register import hooks based on REs which are > used to match the entries in sys.path, e.g. ".*\.zip" for > ZIP importers (caching could help in improving the mapping > performance). > > 2. When Python sees an import request, it scans sys.path and > creates hook objects for each entry which it then calls > to say "go look and check whether you have module X" until > one of the hooks succeeds. > > 3. Python then uses the hook object to complete the import > in much a similar way as e.g. SAX parsers call out to > event handlers. Nice; the hooks can then be cached in a dict, as in iu.py, with the path entry as the key. This makes bootstrapping a bit harder, though, as now we also need re/sre/_sre to be available before hooked imports can work... > The idea is to reuse as much of the existing import machinery > as possible -- writing these hooks in C wouldn't be too hard > either. That's exactly what my patch already does: it leaves most of import.c in tact, it adds no duplication. I'd argue that *implementation*-wise it's simpler to just allow the sys.path entry to handle the request. I also don't see a problem with it design-wise, apart from b/w compatibility issues (which I think are non-issues if we use str subclasses). Are people against of the whole *idea* of having non-strings on sys.path, or is it "only" a b/c compatibility concern? Just From fredrik@pythonware.com Thu Dec 5 10:40:52 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 11:40:52 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEF2FC8.4060700@lemburg.com> Message-ID: <165401c29c4a$c9b91c60$0900a8c0@spiff> mal wrote: > > Why not leave sys.path as list of ordinary strings and add > > all the extra support to the existing import mechanism via > > hooks at the right places ?! since everybody knows that sys.path is used to specify the search path for modules, sys.path is the right place. > Here's a sketch: >=20 > 1. User programs register import hooks based on REs which are > used to match the entries in sys.path, e.g. ".*\.zip" for > ZIP importers (caching could help in improving the mapping > performance). a solution that won't let you wrap up the standard library isn't much of a solution (portions of both RE engines are written in Python) I think this is overkill. let's keep it simple and grokable. From fredrik@pythonware.com Thu Dec 5 10:31:24 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 11:31:24 +0100 Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies References: <3DEF297D.8080603@lemburg.com> Message-ID: <165301c29c4a$c9b48880$0900a8c0@spiff> mal wrote: > Doesn't urllib already support this out of the box ? You can use > its machinery for finding out the proxy settings and then add them > to Transport as needed. urllib doesn't allow you to override content-types and other things that you need to talk proper XML-RPC, and my "request body" [1] patch was rejected by GvR. I'm pretty sure you can use urllib2 for this purpose, though, but I haven't looked deeper into that (mostly because people has been promising me proxy patches for years...) (Bill, note that Python CVS isn't the "master repository" for xmlrpclib; if you come up with a clean and nice urllib2-based proxy transport, you can just mail it to me) 1) http://effbot.org/zone/python-urllib-requestbody.htm From Paul.Moore@atosorigin.com Thu Dec 5 11:03:53 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Thu, 5 Dec 2002 11:03:53 -0000 Subject: [Python-Dev] New Import Hook & New Zip Importer Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E5E@UKDCX001.uk.int.atosorigin.com> On Thu, 5 Dec 2002 02:00:23 +0100, in you wrote: >Phew, it works. > >The attached tgz file contains: > zipper.patch: patches setup.py, Python/import.c, Python/importdl.h > zipimport.c: zipimport module, move to Modules/ > >Do I need to mention it's a rough patch ? How do I build on Windows? I don't know what's needed to build the zipimport module... (Sorry, I'm not being awkward here, I have no idea how to set up the necessary Visual Studio stuff so that I can build it - and someone will need to, before the patch is complete). >import.c changes: > - Allow arbitrary objects on sys.path. I still have reservations about this. Code will break... (if you mean "string subclasses", then I feel slightly better about it) > - If a path entry ends in ".zip", replace that entry with > a zipimporter instance. (obviously, the zipimport module > must be available for this to work.) This seems like a gross hack. How do I write a similar hook for strings ending in ".tar"? This isn't a YAGNI, it's a design constraint which I don't believe is reasonable. > - The search algo is: > [handle builtin modules and frozen] > for p on sys.path: > if hasattr(p, "find_module"): > handler =3D p.find_module(name) > if handler is None: > continue > if isinstance(p, str): > [do builtin import] Who handles checking for pyc/pyo, looking at timestamps, saving compiled source (if possible) etc? Note that Jim Ahlstrom's patch modified the search order so that *both* of pyc and pyo were considered. (This is to allow python -O runs to get compiled modules from a zipfile when only .pyc were available in the zipfile. There was a discussion at the time about why this was good - I can't recall the details just now). Your hooks don't seem to support modified search orders like this. >import.c todo/notes: > - clean up, refactor, debug, document Too right! I don't see any benefits in terms of code readability for this patch (compared to Jim's code). In both cases, import.c is still pretty incomprehensible :-) That's about all the comments I can make, without building the code and trying it out. As a point of reference, could you possibly try the benchmark code I sent you? If for no other reason than to see if it runs under your patch... Paul. From mal@lemburg.com Thu Dec 5 11:15:36 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 12:15:36 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> Message-ID: <3DEF3558.8010006@lemburg.com> Fredrik Lundh wrote: >>Here's a sketch: >> >>1. User programs register import hooks based on REs which are >> used to match the entries in sys.path, e.g. ".*\.zip" for >> ZIP importers (caching could help in improving the mapping >> performance). > > a solution that won't let you wrap up the standard library isn't much > of a solution (portions of both RE engines are written in Python) As I said, this was a sketch. Things like zip imports should be built-in to avoid bootstrapping problems like the one you mention. > I think this is overkill. let's keep it simple and grokable. Hmm, that's what I was after :-) Nothing complicated like importers which pretend to be strings or one of the current white magic solutions in use today. You should basically be able to say sys.path.append('/usr/local/lib/python2.3/std.zip') and have the Python std lib at your fingertips. From a user POV I don't want to bother about how Python manages to get at the stuff inside the ZIP archive. Also note that any solution using special objects in sys.path is not compatible with the PYTHONPATH environment variable (or registry setting). The above approach is... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Paolo Invernizzi Thu Dec 5 11:21:11 2002 From: Paolo Invernizzi (Paolo Invernizzi) Date: Thu, 5 Dec 2002 12:21:11 +0100 Subject: Re[2]: [Python-Dev] New and Improved Import Hooks In-Reply-To: <165401c29c4a$c9b91c60$0900a8c0@spiff> References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> Message-ID: <013394900.20021205122111@dmsware.com> Hello Fredrik, Thursday, December 5, 2002, 11:40:52 AM, you wrote: FL> a solution that won't let you wrap up the standard library isn't much FL> of a solution (portions of both RE engines are written in Python) FL> I think this is overkill. let's keep it simple and grokable. try / except ? The first succeed is the winner. I agree with Paul that Gordon's ui is still the best startup point. from ui.py for klass in self.ownertypes: try: # this may cause an import, which may cause recursion # hence the protection owner = klass(path) except: pass else: break And, about sys.path, if not plain string/unicode, what's in package __path__ ? -- Best regards, Paolo mailto:paoloinvernizzi@dmsware.com From mal@lemburg.com Thu Dec 5 11:26:36 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 12:26:36 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <3DEF37EC.4030902@lemburg.com> Just van Rossum wrote: > M.-A. Lemburg wrote: > > >>Here's a sketch: >> >>1. User programs register import hooks based on REs which are >> used to match the entries in sys.path, e.g. ".*\.zip" for >> ZIP importers (caching could help in improving the mapping >> performance). >> >>2. When Python sees an import request, it scans sys.path and >> creates hook objects for each entry which it then calls >> to say "go look and check whether you have module X" until >> one of the hooks succeeds. >> >>3. Python then uses the hook object to complete the import >> in much a similar way as e.g. SAX parsers call out to >> event handlers. > > > Nice; the hooks can then be cached in a dict, as in iu.py, with the path entry > as the key. Right. > This makes bootstrapping a bit harder, though, as now we also need re/sre/_sre > to be available before hooked imports can work... Right again. We need to provide a basic set of built-in solutions which work without requiring any .py modules being loaded... or maybe trim down the registration logic to just look for suffixes in sys.path, e.g. '.zip' is for the ZIP importer hook, '.exe' for the resource importer hook, etc. >>The idea is to reuse as much of the existing import machinery >>as possible -- writing these hooks in C wouldn't be too hard >>either. > > That's exactly what my patch already does: it leaves most of import.c in tact, > it adds no duplication. That's good :-). I haven't followed the thread too closely, but since this debate has been going on for years, I thought, I just drop in a few lines ;-) > I'd argue that *implementation*-wise it's simpler to just allow the sys.path > entry to handle the request. I also don't see a problem with it design-wise, > apart from b/w compatibility issues (which I think are non-issues if we use str > subclasses). > > Are people against of the whole *idea* of having non-strings on sys.path, or is > it "only" a b/c compatibility concern? Since sys.path is used by quite a few applications directly, the backward compatibility argument is a strong one. I also think that adding too much magic (like having importers being subclasses of str) will only result in people not using the new techniques. From a user POV, I would like to add the path to a ZIP archive to sys.path and be done with it. From a software vendor POV, I may also be interested in adding support for signed/encrypted PYC files within those ZIP archives, so appropriate hooks to be able to add that support would be nice as well. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Dec 5 11:34:13 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 12:34:13 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF3558.8010006@lemburg.com> References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > You should basically be able to say > > sys.path.append('/usr/local/lib/python2.3/std.zip') > > and have the Python std lib at your fingertips. From a user > POV I don't want to bother about how Python manages to get > at the stuff inside the ZIP archive. > > Also note that any solution using special objects in sys.path > is not compatible with the PYTHONPATH environment variable > (or registry setting). The above approach is... It appears you haven't studied Just's patch at all. It allows you to do the sys.path.append, put the zipfile into PYTHONPATH, and *still* uses special objects on sys.path for zipimports. Regards, Martin From mal@lemburg.com Thu Dec 5 11:40:51 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 12:40:51 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> Message-ID: <3DEF3B43.4080108@lemburg.com> Martin v. L=F6wis wrote: > "M.-A. Lemburg" writes: >=20 >=20 >>You should basically be able to say >> >> sys.path.append('/usr/local/lib/python2.3/std.zip') >> >>and have the Python std lib at your fingertips. From a user >>POV I don't want to bother about how Python manages to get >>at the stuff inside the ZIP archive. >> >>Also note that any solution using special objects in sys.path >>is not compatible with the PYTHONPATH environment variable >>(or registry setting). The above approach is... >=20 >=20 > It appears you haven't studied Just's patch at all.=20 That's right; I didn't have time to take a look. > It allows you to > do the sys.path.append, put the zipfile into PYTHONPATH,=20 Nice and thanks for the clarification. > and *still* uses special objects on sys.path for zipimports. But why store the import objects in sys.path and not in some auxilliary dictionary instead ? --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Dec 5 11:44:43 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 12:44:43 +0100 Subject: Re[2]: [Python-Dev] New and Improved Import Hooks In-Reply-To: <013394900.20021205122111@dmsware.com> References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <013394900.20021205122111@dmsware.com> Message-ID: Paolo Invernizzi writes: > try / except ? The first succeed is the winner. > I agree with Paul that Gordon's ui is still the best startup point. Can we please focus on the patch at hand? What specific concerns do you have about Just's patch? I'm not interested in advantages and flaws of infrastructures that one could build, if I need to find out whether a specific infrastructure implemented for the problem has any flaws. Regards, Martin From fredrik@pythonware.com Thu Dec 5 11:46:23 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 12:46:23 +0100 Subject: [Python-Dev] New Import Hook & New Zip Importer References: <16E1010E4581B049ABC51D4975CEDB885E2E5E@UKDCX001.uk.int.atosorigin.com> Message-ID: <171401c29c53$f582bdc0$0900a8c0@spiff> Paul Moore wrote: > > - If a path entry ends in ".zip", replace that entry with > > a zipimporter instance. (obviously, the zipimport module > > must be available for this to work.) >=20 > This seems like a gross hack. How do I write a similar hook for > strings ending in ".tar"? This isn't a YAGNI, it's a design > constraint which I don't believe is reasonable. "Let's make it *really* easy to use zip archives, and reasonably = easy to use other custom importers." really easy: import mymodule reasonably easy: =20 import sys # manipulate path as necessary for index, path in enumerate(sys.path): if isinstance(path, string) and path.endswith(".tar"): sys.path[index] =3D tarimporter(path) import mymodule, ... if you think otherwise, please provide use cases. the implied "there = will be a separate category of people who provide hooks and repackage existing programs without having access to source code for any part of the = program or the standard library" case is totally bogus, and so is the "there = will be programs that add importers to the path without knowing what they add" case. > >import.c changes: > > - Allow arbitrary objects on sys.path. >=20 > I still have reservations about this. Code will break... (if you > mean "string subclasses", then I feel slightly better about it) that's not much of an argument: fragile code will only break if *you* = add non-string objects to the path in *your* programs. if you don't want your programs to break, don't break them (or fix the breakage). From guido@python.org Thu Dec 5 11:46:36 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 06:46:36 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Thu, 05 Dec 2002 09:21:29 GMT." <16E1010E4581B049ABC51D4975CEDB88619970@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619970@UKDCX001.uk.int.atosorigin.com> Message-ID: <200212051146.gB5Bkaq11808@pcp02138704pcs.reston01.va.comcast.net> > From: Guido van Rossum [mailto:guido@python.org] > > I still think I'd like sys.path to be a real list of real strings > > (not necessarily representing files or directories though), with > > a configurable set of handlers that are given a chance to deal > > with each string in turn. [Paul Moore] > That's exactly what I am arguing for. > > Jim Ahlstrom's existing patch makes this happen, for the extremely > specific case of zip files only. > > Gordon McMillan's iu.py code offers this in a general way, but in > Python - and hence with a (fairly small!) efficiency hit, and no > obvious way of writing hooks in C. > > I believe that taking Gordon's design and implementing it in C, > along with a standardised zip import hook, satisfies both Just's > requirements and the requirements of PEP 273. But if it's not > possible to do this in the available timescale, we should just go > with Jim's patch for now (and maybe add iu.py to the standard > library, as an interim pure-python solution), and look at import > hooks in the longer term. I see no obvious value in going through > the design process again, from scratch. OK, except for adding iu.py -- we already have ihooks.py and imputils.py, we don't need another one until we know what we will do long term. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Thu Dec 5 11:47:16 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 12:47:16 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <3DEF3B43.4080108@lemburg.com> Message-ID: <171a01c29c54$1060a3f0$0900a8c0@spiff> mal wrote: > > and *still* uses special objects on sys.path for zipimports. >=20 > But why store the import objects in sys.path and not in > some auxilliary dictionary instead ? "Things should be as complex as necessary but not more complex."=20 From martin@v.loewis.de Thu Dec 5 11:49:57 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 12:49:57 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF3B43.4080108@lemburg.com> References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <3DEF3B43.4080108@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > But why store the import objects in sys.path and not in > some auxilliary dictionary instead ? What would be the advantage of doing so? And, if this is done: How can I add my own custom import mechanism to sys.path? The machinery to replace strings in sys.path is only available for zip files, all other hooks must add their objects to sys.path directly. Regards, Martin From fredrik@pythonware.com Thu Dec 5 11:57:34 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 12:57:34 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <177e01c29c55$80b89990$0900a8c0@spiff> Guido wrote: > I still think I'd like sys.path to be a real list of real strings (not > necessarily representing files or directories though), with a > configurable set of handlers that are given a chance to deal with each > string in turn. (Or was this already rejected? I've rejected it. Does that count? ;-) From mal@lemburg.com Thu Dec 5 12:00:25 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 13:00:25 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <3DEF3B43.4080108@lemburg.com> Message-ID: <3DEF3FD9.7050700@lemburg.com> Martin v. L=F6wis wrote: > "M.-A. Lemburg" writes: >=20 >=20 >>But why store the import objects in sys.path and not in >>some auxilliary dictionary instead ? >=20 >=20 > What would be the advantage of doing so? Real strings in a real dictionary. > And, if this is done: How can I add my own custom import mechanism to > sys.path? The machinery to replace strings in sys.path is only > available for zip files, all other hooks must add their objects to > sys.path directly. Well, that's what I tried to solve in the sketch I posted earlier (slightly modified to meet Fredrik's requirements): 1. User programs register import hooks based on suffixes which are used to match the entries in sys.path, e.g. ".zip" for ZIP importers (caching could help in improving the mapping performance; this is where the auxilliary dictionary comes into play). 2. When Python sees an import request, it scans sys.path and creates hook objects for each entry which it then calls to say "go look and check whether you have module X" until one of the hooks succeeds. 3. Python then uses the hook object to complete the import in much a similar way as e.g. SAX parsers call out to event handlers. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Thu Dec 5 12:01:28 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 13:01:28 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> Message-ID: <3DEF4018.5020003@lemburg.com> Fredrik Lundh wrote: > Guido wrote: > > >>I still think I'd like sys.path to be a real list of real strings (not >>necessarily representing files or directories though), with a >>configurable set of handlers that are given a chance to deal with each >>string in turn. (Or was this already rejected? > > > I've rejected it. Does that count? ;-) I'll add my +1 to Guido's plan, if that helps ;-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Dec 5 12:18:53 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 13:18:53 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <177e01c29c55$80b89990$0900a8c0@spiff> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> Message-ID: "Fredrik Lundh" writes: > I've rejected it. Does that count? ;-) Certainly. The question is: how much :-) Martin From martin@v.loewis.de Thu Dec 5 12:21:45 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 13:21:45 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF3FD9.7050700@lemburg.com> References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <3DEF3B43.4080108@lemburg.com> <3DEF3FD9.7050700@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > Well, that's what I tried to solve in the sketch I posted earlier > (slightly modified to meet Fredrik's requirements): > > 1. User programs register import hooks based on suffixes which are > used to match the entries in sys.path, e.g. ".zip" for > ZIP importers (caching could help in improving the mapping > performance; this is where the auxilliary dictionary comes into > play). I don't like this idea. How could I then make a HTTP importer, where no suffix is available? Regards, Martin From guido@python.org Thu Dec 5 12:31:10 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 07:31:10 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Thu, 05 Dec 2002 12:57:34 +0100." <177e01c29c55$80b89990$0900a8c0@spiff> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> Message-ID: <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> > Guido wrote: > > > I still think I'd like sys.path to be a real list of real strings (not > > necessarily representing files or directories though), with a > > configurable set of handlers that are given a chance to deal with each > > string in turn. (Or was this already rejected? > > I've rejected it. Does that count? ;-) > > Why don't you care about the backwards incompatibilities? --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Thu Dec 5 12:34:38 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 13:34:38 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Message-ID: Martin v. L=F6wis wrote: > "M.-A. Lemburg" writes: >=20 > > But why store the import objects in sys.path and not in > > some auxilliary dictionary instead ? >=20 > What would be the advantage of doing so? >=20 > And, if this is done: How can I add my own custom import mechanism to > sys.path? The machinery to replace strings in sys.path is only > available for zip files, all other hooks must add their objects to > sys.path directly. +1 With MAL's idea you'd need to write p =3D ... sys.importers[p] =3D myimporter(p) sys.path.append(p) instead of sys.path.append(myimporter(p)) Just From just@letterror.com Thu Dec 5 12:31:36 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 13:31:36 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <177e01c29c55$80b89990$0900a8c0@spiff> Message-ID: Fredrik Lundh wrote: > Guido wrote: > > > I still think I'd like sys.path to be a real list of real strings (not > > necessarily representing files or directories though), with a > > configurable set of handlers that are given a chance to deal with each > > string in turn. (Or was this already rejected? > > I've rejected it. Does that count? ;-) Let's rephrase that as "-1" and I'll gladly add wy own -1 ;-) Just From just@letterror.com Thu Dec 5 12:36:05 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 13:36:05 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > Why don't you care about the backwards incompatibilities? Because it's addressed by using a str subclass. Just From mal@lemburg.com Thu Dec 5 12:39:10 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 13:39:10 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <3DEF3B43.4080108@lemburg.com> <3DEF3FD9.7050700@lemburg.com> Message-ID: <3DEF48EE.8020108@lemburg.com> Martin v. L=F6wis wrote: > "M.-A. Lemburg" writes: >=20 >=20 >>Well, that's what I tried to solve in the sketch I posted earlier >>(slightly modified to meet Fredrik's requirements): >> >>1. User programs register import hooks based on suffixes which are >> used to match the entries in sys.path, e.g. ".zip" for >> ZIP importers (caching could help in improving the mapping >> performance; this is where the auxilliary dictionary comes into >> play). >=20 >=20 > I don't like this idea. How could I then make a HTTP importer, where > no suffix is available? I first had "based on RE matching" in there, but Fredrik pointed out that this would cause bootstrapping problems. The suffix approach is easily implementable in plain C. Its open for discussion, of course. If you have a better idea, please speak up. We could also use a combined prefix/suffix approach so that URLs can be recognized by prefix, while ZIP files are by suffix. To answer your question: web-imports could use a special suffix to signal their web-import nature (I'm sure you'll want to apply some more security checks on such modules and or packages), e.g. ".py.web". --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Thu Dec 5 12:42:10 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 07:42:10 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Thu, 05 Dec 2002 13:36:05 +0100." References: Message-ID: <200212051242.gB5CgAm12017@pcp02138704pcs.reston01.va.comcast.net> > Guido van Rossum wrote: > > > Why don't you care about the backwards incompatibilities? > > Because it's addressed by using a str subclass. Which strikes *me* as an ugly hack. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Thu Dec 5 12:45:30 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 05 Dec 2002 13:45:30 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <3DEF4A6A.4010905@lemburg.com> Just van Rossum wrote: > Martin v. L=F6wis wrote: >=20 >=20 >>"M.-A. Lemburg" writes: >> >> >>>But why store the import objects in sys.path and not in >>>some auxilliary dictionary instead ? >> >>What would be the advantage of doing so? >> >>And, if this is done: How can I add my own custom import mechanism to >>sys.path? The machinery to replace strings in sys.path is only >>available for zip files, all other hooks must add their objects to >>sys.path directly. >=20 >=20 > +1 >=20 > With MAL's idea you'd need to write >=20 > p =3D ... > sys.importers[p] =3D myimporter(p) > sys.path.append(p) Hmm, I was thinking of: sys.register_importer(myimporter) sys.path.append(p) myimporter would then either be asked for a suitable suffix or prefix to use in matching or the importer would simply call myimporter.match() during the sys.path scan. > instead of >=20 > sys.path.append(myimporter(p)) Sounds like the classical "you need one more line than me" argument ;-) --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Dec 5 12:46:04 2002 From: martin@v.loewis.de (Martin v. Löwis) Date: Thu, 5 Dec 2002 13:46:04 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <000a01c29c5c$46b61ea0$3c1ee8d9@mira> > Why don't you care about the backwards incompatibilities? There is no strict backwards incompatibility: Programs that work now continue to work, as they will find only strings on sys.path. You only get problems if you actually add a zipfile to sys.path. It is unclear to me what the typical breakage would be, and whether it could be solved in most cases by having the import hooks a string subclass. I can imagine a number of potential problems. Code may choke if it encounters a Unicode string on sys.path (which is possible already), it may choke if it encounters a non-ASCII byte string on sys.path, it may choke if it encounters a string that is not a directory name (in which case you simply can't add zipfile imports without breaking backwards compatibility). Regards, Martin From sjoerd@acm.org Thu Dec 5 12:55:25 2002 From: sjoerd@acm.org (Sjoerd Mullender) Date: Thu, 05 Dec 2002 13:55:25 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <3DEF3B43.4080108@lemburg.com> <3DEF3FD9.7050700@lemburg.com> Message-ID: <20021205125525.44BFC74B08@indus.ins.cwi.nl> On Thu, Dec 5 2002 Martin v. =?iso-8859-15?q?L=F6wis?= wrote: > "M.-A. Lemburg" writes: > > > Well, that's what I tried to solve in the sketch I posted earlier > > (slightly modified to meet Fredrik's requirements): > > > > 1. User programs register import hooks based on suffixes which are > > used to match the entries in sys.path, e.g. ".zip" for > > ZIP importers (caching could help in improving the mapping > > performance; this is where the auxilliary dictionary comes into > > play). > > I don't like this idea. How could I then make a HTTP importer, where > no suffix is available? Instead of a regexp, I would suggest using a function that returns true if the hook can deal with the path component. This function could then use a regexp to determine whether it can. But this is about the same complexity as just calling the hook and let it return quickly if it can't deal with the path component. So this step would just become: 1. User programs register import hooks. -- Sjoerd Mullender From just@letterror.com Thu Dec 5 13:04:13 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 14:04:13 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <200212051242.gB5CgAm12017@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > > Guido van Rossum wrote: > > > > > Why don't you care about the backwards incompatibilities? > > > > Because it's addressed by using a str subclass. > > Which strikes *me* as an ugly hack. :-( True, but you can't have your cake and eat it, too: either we implement an elaborate scheme of registering handlers for sys.path items, or we keep it simple and do our best to avoid breakage. Jim Ahlstrom & Paul Moore's patch implement an elaborate scheme which isn't exposed to Python, and it adds a fair amount of spaghetti code to an already messy part of Python. I'm still a -1 on it. Allowing arbitrary objects on sys.path is simple to explain, document and implement, solves the zip import problem and offers a clean high level import hook scheme. How much better can it get? Just From just@letterror.com Thu Dec 5 13:07:33 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 14:07:33 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <20021205125525.44BFC74B08@indus.ins.cwi.nl> Message-ID: Sjoerd Mullender wrote: > Instead of a regexp, I would suggest using a function that returns > true if the hook can deal with the path component. This function > could then use a regexp to determine whether it can. But this is > about the same complexity as just calling the hook and let it return > quickly if it can't deal with the path component. > > So this step would just become: > 1. User programs register import hooks. That's a great idea, but I think we should tackle this after 2.3 final is released. It's orthogonal to the objects-on-sys.path-or-not dilemma, so can easily be added later. The does-this-item-ends-with-.zip hack is good enough for now. Just From fredrik@pythonware.com Thu Dec 5 13:20:46 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 14:20:46 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <194d01c29c61$20193980$0900a8c0@spiff> Guido wrote: > > > I still think I'd like sys.path to be a real list of real strings = (not > > > necessarily representing files or directories though), with a > > > configurable set of handlers that are given a chance to deal with = each > > > string in turn. (Or was this already rejected? > >=20 > > I've rejected it. Does that count? ;-) > >=20 > > >=20 > Why don't you care about the backwards incompatibilities? hey, I always care about backwards (in)compatibility (and you should know that by now ;-). But I've also written tons of code that uses sys.path in various ways, and probably more different code archivers than anyone else (Gordon keeps working on the same installer all the time; I usually invent a new one for each new project), and I'm not convinced that there are any real incompatibilities... Let's list a couple of use cases (covering roughly 100.00% of the code I've studied over the years): 1) code that doesn't use sys.path directly this is the vast majority of all code. no breakage at all, even if = Python itself would add non-stringy things. 2) code that appends/inserts/removes things from sys.path quite common. won't break if PYTHONPATH isn't changed. very unlikely to break if ZIP files are added to PYTHONPATH. unlikely to break if = string- compatible objects are added to sys.path. and in most normal use cases (insert/append directories), unlikely to break if non-string objects are added to sys.path. 3) code that uses sys.path to search for things in the file system, but doesn't expect all items to exist and/or be valid directories. relatively uncommon. won't break if PYTHONPATH isn't changed. unlikely to break if string-compatible objects are added to sys.path. likely to = break if non-string objects are added to sys.path. 4) code that assumes that everything on sys.path points to a directory not that common. won't break if PYTHONPATH isn't changed. might break if ZIP files are added to the path (whether they're real strings, string = sub- types, or non-string objects). likely to break if non-string objects = are added to sys.path. 5) code that assumes that everything on sys.path points to an existing = directory already broken (but I've fixed such bugs more than once, also in my own = code...). won't break if PYTHONPATH isn't changed. will break if ZIP files are = added to the path (no matter how they're represented; see above). ::: summing up, I don't think *any* code will break after an upgrade, as = long as PYTHONPATH isn't modified. some code may break if ZIP-files are added to PYTHONPATH or sys.path (no matter how they're represented), and some more code may break if non-string objects are added to sys.path. But as long as Python itself won't add non-stringy objects by magic, = that's not really a problem. From just@letterror.com Thu Dec 5 13:22:44 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 14:22:44 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Message-ID: --1439855-2295726080-3248086973=:9001 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit I've attached a newer version of the patch. It fixes two important issues: - fix the uncompressed .zip bug that /F pointed out - Avoid infinite recursion (and crash) if a *.zip file is on sys.path before the item that finds the zipimport and zlib modules. Thanks to Paul Moore for reporting this. (Right now the zipimport module must initially be available on sys.path for the first zip import to work; so you can only replace sys.path with ["myarchive.zip"] *after* you've done "import zipimporter". This could be fixed by making zipimporter a builtin module.) Just --1439855-2295726080-3248086973=:9001 Content-Type: application/octet-stream; Name="zipper2.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000" Content-Transfer-Encoding: base64 Content-Disposition: attachment; Filename="zipper2.tgz" H4sIAAAAAAAAA+w8a3Pbxq79Kv2KjTK1JZt2JPmRxDrJGcWWG986lsd2etKmGQ5N rSyeSCRDUnHcxv/9AtgnH5KVe/qYufdq2shcLrBYAIsFsFj9FsQxT7pPvvsTP+32 bvvp3h58t9tP98V3u7srvsXnO+jwdGd/r727+/S7dqezs7v/Hdv7M4lSn3maeQlj 3/0b/ljWD7qNx38FQX/t5zcpf/gOZnGUZNv+Hz5Gu9Nu7+/uLpD/bqe7v1+Q/84e qAtr/+GUVHz+j8v/cRD60/mIs8b5XTaJwu1Jo27a0iyZ+9mMz655kn8TpSM+TvNt My9JJ9403+hHsziYcmys1x8DUBBydv6zO3h3xdpWwyG1dKyWIbV0dcur/hG1bHUA E4gjC3wWhBlLuZf4EzdKRjx5/4G9YL9LbI5C4sgB73s23MT7zN3fpsE1gLR7gPPJ BtPLgCcsuv439zNGowdZEIXMC0csncf4nm08qdezu5jDaybYlAP+JYhP5N89pLbU 4fd67fxuSEO4rwf9I/PINnA+wWfeYwxIir1sEnozzqIxyyYckTDZAYmwwLLIBxD8 AFjmXU8Jxo/CjIdZykaBnxUg4gQm96UnIBrj+XR6t/1p7k2DccBH27GXACB8+R+9 G76NRGw3EMN9Twvg/O4KmCDxWZN2sblHPB1Hya2XjICR/hShNaSi4oZnLtDe9Cew EtXkW71yx2nkjVyYvzuLRvMpb1rjsY2UT8cOEzhwJkiuU6/Bh+VY5MKckjtArxXL Jvtwwv2PzShuaSCaiWp22Fpxkq16mdC6JWk35LfNHJs2/GnqMFvcN7nnj7ejtFUH BSnNr2cLbzSfze6gRY4uZv7xdhqkmVgGDcnKhsMaMHH8EgKHv9oow1owZs1H53f9 5MY9h8XLr+bxlPfD0Y/87hbWU9oUpCFBADx89HU4PLCm1hD8rTExKjDn/O4yS4Lw hjgDz0Sj6iWf5HerBc0Jz+ZJyM7enp4iPThHoDwv2Bawa+tlFrvedIpagsxrg/yI eoJ49IIwtHBREY6tl2qFvJC4VTvwodQmmGKa7+uKrqbhdkvw/942PTkxo5Go0sgV 5Pz3SXDbJbpd92FR5viqRaq5qnsIAm3pbnWQ0vM79+Ts8GJw3MxhapFOu+9y7yQO S8ZScmVBC4EqE6KJ7qfij8JgiHIBTpyRRTBqAZ+mnIYq0g5wrZyitG3N+BwFo5xq jLhQ3ZJ2kPRx9keDxZzJvZMjl9oNx8RzdO3i7kTrZpxwTt1aNpULrBVYxJGyr6YL 7IVFTa62UOXVC6A5s0V22ZjohUosNRd0NqewFn1CZ8WOV1JXxF1lYozw5bKXxoOt rbFiO3D5LAq5UDdFMMP2szm6RG5/NMohcxgNq5TMgJR1zNgarWXWCIIztuKpl0Lv xDzO7954cYyTfu2lsOKb1mrU/VsVGkwULrRxSBTs24LJLIwy2MLn4PoI30HhUczp 6UUgW1ZQMtrI/0wl0w4AbPmOte3D3zCsMfXoBwbfqoIW9UtV0IigrIaaIlIn6Wn8 wLOTjM8qxShtoQVndKpCxWEuNAXp0Bi/h339ytS7y+A3br8Cld9R6jJIEveSZ9KK wvMXn+YF7VEizD9uSeyzN53zlM0gfGLXnGWIOEW/c8rDm2zCdhqkbDdRFrERaQep LziFrBmQ580C9g+2A1+bm/bgh1Pw65sEnBGTBM2KR5ZIA73gMuSKUkykEZ3fIJzz XAda7sCGIo8qeEM8eSrIWo0ptQaf8hk53Oiwr8Chp4JDORbhEqwJTS06vcqrKG50 WlMclmmGIIpHlvG5Trj30bIgNIQxgI/EFIe+P08SPmqS5lZPWyw+MXGI8rxwPSNS ldkYJ9HMjlVwkji5A7Ihcu+y3XGlxQCftyBvOASmoyOIs2wTMqPWVPhJ9drvjdzG wBZsaQ57M7h67f7Uv+hf/HAJhL97965x7yB8blWzBdZqMTyy0KnV8OserWfKUfP4 tBQvvaFoujwdbLWmYzy/K3f46r8GhxDFRuNxyrNobBs/RzG4hS4hzUR4ivh5CBS9 CdFTgmrv8kFQ6XA4CpQmTjM9vzsCsV5eXQCXmjk3CAxaHTkGWu5jWFgRbjfswOxo cDy4uBgcuf2jo4vB5WUTv1usvVr0KTiZC7Jh7zq5apbQrgk8IpwDlW/DhBomI1aI eFKwDHlutByEodWPkXfsBmChsBttmc0RF/F/lLQqvEJHwshHAskhixPcpUrN4PJ6 WZaUX6SLXmAqBkL68ouExxXdvdQNyc2pfJXyT3Me+hXY4OVMuCXldxMvnVQQBjOv mIecg7U9hjwJfNgB+jA/p8CIaCEnKt4Ajdfz8ZirEdyr8+PT/g+XYJiO+29Prxhu BLrxVf9ycPXz+cBhAno89W5Sgiyqtx5gVCXJLPE+wyKv4JmPm12FYAJ/slBqt2DK MVYT67P8PsiqRIetIf+SlciXBtVMQTZUdCRTZXekhkoVraTs2qvigc5P5Vt56ieI adGryhEQl8WXJga5cQLWrhixay3CB6kMZAuksvXFEtV6o5doIcNjusBDXm2P+NQx lGEwpvYEaAEcwl3wKMOIWUUrv4BhLchKZsYwE0gOsXgcRRnsmvAvGDpYK7ALJE3s 4rD17XXlL9LrkqcoU6k9GXyQwwgo/FmMALCtbcd3DdttFfnTpf39AsDhgxBRAWIo IYpE5nIuFOrLlC/s6/xLExuBScQZ6c8vdS9FBGjljIMPyCJCYeKzIBffY0bgHgVW h71SJRvUxsu2XuJOWsd3P0BDyDzK2kIj/GX5QA67ngfTETSifjpMmUny3e6Yzry2 EJMKv8ClSxETupLoKwU83aah+swEA0EKSLMJ6NYW+ZcO454/YbgPoad5OwnwKWU8 yCZgFdA/ZsAiD/GkHEgWYAeEGD9NlXYW7gF7TDNyo7GrMmubrPGoAf9SexCqdoWA od1KeJpqBKoB9fwjiK7H2uQEzEP1ho809MjLPBe3UEdC03YKU7F7Uy8NMg7Aba8C WQogrIRDANJiAAj2Ey4sHg9RU3F+QRp/vHHUE0BfR9EUXKpkTuqOqXopQhKOzKKD 5BONAs2w9XmM+g3+VkhUiyXnsOJhhoJukayuYBiSO8k6VYcERjNIrgw3ECn7FG0W 6GBqRH3nu1Y4BQsz/2ieSDHR4YcJ3vJ1QJrMwxBVOAhZFGfBDFhOIQAIITdCAaW/ eAQEOosynJaH6aIIzGYw4yiIWcx8DNVowJRipQMkxWMbaIBIoPiC0hWIh1BAQzSF dY5PIXEHezv6Lx97pFkAPgh6+ny0Xa86pahXH1PUfy8cwRRTDibZcHxyOmAb4xj+ nEYwBbNC/ATALJW3VDmnpNAFmpAdCseEA8WJ7oArVgLKN/igOgcOAw0VYScMClzK ZA4ENNERGt2T2wsu6/dv+u/O+1evTwdnsND3Pqh3OMqidxuxo7YmMrRZAuMZU/mS GbhlqYYhOErjaXRrpxvQkpJdpSRZgFYxYjixhp3PMok13G/iO7knmqQq7A4xCGUc xYYw2JCS64baNPG9nWleEAEPVfSrfYEDJgJhRE2GH6W3gDoQGP8IQzlsq9t1WLfV Qw/hEhqF4eHhCFcz6TT6EyrhR2es7gVI9xSmfgxdj6ELIKJcRftLe7+9t/tqry1I B5SvvNEBGwhsh6iS3pQdBWAIgpvQA6IE+trYn0Yp4emtGPTjBNW8MUvoPTBlynpR KucItj/3jN82rbTWkoSWxap94lS9ltN7wrmMMz1KaF4qU25xgWNYdkcckOmYF6yJ OtWSmktOFyo3rhZxUiy8i15P5oU1cYW12G6JA1w9LIlSdBIsn65COPFnKmXbbXdI tiaXoyVsixbB2WtrpAVEwsrttOVpWk2ZoxJRlxMQcZEqsskr9USjtVpPMIMrcURb ypV6a2O6Um9tQVcj2bKyALC7Dxw1GDbrlBt+AM2K3XpLxbjbVWK0NoyV1cuQnLPP qGU2P8wrXJW12DqoKDrcGkw43ohpI97cxNWF20QLNlNfUbARExxqcn80QsOHG3wI PjG7vsukeSpM+EV+f4MO6EyJg0CMmHLnMPTqhQooiBgYC00WOQLkAwD9GJL0WPoR jBgNaWWPhSWlPDH64jPvIzgkaDrRiYCQENadvbphb/KjJOEyoLXkltvOhcC+3Qrs tne0FdA0akNwSuQsMQG2hmyy7r5NyE4bmv4ghUWKBG8kW3RKLKekIMtpr1YylOSC U2dyRYR1xvTo+qN1OrvPklBt70CztN5g0IQX5FjayrbU6y3WaSlpitMvHKslPYqQ AjYr8BLj50NsgZxCbDLv3vTWuwMHcu77nMt0SW1DwKz/2l7XJ84l+Fxmng7cSKHA ScUQADEEwl+lRVatmPlVCHEeUN7ciOWJBY67Qc7M+pN1GkYsNhidjhxgRarDBkxZ w1YfM1URIIYin1Bue4sZIaeI73O1GAghhqiVMwC69EAeUVLvra2eIDPPQENIhx7v iWpB9gWHtTbK1UWJ8C1Clx+WohQozSd/7kZ+iDxRko4OefA561E+crOPg6xjqgJy 9xUG+z/h6U+zMRwOG446mir/0dI8kkRdVhPlMOu47h9MmABxcsTRK+sJwirPV4hp WZm0NKAPkCc8ZROWmIiEhmEL4xIZC2MKpXTS1qqXKNRnbOWzvFJ2B3GqE62KmEr4 ZaRj6FPbPqzgpvsTT66h8Xjq3YjTrMu71P1XAuHyZQZGKWk2HjPbk/VgC/p+OpKJ F3LFv09/DcVJL41mBxPKVcXauzpNsHjAVSjXqDryEnOpPDVHfoy4Eok7nod+E+tL CnGnUHHhugknHdX6DbWiPjVbueNxrHpEMyirHysXgkSJfj700rGRAFngqptKAHOi jbl6iRJhAZ+ZT2NRRQpOG888UQPzATeIQaRfwBs3mopPVvyMj5au6kehrwXeJd4t jYNxK9oNMTd80nTq4PZ6Ps4H8hQ8JxANoaeSuuCSj2Q56YNR5qNxbIeXx1Ey87JC bCk22kKASZ4LhZi0UX6fNnIaWRF3AdnK82IbLTqDdN+ImiTjS2/S3kikUf+HI2BA A8FTIUTPR8I0CAYA4rhFBIb5ULNMLjAUWbfEaSKvDnu9EDYwx/0xfuMcHHQG7LQK DVgyEojo61dbgrCJabBVUgALJk+4EFFDxtTI92OsxgLiFswdCdKRGLp7YnwAeK9J Ii/ol3Xhf6TerdjmAko5IK/AkZWei4JQBlLpOi16WUGAPpv4sx+OqOyBWGcYQKaj SDrSabAtrUUxgSVNB4kmKKAZNdlKzxLREoNCjlgA4q3O+7LbADw5skMqL/JI13Iv lZaVvVggMZNdJhJ7rCE74miU4/jsBVMsrbbKWtS+Bvw1JkPGIkXbrc9luM0Vxbwc Oi0naUsPYS0dAw48KLLgsT5pChbAmLGtzp7cdOSeYzq3qjYpBWntSILxlaZ5Hso6 f9ePRrxZKLhm5qhKPFv1yjhE3vQiBtu22mqpC1sUaVTrrwJz2YnUVXWgHCfmlmfe TeAX90JsE2V7zZYuBER0/3jBnn+b2lzDssaY0Szt6nWMa+V9+wNbg6BtPCZ/v2kR qJq/fhV7GnXv5Ls3TfeXLxl71qqG6i6D6uwvgNpZBtXd1VCLt6k8a2hjbrzCIiDC IssHwDB9v91tt1PpYmrvupQTRX0oRcNCWYyVot1pkz1zhDZsAVN6ytCM+GLfRJbF HUInWfGF/U19pFwN1Nirnm+u1kvMFl1HecdF1z6J2Sr75jGiS9a3aMjlnJANYoFU rsNx8MWdBiF3Rb4ktQop02ie+Nw6Kv4E3kxcubhkV7m8dAo0v8BMJ6vCMvjCR654 o5dTJg6hISr8NYH4tiwK09+Gr96NmrFep/nOCzebT5VTtIFbxeytFSETyRTZbXyi 0DNc11HZRjMm/0h0DFVHFUDXZPJYRq/3MtjCo2+NbyPuFcZrr+dq8ggZ+6S26qYu ZtflubmZUI23tIdidGO+88KpVB+ps7LTN5hxS7nyhtwpK0VByCWVtbRvdSlrI+Ee ijloa71U8qaWVlhxgCe/MgjjeVZJAthECVswEUWNWmm9fvvlpcp7S3m+y7hFsh8w Y9gCsZzeVJUE5QKviu1XiOo1OnOt5qGbFmrpaoiHypULpddWTkBlKA4KLMwVXxtV XTOTXLNmuWamaYHlJrwmZ7yGSYcKSqVHlgtO7Qsx+bKHBzmsfcHlLnQK/q4/oSw2 KaKP1VOiOuCAEiC0HArr2SiRYYzyk7TFkZgOJSr5OCxgLjh8SzGXdmFhJ6mEcdxs DC4uhhcynZLfAOVue44dmzpbpQgd8bE3n2YHK/louXAZXTVM/1MRdkOec43se2Gm JNr2mJXP+ZBXUZdV2trVHHzhProYIvcy+GLxS6xSe9plr2PlpJWYIPASS3MSrl0Q XXdtklZlienVKbKqa2vMLhRXB8eYCXNdqu9xqQLJlLRQoGbdI4U+WH9CKqAyIjgQ bEzEcqoooIphq27gA2XiZTeRWMYTV/vKgz6dwh6brMtevigVEKzqtufCPX3XJl9D UEhU3tvnDPKGj+GmpIsA5eRk0weTY8+/QCfig5XXVoxbGOfZt3NKV09lODIcomed v5/HrDsk5m4YsdOMuTBBS6WY5IeTuDF+oiwiSJ/QiARNZapaVLY1lN7oe4WsmHZ+ P/wgjQAS05Iut1oPul1RjUM+ktmeUthtXx0ox7eS6gXJVsymMjqJwCC/KrlaypbK pS6+BIuaKktaE0WduDZzKQqTnLDvnXfys6ZOvfytsMI19fK5w/03pbpdQz7MGTTq QNBpzIUZ8J+sYfIdB6zx9qz/U//ktP/qdFDM3NqXNGZx6XZG7nZE1fWIwqWBWbzs toC0QnRXgC5bovj02wX5ce0ZKXZRmS8GmHfN8qVudbgilMYAuUNZ5UZMVpbygsu6 bjpKxp8JwL9E3XktV2vaRo6YUtf8y454qSpn763dhaQlVG23aQwZ3VQx/JZPxDqx 6EQq+/znq9fDM7d/fuL+NLi4PBmeoeI8sMrJNaEuhQTVGFTCcGSxDcjdm2BLLZnm d34AvSpR0tgK6xr17u/+1Yz/PR/r91/wexvsrj/5g8dY/vsv3f09+/dfdvbw91+e Pt35/99/+Ss+J3jIegDmOJvHYLLqL/7zT/3i8JLCnAP2xP+cJlGUPYnpx2XU1yhI sydp4j9Rwzqf62Bok4B/xsqfBL6oSryz3el26qNgPGZbPttK6NnQurGxoR9q3afs LPrMuqBQrLNzsAv/7bMtVLCaQLO1tWV677Ej7uvO2P+p6IxI7Q8N0n2+7+y0O4wa EI9o6DL4e0tXc1M9d7rtwUIKR002+JLxEKfRXHcnUZbC/+sOe68ftv31Dy3WWgUc b/qADyfA5cPq4F/wGArTLikhMI9iF1WINh9EpO054bF+MUpRYhHzmHlJ4t3JrGe6 ApnUnzDTX3nipJqK3yh6IsYdTbcnf7G+lsZfoLjd7c4zS2/xkVlEo06VUNW6++yY X0ul7Bzsdg52dqUGEz5UvDLUt2lyZ9/pdoUeM1Y7//EH9+jkYnB4Nbz42cGWQ/fV 25PTq5Mzejr/2T2+GP4ygKdH9HQ4PBpcDC6Hby8OB3iM98bzWRRO6bSRMfDjSAfk rx0hS+mKFnhJSDuOvaOWzH82tlMcHLqcvDl3Xw+HPy4npEqRtv2/U422/YVK1O10 cloEz0yTXFKibb/WfW6MYLcNqnCw29YqhNhKOgRAZQ16tkSDnrX3nWedHWMLRcO+ EmydzneP5W8+kV5U/46TTIOKvKLDRE2FyoFSzQY4fZtLgbslaNNNYdDDl5Rhw/6p E4XIQoBaqBKF6jXm8NxMjbdhU7gUf7dygE0zgNW+YaMvku+Ok+g3Hkrk4kHiljkT JYJTvIxv7h1hUKKutmW6Jk7+xtm/Tq5eM/pdjzeDs6vBUbXknz9znrctyT9vd5zn O3tK8mV2wk5TymTLGjN14qtYivUhUx4S05EdkgOxO45xZ/kdeWWQTGA2U7yFrv4S V5hVUg/6lqWR8JTeyLw4/o2PCVVN5URlyBbErunhiBzCQuUP3Dq8R1nmMgwCfYE+ neS5gOemfGkw5nubdJxAntoTzOVA4Pke/6k+47OxmrHi/8kA8koNCU5wQqZbSDwq jZGbl1X7ViZE1XoI2dwLpi1cUPCuYk0t1jJrTeHVcyFEHLakYZvAA9A8uj7qsBDB cSXVpHKiZpp8JqtSr/Eo1tyrWj3PO23nebdtrZ7drvN8T6+e2qJZj0cu3hrNgpB+ EQx/Awz+15vlfe8BYGUALODc3isQVN3r2vzv9o61p40j+Jl/caZKwbGh3OHDdlBa VSqpLDUP1UFtKlWWMSaxANvyGadU4r93Xrs7ew/HJJSm0s6HKNxjb2d2dmZ2Xo7/ dDsG8wWKt7zN61UINGIc9xvgNG7mcL775t3r/uB1P6krwmbD1fh8YKmqL2py80Lz dU4TYULDBzCUeIFTGWbZeLHctQtt3dkiEnU5mmGZYj0ayvyNKtJK17h12Oy2uta+ IveuN1kMCOMNjjU6Cc6TUQ74GqoGvdusdHclyt9emPf5+p2afXWaBvFeG0R5p2V5 70sn6rZXRSj/wWZfSvYUyH50aMl+Z1absXJzNQhNMrOldm2LrE0JLi/qOVOqT4FI D7KcvFodWK1ux67W5+P3uev0BUg/FGuUr/xR2uy2lVCNwWxtorvJsfYFK8XfJtPB C5jur+P3YIaPF+NzcdxL0BjkN5bnXzRFmMmUpBy05hN5i9QG2g1z1pWfRnN7O4ck fNBhJ2KsFMcOKI5OqnGEk1R8kFj3Q7H4ivSX6XjA28plasxv1U7Xt1Y09V+APLbL FuvRCStrL/vHquHBh9nsUmQ9NsMFfX/a7736eXD6qodnNkvGN7en08nIZV+t6pac Min3xMmUYs3uwo99Mx68VyEGukCoblsTKmkDoVLFDMy7It2c7jDVL3D8ym7ORleg S/Do0pCZ1+yC0sxP/hqOljJ9Wvw8NXI93VQG/AoUsG5UJYR1YlaPUvPtM01/Y8xu GbOtPHqYHxJzKbat8DGDurQW/2n7SKmxu1Wwd/WjfvdEeb7KSsGP8f+eR+bZLWut mIP93bG96e02RQsrQOyI5s5dAVt4zSAoN4ktzGNejK0R8UOWl3McgcxgONlUZsnT bBf5+XQrJVxMYLsRuRAxmCT90xcver/3e3+cYMhbRJLZLsqkxp1TL7c4QQwmKAu9 HXEUN+P4QGn+ksF8RPAsuaIu0sPJNKPINR/rhSQNhcj3UQuTCdBWhD/3YmqssjPf 0RcTvjjBi/QyGuPm5iHf/Nt7o8UX93csK1FXaWxJiZ0fTFsAPPGjCUjsZLavDuNS oIwCgSoyhWdI2hlP6SSDXafM2Y2+83aMDw4Xk6tb4LT51XDERZ8UQadM9OGUD0Ei QfhNbD4xi4YSXKYO4mN4f3SzQN/OftS7EE8Oyhw7HfsuxrqwrGUJn7i6whPRbTSb RtlthhGaD/sWPTvpvNoRE8psjO0nGTCB/dDedLbcc7FdUnpaFqlh/T1Ph0JFPnM6 NCK+L23BbG2XrEwzskNKrO9T47md6t6UW2Yl1wTjbYBUY+Xe85HSjd/9LH9L5zUT 1U3sKgW/Yz4vLFovIGse9OetvrDpeuQVkacYvA2wjaWJq7Kp2If0bHjgh2ILUjT3 5QitnSaYdNvIy15/gGd2O2M+wYSMFJUsgSkykrY0W/ygREdBCXjfENONzZ7r4Qjr h7MPJBm/4/MfSOH+fDw+v5k/44ZMZgOz+KCiHerHBnwB+qxKjHfR3IvdsfIpij9s aZPBiXYWfcRiYaxewcwNFDtnM+r0s1zconRZYoLT7NK8SVXOcP+acipAC1LCYFbf 5wfIl75lRUmP5tebvkHBh40S2Tw8eTvovT15ae1DIA++pi3I/CPHyhB8ORyJMc5f l21bzHhembJXlE9W/1XZEYCM++O5+G+d1+PdwMQPTICgWv0ZQ0IPmT/+ODww4e/+ uJBGBpUMGhlW12rkf391ZU+WmLTm1PaZ6/8JBqhVkO+rZIPNjnWsNx+NZcoFRHKI AqKj7LwYL8VJV9l5HgV0MO0xqbDmnAtWKaLR8tBotQCN9MChgcYinVMoN0iiDfh5 NBgVYBYebpnM9jOQokgzCCY7D2aX7I3R/l05HgG1qbsDGJmUF32NiTHjzAzxkKRy TlpNKDyMlNMpRbM+1WZ9nCKdjjak031o8EjIVnNFfJgCai3t40kOO804aalDjHcQ NR6ynI9m88n6DppK5xMwJzBsK1V5IbAO7Wactq00L5aBRFEx+sl73FSVe9ExNEuW 9CszVUGwtdh4L1zn40xro1dbmJTsBVt5mtR9yATEaGomzFMW/TGZucXwjqJJIyoJ 6uaJUhL9c8SR+E2hIcJx7uqxifKAqaeigTqmwGc/amGBTaRrz60DgslUVabq0GZp ilHwySJbcstCbkWIinu8k+GR0dXv71IzxWgKdmJE/hKRMabugzCskv1pcgAMl2j3 4FELLh2lanu4H224lu0oRQ8R4UryzSD5TM7YuYD7ereWjq0xcHGk7qkuQoNcBW5E 7xhR8CdV58XbIYouLcFNQpaIri0f2STCEEV2+j+R3XU5nX2EY8pHNLVM1QUXeDKH UH3Ik/P6drlbMu62QYJ1Oy7tprgeJK8k47vmEZ8jkXitKgqHA5aGI/mW2UDK58sp bKBqmwmYn2vYpHpaLgBvQ6X3nUQ5rbox0KrbcbQS44mkcI23HydRcwkFXuLQq04/ yG7OdACfJKkmG/r753Xj/7a6TXr9Wb//+TwfXTHOP2Qh4JzxHJnxJZsGRWaq14XU cRdInaTKGsthVUAhyePgkhAea+plCwRsHzfhHxXjLP1JGK3y7c/CMMY12ddaDbqa mgsThQEM9r4X9eICMeJl1S08zE2pYFJe50jO/L0FGKNz4FusQ5QGudSuK5N+uXAO x9k0ua/b0IqAxfgC5DU2ucetb9tx8e/i2cwayVSUpU46sNStnIP1C+mT5AhkWUET iTdoUQmvJZ6RyNdlDntXyYKCNqMmwtR65uxq8h6UFRKFPa5AFeu6FLeLLYanEyn6 PUQlO9+lMKT/G0b57kPe7xgZ53xJzUHhBVhN379/9/Wwyn+d9R0gQIAAAQIECBAg QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE+D/DP6bqvacAoAAA --1439855-2295726080-3248086973=:9001-- From guido@python.org Thu Dec 5 13:24:04 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 08:24:04 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Thu, 05 Dec 2002 13:46:04 +0100." <000a01c29c5c$46b61ea0$3c1ee8d9@mira> References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> <000a01c29c5c$46b61ea0$3c1ee8d9@mira> Message-ID: <200212051324.gB5DO4p12167@pcp02138704pcs.reston01.va.comcast.net> > > Why don't you care about the backwards incompatibilities? > > There is no strict backwards incompatibility: Programs that work now > continue to work, as they will find only strings on sys.path. You only > get problems if you actually add a zipfile to sys.path. I've written code that searches the path for data files. This has been brought up as an example before; Zope 3 for example looks for page templates and configuration files on the path. I'd like such code to continue to work even if the standard library is loaded from a zip file. Such code is typically robust if one of the sys.path entries doesn't refer to a valid directory; but it may only catch os.error, and not the TypeError resulting from attempts at string concatenation. Obviously if the data files get stored in the zip file too, the code searching for it needs to be made zip file aware; but if zip files are used to provide *other* libraries, that shouldn't break code that looks for data files along sys.path. > It is unclear to me what the typical breakage would be, and whether it > could be solved in most cases by having the import hooks a string > subclass. A string subclass would work. But it sounds like the requirement to subclass str makes writing hooks more difficult compared to a mechanism that places the hook in a separate hook registry. > I can imagine a number of potential problems. Code may choke if it > encounters a Unicode string on sys.path (which is possible already), it > may choke if it encounters a non-ASCII byte string on sys.path, it may > choke if it encounters a string that is not a directory name (in which > case you simply can't add zipfile imports without breaking backwards > compatibility). The main breakage I am intersted in avoiding is brekage when you concatenate an ASCII string with a sys.path item. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Dec 5 13:27:29 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 05 Dec 2002 08:27:29 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Thu, 05 Dec 2002 14:04:13 +0100." References: Message-ID: <200212051327.gB5DRTC12201@pcp02138704pcs.reston01.va.comcast.net> > > > > Why don't you care about the backwards incompatibilities? > > > > > > Because it's addressed by using a str subclass. > > > > Which strikes *me* as an ugly hack. :-( > > True, but you can't have your cake and eat it, too: either we implement an > elaborate scheme of registering handlers for sys.path items, or we keep it > simple and do our best to avoid breakage. > > Jim Ahlstrom & Paul Moore's patch implement an elaborate scheme which isn't > exposed to Python, and it adds a fair amount of spaghetti code to an already > messy part of Python. I'm still a -1 on it. I know. :-) > Allowing arbitrary objects on sys.path is simple to explain, document and > implement, solves the zip import problem and offers a clean high level import > hook scheme. How much better can it get? I fear that the requirement that all handlers are str subclasses raises the bar for writing a handler too much. But maybe that's not a problem. (It's not like writing import hooks is currently a piece of cake. :-) So I hereby change my position to -0 on str subclass handlers, which means you can go ahead. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Thu Dec 5 13:33:57 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 14:33:57 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> <000a01c29c5c$46b61ea0$3c1ee8d9@mira> <200212051324.gB5DO4p12167@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <19a301c29c62$fa298bb0$0900a8c0@spiff> Guido wrote: > Obviously if the data files get stored in the zip file too, the code > searching for it needs to be made zip file aware; but if zip files are > used to provide *other* libraries, that shouldn't break code that > looks for data files along sys.path. as I might have mentioned before, it would be really nice if custom importers supported optional "load_data" and "get_module_names" hooks. the former can be used by code that looks for data files on the path (Zope, PIL, most likely hundreds of other tools/programs): for path in sys.path: if hasattr(path, "load_data"): try: data =3D path.load_data(name) except IOError: pass else: return data # got it if isinstance(path, types.StringTypes) and os.path.isdir(path): try: data =3D open(os.path.join(path, file) except IOError: pass else: return data # ... the latter can be used by code that looks for plugins using a name pattern (PIL's *ImagePlugins, for example): for path in sys.path: if hasattr(path, "get_module_names"): names =3D path.get_module_names() elif isinstance(path, types.StringTypes) and = os.path.isdir(path): names =3D os.listdir(path) else: continue for name in names: if name.endswith("ImagePlugin"): __import__(name) (stupid idea: why not change os.path.isdir to return False for non- string arguments? it's a predicate function, after all...) From just@letterror.com Thu Dec 5 13:35:00 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 14:35:00 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <200212051327.gB5DRTC12201@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > I fear that the requirement that all handlers are str subclasses > raises the bar for writing a handler too much. But maybe that's not a > problem. /F's compromise to the rescue: the zipimporter will be a str subclass, but *any* object is allowed. This makes sure that *we* don't break code, and makes it easy for others to write import hooks (really really easy if they don't mind 3rd party breakage, slightly less easy if they do, and somewhat less yet if they must write it in C). > So I hereby change my position to -0 on str subclass handlers, which > means you can go ahead. Ok. Just From mwh@python.net Thu Dec 5 13:43:38 2002 From: mwh@python.net (Michael Hudson) Date: 05 Dec 2002 13:43:38 +0000 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: "Fredrik Lundh"'s message of "Thu, 5 Dec 2002 14:33:57 +0100" References: <3DEDF004.25379.630FD638@localhost> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> <000a01c29c5c$46b61ea0$3c1ee8d9@mira> <200212051324.gB5DO4p12167@pcp02138704pcs.reston01.va.comcast.net> <19a301c29c62$fa298bb0$0900a8c0@spiff> Message-ID: <2m3cpcwph1.fsf@starship.python.net> "Fredrik Lundh" writes: > Guido wrote: > > > Obviously if the data files get stored in the zip file too, the code > > searching for it needs to be made zip file aware; but if zip files are > > used to provide *other* libraries, that shouldn't break code that > > looks for data files along sys.path. > > as I might have mentioned before, it would be really nice if custom > importers supported optional "load_data" and "get_module_names" > hooks. I haven't seen that mentioned before, but it certainly would be useful. (pyrepl currently has some nasty code to allow tab completion of import statements, bits of which I'd be most happy to see the back of). Cheers, M. -- Hey, if I thought I was wrong, I'd change my mind. :) -- Grant Edwards, comp.lang.python From gmcm@hypernet.com Thu Dec 5 12:45:50 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 5 Dec 2002 07:45:50 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> References: Your message of "Wed, 04 Dec 2002 15:49:34 PST." <3DEE948E.3010802@ActiveState.com> Message-ID: <3DEF042E.22672.67469833@localhost> On 5 Dec 2002 at 4:11, Guido van Rossum wrote: > Note that I've seen people write > > sys.path = ... That works with iu.py. > I still think I'd like sys.path to be a real list of > real strings (not necessarily representing files or > directories though), with a configurable set of > handlers that are given a chance to deal with each > string in turn. Which is what iu.py does. > (Or was this already rejected? ... ) Evidently, but I've never heard the rationale. -- Gordon http://www.mcmillan-inc.com/ From gmcm@hypernet.com Thu Dec 5 13:46:36 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 5 Dec 2002 08:46:36 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: References: <3DEE7B87.5090602@ActiveState.com> Message-ID: <3DEF126C.22266.677E3DF5@localhost> On 4 Dec 2002 at 23:32, Just van Rossum wrote: > (I'd still like to see more examples of code > breaking if a sys.path item isn't a string. Not that > I don't believe it, but I'd like to get an > impression of the severeness of the damage.) Greg's imputil put objects on sys.path. I know of two imputil users (at the time - me & /F) who got bit by it, and *still* wrote code assuming sys.path consisted of strings. It's Python folklore. You won't get users to change their minds. -- Gordon http://www.mcmillan-inc.com/ From gmcm@hypernet.com Thu Dec 5 13:46:37 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 5 Dec 2002 08:46:37 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996F@UKDCX001.uk.int.atosorigin.com> Message-ID: <3DEF126D.31051.677E3E3B@localhost> On 5 Dec 2002 at 9:07, Moore, Paul wrote: > I read Gordon's comments as the latter, which > implies your approach (A) below. Only the first sentence of (A). Having things that look like strings but aren't on sys.path just means that you have to convert PYTHONPATH, and trap modifications to sys.path. Why bother? There's an easier way. [Just] > > A) Stick with strings. Hooks can be implemented by > > subclassing str. [...] > > B) Allow arbitrary objects on sys.path. [...] > > Regarding PYTHONPATH and > > sys.path.append("/path/to/my/archive.zip"): for now > > I'd suggest that the sys.path traversing code checks > > for a .zip extension, and replace the item with a > > zipimporter instance. This check can be very cheap. > > Later we could add a general extension-checking > > feature, where one could register an import hook for > > a specific extension. This might be a case of YAGNI, > > though... Here's what iu.py does: The first time an entry on sys.path requires examination, the list of importers is traversed until one accepts it. That importer is associated with the name, so subsequent imports are only one dict lookup away from the importer. Search order is as expected (sys.path.insert(0, ...) works). If your app only uses stuff from sys.path[0], no time is wasted finding the right importers for sys.path[1:]. -- Gordon http://www.mcmillan-inc.com/ From gmcm@hypernet.com Thu Dec 5 13:46:37 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 5 Dec 2002 08:46:37 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF2FC8.4060700@lemburg.com> Message-ID: <3DEF126D.28673.677E3E81@localhost> On 5 Dec 2002 at 11:51, M.-A. Lemburg wrote: > Here's a sketch: > > 1. User programs register import hooks based on REs > which are > used to match the entries in sys.path, e.g. > ".*\.zip" for ZIP importers (caching could help in > improving the mapping performance). > > 2. When Python sees an import request, it scans > sys.path and > creates hook objects for each entry which it then > calls to say "go look and check whether you have > module X" until one of the hooks succeeds. > > 3. Python then uses the hook object to complete the > import > in much a similar way as e.g. SAX parsers call out > to event handlers. Except for where the policy that associates a "hook object" with an element of sys.path resides, this is what iu.py does. In your case, you've got it in the manager. In iu.py, each importer is given a chance, until one says "yes, I can handle this". I base this design choice on the observation that managers, no matter how smart they are to start with, tend to become very stupid over time . -- Gordon http://www.mcmillan-inc.com/ From Paul.Moore@atosorigin.com Thu Dec 5 14:43:23 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Thu, 5 Dec 2002 14:43:23 -0000 Subject: [Python-Dev] (no subject) Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E61@UKDCX001.uk.int.atosorigin.com> > > Guido van Rossum wrote: > >=20 > > > Why don't you care about the backwards incompatibilities? > >=20 > > Because it's addressed by using a str subclass. >=20 > Which strikes *me* as an ugly hack. :-( And it's dangerous. Code which assumes strings on sys.path and modifies sys.path will fail silently rather than giving an error. Example (not too realistic, I admit): sys.path =3D [ os.path.abspath(d) for d in sys.path ] which collapses string subclasses to strings :-( Paul. From just@letterror.com Thu Dec 5 14:50:37 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 15:50:37 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <3DEF126D.31051.677E3E3B@localhost> Message-ID: Gordon McMillan wrote: > Here's what iu.py does: > > The first time an entry on sys.path requires > examination, the list of importers is traversed > until one accepts it. That importer is associated > with the name, so subsequent imports are only > one dict lookup away from the importer. > > Search order is as expected (sys.path.insert(0, ...) > works). If your app only uses stuff from sys.path[0], > no time is wasted finding the right importers for > sys.path[1:]. Ok, that's is a much clearer picture than I ever got from reading the code, thanks. (And sorry Paul, you've been saying this all along; it's been an entertaining and very educating trip for me ;-) At the cost of some complexity this indeed solves all our problems: - importers can be associated with sys.path elements matching a pattern - adds a high level import hook mechanism - makes plugging in a zipimporter a snap - doesn't muck with sys.path I'll have a whack at integrating this in import.c. Just From martin@v.loewis.de Thu Dec 5 14:50:35 2002 From: martin@v.loewis.de (Martin v. Löwis) Date: Thu, 5 Dec 2002 15:50:35 +0100 Subject: [Python-Dev] (no subject) References: <16E1010E4581B049ABC51D4975CEDB885E2E61@UKDCX001.uk.int.atosorigin.com> Message-ID: <002801c29c6d$ab8f2a90$aa2ce8d9@mira> > Example (not too realistic, I admit): > > sys.path = [ os.path.abspath(d) for d in sys.path ] > > which collapses string subclasses to strings :-( Of course, for zipfile imports, this won't be a problem, since the zipimport string is restored when the next import statement is executed. Regards, Martin From fredrik@pythonware.com Thu Dec 5 14:50:53 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 15:50:53 +0100 Subject: [Python-Dev] (no subject) References: <16E1010E4581B049ABC51D4975CEDB885E2E61@UKDCX001.uk.int.atosorigin.com> Message-ID: <1a8f01c29c6d$b694ab40$0900a8c0@spiff> Paul Moore wrote: > > > Guido van Rossum wrote: > > >=20 > > > > Why don't you care about the backwards incompatibilities? > > >=20 > > > Because it's addressed by using a str subclass. > >=20 > > Which strikes *me* as an ugly hack. :-( >=20 > And it's dangerous. Code which assumes strings on sys.path and > modifies sys.path will fail silently rather than giving an error. ignoring for a moment that your example do work as expected with the proposed patch, since when is an "ImportError" a silent failure? next example, please. From pedronis@bluewin.ch Thu Dec 5 14:41:07 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 5 Dec 2002 15:41:07 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <027b01c29c6c$58d74860$6d94fea9@newmexico> From: "Just van Rossum" > Gordon McMillan wrote: > > > Here's what iu.py does: > > > > The first time an entry on sys.path requires > > examination, the list of importers is traversed > > until one accepts it. That importer is associated > > with the name, so subsequent imports are only > > one dict lookup away from the importer. > > > > Search order is as expected (sys.path.insert(0, ...) > > works). If your app only uses stuff from sys.path[0], > > no time is wasted finding the right importers for > > sys.path[1:]. > > Ok, that's is a much clearer picture than I ever got from reading the code, > thanks. (And sorry Paul, you've been saying this all along; it's been an > entertaining and very educating trip for me ;-) > > At the cost of some complexity this indeed solves all our problems: > - importers can be associated with sys.path elements matching a pattern > - adds a high level import hook mechanism > - makes plugging in a zipimporter a snap > - doesn't muck with sys.path > > I'll have a whack at integrating this in import.c. what about __path__'s role in this picture (btw this is not a rhetoric question)? regards. From just@letterror.com Thu Dec 5 15:12:01 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 5 Dec 2002 16:12:01 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <027b01c29c6c$58d74860$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > what about __path__'s role in this picture (btw this is not a rhetoric > question)? pkg.__path__ is just another sys.path. Whatever rules we make up for sys.path, they are identical for pkg.__path__. When traversing path, find_module() doesn't even know whether it's dealing with sys.path or a package's __path__. The way I see it, the dict that maps path items to handlers also maps __path__ items; all handlers will live there, and there's just one such dict. I think that a zipimporter would put something like this in __path__ ["/path/to/archive.zip!packagename"] So the zip handler will have to do a little more than just checking for a *.zip extension. I assume you do something like it in Jython, too? (I picked up the '!' from you I think; it makes it easier/quicker to check for zipness.) Just From bbum@codefab.com Thu Dec 5 14:18:45 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Thu, 5 Dec 2002 09:18:45 -0500 Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies In-Reply-To: <165301c29c4a$c9b48880$0900a8c0@spiff> Message-ID: <76A58EC5-085C-11D7-B650-000393877AE4@codefab.com> On Thursday, December 5, 2002, at 05:31 AM, Fredrik Lundh wrote: > mal wrote: >> Doesn't urllib already support this out of the box ? You can use >> its machinery for finding out the proxy settings and then add them >> to Transport as needed. > > urllib doesn't allow you to override content-types and other things > that you need to talk proper XML-RPC, and my "request body" [1] > patch was rejected by GvR. Yes -- I was surprised to find that urllib doesn't have the ability to return the prepared, but no data sent, HTTP instance. This would seem to be very useful. It would also seem to be easy to do -- split the open_http() method into two methods at the 'h = httplib.HTTP(host)' line. The first method would simply return the created HTTP object while the second method would send the request, etc... In any case... > I'm pretty sure you can use urllib2 for this purpose, though, but I > haven't looked deeper into that (mostly because people has been > promising me proxy patches for years...) ... urllib2 seems to be the correct answer for a number of other reasons. In particular, if the implementation is done correctly the developer can easily add a Handler to deal with any proxy/communication situations that I have not yet imagined (off the top of my head, dealing with TCP/IP proxying firewalls...). Well -- I have pretty much coded myself into a corner out of which the only path is to add working proxy support to xmlrpclib. Our product [see www.codefab.com, if curious] includes a first class python based XML-RPC client [open source'd soon]. Given that product is aimed at the corporate world (among others), the client must be able to deal with any existing infrastructure, including HTTP proxies that will so often be encountered in such situations. > (Bill, note that Python CVS isn't the "master repository" for > xmlrpclib; > if you come up with a clean and nice urllib2-based proxy transport, > you can just mail it to me) Will do. > > > 1) http://effbot.org/zone/python-urllib-requestbody.htm thanks, b.bum From pedronis@bluewin.ch Thu Dec 5 15:27:33 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 5 Dec 2002 16:27:33 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <02bb01c29c72$d59df960$6d94fea9@newmexico> From: "Just van Rossum" > > ["/path/to/archive.zip!packagename"] > > So the zip handler will have to do a little more than just checking for a *.zip > extension. I assume you do something like it in Jython, too? (I picked up the > '!' from you I think; it makes it easier/quicker to check for zipness.) yes at some point we used that, but then we switched to '/' on the other hand the extesion .zip (or .jar in our case) is mandatory. From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_04B1_01C29BB8.B665AE40 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable I think we shouldn't use sys.path for zip archives. Why not "copy" importing_modules_from_dlls(pyds) mechanism? Then an archive.zip would be imported if found in ARCHIVES directory or explicitly imported from __init__.py (obviously:archive.zip should = be in the same directory) Wiktor Sadowski art@wiktorsadowski.com ------=_NextPart_000_04B1_01C29BB8.B665AE40 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
I think we shouldn't use sys.path for zip archives.
Why not "copy" importing_modules_from_dlls(pyds) mechanism?
 
Then an archive.zip  would be imported if  = found in=20 ARCHIVES directory
or explicitly imported from  __init__.py = (obviously:archive.zip=20  should be in the same directory)
 
Wiktor Sadowski
 
------=_NextPart_000_04B1_01C29BB8.B665AE40-- From =?windows-1250?Q?Martin_v._L=F6wis?= Thu Dec 5 16:12:52 2002 From: =?windows-1250?Q?Martin_v._L=F6wis?= (=?windows-1250?Q?Martin_v._L=F6wis?=) Date: Thu, 5 Dec 2002 17:12:52 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <04b401c29bb0$5559e7e0$0201010a@wiktor> Message-ID: <007901c29c79$2a2fb260$aa2ce8d9@mira> > I think we shouldn't use sys.path for zip archives. Why do you think so? > Why not "copy" importing_modules_from_dlls(pyds) mechanism? What do you mean by that? What is "importing_modules_from_dlls"? If you mean "one .pyd file is one Python module", then this approach won't work for .zip files: People want many .py modules in a single .zip file. Regards, Martin From bbum@codefab.com Thu Dec 5 17:13:27 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Thu, 5 Dec 2002 12:13:27 -0500 Subject: [Python-Dev] xmlrpclib + proxy support followup Message-ID: , I'm in the midst of implementing the Proxy support in xmlrpclib's Transport pattern and have a [minor] patch that seems to be necessary to my implementation. I say 'seems' because I'm not that far in yet and may discover something else that makes this particular path of implementation invalid [but I don't think so]. In particular, my goal is to implement a subclass of Transport that leverages the capabilities of urllib2 to provide Proxy support, HTTPs support, etc.. in a single class. The goal is to provide the implementation entirely through the methods provided for creating and sending the request. That is, through make_connection(), send_request(), send_host(), send_user_agent(), and send_content(). However, I need a sixth method -- get_reply(). I patched the Transport class in xmlrpclib as such: def request(self, host, handler, request_body, verbose=0): # issue XML-RPC request .... errcode, errmsg, headers = self.get_reply(h) .... def get_reply(self, connection): return connection.getReply() This would seem to be a trivial change to the existing xmlrpclib that greatly increases the ability to modify its behavior via subclassing, etc... Onwards, b.bum From fredrik@pythonware.com Thu Dec 5 17:23:56 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 18:23:56 +0100 Subject: [Python-Dev] Re: xmlrpclib + proxy support followup References: Message-ID: <00c201c29c83$18dd99a0$ced241d5@hagrid> bill wrote: > However, I need a sixth method -- get_reply(). > > I patched the Transport class in xmlrpclib as such: > > def request(self, host, handler, request_body, verbose=0): > # issue XML-RPC request > .... > errcode, errmsg, headers = self.get_reply(h) > > .... > > def get_reply(self, connection): > return connection.getReply() > > This would seem to be a trivial change to the existing xmlrpclib that > greatly increases the ability to modify its behavior via subclassing, > etc... +1 (and feel free to add self.get_file(h) too, if you need that). From gmcm@hypernet.com Thu Dec 5 17:01:47 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 5 Dec 2002 12:01:47 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <027b01c29c6c$58d74860$6d94fea9@newmexico> Message-ID: <3DEF402B.782.6830EE86@localhost> On 5 Dec 2002 at 15:41, Samuele Pedroni wrote: > > Gordon McMillan wrote: > > > > > Here's what iu.py does: [...] > what about __path__'s role in this picture (btw this > is not a rhetoric question)? In iu, __path__ is (by default) treated just like a "local" sys.path. In fact, you *could* say that sys.path is just the __path__ of the package-with-no-name. BTW, iu does allow non string objects in sys.path. If it exposes "getmod(name)", that will be tried in preference to iu's own machinery. -- Gordon http://www.mcmillan-inc.com/ From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_0748_01C29BD3.465E8700 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable > I think we shouldn't use sys.path for zip archives. >Why do you think so? IMO:Why make changes if it's not necessary .? > Why not "copy" importing_modules_from_dlls(pyds) mechanism? >What do you mean by that? What is "importing_modules_from_dlls"? Should be "importing_module_from_dlls". >If you mean "one .pyd file is one Python module", then this approach >won't work for .zip files:People want many .py modules in a single .zip >file. It will work if handled by the zip importer. As for ARCHIVES directory: a "virtual" path for all zip files could be = built for lookup. Importing from __init__.py could be done explicitly with: from MyArchive_with_many_modules import MyModule or import MyArchive_with_many_modules.MyModule where MyArchive_with_many_modules is MyArchive_with_many_modules.zip=20 in the same directory (or in ARCHIVES) Regards, Wiktor=20 ------=_NextPart_000_0748_01C29BD3.465E8700 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
> I think we shouldn't use sys.path for zip archives.
>Why = do you=20 think so?
IMO:Why make changes if it's not necessary .?
 
> Why not "copy" importing_modules_from_dlls(pyds)=20 mechanism?
>What do you mean by that? What is=20 "importing_modules_from_dlls"?
 
Should be "importing_module_from_dlls".

>If you mean "one .pyd file is one Python module", then this = approach
>won't work for .zip files:People want many .py modules = in a=20 single .zip
>file.
 
It will work if handled by the zip importer.
 
As for ARCHIVES directory: a "virtual" path for all zip files could = be=20 built for lookup.
Importing from __init__.py could be done explicitly with:
 
from MyArchive_with_many_modules import MyModule
or
import MyArchive_with_many_modules.MyModule
 
where  MyArchive_with_many_modules is=20 MyArchive_with_many_modules.zip
in the same directory (or in ARCHIVES)
 
Regards,
Wiktor 
 
 
 
 
 
 
 



------=_NextPart_000_0748_01C29BD3.465E8700-- From niemeyer@conectiva.com Thu Dec 5 19:30:53 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 17:30:53 -0200 Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 In-Reply-To: References: Message-ID: <20021205173053.A18494@ibook.distro.conectiva> > ================================ > `Dictionary evaluation order`__ > ================================ > __ http://mail.python.org/pipermail/python-dev/2002-November/030458.html > [...] > Finn Bock said that Jython went f1, f2, f3, f4. In that case Guido didn't > see any reason to fix it. But Tim Peters brought up the point that the [...] I belive the correct sentence here would be: "In that case Guido didn't see any reason to *block* the fix." -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From =?windows-1250?Q?Martin_v._L=F6wis?= Thu Dec 5 20:01:25 2002 From: =?windows-1250?Q?Martin_v._L=F6wis?= (=?windows-1250?Q?Martin_v._L=F6wis?=) Date: Thu, 5 Dec 2002 21:01:25 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <074b01c29bca$e5298dc0$0201010a@wiktor> Message-ID: <016f01c29c99$18967c80$aa2ce8d9@mira> >> Why not "copy" importing_modules_from_dlls(pyds) mechanism? >>What do you mean by that? What is "importing_modules_from_dlls"? >Should be "importing_module_from_dlls". I still don't understand. Is that some feature of Python? >>If you mean "one .pyd file is one Python module", then this approach >>won't work for .zip files:People want many .py modules in a single .zip >>file. >It will work if handled by the zip importer. How will the zip importer know what zipfile to read? If I have "python23.zip", and do import httplib what mechanism will extract httplib.py from python23.zip? > from MyArchive_with_many_modules import MyModule So you could only zip entire packages? This is insufficient. Regards, Martin From niemeyer@conectiva.com Thu Dec 5 21:26:16 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 19:26:16 -0200 Subject: [Python-Dev] Another approach for the import mechanism Message-ID: <20021205192616.B18494@ibook.distro.conectiva> I propose a different approach to the importer hook mechanism: - An importer register itself with sys.register_importer(), as suggested by MAL. - No .zip/.tar/.whatever files are ever included in sys.path - Alternative importers are only considered when the default importer mechanism fails (each entry in the path at a time, so that precedence is preserved). - Alternative importers, when activated, would be given the module name being imported, and would look for entries in the "current" iterated path by themselves. - A zip importer would, for example, look if the current iterated path have a file named "__init__.zip". - The same importer could also be able to look for a file named "mypackage.zip", if "import mypackage.foo" is tried. What's your opinion about that? -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From sholden@holdenweb.com Thu Dec 5 21:28:25 2002 From: sholden@holdenweb.com (Steve Holden) Date: Thu, 5 Dec 2002 16:28:25 -0500 Subject: [Python-Dev] New and Improved Import Hooks References: <074b01c29bca$e5298dc0$0201010a@wiktor> <016f01c29c99$18967c80$aa2ce8d9@mira> Message-ID: <016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com> [MvL] > >> Why not "copy" importing_modules_from_dlls(pyds) mechanism? > >>What do you mean by that? What is "importing_modules_from_dlls"? > [Wiktor] > >Should be "importing_module_from_dlls". > > I still don't understand. Is that some feature of Python? > I think Wiktor is referring to extensions, which under Windows are often provided as .pyd DLLs. > >>If you mean "one .pyd file is one Python module", then this approach > >>won't work for .zip files:People want many .py modules in a single > .zip > >>file. > > >It will work if handled by the zip importer. > > How will the zip importer know what zipfile to read? > > If I have "python23.zip", and do > > import httplib > > what mechanism will extract httplib.py from python23.zip? > > > from MyArchive_with_many_modules import MyModule > > So you could only zip entire packages? This is insufficient. > I agree. What I'd like to see is the ability to replace a directory on sys.path by a reference to a zipped copy, including any subdirectories, and have all imports work just as they did before (whether they be extensions or pure Python modules]. A quick look at my own code base, admittedly puny, reveals no statements that would break were non-strings to be added to sys.path. Is there some way to trap assignments to sys.path so a warning could be issued if sufficiently un-stringy elements were discovered in it? Clearly it isn't enough to trap assignment to and insertion of elements if breakage is to be totaly avoided. We have to know when the name is assigned-to as well. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From sholden@holdenweb.com Thu Dec 5 21:35:52 2002 From: sholden@holdenweb.com (Steve Holden) Date: Thu, 5 Dec 2002 16:35:52 -0500 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva> Message-ID: <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> > I propose a different approach to the importer hook mechanism: > > - An importer register itself with sys.register_importer(), as suggested > by MAL. > > - No .zip/.tar/.whatever files are ever included in sys.path > > - Alternative importers are only considered when the default importer > mechanism fails (each entry in the path at a time, so that > precedence is preserved). > Far too restrictive. You seem to be saying (unless I misunderstand) that an alternative import can't override an element on the traditional sys.path. Is there to be no way to replace existing system components by providing replacements in zip files, in the same way that I could currently replace httplib by putting my own httplib in a directory which I then place at the beginning of sys.path? > - Alternative importers, when activated, would be given the module name > being imported, and would look for entries in the "current" iterated > path by themselves. > Huh? Why restrict alternative importers to using a path? What if my importer wants to provide randomly generated byte codes as part of a genetic programming experiment? > - A zip importer would, for example, look if the current iterated path > have a file named "__init__.zip". > ...and I'm not really sure what you mean by "iterated path". > - The same importer could also be able to look for a file named > "mypackage.zip", if "import mypackage.foo" is tried. > > What's your opinion about that? > I think I must be misunderstanding at a fairly basic level. This would seem pretty inflexible to me. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- > http://mail.python.org/mailman/listinfo/python-dev > > From niemeyer@conectiva.com Thu Dec 5 21:41:18 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 19:41:18 -0200 Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS In-Reply-To: References: <15852.57674.16727.930806@montanaro.dyndns.org> Message-ID: <20021205194118.C18494@ibook.distro.conectiva> > Feel free to only edit Misc/NEWS; before releases I'll go through it > to find things I've missed, so there's no need to update whatsnew23.tex. > I don't mind if people edit whatsnew23.tex to add items, but it's > certainly not necessary. Thanks for mentioning that. I was really wondering about what to do. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Thu Dec 5 21:58:55 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 19:58:55 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> Message-ID: <20021205195855.A19843@ibook.distro.conectiva> > > - Alternative importers are only considered when the default importer > > mechanism fails (each entry in the path at a time, so that > > precedence is preserved). > > Far too restrictive. You seem to be saying (unless I misunderstand) > that an alternative import can't override an element on the > traditional sys.path. Why not? I belive it's exactly the other way around. You have more possibilities with that scheme. > Is there to be no way to replace existing system components by > providing replacements in zip files, in the same way that I could > currently replace httplib by putting my own httplib in a directory > which I then place at the beginning of sys.path? I'm not sure if I understood what you're trying to say. But if I understood that correctly, there are even more possibilities: Having one of: /some/dir/httplib/__init__.zip /some/dir/httplib.zip You'd do: sys.path.insert(0, "/some/dir") > > - Alternative importers, when activated, would be given the module name > > being imported, and would look for entries in the "current" iterated > > path by themselves. > > Huh? Why restrict alternative importers to using a path? What if my > importer wants to provide randomly generated byte codes as part of a > genetic programming experiment? Then you'd have to work with ihooks. IIRC, none of the proposed mechanisms would allow that. Also, I wouldn't say that "randomly generated byte codes" should be *imported*. Just compile it. > > - A zip importer would, for example, look if the current iterated path > > have a file named "__init__.zip". > > ...and I'm not really sure what you mean by "iterated path". for path in sys.path: import_default(path) import_alternate(path) > I think I must be misunderstanding at a fairly basic level. This would > seem pretty inflexible to me. Does it look better now? -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From fredrik@pythonware.com Thu Dec 5 22:01:18 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 5 Dec 2002 23:01:18 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva> Message-ID: <005701c29ca9$d8517ba0$ced241d5@hagrid> Gustavo Niemeyer wrote: > What's your opinion about that? Who's going to call sys.register_importer()? What does "current iterated path" mean? How do I create a ZIP-only distribution? Do I need to restructure my code to use custom importers, or am I missing something? If I want to ship multiple ZIP archives, do I really have to name each one of them __init__.zip and place each one in a different directory? Have you tried Just's patch? What problem does your solution solve that his code doesn't already handle? Etc. From pedronis@bluewin.ch Thu Dec 5 21:54:11 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 5 Dec 2002 22:54:11 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> Message-ID: <06e601c29ca8$d8a31380$6d94fea9@newmexico> From: "Gustavo Niemeyer" > Having one of: > > /some/dir/httplib/__init__.zip > /some/dir/httplib.zip > > You'd do: > > sys.path.insert(0, "/some/dir") > it's confusing (and imcompatible wrt to how Java/Jython do things now). __init__.py exists so that a package has a first-class counterpart, namely a first class-module. Zipfiles should be able to embrace more than just one package and should be transparent. From skip@pobox.com Thu Dec 5 22:09:34 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 5 Dec 2002 16:09:34 -0600 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021205195855.A19843@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> Message-ID: <15855.52894.524073.885235@montanaro.dyndns.org> Gustavo> Having one of: Gustavo> /some/dir/httplib/__init__.zip Gustavo> /some/dir/httplib.zip Gustavo> You'd do: Gustavo> sys.path.insert(0, "/some/dir") Remember, a zip file looks like a directory because of its embedded structure. If you had httplib.zip it would have to contain at least one .py or .so (or .pyd or ...) file. I think you would insert /some/dir/httplib.zip into sys.path. -- Skip Montanaro - skip@pobox.com http://www.mojam.com/ http://www.musi-cal.com/ From martin@v.loewis.de Thu Dec 5 22:14:50 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 23:14:50 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021205192616.B18494@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > - No .zip/.tar/.whatever files are ever included in sys.path [...] > What's your opinion about that? It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they MUST be considered in order with all other items in PYTHONPATH; the order requirement also applies for sys.path. Regards, Martin From bbum@codefab.com Thu Dec 5 22:24:19 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Thu, 5 Dec 2002 17:24:19 -0500 Subject: [Python-Dev] urllib2.build_opener() broken? Message-ID: <4C444805-08A0-11D7-B650-000393877AE4@codefab.com> I'm invoking build_opener() as... opener = urllib2.build_opener(proxyHandler) .... and the handlers appear to be out of order-- the ProxyHandler that is supposed to be at the beginning is not ... [, , , , , , , ] From the comments on build_opener(), it would seem that it doesn't take this into account. However, if it doesn't, then the example used in the documentation at the top of urllib2 could not possibly work. Whose broken? The module or me? thanks, b.bum From skip@pobox.com Thu Dec 5 22:24:55 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 5 Dec 2002 16:24:55 -0600 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <15855.52894.524073.885235@montanaro.dyndns.org> References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> <15855.52894.524073.885235@montanaro.dyndns.org> Message-ID: <15855.53815.906868.938394@montanaro.dyndns.org> Skip> Remember, a zip file looks like a directory because of its Skip> embedded structure. If you had httplib.zip it would have to Skip> contain at least one .py or .so (or .pyd or ...) file. I think Skip> you would insert /some/dir/httplib.zip into sys.path. Just rereading my own note, I realize you might have components after the zip file name. Suppose you had web.zip which contained directories http, gopher, ftp, and nntp, each of which was itself a package (had a __init__.py file). You might modify sys.path like so: newdirs = [os.path.join("/some/dir/web.zip", x) for x in "http gopher ftp nttp".split()] sys.path.extend(newdirs) Skip From bbum@codefab.com Thu Dec 5 22:27:08 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Thu, 5 Dec 2002 17:27:08 -0500 Subject: [Python-Dev] build_opener() broken Message-ID: It is broken, not me. See bug #487471. https://sourceforge.net/tracker/ ?group_id=5470&atid=105470&func=detail&aid=487471 Guess I have to fix that, too. b.bum From niemeyer@conectiva.com Thu Dec 5 22:31:37 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 20:31:37 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <005701c29ca9$d8517ba0$ced241d5@hagrid> References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> Message-ID: <20021205203137.C19843@ibook.distro.conectiva> > > What's your opinion about that? > > Who's going to call sys.register_importer()? Sorry, but I'll answer your question with another question: Who's going to notify to the importer mechanism that zip importers are available, in the current implementation? > What does "current iterated path" mean? for path in sys.path: default_importer(path) alternative_importer(path) > How do I create a ZIP-only distribution? Include __init__.zip in /usr/lib/python2.2. > Do I need to restructure my code to use custom importers, or am > I missing something? Why would you need to? > If I want to ship multiple ZIP archives, do I really have to name each > one of them __init__.zip and place each one in a different directory? It depends on what your zip archives are. If you want to include their content as "top level" modules (as the standard library), yes. If they are packages by themselves (like the email package), no. > Have you tried Just's patch? Do you agree that this is a different approach? > What problem does your solution solve that his code doesn't already > handle? - Avoids including zip files in the path. - Allows one to have zip packages inside other packages. - Allows me to ship a package inside a zip file, without asking the user to change his path. - Allows me to compress a single file (foobar.py.bz2). -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Thu Dec 5 22:33:12 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 20:33:12 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: References: <20021205192616.B18494@ibook.distro.conectiva> Message-ID: <20021205203312.A20315@ibook.distro.conectiva> > It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they Why? Have you promised that to someone? :-)) > MUST be considered in order with all other items in PYTHONPATH; > the order requirement also applies for sys.path. I told that would be onored (perhaps in an uncomprehensible way). Each path is tried with the alternative importers once the default importer has failed on *that* specific path. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From martin@v.loewis.de Thu Dec 5 22:38:50 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 05 Dec 2002 23:38:50 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com> References: <074b01c29bca$e5298dc0$0201010a@wiktor> <016f01c29c99$18967c80$aa2ce8d9@mira> <016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com> Message-ID: "Steve Holden" writes: > > >Should be "importing_module_from_dlls". > > > > I still don't understand. Is that some feature of Python? > > > I think Wiktor is referring to extensions, which under Windows are often > provided as .pyd DLLs. Yes, but then I understand how that is related to zipfile imports: A .pyd provides only a single module. A .zip file must supply many modules. > A quick look at my own code base, admittedly puny, reveals no statements > that would break were non-strings to be added to sys.path. Is there some way > to trap assignments to sys.path so a warning could be issued if sufficiently > un-stringy elements were discovered in it? This isn't really possible, as this is just a dict modification. You'ld also have to catch assignments to sys.__dict__['path'] Regards, Martin From pedronis@bluewin.ch Thu Dec 5 22:28:54 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 5 Dec 2002 23:28:54 +0100 Subject: [Python-Dev] Another approach for the import mechanism Message-ID: <086b01c29cad$b1de3400$6d94fea9@newmexico> From: "Gustavo Niemeyer" > It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they > > Why? Have you promised that to someone? :-)) Jython has implented it that way because it was agreed that this was what PEP 273 was going to allow. regards. From niemeyer@conectiva.com Thu Dec 5 22:44:01 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 20:44:01 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <15855.53815.906868.938394@montanaro.dyndns.org> References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> <15855.52894.524073.885235@montanaro.dyndns.org> <15855.53815.906868.938394@montanaro.dyndns.org> Message-ID: <20021205204401.D19843@ibook.distro.conectiva> > Just rereading my own note, I realize you might have components after the > zip file name. Suppose you had web.zip which contained directories http, > gopher, ftp, and nntp, each of which was itself a package (had a __init__.py > file). You might modify sys.path like so: > > newdirs = [os.path.join("/some/dir/web.zip", x) > for x in "http gopher ftp nttp".split()] > sys.path.extend(newdirs) I'm not sure how useful that would be (i.e. I haven't done that by myself), but that could be handled in the proposed mechanism as well. If /some/dir/web/http/gopher doesn't exist, that mechanism could check if http.zip exists, or web.zip exists (or __init__.zip inside those directories). This also has the advantage that you don't have to enforce a specific format in your code. If you later discover that tar.bz2 has a better compression, just go for it. OTOH, I understand that this would lead to extra tests in comparison with your example. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Thu Dec 5 22:52:38 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 20:52:38 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <06e601c29ca8$d8a31380$6d94fea9@newmexico> References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> <06e601c29ca8$d8a31380$6d94fea9@newmexico> Message-ID: <20021205205238.E19843@ibook.distro.conectiva> > it's confusing (and imcompatible wrt to how Java/Jython do things now). Sorry. I wouldn't like to do something in a different way than Jython does, or to obligate you to rework that. OTOH, I belive that this is a better scheme, and my obligation is to discuss that with you. If it's of common sense that this isn't good, it won't be accepted, and that's all. > __init__.py exists so that a package has a first-class counterpart, > namely a first class-module. That's a very similar idea. But if you don't like __init__.zip, just rename it to __module__.zip, or whatever. > Zipfiles should be able to embrace more than just one package and > should be transparent. Can you please explain how the propose changes that? -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Thu Dec 5 22:56:21 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 20:56:21 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <086b01c29cad$b1de3400$6d94fea9@newmexico> References: <086b01c29cad$b1de3400$6d94fea9@newmexico> Message-ID: <20021205205621.F19843@ibook.distro.conectiva> > > Why? Have you promised that to someone? :-)) > > Jython has implented it that way because it was agreed that this was what PEP > 273 was going to allow. [replicated] Sorry. I wouldn't like to do something in a different way than Jython does, or to obligate you (or someone) to rework that. OTOH, I belive that this is a better scheme, and my obligation is to discuss that with you. If it's of common sense that this isn't good, it won't be accepted, and that's all. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From pedronis@bluewin.ch Thu Dec 5 22:45:47 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 5 Dec 2002 23:45:47 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> Message-ID: <09e901c29cb0$0da094c0$6d94fea9@newmexico> > - Allows one to have zip packages inside other packages. you can achieve this with __path__ > > - Avoids including zip files in the path. I want to support that. > > - Allows me to ship a package inside a zip file, without asking > the user to change his path. > > - Allows me to compress a single file (foobar.py.bz2). > Java has support for zipfiles from day one, from my user experience they are both YAGNI. sys.path content from my point of view is really a deployment time issue. The 2nd point is not directly addressed by your proposal, and supporting it would make for more complicated importer impls. regards. From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_07AE_01C29BF4.AE00AAC0 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable >Should be "importing_module_from_dlls". >I still don't understand. Is that some feature of Python? Under Windows you can put a Module.pyd into Python/DLLs directory and: import Module >>If you mean "one .pyd file is one Python module", then this approach >>won't work for .zip files:People want many .py modules in a single .zip >>file. >It will work if handled by the zip importer. >How will the zip importer know what zipfile to read? >If I have "python23.zip", and do >import httplib >what mechanism will extract httplib.py from python23.zip? As I said: a virtual directory-tree could be built for all zip files in=20 ARCHIVES directory: ZipArchives: ZipArchives/python23.zip: ZipArchives/python23.zip/Lib: ZipArchives/python23.zip/Lib/httplib.pyc ZipArchives/another.zip:=20 ZipArchives/another.zip/another_directory: = ZipArchives/another.zip/another_directory/another_module.pyc =20 > from MyArchive_with_many_modules import MyModule >So you could only zip entire packages?=20 No. "from MyArchive import MyModule" would be resolved by zip importer as: if there is MyModule in MyArchive.zip then then import MyModule (means extract and import) and import Module: if Module.zip contains Module import Module Wiktor ------=_NextPart_000_07AE_01C29BF4.AE00AAC0 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
>Should be "importing_module_from_dlls".

>I still = don't=20 understand. Is that some feature of Python?
 
Under Windows you can put a Module.pyd into Python/DLLs directory=20 and:
import=20 Module

>>If you mean = "one .pyd file=20 is one Python module", then this approach
>>won't work for .zip = files:People want many .py modules in a=20 single
.zip
>>file.

>It will work if handled by = the zip=20 importer.

>How will the zip importer know what zipfile to=20 read?

>If I have "python23.zip", and do

>import=20 httplib

>what mechanism will extract httplib.py from=20 python23.zip?
 
As I said: a virtual=20 directory-tree could be built for all zip files in
ARCHIVES = directory:
 
ZipArchives:
  =20 ZipArchives/python23.zip:
          &nbs= p;    =20 ZipArchives/python23.zip/Lib:
          &nbs= p;        =20 ZipArchives/python23.zip/Lib/httplib.pyc
   ZipArchives/another.zip: 
          &nbs= p;   =20 ZipArchives/another.zip/another_directory:
          &nbs= p;           =20 ZipArchives/another.zip/another_directory/another_module.pyc  &= nbsp;  =20
 
> from = MyArchive_with_many_modules=20 import MyModule

>So you could only zip entire packages? =
No.
"from MyArchive import=20 MyModule"
would be resolved by zip = importer=20 as:
if there is MyModule in = MyArchive.zip=20 then then import MyModule
(means extract and = import)
 
and
import Module:
if Module.zip contains = Module import=20 Module
 
Wiktor
 ------=_NextPart_000_07AE_01C29BF4.AE00AAC0-- From martin@v.loewis.de Thu Dec 5 23:05:18 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 00:05:18 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021205203312.A20315@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > > It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they > > Why? Have you promised that to someone? :-)) Sure. See PEP 273. Also, it is so similar to the Java CLASSPATH feature that it better be identical. > > MUST be considered in order with all other items in PYTHONPATH; > > the order requirement also applies for sys.path. > > I told that would be onored (perhaps in an uncomprehensible way). If so, it was indeed incomprehensible :-( Rereading your proposal, it appears that you are also proposing that you can only package up entire Python packages with your strategy. It also is a requirement that you can zip up the Pythons standard library. Regards, Martin From martin@v.loewis.de Thu Dec 5 23:17:44 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 00:17:44 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <07b101c29bec$4c902f20$0201010a@wiktor> References: <07b101c29bec$4c902f20$0201010a@wiktor> Message-ID: "Wiktor Sadowski" writes: > >I still don't understand. Is that some feature of Python? > > Under Windows you can put a Module.pyd into Python/DLLs directory and: > import Module I see. Notice that there is nothing special about the DLLs directory. Instead, you can have .pyd files anywhere on sys.path, and the DLLs directory is included in sys.path - that's why .pyd files are found in the DLLs directory. > >If I have "python23.zip", and do > > >import httplib [...] > >So you could only zip entire packages? > No. All your explanations indicate otherwise. > "from MyArchive import MyModule" > would be resolved by zip importer as: > if there is MyModule in MyArchive.zip then then import MyModule > (means extract and import) > > and > import Module: > if Module.zip contains Module import Module So if I do import httplib it will check httplib.zip. Correct? That won't help, since httplib is in python23.zip, not in httplib.zip. According to your proposal, I would have to write from python23 import httplib But I don't want to, since httplib is a toplevel module, which is not included in a python23 package. Regards, Martin From niemeyer@conectiva.com Thu Dec 5 23:21:27 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 21:21:27 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <09e901c29cb0$0da094c0$6d94fea9@newmexico> References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> Message-ID: <20021205212127.G19843@ibook.distro.conectiva> > > - Allows one to have zip packages inside other packages. > > you can achieve this with __path__ Indeed. Thanks for mentioning that. > > - Avoids including zip files in the path. > > I want to support that. Well... :-) > > - Allows me to ship a package inside a zip file, without asking > > the user to change his path. > > > > - Allows me to compress a single file (foobar.py.bz2). > > Java has support for zipfiles from day one, from my user experience > they are both YAGNI. If I want to provide packaged RPMs for small systems, I'd like that *every* module/package which can be installed optionally, be packaged inside a compressed file. There are real usage for that right now. > sys.path content from my point of view is really a deployment time > issue. I'm not sure about what you mean here. > The 2nd point is not directly addressed by your proposal, and supporting it > would make for more complicated importer impls. Yes, it is addressed. My propose is to allow the importer mechanism to check if there's something importable with the given name in the given path, instead of telling him what to import. With that in mind, importing foobar.py.bz2 is just a matter of checking for .py.bz2 in the given path. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Thu Dec 5 23:26:04 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 5 Dec 2002 21:26:04 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> Message-ID: <20021205212604.H19843@ibook.distro.conectiva> > If so, it was indeed incomprehensible :-( > > Rereading your proposal, it appears that you are also proposing that > you can only package up entire Python packages with your strategy. > > It also is a requirement that you can zip up the Pythons standard > library. I need some english classes then. :-) /usr/lib/python2.2/__init__.zip (or __module__.zip, or whatever) -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From martin@v.loewis.de Thu Dec 5 23:33:41 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 00:33:41 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021205212127.G19843@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> <20021205212127.G19843@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > Yes, it is addressed. My propose is to allow the importer mechanism to > check if there's something importable with the given name in the given > path, instead of telling him what to import. With that in mind, > importing foobar.py.bz2 is just a matter of checking for > .py.bz2 in the given path. It appears that this can't be really done with the proposed mechanism, as you want to have a multiple hooks for a single source "URL". If you *only* want to have a .py.bz2 importer, you could replace every directory on sys.path with the pybz2importer, which would check for .py.bz2 files in the directory, and, if that fails, calls imp.find_module. Of course, if somebody wants to provide a crypt importer (which decrypts the source before importing it) in addition to the pybz2importer, then you get the same coordination problem as with all prior import hooks. So you are back to calling all registered hooks for all items on sys.path, which might be expensive. Regards, Martin From martin@v.loewis.de Thu Dec 5 23:37:55 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 00:37:55 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021205212604.H19843@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> <20021205212604.H19843@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > I need some english classes then. :-) > > /usr/lib/python2.2/__init__.zip (or __module__.zip, or whatever) So you can have only a single multi-file non-package zipfile in each directory on sys.path? I find this quite ugly: a directory with only a single file in it, and potentially many of these. What is the advantage of this limitation? Regards, Martin From pedronis@bluewin.ch Thu Dec 5 23:31:45 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 6 Dec 2002 00:31:45 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> <20021205212127.G19843@ibook.distro.conectiva> Message-ID: <0d8401c29cb6$79c48b60$6d94fea9@newmexico> > > If I want to provide packaged RPMs for small systems, I'd like > that *every* module/package which can be installed optionally, be > packaged inside a compressed file. There are real usage for > that right now. > Not that I exactly understand what you want, but it seems that some logic in site.py, such that all *.zip found e.g. in site-archives (and similars) (thought as equivalent and parallel to site-packages) are added to sys.path, would solve your problem. You either install a dir hiearchy somewhere or an archive there. I insist that supporting compressing single modules seems YAGNI and requiring too much complexity. regards. From pedronis@bluewin.ch Thu Dec 5 23:52:21 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 6 Dec 2002 00:52:21 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> Message-ID: <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico> From: "Gustavo Niemeyer" > - Allows me to ship a package inside a zip file, without asking > the user to change his path. > btw for single packages (once you can put zipfiles in sys.path or __path__) you can achieve this with __path__ package/ __init__.py __path__[0] = os.path.join(__path__[0],'package.zip') package.zip regards. From brian@sweetapp.com Fri Dec 6 00:14:53 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Thu, 05 Dec 2002 16:14:53 -0800 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: Message-ID: <00a901c29cbc$803d2230$21795418@dell1700> > Of course, if somebody wants to provide a crypt importer (which > decrypts the source before importing it) in addition to the > pybz2importer, then you get the same coordination problem as with all > prior import hooks. Does anybody really believe that this problem can be solved in general? I like the following: 1. only strings representing directories are allowed in sys.path (a zip archive is conceptually a directory) 2. there are two types of hook that can be installed: - an import hook that returns a module object, a stream object or None - a stream hook that has the opportunity to return a different stream if it likes When the import hook returns a stream, every stream hook is given a chance to transform that stream. If a stream hook returns a transformed stream then every other hook is called again. This continues until no stream hook is interested in transforming the stream anymore. I know that this might have performance issues, but it seems like it would allow you to have compressed archives of encrypted modules located on a web server without any of the hooks knowing anything about each other. Cheers, Brian From bac@OCF.Berkeley.EDU Fri Dec 6 00:24:52 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Thu, 5 Dec 2002 16:24:52 -0800 (PST) Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 In-Reply-To: <20021205173053.A18494@ibook.distro.conectiva> References: <20021205173053.A18494@ibook.distro.conectiva> Message-ID: [Gustavo Niemeyer] > > ================================ > > `Dictionary evaluation order`__ > > ================================ > > __ http://mail.python.org/pipermail/python-dev/2002-November/030458.html > > > [...] > > Finn Bock said that Jython went f1, f2, f3, f4. In that case Guido didn't > > see any reason to fix it. But Tim Peters brought up the point that the > [...] > > I belive the correct sentence here would be: > > "In that case Guido didn't see any reason to *block* the fix." > Oops. =) So this brings up something I have been pondering; what to do when a mistake in a summary is found after it is already out the door. I can obviously fix the web version, but is it really worth it since I think most people read the text version that goes to comp.lang.python(.announce)? . Does anyone object if I just leave it as-is unless it is a major error that changes the meaning of the summary? This obviously makes it seems like Guido said something he didn't, but the outcome in the end is the same; the change was made. And I don't want to have to go back and make changes to summaries that are 6 months old or something when the issue has long past. So, keep the errors in the archive or change them? -Brett From paul@pfdubois.com Fri Dec 6 01:02:26 2002 From: paul@pfdubois.com (Paul F Dubois) Date: Thu, 5 Dec 2002 17:02:26 -0800 Subject: [Python-Dev] PEP 242 Numeric kinds Message-ID: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> We had put off consideration of PEP 242 Numeric kinds until June 2002 when a meeting of some interested parties was to occur but the meeting didn't occur. I have a draft implementation and users of it but my feeling is that although correct and useful the PEP is not useful enough to put it in the standard library. Since it really only interests scientific programmers I propose simply making it a separate deliverable on the Numeric site and withdrawing the PEP. From skip@pobox.com Fri Dec 6 01:22:35 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 5 Dec 2002 19:22:35 -0600 Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 In-Reply-To: References: <20021205173053.A18494@ibook.distro.conectiva> Message-ID: <15855.64475.165789.875723@montanaro.dyndns.org> Brett> So this brings up something I have been pondering; what to do Brett> when a mistake in a summary is found after it is already out the Brett> door. Fix what you can. (tm) Skip From niemeyer@conectiva.com Fri Dec 6 02:11:02 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Fri, 6 Dec 2002 00:11:02 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico> References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico> Message-ID: <20021206001101.A21140@ibook.distro.conectiva> > btw for single packages (once you can put zipfiles in sys.path or > __path__) you can achieve this with __path__ > > package/ > __init__.py > __path__[0] = os.path.join(__path__[0],'package.zip') > > package.zip That would kill the __init__.py that could be inside package.zip, right? One could leave package.zip's __init__.py outside it, and hack it as shown above, but it'd be great if this scheme was just an option for setup.py (--compress-packages). Otherwise the scheme must be prepared by the developer, not by the packager. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Fri Dec 6 02:17:36 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Fri, 6 Dec 2002 00:17:36 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> <20021205212127.G19843@ibook.distro.conectiva> Message-ID: <20021206001736.B21140@ibook.distro.conectiva> > > Yes, it is addressed. My propose is to allow the importer mechanism to > > check if there's something importable with the given name in the given > > path, instead of telling him what to import. With that in mind, > > importing foobar.py.bz2 is just a matter of checking for > > .py.bz2 in the given path. > > It appears that this can't be really done with the proposed mechanism, > as you want to have a multiple hooks for a single source "URL". As you have mentioned below, all hooks registered are called, if the previous ones failed. So there shouldn't be any problem. > If you *only* want to have a .py.bz2 importer, you could replace every > directory on sys.path with the pybz2importer, which would check for > .py.bz2 files in the directory, and, if that fails, calls > imp.find_module. > > Of course, if somebody wants to provide a crypt importer (which > decrypts the source before importing it) in addition to the > pybz2importer, then you get the same coordination problem as with all > prior import hooks. Yes, that's the idea behind all proposes after all. > So you are back to calling all registered hooks for all items on > sys.path, which might be expensive. Agreed. But further hooks are only called if the default one failed. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From j_r_fonseca@yahoo.co.uk Fri Dec 6 02:37:28 2002 From: j_r_fonseca@yahoo.co.uk (=?iso-8859-15?Q?Jos=E9?= Fonseca) Date: Fri, 6 Dec 2002 02:37:28 +0000 Subject: [Python-Dev] Re: [Numpy-discussion] PEP 242 Numeric kinds In-Reply-To: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> Message-ID: <20021206023728.GA24182@localhost.localdomain> On Thu, Dec 05, 2002 at 05:02:26PM -0800, Paul F Dubois wrote: > Since it really only interests > scientific programmers I propose simply making it a separate deliverable > on the Numeric site and withdrawing the PEP. I'm not fully aware about the advantages/implications of puting Numeric in Python standard library, but to say that the manipulation of numeric arrays is only of interest to scientific programmers is the same of when in the early computing days engineers saing that computers would only be good for crunching numbers, and that the concept of personal computers was just non-sense... For a non-scientic usage of Numeric see the examples in http://www.pygame.org/pcr/repository.php, but I can imagine the usefullness of Numeric in many more non-scientific applications: imaging, sound visualization plugins, 3D graphics, and probably much more. The use of Numeric can speed up alot of algorithms which would be otherwise very slow in pure Python, and therefore forcing one to write C extensions. That's why IMHO something with the Numeric functionality should exist in the Python standard library. Note that this is not the same that saying that Numeric should be included as is - perhaps it's better to have it seperately to let it mature more, perhaps not - but still, there is much more than a niche interest around it. José Fonseca __________________________________________________ Do You Yahoo!? Everything you'll ever need on one web page from News and Sport to Email and Music Charts http://uk.my.yahoo.com From niemeyer@conectiva.com Fri Dec 6 02:53:52 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Fri, 6 Dec 2002 00:53:52 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> <20021205212604.H19843@ibook.distro.conectiva> Message-ID: <20021206005352.C21140@ibook.distro.conectiva> > So you can have only a single multi-file non-package zipfile in each > directory on sys.path? I find this quite ugly: a directory with only a Well, with the current implementation you'll only allow one single multi-file non-package zipfile in each entry of sys.path. :-) > single file in it, and potentially many of these. What is the > advantage of this limitation? - Don't have to change path to use compressed packages (at least not if you want to provide compressed packages, individual compressed modules or the standard library). - Don't have to specify the compression type hardcoded. - Allows one to ship a package inside a zip file, without asking the user to change his path, and without hacking the package. - Allows one to compress a single file (foobar.py.bz2). I belive that my propose is quite clear now. If there are no additional supporters, there's no reason to go on. Thanks to everyone who discussed. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From niemeyer@conectiva.com Fri Dec 6 03:05:45 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Fri, 6 Dec 2002 01:05:45 -0200 Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 In-Reply-To: References: <20021205173053.A18494@ibook.distro.conectiva> Message-ID: <20021206010545.D21140@ibook.distro.conectiva> [...] > Does anyone object if I just leave it as-is unless it is a major error > that changes the meaning of the summary? This obviously makes it This specific case changes the meaning of the summary. It puts the BDFL in the exact oposite side, turning me into a subversor. ;-) [...] > So, keep the errors in the archive or change them? Well.. I don't really care about that specific case. But IMHO, if it's easy to fix, information should be changed at least in the web, even 6 months later, since there's no point in providing misgiven information if we know it's wrong. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From greg@cosc.canterbury.ac.nz Fri Dec 6 03:09:24 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Dec 2002 16:09:24 +1300 (NZDT) Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <00a901c29cbc$803d2230$21795418@dell1700> Message-ID: <200212060309.gB639OP12627@kuku.cosc.canterbury.ac.nz> Brian Quinlan : > When the import hook returns a stream, every stream hook is given a > chance to transform that stream. If a stream hook returns a transformed > stream then every other hook is called again. This continues until no > stream hook is interested in transforming the stream anymore. How is a stream hook to know what kind of stream it's been given? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From brian@sweetapp.com Fri Dec 6 03:37:19 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Thu, 05 Dec 2002 19:37:19 -0800 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <200212060309.gB639OP12627@kuku.cosc.canterbury.ac.nz> Message-ID: <00b101c29cd8$c7b25f60$21795418@dell1700> Greg Ewing wrote: > > When the import hook returns a stream, every stream hook is given a > > chance to transform that stream. If a stream hook returns a transformed > > stream then every other hook is called again. This continues until no > > stream hook is interested in transforming the stream anymore. > > How is a stream hook to know what kind of stream it's > been given? It will have to figure it out itself, given the original arguments to the import hook plus the stream itself. A zip hook might, for example, check that the first 4 bytes of the stream are 0x04034b50. If they aren't, it would have to repair the original stream (haven't really thought about how to do this) and signal its lack of interest. Otherwise it would open the archive, decompress it and return a new stream. The more I think about this mechanism, the fewer interesting problems it seems to solve. Maybe the real answer is to have a single import hook but write as much of it as possible in modular Python. Then the user can figure out how to manage the competing interests when importing archived, encrypted modules from an arbitrary URL. Cheers, Brian From aahz@pythoncraft.com Fri Dec 6 04:02:58 2002 From: aahz@pythoncraft.com (Aahz) Date: Thu, 5 Dec 2002 23:02:58 -0500 Subject: [Python-Dev] Import: where's the PEP? Message-ID: <20021206040258.GA24214@panix.com> Having just slogged my way through the entire import thread so far, I've got to ask why nobody else has demanded a PEP before we go any further? There are too many competing proposals; we need a good summary of what design we're planning, plus a summary of why all other proposals are rejected. I'm a very strong -1 on anything until there's a PEP. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From skip@pobox.com Fri Dec 6 04:33:59 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 5 Dec 2002 22:33:59 -0600 Subject: [Python-Dev] Import: where's the PEP? In-Reply-To: <20021206040258.GA24214@panix.com> References: <20021206040258.GA24214@panix.com> Message-ID: <15856.10423.787013.625507@montanaro.dyndns.org> aahz> I'm a very strong -1 on anything until there's a PEP. Try PEP 273. I don't believe Just's original proposal meant to go beyond that. Skip From aahz@pythoncraft.com Fri Dec 6 04:39:21 2002 From: aahz@pythoncraft.com (Aahz) Date: Thu, 5 Dec 2002 23:39:21 -0500 Subject: [Python-Dev] Import: where's the PEP? In-Reply-To: <15856.10423.787013.625507@montanaro.dyndns.org> References: <20021206040258.GA24214@panix.com> <15856.10423.787013.625507@montanaro.dyndns.org> Message-ID: <20021206043921.GA10641@panix.com> On Thu, Dec 05, 2002, Skip Montanaro wrote: > > aahz> I'm a very strong -1 on anything until there's a PEP. > > Try PEP 273. I don't believe Just's original proposal meant to go beyond > that. Doesn't look that way to me: Just is, after all, proposing a brand-new import hook mechanism. His proposal is *compatible* with PEP 273, but it's definitely not the same thing. That's not even talking about all the bandwidth that's been consumed with other ideas -- the whole point of the PEP mechanism is to avoid losing that valuable information! -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From python@rcn.com Fri Dec 6 05:32:03 2002 From: python@rcn.com (Raymond Hettinger) Date: Fri, 6 Dec 2002 00:32:03 -0500 Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 References: <20021205173053.A18494@ibook.distro.conectiva> <20021206010545.D21140@ibook.distro.conectiva> Message-ID: <002101c29ce8$d1ba7a00$770ca044@oemcomputer> > > Does anyone object if I just leave it as-is unless it is a major error > > that changes the meaning of the summary? Ideally, it would be best to fix *anything* that is incorrect. > > So, keep the errors in the archive or change them? As a recorder of history, accuracy takes precedence. In the archive, keep the original, add a bracketed section entitled Subsequent Corrections or some such, and make it right. In the next summary, add an errata section, and post the correction. Errata and amendments are like bugfixes, everyone admires the effort and knows they are better off with honest attempts at accuracy than they are with someone asserting that they always had it right to begin with. It's just good journalism Thanks again for your efforts to keep the community informed. Raymond From martin@v.loewis.de Fri Dec 6 06:25:37 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 07:25:37 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021206005352.C21140@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> <20021205212604.H19843@ibook.distro.conectiva> <20021206005352.C21140@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > > So you can have only a single multi-file non-package zipfile in each > > directory on sys.path? I find this quite ugly: a directory with only a > > Well, with the current implementation you'll only allow one single > multi-file non-package zipfile in each entry of sys.path. :-) Yes, but I don't have to clutter my disk for that. > - Don't have to change path to use compressed packages (at least > not if you want to provide compressed packages, individual > compressed modules or the standard library). I thought you just explained that I will need to change the path to provide a compressed standard library, to point to a directory that contains an __init__.zip. > - Don't have to specify the compression type hardcoded. I don't understand that remark. If I have a zipfile, I surely must install a zipfile hook in your approach also - a .tar.bz2 hook won't be able to load the zipfile, no? > - Allows one to ship a package inside a zip file, without asking > the user to change his path, and without hacking the package. > > - Allows one to compress a single file (foobar.py.bz2). This is really the same issue: If you had a mechanism to import a module from a .py.bz2 file, you could use the same mechanism to import a package (or subpackage) from a .zip file. While I think this might be desirable, I also think it was never the goal of PEP 273 to provide such a facility. Regards, Martin From martin@v.loewis.de Fri Dec 6 06:28:12 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 07:28:12 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021206001101.A21140@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico> <20021206001101.A21140@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > That would kill the __init__.py that could be inside package.zip, > right? One could leave package.zip's __init__.py outside it, and > hack it as shown above, but it'd be great if this scheme was just > an option for setup.py (--compress-packages). Otherwise the scheme > must be prepared by the developer, not by the packager. You can easily put an entire package into a .zip file, as the packager: Just provide a .zip file with the entire package contents (and file names starting with the package dir inside the zip file); then provide a .pth file to add the zipfile to sys.path. Regards, Martin From martin@v.loewis.de Fri Dec 6 06:34:16 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 07:34:16 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <20021206001736.B21140@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> <20021205212127.G19843@ibook.distro.conectiva> <20021206001736.B21140@ibook.distro.conectiva> Message-ID: Gustavo Niemeyer writes: > > It appears that this can't be really done with the proposed mechanism, > > as you want to have a multiple hooks for a single source "URL". > > As you have mentioned below, all hooks registered are called, if > the previous ones failed. So there shouldn't be any problem. I mentioned this as a theoretical architecture which is not going to be implemented by Just. Whether or not this is a problem, I don't know. > > So you are back to calling all registered hooks for all items on > > sys.path, which might be expensive. > > Agreed. But further hooks are only called if the default one failed. Again, this is *not* the strategy that Just proposes, and, unless somebody provides an implementation of it, not one that will matter. As for "only called if the default one failed": This will be the normal case. In my standard installation, sys.path is ['', '/home/martin/work', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2', '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload', '/usr/local/lib/python2.3/site-packages', '/usr/local/lib/site-python'] So if I have 5 import hooks (including the default one), and I do "import Fnorb" (which is in site-python), I get 36 hook calls before the Fnorb package is found. For each element in sys.path, first the default hook will fail, and then all additional hooks, then it proceeds to the next element of sys.path. Quite expensive, potentially. Regards, Martin From martin@v.loewis.de Fri Dec 6 06:40:47 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 07:40:47 +0100 Subject: [Python-Dev] Import: where's the PEP? In-Reply-To: <20021206043921.GA10641@panix.com> References: <20021206040258.GA24214@panix.com> <15856.10423.787013.625507@montanaro.dyndns.org> <20021206043921.GA10641@panix.com> Message-ID: Aahz writes: > Doesn't look that way to me: Just is, after all, proposing a brand-new > import hook mechanism. His proposal is *compatible* with PEP 273, but > it's definitely not the same thing. That's not even talking about all > the bandwidth that's been consumed with other ideas -- the whole point > of the PEP mechanism is to avoid losing that valuable information! I agree that a PEP should be written for anything that is going to be accepted; I disagree that the whole point of the PEP mechanism is to avoid losing valuable information. Instead, I think the purpose of the PEP processis to focus the discussion, so all people know they are talking about the same thing, and you can better tell what remarks are off-topic, and avoid endless repetition of the same observations. In this discussion, I think PEP 273 helped to focus the discussion, as every proposal for a new import mechanism had to show that it can provide the feature of that PEP (i.e. zipfile imports). Regards, Martin From python@rcn.com Fri Dec 6 06:56:25 2002 From: python@rcn.com (Raymond Hettinger) Date: Fri, 6 Dec 2002 01:56:25 -0500 Subject: [Python-Dev] PEP 242 Numeric kinds References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> Message-ID: <002701c29cf4$98296c40$770ca044@oemcomputer> > I have a draft implementation and users of it but my > feeling is that although correct and useful the PEP is not useful enough > to put it in the standard library. Since it really only interests > scientific programmers I propose simply making it a separate deliverable > on the Numeric site and withdrawing the PEP. As the PEP author, your feeling is probably the most accurate. And, it is true that Kinds will, in general, appeal to the same crowd as Numeric. OTOH, "import kinds" is entirely unobtrusive and general purpose. The interface is simple; and may help some code become less platform sensitive (in the way the Metafont writes its own portable routines to create consistent lettering); and it has some educational value (making it easy to demonstrate the effects of truncation and floating point round-off). Also, I think accounting applications can benefit from having some positive assurance that multi-billion dollar calculations don't lose their pennies and that the books will still balance. So, the only downside is having an additional block of code to maintain. My vote is for adding it to the standard library. Control over precision is more of a core capability than it is an extension, but, as the PEP author and principal user, you probably know best. 'nuff said, Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From bac@OCF.Berkeley.EDU Fri Dec 6 07:12:44 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Thu, 5 Dec 2002 23:12:44 -0800 (PST) Subject: [Python-Dev] PEP 242 Numeric kinds In-Reply-To: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> Message-ID: [Paul F Dubois] > We had put off consideration of PEP 242 Numeric kinds until June 2002 > when a meeting of some interested parties was to occur but the meeting > didn't occur. I have a draft implementation and users of it but my > feeling is that although correct and useful the PEP is not useful enough > to put it in the standard library. Since it really only interests > scientific programmers I propose simply making it a separate deliverable > on the Numeric site and withdrawing the PEP. > I say add it. If there are already users out there then that demonstrates as least some form of a demand. Besides, if rationals can get into the library (that module was finally accepted to be added to the library, right?) then a module to help do consistent decimal math should at least be included at least until the Python core can move to C99 (if I am remembering a comment from Tim saying that C99 adds more explicit numeric types, e.g., exactly 2 byte float). I just want to be able to add up a bunch of money values and not have a few pennies disappear in the end; usually I am the one who has to eat the loss and I am an poor college graduate who can't afford the rounding. =) -Brett From fredrik@pythonware.com Fri Dec 6 08:20:32 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 09:20:32 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <20021205192616.B18494@ibook.distro.conectiva><005701c29ca9$d8517ba0$ced241d5@hagrid><20021205203137.C19843@ibook.distro.conectiva><09e901c29cb0$0da094c0$6d94fea9@newmexico><20021205212127.G19843@ibook.distro.conectiva><20021206001736.B21140@ibook.distro.conectiva> Message-ID: <006601c29d00$59936d30$0900a8c0@spiff> martin wrote: > Again, this is *not* the strategy that Just proposes, and, unless > somebody provides an implementation of it, not one that will matter. Depends on how you measure success, of course. Working code that addresses problems observed in the wild, or the length of the discussion thread on a mailing list. From Paul.Moore@atosorigin.com Fri Dec 6 09:38:50 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 09:38:50 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E67@UKDCX001.uk.int.atosorigin.com> From: Gordon McMillan [mailto:gmcm@hypernet.com] > BTW, iu does allow non string objects in sys.path. > If it exposes "getmod(name)", that will be tried > in preference to iu's own machinery. I just tried this out. As far as I can see, iu works perfectly with a simple getmod() method. Why does your patch require find_module() and load_module()? I'm not criticising, just trying to understand the requirements... (Anything to make writing an import hook easier!!!) You'll notice I'm wavering over my dislike of having objects on sys.path :-) My interest in import hooks stems from wanting to be able to implement something like Java's 'executable jar file' support, where an application, data and all, can be packaged up as a single file. In theory, there's no real problem with this (I can write import hooks to load from a zip file right now), but in practice, the bootstrapping problems are quite nasty. The more in-core (or at least standard library) support there is for these features, the less of a bootstrap problem I have... Paul. """ Example code for iu (adds an object to sys.path which satisfies *all* requests for modules and packages with a new empty module...) """ import iu import imp import sys iu.ImportManager().install() class Tester: def getmod(self, name): print "Requested", name, locals() m =3D imp.new_module(name) # Either set __path__ to contain an object, # or set __path__ to contain a path, and # __importsub__ to the object... m.__path__ =3D [ Tester() ] sys.modules[name] =3D m return m sys.path.append(Tester()) From fredrik@pythonware.com Fri Dec 6 09:54:18 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 10:54:18 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: <16E1010E4581B049ABC51D4975CEDB885E2E67@UKDCX001.uk.int.atosorigin.com> Message-ID: <018001c29d0d$72a50bf0$0900a8c0@spiff> Paul Moore wrote: >=20 > I just tried this out. As far as I can see, iu works perfectly > with a simple getmod() method. Why does your patch require > find_module() and load_module()? >=20 > I'm not criticising, just trying to understand the > requirements... (Anything to make writing an import hook > easier!!!) the internal import architecture does things in two steps, for some reason. this is also reflected in the public "imp" module interface: http://www.python.org/doc/2.2.2/lib/module-imp.html I'm not sure how Just's patch interacts with "imp", so maybe this is a non-issue. =20 From Paul.Moore@atosorigin.com Fri Dec 6 10:13:24 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 10:13:24 -0000 Subject: [Python-Dev] Another approach for the import mechanism Message-ID: <16E1010E4581B049ABC51D4975CEDB88619976@UKDCX001.uk.int.atosorigin.com> Samuele Pedroni wrote: > btw for single packages (once you can put zipfiles in sys.path > or __path__) you can achieve this with __path__ > > package/ > __init__.py > > __path__[0] =3D os.path.join(__path__[0],'package.zip') > > package.zip Of course, if __path__[0] was originally an object, this would break... I think that the objects-on-sys.path issue has now been resolved, but this probably acts as a reasonable "real world" example of what needs to be supported. (Unless I've misunderstood things again :-)) Paul. From just@letterror.com Fri Dec 6 10:13:10 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 6 Dec 2002 11:13:10 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <018001c29d0d$72a50bf0$0900a8c0@spiff> Message-ID: Fredrik Lundh wrote: > the internal import architecture does things in two steps, for > some reason. this is also reflected in the public "imp" module > interface: > > http://www.python.org/doc/2.2.2/lib/module-imp.html > > I'm not sure how Just's patch interacts with "imp", so maybe > this is a non-issue. The patch doesn't yet, but that's only to keep the patch small for initial review purposes. I have some thoughts about how it _could_ interact with imp, but that needs more thought. The whole find/load separation is still in tact, and is reflected in the zipimport.zipimporter object: it has a find_module() and a load_module() method. I'm not sure if the two-step approach is a big enough deal to keep it in a new import scheme. Just From fredrik@pythonware.com Fri Dec 6 10:40:50 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 11:40:50 +0100 Subject: [Python-Dev] New and Improved Import Hooks References: Message-ID: <025801c29d13$f30f97a0$0900a8c0@spiff> just wrote: > I'm not sure if the two-step approach is a big enough deal to > keep it in a new import scheme. does anyone remember/know why it was done in two steps? From martin@v.loewis.de Fri Dec 6 10:47:46 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 11:47:46 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619976@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619976@UKDCX001.uk.int.atosorigin.com> Message-ID: "Moore, Paul" writes: > > __path__[0] = os.path.join(__path__[0],'package.zip') > > > > package.zip > > Of course, if __path__[0] was originally an object, this would > break... What do you mean with "was originally", and what do you mean with "would break"? __path__[0] cannot be an object originally, because the package loader inserts a string, always. Even if it was a (non-string) object, I can't see how this would break. os.path.join would continue to work if the object is a string subclass, or atleast supports len, __getitem__, and __add__. The result would be a string, which would be converted to a zipimport on the next package contents import. > I think that the objects-on-sys.path issue has now been resolved, > but this probably acts as a reasonable "real world" example of what > needs to be supported. (Unless I've misunderstood things again :-)) I recommend you try the patch, to see how it works in the real world. Regards, Martin From Paul.Moore@atosorigin.com Fri Dec 6 10:56:02 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 10:56:02 -0000 Subject: [Python-Dev] Another approach for the import mechanism Message-ID: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com> From: Martin v. L=F6wis [mailto:martin@v.loewis.de] > What do you mean with "was originally", and what do you mean > with "would break"? [...] Looks like I did misunderstand :-( > I recommend you try the patch, to see how it works in the real > world. At the moment the patch crashes immediately on Windows. I've let Just know, and I'll try the next version when it arrives... Paul. PS If Just's patch *doesn't* stabilise in time for 2.3, and we go back to Jim Ahlstrom's patch, I'd support Just writing a PEP to cover his import hooks mechanism, for future inclusion. (Actually, I think that Just's mechanism has sufficient extra functionality over PEP 273, that it deserves a PEP even now. But if Just takes time out to write a PEP, I suspect that makes it even less likely we'll get working code in time...) From just@letterror.com Fri Dec 6 11:01:23 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 6 Dec 2002 12:01:23 +0100 Subject: [Python-Dev] zipimport & import hooks Message-ID: I think we can separate the zipimport issue from the import hooks issue, but only to an extent. *Some* hook need to be installed in import.c for zipimports to work, but it doesn't neccesarily mean we must add a general new import hook scheme *now*. But then again, it might be a *very* small step from enabling zipimports to exposing the needed hook and have a clean and general solution. It seems the majority doesn't like non-strings on sys.path, which to me is understandable, especially if you take PYTHONPATH into consideration. Let's try to focus on that. Here's a minimal implementation idea that will allow .zip filenames on sys.path: zipimporters = {} # a cache, private to import.c for p in sys.path: if p.endswith(".zip"): z = zipimporters.get(p) if z is None: z = zipimporters[p] = zipimporter(p) loader = z.find_module(name) ...etc... else: ...builtin import... This gets nessesarily more complex for packages, as now we also have to cater for __path__ items of this form: "/path/to/my/archive.zip/packagedir". In this case we need to create a "subimporter" of "/path/to/my/archive.zip" if only because we don't want to reread the zip archive's file index. By now the machinery needed can be written like so: zipimporters = {} # a cache, private to import.c def get_zipimporter(p): if p.endswith(".zip"): return zipimporter(p) if not os.path.exists(p): pos = p.rfind(".zip") if pos > 0: archive = p[:pos + 4] subpath = p[pos + 5:] # skip initial sep z = zipimporters[archive] return z.subimporter(subpath) for p in sys.path: importer = zipimporters.get(p, -1) if importer != -1: importer = zipimporters[p] = get_importer(p) if importer is not None: loader = importer.find_module(name) ...etc... else: ...builtin import... And *this* can easily be rewritten in a slightly more general form: path_importers = {} # exposed in sys import_hooks = [get_zimpimporter] # exposed in sys # get_zimpimporter() implementation left out for brevity def get_importer(p): for hook in import_hooks: importer = hook(path) if importer is not None: return None return None for p in sys.path: importer = path_importers.get(p, -1) if importer != -1: importer = path_importers[p] = get_importer(p) if importer is not None: loader = importer.find_module(name) ...etc... else: ...builtin import... Which, an observant reader might notice, is a stripped down version of Gordon's iu.py, with s/shadowpath/path_importers/ and s/ownertypes/import_hooks/ and without his metapath. (The metapath is a great idea, but I don't think it's possible to do without a *major* rewrite of import.c. The rest is relatively easy to do, in fact I've got it mostly working already.) Just From Paul.Moore@atosorigin.com Fri Dec 6 11:17:36 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 11:17:36 -0000 Subject: [Python-Dev] Another approach for the import mechanism Message-ID: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com> From: Martin v. L=F6wis [mailto:martin@v.loewis.de] > __path__[0] cannot be an object originally, because the > package loader inserts a string, always. Thinking some more, I'd dispute that "always". The *current* package loader inserts a string, yes. Under Just's patch, I think this remains true, as well. But for a general import hook, it certainly isn't enforced. I posted a snippet of code which uses Gordon's iu, which works perfectly but which inserts an object into a package's __path__. I'm not saying that this isn't bad practice, or even relevant to the case being discussed, but it's *certainly* true that this whole area is under-documented and not properly enforced. The only reference I could find to package structure in the standard documentation was "[XXX Can't be bothered to spell this out right now; see the URL http://www.python.org/doc/essays/packages.html for more details, also about how the module search works from inside a package.]". That URL points to a document which is described as "Built-in Package Support in Python 1.5" which doesn't inspire confidence in its accuracy for current Python... [Hey, I just noticed that it documents __all__, which wasn't in Python 1.5, I believe. So maybe it is up to date after all!] Actually, there's a piece of sample code in that document which suggests putting platform-specific code into the search path via dirname =3D __path__[0] # Package's main folder __path__.insert(0, os.path.join(dirname, "plat-" + platform)) That would result in things like ".../lib/package.zip/plat-win32" being on __path__. Will Just's code handle this? (I'd check, but the code doesn't run on Win32 yet. Looking at the code seems to indicate that it will not handle this...) FWIW, Jim's patch does. Paul. From martin@v.loewis.de Fri Dec 6 11:18:10 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 12:18:10 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com> Message-ID: "Moore, Paul" writes: > (Actually, I think that Just's mechanism has sufficient extra > functionality over PEP 273, that it deserves a PEP even now. > But if Just takes time out to write a PEP, I suspect that > makes it even less likely we'll get working code in time...) Indeed. It appears that, if Just completes the patch, we need BDFL pronouncement, anyway, to chose one of the following options: - Apply Jim's patch: Ugly, unreadable code, but provides strictly PEP 273 only, and preserves backwards compatibility. - Apply Just's patch: Nice code, probably provides PEP 273, might break compatibility, does not provide all features it claims to provide (you can't implement a .py.bz2 importer with it). - Wait for YAGNI (yet another great new infrastructure), to invoke all hooks on all sys.path items; this would give backwards compatibility (sys.path could continue to be a list of strings), extensibility (you could add new strings to sys.path, yet the hooks would honor them), feature-richness (you can implement a .py.bz2 importer), and slowness (for invoking every hook on every sys.path item). Regards, Martin From Paul.Moore@atosorigin.com Fri Dec 6 11:26:28 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 11:26:28 -0000 Subject: [Python-Dev] Another approach for the import mechanism Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E6B@UKDCX001.uk.int.atosorigin.com> From: Martin v. L=F6wis [mailto:martin@v.loewis.de] > "Moore, Paul" writes: >=20 > > (Actually, I think that Just's mechanism has sufficient extra > > functionality over PEP 273, that it deserves a PEP even now. > > But if Just takes time out to write a PEP, I suspect that > > makes it even less likely we'll get working code in time...) >=20 > Indeed. It appears that, if Just completes the patch, we need BDFL > pronouncement, anyway, to chose one of the following options: Agreed. BTW, has all the discussion around Just's proposal diverted anyone from trying Jim's patch? I'd like to see some more success reports from using it. I'd hate to have Guido pronounce in favour of Jim's patch only to discover problems in it... Paul. From just@letterror.com Fri Dec 6 11:34:06 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 6 Dec 2002 12:34:06 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > From: Martin v. L=F6wis [mailto:martin@v.loewis.de] > > __path__[0] cannot be an object originally, because the > > package loader inserts a string, always. >=20 > Thinking some more, I'd dispute that "always". The *current* > package loader inserts a string, yes. Under Just's patch, > I think this remains true, as well. The last patch I posted here inserts zipimporter instances in __path__, j= ust as it replaces *.zip sys.path entries with zipimporter instances. Allowing o= bjects on sys.path/pkg.__path__ is still the most lo-tech solution, but it seems there's too much opposition for that, so I'll continue to work on an iu.py-inspired solution. I hope to post a patch later today. (Since I don= 't have access to a Windows box I can't test on windows, so any help from your en= d (or anyone's) will be greatly appreciated.) Just From martin@v.loewis.de Fri Dec 6 11:44:11 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 12:44:11 +0100 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com> Message-ID: "Moore, Paul" writes: > That would result in things like ".../lib/package.zip/plat-win32" > being on __path__. Will Just's code handle this? Good question. I could not make this work, but I agree it is essential to package the standard library in a zip file. Just? Regards, Martin From pedronis@bluewin.ch Fri Dec 6 11:46:28 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 6 Dec 2002 12:46:28 +0100 Subject: [Python-Dev] Another approach for the import mechanism References: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com> Message-ID: <005901c29d1d$1d795ae0$6d94fea9@newmexico> From: "Martin v. L=F6wis" > "Moore, Paul" writes: > > > That would result in things like ".../lib/package.zip/plat-win32" > > being on __path__. Will Just's code handle this? > > Good question. I could not make this work, but I agree it is essential > to package the standard library in a zip file. Just? Btw Jython handle them. regards. From guido@python.org Fri Dec 6 12:36:05 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 07:36:05 -0500 Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS In-Reply-To: Your message of "Thu, 05 Dec 2002 19:41:18 -0200." <20021205194118.C18494@ibook.distro.conectiva> References: <15852.57674.16727.930806@montanaro.dyndns.org> <20021205194118.C18494@ibook.distro.conectiva> Message-ID: <200212061236.gB6Ca5Y15923@pcp02138704pcs.reston01.va.comcast.net> > > Feel free to only edit Misc/NEWS; before releases I'll go through it > > to find things I've missed, so there's no need to update whatsnew23.tex. > > I don't mind if people edit whatsnew23.tex to add items, but it's > > certainly not necessary. > > Thanks for mentioning that. I was really wondering about what to do. Someone should add this to the developer guidelines on python.org/dev/ --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Fri Dec 6 12:52:58 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 13:52:58 +0100 Subject: [Python-Dev] zipimport & import hooks References: Message-ID: <02ac01c29d26$681e00b0$0900a8c0@spiff> Just van Rossum wrote: > It seems the majority doesn't like non-strings on sys.path, which to = me is > understandable, especially if you take PYTHONPATH into consideration. = Let's try > to focus on that. Has anyone presented any hard data, or are people just guessing? As far as I can tell, I'm the only one here who has posted fragile code from a = real shipping product -- and I still believe it's a bogus argument. A quick look in our code repository indicates that most common sys.path operation is insert/append, followed by not using it at all, followed by = not using it at all, followed by not using it at all, followed by = calculating a list of dictionaries and *replacing* the current sys.path, followed by = looping over sys.path to scan for data or plugin modules (most such code is al- ready safe, none will break "silently" if a non-string object would = appear on sys.path out of the blue). Compared to the append changes (lots of work), moving from exception strings to instances (caused some of the trickiest bugs I've ever seen), and other non-compatible changes, the impact here is close to zero. I can dig out more detailed data if necessary (but not today). Anyone else? From fredrik@pythonware.com Fri Dec 6 13:32:59 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 14:32:59 +0100 Subject: [Python-Dev] zipimport & import hooks References: <02ac01c29d26$681e00b0$0900a8c0@spiff> Message-ID: <036701c29d2b$ff9d5210$0900a8c0@spiff> > using it at all, followed by not using it at all, followed by = calculating a list > of dictionaries and *replacing* the current sys.path dictionaries =3D> directories From jacobs@penguin.theopalgroup.com Fri Dec 6 13:32:21 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 08:32:21 -0500 (EST) Subject: [Python-Dev] zipimport & import hooks In-Reply-To: <02ac01c29d26$681e00b0$0900a8c0@spiff> Message-ID: On Fri, 6 Dec 2002, Fredrik Lundh wrote: > Just van Rossum wrote: > > It seems the majority doesn't like non-strings on sys.path, which to me is > > understandable, especially if you take PYTHONPATH into consideration. Let's try > > to focus on that. > > Has anyone presented any hard data, or are people just guessing? As far > as I can tell, I'm the only one here who has posted fragile code from a real > shipping product -- and I still believe it's a bogus argument. I've reviews several hundred thousand lines of code in our repository, and here is the frequency breakdown of 588 usages of sys.path: 84%: sys.path.insert(0, package_path) 10%: sys.path.append(package_path) 2%: sys.path.extend(path_list) 1%: str(package_path) in sys.path <1%: random debugging/unit test code that prints sys.path or compares sys.path with a previous copy of sys.path. The remaining ~1% use this fairly hairy idiom to implement a plug-in system: # in a package __init__.py # ... for filename in os.listdir(__path__[0]): if not filename.endswith('.py') or filename.startswith('__'): continue name = filename[:-3] plugin = __import__('%s.%s' % (base,name), None, None, [name]) # ...do something with plugin... Clearly, the above code can be redone in a much nicer way that doesn't depend on __path__, the qualified base package name, and __import__. Other than the above hard numbers, I have only some vague user-feelings to add to the discussion based on a poll of our in-house developers: 1) They all are very convinced that sys.path.append('/usr/path/stuff.zip') should work just like > mkdir /usr/path/stuff > cd /usr/path/stuff > unzip -d stuff.zip sys.path.append('/usr/path/stuff') Any other proposal that magically ignores the zip file name is right out, even if it makes it marginally more challenging to wrap the python standard library in a zip file. 2) They don't care if Python stores objects, strings, or bananas in sys.path, so long as sys.path=map(str,sys.path) results in a human-readable path and does not change how imports occur. 3) They all want ZIP archives to be a builtin feature, but don't care if they have to write an extra line of code to register a TAR/BZ2/whatever import hook. 4) They don't care if a small amount of code breaks. It is trivial to notice when it does break and easy to fix. Note that these comments are a result of a 15 minute discussion among a very mix bag of programmers. Some have been with Python for many years, others are in their first week of learning. Some come from primarily one of Java/C/C++/VB/Smalltalk/Scheme/Pascal backgrounds, others are extreme language polyglots. So, given the amount of discussion on python-dev, I was surprised how quickly the group converged upon the previous criteria with virtual unanimity. The only dissenting voice was our lone .NET fanatic who wouldn't shut up about how .NET assemblies were by far the superior way of doing things. We all have long since learned to ignore him. Hope this helps, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From sholden@holdenweb.com Fri Dec 6 13:27:59 2002 From: sholden@holdenweb.com (Steve Holden) Date: Fri, 6 Dec 2002 08:27:59 -0500 Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30 References: <20021205173053.A18494@ibook.distro.conectiva> <15855.64475.165789.875723@montanaro.dyndns.org> Message-ID: <005701c29d2b$4c7a1920$7101a8c0@holdenweb.com> > > Brett> So this brings up something I have been pondering; what to do > Brett> when a mistake in a summary is found after it is already out the > Brett> door. > > Fix what you can. (tm) > ...and can what you can't ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From sholden@holdenweb.com Fri Dec 6 13:31:07 2002 From: sholden@holdenweb.com (Steve Holden) Date: Fri, 6 Dec 2002 08:31:07 -0500 Subject: [Python-Dev] New and Improved Import Hooks References: <074b01c29bca$e5298dc0$0201010a@wiktor><016f01c29c99$18967c80$aa2ce8d9@mira><016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com> Message-ID: <006901c29d2b$bc51b0a0$7101a8c0@holdenweb.com> > "Steve Holden" writes: > > > > >Should be "importing_module_from_dlls". > > > > > > I still don't understand. Is that some feature of Python? > > > > > I think Wiktor is referring to extensions, which under Windows are often > > provided as .pyd DLLs. > > Yes, but then I understand how that is related to zipfile imports: A > .pyd provides only a single module. A .zip file must supply many > modules. > > > A quick look at my own code base, admittedly puny, reveals no statements > > that would break were non-strings to be added to sys.path. Is there some way > > to trap assignments to sys.path so a warning could be issued if sufficiently > > un-stringy elements were discovered in it? > > This isn't really possible, as this is just a dict > modification. You'ld also have to catch assignments to > sys.__dict__['path'] > Indeed. But if sys were an instance of some class, rather than a module, we could trap the assignments in its __setattr__ ... hmm, that's far too radical. Pity. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From martin@v.loewis.de Fri Dec 6 13:51:49 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 06 Dec 2002 14:51:49 +0100 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: References: Message-ID: Kevin Jacobs writes: > for filename in os.listdir(__path__[0]): > if not filename.endswith('.py') or filename.startswith('__'): > continue [...] > Clearly, the above code can be redone in a much nicer way that doesn't > depend on __path__, the qualified base package name, and __import__. That would break only if the package becomes part of a zipfile, right? And, if it is in a zipfile, it breaks no matter what mechanism we find for zipfile imports, since it relies on os.listdir to obtain the package contents, right? > 1) They all are very convinced that > > sys.path.append('/usr/path/stuff.zip') > > should work just like > > > mkdir /usr/path/stuff > > cd /usr/path/stuff > > unzip -d stuff.zip > > sys.path.append('/usr/path/stuff') I think both Jim's and Just's patch meet that requirement; Gustavo's plan doesn't. > 2) They don't care if Python stores objects, strings, or bananas in > sys.path, so long as > > sys.path=map(str,sys.path) > > results in a human-readable path and does not change how imports occur. For Just's patch, that is only true for zipfile imports. If you would also provide URL, or .tgz import with his patch, then you'll loose the magic when converting the path back to strings. > 3) They all want ZIP archives to be a builtin feature, but don't care if > they have to write an extra line of code to register a TAR/BZ2/whatever > import hook. Clearly, with Jim's patch, they'ld only get zipfile import, period. > Hope this helps, It sure does. Thanks, Martin From niemeyer@conectiva.com Fri Dec 6 13:51:38 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Fri, 6 Dec 2002 11:51:38 -0200 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> <20021205212604.H19843@ibook.distro.conectiva> <20021206005352.C21140@ibook.distro.conectiva> Message-ID: <20021206115138.B22531@ibook.distro.conectiva> > Yes, but I don't have to clutter my disk for that. /usr/lib/python2.2 already exists, doesn't it? That's probably one of the few __init__.zip we'd ever see, since complete packages would certainly be a majority. OTOH, with the current implementation, the standard library is probably the only compressed package we'll ever see. That's just my opinion, of course, and I hope to be wrong. > > - Don't have to change path to use compressed packages (at least > > not if you want to provide compressed packages, individual > > compressed modules or the standard library). > > I thought you just explained that I will need to change the path to > provide a compressed standard library, to point to a directory that > contains an __init__.zip. No, I didn't. The compressed library would be in /usr/lib/python2.2, which is already in the path. > > - Don't have to specify the compression type hardcoded. > > I don't understand that remark. If I have a zipfile, I surely must > install a zipfile hook in your approach also - a .tar.bz2 hook won't > be able to load the zipfile, no? What I mean is: sys.path = ["/usr/lib/python2.2/stdlib.zip"] vs. sys.path = ["/usr/lib/python2.2"] > > - Allows one to ship a package inside a zip file, without asking > > the user to change his path, and without hacking the package. > > > > - Allows one to compress a single file (foobar.py.bz2). > > This is really the same issue: If you had a mechanism to import a > module from a .py.bz2 file, you could use the same mechanism to import > a package (or subpackage) from a .zip file. While I think this might > be desirable, I also think it was never the goal of PEP 273 to provide > such a facility. My purpose differs from what is in PEP 273, for sure. Anyway, I'd just like to expose the idea. If everybody disagrees, we can safely forget it now. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From fredrik@pythonware.com Fri Dec 6 13:53:31 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 14:53:31 +0100 Subject: [Python-Dev] zipimport & import hooks References: Message-ID: <039e01c29d2e$ddddeb00$0900a8c0@spiff> Kevin wrote: > 2) They don't care if Python stores objects, strings, or bananas in > sys.path, so long as=20 >=20 > sys.path=3Dmap(str,sys.path)=20 >=20 > results in a human-readable path and does not change how imports = occur. quick question: would the following variation of feeling #2 be = acceptable: > 2) They don't care if Python stores objects, strings, or bananas in > sys.path, so long as=20 >=20 > sys.path=3Dmap(str,sys.path)=20 + + on a sys.path set up by the Python interpreter itself > results in a human-readable path and does not change how imports = occur. (note that today's Python already allows you to violate the above rule, = by adding non-string objects with cunningly constructed __str__ methods...) From jacobs@penguin.theopalgroup.com Fri Dec 6 13:59:41 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 08:59:41 -0500 (EST) Subject: [Python-Dev] zipimport & import hooks In-Reply-To: Message-ID: On 6 Dec 2002, Martin v. [iso-8859-15] L=F6wis wrote: > Kevin Jacobs writes: >=20 > > for filename in os.listdir(__path__[0]): > > if not filename.endswith('.py') or filename.startswith('__'): > > continue > [...] > > Clearly, the above code can be redone in a much nicer way that doesn'= t > > depend on __path__, the qualified base package name, and __import__. >=20 > That would break only if the package becomes part of a zipfile, right? > And, if it is in a zipfile, it breaks no matter what mechanism we find > for zipfile imports, since it relies on os.listdir to obtain the > package contents, right? Yup -- which is why I've just added an item to our task queue to avoid th= e filesystem entirely for this kind of code. > > 2) They don't care if Python stores objects, strings, or bananas in > > sys.path, so long as=20 > >=20 > > sys.path=3Dmap(str,sys.path)=20 > >=20 > > results in a human-readable path and does not change how imports= occur. >=20 > For Just's patch, that is only true for zipfile imports. If you would > also provide URL, or .tgz import with his patch, then you'll loose the > magic when converting the path back to strings. Hmmm. I haven't looked that closely at Just's proposal. Why should .tgz= or http://xyz be treated any differently than .zip? > > 3) They all want ZIP archives to be a builtin feature, but don't ca= re if > > they have to write an extra line of code to register a TAR/BZ2/w= hatever > > import hook. >=20 > Clearly, with Jim's patch, they'ld only get zipfile import, period. The sentiment seemed to be that zipfiles were an 80% solution and the res= t was icing on the cake. Of course, as their boss, I can overrule them and say that I also need an import mechanism that can also gracefully support cryptographically signed modules (native and in zipfiles) too. ;) -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From sholden@holdenweb.com Fri Dec 6 13:56:52 2002 From: sholden@holdenweb.com (Steve Holden) Date: Fri, 6 Dec 2002 08:56:52 -0500 Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS References: <15852.57674.16727.930806@montanaro.dyndns.org> <20021205194118.C18494@ibook.distro.conectiva> <200212061236.gB6Ca5Y15923@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <007d01c29d2f$55d52150$7101a8c0@holdenweb.com> [amk] > > > Feel free to only edit Misc/NEWS; before releases I'll go through it > > > to find things I've missed, so there's no need to update whatsnew23.tex. > > > I don't mind if people edit whatsnew23.tex to add items, but it's > > > certainly not necessary. > > [gustavo] > > Thanks for mentioning that. I was really wondering about what to do. [guido] > > Someone should add this to the developer guidelines on python.org/dev/ > I added a couple of paragraphs under "Recording Change" in http://www.python.org/dev/process.html I hope they'll serve. Better than nothing, anyhway. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From jacobs@penguin.theopalgroup.com Fri Dec 6 14:09:22 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 09:09:22 -0500 (EST) Subject: [Python-Dev] zipimport & import hooks In-Reply-To: <039e01c29d2e$ddddeb00$0900a8c0@spiff> Message-ID: On Fri, 6 Dec 2002, Fredrik Lundh wrote: > quick question: would the following variation of feeling #2 be acceptable: > > > 2) They don't care if Python stores objects, strings, or bananas in > > sys.path, so long as > > > > sys.path=map(str,sys.path) > > + > + on a sys.path set up by the Python interpreter itself > > > results in a human-readable path and does not change how imports occur. I suppose, though the motivation behind it is not transparant to me. There are many places in Python where a user can do fruity things, and the general response has been "DON'T DO THAT!". Is your qualification an attempt to close such a loophole or is there a realistic use-case that is affected? For discussion, here is another use-case that I care about (a little): Consider a PYTHONPATH includes a mix of filesystem paths, zip files, maybe some tgz or tar.bz2 files. We would expect that: ':'.join(map(str,sys.path)) == os.environ['PYTHONPATH'] modulo path normalization and any "extra" items that Python decided to toss into sys.path. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From bckfnn@worldonline.dk Fri Dec 6 14:22:07 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Fri, 06 Dec 2002 15:22:07 +0100 Subject: [Python-Dev] zipimport & import hooks References: Message-ID: <3DF0B28F.2060400@worldonline.dk> [Just van Rossum] > This gets nessesarily more complex for packages, as now we also have to cater > for __path__ items of this form: "/path/to/my/archive.zip/packagedir". In this > case we need to create a "subimporter" of "/path/to/my/archive.zip" if only > because we don't want to reread the zip archive's file index. and > The last patch I posted here inserts zipimporter instances in > __path__, just as it replaces *.zip sys.path entries with zipimporter > instances. Allowing objects on sys.path/pkg.__path__ is still the most > lo-tech solution, but it seems there's too much opposition for that, > so I'll continue to work on an iu.py-inspired solution. You are getting mighty close to what Jython has hardcoded for its .zip support . If the framework you come up with should be usefull from jython, then it is very important that it also somehow support cleanup. If objects are added to sys.path and __path__ then cleanup is easily handled in the __del__ method of the object. If the object is placed on some kind of shadowpath then some other cleanup means should be available. From jython I can also note that: - The automatic conversion of a '*.zip' string to a SyspathArchive (aka zipimporter) object works, but it works by magic and I'm not happy with this kind of identity change. It was just the best I could come up with that allowed easy cleanup as well as only-strings-on-sys.path. - Not all jython's potential importers can be represented by a string. Obviously such an importer can't be added on our PYTHONPATH, but must be added by program itself. An importer that is based around a java classloader is an obvious example of such an importer. So I would prefer that: - non-string objects was acceptable values on sys.path. If string-subclasses becomes a requirement, that is also fine. - if strings on sys.path are changed to something else, it should be done explicit and not implicit at the time of the first import that uses the sys.path entry. It would also be nice if user code could use the import framework to access resources, data files, gifs etc. That would match java's ClassLoader.getResourceAsStream() method. regards, finn From =?iso-8859-1?Q?Martin_v._L=F6wis?= Fri Dec 6 14:26:37 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Fri, 6 Dec 2002 15:26:37 +0100 Subject: [Python-Dev] zipimport & import hooks References: Message-ID: <000b01c29d33$7d692190$3216e8d9@mira> > Hmmm. I haven't looked that closely at Just's proposal. Why should .tgz or > http://xyz be treated any differently than .zip? There is a magic, in import.c, which, when an import statement is executed, looks for sys.path entries ending in .zip, and silently replaces them with zipimport objects. No such magic is available for any other form of sys.path entries. So while you can add a urlimport object to sys.path, you lose the hook when you convert the object to a string, as nobody will recreate the hook. Regards, Martin From guido@python.org Fri Dec 6 14:21:00 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 09:21:00 -0500 Subject: [Python-Dev] Import: where's the PEP? In-Reply-To: Your message of "Thu, 05 Dec 2002 23:02:58 EST." <20021206040258.GA24214@panix.com> References: <20021206040258.GA24214@panix.com> Message-ID: <200212061421.gB6EL0B16459@pcp02138704pcs.reston01.va.comcast.net> > Having just slogged my way through the entire import thread so far, I've > got to ask why nobody else has demanded a PEP before we go any further? > There are too many competing proposals; we need a good summary of what > design we're planning, plus a summary of why all other proposals are > rejected. > > I'm a very strong -1 on anything until there's a PEP. Well, there's PEP 273 which only discusses a feature (import from zip files) and an API (strings in sys.path that reference zip files). The PEP stands (I pronounce it accepted if it helps). But I agree that any API changes beyond that (either in C or in Python) need more consideration, and a PEP would be fine. BTW, it's curious to see the discussion follow the path of the sun around the earth. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From niemeyer@conectiva.com Fri Dec 6 14:27:01 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Fri, 6 Dec 2002 12:27:01 -0200 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: <02ac01c29d26$681e00b0$0900a8c0@spiff> References: <02ac01c29d26$681e00b0$0900a8c0@spiff> Message-ID: <20021206122700.A24104@ibook.distro.conectiva> > > It seems the majority doesn't like non-strings on sys.path, which to > > me is understandable, especially if you take PYTHONPATH into > > consideration. Let's try to focus on that. > > Has anyone presented any hard data, or are people just guessing? As far > as I can tell, I'm the only one here who has posted fragile code from a real > shipping product -- and I still believe it's a bogus argument. What Just said is that "majority doesn't like", not that there is breaking code. You can destroy the interpreter without breaking old code. I, personaly, *don't like* that, since it looks like an ugly hack to me. You may not care about this, and I don't expect to change the world with my feelings. Indeed, I've even tried to expose a different approach, as a way to avoid saying just "I don't like", but it looks like you don't care anyway, right? My intention was not to just create another flamewar in python-dev, I can guarant you. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From guido@python.org Fri Dec 6 14:32:43 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 09:32:43 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Fri, 06 Dec 2002 11:40:50 +0100." <025801c29d13$f30f97a0$0900a8c0@spiff> References: <025801c29d13$f30f97a0$0900a8c0@spiff> Message-ID: <200212061432.gB6EWh616558@pcp02138704pcs.reston01.va.comcast.net> > > I'm not sure if the two-step approach is a big enough deal to > > keep it in a new import scheme. > > does anyone remember/know why it was done in two steps? So you can check whether a module or package exists without importing it? For the relative import ambiguity? --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Fri Dec 6 14:32:17 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 09:32:17 -0500 (EST) Subject: [Python-Dev] zipimport & import hooks In-Reply-To: <000b01c29d33$7d692190$3216e8d9@mira> Message-ID: On Fri, 6 Dec 2002, Martin v. L=F6wis wrote: > > Hmmm. I haven't looked that closely at Just's proposal. Why should = .tgz or > > http://xyz be treated any differently than .zip? >=20 > There is a magic, in import.c, which, when an import statement is > executed, looks for sys.path entries ending in .zip, and silently > replaces them with zipimport objects. No such magic is available for an= y > other form of sys.path entries. So while you can add a urlimport object > to sys.path, you lose the hook when you convert the object to a string, > as nobody will recreate the hook. Don't like that -- a lazy and incremental replacement method seems much m= ore sensible, and preserves the semantics that non-python-dev-subscribers (an= d I) desire. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From fredrik@pythonware.com Fri Dec 6 14:42:25 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 15:42:25 +0100 Subject: [Python-Dev] zipimport & import hooks References: <000b01c29d33$7d692190$3216e8d9@mira> Message-ID: <047201c29d35$b308c880$0900a8c0@spiff> Martin wrote: > There is a magic, in import.c, which, when an import statement is > executed, looks for sys.path entries ending in .zip, and silently > replaces them with zipimport objects. No such magic is available for = any > other form of sys.path entries. note that there's nothing that prevent anyone from adding hooks for this in a later stage. (on the other hand, does anyone seriously believe that many people will use tar-based formats to store code packages? and everyone who think there's such a thing as a transparent network transport should check out http://www.bricklin.com/chiapaint.htm ;-) From just@letterror.com Fri Dec 6 14:55:25 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 6 Dec 2002 15:55:25 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <200212061432.gB6EWh616558@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > > > I'm not sure if the two-step approach is a big enough deal to > > > keep it in a new import scheme. > > > > does anyone remember/know why it was done in two steps? > > So you can check whether a module or package exists without importing > it? I have no idea how useful that is in practice. > For the relative import ambiguity? This is not true at find-module-time: if module x.y does "import z" two things are tried: - Find 'z' on x.__path__ We know up front that *if* that suceeds that the full name is "x.z", so you might as well attempt to *load* "x.y". if the above failed: - Find 'z' on sys.path, in which case the full name is "z". It doesn't matter much, though, in that as long as we don't want to rewrite import.c as a whole we're stuck with the two-step scheme anyway. Just From guido@python.org Fri Dec 6 14:56:15 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 09:56:15 -0500 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: Your message of "Fri, 06 Dec 2002 13:52:58 +0100." <02ac01c29d26$681e00b0$0900a8c0@spiff> References: <02ac01c29d26$681e00b0$0900a8c0@spiff> Message-ID: <200212061456.gB6EuGI16697@pcp02138704pcs.reston01.va.comcast.net> > Has anyone presented any hard data, or are people just guessing? I found two uses in the Zope 3 codebase that would break for non-strings. cwd = os.getcwd() for path in sys.path[1:]: # Yeah, we need to exclude the cwd itself if path != '' and cwd.startswith(path): name = os.path.join(cwd[len(path)+1:], name) name = os.path.normpath(name) break d['PYTHONPATH'] = os.pathsep.join(sys.path) def getPythonPath(self): 'See Zope.App.OFS.ApplicationControl.IRuntimeInfo.IRuntimeInfo' return tuple(sys.path) # The interface promises a tuple of strings --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Dec 6 15:04:01 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 10:04:01 -0500 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: Your message of "Fri, 06 Dec 2002 11:51:38 -0200." <20021206115138.B22531@ibook.distro.conectiva> References: <20021205192616.B18494@ibook.distro.conectiva> <20021205203312.A20315@ibook.distro.conectiva> <20021205212604.H19843@ibook.distro.conectiva> <20021206005352.C21140@ibook.distro.conectiva> <20021206115138.B22531@ibook.distro.conectiva> Message-ID: <200212061504.gB6F41D16796@pcp02138704pcs.reston01.va.comcast.net> > Anyway, I'd just like to expose the idea. If everybody disagrees, we > can safely forget it now. Amen, brother! --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Fri Dec 6 15:04:34 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 16:04:34 +0100 Subject: [Python-Dev] zipimport & import hooks References: <02ac01c29d26$681e00b0$0900a8c0@spiff> <200212061456.gB6EuGI16697@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <049601c29d38$ca818350$0900a8c0@spiff> Guido wrote: > > Has anyone presented any hard data, or are people just guessing? >=20 > I found two uses in the Zope 3 codebase that would break for > non-strings. >=20 > cwd =3D os.getcwd() > for path in sys.path[1:]: # Yeah, we need to exclude the = cwd itself > if path !=3D '' and cwd.startswith(path): > name =3D os.path.join(cwd[len(path)+1:], name) > name =3D os.path.normpath(name) > break >=20 > d['PYTHONPATH'] =3D os.pathsep.join(sys.path) >=20 > def getPythonPath(self): > 'See = Zope.App.OFS.ApplicationControl.IRuntimeInfo.IRuntimeInfo' > return tuple(sys.path) > # The interface promises a tuple of strings can Zope users modify the path? (is this "application code", or is this part of a package that can be used by applications other than the main Zope programs?) From fredrik@pythonware.com Fri Dec 6 15:09:03 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 16:09:03 +0100 Subject: [Python-Dev] zipimport & import hooks References: Message-ID: <049901c29d39$6aff05f0$0900a8c0@spiff> kevin wrote: > > quick question: would the following variation of feeling #2 be = acceptable: > >=20 > > > 2) They don't care if Python stores objects, strings, or bananas = in > > > sys.path, so long as=20 > > >=20 > > > sys.path=3Dmap(str,sys.path)=20 > >=20 > > + > > + on a sys.path set up by the Python interpreter itself > >=20 > > > results in a human-readable path and does not change how = imports occur. >=20 > I suppose, though the motivation behind it is not transparant to me. under the proposed scheme, Python will never mess things up for you by itself. by default, and as long as you only use the standard = mechanisms, all items in the path behaves like strings, and point to things = (directories or archives) in the filesystem. > There are many places in Python where a user can do fruity things, and > the general response has been "DON'T DO THAT!". Is your qualification > an attempt to close such a loophole or is there a realistic use-case = that > is affected? but what if I *want* to do fruity things? that's the whole point here: if the framework delegates imports to objects in the sys.path list, it's dead simple to experiment. for some reason, all counter-proposals seem to be obsessed with control ("don't do that" in lower or upper case, "don't write code that I may = find ugly", "we'll break your programs in the future if you do that", "if you cannot think of a string representation, you're not qualified to write custom importers", "all your ZIP archives must use the same name", etc). why are we spending this much time and effort on making things hard to use? if I know that I can access all modules needed by my program through a five-line custom importer, why do I have to invent a string naming scheme, make sure that scheme is unique so no future version of Python may mistake it for a builtin importer, register the importer using some function that I can remember the name and signature of, and pray I've understood the semantics. isn't it much easier, and much more obvious for anyone who's survived Python 101, if I can just write sys.path =3D [myimporter()] and leave early that day. > For discussion, here is another use-case that I care about (a little): >=20 > Consider a PYTHONPATH includes a mix of filesystem paths, zip files, = maybe > some tgz or tar.bz2 files. >=20 > We would expect that: >=20 > ':'.join(map(str,sys.path)) =3D=3D os.environ['PYTHONPATH'] >=20 > modulo path normalization and any "extra" items that Python decided = to > toss into sys.path. the proposed scheme supports this for zip files. you probably don't want to use tgz or tar.bz2 (no random access), but if = you insist, you just need to add 2-3 lines to your startup code to activate = your custom importer (Python only reads PYTHONPATH on startup anyway). From guido@python.org Fri Dec 6 15:12:08 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 10:12:08 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Fri, 06 Dec 2002 15:55:25 +0100." References: Message-ID: <200212061512.gB6FC8g16874@pcp02138704pcs.reston01.va.comcast.net> > > > > I'm not sure if the two-step approach is a big enough deal to > > > > keep it in a new import scheme. > > > > > > does anyone remember/know why it was done in two steps? > > > > So you can check whether a module or package exists without importing > > it? > > I have no idea how useful that is in practice. Me neither. > > For the relative import ambiguity? > > This is not true at find-module-time: if module x.y does "import z" > two things are tried: > > - Find 'z' on x.__path__ > We know up front that *if* that suceeds that the full name is "x.z", > so you might as well attempt to *load* "x.y". > > if the above failed: > > - Find 'z' on sys.path, in which case the full name is "z". > > It doesn't matter much, though, in that as long as we don't want to > rewrite import.c as a whole we're stuck with the two-step scheme > anyway. The problem with trying to load x.y right away is that then you can't distinguish between "x.y doesn't exist" (and then you go to step 2) as opposed to "x.y exists, but it tries to import something else that raised ImportError" (and then you should simply propagate the exception). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Dec 6 15:22:13 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 10:22:13 -0500 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: Your message of "Fri, 06 Dec 2002 16:04:34 +0100." <049601c29d38$ca818350$0900a8c0@spiff> References: <02ac01c29d26$681e00b0$0900a8c0@spiff> <200212061456.gB6EuGI16697@pcp02138704pcs.reston01.va.comcast.net> <049601c29d38$ca818350$0900a8c0@spiff> Message-ID: <200212061522.gB6FMDp16946@pcp02138704pcs.reston01.va.comcast.net> > > def getPythonPath(self): > > 'See Zope.App.OFS.ApplicationControl.IRuntimeInfo.IRuntimeInfo' > > return tuple(sys.path) > > # The interface promises a tuple of strings > > can Zope users modify the path? (is this "application code", or is this > part of a package that can be used by applications other than the main > Zope programs?) I dunno. The interface is not that well documented. :-( I just show that the assumption of strings (or str-like behavior) is thoroughly established. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Dec 6 15:23:51 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 10:23:51 -0500 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: Your message of "Fri, 06 Dec 2002 16:09:03 +0100." <049901c29d39$6aff05f0$0900a8c0@spiff> References: <049901c29d39$6aff05f0$0900a8c0@spiff> Message-ID: <200212061523.gB6FNp216966@pcp02138704pcs.reston01.va.comcast.net> > but what if I *want* to do fruity things? that's the whole point here: > if the framework delegates imports to objects in the sys.path list, it's > dead simple to experiment. That's a good point. The question then is, if you want to use some 3rd party code that assumes strings, and it breaks as a result, what is broken? Your code or the 3rd party code? --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Fri Dec 6 15:31:26 2002 From: aahz@pythoncraft.com (Aahz) Date: Fri, 6 Dec 2002 10:31:26 -0500 Subject: [Python-Dev] Import: where's the PEP? In-Reply-To: <200212061421.gB6EL0B16459@pcp02138704pcs.reston01.va.comcast.net> References: <20021206040258.GA24214@panix.com> <200212061421.gB6EL0B16459@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021206153125.GA17821@panix.com> On Fri, Dec 06, 2002, Guido van Rossum wrote: >Aahz: >> >> Having just slogged my way through the entire import thread so far, I've >> got to ask why nobody else has demanded a PEP before we go any further? >> There are too many competing proposals; we need a good summary of what >> design we're planning, plus a summary of why all other proposals are >> rejected. >> >> I'm a very strong -1 on anything until there's a PEP. > > Well, there's PEP 273 which only discusses a feature (import from zip > files) and an API (strings in sys.path that reference zip files). > The PEP stands (I pronounce it accepted if it helps). > > But I agree that any API changes beyond that (either in C or in > Python) need more consideration, and a PEP would be fine. Yeah, I should have been clearer that I meant "any changes beyond what's in PEP 273"; I did read PEP 273 before posting to see whether there was coverage of the thread. I think this really needs a new PEP rather than updating PEP 273, because (as you say) PEP 273 covers its domain clearly. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From Paul.Moore@atosorigin.com Fri Dec 6 15:33:35 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 15:33:35 -0000 Subject: [Python-Dev] zipimport & import hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB8861997C@UKDCX001.uk.int.atosorigin.com> Fredrik Lundh wrote: > but what if I *want* to do fruity things? that's the whole > point here: if the framework delegates imports to objects in > the sys.path list, it's dead simple to experiment. > > for some reason, all counter-proposals seem to be obsessed > with control ("don't do that" in lower or upper case, "don't > write code that I may =3D find ugly", "we'll break your programs > in the future if you do that", "if you cannot think of a > string representation, you're not qualified to write custom > importers", "all your ZIP archives must use the same name", > etc). > > why are we spending this much time and effort on making things > hard to use? > > if I know that I can access all modules needed by my program > through a five-line custom importer, why do I have to invent > a string naming scheme, make sure that scheme is unique so > no future version of Python may mistake it for a builtin > importer, register the importer using some function that I can > remember the name and signature of, and pray I've understood > the semantics. Having listened to the arguments, and tried a few things out myself with iu.py, I now agree with this. Importer objects on sys.path are *very* easy to use, and nice and clean. I have no real code myself which will break - and I concede that if others do, they can speak for themselves. In fact, I've gone to the other extreme. I think I'd rather see general objects on sys.path than string subclasses. Basically, the breakage happens sooner if someone *does* assume that sys.path items are strings. I'll drop out of any more discussion on this issue now... Paul. From Paul.Moore@atosorigin.com Fri Dec 6 15:35:45 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 6 Dec 2002 15:35:45 -0000 Subject: [Python-Dev] New and Improved Import Hooks Message-ID: <16E1010E4581B049ABC51D4975CEDB8861997B@UKDCX001.uk.int.atosorigin.com> > It doesn't matter much, though, in that as long as we > don't want to rewrite import.c as a whole we're stuck > with the two-step scheme anyway. I don't wish to seem difficult here, but didn't this start from the basis that import.c was a mess, Jim Ahlstrom's patch was going to make it worse, and we should clean it up rather than making it worse? Don't get me wrong, I'd like to see a better import hook mechanism. And I'd like to see zipfile imports. And I'd like to see the import code cleaned up. But iu.py does all I need from import hooks. Jim Ahlstrom's patch does zip imports. If Just's code doesn't clean up the import code, I quite honestly don't understand the point. I think this reinforces the need for a PEP. It would clearly describe the benefits of the patch - both in overall terms, and in comparison to the (more limited) scope of PEP 273. Paul. From guido@python.org Fri Dec 6 15:46:07 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 06 Dec 2002 10:46:07 -0500 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: Your message of "Fri, 06 Dec 2002 15:33:35 GMT." <16E1010E4581B049ABC51D4975CEDB8861997C@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB8861997C@UKDCX001.uk.int.atosorigin.com> Message-ID: <200212061546.gB6Fk7J17128@pcp02138704pcs.reston01.va.comcast.net> > Having listened to the arguments, and tried a few things out > myself with iu.py, I now agree with this. Importer objects on > sys.path are *very* easy to use, and nice and clean. I have no > real code myself which will break - and I concede that if others > do, they can speak for themselves. > > In fact, I've gone to the other extreme. I think I'd rather see > general objects on sys.path than string subclasses. Basically, > the breakage happens sooner if someone *does* assume that sys.path > items are strings. This is a convincing argument. Maybe we should just start warning that sys.path may contain objects in the future. The str subclass hack still smells ugly to me. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Fri Dec 6 15:54:42 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 06 Dec 2002 16:54:42 +0100 Subject: [Python-Dev] zipimport & import hooks References: <02ac01c29d26$681e00b0$0900a8c0@spiff> Message-ID: <3DF0C842.1090408@lemburg.com> Fredrik Lundh wrote: > Just van Rossum wrote: > > >>It seems the majority doesn't like non-strings on sys.path, which to me is >>understandable, especially if you take PYTHONPATH into consideration. Let's try >>to focus on that. > > Has anyone presented any hard data, or are people just guessing? As far > as I can tell, I'm the only one here who has posted fragile code from a real > shipping product -- and I still believe it's a bogus argument. Have a look at distutils/command/install.py for an example: sys_path = map(os.path.normpath, sys.path) sys_path = map(os.path.normcase, sys_path) install_lib = os.path.normcase(os.path.normpath(self.install_lib)) if (self.warn_dir and not (self.path_file and self.install_path_file) and install_lib not in sys_path): self.warn(("modules installed to '%s', which is not in " + "Python's module search path (sys.path) -- " + "you'll have to change the search path yourself") % self.install_lib) Most code we use is of the form: sys.path.append('dir') sys.path.insert(0, 'dir') sys.path = ['/usr/lib/python1.5/site-packages'] + sys.path How I wish that we had some kind of code base archive for third party code... e.g. the compile farm would probably be great for this. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From fredrik@pythonware.com Fri Dec 6 16:18:07 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 6 Dec 2002 17:18:07 +0100 Subject: [Python-Dev] zipimport & import hooks References: <049901c29d39$6aff05f0$0900a8c0@spiff> <200212061523.gB6FNp216966@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <051501c29d43$1158b0f0$0900a8c0@spiff> Guido wrote: > > but what if I *want* to do fruity things? that's the whole point = here: > > if the framework delegates imports to objects in the sys.path list, = it's > > dead simple to experiment. >=20 > That's a good point. The question then is, if you want to use some > 3rd party code that assumes strings, and it breaks as a result, what > is broken? Your code or the 3rd party code? PIL used to assume strings, and there are a few bug reports on this topic in the PIL issue database. they're all very polite, so I think the answer might be "I did something fruity, so it's probably my fault, and I'm prepared to work around the problem, but it would be nice if the 3rd party library was fixed too". or in other words, we should warn application developers that explicitly adding non-string objects to the path may cause existing code (including 3rd party libraries) to break, and encourage 3rd party developers to = write robust code. From mclay@nist.gov Fri Dec 6 16:38:29 2002 From: mclay@nist.gov (Michael McLay) Date: Fri, 6 Dec 2002 11:38:29 -0500 Subject: [Python-Dev] PEP 242 Numeric kinds In-Reply-To: References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> Message-ID: <200212061138.30065.mclay@nist.gov> On Friday 06 December 2002 02:12 am, Brett Cannon wrote: > [Paul F Dubois] > > > We had put off consideration of PEP 242 Numeric kinds until June 2002 > > when a meeting of some interested parties was to occur but the meeting > > didn't occur. [...] > I just want to be able to add up a bunch of money values and not have a > few pennies disappear in the end; usually I am the one who has to eat the > loss and I am an poor college graduate who can't afford the rounding. =) Does the kinds implementation work with binary floats and decimal numbers? >From a quick scan of the PEP it looks like it is only binary floats. If it only applies to binary numbers then the kinds capability will not eliminate decimal number rounding errors. That problem will require extending Python to include a decimal number type. The abstract mentions a clarification of decimal literals, but I suspect Paul was not suggesting that the numbers defined by kinds will use decimal arithmetic instead of binary arithmetic. From bbum@codefab.com Fri Dec 6 16:04:31 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Fri, 6 Dec 2002 11:04:31 -0500 Subject: [Python-Dev] Re: Patched Transport In-Reply-To: <011501c29d08$eab820f0$0900a8c0@spiff> Message-ID: <67C7F84A-0934-11D7-B650-000393877AE4@codefab.com> On Friday, December 6, 2002, at 04:21 AM, Fredrik Lundh wrote: > hi bill, > > >> As I move to a Transport subclass that uses urllib2, I first created a >> Transport subclass whose request() method is patched to delegate all >> interaction with the connection to methods on the Transport class. > > looks fine. > > just one nit: changing the make_connection signature may break > existing programs. how about adding yet another hook: > > h = self.make_connection(host) > if verbose: > self.set_debuglevel(h, verbose) > > ... > > def set_debuglevel(self, connection, verbosity): > ... > > cheers /F I made the change as recommended -- I was being lazy... XMLRPC proxying now works and, along with it, so does ProxyHandler ordering. The fix I have does not modify urllib2 -- for my current needs, I'm trying to stick with stock python as much as possible. As such, I created a _fixUpHandlers() that will fix the handler order of any Opener instance -- likely, it should be eliminated and a similar fix integrated into add_handler() on Opener directly. I will test to make sure it works with a mix of HTTPS and with proxy servers that require authentication [I'm currently testing without a network connection on my laptop while on the train -- thankfully, OS X's apache configuration includes mod_proxy in the apache configuration... just have to turn it on]. Caveats are noted in the code. In particular, using urllib2 really wants the full URL so that it can do protocol resolution internally. I also ended up with a dummy class (_TransportConnection) that can cache-and-carry the hunks of information passed to the various send_* and get_* methods as the granularity of those methods simply does not match the calling semantics of the urllib2 library. To initialize a ServerProxy with HTTPTransport, I use the following function. Note that the arguments passed in are identifical to the initializer for ServerProxy. proxyUrl, proxyUser, proxyPass are currently stored as module variables -- this will change in the very near future, but is largely irrelevant in the context of xmlrpclib / urllib2. Of relevancy, I believe that proxyUrl could be of the form http://:@host:port/path/ and it would be no different than creating a HTTPTransport by passing in the user/pass seperately. def serverProxyForUrl(uri, transport=None, encoding=None, verbose=0): if not transport: transport = HTTPTransport.HTTPTransport(uri, proxyUrl, proxyUser, proxyPass) return xmlrpclib.ServerProxy(uri, transport, encoding, verbose) (4 space indent, not tabs-- I believe this is the recommended standard for the python library?) In operation, HTTPTransport could completely replace the Transport and SafeTransport classes found within xmlrpclib. However, the one incompatible change is in the way the transport class is instantiated. The __init__ method requires a single argument; the full URI of the XML-RPC server. ServerProxy's __init__() could easily be modified to address this issue. Next, I'll probably and compressed content support -- in my app, I can reduce the bytecount sent between client and server by upwards of 90% by simply gzip'ing the XML prior to dropping it on the wire. I haven't even remotely looked at how to do this. The actually compression/decompression is trivial, but there are a lot of possible spots to stick the compression/decompression mechanism. b.bum --- """ HTTPTransport provides an urllib2 based communications channel for use in xmlrpclib. Created by Bill Bumgarner . Using HTTPTransport allows the XML-RPC library to take full advantage of the features perpetuated by urllib2, including HTTP proxy support and a number of different authentication schemes. If urllib2 does not provide a handler capable of meeting the developer's need, the developer can create a custom Handler without requiring any changes to the XML-RPC code. Example usage: def serverProxyForUrl(uri, transport=None, encoding=None, verbose=0): if not transport: transport = HTTPTransport.HTTPTransport(uri, proxyUrl, proxyUser, proxyPass) return xmlrpclib.ServerProxy(uri, transport, encoding, verbose) """ import xmlrpclib from xmlrpclib import ProtocolError from urllib import splittype, splithost import urllib2 import sys class _TransportConnection: pass def _fixHandlerArrayOrder(handlers): insertionPoint = 0 for handlerIndex in range(0, len(handlers)): aHandler = handlers[handlerIndex] if isinstance(aHandler, urllib2.ProxyHandler): del handlers[handlerIndex] handlers.insert(insertionPoint, aHandler) insertionPoint = insertionPoint + 1 def _fixUpHandlers(anOpener): ### Moves proxy handlers to the front of the handlers in anOpener # # This function preserves the order of multiple proxyhandlers, if present. # This appears to be wasted effort in that build_opener() chokes if there # is more than one instance of any given handler class in the arglist. _fixHandlerArrayOrder(anOpener.handlers) map(lambda x: _fixHandlerArrayOrder(x), anOpener.handle_open.values()) class HTTPTransport(xmlrpclib.Transport): """Handles an HTTP transaction to an XML-RPC server using urllib2 [eventually].""" def __init__(self, uri, proxyUrl=None, proxyUser=None, proxyPass=None): ### this is kind of nasty. We need the full URI for the host/handler we are connecting to # to properly use urllib2 to make the request. This does not mesh completely cleanly # with xmlrpclib's initialization of ServerProxy. self.uri = uri self.proxyUrl = proxyUrl self.proxyUser = proxyUser self.proxyPass = proxyPass def request(self, host, handler, request_body, verbose=0): # issue XML-RPC request h = self.make_connection(host) self.set_verbosity(h, verbose) self.send_request(h, handler, request_body) self.send_host(h, host) self.send_user_agent(h) self.send_content(h, request_body) errcode, errmsg, headers = self.get_reply(h) if errcode != 200: raise ProtocolError( host + handler, errcode, errmsg, headers ) self.verbose = verbose return self.parse_response(self.get_file(h)) def make_connection(self, host, verbose=0): return _TransportConnection() def set_verbosity(self, connection, verbose): connection.verbose = verbose def send_request(self, connection, handler, request_body): connection.request = urllib2.Request(self.uri, request_body) def send_host(self, connection, host): connection.request.add_header("Host", host) def send_user_agent(self, connection): # There is no way to override the 'user-agent' sent by the UrlOpener. # This will cause a second User-agent header to be sent. # This is both different from the urllib2 documentation of add_header() # and would seem to be a bug. # # connection.request.add_header("User-agent", self.user_agent) pass def send_content(self, connection, request_body): connection.request.add_header("Content-Type", "text/xml") def get_reply(self, connection): proxyHandler = None if self.proxyUrl: if self.proxyUser: type, rest = splittype(self.proxyUrl) host, rest = splithost(rest) if self.proxyPass: user = "%s:%s" % (self.proxyUser, self.proxyPass) else: user = self.proxyUser uri = "%s://%s@%s%s" % (type, user, host, rest) else: uri = self.proxyUrl proxies = {'http':uri, 'https':uri} proxyHandler = urllib2.ProxyHandler(proxies) opener = urllib2.build_opener(proxyHandler) _fixUpHandlers(opener) try: connection.response = opener.open(connection.request) except urllib2.HTTPError, c: return c.code, c.msg, c.headers return 200, "OK", connection.response.headers def get_file(self, connection): return connection.response From just@letterror.com Fri Dec 6 15:48:17 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 6 Dec 2002 16:48:17 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861997B@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > But iu.py does all I need from import hooks. Jim Ahlstrom's > patch does zip imports. If Just's code doesn't clean up the > import code, I quite honestly don't understand the point. I won't clean up import.c, but I don't want to make it any worse, which is IMO what the current zip import patch does. I *might* open doors for future cleansing, but that's all. My main motivation are still: - the mechanics of reading zip files should _not_ be in import.c - ditto for a directory caching scheme - to try to minimally patch import.c so that zipimport can relatively cleanly be plugged in (if this leads to a new & approved import scheme, that's merely a nice side effect) Just From gmcm@hypernet.com Fri Dec 6 17:11:32 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 6 Dec 2002 12:11:32 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <025801c29d13$f30f97a0$0900a8c0@spiff> Message-ID: <3DF093F4.13309.6D603860@localhost> On 6 Dec 2002 at 11:40, Fredrik Lundh wrote: [find_module, load_module] > does anyone remember/know why it was done in two > steps? When Greg Stein was arguing for a one-step get_code, I believe Guido's response to the same question was "it seemed logical at the time". -- Gordon http://www.mcmillan-inc.com/ From gmcm@hypernet.com Fri Dec 6 17:11:32 2002 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 6 Dec 2002 12:11:32 -0500 Subject: [Python-Dev] Another approach for the import mechanism In-Reply-To: References: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com> Message-ID: <3DF093F4.4309.6D6038A6@localhost> On 6 Dec 2002 at 12:18, Martin v. L=F6wis wrote: > - Wait for YAGNI (yet another great new > infrastructure), to invoke all hooks on all sys.path > items; this would give backwards compatibility > (sys.path could continue to be a list of strings), > extensibility (you could add new strings to > sys.path, yet the hooks would honor them), > feature-richness (you can implement a .py.bz2 > importer), and slowness (for invoking every hook on > every sys.path item). A .py.bz2 importer is an example of what Greg Stein called a "suffix importer". That's orthogonal to "path importer"s. I omitted "suffix importer"s from iu, precisely because of the slowness. If you limit yourself to hooking nodes on sys.path (or on __path__), a pure Python __import__ can beat the builtin import (if you use efficient archives). -- Gordon http://www.mcmillan-inc.com/ From Chris.Barker@noaa.gov Fri Dec 6 17:31:50 2002 From: Chris.Barker@noaa.gov (Chris Barker) Date: Fri, 06 Dec 2002 09:31:50 -0800 Subject: [Python-Dev] Re: [Numpy-discussion] PEP 242 Numeric kinds References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> <20021206023728.GA24182@localhost.localdomain> Message-ID: <3DF0DF06.835F72A3@noaa.gov> José Fonseca wrote: > to say that the manipulation of numeric > arrays is only of interest to scientific programmers is the same of when > in the early computing days engineers saing that computers would only be > good for crunching numbers, and that the concept of personal computers > was just non-sense... I absolutely concur. > For a non-scientic usage of Numeric see the examples in > http://www.pygame.org/pcr/repository.php, but I can imagine the > usefullness of Numeric in many more non-scientific applications: > imaging, sound visualization plugins, 3D graphics, and probably much > more. It goes MUCH farther than this. All these examples are what I would call serious number crunching. Perhaps not strictly scientific, but certainly the realm of numeric programming, and all places where the "kinds" concept would be useful. However, there is a great deal of usefulness in Numeric for applications that are not doing a lot of number crunching. Performance is only one reason to use Numeric. The other reason is much cleaner and easier syntax for manipulating arrays of numbers, especially ones of more than one dimension. Being able to slice across multiple dimensions, array oriented arithmetic, overloaded comparisons, etc. I use Numeric for virtually every program I write, whether performance is an issue or not. Not having to write all those ugly, confusing and error prone loops is a godsend. All that being said, the "kinds" concept is probably mostly of concern to the Number crunching community. However, it is also sort of a low level concept, and having it part of core language makes sense, just like it makes sense to have complex number and long integers in the core language. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov From paul@pfdubois.com Fri Dec 6 18:44:46 2002 From: paul@pfdubois.com (paul@pfdubois.com) Date: Fri, 6 Dec 2002 11:44:46 -0700 Subject: [Python-Dev] PEP 242 Numeric kinds In-Reply-To: <200212061138.30065.mclay@nist.gov> Message-ID: <3DED0AB000003A8F@mta8.wss.scd.yahoo.com> PEP-242 does not have anything to do with: a. Whether or not some future issue of Numeric goes into the core b. Decimal arithmetic c. Rational arithmetic THe current implementation lives in the optional Packages area of the Num= eric distribution purely because it was convenient for me to put it there, and= the most likely customers use Numeric anyway. The only tie to Numeric is the ability of a kind to tell you the "type le= tter" that modules array and Numeric accept as types. Background: In Numeric there is an issue that if a Numeric.array a has ty= pe 'f', corresponding to C 'float' rather than C 'double', it is inconvenien= t to maintain that type at 'f' during operations with scalars. It used to be impossible, now it is inconvenient, and with Numarray it won't be a probl= em. Historically then, a lot of us were interested in putting a floating poin= t type into Python that corresponded to C 'float'; otherwise something like= a[i] * b was leading to a double result because a[i], a Python scalar, is= double. If we had an 'f' scalar, we could make a[i] return that instead of a 'd' scalar. Yes, there would be no literal equivalent (unless we got= into the usual fistfight over the literals) but that didn't matter so muc= h. With Numarray there is no need for this so the interest in making an 'f' scalar has declined. Still, kinds has its uses anyway. It appears that maybe I'm being too shy= about its virtues. Certainly many of the standard library modules are equ= ally obscure and you could view kinds as a wrapping of a standard C header fil= e and therefore of sufficient interest. So I will restate my position as, I'd like to push 242 forward or backwar= d. It shouldn't sit there in limbo. Advice from the PEPczar about process wo= uld be welcomed by me. From tim@zope.com Fri Dec 6 21:47:09 2002 From: tim@zope.com (Tim Peters) Date: Fri, 6 Dec 2002 16:47:09 -0500 Subject: [Python-Dev] We got leaks! Message-ID: If someone is up for a good weekend puzzle, try this on for size. I'd like to multitask this, if possible. I'm attacking the datetime module in the CVS sandbox, aiming to get it into good shape for 2.3a1. There's a pure Python implementation and a pure C implementation (incomplete but already quite hefty). You build the latter like so, for release mode: setup.py build_ext -i or for debug mode: setup.py build_ext -i --debug test_both.py is a test suite. To test the Python implementation, test_both.py and to test the C implementation, test_both.py c The tests should all pass. Running this in a loop, I didn't notice any memory growth after a minute of CPU time. To be sure, I replaced test_main like so: def test_main(): import gc r = unittest.TextTestRunner(stream=sys.stdout, verbosity=2) s = test_suite() while True: r.run(s) gc.collect() print gc.garbage # this is always an empty list print '*' * 10, 'total refs:', sys.gettotalrefcount() and ran a debug build, watching the printout of total outstanding refs. Alas, the C implementation shows a very slow leak this way: ********** total refs: 29633 ********** total refs: 29643 ********** total refs: 29653 ********** total refs: 29663 ********** total refs: 29673 So the C implementation is leaking 10 references per test cycle. Fine, so I screwed up the C code. However, the Python implementation leaks close to 4x faster, 37 references per test cycle. Jeremy suggests I look for the missing decref in datetime.py : ********** total refs: 33921 ********** total refs: 33958 ********** total refs: 33995 ********** total refs: 34032 ********** total refs: 34069 Likely suspects are "newer stuff": new-style classes and class methods. Possibly unittest too, but hard to imagine how. Even if you can't make time to dig into this, it would help a little to know what the C and Python leak rates are on a different platform. Heck, it would help to know whether the C code even compiles and the tests pass on a different platform (I'm running Win2K + current CVS Python + MSVC 6 here). From jacobs@penguin.theopalgroup.com Fri Dec 6 22:50:41 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 17:50:41 -0500 (EST) Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: On Fri, 6 Dec 2002, Tim Peters wrote: > If someone is up for a good weekend puzzle, try this on for size. I'd like > to multitask this, if possible. I've found the same leak under Linux+GCC 3.2 (Redhat 8) with CVS HEAD Python, compiled with debugging: Testing the Python implementation. ********** total refs: 35554 ********** total refs: 35591 ********** total refs: 35628 ********** total refs: 35665 ********** total refs: 35702 (leaking 37 refs per run) Testing the C implementation. ********** total refs: 31262 ********** total refs: 31272 ********** total refs: 31282 ********** total refs: 31292 (leaking 10 refs per run) HOWEVER, the leak disapears when Python is compiled normally, without debugging. I've also added some instrumentation to the interpreter and have determined that the leak is not occuring in a garbage collected object (anything in gc.get_objects()) that exists after the test-suite has been run. > Even if you can't make time to dig into this, it would help a little to know > what the C and Python leak rates are on a different platform. Yes -- at least on Linux+GCC 3.2 (Redhat 8.0). > Heck, it would help to know whether the C code even compiles and the tests > pass on a different platform (I'm running Win2K + current CVS Python + > MSVC 6 here). Yes on that too. I'm going to start a binary search on the test suite to see if I can isolate a minimal sequence that shows the leak. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jeremy@zope.com (Jeremy Hylton) Fri Dec 6 22:54:20 2002 From: jeremy@zope.com (Jeremy Hylton) (Jeremy Hylton) Date: Fri, 6 Dec 2002 17:54:20 -0500 Subject: [Python-Dev] We got leaks! In-Reply-To: References: Message-ID: <15857.10908.925878.557147@slothrop.zope.com> >>>>> "KJ" == Kevin Jacobs writes: KJ> HOWEVER, the leak disapears when Python is compiled normally, KJ> without debugging. Interesting clue. KJ> without debugging. I've also added some instrumentation to the KJ> interpreter and have determined that the leak is not occuring in KJ> a garbage collected object (anything in gc.get_objects()) that KJ> exists after the test-suite has been run. Yep. We tried that too :-). Jeremy From jacobs@penguin.theopalgroup.com Fri Dec 6 23:09:48 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 18:09:48 -0500 (EST) Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: First minimal test case: class Leaker(object): def __hash__(self): return 1 for i in range(10): hash( Leaker() ) print '*' * 10, 'total refs:', sys.gettotalrefcount() In the context of the datetime objects, this leak only affects the Python version. It also requires that the Leaker class is a new-style object. I'm in the process of isolating the other leak that hits both the Python and C verions of datetime in the test_pickling() function, though I suspect that may be other incarnations of the above problem. Fun, fun! -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From tim.one@comcast.net Fri Dec 6 23:20:42 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 06 Dec 2002 18:20:42 -0500 Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: [Kevin Jacobs] > First minimal test case: > > class Leaker(object): > def __hash__(self): return 1 > > for i in range(10): > hash( Leaker() ) > print '*' * 10, 'total refs:', sys.gettotalrefcount() > > In the context of the datetime objects, this leak only affects the > Python version. It also requires that the Leaker class is a new-style > object. Outstanding! Thanks, Kevin. I note that this variant shows the same growth: x = Leaker() for i in range(10): hash(x) print '*' * 10, 'total refs:', sys.gettotalrefcount() > I'm in the process of isolating the other leak that hits both the > Python and C verions of datetime in the test_pickling() function, > though I suspect that may be other incarnations of the above problem. Possibly. I'll try to figure out what's going on above first. > Fun, fun! The thrill fades a bit after the three hundredth time . From jacobs@penguin.theopalgroup.com Fri Dec 6 23:25:57 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 18:25:57 -0500 (EST) Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: On Fri, 6 Dec 2002, Kevin Jacobs wrote: > I'm in the process of isolating the other leak that hits both the Python and > C verions of datetime in the test_pickling() function, though I suspect that > may be other incarnations of the above problem. Here it is: import sys import cPickle dumped = cPickle.dumps(object()) for i in range(10): cPickle.loads(dumped) print '*' * 10, 'total refs:', sys.gettotalrefcount() With output: ********** total refs: 10929 ********** total refs: 10930 ********** total refs: 10931 ********** total refs: 10932 ********** total refs: 10933 [...] The leak occurs in cPickle.loads, but not in pickle.loads. It also does not depend on binary or non-binary mode of dumps(). It seems to require a new-style object to occur, but not one that implements a custom hash function as in the previously reported leak. This is all I have time for this evening... If the case(s) are not cracked tomorrow, I'll take a stab at dissecting the responsible C code. Enjoy! -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From barry@python.org Fri Dec 6 23:26:40 2002 From: barry@python.org (Barry A. Warsaw) Date: Fri, 6 Dec 2002 18:26:40 -0500 Subject: [Python-Dev] We got leaks! References: Message-ID: <15857.12848.917998.329690@gargle.gargle.HOWL> >>>>> "TP" == Tim Peters writes: >> Fun, fun! TP> The thrill fades a bit after the three hundredth time . You meant the 300th reboot <2000 & 98 winks> -B From tim.one@comcast.net Fri Dec 6 23:32:18 2002 From: tim.one@comcast.net (Tim Peters) Date: Fri, 06 Dec 2002 18:32:18 -0500 Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: > First minimal test case: > > class Leaker(object): > def __hash__(self): return 1 > > for i in range(10): > hash( Leaker() ) > print '*' * 10, 'total refs:', sys.gettotalrefcount() This one is clearly a missing decref in slot_tp_hash(), leaking a reference to the integer result object. If SourceForge wakes up, I've got a fix ready to check in. From jacobs@penguin.theopalgroup.com Fri Dec 6 23:35:53 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 18:35:53 -0500 (EST) Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: On Fri, 6 Dec 2002, Tim Peters wrote: > > First minimal test case: > > > > class Leaker(object): > > def __hash__(self): return 1 > > > > for i in range(10): > > hash( Leaker() ) > > print '*' * 10, 'total refs:', sys.gettotalrefcount() > > This one is clearly a missing decref in slot_tp_hash(), leaking a reference > to the integer result object. If SourceForge wakes up, I've got a fix ready > to check in. Just caught that too. FYI, it is also missing from the last Python 2.2 checkout that I did. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From just@letterror.com Fri Dec 6 23:55:13 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 7 Dec 2002 00:55:13 +0100 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: <200212061512.gB6FC8g16874@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > The problem with trying to load x.y right away is that then you can't > distinguish between "x.y doesn't exist" (and then you go to step 2) > as opposed to "x.y exists, but it tries to import something else that > raised ImportError" (and then you should simply propagate the > exception). Also not true: a function that finds *and* loads a module can return None for "module not found", a new module object upon succes, and propagate an error if one did occur during import. I'm tempted to make my new import hook scheme work that way: it avoids a few hoops I have to go through with import.c, as right now I'm quite limited what zipimporter.find_module() can communicate to zipimporter.load_module(). (It would also make it even closer to iu.py ;-) Just From Jack.Jansen@oratrix.com Sat Dec 7 00:17:43 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Sat, 7 Dec 2002 01:17:43 +0100 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: <02ac01c29d26$681e00b0$0900a8c0@spiff> Message-ID: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com> On vrijdag, dec 6, 2002, at 13:52 Europe/Amsterdam, Fredrik Lundh wrote: > Just van Rossum wrote: > >> It seems the majority doesn't like non-strings on sys.path, which to >> me is >> understandable, especially if you take PYTHONPATH into consideration. >> Let's try >> to focus on that. > > Has anyone presented any hard data, or are people just guessing? As > far > as I can tell, I'm the only one here who has posted fragile code from > a real > shipping product -- and I still believe it's a bogus argument. There are 3 distinct types of problems in Mac/Lib. 1. macresource locates .rsrc files on sys.path. I think this is a similar use case Fredrik presented: finding what is essentially a data file on sys.path. This is going to be a real problem if everything is packed into a zipfile. 2. buildtools loops over sys.path looking for a specific module (using os.path calls). This is a problem, but these uses can be converted to case (3). 3. buildtools (in another place) and bundlebuilder use imp.find_module to find modules and then proceed to do things like use the resulting pathname (like copying the file found). These will probably fail if there are zipfiles on sys.path, but at least there is a chance of fixing them. imp.find_module should return a recognizable magic cookie in stead of the pathname for zipfiles (and, later, for any other non-filesystem importers). I think we have to leave it at "magic cookie" and declare this Somebody Else's Problem, every time I try to think of functionality to put in the cookie I end up with something resembling a complete virtual filesystem interface. As a partial fix the file object returned by find_module could be an object that would "work". -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From DavidA@ActiveState.com Sat Dec 7 00:33:32 2002 From: DavidA@ActiveState.com (David Ascher) Date: Fri, 06 Dec 2002 16:33:32 -0800 Subject: [Python-Dev] zipimport & import hooks In-Reply-To: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com> References: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com> Message-ID: <3DF141DC.7090508@ActiveState.com> A question: Is 'import hook chaining' a requirement? Should one be able to import from a bzip'ed file inside a zip file at a URL? A (probably OT) thought: it feels like this should be viewed in the context of an even more ambitious (ha!) effort like Tcl's VFS (virtual file system) -- see http://mini.net/tcl/vfs. That is a more ambitious approach, but one that I like a lot at least in theory =). --da From jacobs@penguin.theopalgroup.com Sat Dec 7 01:31:45 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Fri, 6 Dec 2002 20:31:45 -0500 (EST) Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --416156269-869349433-1039224705=:12728 Content-Type: TEXT/PLAIN; charset=US-ASCII On Fri, 6 Dec 2002, Kevin Jacobs wrote: > Here it is: > > import sys > import cPickle > > dumped = cPickle.dumps(object()) > for i in range(10): > cPickle.loads(dumped) > print '*' * 10, 'total refs:', sys.gettotalrefcount() > > The leak occurs in cPickle.loads, but not in pickle.loads. It also does not > depend on binary or non-binary mode of dumps(). It seems to require a > new-style object to occur, but not one that implements a custom hash > function as in the previously reported leak. > > This is all I have time for this evening... If the case(s) are not cracked > tomorrow, I'll take a stab at dissecting the responsible C code. I lied. Attached is a patch for this leak and several others in cPickle.c (the diff around line 3118 to Instance_New is the culprit for the above test case). Someone should check over my changes, since I'm getting a little cross-eyed by now. This takes care of all leaks from the Python test suite, though there are still 2 ref-counts leaking per loop when using the C version of datetime. Enjoy, -Kevin PS: If Tim doesn't apply this patch fairly instantly, I will, of course, submit it to sourceforge. blah... blah... blah... -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com --416156269-869349433-1039224705=:12728 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="cpickle-refcnt.diff" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="cpickle-refcnt.diff" SW5kZXg6IE1vZHVsZXMvY1BpY2tsZS5jDQo9PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09DQpSQ1MgZmlsZTogL2N2c3Jvb3QvcHl0aG9uL3B5dGhvbi9kaXN0L3Ny Yy9Nb2R1bGVzL2NQaWNrbGUuYyx2DQpyZXRyaWV2aW5nIHJldmlzaW9uIDIu OTYNCmRpZmYgLXUgLXIyLjk2IGNQaWNrbGUuYw0KLS0tIE1vZHVsZXMvY1Bp Y2tsZS5jCTE2IFNlcCAyMDAyIDE3OjI2OjIzIC0wMDAwCTIuOTYNCisrKyBN b2R1bGVzL2NQaWNrbGUuYwk3IERlYyAyMDAyIDAxOjIxOjI4IC0wMDAwDQpA QCAtMjI1LDEwICsyMjUsNiBAQA0KIAlyZXR1cm4gcjsNCiB9DQogDQotI2Rl ZmluZSBQREFUQV9BUFBFTkRfKEQsTyxFUikgeyBcDQotICBpZiAoUGRhdGFf QXBwZW5kKCgoUGRhdGEqKShEKSksIE8pIDwgMCkgcmV0dXJuIEVSOyBcDQot fQ0KLQ0KICNkZWZpbmUgUERBVEFfQVBQRU5EKEQsTyxFUikgeyAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIFwNCiAgICAgaWYgKCgoUGRhdGEq KShEKSktPmxlbmd0aCA9PSAoKFBkYXRhKikoRCkpLT5zaXplICYmICAgICAg ICBcDQogICAgICAgICBQZGF0YV9ncm93KChQZGF0YSopKEQpKSA8IDApICAg ICAgICAgICAgICAgICAgICAgICAgICAgXA0KQEAgLTg5Nyw2ICs4OTMsNyBA QA0KIAkJaWYgKGtleSA9PSBOVUxMKQ0KIAkJCXJldHVybiAwOw0KIAkJaWYg KFB5RGljdF9HZXRJdGVtKHNlbGYtPmZhc3RfbWVtbywga2V5KSkgew0KKwkJ CVB5X0RFQ1JFRihrZXkpOw0KIAkJCVB5RXJyX0Zvcm1hdChQeUV4Y19WYWx1 ZUVycm9yLA0KIAkJCQkgICAgICJmYXN0IG1vZGU6IGNhbid0IHBpY2tsZSBj eWNsaWMgb2JqZWN0cyBpbmNsdWRpbmcgb2JqZWN0IHR5cGUgJXMgYXQgJXAi LA0KIAkJCQkgICAgIG9iai0+b2JfdHlwZS0+dHBfbmFtZSwgb2JqKTsNCkBA IC05MDQsOSArOTAxLDExIEBADQogCQkJcmV0dXJuIDA7DQogCQl9DQogCQlp ZiAoUHlEaWN0X1NldEl0ZW0oc2VsZi0+ZmFzdF9tZW1vLCBrZXksIFB5X05v bmUpIDwgMCkgew0KKwkJCVB5X0RFQ1JFRihrZXkpOw0KIAkJCXNlbGYtPmZh c3RfY29udGFpbmVyID0gLTE7DQogCQkJcmV0dXJuIDA7DQogCQl9DQorCQlQ eV9ERUNSRUYoa2V5KTsNCiAJfQ0KIAlyZXR1cm4gMTsNCiB9DQpAQCAtOTE5 LDggKzkxOCwxMCBAQA0KIAkJaWYgKGtleSA9PSBOVUxMKQ0KIAkJCXJldHVy biAwOw0KIAkJaWYgKFB5RGljdF9EZWxJdGVtKHNlbGYtPmZhc3RfbWVtbywg a2V5KSA8IDApIHsNCisJCQlQeV9ERUNSRUYoa2V5KTsNCiAJCQlyZXR1cm4g MDsNCiAJCX0NCisJCVB5X0RFQ1JFRihrZXkpOw0KIAl9DQogCXJldHVybiAx Ow0KIH0NCkBAIC0zMTE1LDYgKzMxMTYsNyBAQA0KIAkJCVB5X1hERUNSRUYo c2FmZSk7DQogCQkJcmV0dXJuIE5VTEw7DQogCQl9DQorCQlQeV9ERUNSRUYo c2FmZSk7DQogCX0NCiANCiAJaWYgKGFyZ3M9PVB5X05vbmUpIHsNCkBAIC0z MjE4LDcgKzMyMjAsMTAgQEANCiAJaWYgKCFtb2R1bGVfbmFtZSkgIHJldHVy biAtMTsNCiANCiAJaWYgKChsZW4gPSAoKnNlbGYtPnJlYWRsaW5lX2Z1bmMp KHNlbGYsICZzKSkgPj0gMCkgew0KLQkJaWYgKGxlbiA8IDIpIHJldHVybiBi YWRfcmVhZGxpbmUoKTsNCisJCWlmIChsZW4gPCAyKSB7DQorCQkJUHlfREVD UkVGKG1vZHVsZV9uYW1lKTsNCisJCQlyZXR1cm4gYmFkX3JlYWRsaW5lKCk7 DQorCQl9DQogCQlpZiAoKGNsYXNzX25hbWUgPSBQeVN0cmluZ19Gcm9tU3Ry aW5nQW5kU2l6ZShzLCBsZW4gLSAxKSkpIHsNCiAJCQljbGFzcyA9IGZpbmRf Y2xhc3MobW9kdWxlX25hbWUsIGNsYXNzX25hbWUsIA0KIAkJCQkJICAgc2Vs Zi0+ZmluZF9jbGFzcyk7DQo= --416156269-869349433-1039224705=:12728-- From tim@zope.com Sat Dec 7 03:27:10 2002 From: tim@zope.com (Tim Peters) Date: Fri, 6 Dec 2002 22:27:10 -0500 Subject: [Python-Dev] We got leaks! In-Reply-To: Message-ID: [Kevin Jacobs] > ... > Attached is a patch for this leak and several others in cPickle.c > ... Thanks, Kevin! THe patch was spot-on and I checked it in (and marked as a bugfix candidate since the same cPickle leaks appear to exist in 2.2.2). I already backported the tp_slot_hash leak fix to the 2.2 maintenance branch. I checked in the leak-detector code to test_both.py, so this will be easier to try the next time around. The remaining leaks in the C implementation were my errors, in the date and datetime pickle support functions. There are no visible leaks remaining in the Python or C implementations of the sandbox datetime code. BTW, running hefty tests in a loop and staring at sys.gettotalrefcount() each trip is, as I think this exercise demonstrated, a powerful leak detector. I think we plugged about 10 slow leaks here, 8 of them long-standing leaks in Python (of course, whether they're "slow" or "fast" leaks depends on the app! for the test suite, they were slow). Fun, fun . From arigo@tunes.org Sat Dec 7 16:02:18 2002 From: arigo@tunes.org (Armin Rigo) Date: Sat, 7 Dec 2002 08:02:18 -0800 (PST) Subject: [Python-Dev] os.path.commonprefix Message-ID: <20021207160218.1528F49BA@bespin.org> Hello everybody, I recently discovered that os.path.commonprefix(list-of-strings) returns the longest substring that is an initial segment of all the given strings, and that this has nothing to do with the fact that the strings might be paths. I wonder why this has been put in os.path and not in the string module in the first place. This location misled me for a long time into thinking that commonprefix() would return the longest common *path*, a feature whose interest I could see. But: >>> commonprefix(['/home/users/arigo', '/home/users/alal']) '/home/users/a' What is the use for such a thing ?? I suggest that a more descriptive comment be added in the documentation. Armin From mwh@python.net Sat Dec 7 16:20:25 2002 From: mwh@python.net (Michael Hudson) Date: 07 Dec 2002 16:20:25 +0000 Subject: [Python-Dev] os.path.commonprefix In-Reply-To: Armin Rigo's message of "Sat, 7 Dec 2002 08:02:18 -0800 (PST)" References: <20021207160218.1528F49BA@bespin.org> Message-ID: <2m4r9pssvq.fsf@starship.python.net> Armin Rigo writes: > >>> commonprefix(['/home/users/arigo', '/home/users/alal']) > '/home/users/a' > > What is the use for such a thing ?? I seem to recall a discussion which decided that there is no use for such a thing, but that changing it might break code for people who found one. ICBW. > I suggest that a more descriptive comment be added in the documentation. No argument from here. Cheers, M. -- We did requirements and task analysis, iterative design, and user testing. You'd almost think programming languages were an interface between people and computers. -- Steven Pemberton (one of the designers of Python's direct ancestor ABC) From bh@intevation.de Sat Dec 7 16:32:32 2002 From: bh@intevation.de (Bernhard Herzog) Date: 07 Dec 2002 17:32:32 +0100 Subject: [Python-Dev] os.path.commonprefix In-Reply-To: <20021207160218.1528F49BA@bespin.org> References: <20021207160218.1528F49BA@bespin.org> Message-ID: <6qptsdvlgf.fsf@salmakis.intevation.de> Armin Rigo writes: > >>> commonprefix(['/home/users/arigo', '/home/users/alal']) > '/home/users/a' > > What is the use for such a thing ?? I suppose you could use it for something like filename completion as found in many unix shells. That's perhaps the reason why it's in os.path and not in string. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From skip@pobox.com Sat Dec 7 16:33:49 2002 From: skip@pobox.com (Skip Montanaro) Date: Sat, 7 Dec 2002 10:33:49 -0600 Subject: [Python-Dev] os.path.commonprefix In-Reply-To: <20021207160218.1528F49BA@bespin.org> References: <20021207160218.1528F49BA@bespin.org> Message-ID: <15858.8941.920830.141997@montanaro.dyndns.org> Armin> I recently discovered that os.path.commonprefix(list-of-strings) Armin> returns the longest substring that is an initial segment of all Armin> the given strings, and that this has nothing to do with the fact Armin> that the strings might be paths. :-) I discovered the same thing over a year ago, fixed the problem in CVS, then got roundly booed because I'd changed the semantics. I think it broke some bit of code of Mark Hammond's. I recently used this function to find the common prefix in a bunch of email addresses and prefaced it with this comment: # don't be fooled by "os.path." - commonprefix operates char-by-char! pfx = os.path.commonprefix(all_addrs) The Tools/scripts/trace.py file has a component-wise implementation of commonprefix(). Skip From skip@pobox.com Sat Dec 7 16:34:34 2002 From: skip@pobox.com (Skip Montanaro) Date: Sat, 7 Dec 2002 10:34:34 -0600 Subject: [Python-Dev] os.path.commonprefix In-Reply-To: <2m4r9pssvq.fsf@starship.python.net> References: <20021207160218.1528F49BA@bespin.org> <2m4r9pssvq.fsf@starship.python.net> Message-ID: <15858.8986.387636.729697@montanaro.dyndns.org> Michael> ICBW. Nope, you were spot on. Skip From lists@morpheus.demon.co.uk Sat Dec 7 15:43:52 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sat, 07 Dec 2002 15:43:52 +0000 Subject: [Python-Dev] VFS (Was: zipimport & import hooks) References: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com> <3DF141DC.7090508@ActiveState.com> Message-ID: David Ascher writes: > A (probably OT) thought: it feels like this should be viewed in the > context of an even more ambitious (ha!) effort like Tcl's VFS > (virtual file system) -- see http://mini.net/tcl/vfs. That is a > more ambitious approach, but one that I like a lot at least in > theory =). I agree - this is definitely where my thoughts are heading. I originally started from the idea of Java's JAR file support - packed archives of code, which were also executable. Java also supports getting resources out of JAR files. I later found Tcl's VFS, and Starkits. These are, in effect, the same concept. It's worth remembering,though, that all of these things can be done now, with varying degrees of success, in Python by user-level code. All core support adds, really, is a solution to the bootstrapping problem (you can't import zipfile support from a zip file, so it can't be included in an application packaged as a zip file). Another feature I'd find useful is a token (like Perl's __END__) which tells the parser to stop parsing at that point. It allows data to be embedded at the end of a main script. As zipfiles can have arbitrary data added at the beginning, you can then zip up resources and access them via zipfile.ZipFile(sys.argv[0]), or even sys.path.append(sys.argv[0]), allowing you to package code and data together. [Actually, this last approach won't work under PEP 273, as it mandates a file extension of .zip rather than checking the file signature. Rats.] But anything like this is way OT for zipfile imports. It almost certainly needs a PEP if it is to be done right, as simply stealing another language's feature without thinking it through is not going to work... Paul. -- This signature intentionally left blank From lists@morpheus.demon.co.uk Sat Dec 7 15:56:00 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sat, 07 Dec 2002 15:56:00 +0000 Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension Message-ID: PEP 273 does not mandate a particular file extension for zip files. However, both current implementations (Jim Ahlstrom's patch and Just's import hooks) recognise zip files via the extension ".zip". I'd like to suggest that this should not be the case, and that zipfiles should be recognised by the embedded signature, as is done by zipfile.is_zipfile(). This will (slightly) slow down the initial checking for zipfiles, as it requires a file read, but if the result of the check is cached the performance impact will be minimal. The main advantage of this is in generality. However, I have a specific usage in mind - namely, concatenating a Python script with its supporting code as a zip archive. (Zip archives can have arbitrary data prepended without affecting their validity). Then, simply by executing sys.path.append(sys.argv[0]) the script can access its own embedded modules. One problem with getting this to work is that at present, the Python parser will choke on data appended to the script. This can be avoided under Windows by a system-specific hack, of embedding a ^Z character in the script (which text-mode I/O treats as EOF) just before the appended data. In the longer term, a language-mandated "stop parsing here" token, like Perl's __END__, might be useful. But that's not something I'm proposing right now (unless an expert in Python's parser steps up and says "oh, that's easy - I can implement that in a couple of lines of code" :-)) Any comments? Paul. -- This signature intentionally left blank From martin@v.loewis.de Sat Dec 7 17:26:30 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 07 Dec 2002 18:26:30 +0100 Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension In-Reply-To: References: Message-ID: Paul Moore writes: > Any comments? That's quite expensive. Unless I'm mistaken, it causes an additional stat call per sys.path item (to find out that it is a directory). If Just's patch is accepted, you get that feature for free import zipimport,sys sys.path.append(zipimport.zipimporter(sys.argv[0])) does what you want. For Jim's patch, I'd say you lose: We should tighten the PEP saying that only items in sys.path are treated as zipfiles that have ".zip" in their name (with a yet-to-be-specified separator for putting directory in the zipfile into sys.path). Regards, Martin From just@letterror.com Sat Dec 7 17:56:10 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 7 Dec 2002 18:56:10 +0100 Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension In-Reply-To: Message-ID: Martin v. L=F6wis wrote: > That's quite expensive. Unless I'm mistaken, it causes an additional > stat call per sys.path item (to find out that it is a directory). >=20 > If Just's patch is accepted, you get that feature for free >=20 > import zipimport,sys > sys.path.append(zipimport.zipimporter(sys.argv[0])) >=20 > does what you want. For Jim's patch, I'd say you lose: We should > tighten the PEP saying that only items in sys.path are treated as > zipfiles that have ".zip" in their name (with a yet-to-be-specified > separator for putting directory in the zipfile into sys.path). Note that I'm currently working on a variation of the patch that implemen= ts a subset of iu.py (I think I mentioned this a few times already ;-). For ea= ch path item, each item in a list of hooks is asked whether it can handle the pat= h item; the first one that can is used. This is done *once* per path item, the re= sults are cached in a dict (iu's shadowpath): {path_item: path_handler, ...}. T= his gives the zipimporter a chance no matter what the extension is and it's s= till cheap as it's only done once. Just From aahz@pythoncraft.com Sat Dec 7 20:21:28 2002 From: aahz@pythoncraft.com (Aahz) Date: Sat, 7 Dec 2002 15:21:28 -0500 Subject: [Python-Dev] Import compromise Message-ID: <20021207202128.GA28924@panix.com> Guido, I call for a Pronouncement: I'm seeing some convergence on basic criteria needed for a Grand Import Redesign, but I'm not seeing convergence on the actual design itself. Time is running short for 2.3, and people making hasty decisions make poor decisions. OTOH, the complaints raised about the current patch for PEP 273 have merit, I think. On the gripping hand, we really want PEP 273 in some form for 2.3. So here's my proposed compromise: Split the PEP 273 patch into two parts. The first part is the absolutely critical modifications to import.c, which basically boils down to making a function call every time you access a '*.zip' string on sys.path. Everything else goes into zipimport.c, a compiled-in module like socket. (I'd even support making zipimport.c an include file for import.c) Yes, this means we lose the caching for non-ZIP files for 2.3 -- I vote YAGNI. Yes, this makes more work over the long haul, but it saves stress and time now. Yes, it's more work than just accepting the current PEP 273 patch, but I think there are too many good arguments against that. zipimport will *NOT* be a publicly exposed library for 2.3, possibly never. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From fredrik@pythonware.com Sat Dec 7 20:41:19 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 7 Dec 2002 21:41:19 +0100 Subject: [Python-Dev] Import compromise References: <20021207202128.GA28924@panix.com> Message-ID: <026301c29e31$00632980$ced241d5@hagrid> Aahz wrote: > Time is running short for 2.3, and people making hasty decisions make > poor decisions. speak for yourself. From martin@v.loewis.de Sat Dec 7 20:54:09 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 07 Dec 2002 21:54:09 +0100 Subject: [Python-Dev] Import compromise In-Reply-To: <20021207202128.GA28924@panix.com> References: <20021207202128.GA28924@panix.com> Message-ID: Aahz writes: > I'm seeing some convergence on basic criteria needed for a Grand Import > Redesign, but I'm not seeing convergence on the actual design itself. > Time is running short for 2.3, and people making hasty decisions make > poor decisions. I disagree with this analysis. Time is not running short for Python 2.3: there isn't even a release schedule, yet, and I question promises that an alpha release will be produced this year. > Split the PEP 273 patch into two parts. The first part is the > absolutely critical modifications to import.c, which basically boils > down to making a function call every time you access a '*.zip' string on > sys.path. Everything else goes into zipimport.c, a compiled-in module > like socket. (I'd even support making zipimport.c an include file for > import.c) Good plan. Who is going to provide the implementation? I think Just would be happily drop his implementation if such a change would magically appear, but I think that is unlikely to happen. Regards, Martin From just@letterror.com Sat Dec 7 21:05:30 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 7 Dec 2002 22:05:30 +0100 Subject: [Python-Dev] Import compromise In-Reply-To: Message-ID: Martin v. L=F6wis wrote: > I think Just would be happily drop his implementation if such a change > would magically appear, but I think that is unlikely to happen. Oh, Aahz's proposal is *very* close to what my current implementation loo= ks like, except that it exposes a couple of things here and there ;-) I'll try to post code later today (it works for me now, but I'd like to g= o through it one more time, do some more testing), and will then write a sh= ort piece about what it does, why I think it's good that it does that, what t= he impact is on import.c, etc. etc. (And depending on the size if that piece= I might call it a "draft PEP"...) Just From aahz@pythoncraft.com Sat Dec 7 21:09:05 2002 From: aahz@pythoncraft.com (Aahz) Date: Sat, 7 Dec 2002 16:09:05 -0500 Subject: [Python-Dev] Import compromise In-Reply-To: References: Message-ID: <20021207210905.GA10103@panix.com> On Sat, Dec 07, 2002, Just van Rossum wrote: > Martin v. Löwis wrote: >> >> I think Just would be happily drop his implementation if such a change >> would magically appear, but I think that is unlikely to happen. > > Oh, Aahz's proposal is *very* close to what my current implementation > looks like, except that it exposes a couple of things here and there Would you have any objection to restricting the scope of your work so that it *only* covers PEP 273? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From aahz@pythoncraft.com Sat Dec 7 21:11:38 2002 From: aahz@pythoncraft.com (Aahz) Date: Sat, 7 Dec 2002 16:11:38 -0500 Subject: [Python-Dev] Import compromise In-Reply-To: References: <20021207202128.GA28924@panix.com> Message-ID: <20021207211138.GB10103@panix.com> On Sat, Dec 07, 2002, Martin v. Löwis wrote: > Aahz writes: >> >> I'm seeing some convergence on basic criteria needed for a Grand Import >> Redesign, but I'm not seeing convergence on the actual design itself. >> Time is running short for 2.3, and people making hasty decisions make >> poor decisions. > > I disagree with this analysis. Time is not running short for Python > 2.3: there isn't even a release schedule, yet, and I question promises > that an alpha release will be produced this year. You may be right; however, the tone of the discussion seems to be partly predicated on a short timeframe. I'm fairly confident Guido won't hold 2.3 for lack of a new import facility. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From aahz@pythoncraft.com Sat Dec 7 21:23:26 2002 From: aahz@pythoncraft.com (Aahz) Date: Sat, 7 Dec 2002 16:23:26 -0500 Subject: [Python-Dev] Proposed changes to linuxaudiodev In-Reply-To: <20021130164818.GA25894@cthulhu.gerg.ca> References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca> Message-ID: <20021207212326.GA13707@panix.com> On Sat, Nov 30, 2002, Greg Ward wrote: > On 29 November 2002, Guido van Rossum said: >> >> I recommend that you simply check in your new code as ossaudiodev and >> we'll mark the linuxaudiodev module as deprecated. > > Sounds fine to me. Should I worry about adding an official > DeprecationWarning to linuxaudiodev.c? Or does that wait until 2.4? Given its lack of official documentation, I'd vote for DeprecationWarning now. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From lists@morpheus.demon.co.uk Sat Dec 7 21:34:25 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sat, 07 Dec 2002 21:34:25 +0000 Subject: [Python-Dev] Import compromise References: <20021207202128.GA28924@panix.com> Message-ID: Aahz writes: > Split the PEP 273 patch into two parts. The first part is the > absolutely critical modifications to import.c, which basically boils > down to making a function call every time you access a '*.zip' > string on sys.path. Everything else goes into zipimport.c, a > compiled-in module like socket. (I'd even support making > zipimport.c an include file for import.c) Who's going to write this code? I can't - my involvement has been in updating Jim's patch to make it apply cleanly to current CVS, testing it a little, and making some bugfix-type changes. I don't feel qualified to do the major surgery you're suggesting. Frankly, there are only 2 relevant chunks of code - Jim's patch and Just's. The disadvantages of Jim's have been spelled out, and remain true. Its main advantage is that it works now. Just's patch is far better in theory. It remains to be seen if he can deliver working code in the necessary timeframe (whatever that is...). Of course, in the absence of code, we all get to have a lot of fun talking about what might go wrong if it works the way we assume it does, but we've got to while away the time somehow :-) Paul. -- This signature intentionally left blank From bac@OCF.Berkeley.EDU Sat Dec 7 22:08:32 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sat, 7 Dec 2002 14:08:32 -0800 (PST) Subject: [Python-Dev] Import compromise In-Reply-To: References: <20021207202128.GA28924@panix.com> Message-ID: [Paul Moore] > Of course, in the > absence of code, we all get to have a lot of fun talking about what > might go wrong if it works the way we assume it does, ... and drive the person who has to summarize all of this bloody long thread insane. At least it helped give me some inspiration (along with one of Martin's emails) on how to implement ``dummy_threading`` rather easily. -Brett From fredrik@pythonware.com Sat Dec 7 23:47:00 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sun, 8 Dec 2002 00:47:00 +0100 Subject: [Python-Dev] Import compromise References: <20021207202128.GA28924@panix.com> Message-ID: <02ed01c29e4a$f82e7d40$ced241d5@hagrid> Martin wrote: > I disagree with this analysis. Time is not running short for Python 2.3 In addition to the schedule issues (and nothing says that the zip importer has to be in the first alpha), Aahz also ignores the fact that the people involved has long experience of python code archives (I wrote squeeze in 1997, and that wasn't my first archiver; Gordon has been shipping his installer for several years, the mac folks have been using this stuff since the early days; that's easily 10-15 years of experience of using archive technology in real applications. hasty indeed); he also ignores the fact that we have two working implementations to chose from, each under- stood by more than one programmer; that we have input from large-scale Python users; and that this isn't some groundbreaking change to the Python language, but merely an extension to the way the CPython implementation uses the sys.path variable. Etc. If this isn't enough to get this feature into CPython 2.3, there's something wrong with the python-dev process. From fredrik@pythonware.com Sat Dec 7 23:49:47 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sun, 8 Dec 2002 00:49:47 +0100 Subject: [Python-Dev] Import compromise References: <20021207202128.GA28924@panix.com> Message-ID: <02f901c29e4b$5561be00$ced241d5@hagrid> Brett Cannon wrote: > ... and drive the person who has to summarize all of this bloody long > thread insane. maybe you could team up with Ping and turn his zest tool into a brettbot? http://www.sims.berkeley.edu/~ping/vertex-2002/zest.html From bac@OCF.Berkeley.EDU Sat Dec 7 23:55:15 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sat, 7 Dec 2002 15:55:15 -0800 (PST) Subject: [Python-Dev] Import compromise In-Reply-To: <02f901c29e4b$5561be00$ced241d5@hagrid> References: <20021207202128.GA28924@panix.com> <02f901c29e4b$5561be00$ced241d5@hagrid> Message-ID: [Fredrik Lundh] > Brett Cannon wrote: > > > ... and drive the person who has to summarize all of this bloody long > > thread insane. > > maybe you could team up with Ping and turn his zest tool > into a brettbot? > > http://www.sims.berkeley.edu/~ping/vertex-2002/zest.html > Perhaps, but part of the reason I wanted to do the Summaries was to force myself to read the emails so as to make sure I learn all the little nuggests of knowledge embedded in each email. I just needed a slight venting. =) I will have a look, though. Thanks, Frederik. -Brett From just@letterror.com Sun Dec 8 00:44:47 2002 From: just@letterror.com (Just van Rossum) Date: Sun, 8 Dec 2002 01:44:47 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) Message-ID: --7584767-3623878656-3248300694=:9002 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit Here's the current state of my stuff. I'd rather post it now than to fiddle more: I'm too tired, I'd only break stuff... I currently build the zipimport module as a builtin to avoids some bootstrapping pitfalls (*). It should still work as a shared lib *provided* it is available on sys.path *before* site.py is run. The same goes more or less for zlib unless you're using uncompressed archives. The test suite ran fine with the std lib packaged as a zip file (minus the test suite it, as it globs for *.py in Lib/test/), apart from three tests that do stuff with module.__file__ (eg. email can't find it's data files ;-). Highlights: - zip file names on sys.path work - zip file names PYTHONPATH work - sys.path still contains strings - module.__file__ is still a regular string, looking like this: '/path/to/archive.zip/path/to/module.pyc' - pkg.__path__ still contains a regular string, that looks like this: '/path/to/archive.zip/path/to/packagedir/' I'll write a bit about the mechanics tomorrow, but in the meantime I would like to see some test results/bug reports/bug fixes from Windows developers. Please don't start with the full monty (ie. the std lib), try a small test archive first please ;-) There's still plenty to do regarding imp module integration, minor things like reload() not being supported yet, etc. Nevertheless I think the import.c patch is remarkably small for what it accomplishes. Needless to say, the zipimport module itself also needs work. Fun, fun! Just *) It's mostly a problem when you're running Python from the source directory, when site.py adds stuff to sys.path to get a running system: by then it's too late to find zipimport. It should be less of a problem with a regular install, but I haven't tried. --7584767-3623878656-3248300694=:9002 Content-Type: application/octet-stream; Name="zipper3.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000" Content-Transfer-Encoding: base64 Content-Disposition: attachment; Filename="zipper3.tgz" H4sIAAAAAAAAA+w8a1fbSLLz1f4VHecEbGPA5pUsXjLHCZCwlyScQGazm+ToCKkN WoSkkWQSzwz//VZVvyUZyJnHfrjXJyfG3VXd1dXV9eqSfomyjOeb6z/8iZ/hcGv4 dHsbvofDpzvie7ixJb7F5wcAeLq1Dd2b0D4abW49/YFt/5lEqc+sKP2csR/+A3/c BQdg0+lfQdBf+/lF7j98R9dZmpdrwR8+x3A0HO5sbS3Y/62NrZ3q/m9ub2/+wIZ/ OCUNn//j+/84SoJ4FnLWOZmXl2mydtlpm7aizGdBec2vz3nu9qRFyKeF23bt58Wl H7uNQXqdRTHHxnYbmFhGAQsu/bzPsrnHv5U8KaI0KT59YXvs185aNg86A4bfqfzu 3I4Bc73PtIjynKXn/+FByYCGKIlKGID5SciKWYb9rL/ebpfzjEM3E0twkP8dZUfy 7zHSVAP4td06mb+jKbzXB5N985P1/Ty4jG74mDEgKfPLy8S/5iydsvKS4yBMAiAR FtoUeFAAEqGFETRd8ITnfsywh0XJNK1gZDks7ttYYBCQaNllHX+9mJ2vh1EOgGk+ X+8g6u1Y8/dkfgarlwNZq/WweUzMnKb5Vz8PgYNBjNgaU01/wUuvTIMu7pVedW9c B4xTP/Rg4d51Gs5i3rXmY/2Cx9MBE2NMZ3GMzBq0W6IBEJCBA1h9yaIi84Mr/4Kr n1JwwoE1FxDk8aTM50BI+zHtPncW+PKSB1fdNOtpJFqzah6wpSo7eu36ktqWMHgJ /9p1GNoP4sImys8vnN9XX8Oi1wYZAjYfXSRpzkEmLmbXQDjA2UOj7LLSv+IFC/xc SpFfAoNg11Ggv3IWpslyCT8AMGXXAMviaMoZMDAEBsM2gtSfB7FfFLxYIxGqbQDw qoXfcMLc3enBSlafl5nnxzFuNa5rCKxtRVPWJYxHe+zth+PjHh4JGmP1uZJv0TPW 7SThtVYhs6b5tt3KeTnLE9Y1HOsJKm/1VoAEtKuMahKsB2yCEDWUM1wXsDHmifgl lvnoZD7JL7wT0F38bJaBAIuhOu8e7VokrHlEhOd1YHdauD/wWTqZn5Z5lFyQJIFw ORzq9QBSLnZ1hPOdzL2jty/fHxx2XUCgDEkCLukBJ4X4ow4p6beBT6NfeA1QLE9B P99jbyYfTyZnr48P3ortPJkf5Ll3yks5E/z+FniCxdCT5rTSVkdtOI1TpimL0+Si g6TYq4OdbVckQakQRLSlSkMIqWBLS+yRIAYI4Bmq8zd+GVzyQtH0juipMnTRgGJ5 cPoU5df+nJ1rJQ1n5rqttpB11pG89TJdl9BrsOuoX0G9kl5t2ToZQD0lEiAkfW0y jNyTwJ3Ppp8Mv1dGX3RPH5RQv3oqWhn8LWW0hWe6Ox6LVbRI+YMqQMrhlJcsDYJZ nvtJQAvqILmSThoFzBn862YD2dUTlurjx4/Mj4uUrf376ESCI/syzTTa7POc+1dj 3flp6wv2nx6c9PQov/3GJsdn0CRHaenFZGNBxMoe28I/4ayLYSSA3G4YQDWt0qJ7 2KUldZVt9czaE6AYreo0nQETgDP0KyqYXzIOLYIGIxRIpRQXhCrKKI5ZwUsBeKvF /mXM/bxLQmwOlEsWWwFKaHNEKyxrmxRbmSdBNu/CHg+YMGBqiBU2IgzcftmGfs3y 5+GyOxOMAT/kfIRiGPXJcGL0BTUwsZ8Y4gLg0NA3rvcgIfbEuG45M5BmtLPR5VqX HObptVQI6txWVXkjrFqJXEplfLP17liW7DmaErf+LJ+jzQMtwnx0IUAhhI6bNoXp WTEv1txjuSYO9z+17QxDHAcPUICDsHQGuj6+Ufay5aKLBc4L7xUvxbnvdlyIjtkw F/ORUWgn831w8qQ/4oIp6bZ0h4SGGY9Kfl2BF2oY11SzBYIKPZA1f90pUlBqemRx kaCeJX+0EOzUXAqVjy1PuavauxVHQg8uQYg2tHcfHYNHfYLw20XH8U6lfo+FWYxN 9Ni2Ayc3zsjQ9j5u0ih03I+QCwep5oGQh4Gr3D9oNuu1Ps2BWo/RBeJ3eu5hDEP+ 2TTnwsD3ql4SeoQeDoqe9WLfm/xu+TfpLfFnP/OMBw4/tBO+0HXCwykNgbFvMBpI GKgBirxQQWr9eJ9DIz0aoxIBC9ukBtUqrt2StEpDKTQZ6DlEBm3neANSu7jOqybK IC0mT26Gsg+o+5u8LbNnt+0FMyDna4tU8orfzjJ1y4qtv3FlJ/M3fpbh/K/94n/4 3CGXpEoYI3m87c2EgUZjwwJpIDSFHe3a2sv9m1gUjwveNOBwLBWnIl+Sao4UCcvt fdEV+jYqcjQgoHqqzj1JZE2+6yENoI6V6IIXJwS/L6Tc9cq+SKmOyH4PTAR6b2hQ OJGBtQQnOJCz216rcPekqJY5nIG8q4lcXlvuuTLb7KH/5MczbjvocgR2PQP3MElL 9HV9MH5lycM17HGcdR2IKZl0VYjQGpooOsRLgn1LRj2MdXTB/g4hY32F5MRGJCgs AphN+FpZcY6ikcNBJRcUfdGW4DvFvhJiCZuwMOBURgAUmtg/4p5wNUUyRg0Gf71N E1q3HE22PEDEKUHyZ4m4TqpY0l4X85om/70yb63JkXlFziKhf6S1qHBKXGMA/iGl Ppr1uIwTG/CVtv6uU6N8U8yzyIn8EjDOZyX4DXSW4BwVNEax6ASpo0eanY6z2RA8 z8osKTDjjRhMhTEW2tb0rayI2K24gpAHRiPr+jtP7XqfOKwsqYxsFCb7tAIBsbYH X6AHjiVh1O0XNaO129OB9gpaMfgf+8Fir9+tBxryejIkHlByUMfHD9YZAGgPZZSH 9KwXag+lbgy2o4hbbkoSj2SEEBtCwQhaq4lQsSdGHmhTTJqzLZ16/NjZTjvHKRSa dBMrHUguTbyndJHl2J/7IUjIRZSwZIbJe3Dw2Pm8BN8+DYEUXGACzFPOvZkERhTD t4I0KaNkxrW3rsQfQFRMWXXeb9sPyCqxjtYkuxBrJMQ4qYA7loYVe2+r1ze8vEzD fT51sqjX1CpvENqtXzuOQWYLvI0Be3Nw9tr7afJ+8v7VKRAFLm3ndoD4jnJjC1T5 YnwS4FYLv24pyOLISB7XkvRv6F6lvhxstZYjDxeQws68dy/+cfDybMDS6bTgZTq1 TcNAncMegLKhWAxJOeLejy1iEwJV2EJBAPp9uFIFDxQmrZ6WC7EtCO7p2XtgVdeJ qtJg0Ea2gasSUCa7fsPTsRP9+weHB+/fH+x7k/399wenp1387rHhw+49BDudex2w 7kdn3dqwS2IccT0Aam4ICzIyu2ZRibavgHDA5UZvgDgteRQhfItA9SAYHbduyMWV U5r3GoLMgcSRPwnFGSzLKVirNkMojOar3lEs6kCVg7cOtY6cZw3gfuFJXdLUVfCf Zxxzkk2d10IL1/su/eKygTBYecM65Br0Dr7CC7QoANU+gfUNKoxIF3KioQdoBO08 5WoG7+zk8Hjy6hS04uHkw/EZ+41ZjS8mpwdn/zo5GDCBPY39i4Iwq+KtJwibdrLM /Rs46Q08C1CjNmxMFFwu3LWv3L+Ko6IU57PeH5VNW4et2hI0aFWzBNnQAEj6ygak hkYRbaTs3G/iAV2S1lt5EeQ40qKuxhlwLIsvXfRvsjwNnAOIjVqK6GpOCAPpAils E3FEtdzoI1q5MTQg8MMV230eDwxlmNlRhgHvZWGQmONVoU+32ujxgcsGClmlurSv vPpcKOw29r6CpgSCPnlHA39ZV9HoekRxiDEh8GHA1HGkS2X0TQocogtL8WOVKdWo swJB0a3zCxih4CB9PuougoTVo0sS8WKN6JgY94WS9BhM5aslBhK7BEDePcpwzotC mBr2mKkGXPIV2OgxG5JBmCWqh4cKm4V+6XuoTgcSm1RrOmUGlmAYDBZGxZVGxPU2 Ioa8guqiCMkZEIqUIkTCKUQKuvRzapJMU9g9WvG+vKaP6JY3waACGtKLBCYPwSkj doMqiGLFaPDVsE3cldsLpljFIomhIhiutVFK6jFo8w1++9daWQJ42yW4XYdHxwfQ grc4eMPHzC4FeTCw2W4x0mEQgKBzGVEsQ2Nccj+EE6EAcEkSUfbgDwWMQSnFpRcY twQQhpcqdqVMk5VwXGHbOq7FQRf19TMr3YKRiz4+zx94H/oOVPQ0Tr/aoSOeLTpp FDOBnDfeipobbyt4MT4aZUnwxm6aZoYwcCTz846KG6eZm/4WJB6m+bVfunejNa96 GSQSxiU9gJu0yzo6Tu8sP1nbGA6L5Y5NTgPhsGv8CqgYsNWNjQGGO6i2TqFRSD7e wIHckyJBJacyNlQH5L2HLT4GruA10SGAwECU4Bp+G+4Mt7debA/1Te0LP9xlB2K0 l6g+QBPBwYETepH4QJQYvjUN4rSgccbfwQ1M6vi/ixHACXnjIG9q3vKvXb1J1YuG airMMHFnIEJG51jIQHUxzyhsZ6dKy1j8EfU/xBtxbDAyRUHsVRLNdJhk2ta5XzbE VY7qUF4c62lpkwWQ2Iz4IYSLNKHc9Y3hiHYd+S9umvXe25uO6Oy1NdMCIjF9PZQV Ky2lrWpEnV6CEFSpQh31MEjUaQ+DBC35II4Ydf4QaK1rHwStFezDSLaUMCBs7QBH zQgr4h71nmEeCDa+cxu3NtQ22tbtoeJlSHaUOkqZzQ/TRYkiVL0y7VbLUmk0K1vV 6iMGWdMemNZAUUAk9ElRL68vi4tGAtVX8ysrKl9C7UOS+gneSvP8OgL9hoY/mUHQ QykaEvkKh/ZceymUJqVAdVlYgVqSqkTIj7OPK1ioIM1zfZtrNsIx32IHvv9Ybw03 9bHWWSN9so+JnDvOtL3lK2xjxyZkcwhNf5AEIkWCN5ItOiR3pA54HY9bNc1H7h5d b+prrwunBEOVhZgU5YUoxBBuz8C+L11V3atUMIIKibjuvUBHnVLV3U4En87AcsMa PbC2yCJaKzDJTLvuQhquUycRKt0/QV/ZU3c5rYsUpIqjJRV2VqcgBbFkTUiq0Uew bbJwALyfeH4OjYcQGPdoAKyn+GcOweZpCZzPu53HzDLQWKHxBAIUikWEZ/Gk+JyI mwWazDHP0r7KQgMic7dtUanv2BcmEh0f2Xj/3hQCji5e/5ObXAM3kKZ0y9zFs0sI 571f4ugcekUZC7LDasVmvMWwx3HdOzUTwiPr5Wgnc5FbknnUNyLB3MHejhBswUpG 8CTSOLWa1Upk2zQOZYK5UoCBikpf+zoIdIvccrhgZWMoDSMFCxHACTOgnWo2GyH0 dM1is0hurrXgEF1GVCxif2Qd/wZiKv885h1w+T68nfw0OTqevDg+6PTM1XbbKVIx 5JKcS/ExrXcIEeb98Xy60Zao5GUkHeYc4y/rLONPK6LSP+VpdgO23P9K80DMRhpJ 35fYVLZ0oYcb1uHQwMUB2ZnCAw8sFGJwfyTyCBXtPSGIUMPV7D563xSJkBp9Utzr aosrJcnIHqXJvTeiAMe4TrLSDkkj+PuvzWEY8JUrYZwbMNEk6O+JZKCQWTfmqJML DEXW3WFSkUiC2hPq1eH+FL9FrczIVfE0YU274kCgzq0xwAhrtDsre9/duXgaCwfq yOAK+X6IhUeyJKdh7UiQVgboDIj58QJak0TFIf9elleZ/lfwUcAfiSgyRV6tZSKA MX6xsixK1ptLDidJSJU4xDrDgHG7gXSk04y2MFJzlkO7JR7XQCygGSXZShQR0XIE Nbgsj9o3KvJrBGZeq2VHdzYZH0XtYhPxsKLt2vZaAxKhIgpukQ0REbJWl7TZjgsA m6E3Qur7l3BYDoFiTFNa1GLhfAxH3Oip1dG2rHeTmt/RsnXrrTAtAy4426h7Z4l8 6sjDm81utSjCFD6I35Ytxylc3Yoj2MqzsR5fkYa6VAdadmWYAqCUVjYPPLyHDWwj DqbyDba9peuUrq7Vp+H+vsf+9n1bjXe9MI91dpsPKh6GT8MvbAl89umUMjFdi0DV rGswEHzkgncN+PPnjD3rNWNt3IU12lmAtXkX1saWxhLcebjL8ELehQfWXbhI+3T0 JbySEKn6Gsp+qnU/Qp1heipmeGEF42JCPJ+DFcE7FRBU+RDYLaZfwnpELsTOqqEW lYPPBkKuVoG9Y6WTQr5Qbam6mpcAJKtiEF4Xw6hzRY0L0mZ4xWHbcGHFma53kNVR gmtKFfpUTKAuajWmw8paRlE2iKPWeKKn0TcvjhLu8STE0hurZqpIZ3lg16b+jKV9 zY/NCNCxLly9aHpkRgPZGfFvPPREDzlG1u+7DVE2sII+kxy00Rdanp8bF2Ejyyxg zrPYh6E6n/PPyY8dvD/ogPvLdK2NSuxZWYnP+bJKYfy8skK1msmySV10MyJYgCYK VKUtWjKzuLqqC0FUgGCN2M/GlTmHy1aar62GA1SbBjpEVJ+v7j8+050XAJmS0m5X cUaX7zmMoVJpqYrNBthxor2jjTInBV0CfYcVsSTStSODeyWpJueWyD5ccLRm8V6K NWhjcacwWXVJqgCI/NYoyWala66rQnj/Cf7vPBHq7oGMkeRWwLgYIkWBvlDqNzz6 sviZOzd+u/cmTBn0yuMvi2siKyWXelHgTcks0JKZfsnMry8y2JJFyZIdPorjXPMI tEPnBK8mbBXjfseateN6v4ttdlIQJ2W44sqZ46elQrlWdl26RK6c4LuRLV9UeXMV K4shlvqtnACzGCH7ajGqulI/dQj6zKNnFTxPXadLA9o/56CkeV88ZoPjY02EsJz8 Gw9mpYorZKhO9wokr/jsX8Ez3aBmcCt926oYUTubkzCU6SJdm6tUtawetLJDDd5C RXTIm5QF07KINBUPYK2FUU61qLKnJxYi7z4VuXpDRIpXLMoqnzVw+NRb5dlF8dzh jybBpfAfWU/59FXrCto0aUQwvMR6EvLBiB3ohWNGVFc8iji+MU0qyiY6ijbjNlZT tp+KL3BWFVxPcxrHfrTXlFwFaaxs2AHIAfpygsyDb5YsC21mMbC+Xw9OwMqE4ZOC PaYCTBC8JzL3iq6fSaqZ6a0C2B9ZBzMQOGuH7bKOOHYdm7i2Uy9aj/Mk3x9U7Hmd 1ao8nSrLpjLLSt3hdXZXwaFVfgqIb97tY2Wgd/jh7cs2VgJpOJMcNqbGNi5qA6hm CJ38ebf+xgH3kQmD5L3LyugaFCttm1In77ksEsN3Y1AFCP4litikbiivSS/AF9Z0 O2XRQ3ryuNpWAxs1gNETSGJoPPRKTL0jYIgQzq2uyTNQpazZJ/mLWC6OishTnvzr 7PW7t97k5Mj76eD96dG7tygE95xNsisEYs45naIpeI2Gf4tPrlOy6TwIsnh3rAnw f5BdlM//9vta/uiP9f4f/EZNHlz+wXPc/f6fzdHO9si8/2c0wvf/PN3Z+P/3//wV n6Mk5N92mTh3xTqcnlkGlrwo23u//9N+//JU1tysBzdFnqblekavGVJfONN6kQfr dQIGN21QkHnEbzAuzOGLygNHa5ub7TCaTtlqwFZz/Mksqvv9fsNiWhuYkmUbIHZs ONzdeLo7GrJVlL4Wjbe6utqE9czBGm3vwlyEhdPYH5p2NBoNRqMdRg04omh4yuDv 1TZjWVpE38T/wtqsBa3WY9nc/fD26GMPn2YH5UUJJSzKBKuZpOL/hSgC5gYdEETx ipzTfwgKsIc5D/Poih3PkvByuWBYj5rzi1kMUs+/qarLor1iGUPrZWA4hHoWCqtI wcSS5aR6qgw8gaIMMYMMuvJrmmOtZRvLJM8uscqxKNU7NwT1BcMCZUCiCseoEFdC WMSIFd/gXGDhKz55PoP55jRQyKf+LC7XGPtQzACKyiXnqpAy5OCDIAaaRSIPq1TD OfgrEfJw3pbyLV5uta5W9RcLd2X2BZK9sbYx2rBEm34zTTKKWGWg1o4R0NHG7taz 3W0poC0xGEphFec7hXp7OBhtb1tCjQ3PtpVQk2mnSCbIuS/3NsHcYoy3frkoY4bt 5/SyBUSwc+JvJq+OXop0DXTdghTCPxlEIXPw7Q6Ccu8yTa8K8Zqv2isfBviKDyqj mWWWHMOEMNqv+J/lr92AtyZcFXLc7IdO1K0+IBBSkzeN7c0301ECRiKmNJahAckG V7pnBlVz33Gvr50qgSYu8xWa9WQa0eIuYNGFvF6w4wzJ8e17eQkoe26t+c00Ot6y mHFqXpdh7xj4Wzf12pZP7yBQEtjOAnpyWkovfsfgx3DUqChzqEaw3q9wU2+ypxw3 LnGvwuUGUVBd3ykN5FTycFdLhS7COMFHe7qq8VYxwKnPMJtS40zlLSWKN6ZgdSFr buH03QrdTS+/aNIEz4bPBs+UeSPQNkXnh/JNcnDYHrHmt8PJPKAIVgZMVCSoJCBl 93p4/iWyfCEfqlZ6pAMzqUltEDPFAOZtqUSY6sb8lFequfoN43sQ6/7Ck/9t71h7 27aBn71fwbhYkIfaWbJsSwk6YEvc1WgbF3a6oRsGw41dJFn8QJwHgiL/fbzjkTxK lPNojPSDCCSIZYniHY/3vgvNrj5s2G4Z+ISC8D3UJ0JthWp+ACyIrNaTi4V27VC3 lL86h28FRo8+tA8O2/vIM5O6RF3c1DzzaVHHUbFqNPL7nxGnXvKMJY4bsSZPst+g kGp6OSfpIn9RDFT7hSTIX0eY+sJwYxti6P5OmGNAhX6QUgA+1jlCo3z1IydPXxjJ saynmY3jbuqHsmcdaacl4UpiK29XDlegAFk5dN5dbLSCpFm3u1iZGKFcALwEdaKr rwlQ5a4CSBBACcDLX6GXDoOJ+QxhXp5mg/SvXrGjtiCRi0rrdguedFE5dN93aV78 JbGkltSqa2kYB2EttPqabXhELH9b2PRBFE+siyqqY/kAzzZGDRxxw3wqWdXMfsNF N78+B0UChWn2XujQKi+DN/w0EFOlAxqBj2FXPilKM3zgvr21xFxJQF8fLS3gbT5A RmfQHIsv9VaBkZHZpNVCfrG7gH/m/4LtBAEGrNsD8Qv1YZfnC6r/o8W57q08GIHt AqA8zWyBWqmFaxgkPlWp7qeQ6o6Ik3+rPHdXVUZ6MOjWKHS38ZQpUOp+RzmkADDe 4dFSnTQieBwSiKpmW9x9+X6d7Cm1MNYhwdWUOR2Z4FEWGVbjziqiHmPkNkeir4tI 1Kas5FBODWtovhcnX6E/8tvf/mwP+oe/HfpewRpLwGw2Du4Nm+o9I2UA+Ab+wjxU jRz1+Po6VOQNL6gmbp1uU7QLwbe1/qDT3+/0NuibV5L+JpjWYraY+i/rdJTxZH5x g4qEvLAlr2xhwauukaKiSvQ4KC8L2R+SpS5UkE77N3RIYCzhnZ4sJuIa+whCV6Uv Z3hMoThfZfadXLwy0/TGk9mVqsHA84yZGsezsxG2ZgDb1SFAaPUrBkfD6QDnM9PQ vNS19Gg2GdvcPHE2hPY7xBcqnDPsj88KGBxjCDmWQHSLBKGNLkh5RIj/m86uxTUs E2CeCfm26zHlE2LLN3QFSRy/nGKQWJ4OCb9cOvA5zdVwqS8gyeGrj9ESibGeiUtY nDkqJtl06VHKwoqgepj2bbGgY/rTADbGJ/D0Jd5szyPRlE4seV0gts6hcoEYsr5c bLzbFzi9TazkYnNk+UKObRWximxfK4uvRSHDZiBVweAkJR/BtvkjdI8VzpCN63Z2 VAtYZNgUW+v5eLGZ20PC4pLtAz0Fo5vnzAxxVI8nVVYCZbVprcUy37y+lH2x+zqf VsJSE3w8ulAjUStybYEMLdv1+3bQPVW6vZ+B89YYdf0xJJAgA0TOvEHeW+Ox2yTf MTmzgWXiZPIJVTSPZWqL+QxTopjtqkRKgOx94+pkqI1F01h9E+fwqsRpGqRhyPwW hQbyT0LkbWQpss98VOO001R2Mmj20JoK1H+Ahqzm+UAq73JuNIOo+xr1IJcX6FFs aQXvwb6JcD1vv6NFZ+wOyqGS308wH0Ve0rUe7DYp5qcg53ud/l63zyZ2xbPQLBp9 vLWoIa2GesI8Fs+BNMfdMKeD/EyY9Jx3fVgzTOQxWPcRbhomQRrVmeu9FsO2NCJj zBHECyl2RwMOMvW/xOsqk93QgyOJNVJdj67BNXtOm6dUWK43ON9IQRixc0crBS/M LQlzwmEOa60gDKMWs74rVc1PTGhqiJElbMthND6sJqwqw9pVBViowQkueDo/O97m LFPWj1nGvEamEm+5TKapK5KXpvVvWyidF+Whrep7mWKp1+lXfx7T7FoTS+ZRB+zl 7a4fDnjmZRZ0SOlYBiHsrv0PCdZPQGlJ8K06KLadofFN5gvBkbnoInAvxab1IE0T TrF1oNiY+YvQQ6diGvwA6kbKqq0/Ol0upycXN6RtX/HOsmqNOoTma6V5tan8Rcws hY8KUDfv/arA+xbWasBgwhqHBuJ9YYt7v1j84Gg2v6ETZt8LqsCVgI/Dk+kCPZJ6 4frYVTI+J7JnvCxJ23uuZ5yUm4pVb5iup0TPchVLR4VUIiFN4rguHPJCBQez2Bzj y641YyurVFBSk8S1NP9MX1n1dZFAheWpv17TnZVvVfhfSPKn8+Hj4G23++5Weyw4 p3awote+bqbL+jO0dkdSddcY1RYXBBI0gcpZxwQHnThyJkyGR1IqzxbHuNm/KBeu JKz+fDweXc53xHgo9UPDvaEvnsBiQOxCNkL9zk+ZcRwHYdyoaTUOimrvDBSIHrT2 ax/stcVe99PBoaJCq8wwU2FN5MMtioq0JuCoLqB7oEdZKyNOOuAlFqa7V5U3P2yC OGuG5jj9IHBwTSujZt0DMu+ONaJ6IH9xzthMmkHYqjFZbg28CbERcpGhY7pyBJ3h NNHv5NyTCCoQuCJew0Iyp9kykHsKIEecoj8GMlLUrMZb6HekVMyais15NZE80bzR KPN3sikcfpR746TQMDdT5AzzzMnHm+gb653UcpTstx2m1LmV3xx3QujoYXXfuo+O 5Y+UZzrdWFWQbQDNqbSRn0ebVb8uGIIyGKYJi+nlyAUlK9UCrDm0YayCwn/II6yr 0jUN1Fc5CwBIOAobrSAKm/UlJLx8TcxF8JSL8yMwbQQS36FFINnqFU+80BMwNP2k WbSQr5OFQblSYDWtovihVmDuG0Ek3Ldiifuk5ipWTwyO/GUk4urh8m2b3DEJp9HC tMqVc9oJxhOM10xhZY2YA5cFNo+/KDBK4UdSI924o60YcVUdtTFJXW5MWnN1xFUu OBBmk9bcTXI27xEA+XckqckdSd0dcTyrD4nDP+BYeakvS3x5A/eRMXFfwNlUcRAD jJoSFVHL3evVooJHyFePkHw8/g60eCkmihKJprqVXbp6lKNEWLva8eRilqSyYM3y 87jjiLO1qQWEdDeK8ukH+njHNQlLHNoclGeDZVlqyj0h8u5WI5Uct1lj5zvTfdPh XcyJtOb4w/OkZWExZ+22SHVHXLdAxrW4jFvJShgiC9fjxVQrasr11UOr1UdpTV5K Q+uVVP+I6QKi6JBvujf4/VPn/WHnQOqh5m9VZiNMkc3l+XjX8+zHz4M3ve7fbXjW /H3/Z/e6+9Ju6nc/9fbaagZ+xTvPdm4ebX8w+7tgBWjP0WfUmXe9+eejs1fHz5mB Du8vzEEPEycFPUwEW/RWNgcdrlaipngz/kJp6OFOHO5IvqfT0OV8uSx0fOqBeeiS xiJzPD+++2Ow3+m19w67vc9oeRjCwk+GVDCqkNl2cOR8GB6J2fTsRlnSt7v6DGQU 7m9ktwaRVfi/691B9uXyFk1Uyxfy3IVC5ShHOcpRjnKUoxzlKEc5ylGOcpSjHOUo RznKUY5ylKMc5ShHOcpRjnKU44cd/wOYRuFDAKAAAA== --7584767-3623878656-3248300694=:9002-- From just@letterror.com Sun Dec 8 01:00:35 2002 From: just@letterror.com (Just van Rossum) Date: Sun, 8 Dec 2002 02:00:35 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: Message-ID: --7641475-0-3248301639=:9003 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit Just van Rossum wrote: > - zip file names PYTHONPATH work Doop, I forgot to include a site.py patch: unpatched it filters out non-directories... (site.py's sys.path filtering needs to be changed more thoroughly than this, but the attached patch is enough for zip files on PYTHONPATH to work.) Just --7641475-0-3248301639=:9003 Content-Type: application/octet-stream; Name="site.py.patch"; X-Mac-Type="00000000"; X-Mac-Creator="00000000" Content-Transfer-Encoding: base64 Content-Disposition: attachment; Filename="site.py.patch" SW5kZXg6IExpYi9zaXRlLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KUkNTIGZpbGU6 IC9jdnNyb290L3B5dGhvbi9weXRob24vZGlzdC9zcmMvTGliL3NpdGUucHksdgpy ZXRyaWV2aW5nIHJldmlzaW9uIDEuNDYKZGlmZiAtYyAtcjEuNDYgc2l0ZS5weQoq KiogTGliL3NpdGUucHkJOSBOb3YgMjAwMiAwNTowODowNiAtMDAwMAkxLjQ2Ci0t LSBMaWIvc2l0ZS5weQk4IERlYyAyMDAyIDAwOjU3OjIzIC0wMDAwCioqKioqKioq KioqKioqKgoqKiogNzUsODggKioqKgogIF9kaXJzX2luX3N5c19wYXRoID0ge30K ICBmb3IgZGlyIGluIHN5cy5wYXRoOgogICAgICAjIEZpbHRlciBvdXQgcGF0aHMg dGhhdCBkb24ndCBleGlzdCwgYnV0IGxlYXZlIGluIHRoZSBlbXB0eSBzdHJpbmcK ISAgICAgIyBzaW5jZSBpdCdzIGEgc3BlY2lhbCBjYXNlLiBXZSBhbHNvIG5lZWQg dG8gc3BlY2lhbC1jYXNlIHRoZSBNYWMsCiEgICAgICMgYXMgZmlsZSBuYW1lcyBh cmUgYWxsb3dlZCBvbiBzeXMucGF0aCB0aGVyZS4KISAgICAgaWYgc3lzLnBsYXRm b3JtICE9ICdtYWMnOgohICAgICAgICAgaWYgZGlyIGFuZCBub3Qgb3MucGF0aC5p c2RpcihkaXIpOgohICAgICAgICAgICAgIGNvbnRpbnVlCiEgICAgIGVsc2U6CiEg ICAgICAgICBpZiBkaXIgYW5kIG5vdCBvcy5wYXRoLmV4aXN0cyhkaXIpOgohICAg ICAgICAgICAgIGNvbnRpbnVlCiAgICAgIGRpciwgZGlyY2FzZSA9IG1ha2VwYXRo KGRpcikKICAgICAgaWYgbm90IGRpcmNhc2UgaW4gX2RpcnNfaW5fc3lzX3BhdGg6 CiAgICAgICAgICBMLmFwcGVuZChkaXIpCi0tLSA3NSw4NSAtLS0tCiAgX2RpcnNf aW5fc3lzX3BhdGggPSB7fQogIGZvciBkaXIgaW4gc3lzLnBhdGg6CiAgICAgICMg RmlsdGVyIG91dCBwYXRocyB0aGF0IGRvbid0IGV4aXN0LCBidXQgbGVhdmUgaW4g dGhlIGVtcHR5IHN0cmluZwohICAgICAjIHNpbmNlIGl0J3MgYSBzcGVjaWFsIGNh c2UuIEZpbGVuYW1lcyBhcmUgYWxsb3dlZCBvbiBzeXMucGF0aC4KISAgICAgIyBY WFggd2l0aCBpbXBvcnQgaG9va3MgaXQncyBkdWJpb3VzIHRoZSBldmVuIGZpbHRl ciBvdXQgcGF0aCBpdGVtcwohICAgICAjIHRoYXQgZG9uJ3QgZXhpc3QgaW4gdGhl IGZpbGUgc3lzdGVtLgohICAgICBpZiBkaXIgYW5kIG5vdCBvcy5wYXRoLmV4aXN0 cyhkaXIpOgohICAgICAgICAgY29udGludWUKICAgICAgZGlyLCBkaXJjYXNlID0g bWFrZXBhdGgoZGlyKQogICAgICBpZiBub3QgZGlyY2FzZSBpbiBfZGlyc19pbl9z eXNfcGF0aDoKICAgICAgICAgIEwuYXBwZW5kKGRpcikK --7641475-0-3248301639=:9003-- From barry@python.org Sun Dec 8 02:33:23 2002 From: barry@python.org (Barry A. Warsaw) Date: Sat, 7 Dec 2002 21:33:23 -0500 Subject: [Python-Dev] Import compromise References: <20021207202128.GA28924@panix.com> <02f901c29e4b$5561be00$ced241d5@hagrid> Message-ID: <15858.44915.895749.878026@gargle.gargle.HOWL> >>>>> "FL" == Fredrik Lundh writes: FL> maybe you could team up with Ping and turn his zest tool FL> into a brettbot? FL> http://www.sims.berkeley.edu/~ping/vertex-2002/zest.html Zest rocks. Has anything more happened with it in the last 8 months or so? -Barry From brett@python.org Sun Dec 8 03:43:11 2002 From: brett@python.org (Brett Cannon) Date: Sat, 7 Dec 2002 19:43:11 -0800 (PST) Subject: [Python-Dev] dummy_thread (#622537), part 2 Message-ID: Wrote ``dummy_threading``. Thanks to the recent importing discussion and one of Martin's replies to my previous announcement for this patch I figured out how to do it while still using ``threading``. ``sys.modules`` is your friend (as long as you treat it nicely). So the patch is ready for review again. Tim mentioned he was going to have a look, but I will let him say whether he is going to still look at it now. All relevent info is in the patch. -Brett From skip@manatee.mojam.com Sun Dec 8 13:00:17 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 8 Dec 2002 07:00:17 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200212081300.gB8D0HfZ011168@manatee.mojam.com> Bug/Patch Summary ----------------- 316 open / 3105 total bugs (+2) 94 open / 1822 total patches (-2) New Bugs -------- ext module generation problem (2002-08-23) http://python.org/sf/599248 Tkinter: BitmapImage vanishes if not stored in non-local var (2002-11-01) http://python.org/sf/632323 test_poll fails on FreeBSD (2002-12-01) http://python.org/sf/646547 netrc & special chars in passwords (2002-12-01) http://python.org/sf/646592 Can't run script from IDLE (2002-12-01) http://python.org/sf/646730 more email ASCII decoding errors (2002-12-03) http://python.org/sf/648119 xmlrpc can't do proxied HTTP (2002-12-04) http://python.org/sf/648658 nturl2path.url2pathname() mishandles /// (2002-12-07) http://python.org/sf/649961 All URL funcs mishandle file://localhost (2002-12-07) http://python.org/sf/649962 urllib.url2pathname('file://host/')... (2002-12-07) http://python.org/sf/649974 New Patches ----------- asyncore loop_once and remove poll2 (2002-12-04) http://python.org/sf/648322 tuple arg for issubclass (2002-12-06) http://python.org/sf/649608 urllib2.Request's headers are case-sens. (2002-12-06) http://python.org/sf/649742 Fix: asynchat.py: endless loop (2002-12-06) http://python.org/sf/649762 Complementary patch for OpenVMS (2002-12-07) http://python.org/sf/649997 posixfy some things (2002-12-08) http://python.org/sf/650412 avoid redefinition of pthread functions (2002-12-08) http://python.org/sf/650415 Closed Bugs ----------- Removing _tkinter considered harmful (2002-03-25) http://python.org/sf/534748 LINKCC incorrectly set (2002-06-16) http://python.org/sf/569668 list slice ass ignores subtypes of list (2002-11-04) http://python.org/sf/633152 --with-dl-dld section of the readme (2002-11-10) http://python.org/sf/636313 __all__ not in RefMan index (2002-11-24) http://python.org/sf/643227 typo in Object/abstract.c : ternary_op (2002-11-24) http://python.org/sf/643260 old UnicodeData.txt (2002-12-01) http://python.org/sf/646408 Closed Patches -------------- Start of a bgen tutorial (2001-09-04) http://python.org/sf/458383 use readline in pydoc.help if available (2002-04-28) http://python.org/sf/549901 Regression exceptions for cygwin (2002-05-03) http://python.org/sf/551977 PyBufferObject fixes (2002-05-04) http://python.org/sf/552438 db4 include not found (2002-08-02) http://python.org/sf/590377 OpenVMS patches (2002-09-24) http://python.org/sf/614055 make some type attrs writable (2002-11-09) http://python.org/sf/635933 Allow device name in linuxaudiodev.open (2002-11-27) http://python.org/sf/644977 Give some clue why modules fail... (2002-11-28) http://python.org/sf/645383 From ping@zesty.ca Sun Dec 8 18:21:25 2002 From: ping@zesty.ca (Ka-Ping Yee) Date: Sun, 8 Dec 2002 12:21:25 -0600 (CST) Subject: [Python-Dev] Import compromise In-Reply-To: <15858.44915.895749.878026@gargle.gargle.HOWL> Message-ID: On Sat, 7 Dec 2002, Barry A. Warsaw wrote: > Zest rocks. Has anything more happened with it in the last 8 months > or so? I brought a more recent revision to CSCW 2002 and demonstrated it there. It's not what i would call production code, though. The most significant recent change is that little speech-bubble icons now appear in the message body at the ends of paragraphs that were subsequently quoted and replied to, so you can navigate forwards from messages to their replies. I'm sorry, i wish it were moving along faster too. Maybe after the next clump of deadlines passes (CHI demo Dec 16, CHI short Jan 3, CHI long Jan 10, Usenix Security Jan 27)... -- ?!ng From lists@morpheus.demon.co.uk Sun Dec 8 17:34:15 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sun, 08 Dec 2002 17:34:15 +0000 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: (Just van Rossum's message of "Sun, 8 Dec 2002 01:44:47 +0100") References: Message-ID: Just van Rossum writes: > I'll write a bit about the mechanics tomorrow, but in the meantime I would like > to see some test results/bug reports/bug fixes from Windows developers. Please > don't start with the full monty (ie. the std lib), try a small test archive > first please ;-) The patch applies fine to my copy of Python (downloaded from CVS Dec 1st - I can't get cvs update to cooperate for me :-() with an offset of -3 in most of the import.c changes (I assume that's just because I'm a bit out of date). To get zipimport as a builtin, apply the following patch, and add zipimport.c to the pythoncore project. --- PC\config.c.orig Sun Jul 07 03:59:34 2002 +++ PC\config.c Sun Dec 08 15:37:03 2002 @@ -43,6 +43,7 @@ extern void init_weakref(void); extern void init_hotshot(void); extern void initxxsubtype(void); +extern void initzipimport(void); /* XXX tim: what's the purpose of ADDMODULE MARKER? */ /* -- ADDMODULE MARKER 1 -- */ @@ -96,6 +97,7 @@ {"_hotshot", init_hotshot}, {"xxsubtype", initxxsubtype}, + {"zipimport", initzipimport}, /* XXX tim: what's the purpose of ADDMODULE MARKER? */ /* -- ADDMODULE MARKER 2 -- */ Zipimport doesn't work at all for me as a dynamic extension. Is this just a temporary problem, an issue specific to the zipimport module, or a limitation in the hook mechanism? (I assume not the last, as that would make it pretty useless as a general mechanism...) As a builtin, it works fine in a release build. But a debug build asserts in fopen() as the filename is "". An easy fix for this is to add a check in zipimporter_init(), if (lenpath == 0) { PyErr_SetString(PyExc_ImportError, "archive path is empty"); return -1; } Do we need to set an error string there? It's not really an error, as such. BTW, I don't see any code for checking timestamps of .pyc files against .py files. Did you omit that (and if so, was it deliberate or accidental)? I hit an odd case when I was testing interactively. I set sys.path to a list containing just 'ziptest.zip', and then tried to import a module. Zipimport failed because zlib wasn't accessible. That's OK, but when I reset sys.path, zipimport *still* claimed it couldn't decompress as zlib wasn't available. I see why - the optimisation in get_decompress_func() only checks for zlib once - but nevertheless it is annoying that I can only "fix" my mistake by restarting the interpreter. This is never going to happen "in real life", so it's not crucial. But maybe reload(zipimport) could be made to retry the check? I also tried a Python import hook. It seemed to be doing the right things. I didn't try too hard - just dummy {find,load}_module methods, and an __init__. It looked like errors were generated as a result of me not implementing the protocol right, which is good :-) Paul. PS The code *still* looks messy. I think that's just C though - I've got so used to writing Python code that *any* C looks messy :-) -- This signature intentionally left blank From just@letterror.com Sun Dec 8 20:24:50 2002 From: just@letterror.com (Just van Rossum) Date: Sun, 8 Dec 2002 21:24:50 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: Message-ID: Paul Moore wrote: > To get zipimport as a builtin, apply the following patch, and add > zipimport.c to the pythoncore project. Ah, thanks, will add that to the next version. > Zipimport doesn't work at all for me as a dynamic extension. Is this > just a temporary problem, an issue specific to the zipimport module, > or a limitation in the hook mechanism? (I assume not the last, as that > would make it pretty useless as a general mechanism...) I don't know, as you don't give specifics about what error you get, if any. Maybe -v tells us more. (Although zipimport.c can sure use better -v reporting.) > As a builtin, it works fine in a release build. But a debug build > asserts in fopen() as the filename is "". > > An easy fix for this is to add a check in zipimporter_init(), Thanks, done. > Do we need to set an error string there? It's not really an error, as > such. I'm not sure. I'd like to make it easy to write hooks, and it makes most sense to use classes as import_hooks, and since an __init__ can't return anything, an exception is the best you can get. It's _possible_ to write a __new__ method that returns None when it can't handle the path, but that raises the bar somewhat for writing hooks. Lots of stuff in import.c raises ImportError when it really means: "I can't find what you're asking for". (Btw. I intend to add a ZipImportError exception to the zipimport module, which would be a subclass of ImportError.) > BTW, I don't see any code for checking timestamps of .pyc files > against .py files. Did you omit that (and if so, was it deliberate or > accidental)? I copied that verbatim from the PEP 273 patch ;-) I'm sure this could be fixed, but there are more important issues at the moment. > I hit an odd case when I was testing interactively. I set sys.path to > a list containing just 'ziptest.zip', and then tried to import a > module. Zipimport failed because zlib wasn't accessible. That's OK, > but when I reset sys.path, zipimport *still* claimed it couldn't > decompress as zlib wasn't available. I see why - the optimisation in > get_decompress_func() only checks for zlib once - but nevertheless it > is annoying that I can only "fix" my mistake by restarting the > interpreter. This is never going to happen "in real life", so it's not > crucial. But maybe reload(zipimport) could be made to retry the check? Right, but maybe I simply shouldn't remember that a previous zlib import failed, and just try again. > I also tried a Python import hook. It seemed to be doing the right > things. I didn't try too hard - just dummy {find,load}_module methods, > and an __init__. It looked like errors were generated as a result of > me not implementing the protocol right, which is good :-) > > Paul. > > PS The code *still* looks messy. I think that's just C though - I've > got so used to writing Python code that *any* C looks messy :-) import.c, zipimport.c or both? ;-) Yeah, it needs work, but you're also right about C: there's a lot of string manipulation going on which is a huge pain in the butt. But: I admit not very _good_ at doing string manipulations in C, so once it's more or less bugfree I'm sure more experienced people can improve it if needed. Just From lists@morpheus.demon.co.uk Sun Dec 8 22:38:37 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sun, 08 Dec 2002 22:38:37 +0000 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: (Just van Rossum's message of "Sun, 8 Dec 2002 21:24:50 +0100") References: Message-ID: Just van Rossum writes: > Paul Moore wrote: >> Zipimport doesn't work at all for me as a dynamic extension. Is >> this just a temporary problem, an issue specific to the zipimport >> module, or a limitation in the hook mechanism? (I assume not the >> last, as that would make it pretty useless as a general >> mechanism...) > > I don't know, as you don't give specifics about what error you get, > if any. Maybe -v tells us more. (Although zipimport.c can sure use > better -v reporting.) Sorry. Basically it just didn't find the module in the zipfile. I don't think it's a big deal. If you want, I can rebuild as a dynamic module and try with -v. >> Do we need to set an error string there? It's not really an error, >> as such. > > I'm not sure. I'd like to make it easy to write hooks, and it makes > most sense to use classes as import_hooks, and since an __init__ > can't return anything, an exception is the best you can get. Ah. Now I see the logic (actually, it's the same as iu.py). I'd say stick with the exception method. What I was really thinking was in the C code, could I just return -1 without setting the exception. But that's not good practice, even if it did work. > It's _possible_ to write a __new__ method that returns None when it > can't handle the path, but that raises the bar somewhat for writing > hooks. Too right. Leave it as it is. >> BTW, I don't see any code for checking timestamps of .pyc files >> against .py files. Did you omit that (and if so, was it deliberate >> or accidental)? > > I copied that verbatim from the PEP 273 patch ;-) I'm sure this > could be fixed, but there are more important issues at the moment. It's a fair cop guvnor :-) To be honest, I think that this is one of the nastiest hurdles to writing a good import hook - replicating (or even understanding the logic for!) the hairy extension and timestamp checking code used for normal imports. Refactoring and encapsulating that would potentially help in a lot of cases. Except that maybe it's not always exactly the same (you can't write a compiled .pyc file back to a zip file, you can't load a C extension from a zipfile, ...) But that isn't an issue for this patch. >> I hit an odd case when I was testing interactively. I set sys.path to >> a list containing just 'ziptest.zip', and then tried to import a >> module. Zipimport failed because zlib wasn't accessible. That's OK, >> but when I reset sys.path, zipimport *still* claimed it couldn't >> decompress as zlib wasn't available. I see why - the optimisation in >> get_decompress_func() only checks for zlib once - but nevertheless it >> is annoying that I can only "fix" my mistake by restarting the >> interpreter. This is never going to happen "in real life", so it's not >> crucial. But maybe reload(zipimport) could be made to retry the check? > > Right, but maybe I simply shouldn't remember that a previous zlib > import failed, and just try again. I thought of that - it slows things down when zlib isn't available, as it checks (and fails) every time. How bad this is, I'm not sure... >> PS The code *still* looks messy. I think that's just C though - I've >> got so used to writing Python code that *any* C looks messy :-) > > import.c, zipimport.c or both? ;-) import.c hasn't got much better - but at least it hasn't got any worse :-) > Yeah, it needs work, but you're also right about C: there's a lot of > string manipulation going on which is a huge pain in the butt. But: > I admit not very _good_ at doing string manipulations in C, so once > it's more or less bugfree I'm sure more experienced people can > improve it if needed. It wasn't a criticism. Every time I look at C code which manipulates strings (you're right, that's the bulk of the nastiness) it reminds me how *nice* virtually any other language is. But you let me write import hooks in Python and avoid C, so you're my friend for life :-) Paul. -- This signature intentionally left blank From fredrik@pythonware.com Sun Dec 8 22:53:28 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sun, 8 Dec 2002 23:53:28 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) References: Message-ID: <000b01c29f0c$a192eda0$ced241d5@hagrid> paul wrote: > To be honest, I think that this is one of the nastiest hurdles to > writing a good import hook - replicating (or even understanding the > logic for!) the hairy extension and timestamp checking code used for > normal imports. Refactoring and encapsulating that would potentially > help in a lot of cases. Except that maybe it's not always exactly the > same (you can't write a compiled .pyc file back to a zip file, you > can't load a C extension from a zipfile, ...) But that isn't an issue > for this patch. do you have real use cases for this? I would have thought that the vast majority of users of a feature like this would use it to "freeze" a collection of modules for use with their application. emulating a file system when you already have a file system seems a bit like overkill... From just@letterror.com Sun Dec 8 22:59:37 2002 From: just@letterror.com (Just van Rossum) Date: Sun, 8 Dec 2002 23:59:37 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: Message-ID: Paul Moore wrote: > Sorry. Basically it just didn't find the module in the zipfile. I > don't think it's a big deal. If you want, I can rebuild as a dynamic > module and try with -v. Don't worry, I've found (and solved) the problem and am playing with a dynamic build as we speak. [time stamp checking] > To be honest, I think that this is one of the nastiest hurdles to > writing a good import hook - replicating (or even understanding the > logic for!) the hairy extension and timestamp checking code used for > normal imports. Not all import hooks will need to support all of .pyc, .pyc and .py. I for one would've been more than happy for zipimport to only support .pyc ;-) > Refactoring and encapsulating that would potentially > help in a lot of cases. Except that maybe it's not always exactly the > same (you can't write a compiled .pyc file back to a zip file, you > can't load a C extension from a zipfile, ...) But that isn't an issue > for this patch. Right; I need to peek in the zipfile.py module to see how the time/date shorts from the zip archive translate to a usable timestamp. I've updated my code so it at least doesn't throw away those values ;-) > > Right, but maybe I simply shouldn't remember that a previous zlib > > import failed, and just try again. > > I thought of that - it slows things down when zlib isn't available, as > it checks (and fails) every time. How bad this is, I'm not sure... That was my original rationale, too, but it's wrong: it only checks if compression is *needed* and *if* zlib is not available then, the program will abort with a traceback anyway... I've changed it in my local copy. [ ... ] > But you let me write import hooks in Python and avoid C, so you're my > friend for life :-) Heh. Make sure to buy me a beer some time -- and I'll be sure to buy you one right back ;-) Just From just@letterror.com Sun Dec 8 23:04:16 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 00:04:16 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: <000b01c29f0c$a192eda0$ced241d5@hagrid> Message-ID: Fredrik Lundh wrote: > emulating a file system when you already have a file system seems a > bit like overkill... True, but zipimport (just like the PEP 273 implementation) *does* handle any of .py, .pyc and .pyo. It prefers byte code over source, but ignores the time stamp. That *could* be confusing. Just From gward@python.net Sun Dec 8 23:09:41 2002 From: gward@python.net (Greg Ward) Date: Sun, 8 Dec 2002 18:09:41 -0500 Subject: [Python-Dev] __getstate__() not inherited when __slots__ present Message-ID: <20021208230941.GA12533@cthulhu.gerg.ca> If a class and its superclass both define __slots__, it appears that __getstate__() is not inherited from the superclass. Example: """ class Base (object): __slots__ = [] def __getstate__ (self): return tuple([getattr(self, attr) for attr in self.__slots__]) class Thing (Base): __slots__ = ['a', 'b'] def __init__ (self): self.a = 42 self.b = "boo!" abstract = Base() thing = Thing() print abstract.__getstate__ print thing.__getstate__ """ When I run this with a not-quite-current CVS Python: > The upshot of this is that I can't just define one __getstate__() in the superclass of a bunch of __slots__-using classes -- I guess I'll have to set __getstate__ manually for each class. ;-( If this is a feature, is it documented anywhere? (BTW, I see the same behaviour with Python 2.2.2.) Greg -- Greg Ward http://www.gerg.ca/ "Very funny, Scotty. Now beam my *clothes* down." From skip@pobox.com Sun Dec 8 23:53:24 2002 From: skip@pobox.com (Skip Montanaro) Date: Sun, 8 Dec 2002 17:53:24 -0600 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: References: <000b01c29f0c$a192eda0$ced241d5@hagrid> Message-ID: <15859.56180.914141.702803@montanaro.dyndns.org> Just> It prefers byte code over source, but ignores the time stamp. That Just> *could* be confusing. Not to mention a change of semantics. If the .py file is newer than the .pyc or .pyo files, why not just force compilation but don't save the result (maybe also toss a Warning)? Isn't that what importing from a read-only directory or filesystem does today? Sure, it would slow things down, but it would better match the semantics of importing from the filesystem. Presumably testing before delivery of the zip archive would identify any time order problems. Skip From barry@python.org Mon Dec 9 04:28:24 2002 From: barry@python.org (Barry A. Warsaw) Date: Sun, 8 Dec 2002 23:28:24 -0500 Subject: [Python-Dev] Import compromise References: <15858.44915.895749.878026@gargle.gargle.HOWL> Message-ID: <15860.7144.998931.300423@gargle.gargle.HOWL> >>>>> "KY" == Ka-Ping Yee writes: KY> I'm sorry, i wish it were moving along faster too. Maybe KY> after the next clump of deadlines passes (CHI demo Dec 16, CHI KY> short Jan 3, CHI long Jan 10, Usenix Security Jan 27)... I can sympathize! Zest is one of those Cool Projects to keep an eye on. -Barry From just@letterror.com Mon Dec 9 08:54:45 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 09:54:45 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: <15859.56180.914141.702803@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > Just> It prefers byte code over source, but ignores the time stamp. That > Just> *could* be confusing. > > Not to mention a change of semantics. Hm, questionable, as we don't currently *have* zip import ;-) > If the .py file is newer than the > ..pyc or .pyo files, why not just force compilation but don't save the result > (maybe also toss a Warning)? Isn't that what importing from a read-only > directory or filesystem does today? Yes, this is what it should do. However, I'd be *very* grateful for a recipe that translates a dos time/date (it's two shorts, zipfile.py contains code that translates it to a time tuple) to something compatible with st.st_mtime. And it can't use the time module ;-) > Sure, it would slow things down, but it > would better match the semantics of importing from the filesystem. > Presumably testing before delivery of the zip archive would identify any > time order problems. Agreed. Just From mal@lemburg.com Mon Dec 9 10:15:38 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 09 Dec 2002 11:15:38 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding References: <20021206025220.E17658@vestdata.no> Message-ID: <3DF46D4A.2070902@lemburg.com> Ragnar Kj=F8rstad wrote: > Hi >=20 > The dumps-method in xmlrpclib has the following comment: > All 8-bit strings in the data structure are assumed to use the > packet encoding. Unicode strings are automatically converted, > where necessary. >=20 >=20 > This doesn't work very well. In our particular case we're using latin_1 > as our default encoding, and we're using UTF-8 for the packet encoding. > We can't really change the default encoding, because the sql-modules > transfer latin_1 encoded data and we can't change the packet encoding t= o > latin_1 because the xmlrpc-client (php) doesn't work with that. > > The attached patch changes xmlrpclib to convert strings to unicode usin= g > the default encoding, and then convert them back to strings with the > packet encoding. If unicode is not available it falls back to the old > behaviour. I believe this is overkill. If you need this behaviour, subclass the Marshaller in xmlrpclib and add your feature to that subclass. Then replace the Marshaller class in xmlrpclib with your subclass. Aside: xmlrpclib should support subclassing the Marshaller and Unmarshaller more transparently. Currently, the two are hard-coded into the rest of xmlrpclib without the possibility to provide your own subclasses without tweaking xmlrpclib from the outside. > I guess for performance it could check if the defaultencoding is the > same as the packet-encoding, but my guess is that it hardly ever is, so > no reason to optimize for it. >=20 > Note; I'm not at all sure this is the best way to fix the problem. If > it's not, please feel free to ignore this patch, or even better - tell > me what the preferable fix is. Please post patches using the SourceForge patch manager. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mchermside@ingdirect.com Mon Dec 9 13:33:02 2002 From: mchermside@ingdirect.com (Chermside, Michael) Date: Mon, 9 Dec 2002 08:33:02 -0500 Subject: [Python-Dev] Re: Another approach for the import mechanism Message-ID: <902A1E710FEAB740966EC991C3A38A8903C2791A@INGDEXCHANGEC1.ingdirect.com> Gustavo Niemeyer writes: > - Don't have to change path to use compressed packages (at least > not if you want to provide compressed packages, individual > compressed modules or the standard library). >=20 > - Don't have to specify the compression type hardcoded. >=20 > - Allows one to ship a package inside a zip file, without asking > the user to change his path, and without hacking the package. >=20 > - Allows one to compress a single file (foobar.py.bz2). >=20 > I belive that my propose is quite clear now. If there are no > additional supporters, there's no reason to go on. I think this proposal would make sense IF compression were an important goal here. But to me, it isn't. Zip does two things... it aggregates into a single file (maintaining directory structure) and it compresses. Of the two, I find the aggregation important and the compression a mere side effect. This is why I really don't care much about switching to a different compression (or aggregation) format -- one standard way to do it is more useful to me than a BETTER way. It is why I don't care about compressing a single file. Basically, disk space is cheap, but effort to keep track of (and distribute) complex file hierarchies isn't. > Thanks to everyone who discussed. And thanks for your contributions too. I don't realize exactly what I was looking for from .zip until you explained so clearly the benefits of your proposal. -- Michael Chermside From bkline@rksystems.com Mon Dec 9 14:33:56 2002 From: bkline@rksystems.com (Bob Kline) Date: Mon, 9 Dec 2002 09:33:56 -0500 (EST) Subject: [Python-Dev] Re: Another approach for the import mechanism In-Reply-To: <902A1E710FEAB740966EC991C3A38A8903C2791A@INGDEXCHANGEC1.ingdirect.com> Message-ID: On Mon, 9 Dec 2002, Chermside, Michael wrote: > I think this proposal would make sense IF compression were an > important goal here. But to me, it isn't. Zip does two things... it > aggregates into a single file (maintaining directory structure) and > it compresses. Of the two, I find the aggregation important and the > compression a mere side effect. Our *three* chief weapons are .... Oops, wrong show. Zip (and tar + bzip2/gzip) provide at least three services: 1. aggregation 2. compression 3. integrity validation (CRC, etc.) I'm leaving off encryption, so someone else can continue with the SI gag. Some do a better job than others at letting you get to the rest of the archive if one spot is corrupted. > Basically, disk space is cheap ... True in many contexts, but not all. And the differences in compression ratios between zip and bzipped tar can be surprisingly significant. -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com From fredrik@pythonware.com Mon Dec 9 14:29:12 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 9 Dec 2002 15:29:12 +0100 Subject: [Python-Dev] Re: Another approach for the import mechanism References: Message-ID: <012b01c29f8f$58ed6d90$0900a8c0@spiff> Bob Kline wrote: > > Basically, disk space is cheap ... >=20 > True in many contexts, but not all. And the differences in = compression > ratios between zip and bzipped tar can be surprisingly significant. tried doing random access in a bzipped tar file? From skip@pobox.com Mon Dec 9 14:36:01 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 9 Dec 2002 08:36:01 -0600 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: References: <15859.56180.914141.702803@montanaro.dyndns.org> Message-ID: <15860.43601.620525.747883@montanaro.dyndns.org> Just> However, I'd be *very* grateful for a recipe that translates a dos Just> time/date (it's two shorts, zipfile.py contains code that Just> translates it to a time tuple) to something compatible with Just> st.st_mtime. And it can't use the time module ;-) Looking at the code in zipfile.py which converts a time tuple to DOS date and time shorts (I didn't see code that went the other way), it appears that dosdate is 7 bits for the year, 4 bits for the month and 5 bits for the day. Dostime looks like 5 bits for the hour, 6 bits for the minute and 5 bits for half the number of seconds. You should be able to extract those chunks into a tm structure then call the C lib mktime() function to convert to a time in seconds since the epoch (this is just off the top of my head): stm = malloc(sizeof(struct tm)); stm->tm_sec = (dostime & 0x1f) * 2; stm->tm_min = (dostime >> 5) & 0x3f; stm->tm_hour = dostime >> 11); stm->tm_mday = dosdate & 0x1f; stm->tm_mon = (dosdate >> 5) & 0xf; stm->tm_year = dosdate >> 9; mtime = mktime(stm); Here's the layout of dosdate and dostime as I understand it from reading the zipfile code: dosdate = yyyyyyymmmmddddd dostime = hhhhhmmmmmmsssss where "sssss" is actually the seconds divided by 2. I'll let others check my code. I have to dash off to a class. Skip From just@letterror.com Mon Dec 9 14:35:50 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 15:35:50 +0100 Subject: [Python-Dev] Re: Another approach for the import mechanism In-Reply-To: Message-ID: Bob Kline wrote: > 3. integrity validation (CRC, etc.) This is another thing that zipimport doesn't do (nor does the PEP 273 implementation btw.). It could in theory be added but it doesn't mean much: someone can still put bogus data in a zip file and make marshal crash. So I'm tempted to blame marshal and forget about CRC checking... Just From just@letterror.com Mon Dec 9 14:56:03 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 15:56:03 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: <15860.43601.620525.747883@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > Looking at the code in zipfile.py which converts a time tuple to DOS date > and time shorts (I didn't see code that went the other way), it appears that > dosdate is 7 bits for the year, 4 bits for the month and 5 bits for the day. > Dostime looks like 5 bits for the hour, 6 bits for the minute and 5 bits for > half the number of seconds. You should be able to extract those chunks into > a tm structure then call the C lib mktime() function to convert to a time in > seconds since the epoch (this is just off the top of my head): > > stm = malloc(sizeof(struct tm)); > stm->tm_sec = (dostime & 0x1f) * 2; > stm->tm_min = (dostime >> 5) & 0x3f; > stm->tm_hour = dostime >> 11); > stm->tm_mday = dosdate & 0x1f; > stm->tm_mon = (dosdate >> 5) & 0xf; > stm->tm_year = dosdate >> 9; > mtime = mktime(stm); > > Here's the layout of dosdate and dostime as I understand it from reading the > zipfile code: > > dosdate = yyyyyyymmmmddddd > dostime = hhhhhmmmmmmsssss > > where "sssss" is actually the seconds divided by 2. I'll let others check > my code. I have to dash off to a class. Thanks! How portable is this? Which header(s) should I include? Just From guido@python.org Mon Dec 9 15:13:45 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 09 Dec 2002 10:13:45 -0500 Subject: [Python-Dev] __getstate__() not inherited when __slots__ present In-Reply-To: Your message of "Sun, 08 Dec 2002 18:09:41 EST." <20021208230941.GA12533@cthulhu.gerg.ca> References: <20021208230941.GA12533@cthulhu.gerg.ca> Message-ID: <200212091513.gB9FDjb19635@odiug.zope.com> > If a class and its superclass both define __slots__, it appears that > __getstate__() is not inherited from the superclass. Example: > > """ > class Base (object): > __slots__ = [] > > def __getstate__ (self): > return tuple([getattr(self, attr) for attr in self.__slots__]) > > class Thing (Base): > __slots__ = ['a', 'b'] > > def __init__ (self): > self.a = 42 > self.b = "boo!" > > abstract = Base() > thing = Thing() > > print abstract.__getstate__ > print thing.__getstate__ > """ > > When I run this with a not-quite-current CVS Python: > > > > > > The upshot of this is that I can't just define one __getstate__() in the > superclass of a bunch of __slots__-using classes -- I guess I'll have to > set __getstate__ manually for each class. ;-( If this is a feature, is > it documented anywhere? > > (BTW, I see the same behaviour with Python 2.2.2.) It's a feature. When a class defines __slots__ and not __getstate__, it gets a bogus __getstate__ that always raises an exception. The assumption is that the base class __getstate__ doesn't know about the subclass slots and hence is unlikely to be able to retrieve them correctly. BTW, your code for accessing the slots by saying self.__slots__ doesn't work in general: self.__slots__ returns the __slots__ variable of the most derived class only. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Mon Dec 9 15:20:09 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 9 Dec 2002 10:20:09 -0500 (EST) Subject: [Python-Dev] __getstate__() not inherited when __slots__ present In-Reply-To: <200212091513.gB9FDjb19635@odiug.zope.com> Message-ID: On Mon, 9 Dec 2002, Guido van Rossum wrote: > It's a feature. When a class defines __slots__ and not __getstate__, > it gets a bogus __getstate__ that always raises an exception. The > assumption is that the base class __getstate__ doesn't know about the > subclass slots and hence is unlikely to be able to retrieve them > correctly. > > BTW, your code for accessing the slots by saying self.__slots__ > doesn't work in general: self.__slots__ returns the __slots__ variable > of the most derived class only. Here is my code for dealing with this "feature": def __getstate__(self): state = getattr(self, '__dict__', {}).copy() for obj in type(self).mro(): for name in getattr(obj,'__slots__',()): if hasattr(self, name): state[name] = getattr(self, name) return state def __setstate__(self, state): for key,value in state.items(): setattr(self, key, value) I also have a meta-class that adds these methods for derived objects, since the core decided to play it overly safe on this issue. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Mon Dec 9 15:29:37 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 09 Dec 2002 10:29:37 -0500 Subject: [Python-Dev] Proposed changes to linuxaudiodev In-Reply-To: Your message of "Sat, 07 Dec 2002 16:23:26 EST." <20021207212326.GA13707@panix.com> References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca> <20021207212326.GA13707@panix.com> Message-ID: <200212091529.gB9FTbG19825@odiug.zope.com> > > Sounds fine to me. Should I worry about adding an official > > DeprecationWarning to linuxaudiodev.c? Or does that wait until 2.4? > > Given its lack of official documentation, I'd vote for DeprecationWarning > now. +1. Somebody check this in please. (But make sure that running the unit tests doesn't issue the warning.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 9 15:30:33 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 09 Dec 2002 10:30:33 -0500 Subject: [Python-Dev] Import compromise In-Reply-To: Your message of "Sat, 07 Dec 2002 16:11:38 EST." <20021207211138.GB10103@panix.com> References: <20021207202128.GA28924@panix.com> <20021207211138.GB10103@panix.com> Message-ID: <200212091530.gB9FUXA19837@odiug.zope.com> [Aahz] > You may be right; however, the tone of the discussion seems to be partly > predicated on a short timeframe. I'm fairly confident Guido won't hold > 2.3 for lack of a new import facility. No, I'll push the zipfile import developers harder to be ready before Xmas. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 9 15:34:38 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 09 Dec 2002 10:34:38 -0500 Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension In-Reply-To: Your message of "Sat, 07 Dec 2002 15:56:00 GMT." References: Message-ID: <200212091534.gB9FYcL19877@odiug.zope.com> > PEP 273 does not mandate a particular file extension for zip > files. However, both current implementations (Jim Ahlstrom's patch and > Just's import hooks) recognise zip files via the extension ".zip". > > I'd like to suggest that this should not be the case, and that > zipfiles should be recognised by the embedded signature, as is done by > zipfile.is_zipfile(). This will (slightly) slow down the initial > checking for zipfiles, as it requires a file read, but if the result > of the check is cached the performance impact will be minimal. > > The main advantage of this is in generality. However, I have a > specific usage in mind - namely, concatenating a Python script with > its supporting code as a zip archive. (Zip archives can have arbitrary > data prepended without affecting their validity). Then, simply by > executing > > sys.path.append(sys.argv[0]) > > the script can access its own embedded modules. > > One problem with getting this to work is that at present, the Python > parser will choke on data appended to the script. This can be avoided > under Windows by a system-specific hack, of embedding a ^Z character > in the script (which text-mode I/O treats as EOF) just before the > appended data. In the longer term, a language-mandated "stop parsing > here" token, like Perl's __END__, might be useful. But that's not > something I'm proposing right now (unless an expert in Python's parser > steps up and says "oh, that's easy - I can implement that in a couple > of lines of code" :-)) > > Any comments? -1. Too much of a hack. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Mon Dec 9 15:37:37 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 9 Dec 2002 16:37:37 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) References: Message-ID: <001b01c29f98$f08a3530$ced241d5@hagrid> just wrote: > > Looking at the code in zipfile.py which converts a time tuple to DOS date > > and time shorts (I didn't see code that went the other way), it appears that > > dosdate is 7 bits for the year, 4 bits for the month and 5 bits for the day. > > Dostime looks like 5 bits for the hour, 6 bits for the minute and 5 bits for > > half the number of seconds. You should be able to extract those chunks into > > a tm structure then call the C lib mktime() function to convert to a time in > > seconds since the epoch (this is just off the top of my head): > > > > stm = malloc(sizeof(struct tm)); > > stm->tm_sec = (dostime & 0x1f) * 2; > > stm->tm_min = (dostime >> 5) & 0x3f; > > stm->tm_hour = dostime >> 11); > > stm->tm_mday = dosdate & 0x1f; > > stm->tm_mon = (dosdate >> 5) & 0xf; > > stm->tm_year = dosdate >> 9; I could have sworn zip files stored UTC time, but the specification only says "time", and zipfile.py uses localtime. oh well... here's a slightly more robust (but still untested) version: #include struct tm stm; time_t mtime; stm.tm_sec = (dostime & 0x1f) * 2; stm.tm_min = (dostime >> 5) & 0x3f; stm.tm_hour = (dostime >> 11) & 0x1f; stm.tm_mday = dosdate & 0x1f; stm.tm_mon = ((dosdate >> 5) & 0xf) - 1; stm.tm_year = ((dosdate >> 9) & 0x7f) + 80; stm.tm_isdst = 0; /* wday/yday is ignored */ mtime = mktime(&stm); you may have to add 1 to the mday (mktime expects 1..31) and/or remove the -1 from the mon value (mktime expects 0..11). > Thanks! How portable is this? Which header(s) should I include? mktime is part of ANSI C. for the header, see above. From pedronis@bluewin.ch Mon Dec 9 15:41:54 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 9 Dec 2002 16:41:54 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) References: Message-ID: <023401c29f99$80358900$6d94fea9@newmexico> I haven't looked at the code. Is that true that this version requires all import hooks to be triggered by some kind of string in sys.path which is then recognized by the import hook.? Finn has already said that but I repeat it, there are cases for Jython for which this cannot work, involving import hooks that correspond to runtime only objects. regards. From guido@python.org Mon Dec 9 16:02:48 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 09 Dec 2002 11:02:48 -0500 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: Your message of "Mon, 09 Dec 2002 16:41:54 +0100." <023401c29f99$80358900$6d94fea9@newmexico> References: <023401c29f99$80358900$6d94fea9@newmexico> Message-ID: <200212091602.gB9G2ms20175@odiug.zope.com> > Finn has already said that but I repeat it, there are cases for Jython for > which this cannot work, involving import hooks that correspond > to runtime only objects. Can you give an example? --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Mon Dec 9 16:33:33 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 9 Dec 2002 17:33:33 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) References: <023401c29f99$80358900$6d94fea9@newmexico> <200212091602.gB9G2ms20175@odiug.zope.com> Message-ID: <025201c29fa0$b753c760$6d94fea9@newmexico> From: "Guido van Rossum" > > Finn has already said that but I repeat it, there are cases for Jython for > > which this cannot work, involving import hooks that correspond > > to runtime only objects. > > Can you give an example? Java class loaders, http://java.sun.com/j2se/1.4.1/docs/api/java/lang/ClassLoader.html from them one can retrieve java classes or data resources. E.g. when Jython is embedded, the embedding enviroment can have setup classloaders to access Jython code and other bundled data ( getResource, getResourceAsStream). And our users would like to bundle and access Python code from there. They are purely runtime object, and an abstraction over the filesystem, zip files or some remote code repository of whatever... regards. From jim@interet.com Mon Dec 9 17:24:11 2002 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 09 Dec 2002 12:24:11 -0500 Subject: [Python-Dev] Complexity of import.c Message-ID: <3DF4D1BB.9030400@interet.com> I see that my patch to import.c has been criticized as adding to the complexity of an already complex module. I don't entirely agree. If you want to have the standard library available as a zip module, you must have zip file import available in C. The patch I proposed in PEP 273 was a minimalist patch; it adds as little as possible while still keeping package import and preserving all import semantics. IMHO, the real complexity of import.c comes not from the double loop for path in sys.path: for suffix in (".pyc", ".pyo",...): fopen(...) and not from the caching of zip archive names. It comes from the use of import.c to perform package imports. Package imports are performed by creating recursive calls to functions that would otherwise be flat, and by replacing sys.path with the package path. This is darned confusing. At the time PEP 273 was discussed, I proposed moving package imports out of import.c into a Python module such as Gordon's iu.py or Greg's imputil.py. This was rejected because package users feared that package imports would be slowed down. The speed of Python startup and import was a concern. I understand that people want to generalize zip imports, and create a better import hook mechanism. Some feel that now is the time to do it. This is my humble opinion: Replacing import.c plus zip import with an equally complex import.c is a fundamental mistake. The current import.c is understood (by few), and the zip file import patch adds the minimum required. If a different import.c is accepted, then it should be MUCH simpler, and support only flat directory searches and zip directories. Package imports and import hooks should be moved to a Python module such as Gordon's iu.py. And import.c must bootstrap iu.py and all its imports from either a directory search or a zip file. An improved import hook mechanism should have its own PEP. I tried my best not to break the Macintosh port, which has a lot of special code. A replacement import.c should do the same. My patch modified many other Python .c and .py files to solve several difficult bootstrap problems. If you replace just import.c you will get a painful lesson in bootstrapping. You probably need these other patches even if you reject my import.c patch. Jim Ahlstrom From guido@python.org Mon Dec 9 17:39:07 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 09 Dec 2002 12:39:07 -0500 Subject: [Python-Dev] New and Improved Import Hooks In-Reply-To: Your message of "Sat, 07 Dec 2002 00:55:13 +0100." References: Message-ID: <200212091739.gB9Hd7121989@odiug.zope.com> > Guido van Rossum wrote: > > > The problem with trying to load x.y right away is that then you can't > > distinguish between "x.y doesn't exist" (and then you go to step 2) > > as opposed to "x.y exists, but it tries to import something else that > > raised ImportError" (and then you should simply propagate the > > exception). [His brother replied] > Also not true: a function that finds *and* loads a module can return > None for "module not found", a new module object upon succes, and > propagate an error if one did occur during import. I'm tempted to > make my new import hook scheme work that way: it avoids a few hoops > I have to go through with import.c, as right now I'm quite limited > what zipimporter.find_module() can communicate to > zipimporter.load_module(). (It would also make it even closer to > iu.py ;-) OK, sounds like it can work, if b/w compat issues are taken care of. --Guido van Rossum (home page: http://www.python.org/~guido/) From =?iso-8859-1?Q?Martin_v._L=F6wis?= Mon Dec 9 17:43:18 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Mon, 9 Dec 2002 18:43:18 +0100 Subject: [Python-Dev] Complexity of import.c References: <3DF4D1BB.9030400@interet.com> Message-ID: <018701c29faa$75f7b560$f92fe8d9@mira> > I see that my patch to import.c has been criticized > as adding to the complexity of an already complex module. > I don't entirely agree. If you want to have the standard > library available as a zip module, you must have zip file > import available in C. The patch I proposed in PEP 273 > was a minimalist patch; it adds as little as possible > while still keeping package import and preserving all > import semantics. This is not the case. Much of the complexity of the patch stems from an (atleast perceived) attempt to reorganize import.c. For example: - why did is_builtin have to go? - Was the ExternalNames dictionary really necessary? - Why was the HAVE_READLINK chunk moved? - What is the purpose of Get/SetScriptName? Regards, Martin From just@letterror.com Mon Dec 9 18:03:59 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 19:03:59 +0100 Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?) In-Reply-To: <025201c29fa0$b753c760$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > Java class loaders, > > http://java.sun.com/j2se/1.4.1/docs/api/java/lang/ClassLoader.html > > from them one can retrieve java classes or data resources. > > E.g. when Jython is embedded, the embedding enviroment can have setup > classloaders to access Jython code and other bundled data > ( getResource, getResourceAsStream). > And our users would like to bundle and access Python code from there. > > They are purely runtime object, and an abstraction over the filesystem, zip > files or some remote code repository of whatever... With my current patch you could just add a "special" string to sys.path, and associate an importer object with it. However, this sounds like a job for iu's metapath (which I also intend to implement, if only to show it's possible). Very much simplified, this allows you to add inport hooks that *don't* correspond to sys.path items. I'm almost done with my note about this stuff; will explain in more detail there. Just From just@letterror.com Mon Dec 9 18:21:24 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 19:21:24 +0100 Subject: [Python-Dev] Complexity of import.c In-Reply-To: <3DF4D1BB.9030400@interet.com> Message-ID: James C. Ahlstrom wrote: > I see that my patch to import.c has been criticized > as adding to the complexity of an already complex module. > I don't entirely agree. If you want to have the standard > library available as a zip module, you must have zip file > import available in C. This has not been questioned. > The patch I proposed in PEP 273 > was a minimalist patch; it adds as little as possible > while still keeping package import and preserving all > import semantics. I think my counterpatch (even if it's not done yet) shows this is simply not true. > IMHO, the real complexity of import.c comes not from > the double loop > for path in sys.path: > for suffix in (".pyc", ".pyo",...): > fopen(...) > and not from the caching of zip archive names. It comes > from the use of import.c to perform package imports. > Package imports are performed by creating recursive calls > to functions that would otherwise be flat, and by replacing > sys.path with the package path. Erm, I think package import is very recursive by *nature*. I'm not sure what function you mean that could be flat. > This is darned confusing. So true! > At the time PEP 273 was discussed, I proposed moving package > imports out of import.c into a Python module such as Gordon's > iu.py or Greg's imputil.py. This was rejected because package > users feared that package imports would be slowed down. The > speed of Python startup and import was a concern. It's also dubious how this would help: the current code works well and is stable. I would fear that moving any of this to Python would cause bootstrapping problems. > I understand that people want to generalize zip imports, and > create a better import hook mechanism. Some feel that > now is the time to do it. > > This is my humble opinion: > > Replacing import.c plus zip import with an equally complex > import.c is a fundamental mistake. Yes, and luckily noone is proposing to do that. [ ... ] > My patch modified many other Python .c and .py files to > solve several difficult bootstrap problems. If you replace > just import.c you will get a painful lesson in bootstrapping. I've learned them indeed over the last couple of days, but I didmanage to get it working by only touching import.c and importdl.h. But my current working version also touches pythonrun.[hc], as it has a new _PyImportHooks_Init function. (The reason this was neccesary was I wanted a subclass of ImportError in the zipimport module; I learned the hard way that you can't create new exceptions from C before the exceptions module is loaded... So I had to move my init code from _PyImport_Init to a separate function, which is called by Py_Initialize().) > You probably need these other patches even if you reject my > import.c patch. I doubt it. Just From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_005D_01C29EF2.8D2854A0 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable Just van Rossum writes: > I'll write a bit about the mechanics tomorrow, but in the meantime I = would like > to see some test results/bug reports/bug fixes from Windows = developers. Please > don't start with the full monty (ie. the std lib), try a small test = archive > first please ;-) I have managed to apply the patch (import.c) to Python2.2.2 (wasn't too = hard) Could you please email me the site.py patch,or post it again?=20 >>>base64.decodestring(site.patch) >>>binascii.Error: Incorrect padding Python 2.2.2 (#37, Dec 7 2002, 19:19:43) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. >>> import sys >>> print sys.import_hooks [] >>> from PIL import Image Traceback (most recent call last): File "", line 1, in ? from PIL import Image ImportError: No module named PIL >>> sys.path.append('C:\Python22\PIL.zip') >>> from PIL import Image now it works >>> from zipimport import zipimporter >>> arc=3Dzipimporter('C:\Python22\Modules.zip') #Modules.zip - zipped = non-package directory >>> m=3Darc.load_module('module1') ok >>> arc1=3Dzipimporter('C:\Python22\Modules.zip\Folder1') >>> m=3Darc1.load_module('fmodule1') ok Regards Wiktor art@wiktorsadowski.com ------=_NextPart_000_005D_01C29EF2.8D2854A0 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
Just van Rossum <just@letterror.com> = writes:
 
> I'll write a bit about the mechanics tomorrow, but in the = meantime I=20 would like
> to see some test results/bug reports/bug fixes from = Windows=20 developers. Please
> don't start with the full monty (ie. the std = lib),=20 try a small test archive
> first please ;-)
 
I have managed to apply the patch (import.c) to Python2.2.2 = (wasn't=20 too hard)
 
Could you please email me the site.py patch,or post it again?=20
>>>base64.decodestring(site.patch)
>>>binascii.E= rror:=20 Incorrect padding
 
Python 2.2.2 (#37, Dec  7 2002, 19:19:43) [MSC 32 bit (Intel)] = on=20 win32
Type "copyright", "credits" or "license" for more=20 information.

>>> import sys
>>> print=20 sys.import_hooks
[<type 'zipimport.zipimporter'>]
 
>>> from PIL import Image
Traceback (most recent call=20 last):
  File "<pyshell#2>", line 1, in = ?
   =20 from PIL import Image
ImportError: No module named PIL
 
>>> sys.path.append('C:\Python22\PIL.zip')
>>> = from=20 PIL import Image
now it works
 
>>> from zipimport import zipimporter
>>>=20 arc=3Dzipimporter('C:\Python22\Modules.zip')  #Modules.zip - zipped = non-package directory
>>> = m=3Darc.load_module('module1')
ok
 
>>>=20 arc1=3Dzipimporter('C:\Python22\Modules.zip\Folder1')
>>>=20 m=3Darc1.load_module('fmodule1')
ok
 

Regards
Wiktor
art@wiktorsadowski.com
------=_NextPart_000_005D_01C29EF2.8D2854A0-- From jim@interet.com Mon Dec 9 21:31:37 2002 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 09 Dec 2002 16:31:37 -0500 Subject: [Python-Dev] Complexity of import.c References: <3DF4D1BB.9030400@interet.com> <018701c29faa$75f7b560$f92fe8d9@mira> Message-ID: <3DF50BB9.2080603@interet.com> Martin v. Löwis wrote: >>I see that my patch to import.c has been criticized >>as adding to the complexity of an already complex module. > This is not the case. Much of the complexity of the patch stems from an > (atleast perceived) attempt to reorganize import.c. For example: I think package imports really are to blame, because they require the zip import mechanism to be within import.c. Remember that PEP 273 semantics require that a zip archive is equivalent to a directory hierarchy, not just a single directory. Then this must work on __path__: /a/b/c/myarchive.zip/subdir/ And consider a package import which changes the path, and then imports subpackages, each of which may also alter the path. I don't see how you can jump out of import.c on a sys.path item and make this work, because the new path is coming from package import, not the original sys.path. > - why did is_builtin have to go? I introduced a dictionary of zip file contents to avoid repeated linear searches of the zip file. Since builtins are a constant list, it was easy to add builtin names too. I was not necessary to ditch is_builtin, but it was more natural to do so. > - Was the ExternalNames dictionary really necessary? Yikes! that is where I record the zip file contents. That avoids the linear search. OK, so once I had a dictionary of zip names, it was natural to add other info, such as directory file names too. > - Why was the HAVE_READLINK chunk moved? > - What is the purpose of Get/SetScriptName? These changes and several others were necessary to be able to import the Python standard lib from a zip file. The problem was the attempt to import site.py before the sys.path was fully set up. This is arguably a bug in Python. The changes moved all sys.path setup to getpath and out of sysmodule.c. This isn't an issue of zip importing per se. It is the avoidance of phase errors during start up. For example, you need a full valid sys.path to be able to import zlib.pyd (on Windows) so you read a compressed zip archive. Then you import site.py plus whatever it imports. This must be available in the zip standard lib file, so zip importing must already work. Thus the need for careful bootstrapping. JimA From =?iso-8859-1?Q?Martin_v._L=F6wis?= Mon Dec 9 21:44:33 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Mon, 9 Dec 2002 22:44:33 +0100 Subject: [Python-Dev] Complexity of import.c References: <3DF4D1BB.9030400@interet.com> <018701c29faa$75f7b560$f92fe8d9@mira> <3DF50BB9.2080603@interet.com> Message-ID: <029f01c29fcc$29aa3300$f92fe8d9@mira> > > - Was the ExternalNames dictionary really necessary? > > Yikes! that is where I record the zip file contents. That > avoids the linear search. So what would have been the problem with linear search that makes this extra data structure strictly necessary to have? > This isn't an issue of zip importing per se. It is the > avoidance of phase errors during start up. For example, > you need a full valid sys.path to be able to import zlib.pyd > (on Windows) so you read a compressed zip archive. Then > you import site.py plus whatever it imports. This must be > available in the zip standard lib file, so zip importing > must already work. Thus the need for careful bootstrapping. I think you can't really cover all cases. For example, your patch fails to find zlib in the Unix build tree, as the build directory is added to sys.path only in site.py, and your patch won't attempt to import zlib outside InitZip. So it might have been good to explain somewhere what bootstrapping scenarios this patch is meant to support. Regards, Martin From arigo@tunes.org Mon Dec 9 22:48:15 2002 From: arigo@tunes.org (Armin Rigo) Date: Mon, 9 Dec 2002 14:48:15 -0800 (PST) Subject: [Python-Dev] os.path.commonprefix In-Reply-To: <15858.8941.920830.141997@montanaro.dyndns.org>; from skip@pobox.com on Sat, Dec 07, 2002 at 10:33:49AM -0600 References: <20021207160218.1528F49BA@bespin.org> <15858.8941.920830.141997@montanaro.dyndns.org> Message-ID: <20021209224815.8F7D94B52@bespin.org> Hello Skip, On Sat, Dec 07, 2002 at 10:33:49AM -0600, Skip Montanaro wrote: > :-) I discovered the same thing over a year ago, fixed the problem in CVS, > then got roundly booed because I'd changed the semantics. Can't we deprecate the thing and move it elsewhere? (where?) In all cases, wouldn't it be worth a documentation note? I could submit a patch. Armin From just@letterror.com Mon Dec 9 22:51:21 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 9 Dec 2002 23:51:21 +0100 Subject: [Python-Dev] zipimport, round 4 Message-ID: --11364638-0-3248466697=:9007 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit Here's a new patch. Changes include: - pyc time stamp checking (thanks Skip and !) - better python -v output - also works when zipimport is built dynamically I've written a note about various aspects of the patch (pasted below) but I'm not sure it's PEP-ready yet. Comments are more than welcome! Just --------------------- This note is in a way an addendum to PEP 273. I fully agree the points and goals of the PEP, except for the sections "Efficiency", "Directory Imports" and "Custom Imports". However, I disagree strongly with much of the design and implementation of PEP 273. This note presents an alternative, with a matching implementation. A brief history of import.c -- digging through its cvs log. When Python was brand new, there were only builtin modules and .py files. Then .pyc support was added and about half a year later support for dynamically loaded C extension was implemented. Then came frozen modules. Then Guido rewrote much of import.c, introducing the filedescr struct {suffix, mode, type}, allowing for some level of (builtin) extension of the import mechanism. This was just before Python 1.1 was released. Since then, the only big change has been package support (in 1997), which added a lot of complexity. (The __import__ hook was quietly added in 1995, it's not even mentioned in the log entry of ceval.c r2.69, I had to do a manual binary search to find it...) All later import extensions were either implemented using the filedescr mechanism and/or hardcoded in find_module and load_module. This ranges from reading byte code from Macintosh resources to Windows registry-based imports. Every single time this involved another test in find_module() and another branch in the load_module() switch. "This has to stop." The PEP 273 implementation. Obviously the PEP 273 implementation has to add *something* to import.c, but it makes a few mistakes: - it's badly factored (for example it adds the full implementation of reading zip files to import.c.) - it adds a questionable new feature: directory caching. The original author claimed this was needed for zip imports, but instead solving the problem locally for zip files the feature is added to the builtin import mechanism as a whole. Although this causes some speedup (especially for network file system imports), this is bad, for several reasons: - it's not strictly *needed* for builtin import - it's not a frequent feature request from users (as far as I know) - it makes import.c even more complicated than it already is (note that I say "complicated", not "complex") - it changes semantics: if a module is added to the file system *after* the directory contents has been cached, it will not be found. This might only be a real issue for an IDE that runs code inside the IDE process, but still. A different approach. An __import__ hook is close to useless for the problem at hand, as it needs to reimplement much of import.c from scratch. This can be witnessed in Guido's old ihooks.py, Greg Stein's imputils.py and Gordon McMillan's iu.py, each of which are failry complex, and not compatible with any other. So we either have to add just another import type to import.c for zip archives, or we can add a more general import hook. Let's assume for a moment we want to do the *former*. The most important goal is for zip file names on sys.path and PYTHONPATH to "just work" -- as if a zip archive is just another directory. So when traversing sys.path, each item must be checked for zip-file-ness, and if it is, the zip file's file index needs to be read so we can determine whether the module being imported is in there. I went for an OO approach, and represent a zip file with an instance of the zipimporter class. Obviously it's quite expensive to read the zip file index again and again, so we have to maintain a cache of zipimporter objects. The most Pythonic approach would be to use a dict, using the sys.path item as the key. This cache could be private to the zip import mechanism, but it makes sense to also cache the fact that a sys.path item is *not* a zip file. A simple solution is to map such a path item to None. By now it makes more sense to have this cache available in import.c. The zipimporter protocol. The zipimporter's constructor takes one argument: a path to a zip archive. It will raise an exception if the file is not found or if it's not a zip file. The import mechanism works in two steps: 1) find the module, 2) if found, load the module. The zipimporter object follows this pattern, it has two methods: find_module(name): Returns None if the module wasn't found, or the zipimporter object itself if it was. load_module(fullname): Load the module, return it (or propagate an exception). The main path traversing loop in import.c will then look like this (omitting the caching mechanics for brevity): def find_module(name, path): if isbuiltin(name): return builtin_filedescr if isfrozen(name): return frozen_filedescr if path is None: path = sys.path for p in sys.path: try: v = zipimporter(p) except ZipImportError: pass else: w = v.find_module(name) if w is not None: return w ...handle builtin file system import... Packages. Paths to subdirectories of the zip archive must also work, on sys.path for one, but most importantly for pkg.__path__. For example: "Archive.zip/distutils/". Such a path will most likely be added *after* "StdLib.zip" has been read (after all, the parent package is *always* loaded before any submodules), so all I need to do is strip the sub path, and look up the bare .zip path in the cache. A *new* zipimporter instance is then created, which references the same (internal, but visible) file directory info as the "bare" zipimporter object. A .prefix contains the sub path: >>> from zipimport import zipimporter >>> z = zipimporter("Archive.zip/distutils") >>> z.archive 'Archive.zip' >>> z.prefix 'distutils/' >>> Beyond zipimport. So there we are, zipimport works, with just a relatively minor impact on import.c. The obvious next question is: what about other import types, whether future needs for the core, or third party needs? It turns out the above approach is *easily* generalized to handle *arbitrary* path-based import hooks. Instead of just checking for zip-ness, it can check a list of candidates (again, caching cruft omitted): def find_module(name, path): if isbuiltin(name): return builtin_filedescr if isfrozen(name): return frozen_filedescr if path is None: path = sys.path for p in sys.path: v = None for hook in sys.import_hooks: try: v = hook(p) except ImportError: pass else: break if v is not None: w = v.find_module(name) if w is not None: return w ...handle builtin file system import... Now, one tiny step further, and we have something that fairly closely mimics Gordon McMillan's iu.py. That tiny step is what Gordon calls the "metapath". It works like this: def find_module(name, path): for v in sys.meta_path: w = v.find_module(name, path) if w is not None: return w # fall through to builtin behavior if isbuiltin(name): return builtin_filedescr [ rest same as above ] An item on sys.meta_path can override *anything*, and does not need an item on sys.path to get invoked. The find_module() method of such an object has an extra argument: path. It is None or parent.__path__. If it's None, sys.path is implied. The Patch. I've modified import.c to support all of the above. Even the path handler cache is exposed: sys.path_importers. (This is what Gordon calls "shadowpath"; I'm not happy with either name, I'm open to suggestions.) The sys.meta_path addition isn't strictly neccesary, but it's a useful feature and I think generalizes and exposes the import mechanism to the maximum of what is possible with the current state of import.c. The protocol details are open to discussion. They are partly based on what's relatively easily doable in import.c. Other than that I've tried to follow common sense as to what is practical for writing import hooks. The patch is not yet complete, especially regarding integration with the imp module: you can't currently use the imp module to invoke any import hook. I have some ideas on how to do this, but I'd like to focus on the basics first. Also: the reload() function is currently not supported. This will be easy to fix later. I *thought* about allowing objects on sys.path (which would then work as an importer object) but for now I've not done it as sys.meta_path makes it somewhat redundant. It would be easy to do, though: it would add another 10 lines or so to import.c. I've tested the zipimporter module both as a builtin and as a shared lib: it works for me in both configurations. But when building it dynamically: it _has_ to be available on sys.path *before* site.py is run. When running from the build dir on unix: add the appropriate build/lib.* dir to your PYTHONPATH and it should work. --11364638-0-3248466697=:9007 Content-Type: application/octet-stream; Name="zipper4.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000" Content-Transfer-Encoding: base64 Content-Disposition: attachment; Filename="zipper4.tgz" H4sIAAAAAAAAA+09+VMbR7P5VforBqUCEixYEhJn7BQxEPM+GyjAuV1bi7SCjSWt vLviSML//vqYc3fFkcTOq+9ZlVjs7ExPT09PX9Mz+j2aTMKk8+yLj/hpNjvN9W4X vpvN9TX+brY7/M2fL6DCerfdWmu11r9otlqrnfYXovsxkVKfaZoFiRBf/AZ/3FcP qg0GnwKhT/v5Xc4/fEejSZxkK71/vI9mq9lc63RmzH9nrd3t5OZ/tdtd/UI0/3FM Sj7/z+f/y2jcG077oagd32aX8XjlslY1ZWmWTHvZKBydh4n7Jk774SB1y0ZBkl4G Q7ewF48m0TB0Cr/OohGUvKhWq0DWLOqJ3mWQLIrJrR/eZOE4jeJx+ss78Vz8UVuZ 3PZqnsDvWH7X7rah5bNFoZk2TER8/lvYywRgFY2jDACIYNwX6XSC78Xis2o1u52E 8FrwoJzGP0eTA/n3NuJUqPBHtXJ8e0Rd+K/2dnbNo1gMkt5ldBVuCwEoTYLschyM QhEPRHYZIhAhKyASVrMBUCWFRtSsH0HRRTgOk2Ao8I2IxoM412KSwOButrkFVeKS LVELnqXT82f9KIGKcXL7rIZN77Y1fY9vz2D0EpA1Wh+Lt00t1ZWuspckMdIEe4yT 6yDpA4l7QwRfaJSEQd/XONRxTjV1GiWdDGOoDwTyR3F/OgzrFl5iMQ2HA08wjMF0 OESietUKF0ADJLQHVMpElE6C3vvgIlSPkuX6nkA+8zMxwm/PJmUW9/xwnCW3Bi9u 66fhhDGHP+Bl9UviqNAh2svLsPe+Hk8aejBER1Xsifk8iRvV4vCrFoP54/C67kzS Ym+YesLmsgvn+f11P21UgS9hZg4uxnESAp9dTEcwKqhng8b1ILLgfZiKXpBIzgwy oAdwEi6S61D04/FCBg9QMRYjqCuG0SAUQIc+TAbMPKyk894wSNMwXSG2LEwW0KqC 37Bq3ZlswEiWX2QTPxgO416dxtUE0laigahTi7nn4vDt69cNXGYEY/mFWjP8ZluX 06oplPI6MMV31UoSZtNkLOqGYg3G8q5qzXg1T6gyJnzEJFiLFPjSVyCh8qKWIRo7 Yq/z6eCXNzs/Hu+cvXq9d7jUeqdeLDJnL+bHhPw5DMdIZiTc3PHtTnLhH4PMDc+m E1g+jFztaG7LGtSKT8Py/RrMdwVnHD7zx7enWRKNL4g3gV0dmjcaUFOSb7mF/R3f +geHL0/29utuRcAKkQUcNcCdlP8o1gTc7Yqn0e9hoRIPjWo+Bx4hfji+BRnkn4aZ BOyKJhiwYhVCJUpFOJpktzXs0h4F8oQC/uK5MKR/VC9Iu4rbUxbHYhiPL2b2VMdq v0B/y613OJ6FyYL480+RLz1eaIj5eVqMwjRp88vIbSJLD0qbrPLL390msvRnq4l5 2eGXKwtMBM1yNMAlYjdSNkRSVJxAGlr9MMRwmIbUCt4rwoyCW3GudR+IjVGVeK72 DCE+y+JnsuYKsCiqLNAWpKoqivdl50hSlDr17W1GrUIqD4QVAl4EQZQtirjXmyZJ MO5RjzWEKYFJaB8AGiAN/9UnnqxAs0Xz8wHHTnKHZjdf+eeD49mVz0HXvaeXH8TS c9HR9aQGWfzQ4IqapB+oykT9dafZRNWwZSAMdhwLxBeIOIVR06iYR/eBqkFWzqDj GIQ42RwoJ7fEVyvtZjMFswlpWsanvCgVDsu6HlBh3Jvc1kFKcWMPmQFYoqUEtxoo N23YDLS0xFyTvgdETveOGXsUeAADrbqFX5sLiAz3Dn3BHxIHQpLkm+yAi7EloNd6 JzuqmGIECJ1su6WIqt0V01tKK0Blu1pQNFo07SfxSIoBTQ9XzZRWVfgbxWZB59nF lemCMkzlSFyg3llyi8r4IsQpBX7uXYZ9xyYdQN8ivU1XXJWzQov8B63T+30Eg8um hzBEPAWNMbxSetxtzGO7Tf3vwow1Wr3m1qgpBsg1lDochAxA2AVzVlpJbjU5fZZO lJWhv4MsHOWqM1vDeArahHlQgbE6L9ppqlbDSJJ0HGS9S7K6U6ajpk9feRIsSFyr o56zbDRoWYWYELTlj466pFcNw4W8jl8OwyCp23aQ7scWBfgSIH4fJudxGu4Pg4sG d4Kz9EMSZeFpBjZaUq99abhjS6AtjsKa3ISaImNNW+c86oWv0oVfx7Z8cMebs+h1 rVkYYz85CSMfm7bhdRVHfcfy6odsGxaML2lc+T/u7pXbH4V3mtiFN0bA8HN87qNL SKbpIAnZGmnkDUQ0hn0Eig7IbBeF3BP5NwlN/nNx4htHBR60r0IDM/aea98Bh/74 448e/kMGOjDnWKQxeJVQsiXGYUhruh9bZgx0CMwu9TMJcC2/HzLOjHUmRTG0wjJL YpOArVbkcJSFQJKU7QSUtw4rSwHnmvYaKdNoNnpKsyhNsVRuPZppvavO6AEnpzBI xczKGDJ6RJUs2RqE1+GbYDLB/l8F6X/CWwddYjy5kni49nwDoNa2IYExsBjDmjbT 7eFubruWVg5gE6va6EtUzaojfrp7yPdEu0r54KYKiMG860NMW1gCRYcPmmpvBow8 XhvSr3H9nneS8SOyLzzjyz/o5qSOl2MNwXF0ZO+2S8PulGTVLIE1kNQ1kmQKOzyb dwzg+abnfx8Mp6FteUkIYjRNM4FmGJjBAazQLAv7K/jGcRO0m6p40pUyLFg0UrSI 55l880aCbBuH5mtwloojJAM6IkYREdRZha+lJWcpGj70ctG36J2W9E9k+5y7yDb0 THdcKUWQeTx/RD1j9Rpg8NdhPKZxS2iy5BEsTqGmf57FSwJKmvF10MpaA0XmL6iA kpVQscNYT1wY1sCdhaGwm7Uy5rSoZSPK1Rhgx0rvs0zYoxlW3l6J9CctLWVDoyaU HQUZtDifZmB60IKDxcbOaTprman1SeKf1ryZH1z0SnepasagMS1Vi20Wyead4/AA NFLBf3NpP1skCit1y+NeUi3FL0vgVGul8Q7ewNqlFkUlR8WoEp9rR32J/LIlAg9q 3bIYbIlAwXZERnM3GexKGkibfaY4kDQ1jR0fgZhWcocJx+oghapAmt56DzOzxuxD y4EQOhhnYDy8jomHuJnyJ3RLT3TYlAChIc1COwis9KwFt8mTio745BbMr2nSAya8 CqJhcD7kaP59EtbIBlh+Ooz3aNELFf8y1cvJLqWzGyNHyRZhjTbLacY1H5lnrjUr htjWxN2Ve4EfO/zOcXfh0HnmBDDW1P9zJdklyqgczoM+LKULEPHjKW5HgUUtzm8z 8NriPvYB4xwDDaXbZnUCEBl8pRePs2g8DenpztJKUEUqooJvpt21ewKQltvVC9Dn RvJJbVaz1BVzgK2r3oTZZdzfDQdOwH5EpXIDrFr5o+ZYN2KG6eaJN3tnr/zvd052 Tr47BbTAV6jdedjeUQJihl6c3R7x9ioV/Loj7zlEOobDwh7TG9ooLA4HS63hSCEE qIgz/+jb/9l7eeaJeDBIwywe2HrWU/Kq4VVO9nZ2jw5f/8QjIoZHAA+DYIcwD4AF KkB4qLlUWbn2RAsa/PHtLnDx6dkJEK7u+LRxz6siEcEK7NEWSnG7smbvMO3u7e+d nOzt+ju7uyd7p6d1/G6I5uM28Zi4ziYlGE4HZ/UC2HmGw/tSoByaMCDDwysWlmgx pCB/XZo0PGxTkesSnOcIxBFWo7VX74e8fxonjRIX35Nt5CM1cYBNEnKV88UXYYZK v/ginfUCxRBudxVeJOGkpHqQ+lKwlL1Kww/TEAPNZS9HLJmL7y6D9LIEMRh5yTjk GPQMfoe7wVEPxP0OjM/LESKeSYmSN4AjSOxBqHrwz473X+98dwoicn/n7esz8aew Cr/dOd07++l4zxPcejAMLlJqmWdv3UG/bCazJLiCdV9Csx6K15KJiXqXM2ftOgze D6M041VafB9lZVOHpVotlMhYMwRZUFKRpJddkQpKWbQUs/OgjAa0418sDdNegpBm vSrtAWFZdKmjVThJ4p6zALFQcxHtCTMzkCyQzLbDS1TzjV6iua1qUwUeXLbdDYee wQzjakpNYA4BABmGuEcdUIoGpWWwczNBRwYYKcXnOsoA+LsfZHJXX74g90zmZ2Qj sHJHtO2cjVayEdCmh+aMrCrmRfOmNWiIRdHe1nVG0diu8+KF6Dao5urAVLoEWy9X q9VqSIAWrH6AppnEs/g6pq7q6r3pC5BapnCQrHmLyyFXdZOrrkPdJbHRNJWjtA8e D5mcQNBrwOHZLSISpSKiPIA+RwCVbfOeCDoPrTm2CY4FaM1cZFc7dssvWGFWsdJ3 UDQWgdrvlFtLsi5agdGwj1EOYD8w9qQUpIgzmokpgqgDBwVDtQOhm05TrIo+SJAC hDSE6Q9QZVBNYDo0C6MwXSE8dowJicMMRIY2/ha9Iy8UpUYSpqnHz18KVYBM9h5s pG3RJBU8Has3YV+1FkDywEcF5snWpMzigTB1qY4AYP0ofa8b4lBLG/bDXFO3Ca9V j5rIdYuNsAve1MmChIokvXRraVCrz5doZFKSC1ZG0LpmL+mZekSRpMch4nQ6ytdu ECV3JTdElCgyRqcaCuKLMQyqD7Y2zSAI9Wio5g7WEpZxao5NSPLVraEKFOnNlSoy ZjFQc2/ekJtdkUv/2D94vQeFE4ztgvcnDB/Q+K2JtabKmQKWMBxYYRiXgA1IOVUB Bycbyjf4UAjKcFsM3lD85gId+l48HWcqxkNxWitcvyS6Ov6Dfcx6tzixgpXo0uul +uLBJAYOqByBFh4M42s7poLrmFY1BRNgTZVmM5hsGstnNUY5xRhxR3sQTwxi4Dkk 53qXEN8Xw6nle9glnhQCtvazazqCVVvgve2Fmo1PCeYwi+F7QMMTy+22h24uqqZT KOS1FgKnyuVAikwFPCmV0T+BKce4Au7z7kMVAETx4eZNc63Z7XzbbeoN+2+D/pbY Y2gvUVaB2IMlBTLhYhwAUgy+MugN45TgbD+FHPmd/adTAkghF4/ccj0Mr/X2Y2Eb 7yIGQRzK5D+biGsehwqcZSIDFLNpRgEtcarkmkUfuR2JtOF1g4oQObGR26eh1SR3 PZzUECsIS7EPqAWIUwTF4J1b1c0GB3c0RjT/XInnafiYMXEAXjJEu9kihsCZ4fwQ zRY2P2Bz8crqaQaSuDHUlJlyFSXYCkidXgJ75LHSgbEHa5K18aiaqD8eQxGjAx5T W4vlR9XWsvhxKFvyGhp01oCiBsISZyY9AOaR1bbvncZOW02jrRIfy14GZUfgI5fZ 9DCviPNRLMtYdSFKqZtZ0coKpV6R7m2A+9FTGBAKiyTEF54tcE4TVdUpN0tLKnZG 5U3i+h3MOgkTsLZB2YNiHE/B56VwHbF8jkLPXdXK8pQiwzodNUUBisYG25P2cgXt 1YuTROdsmIlwND3PwNOXdae5qpe1jiDqlf2a0LlnTdtTviTaazYiq00o+oc4EDFi 2kiy6IiMw3VA6+F2pSD5yPYkt0FvKIMotlKrVEKYiVpf0B9yd92zMxGW1etlShez Ququk2dsL5ZaNDX+t+hV0CZQvRbBZxjVPMuwK7Xpqhx/duZb2sogt0xI3ElUsVVc pQJ/GNV46sTYZXSdx5nlNxE0dISALFMCnHQXLZS76oyMnocSenhH9CtwuMi3Qtv7 qxRzd5gvAb6j/qX2l1lJhMhW1TU+rAQZnTQzM1KtjXUQDr5xb/wBeFR1zOeRnnmu uqlp7YAgAewXrn2omv4+jM6RdvjNuzxECZ+/3vDGRA3f1pj7mVKC6hPfYz/05G6A ODhZ4TaKs8kpx2ZggJmqtfzeBdZQOV1q56iwe/DE7K2Rnm1C28xvxR6G+EbU9DZU DUzBt4c73+8cvN759vVezd3KslLRzFBoScqJNqX3TDduBOGycz0zTyYTo71llifF aswSxUfL9dKPvEhzzl0SXFM/HvANyiMermdzkfaKOIXT8v+k2eeRlkl99CnZWHzY R5lDMVtwTth5OjjSbhPa2q6DQlY5uSgkRL9KHzTB1RDLc0h3xn3a7pSEbvDwNTnB ZmWEDRSbsV3PouAI8QZwMdlKASO2YCl4nwZVku65lHQOuQf4zUlnLVdUE1IOhgrQ n3/aUwY8bsZ7j1NrzUulUvQbCRYCquXEtTXasvlBpLR0mNNnEDBvQ6NFmVY/L8gt /+AazJKI5DFggfTC3WKSP7qFLfltsQfQ6fAXTiWAQG6y4kYEQ6Ko0JbZgbtGhF1H oGi1zHNkW5moVpSfLX8fcQ6iQG4LGmEpM05JOrPrquUVEd7RjkAYvSSkMH4ZDIf7 gC4GiS1U8XTLENaYERTLra5UZXJ+HTGn1F759CsRSGQ1wg99TxJ7+EddSxsiDwV5 bqDtjQB0KZ2n+Y5DugMq/RNsaCxtqdKG+PprITacl23nZWvNebnqvGx3DKI3+dTU 8INPRk5dhqKylj7xlrVJtsonXJpZC8yxrK2mX6WM4avlPrvnKtwdj4d42CMGKooQ I8ApEHXcB/KnMUYFwbqLQrlJp4govqYEx1IdMh3Lk6E+7tXX8+lRJgWKny3rgebY DrTlNAbCs1VCqYBT840aQjuPdh6pln4UZcMaMJjNJ6wHzE2Y3PYsgVMuWTRXET/N WUYNWB9vMLnhkLYg642n535/lYrLIDVpEtp0qAh9KFXJwmJCXT6jjmUbRq6GAvcr wZ/AwHxyK4IhbqnBFMsDrWpsbOKjyKT0L82b9pjRF25Iy/xvDhFBfMohIqcVXEdm Q+sMCI9xw2M+WxYbStxy88JhD3W0jxPuXkIlmRCF9XWWnBJeVFiIGrI+xF0821Jh W0XoNB+ZW8mhQqVsAkqeUbkIuqVDy4IdIQt46ZWu90F04w+jceiHYzyDkFoZl5xF ZSW/f/D40FdJEjhX3daZ8RfuSUNOAedKMvvc3ie4Cfs+vyUT0Hq+3/aaqPi9CY3a TWdMYZWOjxUHYTeWQdAknAwDAFX7Nfl1/E0NN1ZqwMkmn0zFNa3Iy6/JggrTfFha okzv8YIJz9QndDKLq45VVRWaqcjA6vLytkp8sn0WCRH3Gdw+mwtWKLOqwEFTGwda RXTGSG0M/UrbuhU6fCcT0n1NmJMQV0Z93qaLTXDtttrTV8pkkrNlpScoFYsFXUXi Pcg2Bca2ePTxnKJFif+Sx6AN23u5x8q/o0mF9mSdR+PJNHONoDzXPbxkn34UX7gn 8WXy8iOP4ouSxOncjEhHUE4MdIN+YNTT+2mLJQf37jl87O4OPrgxqJpgiErHkZSJ kDvUNzuDOpegbeWiUmQrwsjWvMFq3qCld3fEvIXgvOOOzRvk5jHKxcu+YHlo69px 5Y0Tzx09gTbahbA90HIaGAZg5CTr5wxCw1P6Xgee/ZGKDdqHYCSM3PqfAUOncGr/ QBmDOaWMbqh6VkaDGROvHDUmlaWtTz+D+PPpYJTvq0wHqW8Xz0OQ6eEiny9E+Jgl xIo2vAl700w5enrbSh1exjPOaTjRBaoH9wBBVaXsalNyp9+XwTGd468ku0yutWJh JcZFjoPI+pGnM2QyesxHTlf6UUI57fJNgwcit4oVunpCOOrNg7LS8E29071jqPNl NMBLWnZen8GzCsWrVlb09j5I3BiBoaAe5KDMWVAWVekSqk+pr9AtwuwsMveIlGif Y2BYJxPfEy3mbJiawsaYqPng9i/pO1j/ql7jEXFk4OTcZO8BD6HZyGju3VjrgCWn RfziXD86Ei1Dq2CAf0nJzcC0X8mDs2hlmkCl6d5KMf9G1DDCg73WxJao8ZKt2ciZ dCVMxS767ZLuM8LT6Aznr25BbcKnEYlvkJlyrIVij85jUwVZqjiG02L0Ymhtl59k dVO4R5NC7raTO12WPJ3LHx5N7ksctpLKoeGbo13M8PX33x6+rGJGn65nIvLOKQQl wmTSH7owt/XiXTXucTLTyD+agEAGdUCcoqTfSSizPPGmJkr8wb9i+1aHbERiDL5w D8g569B8R6ZlrqxQrVVSjU5nMmiUUWpl+AdACV4PnboJVVHiu5kg+US05tXJwdbj n85eHR36O8cH/vd7J6cHR4fEna7bT/2gC3YYXoPzFU4oTmXlULvVLREgZPTSTvMg YSRXY76fvHZ1b2Nxa5sVLYUWaAJ5hp/Mp1oZVjIDLAepyAGm0yK/PNRvLqNcd2of BnyYC++q//Zdaf+NH+v+P/xGxd67/If7eOD+v1a7s67v/2vi+9bq+tr65/v/PsXn YNwPb7ZAK2XTCYjt6vO//6mevDyVWWrPeldpEsfZswldLqi++lGaPUuT3jPVrXdV hWWeROEVRhAS+KLM3dZKq92ugj4eiOWeWE7o2eC6uLioHypruCki2sBYotXcane3 4I9l5K8KQ1leXjaVN03ldntrdX2r2eXKCNP+UB/tzTVvtdkSVIBwuADgw0dn3KJB n6UrAayjcb8u9pSiqi/4l3GWwv8LnvhFP6z0Ft41wMR4RHM8+pCEA24uHx7f/Ab3 wjBqkRIA88i2hAK09CAgLcYJjnVhqMLEQuZLESRJcCtDi+kj0KT6BJn+cpGTXHrA 90dKNkqm45XLT8ywJRjM4N32SrdrsS4+ChtvZKwSaJX2mjgCjciM3Nla3dxqbkpG JpDIf2XtnsbSm5teq9kxHE3PXcXQx7do/6BZaato4bNngPYVG5jbbl0qw1oqU+PB imQLWbWW7gH3Ko7f5zuvUvjx+yCJ4iluwVLcHrPD4F+wU/mUzuyu98FoVsAkix3b 846U7X1iDisiMJPBWmubDofBs7ARx4kugqu01sVhfCU5bH2r297qdDSHIUxkiJJ2 RQ5bu4fDWmstr7W2ZliMCzYUj6HrMLnEo1FtzELDsyTgF8kJqxgm2o9uphMjJ8Cv llXRtRbOI7PEktXa4hniLgSNjlIaXaRQQhnAvAN0CY4Lha/TbDrAw820xLCArpMd Z0m9wbgpGKMgGksYbPeCK49lvo/1XHZiSdkffnJ5Veh/NjNtuLy0ISykLU4ypSiq 9sNzyUitrU5ra9VipA2bj6xWT2UjD1Q+PQLdj//znb97cLL38uzo5Cc8clJ56X/7 9uD12cEhPR3/5O+fHP28B09z9PTyaHfvZO/06O3Jyz2aqaDH29s0lXdShsgDZUhS OmUHri1zrNde1fz6t/r28p1DlYM3x/6ro6P/3I9IGSP9W0JJ9T6TidqttsNF8Cw0 ygUmApliW20ghza2uquagxBYgYVK5dD6fQzUXfVa3Q1LDkEBNFbzeseUpyvJlmZr G3j3B4oWow+vPLHIEQXam7APd6tUxyWhwrqcirZE0okjxiRrbtMVbgRWIfTHV1YX bvTz0HClNOrpxIoFARcxwLIwHhSXb6KDjM2CIYk6Awp7/3VcaygUVXgT25+amwBH YRZQuBIk7xVVeA0cQgdJmg1qbCeVXnHJDFi5WwUVQHMyZRY8NwHNjNOCbdN0NqoI 6MqOhyOsWbQsJ+aseYwTUeNmdOte7qrEQYDrW9G7kDWKRTLoIS8gyS6xl+uQcg77 Mc985Q7/URx4T4asjoCZYetWc+7oXSaelRurmd6J7OjB6AxZWY9f3Jm+TR+57jE8 miOlchWsVg05fs1ZNLM7XO/Kud+iBKfcuzubpebMXpBDib/DIjPX2yPY4E6uAGcB 3PEaJYElhG0oKWMWiv8gjWTWjgmbAoxSH7cDyq5r+7gbG157s6PVX02u+1qQXFzh 9yRt8Vcbv8KbKMMEwxrpQnjs0eWK/KrnX+GWh3rIkqAXnge997IybsHo2vSgq/Mr qz4QJL+8i6LEFlTYA86e1rJlo99obngbrTVlZ2jPYl9eeU8Ku/z+erlhzoFuT3BG stodp21wMktl44KCX7Qv31ONTBdkS6iIqXqNO7J+pvpaLIHvD5L493AsofND3VxU aflOr/GqGjzhzZcKogKS+xxgJatdTHkj6g8HZ68ERYHf7B2e7e0Sn2x0W95Gd1Px yT9LOpsUH5uMdv1/kaal7NkB9ux2tBnMu4J4i8Z4OsF+K2jFVGQ6qtrGhCEP+pT6 btHGXDSp7nWmbG15ywvmJWKawYRGw0ktfecEr9ACEIWV3vx4g2HjMK0XtjcaqlFe vBHvbALvbK5rGfPxx6V3Wz7y6EpnsbvubaytmlmsjLSlOGPwMNSRuo5LDpR3V3Ek NEAYwPKLjHdh1JisLW6Ea2fdE/9zF1sc+Gk2vU30wNQU/KNIFcj9WNRK6bfR8TY2 Nq2gVXvdazVb2pS3dmilXb8kzLkg0rnW78GQFi1mQi1RnoyjUqwgWN4wN29spWSX T0pcBwVAewi/eWLMlqO2ZygZ0QaqjXPHQrvv8mwxMRbfLJPL7OvlbCYlsWxUpT2S M1OkT4OnC10Efpm8w3QYzKWh20PQOMErJaZJKi9/0c6LnUdRHIZn7oOb01a/kwvA WFD65G980PU3POhKhIO/+ZQrj1dNAvGDJrcioTuNv0liIJJcX3EvGXw5FHL3mped ZUAgeIqhpifHnZ2cqakSyZ4gjdjrKLFR9bsZdmrBMt0ypqkWcMd4O1ddl97J7wIh 1JsS18a6hg8f74pEmLNSsKiFTEF1Tfkcv5Y533cF7n8+i/tNonhhHuUdsxKeTCJ5 tfP9nn96tnNW1oV1eyFCM7mopcmKihGknYEiif6hI26KONx8fh6vAQkyeRHHvKzG ywLdlrlT/+B09+CkLt+sAGuPKJdcz7/8kSqVA84/7YE2ChQsQski3eij7mWQV7zQ +TbwzWCJeAwGpHXKqW4yAKs4aBTCeMdROhLX9AMEeBEy3peJF5aHwZAPLEXZigZz Eo7iKz7cTaKC0qMv42GfLgDEoIjDnPjbRcIHT9gneBqMhCt/g6SHt6WbyzqHAV6G qxxHW+jshsMZstOSNSUcfqcZghKpWQbyTy68H8fX4hrR5EvaoTf23eEdXcFDST1A 4+UxZVzC6oDxA+ooQpXAJFRlNlKJDJcsZv3cwj3S03jO6kjbvUspP1Yaaok+uLtf h1KiWGKZyI5a/EcVqccehdKoZvUWdXm+Y7e7Mo1pZYiWLfKZ2pIxyqsCzherG8xz t3vW0IGVDoXj/buTpC5416O+0+7HaYjJvbSeaKHXQV4F02GmI4sN4r9AeiIruAIJ GLTgjVq6TiGdxGP9ExLsZbGE8kha1K+iQLk1+ofHGgSj1HjbXPM2Wy3Lw7Z+dq0X pKEfv687nh/9Iw/Cbrv1iWJk7jseH2/GWI6n3U/RZawKUfQaQdMMy3jV+W0H9hzR 1sXbe9EgRqpJP3LigzmrQjDyRu8xX/ANBbIp3fqL/dAN/Vhe9GjJx9GWuEy/h/cj SkaGInX62aoG2mmM6unk4PTl0akF2NUqQkkWCog3W2hHr3YsH/4TTM7SzJlhTscV yc/4M5C4MuB/tWdiNkw+/eQ6gYKJFHP/0ow70lBHvby8FM2J27/CKWWLerO14W22 V629lWanhYkFZpdXjj4FDdf37eHLX4egcj65qXnYUXqKwMZxRHmr6W61U06mvDdK TXbxojQhZp0qz12VVjrmNgiy1ZY95i6OeUOnA3HQQqbBM7tpXNQOtYx06xlzLBzj GBUGL1W3aVfye00m+Gl5GFYLo87mpPPDZw51FWMrzjh7bx+Ale5BzR2OupU/EHhn K/08m65pDTXVDka5fVUZ60Eap9jgyTUL1xvR0lNXG8lx2IFHqZ8fdgatBRUpRO/V 6tK9y2v0I6nSNbUqzkx8o625LWGNiw8tyO4cv9OhFv9AH6UL2+atYZycN8JHVqTl IK7BwNY/tsGv7bXlkEt1O58T0HmXT1ksUjLm7GW1TIkEzi8COD8YoRlR8MbQ2aU8 mI5xJWSodHouQ6l4Te112MetlqiPPFfj6GyN2tKBVhmJxV8/BQuFfjsNY7kMgPbr ZB3+VbTShb8Owm7D3khutTe9VqvbtYJnvA5mLAHjVdEVPpyvkmN9IYmkzVjfxIVy 69zZBsnbrarZrMXuRJaMR/DI5V4r7HjOXvCl673UxfgrP0W3ZJjJbuoM+/5fo3v6 wHOdmaHj0Z/7RoizWybR5KLHt6whhf4BD601HhBzwolt9eLJrRPGtdIWimLOlnBs GaBhAOv47enB4Xf+28MDTGWxguNvx1HPnFW/MlFw2a+psTemE36mYOdUwYN25Rq2 1Wx2Oa5rLbX1Niw1azuyYu9uYrdyMZnoEorFK4GPAeaV4d6BTC7TK6ySiw5LEVim eVFCOQxvyqmNOjdf/EXEh5SQlrWWz8xWw/2u6tV/pZrga+zUYCROeF90Ic4kEblz mHYU9MAAj9NLqTboC3jqdBKG/elkS4QBuMZaRuMvF9DF2ZQ5iseUwbUtZ8pOp+O1 Ot2m8mDx5qsHd/PECf74wt7hyz3x8ujt4VneKbViJnOiuCfKbKCMfsdLQTeDtn3M Zr910GtKt765pdsyp6294bXWjLP3f2QctlOV86geMbLSGeu2MfOrbWvs9SZ4uutt Y6pbEeCRlCAy2Mx2Ojq/2uncshJAcgvuLxjMtuDB2GU8zSTnP2QUj2ZYn9wasyac 33vJRZSsaLrSSTJAtGV5Ru4laDb+QqiN9NquCXdewv9ZrNamvGakTrYaHXv+qt+o zRD3aFi1Njes7e3CpJBHJU+AzznqTLvZM39VXZjQuutr86uCS03JL6hv4J+Nexjl fpysiOQ/iVw5ATe7mNHYMgSUwcBKydZ5yd65/jUya+PcxtPKCLC1rrFaZm2lP3X7 StJ+YxNov2mZ1R9hOPCP1jsff1xl0wYz1oFpazWthICy/ShhSQK9GcVUmZPCwJa4 5gT2rBwBuRPPO/C5LXhzT4BrEfDEbG547Xaz6xphHxNhT+hJmnMnyZm8vzCg8hnZ aMKMbLoz4mxgPiUl5QnLqpT78sxXdBb/YnpIWe6FPn8vBWB7HWRKe8Od649LCjtZ 5OMTpJia8gBZSjkGSARkWjW6S904ZJPEDnjYG0fYnfQGNfpF2tmEM/cZzWCkh0lU zMRRy7sD9hL8o1P5/r2x3Jel9cgRlc5WdxMk7lrTWt+5n6hwZJcVkJlzNtyKrGXG otfa3SwDmWi9jjpu3dZxHwUTi5Az8Sml1Hp7DfBbdc68tlse/GOy9fh2DTwbVcdc cH0aBqxN/Tef4heURYZZJdMk3C5pq8/O1PAGBvn349vaJ20Ygl1SCmepAEdZ+dYu 0wwMyGuSz2Qz69ODr6PzZyk4lZ/+JLXV88zD1J015yx1Z00oXHHOLQiVTXNCsNnd am7oM1oVgoL84FZ/yokc0CwqjCoE/u5P6kdjH3xytaPxB/KqTJHABBTlrm9V1eHi /WhIP6I45ZSzlJNCOPsivAGaoLTPVGIFp7Bwogv/TDAsIwaURvijflG2QL9CNQl7 Ed0GmYYr4gcMLKexGIdhH10b+XaZXEIE+CboeRoQxkf0D19RoFlFpWMzAGyWhCuy EfAfvRgGGQx2hNJrYRT0Frbke1kHiYAb7nTXtLx5KUqhtA7/N+zKvHXP4S9ZjP7J I+ARzdL7AfIjVPHwH6IC/7gxQqCmVd0HwlaVgPyFSVYziZ/X6ogHgUDWQgbRGan/ dxlkX/6m88zpXtFQ8DKfHyJ3my9lqP3pOR1exmuervgmcRtzClGlGlB+GCX5WVvq B9souNWLp8M+R6ffnrzWcJB44U2Av8ZnseMTWOMTs8W/fU3G58/nz+fP58/nz+fP 58/nz3/N538B2FEpBwCgAAA= --11364638-0-3248466697=:9007-- From skip@pobox.com Mon Dec 9 23:18:42 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 9 Dec 2002 17:18:42 -0600 Subject: [Python-Dev] os.path.commonprefix In-Reply-To: <20021209224815.8F7D94B52@bespin.org> References: <20021207160218.1528F49BA@bespin.org> <15858.8941.920830.141997@montanaro.dyndns.org> <20021209224815.8F7D94B52@bespin.org> Message-ID: <15861.9426.521685.951820@montanaro.dyndns.org> >>>>> "Armin" == Armin Rigo writes: Armin> Can't we deprecate the thing and move it elsewhere? (where?) How about just adding commonpathprefix? As I mentioned before, there's an implementation in Tools/scripts/trace.py. Armin> In all cases, wouldn't it be worth a documentation note? The docs for the function state, "Note that this may return invalid paths because it works a character at a time." I suspect that's sufficient. Skip From python@ragnark.vestdata.no Tue Dec 10 03:45:39 2002 From: python@ragnark.vestdata.no (=?iso-8859-1?Q?Ragnar_Kj=F8rstad?=) Date: Tue, 10 Dec 2002 04:45:39 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding In-Reply-To: <3DF46D4A.2070902@lemburg.com>; from mal@lemburg.com on Mon, Dec 09, 2002 at 11:15:38AM +0100 References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> Message-ID: <20021210044539.H26400@vestdata.no> On Mon, Dec 09, 2002 at 11:15:38AM +0100, M.-A. Lemburg wrote: > > The dumps-method in xmlrpclib has the following comment: > > All 8-bit strings in the data structure are assumed to use the > > packet encoding. Unicode strings are automatically converted, > > where necessary. > >=20 > > This doesn't work very well. In our particular case we're using latin= _1 > > as our default encoding, and we're using UTF-8 for the packet encodin= g. > > We can't really change the default encoding, because the sql-modules > > transfer latin_1 encoded data and we can't change the packet encoding= to > > latin_1 because the xmlrpc-client (php) doesn't work with that. > > > > The attached patch changes xmlrpclib to convert strings to unicode us= ing > > the default encoding, and then convert them back to strings with the > > packet encoding. If unicode is not available it falls back to the old > > behaviour. >=20 > I believe this is overkill. If you need this behaviour, subclass > the Marshaller in xmlrpclib and add your feature to that subclass. > Then replace the Marshaller class in xmlrpclib with your subclass. Well, we replaced the xmlrpclib.Marshaller.dump_string method from our application. That works as a workaround for us, but my point was not not to merely make our application work but to fix this problem for other python users as well. The library makes an assumption that is (IMHO) just not valid. There is simply no reason to assume strings use the packet encoding.=20 Why would you not like to fix this? Because of the performance? It would be possible to have both functions available in the class, and only use the encoding-convertion when the encodings are actually different. This could be done with no other performance penalty than a simple check when the encoding is set. (The constructor?) I simply didn't include this code in my patch because it would make the code harder to read and I think most people use ascii or latin_1 for thei= r=20 string-encoding and UTF-8 as their packet-encoding. > Aside: xmlrpclib should support subclassing the Marshaller and > Unmarshaller more transparently. Currently, the two are hard-coded > into the rest of xmlrpclib without the possibility to provide your > own subclasses without tweaking xmlrpclib from the outside. In principle I agree, but it should not be neccessary to subclass the Marshaller for most applications, and tweaking it from the outside can be done pretty easily in python :) > Please post patches using the SourceForge patch manager. Didn't you just write that the patch was overkill and you didn't want it? Do you want me to post it anyway? Or did you just mean for any potentail future patches? --=20 Ragnar Kj=F8rstad Zet.no From martin@v.loewis.de Tue Dec 10 06:10:37 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 10 Dec 2002 07:10:37 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding In-Reply-To: <20021210044539.H26400@vestdata.no> References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> Message-ID: Ragnar Kj=C3=B8rstad writes: > The library makes an assumption that is (IMHO) just not valid. There is > simply no reason to assume strings use the packet encoding.=20 There isn't a reason to assume that strings use any other encoding, either. Reliance on the default encoding should be avoided as much as possible. Users wanting to transmit non-ASCII strings with xmlrpclib should pass Unicode objects in the first place. Use byte strings to represent bytes, use Unicode strings to represent characters. Regards, Martin From python@ragnark.vestdata.no Tue Dec 10 08:25:14 2002 From: python@ragnark.vestdata.no (=?iso-8859-1?Q?Ragnar_Kj=F8rstad?=) Date: Tue, 10 Dec 2002 09:25:14 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding In-Reply-To: ; from martin@v.loewis.de on Tue, Dec 10, 2002 at 07:10:37AM +0100 References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> Message-ID: <20021210092514.L26400@vestdata.no> On Tue, Dec 10, 2002 at 07:10:37AM +0100, Martin v. L=F6wis wrote: > Ragnar Kj=C3=B8rstad writes: >=20 > > The library makes an assumption that is (IMHO) just not valid. There = is > > simply no reason to assume strings use the packet encoding.=20 >=20 > There isn't a reason to assume that strings use any other encoding, > either.=20 Isn't that what sys.{get,set}defaultencoding is for? > Reliance on the default encoding should be avoided as much as > possible. Users wanting to transmit non-ASCII strings with xmlrpclib > should pass Unicode objects in the first place. >=20 > Use byte strings to represent bytes, use Unicode strings to represent > characters. So perhaps it would be better to change the postgresql-modules to return unicode-objects instead of strings? --=20 Ragnar Kj=F8rstad Zet.no From fredrik@pythonware.com Tue Dec 10 08:40:26 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 10 Dec 2002 09:40:26 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> <20021210092514.L26400@vestdata.no> Message-ID: <005f01c2a027$ca50d920$0900a8c0@spiff> Ragnar Kj=F8rstad wrote: > > There isn't a reason to assume that strings use any other encoding, > > either.=20 >=20 > Isn't that what sys.{get,set}defaultencoding is for? Tthe default encoding stuff was added for experimental purposes only; in = the early days, there was some disagreement on whether the default encoding should be UTF-8, ISO-Latin-1 (8-bit subset of unicode), or ASCII (7-bit = sub- set of unicode). We finally settled on ASCII, for a couple of reasons. Robust application code should assume that the default encoding will = *always* be ASCII, and use Unicode strings and ASCII-only strings for text. > So perhaps it would be better to change the postgresql-modules to = return > unicode-objects instead of strings? That's the correct solution, yes. Checking for chars>=3D128 and returning Unicode only if the string = contains non-ASCII characters is also a valid approach. From brian@sweetapp.com Tue Dec 10 08:48:26 2002 From: brian@sweetapp.com (Brian Quinlan) Date: Tue, 10 Dec 2002 00:48:26 -0800 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding In-Reply-To: <20021210092514.L26400@vestdata.no> Message-ID: <002101c2a028$e803e2e0$21795418@dell1700> > > There isn't a reason to assume that strings use any other encoding, > > either. > > Isn't that what sys.{get,set}defaultencoding is for? If that is all you want to support, then: def dump_string(self, value, escape=escape): if unicode: self.dump_unicode(unicode(value), escape) else: self.write("%s\n" % escape(value)) dispatch[StringType] = dump_string If you have changed the default encoding (not recommended) then everything should work fine. > So perhaps it would be better to change the postgresql-modules to return > unicode-objects instead of strings? Maybe. Cheers, Brian From =?utf-8?Q?Martin_v._L=C3=B6wis?= Tue Dec 10 08:56:17 2002 From: =?utf-8?Q?Martin_v._L=C3=B6wis?= (=?utf-8?Q?Martin_v._L=C3=B6wis?=) Date: Tue, 10 Dec 2002 09:56:17 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> <20021210092514.L26400@vestdata.no> Message-ID: <000801c2a02a$00edf2e0$ad25e8d9@mira> > > There isn't a reason to assume that strings use any other encoding, > > either. > > Isn't that what sys.{get,set}defaultencoding is for? No. If it was for me, those functions would not be there in the first place, and the Python system encoding would be ASCII. I consider it an advantage that setdefaultencoding isn't really there. Byte strings are used in many places in Python, e.g. when writing to files. You can't possibly know the encoding of all files that you may encounter in advance, so the notion of a single default encoding is too limited in practice - as you have just discovered. The purpose of the default encoding is to define automatic coercion between byte strings and Unicode strings, e.g. when comparing or concatenating them. This coercion is necessary primarily because of string literals; if they were Unicode strings instead of byte strings, coercion and the default encoding could be deprecated. > So perhaps it would be better to change the postgresql-modules to return > unicode-objects instead of strings? I'm without context here to answer the question. If you are talking about a character string type, and if postgresql has a well-defined notion of what the encoding of character strings is, then yes (except for backwards compatibility concerns). If you are talking about BLOBs, then no: these are byte strings, and don't have any inherent encoding. I don't know how SQL deals with non-ASCII strings. If (as I assume) there are no clear specifications, it might be best to introduce the notion of a "table encoding" or "driver encoding" in the DB ABI. If this is not set, the driver would return byte strings. If it is set, the driver would return Unicode strings, decoding the data it receives from the underlying database appropriately. I don't use relational database or SQL regularly, so take this advise with a grain of salt. My 0.02 €, Martin From Jack.Jansen@cwi.nl Tue Dec 10 10:15:15 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Tue, 10 Dec 2002 11:15:15 +0100 Subject: [Python-Dev] Complexity of import.c In-Reply-To: <3DF4D1BB.9030400@interet.com> Message-ID: <46B467C6-0C28-11D7-AE16-0030655234CE@cwi.nl> On Monday, Dec 9, 2002, at 18:24 Europe/Amsterdam, James C. Ahlstrom wrote: > I tried my best not to break the Macintosh port, which has > a lot of special code. A replacement import.c should do > the same. Actually, if the new improved import has the right set of hooks then the macintosh stuff can go away. 90% of it does basically the same as what zip-import does: it allows you to put files on sys.path and looks in those files for modules. The mumbo-jumbo about interning sys.path strings is so the code can cache whether a sys.path entry is a file or a directory, without that normal imports were slowed down by 30% or so. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From mal@lemburg.com Tue Dec 10 11:04:53 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 10 Dec 2002 12:04:53 +0100 Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> Message-ID: <3DF5CA55.1070505@lemburg.com> Ragnar Kj=F8rstad wrote: > On Mon, Dec 09, 2002 at 11:15:38AM +0100, M.-A. Lemburg wrote: >=20 >>>The dumps-method in xmlrpclib has the following comment: >>> All 8-bit strings in the data structure are assumed to use the >>> packet encoding. Unicode strings are automatically converted, >>> where necessary. >>> >>>This doesn't work very well. In our particular case we're using latin_= 1 >>>as our default encoding, and we're using UTF-8 for the packet encoding. >>>We can't really change the default encoding, because the sql-modules >>>transfer latin_1 encoded data and we can't change the packet encoding = to >>>latin_1 because the xmlrpc-client (php) doesn't work with that. >>> >>>The attached patch changes xmlrpclib to convert strings to unicode usi= ng >>>the default encoding, and then convert them back to strings with the >>>packet encoding. If unicode is not available it falls back to the old >>>behaviour. >> >>I believe this is overkill. If you need this behaviour, subclass >>the Marshaller in xmlrpclib and add your feature to that subclass. >>Then replace the Marshaller class in xmlrpclib with your subclass. >=20 > Well, we replaced the xmlrpclib.Marshaller.dump_string method from our > application. That works as a workaround for us, but my point was not no= t > to merely make our application work but to fix this problem for other > python users as well. The first point I want to make is that non-ASCII text doesn't belong in strings, it belongs in Unicode objects. I don't think there's anything to argue here. My second point is that I would like xmlrpclib to be more flexible w/r to using user-supplied marshallers and unmarshallers. The reasoning here is that applications will want to send custom types/classes over the wire and need to provide custom mappings for these via those adapted marshallers, e.g. a product I'm working on uses mxDateTime objects to store date/time values and passes in buffer() objects to signal: this is binary data. > The library makes an assumption that is (IMHO) just not valid. There is > simply no reason to assume strings use the packet encoding.=20 It is valid with respect to what the whole standard lib assumes: strings contains ASCII text data or binary data. > Why would you not like to fix this? Because of the performance? It woul= d > be possible to have both functions available in the class, and only use > the encoding-convertion when the encodings are actually different. This > could be done with no other performance penalty than a simple check whe= n > the encoding is set. (The constructor?) >=20 > I simply didn't include this code in my patch because it would make the > code harder to read and I think most people use ascii or latin_1 for th= eir=20 > string-encoding and UTF-8 as their packet-encoding. A correct fix would be to check whether the strings is indeed using the xmlrpc encoding. That is IMHO overkill, since this is something to think about when designing the application and not a runtime check that's always needed. >>Aside: xmlrpclib should support subclassing the Marshaller and >>Unmarshaller more transparently. Currently, the two are hard-coded >>into the rest of xmlrpclib without the possibility to provide your >>own subclasses without tweaking xmlrpclib from the outside. >=20 >=20 > In principle I agree, but it should not be neccessary to subclass the > Marshaller for most applications, and tweaking it from the outside can > be done pretty easily in python :) Sure, but it's not a clean solution :-( I've done that too in the past, but reverted to writing my own code for the dumps() and loads() APIs to be able to use subclassing to implement the above mentioned mappings. >>Please post patches using the SourceForge patch manager. >=20 >=20 > Didn't you just write that the patch was overkill and you didn't want > it? Do you want me to post it anyway? Or did you just mean for any > potentail future patches? No, I meant that patches should always be posted there. SF is what we use to track patches and bugs. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From erik@pythonware.com Tue Dec 10 11:54:34 2002 From: erik@pythonware.com (Erik Heneryd) Date: Tue, 10 Dec 2002 12:54:34 +0100 (CET) Subject: [Python-Dev] anyone tried to use optparse with something else than sys.argv? Message-ID: hi snuck a peek at the newly included optparse module yesterday and it seems option parsing will be a breeze in the future. good work! however, imho, it seems a bit too focused on initial sys.argv parsing. for anything else, you basically have to subclass OptionParser etc to do anything useful. for example, you can supply your own argument list to the parse_args() method, but the autohelp/version/error reporters still use sys.argv[0] as command name (probably not what you want). the way optparse gets the command name (a global function) indicates that you're not supposed to override it? also, even if you tell OptionParser to use your user defined Option class (as explained in the docs: op = OptionParser(option_class=MyOption)), it still adds standard Option instances to handle autohelp/version. per default, the autohelp/version/error handlers call sys.exit(). the docs says that if you want to change this behaviour, simply subclass OptionParser and/or Option and override/extend the corresponding methods. ofcourse, this has no effect, since standard instances are used. also, supressing the actual printing of autohelp/version/error requires subclassing OptionParser and Option (or sys.stdout = StringIO fiddling). so? some minor glitches and (imho) annoyances, whats the deal? well, to me, it's pretty obvious that the design effort on optparse has pretty much been focused strictly on initial sys.argv parsing and while it's extendable, other use cases (embedded consoles etc) seem to have been given little or no real thought. the things i mention is probably easy to fix: * make parse_args() take the whole "sys.argv" list and use the first element as the command name. breaks backward compatibility, i know. why was sys.argv[1:] used in the first place? the getopt legacy? * add parse_args_ex() that raises an exception containing the help/error message instead of SystemExit (SystemExit containing something means something went wrong, don't it?) * (simple bug) dynamically populate the OptionParser using the option_class and not some pre-made Option. * (simple bug) replace all %prog in help texts etc, not just the first occurance. erik From jim@interet.com Tue Dec 10 15:09:46 2002 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 10 Dec 2002 10:09:46 -0500 Subject: [Python-Dev] Complexity of import.c References: <3DF4D1BB.9030400@interet.com> <018701c29faa$75f7b560$f92fe8d9@mira> <3DF50BB9.2080603@interet.com> <029f01c29fcc$29aa3300$f92fe8d9@mira> Message-ID: <3DF603BA.5090504@interet.com> Martin v. Löwis wrote: > So what would have been the problem with linear search that makes this > extra data structure strictly necessary to have? Speed. The linear search would be repeated with each import. > I think you can't really cover all cases. For example, your patch fails > to find zlib in the Unix build tree, as the build directory is added to > sys.path only in site.py, and your patch won't attempt to import zlib > outside InitZip. Hmm, you are right. But easily fixed. JimA From guido@python.org Tue Dec 10 15:31:06 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 10 Dec 2002 10:31:06 -0500 Subject: [Python-Dev] anyone tried to use optparse with something else than sys.argv? In-Reply-To: Your message of "Tue, 10 Dec 2002 12:54:34 +0100." References: Message-ID: <200212101531.gBAFV6h31663@odiug.zope.com> > snuck a peek at the newly included optparse module yesterday and it seems > option parsing will be a breeze in the future. good work! > > however, imho, it seems a bit too focused on initial sys.argv parsing. > for anything else, you basically have to subclass OptionParser etc to do > anything useful. > > for example, you can supply your own argument list to the parse_args() > method, but the autohelp/version/error reporters still use sys.argv[0] as > command name (probably not what you want). the way optparse gets the > command name (a global function) indicates that you're not supposed to > override it? > > also, even if you tell OptionParser to use your user defined Option class > (as explained in the docs: op = OptionParser(option_class=MyOption)), it > still adds standard Option instances to handle autohelp/version. > > per default, the autohelp/version/error handlers call sys.exit(). the > docs says that if you want to change this behaviour, simply subclass > OptionParser and/or Option and override/extend the corresponding methods. > ofcourse, this has no effect, since standard instances are used. > > also, supressing the actual printing of autohelp/version/error requires > subclassing OptionParser and Option (or sys.stdout = StringIO fiddling). > > so? some minor glitches and (imho) annoyances, whats the deal? > > well, to me, it's pretty obvious that the design effort on optparse has > pretty much been focused strictly on initial sys.argv parsing and while > it's extendable, other use cases (embedded consoles etc) seem to have been > given little or no real thought. > > the things i mention is probably easy to fix: > > * make parse_args() take the whole "sys.argv" list and use the first > element as the command name. breaks backward compatibility, i know. > why was sys.argv[1:] used in the first place? the getopt legacy? > > * add parse_args_ex() that raises an exception containing the help/error > message instead of SystemExit (SystemExit containing something means > something went wrong, don't it?) > > * (simple bug) dynamically populate the OptionParser using the > option_class and not some pre-made Option. > > * (simple bug) replace all %prog in help texts etc, not just the first > occurance. You make some good points. A patch uploaded to SF would be welcome. --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@python.net Tue Dec 10 16:32:35 2002 From: gward@python.net (Greg Ward) Date: Tue, 10 Dec 2002 11:32:35 -0500 Subject: [Python-Dev] Proposed changes to linuxaudiodev In-Reply-To: <200212091529.gB9FTbG19825@odiug.zope.com> References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca> <20021207212326.GA13707@panix.com> <200212091529.gB9FTbG19825@odiug.zope.com> Message-ID: <20021210163235.GA8067@cthulhu.gerg.ca> [me] > Sounds fine to me. Should I worry about adding an official > DeprecationWarning to linuxaudiodev.c? Or does that wait until 2.4? [Aahz] > Given its lack of official documentation, I'd vote for DeprecationWarning > now. [Guido] > +1. > > Somebody check this in please. (But make sure that running the unit > tests doesn't issue the warning.) Two points: * ossaudiodev isn't 100% compatible with linuxaudiodev -- does this affect your decision? (Currently the incompatible is limited to subtle behavioural stuff, but I plan to change some peculiar method signatures -- setparameters(), in particular, takes an 'ssize' argument that serves no purpose as near as I can tell.) * if I remove test_linuxaudiodev.py, which should be fine since I've just created and checked in test_ossaudiodev.py, then the warnings from the test suite won't be an issue. OK if I remove it? Greg -- Greg Ward http://www.gerg.ca/ I am NOT a nut.... From gward@python.net Tue Dec 10 16:40:48 2002 From: gward@python.net (Greg Ward) Date: Tue, 10 Dec 2002 11:40:48 -0500 Subject: [Python-Dev] anyone tried to use optparse with something else than sys.argv? In-Reply-To: References: Message-ID: <20021210164048.GB8067@cthulhu.gerg.ca> [cc'ing optik-users@lists.sourceforge.net, since this concerns Optik's reincarnation as optparse in the Python standard library] On 10 December 2002, Erik Heneryd said (on python-dev): > snuck a peek at the newly included optparse module yesterday and it seems > option parsing will be a breeze in the future. good work! Thanks! > well, to me, it's pretty obvious that the design effort on optparse has > pretty much been focused strictly on initial sys.argv parsing and while > it's extendable, other use cases (embedded consoles etc) seem to have been > given little or no real thought. Guilty as charged. My intention was always to support that kind of stuff, but I've never had occasion to test it myself. Sounds like I carefully hid lots of little bugs in there -- congrats for finding them. ;-) > the things i mention is probably easy to fix: > > * make parse_args() take the whole "sys.argv" list and use the first > element as the command name. breaks backward compatibility, i know. Since optparse is new to the stdlib as of Python 2.3, backwards compatibility is not a huge concern; anyone switching from Optik to optparse will have to make changes anyways. So this is an option. > why was sys.argv[1:] used in the first place? the getopt legacy? Gee, good question. I can assure you it had nothing to do with getopt. I think it was because, despite Unix/libc conventions, I don't see the program name as part of the argument list. That's a perfectly valid interpretation, but I failed to allow a clean/obvious way to override argv[0]. Idea: add a keyword arg 'prog' to OptionParser.__init__() that sets self.prog, and then make get_prog_name() a method that returns self.prog or sys.argv[0]. That feels right. The other possibility is to add a 'prog' argument to parse_args(), but I'm not sure offhand what would happen to it (and how it would find its way to down to usage/version strings.) Anyways, thanks for the bug reports. I'll use your message as a to-do list for Optik 1.4.1. Greg -- Greg Ward http://www.gerg.ca/ Think honk if you're a telepath. From guido@python.org Tue Dec 10 16:40:10 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 10 Dec 2002 11:40:10 -0500 Subject: [Python-Dev] Proposed changes to linuxaudiodev In-Reply-To: Your message of "Tue, 10 Dec 2002 11:32:35 EST." <20021210163235.GA8067@cthulhu.gerg.ca> References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca> <20021207212326.GA13707@panix.com> <200212091529.gB9FTbG19825@odiug.zope.com> <20021210163235.GA8067@cthulhu.gerg.ca> Message-ID: <200212101640.gBAGeA032131@odiug.zope.com> > [me] > > Sounds fine to me. Should I worry about adding an official > > DeprecationWarning to linuxaudiodev.c? Or does that wait until 2.4? > > [Aahz] > > Given its lack of official documentation, I'd vote for DeprecationWarning > > now. > > [Guido] > > +1. > > > > Somebody check this in please. (But make sure that running the unit > > tests doesn't issue the warning.) > > Two points: > > * ossaudiodev isn't 100% compatible with linuxaudiodev -- does this > affect your decision? (Currently the incompatible is limited > to subtle behavioural stuff, but I plan to change some peculiar > method signatures -- setparameters(), in particular, takes an > 'ssize' argument that serves no purpose as near as I can tell.) I don't care about API compatibility -- as long as oss lets you do the same stuff (possibly by making different calls) I think it's a good replacement. The deprecation warning for linuxaudiodev doesn't mean it's illegal to use it today -- it means it will go away in the future. > * if I remove test_linuxaudiodev.py, which should be fine since I've > just created and checked in test_ossaudiodev.py, then the warnings > from the test suite won't be an issue. OK if I remove it? Typically, we don't remove tests for deprecated code until we remove the deprecated code itself. This saves us from accidentally breaking the deprecated code before it's time to kill it. --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@python.net Wed Dec 11 13:49:58 2002 From: gward@python.net (Greg Ward) Date: Wed, 11 Dec 2002 08:49:58 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: References: Message-ID: <20021211134958.GA10384@cthulhu.gerg.ca> On 11 December 2002, Just van Rossum said: > This checkin makes textwrap fail for me when running python -S ("don't run > site.py"): > > Python 2.3a0 (#42, Dec 11 2002, 10:33:42) > [GCC 3.1 20020420 (prerelease)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import textwrap > Traceback (most recent call last): > File "", line 1, in ? > File "/Users/just/code/python_cvs/Lib/textwrap.py", line 15, in ? > class TextWrapper: > File "/Users/just/code/python_cvs/Lib/textwrap.py", line 56, in TextWrapper > unicode_whitespace_trans[ord(unicode(c))] = ord(u' ') > UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 0: ordinal > not in range(128) > >>> Weird. What the heck is 0xa0 doing in string.whitespace? Oh, waitasec, 0xa0 is non-breaking-space in ISO-8859-1. But no, string.whitespace is hardcoded to the US-ASCII whitespace chars, all of which are in range(128) -- your locale shouldn't matter. Just, can you run the following commands and tell me what you get: python -c "import string; print [hex(ord(c)) for c in string.whitespace]" python -S -c "import string; print [hex(ord(c)) for c in string.whitespace]" LANG=C python -c "import string; print [hex(ord(c)) for c in string.whitespace]" LANG=C python -S -c "import string; print [hex(ord(c)) for c in string.whitespace]" (On my Linux box where LANG=en_CA normally, I always get the six US-ASCII whitespace chars.) Greg -- Greg Ward http://www.gerg.ca/ This quote intentionally left blank. From fredrik@pythonware.com Wed Dec 11 14:04:58 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 11 Dec 2002 15:04:58 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 References: <20021211134958.GA10384@cthulhu.gerg.ca> Message-ID: <017b01c2a11e$4b655c60$0900a8c0@spiff> Greg Ward wrote: > Weird. What the heck is 0xa0 doing in string.whitespace? Oh, = waitasec, > 0xa0 is non-breaking-space in ISO-8859-1. But no, string.whitespace = is > hardcoded to the US-ASCII whitespace chars, all of which are in > range(128) -- your locale shouldn't matter. $ tail Lib/string.py try: from strop import maketrans, lowercase, uppercase, whitespace letters =3D lowercase + uppercase except ImportError: pass $ tail Modules/strop.c DL_EXPORT(void) initstrop(void) { ... /* Create 'whitespace' object */ n =3D 0; for (c =3D 0; c < 256; c++) { if (isspace(c)) buf[n++] =3D c; } s =3D PyString_FromStringAndSize(buf, n); if (s) { PyDict_SetItemString(d, "whitespace", s); Py_DECREF(s); } ... From just@letterror.com Wed Dec 11 14:17:59 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 11 Dec 2002 15:17:59 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: <20021211134958.GA10384@cthulhu.gerg.ca> Message-ID: Greg Ward wrote: > Just, can you run the following commands and tell me what you get: I get this in all four cases: ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20'] But: I can't reproduce the problem with -c, only in an interactive shell or in a script! Here's what I get in an interactive session: sh-2.05a$ ./python.exe -S Python 2.3a0 (#42, Dec 11 2002, 10:33:42) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; print [hex(ord(c)) for c in string.whitespace] ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20', '0xa0'] >>> (Look, now there's an 0xa0 in there!) sh-2.05a$ ./python.exe Python 2.3a0 (#42, Dec 11 2002, 10:33:42) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; print [hex(ord(c)) for c in string.whitespace] ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20'] >>> sh-2.05a$ LANG=C ./python.exe -S Python 2.3a0 (#42, Dec 11 2002, 10:33:42) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; print [hex(ord(c)) for c in string.whitespace] ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20'] >>> sh-2.05a$ LANG=C ./python.exe Python 2.3a0 (#42, Dec 11 2002, 10:33:42) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; print [hex(ord(c)) for c in string.whitespace] ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20'] >>> I have _no_ idea what's going on here. Just From martin@v.loewis.de Wed Dec 11 14:23:10 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 15:23:10 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: <017b01c2a11e$4b655c60$0900a8c0@spiff> References: <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff> Message-ID: "Fredrik Lundh" writes: > > Weird. What the heck is 0xa0 doing in string.whitespace? Oh, waitasec, > > 0xa0 is non-breaking-space in ISO-8859-1. But no, string.whitespace is > > hardcoded to the US-ASCII whitespace chars, all of which are in > > range(128) -- your locale shouldn't matter. [...] > /* Create 'whitespace' object */ > n = 0; > for (c = 0; c < 256; c++) { > if (isspace(c)) > buf[n++] = c; > } Sure. But shouldn't Python/the-C-library startup in the "C" locale? Is this another one of these Mac quirks? Regards, Martin From mwh@python.net Wed Dec 11 14:38:20 2002 From: mwh@python.net (Michael Hudson) Date: 11 Dec 2002 14:38:20 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: Just van Rossum's message of "Wed, 11 Dec 2002 15:17:59 +0100" References: Message-ID: <2mel8owrhf.fsf@starship.python.net> Just van Rossum writes: > I have _no_ idea what's going on here. I do: readline. I wouldn't want to live without readline, but some of the things it does call for the application of thumbscrews. Cheers, M. -- FORD: Just pust the fish in your ear, come on, it's only a little one. ARTHUR: Uuuuuuuuggh! -- The Hitch-Hikers Guide to the Galaxy, Episode 1 From gward@python.net Wed Dec 11 14:39:57 2002 From: gward@python.net (Greg Ward) Date: Wed, 11 Dec 2002 09:39:57 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: <017b01c2a11e$4b655c60$0900a8c0@spiff> References: <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff> Message-ID: <20021211143957.GA10597@cthulhu.gerg.ca> [/F proves beyond a shadow of a doubt that string.whitespace is locale-sensitive] Thanks, Fredrik! That clarifies the behaviour Just is seeing. Hey: I just realized that making textwrap trust string.whitespace is wrong in at least one case: 0xa0 is *non-breaking* space in ISO-8859-1, and converting it to 0x20 (regular ol' space) is clearly wrong -- the "non-break" request will be ignored. So Unicode or not, textwrap should probably just hard-code the US-ASCII whitespace chars. My attitude is that textwrap should work on European languages, whether they are encoded in 8-bit "ASCII" or Unicode. I suspect that passing an arbitrary Unicode string to it is meaningles -- what the heck does it even mean to wrap a string of Chinese or Hebrew or Devangari characters? Beats me, and I think they're out of scope for textwrap. So: do I even need to worry about the cornucopia of Unicode whitespace characters at all? Or can I sweep that can of worms under the rug? (Pardon the horribly mixed metaphor.) Greg -- Greg Ward http://www.gerg.ca/ Do radioactive cats have 18 half-lives? From guido@python.org Wed Dec 11 15:25:41 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 11 Dec 2002 10:25:41 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: Your message of "Wed, 11 Dec 2002 08:49:58 EST." <20021211134958.GA10384@cthulhu.gerg.ca> References: <20021211134958.GA10384@cthulhu.gerg.ca> Message-ID: <200212111525.gBBFPfh09916@pcp02138704pcs.reston01.va.comcast.net> > Weird. What the heck is 0xa0 doing in string.whitespace? Oh, > waitasec, 0xa0 is non-breaking-space in ISO-8859-1. But no, > string.whitespace is hardcoded to the US-ASCII whitespace chars, all > of which are in range(128) -- your locale shouldn't matter. It's overridden in the strop module, which tests for isspace(). --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Wed Dec 11 15:27:09 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 16:27:09 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: References: Message-ID: Just van Rossum writes: > I have _no_ idea what's going on here. Are you using readline in interactive mode? Do you have the current CVS? readline sets LC_CTYPE, but the current CVS is supposed to restore it to its original value. Regards, Martin From guido@python.org Wed Dec 11 15:37:09 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 11 Dec 2002 10:37:09 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: Your message of "11 Dec 2002 14:38:20 GMT." <2mel8owrhf.fsf@starship.python.net> References: <2mel8owrhf.fsf@starship.python.net> Message-ID: <200212111537.gBBFb9v16268@pcp02138704pcs.reston01.va.comcast.net> > I wouldn't want to live without readline, but some of the things it > does call for the application of thumbscrews. Indeed. But I thought I fixed that in 2.3, by restoring the locale after initializing GNU readline. Maybe the Mac IDE environment starts up in a different locale? Would be typical for Mac... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Dec 11 15:39:56 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 11 Dec 2002 10:39:56 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: Your message of "Wed, 11 Dec 2002 09:39:57 EST." <20021211143957.GA10597@cthulhu.gerg.ca> References: <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff> <20021211143957.GA10597@cthulhu.gerg.ca> Message-ID: <200212111539.gBBFduf16609@pcp02138704pcs.reston01.va.comcast.net> > [/F proves beyond a shadow of a doubt that string.whitespace is > locale-sensitive] > > Thanks, Fredrik! That clarifies the behaviour Just is seeing. > > Hey: I just realized that making textwrap trust string.whitespace is > wrong in at least one case: 0xa0 is *non-breaking* space in ISO-8859-1, > and converting it to 0x20 (regular ol' space) is clearly wrong -- the > "non-break" request will be ignored. So Unicode or not, textwrap should > probably just hard-code the US-ASCII whitespace chars. +1 > My attitude is that textwrap should work on European languages, whether > they are encoded in 8-bit "ASCII" or Unicode. I suspect that passing an > arbitrary Unicode string to it is meaningles -- what the heck does it > even mean to wrap a string of Chinese or Hebrew or Devangari characters? > Beats me, and I think they're out of scope for textwrap. Correct -- you can't trust the width of characters to be all the same. (I'm not even sure if that's true for Latin-1, Cyrillic or Greek, but it seems likely.) > So: do I even need to worry about the cornucopia of Unicode whitespace > characters at all? Or can I sweep that can of worms under the rug? > (Pardon the horribly mixed metaphor.) Please shove them under the garage. --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Wed Dec 11 15:45:31 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 11 Dec 2002 16:45:31 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: <200212111537.gBBFb9v16268@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > But I thought I fixed that in 2.3, by restoring the locale after > initializing GNU readline. Maybe the Mac IDE environment starts up in > a different locale? Would be typical for Mac... I thought I clearly showed command line examples. This is a vanilla Unix Python. Martin: yes, I've got readline installed and yes Python was built from CVS. Just From guido@python.org Wed Dec 11 16:01:23 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 11 Dec 2002 11:01:23 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: Your message of "Wed, 11 Dec 2002 16:45:31 +0100." References: Message-ID: <200212111601.gBBG1Nh16796@pcp02138704pcs.reston01.va.comcast.net> > > But I thought I fixed that in 2.3, by restoring the locale after > > initializing GNU readline. Maybe the Mac IDE environment starts up in > > a different locale? Would be typical for Mac... > > I thought I clearly showed command line examples. This is a vanilla > Unix Python. > > Martin: yes, I've got readline installed and yes Python was built from CVS. Hm. Mayb the code in Modules/readline.c somehow has the setlocale() call #ifdef'ed out? Can you put a printf inside the #ifdef SAVE_LOCALE block to see if it is executed? --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Dec 11 16:36:25 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 11 Dec 2002 17:36:25 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 References: Message-ID: <3DF76989.5070700@lemburg.com> loewis@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Include > In directory sc8-pr-cvs1:/tmp/cvs-serv15966/Include > > Modified Files: > compile.h parsetok.h pyerrors.h pythonrun.h symtable.h > Log Message: > Constify filenames and scripts. Fixes #651362. I remember Tim opposing such changes (and Greg Stein insisting on them ;-). AFAIK, this breaks third party code since some compilers simply reject casting char* to const char*. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Dec 11 16:43:59 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 11 Dec 2002 11:43:59 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: Your message of "Wed, 11 Dec 2002 17:36:25 +0100." <3DF76989.5070700@lemburg.com> References: <3DF76989.5070700@lemburg.com> Message-ID: <200212111643.gBBGhxL17148@pcp02138704pcs.reston01.va.comcast.net> > I remember Tim opposing such changes (and Greg Stein insisting > on them ;-). AFAIK, this breaks third party code since some > compilers simply reject casting char* to const char*. That was years ago. It also sounds like a standard-defying compiler. If this is still an issue, we'll find out during the alpha or beta testing and we can revert if really necessary. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Wed Dec 11 16:53:56 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 17:53:56 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: <20021211143957.GA10597@cthulhu.gerg.ca> References: <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff> <20021211143957.GA10597@cthulhu.gerg.ca> Message-ID: Greg Ward writes: > My attitude is that textwrap should work on European languages, whether > they are encoded in 8-bit "ASCII" or Unicode. Please, don't assume any specific encoding. Why is Latin-1 better than KOI8-R? The only encoding that is truly better than all others is ASCII, since virtually all other encodings have ASCII as a subset (except for the EBCDIC ones, and, with limitations, the ISO-2022 ones). Also, you'll find more-and-more European languages encoded in UTF-8, so your support would be useless and give wrong results. [If you meant to suggest no specific processing for   disregard this comment] > I suspect that passing an arbitrary Unicode string to it is > meaningles -- what the heck does it even mean to wrap a string of > Chinese or Hebrew or Devangari characters? Beats me, and I think > they're out of scope for textwrap. Actually, the Unicode database has "line-breaking properties". Those are not yet incorporated into unicodedata, but that could be used to meaningfully extend the module to Unicode. > So: do I even need to worry about the cornucopia of Unicode whitespace > characters at all? Or can I sweep that can of worms under the rug? > (Pardon the horribly mixed metaphor.) Sweep away. Regards, Martin From martin@v.loewis.de Wed Dec 11 16:56:16 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 17:56:16 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: <200212111601.gBBG1Nh16796@pcp02138704pcs.reston01.va.comcast.net> References: <200212111601.gBBG1Nh16796@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > > Martin: yes, I've got readline installed and yes Python was built from CVS. > > Hm. Mayb the code in Modules/readline.c somehow has the setlocale() > call #ifdef'ed out? Can you put a printf inside the #ifdef > SAVE_LOCALE block to see if it is executed? I'm confused by Just's observation that it happens *only* with -S, in interactive mode. I would have expected that "plain" interactive mode behaves the same, but this is not what Just reported. Regards, Martin From tim@zope.com Wed Dec 11 17:01:32 2002 From: tim@zope.com (Tim Peters) Date: Wed, 11 Dec 2002 12:01:32 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: <3DF76989.5070700@lemburg.com> Message-ID: [M.-A. Lemburg, on constification] > I remember Tim opposing such changes (and Greg Stein insisting > on them ;-). AFAIK, this breaks third party code since some > compilers simply reject casting char* to const char*. A non-broken compiler always allowd passing a T actual argument to a const T formal argument, so I never object to those, and indeed I sneak const into prototype arglists whenever it's convenient. That's a different issue than making all of Python "const correct", which has been a near boundlessly low bang-for-the-buck effort in every project I've seen undertake it. Try passing a const T* actual argument to a T* formal argument and a non-broken compiler will complain, and then either casts have to proliferate, or consts have to proliferate, to stop the errors. Sometimes this is called "const poisoning", at least by objective observers . From martin@v.loewis.de Wed Dec 11 17:17:01 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 18:17:01 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: References: Message-ID: "Tim Peters" writes: > Sometimes this is called "const poisoning", at least by objective > observers . For the specific change, I found quite a lot of const poisoning: in the original bug report, the submitter only requested a single change (for RunString). Changing the entire class of functions, and figuring out the closure of needed changes, was my day's ratio of poison that I can stand without becoming more insane :-) I had to cut it short at two places: PyFile_FromFile got not changed, and neither did the tok_state members. Regards, Martin From dave@boost-consulting.com Wed Dec 11 17:44:03 2002 From: dave@boost-consulting.com (David Abrahams) Date: Wed, 11 Dec 2002 12:44:03 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: ("Tim Peters"'s message of "Wed, 11 Dec 2002 12:01:32 -0500") References: Message-ID: "Tim Peters" writes: > A non-broken compiler always allowd passing a > > T > > actual argument to a > > const T > > formal argument, so I never object to those, and indeed I sneak const into > prototype arglists whenever it's convenient. I don't know if C differs from C++ in this respect, but in C++, const in a prototype argument list has no effect whatsoever (unless modifying the referent of a pointer or reference). So: void f(const int x); is identical to void f(int x); In fact, a very useful idiom is: void f(int x); // prototype. No const; it communicates nothing // useful to the caller ... void f(const int x) // implementation. I know I can always use x { // and get the original argument value ... } > That's a different issue than making all of Python "const correct", which > has been a near boundlessly low bang-for-the-buck effort in every project > I've seen undertake it. Try passing a > > const T* > > actual argument to a > > T* > > formal argument and a non-broken compiler will complain, and then > either casts have to proliferate, or consts have to proliferate, to > stop the errors. Sometimes this is called "const poisoning", at > least by objective observers . There's a long discussion even now on comp.lang.c++.moderated about whether const-correctness is a dogma. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From just@letterror.com Wed Dec 11 17:54:40 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 11 Dec 2002 18:54:40 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: Message-ID: Martin v. L=F6wis wrote: > I'm confused by Just's observation that it happens *only* with -S, in > interactive mode. I would have expected that "plain" interactive mode > behaves the same, but this is not what Just reported. One more time: sh-2.05a$ ./python.exe=20 Python 2.3a0 (#42, Dec 11 2002, 10:33:42)=20 [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; string.whitespace '\t\n\x0b\x0c\r ' >>>=20 sh-2.05a$ ./python.exe -S Python 2.3a0 (#42, Dec 11 2002, 10:33:42)=20 [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; string.whitespace '\t\n\x0b\x0c\r \xa0' >>>=20 sh-2.05a$ LANG=3DC ./python.exe -S Python 2.3a0 (#42, Dec 11 2002, 10:33:42)=20 [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string; string.whitespace '\t\n\x0b\x0c\r ' >>>=20 Guys, I don't care about enough about this problem (if it is one) to dig = any deeper. I just happened to notice the textwrap breakage when I was import= ing lots of arbitrary modules to do some import timing... Just From barry@zope.com Wed Dec 11 18:12:55 2002 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 11 Dec 2002 13:12:55 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 References: Message-ID: <15863.32807.655604.178384@gargle.gargle.HOWL> >>>>> "DA" == David Abrahams writes: DA> There's a long discussion even now on comp.lang.c++.moderated DA> about whether const-correctness is a dogma. Wow, and you think some Python-related threads never die. -Barry From martin@v.loewis.de Wed Dec 11 18:58:11 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: Wed, 11 Dec 2002 19:58:11 +0100 Subject: [Python-Dev] Modules/tclNotify.c Message-ID: <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de> Is this file still in use? I stumbled over it while investigating support for threaded Tcl. If it is not used anymore, can it be deleted? Regards, Martin From guido@python.org Wed Dec 11 19:07:27 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 11 Dec 2002 14:07:27 -0500 Subject: [Python-Dev] Modules/tclNotify.c In-Reply-To: Your message of "Wed, 11 Dec 2002 19:58:11 +0100." <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de> References: <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de> Message-ID: <200212111907.gBBJ7RI18188@pcp02138704pcs.reston01.va.comcast.net> > Is this file still in use? I stumbled over it while investigating > support for threaded Tcl. > > If it is not used anymore, can it be deleted? It was my experiment. I don't rememeber what came of it; I guess it wasn't needed. You should also delete the 'license.terms' if you delete tclNotify.c. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Wed Dec 11 19:16:56 2002 From: tim@zope.com (Tim Peters) Date: Wed, 11 Dec 2002 14:16:56 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: Message-ID: [Tim] >> A non-broken compiler always allowd passing a >> >> T >> >> actual argument to a >> >> const T >> >> ... [David Abrahams] > I don't know if C differs from C++ in this respect, but in C++, const > in a prototype argument list has no effect whatsoever (unless > modifying the referent of a pointer or reference). I meant T above to include cases like T = char*. > So: > > void f(const int x); > > is identical to > > void f(int x); Not in the presence of #define int char* it's not. > In fact, a very useful idiom is: > > void f(int x); // prototype. No const; it communicates nothing > // useful to the caller > > ... > > void f(const int x) // implementation. I know I can always use x > { // and get the original argument value > ... > } All seriousness aside, that's useful in C too, but rarely seen, since the meaning of C code is always self-evident. > ... > There's a long discussion even now on comp.lang.c++.moderated about > whether const-correctness is a dogma. I prefer to think of it as a crock, except when I'm working on embedded C applications that need to know which static data is ROMable. So far that hasn't happened. We came close once at Dragon, until the vendor revealed their compiler didn't support static data at all. That said, I'm happy to add const decorations to Python's C API decls where it helps C++ users, provided we get to stop about a thousand miles short of making anyone truly happy. compromise-is-the-art-of-spreading-misery-ly y'rs - tim From martin@v.loewis.de Wed Dec 11 19:54:07 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 20:54:07 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: References: Message-ID: "Tim Peters" writes: > >> A non-broken compiler always allowd passing a > >> > >> T > >> > >> actual argument to a > >> > >> const T > >> > >> ... > > I meant T above to include cases like T = char*. Of course, if T = char*, then const T = char *const != const char*. Regards, Martin From martin@v.loewis.de Wed Dec 11 19:56:15 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 20:56:15 +0100 Subject: [Python-Dev] Modules/tclNotify.c In-Reply-To: <200212111907.gBBJ7RI18188@pcp02138704pcs.reston01.va.comcast.net> References: <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de> <200212111907.gBBJ7RI18188@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > It was my experiment. I don't rememeber what came of it; I guess it > wasn't needed. You should also delete the 'license.terms' if you > delete tclNotify.c. Thanks for the information! I've now removed both files. Regards, Martin From dave@boost-consulting.com Wed Dec 11 20:44:39 2002 From: dave@boost-consulting.com (David Abrahams) Date: Wed, 11 Dec 2002 15:44:39 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11 In-Reply-To: ("Tim Peters"'s message of "Wed, 11 Dec 2002 14:16:56 -0500") References: Message-ID: "Tim Peters" writes: > [Tim] >>> A non-broken compiler always allowd passing a >>> T >>> actual argument to a >>> const T >>> ... > > [David Abrahams] >> I don't know if C differs from C++ in this respect, but in C++, >> const in a prototype argument list has no effect whatsoever (unless >> modifying the referent of a pointer or reference). > > I meant T above to include cases like T = char*. > >> So: >> void f(const int x); >> is identical to >> void f(int x); > > Not in the presence of > > #define int char* > > it's not. Ouch. Now "int const" != "const int" (and I seriously can't think of any other problems with that #define. Macros to the rescue!!! ... uh, ) > All seriousness aside, that's useful in C too, but rarely seen, since > the meaning of C code is always self-evident. Better than that. It always works perfectly and is trivial to write correctly. Who needs Python when we have C? > That said, I'm happy to add const decorations to Python's C API decls > where it helps C++ users, provided we get to stop about a thousand > miles short of making anyone truly happy. That sure would reduce the number of nasty casts we need to make. const_casts always set C++ users' teeth to chatterin'. > compromise-is-the-art-of-spreading-misery-ly y'rs - tim ...somewhat. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From tdelaney@avaya.com Wed Dec 11 21:45:19 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Thu, 12 Dec 2002 08:45:19 +1100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 Message-ID: > From: M.-A. Lemburg [mailto:mal@lemburg.com] > > I remember Tim opposing such changes (and Greg Stein insisting > on them ;-). AFAIK, this breaks third party code since some > compilers simply reject casting char* to const char*. I must admit I caused this recent constifying mess ... someone complained on python-list and I told him to submit a bug report (in the hope that it would get rejected because I knew the problems it would cause). I submit myself for a spanking. Tim Delaney From martin@v.loewis.de Wed Dec 11 22:06:27 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 23:06:27 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 In-Reply-To: References: Message-ID: "Delaney, Timothy" writes: > I must admit I caused this recent constifying mess ... someone complained on > python-list and I told him to submit a bug report (in the hope that it would > get rejected because I knew the problems it would cause). I (obviously) did not feel that the request was misguided. The second on (on RunString) was probably more effort that it will ever do good, but I am convinced that the change is conceptually right, i.e. the strings used as file names and scripts really should be const, as people will pass string literals. OTOH, there are more places left that should accept const strings, but don't, and I was certainly not out to fix them all. Instead, doing it on user request only seems right to me. In the long run, we get happy users because of such changes, which should be weighed against the few unhappy developers that now have to silence compiler warnings :-) Regards, Martin From mal@lemburg.com Wed Dec 11 22:45:54 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 11 Dec 2002 23:45:54 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: Message-ID: <3DF7C022.3060701@lemburg.com> Martin v. L=F6wis wrote: > "Delaney, Timothy" writes: >=20 >>I must admit I caused this recent constifying mess ... someone complain= ed on >>python-list and I told him to submit a bug report (in the hope that it = would >>get rejected because I knew the problems it would cause). >=20 >=20 > I (obviously) did not feel that the request was misguided. The second > on (on RunString) was probably more effort that it will ever do good, > but I am convinced that the change is conceptually right, i.e. the > strings used as file names and scripts really should be const, as > people will pass string literals. > > OTOH, there are more places left that should accept const strings, but > don't, and I was certainly not out to fix them all. Instead, doing it > on user request only seems right to me. In the long run, we get happy > users because of such changes, which should be weighed against the few > unhappy developers that now have to silence compiler warnings :-) I would accept that point if you could demonstrate a single case where the const nature of the filename actually does any good for the *user*. BTW, how can I silence warnings when writing C code that's supposed to compiler with Python 2.1 and 2.3 ? (passing const char * to a char * API doesn't work for obvious reasons) --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mhammond@skippinet.com.au Wed Dec 11 22:59:07 2002 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 12 Dec 2002 09:59:07 +1100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 In-Reply-To: <3DF7C022.3060701@lemburg.com> Message-ID: [Martin] > > OTOH, there are more places left that should accept const strings, but > > don't, and I was certainly not out to fix them all. Instead, doing it > > on user request only seems right to me. In the long run, we get happy > > users because of such changes, which should be weighed against the few > > unhappy developers that now have to silence compiler warnings :-) [MAL] > I would accept that point if you could demonstrate a single > case where the const nature of the filename actually does any > good for the *user*. Well, it depends how you define "user". When working on win32all, I am a developer. However, when knocking up a simple little extension for Python for some silly reason, I feel more like a user. I'm with Martin on this on; these strings should be const. It has bitten me *lots* of times. Sometimes as I am implementing someone elses API, am passed a "const char *", but can't simply pass it to Python API functions that clearly treat the string as const. Similarly, I tend to personally use const for my own APIs where possible, so often I do this just to find my const-ness getting in the way. I haven't been very vocal about it as I see both side of the const argument - but personally lean towards the "const is good" camp. > BTW, how can I silence warnings when writing C code that's > supposed to compiler with Python 2.1 and 2.3 ? (passing const char * > to a char * API doesn't work for obvious reasons) This won't be a problem - continue to personally ignore the "const" qualifier. Then you are passing non-const strings to both the non-const 2.2 and the const 2.3. It is only a problem if you want to take advantage of const - and if you object in principle, you probably won't . For the rest of us we just gotta make the same decision we do these days about Python 1.5 etc - and that probably means generally ignoring it for a few years too. Mark. From brett@python.org Wed Dec 11 23:28:08 2002 From: brett@python.org (Brett Cannon) Date: Wed, 11 Dec 2002 15:28:08 -0800 (PST) Subject: [Python-Dev] Still want an expected skip file for regrtest.py? Message-ID: I just updated my CVS copy of Python and noticed that regrtest.py was patched. That triggered the memory of summarizing the whole expected skip thread from the last summary. The conclusion to that thread was the idea of having a file that would list tests that could be added to the expected skip list at run-time and thus squash the issue of having to ignore skipped test messages for things you knew would be skipped that were not listed on the platform skip list. Well, with ``dummy_thread(ing)?`` sitting there on SF and knowing that I have a lot of PHP coding coming up, I am up for implementing this (with no specified timeframe). The question is how do you want it implemented? I say it should just be a file you keep in your Lib/test/ directory that is formatted like what you pass in for the -f option to regrtest.py (heck, I plan to copy the code that implements that option). It should always be included without a passed-in option since the whole point of this is that these tests are expected to be skipped. Thus it should have a standardized name (how about "ToSkip.txt"? Maybe a leading underscore?) The other issue is whether the listed tests should supplement or extend the expected tests for the platform. I say it should extend, but the thread originally discussing this said supplement. If both ways are desired, the file could specify either ``..add`` or ``..replace`` (or something similar) to signal which to do (default being to add them). It could be required to be at the top of the file, at the end, or just anywhere; I have no preference. But I think it should just extend it since we have those tests listed for a reason and having to possibly re-list them would be a pain. Or you could all just say "surprise me, Brett" and let me sweat the details. =) -Brett From jeremy@alum.mit.edu Wed Dec 11 23:32:42 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Wed, 11 Dec 2002 18:32:42 -0500 Subject: [Python-Dev] trace.py in std library? Message-ID: <15863.51994.549143.421230@slothrop.zope.com> There's a trace.py script sitting in Tools/scripts that works when imported. I used it today to add code coverage to Zope's test runner script. It took about 30 min. (mostly fixing bugs in trace.py) and seems to work pretty well. Why don't we put this script in the std library so it can actually be imported? I can guess at some of the reasons. The code looks very complicated and has been updated piecemeal by many people of the years. The last update has a bunch of comments with Zooko's name on them and a date. They seem like XXX comments. It might be nice to clean it up before putting in the library, but I'd be in favor of having some minimally functional tool in the library regardless. Jeremy From sholden@holdenweb.com Wed Dec 11 23:47:09 2002 From: sholden@holdenweb.com (Steve Holden) Date: Wed, 11 Dec 2002 18:47:09 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: Message-ID: <005d01c2a16f$9ee64c90$6300000a@holdenweb.com> > [Martin] > > > OTOH, there are more places left that should accept const strings, but > > > don't, and I was certainly not out to fix them all. Instead, doing it > > > on user request only seems right to me. In the long run, we get happy > > > users because of such changes, which should be weighed against the few > > > unhappy developers that now have to silence compiler warnings :-) > > [MAL] > > I would accept that point if you could demonstrate a single > > case where the const nature of the filename actually does any > > good for the *user*. > > Well, it depends how you define "user". When working on win32all, I am a > developer. However, when knocking up a simple little extension for Python > for some silly reason, I feel more like a user. > > I'm with Martin on this on; these strings should be const. It has bitten me > *lots* of times. Sometimes as I am implementing someone elses API, am > passed a "const char *", but can't simply pass it to Python API functions > that clearly treat the string as const. Similarly, I tend to personally use > const for my own APIs where possible, so often I do this just to find my > const-ness getting in the way. > > I haven't been very vocal about it as I see both side of the const > argument - but personally lean towards the "const is good" camp. > > > BTW, how can I silence warnings when writing C code that's > > supposed to compiler with Python 2.1 and 2.3 ? (passing const char * > > to a char * API doesn't work for obvious reasons) > > This won't be a problem - continue to personally ignore the "const" > qualifier. Then you are passing non-const strings to both the non-const 2.2 > and the const 2.3. > > It is only a problem if you want to take advantage of const - and if you > object in principle, you probably won't . For the rest of us we just > gotta make the same decision we do these days about Python 1.5 etc - and > that probably means generally ignoring it for a few years too. > > Mark. Folks: I'm just reposting the latest contribution to this thread in an attempt to stop the messages from appearing as attachments. Don't know whether it will work, or why it was happening in the first place ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Previous .sig file retired to www.homeforoldsigs.com ----------------------------------------------------------------------- From martin@v.loewis.de Wed Dec 11 22:51:12 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 11 Dec 2002 23:51:12 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 In-Reply-To: <3DF7C022.3060701@lemburg.com> References: <3DF7C022.3060701@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > I would accept that point if you could demonstrate a single > case where the const nature of the filename actually does any > good for the *user*. See the original bug report. It avoids having to write const_casts. > BTW, how can I silence warnings when writing C code that's > supposed to compiler with Python 2.1 and 2.3 ? (passing const char * > to a char * API doesn't work for obvious reasons) Can you explain where such silencing would be necessary, for the specific patch in question? It is not, because non of the variables that got const now are ever passed to 2.1 API: they are all internal to the core. Regards, Martin From barry@python.org Wed Dec 11 23:54:11 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 11 Dec 2002 18:54:11 -0500 Subject: [Python-Dev] trace.py in std library? References: <15863.51994.549143.421230@slothrop.zope.com> Message-ID: <15863.53283.298026.362146@gargle.gargle.HOWL> >>>>> "JH" == Jeremy Hylton writes: JH> There's a trace.py script sitting in Tools/scripts that works JH> when imported. I used it today to add code coverage to Zope's JH> test runner script. It took about 30 min. (mostly fixing bugs JH> in trace.py) and seems to work pretty well. Didn't Skip work on some code coverage tool? If so, maybe he can tell us how they compare. -Barry From jeremy@alum.mit.edu (Jeremy Hylton) Wed Dec 11 23:55:41 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) (Jeremy Hylton) Date: Wed, 11 Dec 2002 18:55:41 -0500 Subject: [Python-Dev] trace.py in std library? In-Reply-To: <15863.53283.298026.362146@gargle.gargle.HOWL> References: <15863.51994.549143.421230@slothrop.zope.com> <15863.53283.298026.362146@gargle.gargle.HOWL> Message-ID: <15863.53373.521756.396364@slothrop.zope.com> trace.py is Skip's tool. Jeremy From tim.one@comcast.net Wed Dec 11 23:59:23 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 11 Dec 2002 18:59:23 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 In-Reply-To: <005d01c2a16f$9ee64c90$6300000a@holdenweb.com> Message-ID: > Folks: I'm just reposting the latest contribution to this thread in an > attempt to stop the messages from appearing as attachments. Don't know > whether it will work, Sounds as likely to work as gutting a cat and offering the entrails to Barry. > or why it was happening in the first place It wasn't -- I expect you're the only one who saw attachments. My Outlook 2000 gets into that state sometimes, and is then cured by closing it, using Task Manager to make sure it's gone, and then opening it again. From barry@python.org Thu Dec 12 00:10:09 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 11 Dec 2002 19:10:09 -0500 Subject: [Python-Dev] trace.py in std library? References: <15863.51994.549143.421230@slothrop.zope.com> <15863.53283.298026.362146@gargle.gargle.HOWL> <15863.53373.521756.396364@slothrop.zope.com> Message-ID: <15863.54241.591375.904547@gargle.gargle.HOWL> >>>>> "JH" == Jeremy Hylton writes: JH> trace.py is Skip's tool. So I guess that means they're similar. -Barry From barry@python.org Thu Dec 12 00:12:24 2002 From: barry@python.org (Barry A. Warsaw) Date: Wed, 11 Dec 2002 19:12:24 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: <005d01c2a16f$9ee64c90$6300000a@holdenweb.com> Message-ID: <15863.54376.614967.808943@gargle.gargle.HOWL> >>>>> "TP" == Tim Peters writes: >> Folks: I'm just reposting the latest contribution to this >> thread in an attempt to stop the messages from appearing as >> attachments. Don't know whether it will work, TP> Sounds as likely to work as gutting a cat and offering the TP> entrails to Barry. I'm much more into fried snakes in my computer's guts: http://www.uq.edu.au/education/extra/all.html -Barry From tdelaney@avaya.com Thu Dec 12 00:34:16 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Thu, 12 Dec 2002 11:34:16 +1100 Subject: [Python-Dev] trace.py in std library? Message-ID: > From: barry@python.org [mailto:barry@python.org] > > >>>>> "JH" == Jeremy Hylton writes: > > JH> trace.py is Skip's tool. > > So I guess that means they're similar. I think it's premature to put it into the std library - not least because I've got a version I've been working on for a while which I'd eventually like to get in there if I can ;) Of course, part of that will rest on my employer ... The major problems I had with trace.py are that it is *slow* and that the data it collects is very incomplete. If we are going to have a standard tracing module, I think it needs to be *fast* and the data it collects needs to be complete, and easily transformable to many types of reports. My main use case is automated code coverage reports. If my unit tests take 30 minutes to run with no coverage tool, but 15 hours to run under a coverage tool, the coverage tool is useless for my nightly builds and automated testing. The one I've been working on is a *lot* faster (though still in pure python), dropping from about 3000% performance overhead to about 50%. It also collects pretty much all the data it can (calls, lines, which lines belong to which classes/functions/methods), resolves modules down to a single, most-qualified instance (for example, if import a and import b.a end up importing the same module, both will be represented in the report as b.a) and various other things. It also determines which source lines are executable (for example, multi-line strings will only have one of the lines as executable). It also *almost* works with Jython (doesn't work out executable lines properly). It *also* allows outputting of tracing lines as they are executed (and damn that slows things down). But I don't think it's a candidate yet. Among other things, the tracing functionality is not hooked into each new thread. Not an issue for my current project (which must run single-threaded due to being invoked inside something that must run in a single thread) but it is an issue for a version in the standard library. Tim Delaney From jeremy@alum.mit.edu Thu Dec 12 00:38:19 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Wed, 11 Dec 2002 19:38:19 -0500 Subject: [Python-Dev] trace.py in std library? In-Reply-To: References: Message-ID: <15863.55931.936197.13180@slothrop.zope.com> >>>>> "TD" == Timothy Delaney writes: TD> But I don't think it's a candidate yet. Among other things, the TD> tracing functionality is not hooked into each new thread. Not an TD> issue for my current project (which must run single-threaded due TD> to being invoked inside something that must run in a single TD> thread) but it is an issue for a version in the standard TD> library. Maybe you should checkin what you have now. If it's at least as functional as trace.py, then it's an improvement over what we've got in the std library now. You/we can improve it as 2.3 development proceeds. Jeremy From tdelaney@avaya.com Thu Dec 12 00:46:14 2002 From: tdelaney@avaya.com (Delaney, Timothy) Date: Thu, 12 Dec 2002 11:46:14 +1100 Subject: [Python-Dev] trace.py in std library? Message-ID: > From: Jeremy Hylton [mailto:jeremy@alum.mit.edu] > > >>>>> "TD" == Timothy Delaney writes: > > TD> But I don't think it's a candidate yet. Among other things, the > TD> tracing functionality is not hooked into each new thread. Not an > TD> issue for my current project (which must run single-threaded due > TD> to being invoked inside something that must run in a single > TD> thread) but it is an issue for a version in the standard > TD> library. > > Maybe you should checkin what you have now. If it's at least as > functional as trace.py, then it's an improvement over what we've got > in the std library now. You/we can improve it as 2.3 development > proceeds. Ooh - believe me - you don't want to see the code as it is now. Yuk! Needs some major cleanup (esp. after the hacking I've done to make it work with Jython). No way I would check something that disgusting in where other people could see it ;) Unfortunately, I'm really busy right now. Maybe I'll get to spend some time on it over my 2-week break coming up. Plus as I said, I need to check with my employer - it has been partially developed on company time, and for a company project. Maybe I can get it into my goals and objectives ;) I should use the sandbox though. Unfortunately, I can't compile 2.3 at home ... MinGW still doesn't work, and I don't have a BSD or Linux box set up (the only available machine is currently serving as my internet gateway until I fix another machine). Tim Delaney From skip@pobox.com Thu Dec 12 01:33:09 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 11 Dec 2002 19:33:09 -0600 Subject: [Python-Dev] trace.py in std library? In-Reply-To: <15863.55931.936197.13180@slothrop.zope.com> References: <15863.55931.936197.13180@slothrop.zope.com> Message-ID: <15863.59221.238508.808135@montanaro.dyndns.org> TD> But I don't think it's a candidate yet.... Jeremy> Maybe you should checkin what you have now. If it's at least as Jeremy> functional as trace.py, then it's an improvement over what we've Jeremy> got in the std library now. You/we can improve it as 2.3 Jeremy> development proceeds. As the purported author of trace.py, I agree. Skip From skip@pobox.com Thu Dec 12 01:35:21 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 11 Dec 2002 19:35:21 -0600 Subject: [Python-Dev] trace.py in std library? In-Reply-To: References: Message-ID: <15863.59353.890614.697677@montanaro.dyndns.org> Jeremy> Maybe you should checkin what you have now. Tim> Ooh - believe me - you don't want to see the code as it is Tim> now. So check it into the sandbox (well, after your boss says okay). Failing approval from your employer, can you outline the approach you took, especially how it differs from the approach taken by trace.py? Skip From prabhu@aero.iitm.ernet.in Thu Dec 12 04:10:44 2002 From: prabhu@aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 12 Dec 2002 09:40:44 +0530 Subject: [Python-Dev] trace.py in std library? In-Reply-To: <15863.51994.549143.421230@slothrop.zope.com> References: <15863.51994.549143.421230@slothrop.zope.com> Message-ID: <15864.3140.97153.574886@monster.linux.in> >>>>> "JH" == Jeremy Hylton writes: JH> There's a trace.py script sitting in Tools/scripts that works JH> when imported. I used it today to add code coverage to Zope's JH> test runner script. It took about 30 min. (mostly fixing bugs JH> in trace.py) and seems to work pretty well. JH> Why don't we put this script in the std library so it can JH> actually be imported? FWIW I wrote a yet another trace.py script that I occasionally use to look at call graphs. Its pretty simple code and I have no idea how it compares to Skip's version. I do believe that I wrote to Skip about my code a long while ago but can't remember what he said or if he said anything at all. My version pretty prints the call graph and indents things to make things easier to read. There is no 'coverage' class or anything that gives you stats. There are a few options which are fairly well documented. If you are interested maybe you can take a look here: http://av.stanford.edu/~prabhu/download/python/ There is also a test suite for the code in that directory (test_trace.py). cheers, prabhu From mal@lemburg.com Thu Dec 12 09:21:52 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 10:21:52 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: Message-ID: <3DF85530.7040203@lemburg.com> Mark Hammond wrote: > [Martin] > >>>OTOH, there are more places left that should accept const strings, but >>>don't, and I was certainly not out to fix them all. Instead, doing it >>>on user request only seems right to me. In the long run, we get happy >>>users because of such changes, which should be weighed against the few >>>unhappy developers that now have to silence compiler warnings :-) > > [MAL] > >>I would accept that point if you could demonstrate a single >>case where the const nature of the filename actually does any >>good for the *user*. > > Well, it depends how you define "user". When working on win32all, I am a > developer. However, when knocking up a simple little extension for Python > for some silly reason, I feel more like a user. It is definitely useful for the developer and that's what I wanted to hint at: the user doesn't notice any change by introducing "const". > I'm with Martin on this on; these strings should be const. It has bitten me > *lots* of times. Sometimes as I am implementing someone elses API, am > passed a "const char *", but can't simply pass it to Python API functions > that clearly treat the string as const. Similarly, I tend to personally use > const for my own APIs where possible, so often I do this just to find my > const-ness getting in the way. > > I haven't been very vocal about it as I see both side of the const > argument - but personally lean towards the "const is good" camp. I'm all for using "const" in new APIs too (I've added it to most Unicode APIs for that reason), what worries me is that existing APIs get converted. >>BTW, how can I silence warnings when writing C code that's >>supposed to compiler with Python 2.1 and 2.3 ? (passing const char * >>to a char * API doesn't work for obvious reasons) > > This won't be a problem - continue to personally ignore the "const" > qualifier. Then you are passing non-const strings to both the non-const 2.2 > and the const 2.3. > > It is only a problem if you want to take advantage of const - and if you > object in principle, you probably won't . For the rest of us we just > gotta make the same decision we do these days about Python 1.5 etc - and > that probably means generally ignoring it for a few years too. Will do. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Thu Dec 12 09:30:56 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 10:30:56 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: <3DF7C022.3060701@lemburg.com> Message-ID: <3DF85750.7080109@lemburg.com> Martin v. L=F6wis wrote: > "M.-A. Lemburg" writes: >=20 >=20 >>I would accept that point if you could demonstrate a single >>case where the const nature of the filename actually does any >>good for the *user*. >=20 > See the original bug report. It avoids having to write const_casts. Great and now those who want to avoid warnings have to patch their old code :-( >>BTW, how can I silence warnings when writing C code that's >>supposed to compiler with Python 2.1 and 2.3 ? (passing const char * >>to a char * API doesn't work for obvious reasons) >=20 > Can you explain where such silencing would be necessary, for the > specific patch in question? It is not, because non of the variables > that got const now are ever passed to 2.1 API: they are all internal > to the core. They are ? The pythonrun.h changes were public in 2.1 as well and they include some of the more common APIs like the PyRun_*() APIs. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From =?iso-8859-1?Q?Martin_v._L=F6wis?= Thu Dec 12 09:06:41 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Thu, 12 Dec 2002 10:06:41 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: <3DF85530.7040203@lemburg.com> Message-ID: <000901c2a1bd$c9a638c0$741ee8d9@mira> > > Well, it depends how you define "user". When working on win32all, I am a > > developer. However, when knocking up a simple little extension for Python > > for some silly reason, I feel more like a user. > > It is definitely useful for the developer and that's what > I wanted to hint at: the user doesn't notice any change by > introducing "const". I see that my terminology was confusing: user: software author who uses the Python header files. developer: software author who writes them. Strictly speaking, computer users never "use" Python: they always use some Python application. So the Python "users" are always software authors. Of those, only the authors who write C extensions will notice a change. Sorry for the confusion. > I'm all for using "const" in new APIs too (I've added it to most > Unicode APIs for that reason), what worries me is that existing APIs > get converted. No need to worry. Nothing can break because of that. Regards, Martin From =?iso-8859-1?Q?Martin_v._L=F6wis?= Thu Dec 12 09:11:42 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Thu, 12 Dec 2002 10:11:42 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56 symtable.h,2.10,2.11 References: <3DF7C022.3060701@lemburg.com> <3DF85750.7080109@lemburg.com> Message-ID: <001101c2a1be$7cfbcc50$741ee8d9@mira> > Great and now those who want to avoid warnings have to > patch their old code :-( Why is that? Nobody has to change anything. >> Can you explain where such silencing would be necessary, for the >> specific patch in question? It is not, because non of the variables >> that got const now are ever passed to 2.1 API: they are all internal >> to the core. > They are ? The pythonrun.h changes were public in 2.1 as well > and they include some of the more common APIs like the >PyRun_*() APIs. Right. But those changes will no author of extension modules force to change anything. The PyRun_ functions now expect const char*. Existing code will pass char*. No problem, this is well-formed C. In addition, people who want to pass const char* can now do so without using a cast, which they previously couldn't. The only necessary subsequential change is *inside* the PyRun_ functions, where the types of local variables had to be changed, and the types of PyParse_ functions, etc. Developers of the Python core have to keep this in mind when changing the Python core. Nobody else has to change anything. Regards, Martin From mal@lemburg.com Thu Dec 12 10:35:12 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 11:35:12 +0100 Subject: [Python-Dev] make install fails with current Python CVS Message-ID: <3DF86660.6030808@lemburg.com> I've tried to run a "make install" with todays CVS version, but this fails on my Linux box with the following error: Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep263.py ... Sorry: MemoryError: Is that just my setup or reproducable on other machines as well ? The above file looks pretty innocent to me: #! -*- coding: koi8-r -*- assert u"".encode("utf-8") == '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' assert u"\".encode("utf-8") == '\\\xd0\x9f' -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From just@letterror.com Thu Dec 12 10:42:56 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 12 Dec 2002 11:42:56 +0100 Subject: [Python-Dev] new import hooks & zip import Message-ID: After incorporating some suggestions and fixes from Paul Moore and Neal Norwitz and doing more testing myself, I've decided the importhook/zipimport patch is mature enough to post a proper patch: http://www.python.org/sf/652586 I've renamed two of the three new sys items; the patch note explains. Thorough review would be very welcome. Thanks! Just From =?iso-8859-1?Q?Martin_v._L=F6wis?= Thu Dec 12 10:44:17 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Thu, 12 Dec 2002 11:44:17 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> Message-ID: <00bb01c2a1cb$6bd13160$741ee8d9@mira> > I've tried to run a "make install" with todays CVS version, but > this fails on my Linux box with the following error: > > Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep26 3.py ... > Sorry: MemoryError: That is typically an indication that the codec lookup has failed. Can you try to lookup the koi8-r codec interactively? Regards, Martin From mal@lemburg.com Thu Dec 12 11:01:26 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 12:01:26 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> Message-ID: <3DF86C86.7000508@lemburg.com> Martin v. L=F6wis wrote: >>I've tried to run a "make install" with todays CVS version, but >>this fails on my Linux box with the following error: >> >>Compiling >=20 > /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep2= 6 > 3.py ... >=20 >>Sorry: MemoryError: >=20 >=20 > That is typically an indication that the codec lookup has failed. Can > you try to lookup the koi8-r codec interactively? Hmm, I get: Python/Dev-Python> ./python Python 2.3a0 (#1, Dec 12 2002, 11:48:24) [GCC 2.95.3 20010315 (SuSE)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> unicode('x', 'koi8-r') Traceback (most recent call last): File "", line 1, in ? LookupError: no codec search functions registered: can't find encoding >>> Looking at the output of ./python -vv it appears as if the struct cannot be loaded (which is imported by codecs.py). --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mwh@python.net Thu Dec 12 11:03:31 2002 From: mwh@python.net (Michael Hudson) Date: 12 Dec 2002 11:03:31 +0000 Subject: [Python-Dev] make install fails with current Python CVS In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 11:35:12 +0100" References: <3DF86660.6030808@lemburg.com> Message-ID: <2msmx3o5x8.fsf@starship.python.net> "M.-A. Lemburg" writes: > I've tried to run a "make install" with todays CVS version, but > this fails on my Linux box with the following error: > > Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep263.py ... > Sorry: MemoryError: > > Is that just my setup or reproducable on other machines as well ? I saw that too, but thought it must have been my fault. > The above file looks pretty innocent to me: > > #! -*- coding: koi8-r -*- > assert u"".encode("utf-8") == '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' > assert u"\".encode("utf-8") == '\\\xd0\x9f' It might well be the only file that actually uses the pep263 stuff to affect the interpretation of a unicode string. That would seem to be the place to start looking -- though it imports & compiles file when run as part of the test suite... Hmm... >>> exec open("../Lib/test/test_pep263.py").read() sigh, that was one easy guess. Cheers, M. -- Every now and then, Google doesn't throw up what I need so I start checking Altavista, Yahoo, etc. In almost every single case, I am brutally reminded why I use Google in the first place. -- John Riddoch, asr From mwh@python.net Thu Dec 12 11:07:11 2002 From: mwh@python.net (Michael Hudson) Date: 12 Dec 2002 11:07:11 +0000 Subject: [Python-Dev] make install fails with current Python CVS In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 12:01:26 +0100" References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> Message-ID: <2mpts7o5r4.fsf@starship.python.net> "M.-A. Lemburg" writes: > Martin v. L=F6wis wrote: > >>I've tried to run a "make install" with todays CVS version, but > >>this fails on my Linux box with the following error: [...] > Looking at the output of ./python -vv it appears as if the struct > cannot be loaded (which is imported by codecs.py). Is this an out-of-tree build? I have a feeling you may be being bitten by their current breakage. Weird symptom, though. Cheers, M. -- 59. In English every word can be verbed. Would that it were so in our programming languages. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From mal@lemburg.com Thu Dec 12 11:10:46 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 12:10:46 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> Message-ID: <3DF86EB6.1020200@lemburg.com> M.-A. Lemburg wrote: > Martin v. L=F6wis wrote: >=20 >>> I've tried to run a "make install" with todays CVS version, but >>> this fails on my Linux box with the following error: >>> >>> Compiling >> >> >> /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep= 26 >> 3.py ... >> >>> Sorry: MemoryError: Could this error message be made more helpful :-) >> >> That is typically an indication that the codec lookup has failed. Can >> you try to lookup the koi8-r codec interactively? >=20 >=20 > Hmm, I get: >=20 > Python/Dev-Python> ./python > Python 2.3a0 (#1, Dec 12 2002, 11:48:24) > [GCC 2.95.3 20010315 (SuSE)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> unicode('x', 'koi8-r') > Traceback (most recent call last): > File "", line 1, in ? > LookupError: no codec search functions registered: can't find encoding > >>> >=20 > Looking at the output of ./python -vv it appears as if the struct > cannot be loaded (which is imported by codecs.py). Might help: For some reason I don't see a build/ dir in the Python distribution directory. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Thu Dec 12 11:12:33 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 12:12:33 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> Message-ID: <3DF86F21.6070509@lemburg.com> Michael Hudson wrote: > "M.-A. Lemburg" writes: > > >>Martin v. L=F6wis wrote: >> >>>>I've tried to run a "make install" with todays CVS version, but >>>>this fails on my Linux box with the following error: > > [...] > >>Looking at the output of ./python -vv it appears as if the struct >>cannot be loaded (which is imported by codecs.py). > > > Is this an out-of-tree build? This is a CVS checkout from today. I then tried the usual configure; make install dance and the byte-code compile stopped with the error I posted. > I have a feeling you may be being bitten by their current breakage. It's still broken ? I though you cleared that problem out last week ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mwh@python.net Thu Dec 12 11:39:53 2002 From: mwh@python.net (Michael Hudson) Date: 12 Dec 2002 11:39:53 +0000 Subject: [Python-Dev] make install fails with current Python CVS In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 12:12:33 +0100" References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> Message-ID: <2mn0nbo48m.fsf@starship.python.net> "M.-A. Lemburg" writes: > Michael Hudson wrote: > > "M.-A. Lemburg" writes: > > > > > >>Martin v. L=F6wis wrote: > >> > >>>>I've tried to run a "make install" with todays CVS version, but > >>>>this fails on my Linux box with the following error: > > > > [...] > > > >>Looking at the output of ./python -vv it appears as if the struct > >>cannot be loaded (which is imported by codecs.py). > > > > > > Is this an out-of-tree build? > > This is a CVS checkout from today. I then tried the usual > configure; make install dance and the byte-code compile > stopped with the error I posted. Oh. Then maybe it's something else. > > I have a feeling you may be being bitten by their current breakage. > > It's still broken ? I though you cleared that problem out > last week ? No, I think it's still broken. I handed the problem off to AMK -- don't /think/ he's fixed it yet... Cheers, M. -- My hat is lined with tinfoil for protection in the unlikely event that the droid gets his PowerPoint presentation working. -- Alan W. Frame, alt.sysadmin.recovery From mal@lemburg.com Thu Dec 12 12:21:48 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 13:21:48 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> <2mn0nbo48m.fsf@starship.python.net> Message-ID: <3DF87F5C.7060504@lemburg.com> > "M.-A. Lemburg" writes: >>>I've tried to run a "make install" with todays CVS version, but >>>this fails on my Linux box with the following error: > >Looking at the output of ./python -vv it appears as if the struct >cannot be loaded (which is imported by codecs.py). Ok, I've narrowed down the problem to the missing shared mods struct and _codecs. It seems that the distutils setup.py is not run before installing and compiling the .py files. There are two options: 1. tweak the Makefile targets 2. make struct and _codecs built-in modules I think the second option is more attractive. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From =?iso-8859-1?Q?Martin_v._L=F6wis?= Thu Dec 12 13:19:49 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Thu, 12 Dec 2002 14:19:49 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <3DF86EB6.1020200@lemburg.com> Message-ID: <00e301c2a1e1$25fa9d50$741ee8d9@mira> > >>> Sorry: MemoryError: > > Could this error message be made more helpful :-) I don't know. I tried, and failed. I did not try too hard, though. I just knew what the problem was as I investigated this a couple of times before, to track it down to the codecs lookup facility not working. Regards, Martin From niemeyer@conectiva.com Thu Dec 12 13:24:40 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 12 Dec 2002 11:24:40 -0200 Subject: [Python-Dev] Tarfile support Message-ID: <20021212112440.A25044@ibook.distro.conectiva> Hello! Lars Gustabel has opened a patch (#651082) with the implementation of Tarfile, including documentation and everything. I have already reviewed and sent him some considerations which were fixed (besides the constructor stuff we discussed here). It would be nice if other people reviewed it as well. Thank you! -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From skip@pobox.com Thu Dec 12 14:40:58 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 12 Dec 2002 08:40:58 -0600 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: References: Message-ID: <15864.40954.242269.589277@montanaro.dyndns.org> Just> Thorough review would be very welcome. Scanning the patch summary I see: sys.path_hooks is a list of callable objects that take a string as their only argument. A hook will be called with a sys.path or pkg.__path__ item. It should return an "importer" object (see below), or raise ImportError or return None if it can't deal with the path item. By default, sys.path_hooks only contains the zipimporter type, if the zipimport module is available. Why have it raise ImportError or return None if it can't deal with the path item? Shouldn't there be only one way to report failure? (Or do I misread your explanation?) Skip From just@letterror.com Thu Dec 12 14:53:05 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 12 Dec 2002 15:53:05 +0100 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: <15864.40954.242269.589277@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > Scanning the patch summary I see: > > sys.path_hooks is a list of callable objects that take > a string as their only argument. A hook will be called > with a sys.path or pkg.__path__ item. It should return > an "importer" object (see below), or raise ImportError > or return None if it can't deal with the path item. By > default, sys.path_hooks only contains the zipimporter > type, if the zipimport module is available. > > Why have it raise ImportError or return None if it can't deal with the path > item? Shouldn't there be only one way to report failure? (Or do I misread > your explanation?) No, you read correctly... I'm not sure, I just thought if might be convenient to also be able to return None. I'd be happy to stick with just one way, though (which would be raising ImportError, as that's easiest from __init__ methods). Just From =?iso-8859-1?Q?Martin_v._L=F6wis?= Thu Dec 12 13:25:17 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Thu, 12 Dec 2002 14:25:17 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> <2mn0nbo48m.fsf@starship.python.net> <3DF87F5C.7060504@lemburg.com> Message-ID: <00ef01c2a1e1$e998bd50$741ee8d9@mira> > Ok, I've narrowed down the problem to the missing shared mods > struct and _codecs. It seems that the distutils setup.py is > not run before installing and compiling the .py files. > > There are two options: > 1. tweak the Makefile targets > 2. make struct and _codecs built-in modules > > I think the second option is more attractive. There is a third option: 3. Document that the proper build procedure is "configure;make;make install" Actually, that is documented, so I'd vote "pilot error". That said, I'm always in favour of making more modules builtin again, as this simplifies the build process and may slightly improve performance if the module is imported. Regards, Martin From guido@python.org Thu Dec 12 15:56:05 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 10:56:05 -0500 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: Your message of "Thu, 12 Dec 2002 15:53:05 +0100." References: Message-ID: <200212121556.gBCFu5105528@odiug.zope.com> > Skip Montanaro wrote: > > > Scanning the patch summary I see: > > > > sys.path_hooks is a list of callable objects that take > > a string as their only argument. A hook will be called > > with a sys.path or pkg.__path__ item. It should return > > an "importer" object (see below), or raise ImportError > > or return None if it can't deal with the path item. By > > default, sys.path_hooks only contains the zipimporter > > type, if the zipimport module is available. > > > > Why have it raise ImportError or return None if it can't deal with the path > > item? Shouldn't there be only one way to report failure? (Or do I misread > > your explanation?) > > No, you read correctly... I'm not sure, I just thought if might be > convenient to also be able to return None. I'd be happy to stick > with just one way, though (which would be raising ImportError, as > that's easiest from __init__ methods). Hm, I'd prefer None. With an exception (especially when you're reusing an existing exception) you never know if it was raised intentionally or whether it means a real (unexpected) error -- in the latter case, swallowing the traceback would be a bad idea because it makes diagnosing the real problem hard. ("Why does my zipimport not work? I don't get any errors, but it doesn't work...") --Guido van Rossum (home page: http://www.python.org/~guido/) From edcjones@erols.com Thu Dec 12 16:43:42 2002 From: edcjones@erols.com (Edward C. Jones) Date: Thu, 12 Dec 2002 11:43:42 -0500 Subject: [Python-Dev] os.path.walk generator Message-ID: <3DF8BCBE.9000608@erols.com> I feel that "The Right Way" to walk through a directory tree is with a generator. Therefore I suggest that a generator walk2(topdir, hidden=None) be added to os.path. It yields all the files and directories under (and including) "topdir". If "hidden" is true, then hidden files and directories are included. Thanks, Ed Jones From just@letterror.com Thu Dec 12 16:25:56 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 12 Dec 2002 17:25:56 +0100 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: <200212121556.gBCFu5105528@odiug.zope.com> Message-ID: Guido van Rossum wrote: > Hm, I'd prefer None. The reason I prefer an exception (any exception, could be a new one) is that you can't return None from an __init__ method, and a class/type is the *perfect* candidate for a hook (zipimport.zipimporter is one, too). I'd prefer it if people didn't have to write a __new__ method or a separate factory function. > With an exception (especially when you're > reusing an existing exception) you never know if it was raised > intentionally or whether it means a real (unexpected) error -- in the > latter case, swallowing the traceback would be a bad idea because it > makes diagnosing the real problem hard. ("Why does my zipimport not > work? I don't get any errors, but it doesn't work...") (I think import.c should print a warning if the zipimport hook couldn't be installed, similar to the site.py warning. The hook itself doesn't invoke any Python code, so an ImportError is a 100% sure sign zipimporter can't handle the path.) Unless you do imports *in* the function/class __init__ that is the hook, there's no chance of getting ImportErrors after the hook is installed, so I'm not too worried. The problem is perhaps comparable to old-style sequence iteration: the __getitem__ implementation *could* raise a different IndexError than the author intended. Just From aahz@pythoncraft.com Thu Dec 12 16:26:34 2002 From: aahz@pythoncraft.com (Aahz) Date: Thu, 12 Dec 2002 11:26:34 -0500 Subject: [Python-Dev] os.path.walk generator In-Reply-To: <3DF8BCBE.9000608@erols.com> References: <3DF8BCBE.9000608@erols.com> Message-ID: <20021212162634.GA29004@panix.com> On Thu, Dec 12, 2002, Edward C. Jones wrote: > > I feel that "The Right Way" to walk through a directory tree is with a > generator. Therefore I suggest that a generator > > walk2(topdir, hidden=None) > > be added to os.path. It yields all the files and directories under (and > including) "topdir". If "hidden" is true, then hidden files and > directories are included. python-dev is the wrong place for this discussion; you should post to comp.lang.python and/or submit a feature request to SourceForge. Best of all, post a patch to SF. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From mal@lemburg.com Thu Dec 12 16:27:42 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 17:27:42 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> <2mn0nbo48m.fsf@starship.python.net> <3DF87F5C.7060504@lemburg.com> <00ef01c2a1e1$e998bd50$741ee8d9@mira> Message-ID: <3DF8B8FE.7090905@lemburg.com> Martin v. L=F6wis wrote: >>Ok, I've narrowed down the problem to the missing shared mods >>struct and _codecs. It seems that the distutils setup.py is >>not run before installing and compiling the .py files. >> >>There are two options: >>1. tweak the Makefile targets >>2. make struct and _codecs built-in modules >> >>I think the second option is more attractive. >=20 >=20 > There is a third option: >=20 > 3. Document that the proper build procedure is "configure;make;make > install" That's what I did. It still fails: the reason is that shared mods are built *after* python itself is built and since struct and _codecs are shared mods, compiling the .py files doesn't work since that's done when building the python target (before building the shared mods). I'm +1 on option 2. > Actually, that is documented, so I'd vote "pilot error". >=20 > That said, I'm always in favour of making more modules builtin again, a= s > this simplifies the build process and may slightly improve performance > if the module is imported. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Thu Dec 12 16:28:54 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 11:28:54 -0500 Subject: [Python-Dev] os.path.walk generator In-Reply-To: Your message of "Thu, 12 Dec 2002 11:43:42 EST." <3DF8BCBE.9000608@erols.com> References: <3DF8BCBE.9000608@erols.com> Message-ID: <200212121628.gBCGSs005830@odiug.zope.com> > I feel that "The Right Way" to walk through a directory tree is with a > generator. Therefore I suggest that a generator > > walk2(topdir, hidden=None) > > be added to os.path. It yields all the files and directories under (and > including) "topdir". If "hidden" is true, then hidden files and > directories are included. Submit a patch. I think hiding 'hidden' files should not be up to the walk() generator -- os.listdir() and the existing os.path.walk() don't hide them either. Style nit: don't use None to indicate False. Use False. --Guido van Rossum (home page: http://www.python.org/~guido/) From Paul.Moore@atosorigin.com Thu Dec 12 16:32:14 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Thu, 12 Dec 2002 16:32:14 -0000 Subject: [Python-Dev] new import hooks & zip import Message-ID: <16E1010E4581B049ABC51D4975CEDB8861997F@UKDCX001.uk.int.atosorigin.com> Guido wrote: > > Skip Montanaro wrote: > >=20 > > > Scanning the patch summary I see: > > >=20 > > > sys.path_hooks is a list of callable objects that take=20 > > > a string as their only argument. A hook will be called=20 > > > with a sys.path or pkg.__path__ item. It should return=20 > > > an "importer" object (see below), or raise ImportError=20 > > > or return None if it can't deal with the path item. By=20 > > > default, sys.path_hooks only contains the zipimporter =20 > > > type, if the zipimport module is available. =20 > > > > > > Why have it raise ImportError or return None if it can't > > > deal with the path item? Shouldn't there be only one way to > > > report failure? (Or do I misread your explanation?) > > > > No, you read correctly... I'm not sure, I just thought if > > might be convenient to also be able to return None. I'd > > be happy to stick with just one way, though (which would > > be raising ImportError, as that's easiest from __init__ > > methods). > > Hm, I'd prefer None. With an exception (especially when you're > reusing an existing exception) you never know if it was raised > intentionally or whether it means a real (unexpected) error -- > in the latter case, swallowing the traceback would be a bad > idea because it makes diagnosing the real problem hard. ("Why > does my zipimport not work? I don't get any errors, but it > doesn't work...") The problem is that in 99% of cases (and specifically for zipimporter) the "callable object" in question is a class. It's not possible for a class when called to return a value (the constructor either does its job or raises an exception). So an exception signalling "I don't apply to this path item" is the only way of letting classes be put directly on sys.path_hooks. You could mandate wrappers, but I imagine that def my_importer_wrapper(path): try: return my_importer(path) except ImportError: return None sys.path_hooks.append(my_importer_wrapper) is even more offensive... Actually, I'd say that "I can't handle this" should *always* be signalled by an exception (Only One Way To Do It), but rather than reusing ImportError, that should be a new exception type (I don't know what to call it though - ImportHandlerError?) That way, you don't risk masking real exceptions. Paul. From mal@lemburg.com Thu Dec 12 17:32:41 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 18:32:41 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> <2mn0nbo48m.fsf@starship.python.net> <3DF87F5C.7060504@lemburg.com> <00ef01c2a1e1$e998bd50$741ee8d9@mira> <3DF8B8FE.7090905@lemburg.com> Message-ID: <3DF8C839.4040603@lemburg.com> M.-A. Lemburg wrote: > Martin v. L=F6wis wrote: >=20 >>> Ok, I've narrowed down the problem to the missing shared mods >>> struct and _codecs. It seems that the distutils setup.py is >>> not run before installing and compiling the .py files. >>> >>> There are two options: >>> 1. tweak the Makefile targets >>> 2. make struct and _codecs built-in modules >>> >>> I think the second option is more attractive. >> >> There is a third option: >> >> 3. Document that the proper build procedure is "configure;make;make >> install" >=20 > That's what I did. It still fails:=20 Ah, didn't see the separate "make;" in there. You're right, Martin, I ran "./configure; make install" using a script. The analysis below applies to this way of invoking the make process. > the reason is that shared mods > are built *after* python itself is built and since struct and > _codecs are shared mods, compiling the .py files doesn't work > since that's done when building the python target (before building > the shared mods). I'll check in a fix which makes _codecs a builtin and removes the struct dependency from codecs.py. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Dec 12 17:41:04 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 18:41:04 +0100 Subject: [Python-Dev] os.path.walk generator In-Reply-To: <20021212162634.GA29004@panix.com> References: <3DF8BCBE.9000608@erols.com> <20021212162634.GA29004@panix.com> Message-ID: Aahz writes: > > walk2(topdir, hidden=None) > > > > be added to os.path. It yields all the files and directories under (and > > including) "topdir". If "hidden" is true, then hidden files and > > directories are included. > > python-dev is the wrong place for this discussion; you should post to > comp.lang.python and/or submit a feature request to SourceForge. Best > of all, post a patch to SF. I'd take this back a bit. python-dev is the wrong place to ask for new features, yes. However, it is ok to ask "Might it be acceptable if I implement it?", to which possible answers are "yes", "no", or "write a PEP". For that matter, I'd like to steer attention to python.org/sf/619222. I think any walk modification should take file types into account (a feature which isn't implemented, either). I hope that Python adds support for file types one day, and it would be unfortunate if that would cause introduction of walk3 and walk4. Please take into consideration that the os.path.walk callback can control traversal, by modifying the file list. It should be specified if this is also supported for walk2. Regards, Martin From mal@lemburg.com Thu Dec 12 17:43:01 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 12 Dec 2002 18:43:01 +0100 Subject: [Python-Dev] make install fails with current Python CVS References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <3DF86EB6.1020200@lemburg.com> <00e301c2a1e1$25fa9d50$741ee8d9@mira> Message-ID: <3DF8CAA5.1040905@lemburg.com> Martin v. L=F6wis wrote: >>>>>Sorry: MemoryError: >> >>Could this error message be made more helpful :-) >=20 > I don't know. I tried, and failed. I did not try too hard, though. >=20 > I just knew what the problem was as I investigated this a couple of > times before, to track it down to the codecs lookup facility not > working. Could you tell me where this error is generated ? --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Thu Dec 12 17:54:32 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 18:54:32 +0100 Subject: [Python-Dev] make install fails with current Python CVS In-Reply-To: <3DF8CAA5.1040905@lemburg.com> References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <3DF86EB6.1020200@lemburg.com> <00e301c2a1e1$25fa9d50$741ee8d9@mira> <3DF8CAA5.1040905@lemburg.com> Message-ID: "M.-A. Lemburg" writes: > > I don't know. I tried, and failed. I did not try too hard, though. > > I just knew what the problem was as I investigated this a couple of > > times before, to track it down to the codecs lookup facility not > > working. > > Could you tell me where this error is generated ? That's part of the problem: It was difficult to find, and I forgot now. It may be one of the parser error codes. Regards, Martin From mwh@python.net Thu Dec 12 17:58:29 2002 From: mwh@python.net (Michael Hudson) Date: 12 Dec 2002 17:58:29 +0000 Subject: [Python-Dev] make install fails with current Python CVS In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 18:43:01 +0100" References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <3DF86EB6.1020200@lemburg.com> <00e301c2a1e1$25fa9d50$741ee8d9@mira> <3DF8CAA5.1040905@lemburg.com> Message-ID: <2mk7ifnmpm.fsf@starship.python.net> "M.-A. Lemburg" writes: > Martin v. L=F6wis wrote: > >>>>>Sorry: MemoryError: > >> > >>Could this error message be made more helpful :-) > >=20 > > I don't know. I tried, and failed. I did not try too hard, though. > >=20 > > I just knew what the problem was as I investigated this a couple of > > times before, to track it down to the codecs lookup facility not > > working. > > Could you tell me where this error is generated ? I imagine it's somewhere inside PyUnicode_AsEncodedString which is called from compile.c:parsestr. If you want to expose the latter to Python while you're at it, be my guest (sf feature request 617979). Cheers, M. -- I've even been known to get Marmite *near* my mouth -- but never actually in it yet. Vegamite is right out. UnicodeError: ASCII unpalatable error: vegamite found, ham expected -- Tim Peters, comp.lang.python From dave@boost-consulting.com Thu Dec 12 18:01:57 2002 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 12 Dec 2002 13:01:57 -0500 Subject: [Python-Dev] Possible low-hanging optimization Message-ID: In Boost.Python, extension class instance dictionaries are created only on-demand. It's an obvious win for lots of wrapped C++ class instances, which will never have Python attributes tacked on. This also seems like a potential big win for some kinds of Python instances, like those which use __slots__ but may also have a dictionary (IIRC that is a possible configuration). The code to do this in Boost.Python is small and "just works" transparently, so I thought there might be some interest here for the Python core: I've "C-ified" the code here, sort of on-the-fly, just to give an idea, but it's so simple that any errors can't obscure the intent /too/ much ;-) static PyObject* instance_get_dict(PyObject* op, void*) { bpl_instance* inst = (bpl_instance*)op; if (inst->dict == 0) inst->dict = PyDict_New(); Py_XINCREF(inst->dict); return inst->dict; } static int instance_set_dict(PyObject* op, PyObject* dict, void*) { (bpl_instance)* inst = (bpl_instance*)op; Py_XDECREF(inst->dict); Py_XINCREF(dict); inst->dict = dict return 0; } static PyGetSetDef instance_getsets[] = { /* used as tp_getset */ {"__dict__", instance_get_dict, instance_set_dict, NULL}, {0} }; Useful? -Dave -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From gward@python.net Thu Dec 12 17:34:10 2002 From: gward@python.net (Greg Ward) Date: Thu, 12 Dec 2002 12:34:10 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19 In-Reply-To: References: <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff> <20021211143957.GA10597@cthulhu.gerg.ca> Message-ID: <20021212173410.GA1549@cthulhu.gerg.ca> On 11 December 2002, Martin v. Löwis said: > Please, don't assume any specific encoding. Why is Latin-1 better than > KOI8-R? The only encoding that is truly better than all others is > ASCII, since virtually all other encodings have ASCII as a subset > (except for the EBCDIC ones, and, with limitations, the ISO-2022 > ones). Sorry, I should have said *Western* European languages. > [If you meant to suggest no specific processing for   disregard > this comment] Correct: 0xa0 (the ISO-8859-1 non-breaking space) is now just another character, no matter what your locale says. OK, here's a proposed policy statement for textwrap's I18N support (or lack thereof): textwrap is intended for wrapping ASCII-encoded English-language text for use in fixed-width settings such as email messages and plain text files. It also works for Unicode strings that only contain ASCII characters. And it should work for other European languages that use Latin script and have similar line-breaking conventions for monospaced text as English. textwrap is not intended for use with languages or character sets that do not use Latin characters. I should add that to the docs. (Should probably also mention that it doesn't do hyphenation -- speaking of huge cans of worms!) Greg -- Greg Ward http://www.gerg.ca/ This quote intentionally left blank. From jacobs@penguin.theopalgroup.com Thu Dec 12 18:06:17 2002 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Thu, 12 Dec 2002 13:06:17 -0500 (EST) Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: Message-ID: On Thu, 12 Dec 2002, David Abrahams wrote: > In Boost.Python, extension class instance dictionaries are created > only on-demand. It's an obvious win for lots of wrapped C++ class > instances, which will never have Python attributes tacked on. This > also seems like a potential big win for some kinds of Python > instances, like those which use __slots__ but may also have a > dictionary (IIRC that is a possible configuration). I am fairly confident that CPython doesn't allocate instance dictionaries until they are needed. At one point I verified this, though my brain and the code has likely diverged much since there. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Thu Dec 12 18:07:45 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 13:07:45 -0500 Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: Your message of "Thu, 12 Dec 2002 13:01:57 EST." References: Message-ID: <200212121807.gBCI7jL20642@odiug.zope.com> > In Boost.Python, extension class instance dictionaries are created > only on-demand. It's an obvious win for lots of wrapped C++ class > instances, which will never have Python attributes tacked on. This > also seems like a potential big win for some kinds of Python > instances, like those which use __slots__ but may also have a > dictionary (IIRC that is a possible configuration). The code to do > this in Boost.Python is small and "just works" transparently, so I > thought there might be some interest here for the Python core: > > I've "C-ified" the code here, sort of on-the-fly, just to give an > idea, but it's so simple that any errors can't obscure the intent > /too/ much ;-) > > static PyObject* instance_get_dict(PyObject* op, void*) > { > bpl_instance* inst = (bpl_instance*)op; > if (inst->dict == 0) > inst->dict = PyDict_New(); > Py_XINCREF(inst->dict); > return inst->dict; > } > > static int instance_set_dict(PyObject* op, PyObject* dict, void*) > { > (bpl_instance)* inst = (bpl_instance*)op; > Py_XDECREF(inst->dict); > Py_XINCREF(dict); > inst->dict = dict > return 0; > } > > static PyGetSetDef instance_getsets[] = { /* used as tp_getset */ > {"__dict__", instance_get_dict, instance_set_dict, NULL}, > {0} > }; > > Useful? AFAICT, this is already done for new-style classes. For classic classes, it would be too much of a break with history. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Thu Dec 12 18:11:17 2002 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 12 Dec 2002 13:11:17 -0500 Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: <200212121807.gBCI7jL20642@odiug.zope.com> (Guido van Rossum's message of "Thu, 12 Dec 2002 13:07:45 -0500") References: <200212121807.gBCI7jL20642@odiug.zope.com> Message-ID: Guido van Rossum writes: > AFAICT, this is already done for new-style classes. For classic > classes, it would be too much of a break with history. Hmm, I wonder if my extension classes will inherit this from the new-style classes automatically. Was this feature in 2.2 as released? -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From martin@v.loewis.de Thu Dec 12 18:18:37 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 19:18:37 +0100 Subject: [Python-Dev] Tcl, Tkinter, and threads Message-ID: I have now applied changes to _tkinter to make it work when Tcl was built with --enable-threads. For those of you interested in such kind of stuff, here is the full story. This work was triggered by the bug report http://bugs.debian.org/171353 where pydoc -g crashes Tcl. It turns out that pydoc uses threads, and invokes a button configure command from a thread different from the one where the Tcl interpreter was called. Tcl uses thread-local storage to associate a TkDisplay* with a X Display*, and that TLS was uninitialized in this other thread. Discussion with Tcl people at http://sourceforge.net/tracker/index.php?func=detail&aid=649209&group_id=12997&atid=112997 lead to the conclusion that this is intentional; they call it the "appartment model"; you can use the Tcl interpreter only from the thread that has created it, you cannot pass Tcl_Obj* across thread boundaries, and you cannot invoke Tcl commands from other threads. If Tcl is not built for threads, this all is not relevant, since there is only a single set of "thread-local" data (i.e. it isn't thread-local); the thread API is still there, anyway. To overcome this limitation, I now use Tcl Queues to pass commands from one thread the other; this is also how the Tcl thread extension works. You allocate a Tcl_Event, fill some data, put it into a queue, alert the target thread, and block on a Tcl_Condition. The target thread fetches the event from the queue, invokes the callback function, which performs the Tcl action, and notifies the condition. Passing of results happens to stack variables in the calling thread whose addresses are put into the event. This kind of marshalling now happens for the following APIs: - call: passes the PyObject* args to the target thread. There it is converted into Tcl objects, the command is invoked, and the result is converted back to a Python object. - getvar/setvar/unsetvar: pass the variable name and value, and invoke the Tcl API in the target thread. - createcommand/deletecommand: likewise. For a few APIs, this marshalling is not possible in principle: - mainloop/doonevent: event processing must happen in the interpreter thread, by nature of the Tcl threading model For a larger number of APIs, I found the effort not worthwhile: - globalcall, eval, globaleval, evalfile, record, adderrorinfo, exprstring, exprlong, exprdouble, exprboolean, createfilehandler, deletefilehandler For all these functions, _tkinter will now raise an exception if they are invoked in the wrong thread. A tricky question is what happens if the target thread is not processing events right now, either because it mainloop hasn't been invoked, or because it is busy doing something else. The current code raises an exception if the target interpreter is not in the mainloop, and blocks (potentially indefinitely) if the target process does not unqueue its events. This might cause problems if you create multiple interpreters in one thread: it would be sufficient if one of them processes events. Currently, calls to the other interpreters will raise the exception that the mainloop has not been entered. I hope this won't cause problems in practice, since you rarely have more than one interpreter. With these changes, it would now be possible to build Tcl in threaded mode on Windows. This has both advantages and disadvantages: + It allows to get rid of the Tcl lock, and the nearly-busy wait. - It may cause problems for existing applications if they run into one of the limitations. Of course, those applications will run into the same limitations on Unix if Tcl was build with threads enabled. If Tcl wasn't build with threads enabled, behaviour is nearly unmodified. Python will still invoke the Tcl thread API, but that won't do anything, so there should be no user-visible change. Regards, Martin From guido@python.org Thu Dec 12 18:18:56 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 13:18:56 -0500 Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: Your message of "Thu, 12 Dec 2002 13:11:17 EST." References: <200212121807.gBCI7jL20642@odiug.zope.com> Message-ID: <200212121818.gBCIIuE20736@odiug.zope.com> > > AFAICT, this is already done for new-style classes. For classic > > classes, it would be too much of a break with history. > > Hmm, I wonder if my extension classes will inherit this from the > new-style classes automatically. Was this feature in 2.2 as released? I think so, but you better check to be sure. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Thu Dec 12 18:34:52 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 19:34:52 +0100 Subject: [Python-Dev] PEP 298 Message-ID: I intend to work on the PEP 298 (locked buffer interface) implementation, with the hope that it is not too late to bring this into 2.3. Is there still a chance? If it will not make it into 2.3, I will retract the PEP, because 2.4 is too late for me. Thomas From guido@python.org Thu Dec 12 18:39:50 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 13:39:50 -0500 Subject: [Python-Dev] PEP 298 In-Reply-To: Your message of "12 Dec 2002 19:34:52 +0100." References: Message-ID: <200212121839.gBCIdoM21455@odiug.zope.com> > I intend to work on the PEP 298 (locked buffer interface) > implementation, with the hope that it is not too late to bring this > into 2.3. Is there still a chance? > > If it will not make it into 2.3, I will retract the PEP, > because 2.4 is too late for me. Is the PEP accepted? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Thu Dec 12 18:44:23 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 19:44:23 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <200212121839.gBCIdoM21455@odiug.zope.com> References: <200212121839.gBCIdoM21455@odiug.zope.com> Message-ID: Guido van Rossum writes: > > I intend to work on the PEP 298 (locked buffer interface) > > implementation, with the hope that it is not too late to bring this > > into 2.3. Is there still a chance? > > > > If it will not make it into 2.3, I will retract the PEP, > > because 2.4 is too late for me. > > Is the PEP accepted? How could I know? Marked as Draft, I would say it is ready for pronouncement. Only the implementation is missing. Thomas From guido@python.org Thu Dec 12 18:51:46 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 13:51:46 -0500 Subject: [Python-Dev] PEP 298 In-Reply-To: Your message of "12 Dec 2002 19:44:23 +0100." References: <200212121839.gBCIdoM21455@odiug.zope.com> Message-ID: <200212121851.gBCIpkh21556@odiug.zope.com> > Guido van Rossum writes: > > > > I intend to work on the PEP 298 (locked buffer interface) > > > implementation, with the hope that it is not too late to bring this > > > into 2.3. Is there still a chance? > > > > > > If it will not make it into 2.3, I will retract the PEP, > > > because 2.4 is too late for me. > > > > Is the PEP accepted? > > How could I know? Marked as Draft, I would say it is ready for > pronouncement. Only the implementation is missing. There are doubts about its usefulness. (AKA YAGNI.) Would you also provide implementations for common object types? Otherwise there wouldn't be a point, right? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Dec 12 18:56:23 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 13:56:23 -0500 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: Your message of "Thu, 12 Dec 2002 16:32:14 GMT." <16E1010E4581B049ABC51D4975CEDB8861997F@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB8861997F@UKDCX001.uk.int.atosorigin.com> Message-ID: <200212121856.gBCIuNG21865@odiug.zope.com> > Guido wrote: > > > > Skip Montanaro wrote: > > > > > > > Scanning the patch summary I see: > > > > > > > > sys.path_hooks is a list of callable objects that take > > > > a string as their only argument. A hook will be called > > > > with a sys.path or pkg.__path__ item. It should return > > > > an "importer" object (see below), or raise ImportError > > > > or return None if it can't deal with the path item. By > > > > default, sys.path_hooks only contains the zipimporter > > > > type, if the zipimport module is available. > > > > > > > > Why have it raise ImportError or return None if it can't > > > > deal with the path item? Shouldn't there be only one way to > > > > report failure? (Or do I misread your explanation?) > > > > > > No, you read correctly... I'm not sure, I just thought if > > > might be convenient to also be able to return None. I'd > > > be happy to stick with just one way, though (which would > > > be raising ImportError, as that's easiest from __init__ > > > methods). > > > > Hm, I'd prefer None. With an exception (especially when you're > > reusing an existing exception) you never know if it was raised > > intentionally or whether it means a real (unexpected) error -- > > in the latter case, swallowing the traceback would be a bad > > idea because it makes diagnosing the real problem hard. ("Why > > does my zipimport not work? I don't get any errors, but it > > doesn't work...") [Paul Moore] > The problem is that in 99% of cases (and specifically for > zipimporter) the "callable object" in question is a class. It's > not possible for a class when called to return a value (the > constructor either does its job or raises an exception). So an > exception signalling "I don't apply to this path item" is the > only way of letting classes be put directly on sys.path_hooks. > You could mandate wrappers, but I imagine that > > def my_importer_wrapper(path): > try: > return my_importer(path) > except ImportError: > return None > > sys.path_hooks.append(my_importer_wrapper) > > is even more offensive... > > Actually, I'd say that "I can't handle this" should *always* be > signalled by an exception (Only One Way To Do It), but rather than > reusing ImportError, that should be a new exception type (I don't > know what to call it though - ImportHandlerError?) That way, you > don't risk masking real exceptions. OK, point taken. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Thu Dec 12 19:02:24 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 20:02:24 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <200212121851.gBCIpkh21556@odiug.zope.com> References: <200212121839.gBCIdoM21455@odiug.zope.com> <200212121851.gBCIpkh21556@odiug.zope.com> Message-ID: Guido van Rossum writes: > There are doubts about its usefulness. (AKA YAGNI.) Would you also > provide implementations for common object types? Otherwise there > wouldn't be a point, right? Not really, IMO. It is an interface, not an implementation (except that there are 3 helper functions in abstract.c). I have patches ready for string and unicode objects (trivial), and started a patch for arrayobject. This is more involved, because the array must be locked while a buffer pointer is out somewhere. I could also implement it for mmap objects, there are comparable issues. Maybe the most useful object type would be the upcoming bytes object (PEP 296). It seems, this will not be included in 2.3, but it could also be implemented as a separate extension - impossible for the locked buffer interface, which must be included into the core. If you call YAGNI, it's ok for me, that's why I'm asking. Better save the time in this case... Thomas From guido@python.org Thu Dec 12 19:10:50 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 14:10:50 -0500 Subject: [Python-Dev] PEP 298 In-Reply-To: Your message of "12 Dec 2002 20:02:24 +0100." References: <200212121839.gBCIdoM21455@odiug.zope.com> <200212121851.gBCIpkh21556@odiug.zope.com> Message-ID: <200212121910.gBCJAo222634@odiug.zope.com> > Guido van Rossum writes: > > > There are doubts about its usefulness. (AKA YAGNI.) Would you also > > provide implementations for common object types? Otherwise there > > wouldn't be a point, right? > > Not really, IMO. It is an interface, not an implementation (except that > there are 3 helper functions in abstract.c). > > I have patches ready for string and unicode objects (trivial), and > started a patch for arrayobject. This is more involved, because the > array must be locked while a buffer pointer is out somewhere. > > I could also implement it for mmap objects, there are comparable issues. > > Maybe the most useful object type would be the upcoming bytes object > (PEP 296). It seems, this will not be included in 2.3, but it could also > be implemented as a separate extension - impossible for the locked buffer > interface, which must be included into the core. > > If you call YAGNI, it's ok for me, that's why I'm asking. Better save > the time in this case... Well, I have no personal need for it. If you're the only one, that sounds like a big YAGNI to me... --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Thu Dec 12 19:25:13 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 12 Dec 2002 20:25:13 +0100 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: <200212121856.gBCIuNG21865@odiug.zope.com> Message-ID: [Paul Moore] > > Actually, I'd say that "I can't handle this" should *always* be > > signalled by an exception (Only One Way To Do It), but rather than > > reusing ImportError, that should be a new exception type (I don't > > know what to call it though - ImportHandlerError?) That way, you > > don't risk masking real exceptions. [GvR] > OK, point taken. If we should add a new exception, would it be ok for it to live in the imp module? It would be a fairly specialized exception so it's kindof a shame to pollute the builtin namespace for it. ImportHookError? zipimporter currently raises ZipImportError in this case, but that's a subclass of ImportError... (I'm actually -0 on adding a new exception; I don't think reusing ImportError is all that bad.) Just From skip@pobox.com Thu Dec 12 19:47:53 2002 From: skip@pobox.com (Skip Montanaro) Date: Thu, 12 Dec 2002 13:47:53 -0600 Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: <200212121807.gBCI7jL20642@odiug.zope.com> References: <200212121807.gBCI7jL20642@odiug.zope.com> Message-ID: <15864.59369.517000.126606@montanaro.dyndns.org> Guido> For classic classes, it would be too much of a break with Guido> history. In addition, since most classes exist to associate state with methods that operate on it you'd probably only be delaying the creation of self.__dict__ by a few milliseconds in the common case. Skip From guido@python.org Thu Dec 12 19:47:32 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 14:47:32 -0500 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: Your message of "Thu, 12 Dec 2002 20:25:13 +0100." References: Message-ID: <200212121947.gBCJlWf23496@odiug.zope.com> > [Paul Moore] > > > Actually, I'd say that "I can't handle this" should *always* be > > > signalled by an exception (Only One Way To Do It), but rather than > > > reusing ImportError, that should be a new exception type (I don't > > > know what to call it though - ImportHandlerError?) That way, you > > > don't risk masking real exceptions. > > [GvR] > > OK, point taken. > > If we should add a new exception, would it be ok for it to live in the imp > module? It would be a fairly specialized exception so it's kindof a shame to > pollute the builtin namespace for it. > > ImportHookError? > > zipimporter currently raises ZipImportError in this case, but that's a subclass > of ImportError... (I'm actually -0 on adding a new exception; I don't think > reusing ImportError is all that bad.) Let's try raising ImportError (or a subclass) for now. It's okay for it to live in the imp module. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Thu Dec 12 19:54:14 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 20:54:14 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: Thomas Heller writes: > I intend to work on the PEP 298 (locked buffer interface) > implementation, with the hope that it is not too late to bring this > into 2.3. Is there still a chance? Having looked at the PEP for the first time, I have a number of comments: - I think there must be a way to obtain the address of the buffer on the Python level, see http://tinyurl.com/3h79 Using /dev/poll on Solaris requires a way to create a struct whose address you know. - There must be a Python-level API to obtain a locked buffer. - How will string objects implement that interface? In particular, what prevents the refcount of the string dropping to zero? Regards, Martin From theller@python.net Thu Dec 12 20:06:47 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 21:06:47 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: > > I intend to work on the PEP 298 (locked buffer interface) > > implementation, with the hope that it is not too late to bring this > > into 2.3. Is there still a chance? > > Having looked at the PEP for the first time, I have a number of > comments: [Martin, I rearranged your questions] > - How will string objects implement that interface? In particular, > what prevents the refcount of the string dropping to zero? > It is required to unlock the buffer again after you don't need it any longer. You do this by calling the releaselockedbuffer() function with the pointer to the original python object. So the object haas to keep alive. > > - I think there must be a way to obtain the address of the buffer on > the Python level, see > > http://tinyurl.com/3h79 > > Using /dev/poll on Solaris requires a way to create a struct whose > address you know. Hm, the usual hacky way is to use an array object and call buffer_info() on it. > > - There must be a Python-level API to obtain a locked buffer. > That may be, but the PEP only describes the interface. Extensions may implement a 'locked buffer object', or whatever, and expose the address to Python. Dangerous, though, because then you can separate the python oobject pointer and the buffer pointer. Thomas From martin@v.loewis.de Thu Dec 12 20:18:52 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 21:18:52 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: Thomas Heller writes: > > - How will string objects implement that interface? In particular, > > what prevents the refcount of the string dropping to zero? > > > > It is required to unlock the buffer again after you don't need it > any longer. You do this by calling the releaselockedbuffer() function > with the pointer to the original python object. So the object haas to > keep alive. This is flawed. According to the PEP, an object which has acquirelockedbuffer called will live forever: # Retrieving a buffer from an object puts this object in a locked # state during which the buffer may not be freed, resized, or # reallocated. But now you are saying that the object can be released even in locked state. > > Using /dev/poll on Solaris requires a way to create a struct whose > > address you know. > > Hm, the usual hacky way is to use an array object and call buffer_info() > on it. I see. I didn't know that. However, having to copy a string to an array to obtain its address is not obvious; putting it into the buffer objects seems more logical - especially if the buffer claims to be *locked*. > That may be, but the PEP only describes the interface. Extensions > may implement a 'locked buffer object', or whatever, and expose the > address to Python. Dangerous, though, because then you can separate > the python oobject pointer and the buffer pointer. If it is just the interface, I'm -1. You don't need a PEP to define an interface in Python - just establish an interface in the types you care about. If you really mean to implement it primarily for some type of yours, just do so (it' can't be *exactly* this interface, but you can have the same operations, just in a different place). Regards, Martin From guido@python.org Thu Dec 12 20:21:01 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 15:21:01 -0500 Subject: [Python-Dev] Tcl, Tkinter, and threads In-Reply-To: Your message of "12 Dec 2002 19:18:37 +0100." References: Message-ID: <200212122021.gBCKL1T24010@odiug.zope.com> > I have now applied changes to _tkinter to make it work when Tcl was > built with --enable-threads. For those of you interested in such kind > of stuff, here is the full story. Good work. Thanks for taking over maintenance of _tkinter! --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Thu Dec 12 20:22:10 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 21:22:10 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: <65tzasy5.fsf@python.net> martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes: > Having looked at the PEP for the first time, I have a number of > comments: > > - I think there must be a way to obtain the address of the buffer on > the Python level, see > > http://tinyurl.com/3h79 > > Using /dev/poll on Solaris requires a way to create a struct whose > address you know. Do you know my ctypes module? http://tinyurl.com/2tyx It allows to create Python objects which internally look like C data types. Able to access and create structures, unions, pointers, and such. And you do it (IMO) in a natural way. The released package currently only runs under windows, although I have a prototype working under Linux. Thomas From guido@python.org Thu Dec 12 20:26:07 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 15:26:07 -0500 Subject: [Python-Dev] Tarfile support In-Reply-To: Your message of "Thu, 12 Dec 2002 11:24:40 -0200." <20021212112440.A25044@ibook.distro.conectiva> References: <20021212112440.A25044@ibook.distro.conectiva> Message-ID: <200212122026.gBCKQ7x24337@odiug.zope.com> > Lars Gustabel has opened a patch (#651082) with the implementation of > Tarfile, including documentation and everything. I have already reviewed > and sent him some considerations which were fixed (besides the > constructor stuff we discussed here). It would be nice if other people > reviewed it as well. No time to review the code, but the feature set and docs look very good. Is there any chance that this can be contributed to the PSF under the standard PSF license? --Guido van Rossum (home page: http://www.python.org/~guido/) From niemeyer@conectiva.com Thu Dec 12 20:29:54 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 12 Dec 2002 18:29:54 -0200 Subject: [Python-Dev] Tarfile support In-Reply-To: <200212122026.gBCKQ7x24337@odiug.zope.com> References: <20021212112440.A25044@ibook.distro.conectiva> <200212122026.gBCKQ7x24337@odiug.zope.com> Message-ID: <20021212182954.A30768@ibook.distro.conectiva> > No time to review the code, but the feature set and docs look very > good. > > Is there any chance that this can be contributed to the PSF under the > standard PSF license? Lars, can you please give us your position about this? -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From theller@python.net Thu Dec 12 20:31:21 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 21:31:21 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes: > Thomas Heller writes: > > > > - How will string objects implement that interface? In particular, > > > what prevents the refcount of the string dropping to zero? > > > > > > > It is required to unlock the buffer again after you don't need it > > any longer. You do this by calling the releaselockedbuffer() function > > with the pointer to the original python object. So the object haas to > > keep alive. > > This is flawed. According to the PEP, an object which has > acquirelockedbuffer called will live forever: > > # Retrieving a buffer from an object puts this object in a locked > # state during which the buffer may not be freed, resized, or > # reallocated. > > But now you are saying that the object can be released even in locked > state. > But the PEP it continues to say: The object must be unlocked again by releasing the buffer if it's no longer used by calling another function in the locked buffer interface. Also, an easy solution to keep the string object alive forever would be to Py_INCREF() in the acquirebuffer() function, and Py_DECREF() in releasebuffer() again. > > > Using /dev/poll on Solaris requires a way to create a struct whose > > > address you know. > > > > Hm, the usual hacky way is to use an array object and call buffer_info() > > on it. > > I see. I didn't know that. However, having to copy a string to an > array to obtain its address is not obvious; putting it into the buffer > objects seems more logical - especially if the buffer claims to be > *locked*. > > > That may be, but the PEP only describes the interface. Extensions > > may implement a 'locked buffer object', or whatever, and expose the > > address to Python. Dangerous, though, because then you can separate > > the python oobject pointer and the buffer pointer. > > If it is just the interface, I'm -1. You don't need a PEP to define an > interface in Python - just establish an interface in the types you > care about. If you really mean to implement it primarily for some type > of yours, just do so (it' can't be *exactly* this interface, but you > can have the same operations, just in a different place). That's what I have done, since I need it *now*, and in 2.2. I just did not want the PEP to collect too much dust: it should either be implemented or just die. Once again: the whole point of the PEP is to avoid the flaws of the 'old' buffer interface. Thomas From martin@v.loewis.de Thu Dec 12 20:42:17 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 21:42:17 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: Thomas Heller writes: > But the PEP it continues to say: > > The object must be unlocked again by releasing the buffer if it's > no longer used by calling another function in the locked buffer > interface. This is counter-intuitive, and under-specified. What does it mean that "it's no longer used"? My code uses the buffer until system exit, so I never need to release the buffer. I don't release the string, either, since that was an element of ParseTuple, of which I only acquired the buffer. It doesn't talk about the object "being used", just "the buffer". The spec should not prescribe under what conditions some API must be invoked, but instead it should describe what happens if the API is invoked. > Also, an easy solution to keep the string object alive forever > would be to Py_INCREF() in the acquirebuffer() function, and > Py_DECREF() in releasebuffer() again. I would prefer such an implementation strategy, and I would prefer if the spec clarifies that the caller of acquire will keep the object alive atleast until the corresponding release is called. That, of course, means that it is no longer an option to not implement release. > > If it is just the interface, I'm -1. You don't need a PEP to define an > > interface in Python - just establish an interface in the types you > > care about. If you really mean to implement it primarily for some type > > of yours, just do so (it' can't be *exactly* this interface, but you > > can have the same operations, just in a different place). > > That's what I have done, since I need it *now*, and in 2.2. Ok, so this actually *is* an interface to some existing object. Which one? > I just did not want the PEP to collect too much dust: it should > either be implemented or just die. Then I would suggest that you withdraw it. To me, it sounds useless unless significantly enhanced. Regards, Martin From theller@python.net Thu Dec 12 20:48:15 2002 From: theller@python.net (Thomas Heller) Date: 12 Dec 2002 21:48:15 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: Message-ID: > > I just did not want the PEP to collect too much dust: it should > > either be implemented or just die. > > Then I would suggest that you withdraw it. To me, it sounds useless > unless significantly enhanced. > If there is still interest, it could be improved. If not, it should better be withdrawn. I'll wait if more responses com up. Thanks for the review, Thomas From martin@v.loewis.de Thu Dec 12 20:45:34 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 21:45:34 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <65tzasy5.fsf@python.net> References: <65tzasy5.fsf@python.net> Message-ID: Thomas Heller writes: > Do you know my ctypes module? http://tinyurl.com/2tyx Thanks for the pointer. I didn't know it; it looks quite powerful. Regards, Martin From dave@boost-consulting.com Thu Dec 12 21:09:09 2002 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 12 Dec 2002 16:09:09 -0500 Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: <15864.59369.517000.126606@montanaro.dyndns.org> (Skip Montanaro's message of "Thu, 12 Dec 2002 13:47:53 -0600") References: <200212121807.gBCI7jL20642@odiug.zope.com> <15864.59369.517000.126606@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > Guido> For classic classes, it would be too much of a break with > Guido> history. > > In addition, since most classes exist to associate state with methods that > operate on it you'd probably only be delaying the creation of self.__dict__ > by a few milliseconds in the common case. That's why I mentioned that it might help for new-style classes with __slots__. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From guido@python.org Thu Dec 12 21:57:45 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 16:57:45 -0500 Subject: [Python-Dev] Still want an expected skip file for regrtest.py? In-Reply-To: Your message of "Wed, 11 Dec 2002 15:28:08 PST." References: Message-ID: <200212122157.gBCLvjC02221@odiug.zope.com> > I just updated my CVS copy of Python and noticed that regrtest.py was > patched. That triggered the memory of summarizing the whole expected skip > thread from the last summary. The conclusion to that thread was the idea > of having a file that would list tests that could be added to the > expected skip list at run-time and thus squash the issue of having to > ignore skipped test messages for things you knew would be skipped that > were not listed on the platform skip list. > > Well, with ``dummy_thread(ing)?`` sitting there on SF and knowing that I > have a lot of PHP coding coming up, I am up for implementing this (with no > specified timeframe). The question is how do you want it implemented? I > say it should just be a file you keep in your Lib/test/ directory that is > formatted like what you pass in for the -f option to regrtest.py (heck, I > plan to copy the code that implements that option). It should always be > included without a passed-in option since the whole point of this is that > these tests are expected to be skipped. Thus it should have a > standardized name (how about "ToSkip.txt"? Maybe a leading underscore?) I leave the design mostly up to you. I think it should use a file in the current directory -- that's where you'd type "make test" anyway, and when building multiple versions (e.g. a debug version or different platforms) it may actually be useful to be able to have multiple "skips" files. (When using VPATH, that is.) > The other issue is whether the listed tests should supplement or extend > the expected tests for the platform. I say it should extend, but the > thread originally discussing this said supplement. If both ways are > desired, the file could specify either ``..add`` or ``..replace`` (or > something similar) to signal which to do (default being to add them). It > could be required to be at the top of the file, at the end, or just > anywhere; I have no preference. But I think it should just extend it > since we have those tests listed for a reason and having to possibly > re-list them would be a pain. I expect that the most common use would be for the file to augment the built-in list. Maybe entries with a leading minus could be deducted from the built-in list. > Or you could all just say "surprise me, Brett" and let me sweat the > details. =) I'll say "amaze me, Brett." :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Thu Dec 12 22:46:17 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 12 Dec 2002 23:46:17 +0100 Subject: [Python-Dev] Tcl, Tkinter, and threads In-Reply-To: <200212122021.gBCKL1T24010@odiug.zope.com> References: <200212122021.gBCKL1T24010@odiug.zope.com> Message-ID: Guido van Rossum writes: > Good work. Thanks for taking over maintenance of _tkinter! This reminds me that I forgot the acknowledgements in my previous message: I'd like to thank Matthias Klose of the Debian project for coordinating the communication, and recherching various bits of information, and Jeffrey Hobbs and Zoran Vasiljevic of the Tcl project for giving valuable hints on how to approach this problem. Regards, Martin From lars@gustaebel.de Thu Dec 12 23:35:05 2002 From: lars@gustaebel.de (Lars =?iso-8859-1?q?Gust=E4bel?=) Date: Fri, 13 Dec 2002 00:35:05 +0100 Subject: [Python-Dev] Tarfile support In-Reply-To: <20021212182954.A30768@ibook.distro.conectiva> References: <20021212112440.A25044@ibook.distro.conectiva> <200212122026.gBCKQ7x24337@odiug.zope.com> <20021212182954.A30768@ibook.distro.conectiva> Message-ID: <200212130034.39814.lars@gustaebel.de> > > No time to review the code, but the feature set and docs look very > > good. > > > > Is there any chance that this can be contributed to the PSF under the > > standard PSF license? > > Lars, can you please give us your position about this? I must admit that tarfile.py's licensing does not really matter to me. If= =20 changing its license to PSF makes people's life easier, it is absolutely=20 okay. --=20 Lars Gust=E4bel lars@gustaebel.de From guido@python.org Thu Dec 12 23:59:38 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 18:59:38 -0500 Subject: [Python-Dev] Tarfile support In-Reply-To: Your message of "Fri, 13 Dec 2002 00:35:05 +0100." <200212130034.39814.lars@gustaebel.de> References: <20021212112440.A25044@ibook.distro.conectiva> <200212122026.gBCKQ7x24337@odiug.zope.com> <20021212182954.A30768@ibook.distro.conectiva> <200212130034.39814.lars@gustaebel.de> Message-ID: <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net> > > > No time to review the code, but the feature set and docs look very > > > good. > > > > > > Is there any chance that this can be contributed to the PSF under the > > > standard PSF license? > > > > Lars, can you please give us your position about this? > > I must admit that tarfile.py's licensing does not really matter to me. If > changing its license to PSF makes people's life easier, it is absolutely > okay. Great! I strongly prefer contributions under the standard PSF license, so redistributors won't have to worry about separate licensing conditions of individual components. There's a draft contribution agreement at the PSF website: http://www.python.org/psf/psf-contributor-agreement.html --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Fri Dec 13 00:21:32 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Thu, 12 Dec 2002 16:21:32 -0800 (PST) Subject: [Python-Dev] Still want an expected skip file for regrtest.py? In-Reply-To: <200212122157.gBCLvjC02221@odiug.zope.com> References: <200212122157.gBCLvjC02221@odiug.zope.com> Message-ID: [Guido van Rossum] > I leave the design mostly up to you. I think it should use a file in > the current directory -- that's where you'd type "make test" anyway, > and when building multiple versions (e.g. a debug version or different > platforms) it may actually be useful to be able to have multiple > "skips" files. (When using VPATH, that is.) > Using the current directory works for me. As for multiple skips, I guess I could determine possible file names based on the name of the platform ("ToSkipDARWIN.txt", "ToSkipDEBUG.txt", etc.). Think I am going to have to read up on VPATH... > I expect that the most common use would be for the file to augment the > built-in list. Maybe entries with a leading minus could be deducted > from the built-in list. > Sounds reasonable. I can go with that. Might require tests to be on individual lines (-f option just does a ``line.split()``). > > Or you could all just say "surprise me, Brett" and let me sweat the > > details. =) > > I'll say "amaze me, Brett." :-) > Better not mess this up then. =) -Brett From guido@python.org Fri Dec 13 01:05:21 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 12 Dec 2002 20:05:21 -0500 Subject: [Python-Dev] Still want an expected skip file for regrtest.py? In-Reply-To: Your message of "Thu, 12 Dec 2002 16:21:32 PST." References: <200212122157.gBCLvjC02221@odiug.zope.com> Message-ID: <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net> > As for multiple skips, I guess I could determine possible file names based > on the name of the platform ("ToSkipDARWIN.txt", "ToSkipDEBUG.txt", etc.). > Think I am going to have to read up on VPATH... No, you misunderstand. When one builds for multiple platforms, each platform has its own directory, which is the current directory when you build (and test) for that platform, so the skip file can simply have a fixed name in the current directory. --Guido van Rossum (home page: http://www.python.org/~guido/) From brett@python.org Fri Dec 13 01:58:18 2002 From: brett@python.org (Brett Cannon) Date: Thu, 12 Dec 2002 17:58:18 -0800 (PST) Subject: [Python-Dev] Still want an expected skip file for regrtest.py? In-Reply-To: <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net> References: <200212122157.gBCLvjC02221@odiug.zope.com> <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > > As for multiple skips, I guess I could determine possible file names based > > on the name of the platform ("ToSkipDARWIN.txt", "ToSkipDEBUG.txt", etc.). > > Think I am going to have to read up on VPATH... > > No, you misunderstand. When one builds for multiple platforms, each > platform has its own directory, which is the current directory when > you build (and test) for that platform, so the skip file can simply > have a fixed name in the current directory. > Ah, gotcha. So then a single, fixed name will handle it since people can just stick their file in the directories with the appropriate changes for that platform. Nifty. Now I just need a good name. ToSkip.txt? SkipTests.txt? TestsToSkip.txt? Eh, not worth worrying about this exact moment. -Brett From martin@v.loewis.de Fri Dec 13 09:05:28 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 13 Dec 2002 10:05:28 +0100 Subject: [Python-Dev] Still want an expected skip file for regrtest.py? In-Reply-To: References: <200212122157.gBCLvjC02221@odiug.zope.com> <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Brett Cannon writes: > Now I just need a good name. ToSkip.txt? SkipTests.txt? > TestsToSkip.txt? Eh, not worth worrying about this exact moment. The exact file name won't matter, but PleaseNoCamelNames. This is not a Wiki :-) Regards, Martin From mwh@python.net Fri Dec 13 10:33:13 2002 From: mwh@python.net (Michael Hudson) Date: 13 Dec 2002 10:33:13 +0000 Subject: [Python-Dev] Possible low-hanging optimization In-Reply-To: David Abrahams's message of "Thu, 12 Dec 2002 16:09:09 -0500" References: <200212121807.gBCI7jL20642@odiug.zope.com> <15864.59369.517000.126606@montanaro.dyndns.org> Message-ID: <2mhedinr86.fsf@starship.python.net> David Abrahams writes: > Skip Montanaro writes: > > > Guido> For classic classes, it would be too much of a break with > > Guido> history. > > > > In addition, since most classes exist to associate state with methods that > > operate on it you'd probably only be delaying the creation of self.__dict__ > > by a few milliseconds in the common case. > > That's why I mentioned that it might help for new-style classes with > __slots__. And why everyone seems to think we do it already in that case but isn't quite sure :) (I'm in that camp too). Cheers, M. -- Just point your web browser at http://www.python.org/search/ and look for "program", "doesn't", "work", or "my". Whenever you find someone else whose program didn't work, don't do what they did. Repeat as needed. -- Tim Peters, on python-help, 16 Jun 1998 From lars@gustaebel.de Fri Dec 13 11:34:19 2002 From: lars@gustaebel.de (Lars =?iso-8859-1?q?Gust=E4bel?=) Date: Fri, 13 Dec 2002 12:34:19 +0100 Subject: [Python-Dev] Tarfile support In-Reply-To: <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net> References: <20021212112440.A25044@ibook.distro.conectiva> <200212130034.39814.lars@gustaebel.de> <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212131233.47430.lars@gustaebel.de> > Great! I strongly prefer contributions under the standard PSF > license, so redistributors won't have to worry about separate > licensing conditions of individual components. > > There's a draft contribution agreement at the PSF website: > > http://www.python.org/psf/psf-contributor-agreement.html Ok. If I understand all this correctly, the section entitled "Python=20 Contribution Licensing Agreement Version 1.0" is the appropriate one for = this=20 particular case. If this is right, the PSF will soon get a letter from me. Any hints on ho= w to=20 describe the contribution accurately? --=20 Lars Gust=E4bel lars@gustaebel.de From Jack.Jansen@cwi.nl Fri Dec 13 12:33:16 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Fri, 13 Dec 2002 13:33:16 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <200212121910.gBCJAo222634@odiug.zope.com> Message-ID: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> On Thursday, Dec 12, 2002, at 20:10 Europe/Amsterdam, Guido van Rossum wrote: >> If you call YAGNI, it's ok for me, that's why I'm asking. Better save >> the time in this case... > > Well, I have no personal need for it. If you're the only one, that > sounds like a big YAGNI to me... Well... Note that you are currently living dangerously if you're using an s# format and access that buffer again after a Py_BEGIN_ALLOW_THREADS. If the object passed from Python was a mutable object with a buffer interface there's a chance that another thread has moved the underlying data. The PyArg_Parse replacement I would like to do (for which I have had, unfortunately, absolutely no support up to now, and which I don't feel up to designing all by myself) could fix this with PEP298: it would use the locked buffer interface if available on the object, and release the locked buffer after your method returns. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido@python.org Fri Dec 13 13:32:40 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 13 Dec 2002 08:32:40 -0500 Subject: [Python-Dev] Tarfile support In-Reply-To: Your message of "Fri, 13 Dec 2002 12:34:19 +0100." <200212131233.47430.lars@gustaebel.de> References: <20021212112440.A25044@ibook.distro.conectiva> <200212130034.39814.lars@gustaebel.de> <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net> <200212131233.47430.lars@gustaebel.de> Message-ID: <200212131332.gBDDWeF02668@pcp02138704pcs.reston01.va.comcast.net> > > Great! I strongly prefer contributions under the standard PSF > > license, so redistributors won't have to worry about separate > > licensing conditions of individual components. > > > > There's a draft contribution agreement at the PSF website: > > > > http://www.python.org/psf/psf-contributor-agreement.html > > Ok. If I understand all this correctly, the section entitled "Python > Contribution Licensing Agreement Version 1.0" is the appropriate one > for this particular case. If this is right, the PSF will soon get a > letter from me. Any hints on how to describe the contribution > accurately? Yup, that's the one. I'd say "the tarfile.py module and its documentation and unit tests" would cover your contribution adequately. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Fri Dec 13 14:02:02 2002 From: theller@python.net (Thomas Heller) Date: 13 Dec 2002 15:02:02 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> Message-ID: Jack Jansen writes: > On Thursday, Dec 12, 2002, at 20:10 Europe/Amsterdam, Guido van Rossum > wrote: > > >> If you call YAGNI, it's ok for me, that's why I'm asking. Better save > >> the time in this case... > > > > Well, I have no personal need for it. If you're the only one, that > > sounds like a big YAGNI to me... > > Well... Note that you are currently living dangerously if you're using > an s# format and access that buffer again after a > Py_BEGIN_ALLOW_THREADS. If the object passed from Python was a mutable > object with a buffer interface there's a chance that another thread > has moved the underlying data. It can even by the same thread, executing an innocent Py_DECREF() can execute arbitrary Python code. Thinking of it, it's not clear what you are allowed to do after PyArg_Parse() if you still need the 'parsed pointers'. And that's the main point of the PEP: the 'old' buffer interface is nearly useless if you want too be safe. Thomas From skip@pobox.com Fri Dec 13 14:51:18 2002 From: skip@pobox.com (Skip Montanaro) Date: Fri, 13 Dec 2002 08:51:18 -0600 Subject: [Python-Dev] Still want an expected skip file for regrtest.py? In-Reply-To: References: <200212122157.gBCLvjC02221@odiug.zope.com> <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15865.62438.953094.720317@montanaro.dyndns.org> Brett Cannon writes: > Now I just need a good name. ToSkip.txt? SkipTests.txt? > TestsToSkip.txt? Eh, not worth worrying about this exact moment. I'm always suspicious when things get named after me. Will I have to support them? What about bretts_tests.txt? ;-) Skip From guido@python.org Fri Dec 13 15:19:56 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 13 Dec 2002 10:19:56 -0500 Subject: [Python-Dev] PEP 298 In-Reply-To: Your message of "13 Dec 2002 15:02:02 +0100." References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> Message-ID: <200212131519.gBDFJvM03149@odiug.zope.com> > > Well... Note that you are currently living dangerously if you're using > > an s# format and access that buffer again after a > > Py_BEGIN_ALLOW_THREADS. If the object passed from Python was a mutable > > object with a buffer interface there's a chance that another thread > > has moved the underlying data. > > It can even by the same thread, executing an innocent Py_DECREF() > can execute arbitrary Python code. Thinking of it, it's not clear > what you are allowed to do after PyArg_Parse() if you still need > the 'parsed pointers'. > > And that's the main point of the PEP: the 'old' buffer interface > is nearly useless if you want too be safe. Except that we can't force everyone to change at once, so we're stuck with this... Can you find an actual case where this could be used to cause a segfault? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Fri Dec 13 15:28:56 2002 From: theller@python.net (Thomas Heller) Date: 13 Dec 2002 16:28:56 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <200212131519.gBDFJvM03149@odiug.zope.com> References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> <200212131519.gBDFJvM03149@odiug.zope.com> Message-ID: <65tyos3r.fsf@python.net> > > And that's the main point of the PEP: the 'old' buffer interface > > is nearly useless if you want too be safe. > > Except that we can't force everyone to change at once, so we're stuck > with this... > > Can you find an actual case where this could be used to cause a > segfault? I'm not sure about core python, if one doesn't use the buffer object. (Um, I probably shouldn't have mentioned this.) And not anymore in my own extensions, but that's because I implemented my own private scheme. Thomas From guido@python.org Fri Dec 13 15:48:53 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 13 Dec 2002 10:48:53 -0500 Subject: [Python-Dev] PEP 298 In-Reply-To: Your message of "Fri, 13 Dec 2002 13:33:16 +0100." <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl> Message-ID: <200212131548.gBDFmrm09751@odiug.zope.com> > The PyArg_Parse replacement I would like to do (for which I have had, > unfortunately, absolutely no support up to now, and which I don't feel > up to designing all by myself) could fix this with PEP298: it would use > the locked buffer interface if available on the object, and release the > locked buffer after your method returns. I can't give you much more than moral support. :-( It sounds like it would be a Big Change though; are you sure you're up to this for Python 2.3? --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Dec 13 18:19:11 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 13 Dec 2002 13:19:11 -0500 Subject: [Python-Dev] new import hooks & zip import References: Message-ID: <3DFA249F.20706@interet.com> Just van Rossum wrote: > After incorporating some suggestions and fixes from Paul Moore and Neal Norwitz > and doing more testing myself, I've decided the importhook/zipimport patch is > mature enough to post a proper patch: In PEP 273 under "Booting" the directory /usr/local/lib/python23.zip (for Unix) or python23.zip in the directory of python23.dll (for Windows) was added to sys.path so that the Python standard library could be loaded from a zip archive. This is missing in your patch. I am not sure whether this was deliberate. But if you want to add it, it requires patches to site.py, getpath.c, getpathp.c, pythonrun.c, main.c and sysmodule.c. This moves changes to sys.path out of sysmodule.c and into getpath[p].c where they belong. You may want to use my patches for this even if you use your own import.c instead of mine. JimA From just@letterror.com Fri Dec 13 18:55:21 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 13 Dec 2002 19:55:21 +0100 Subject: [Python-Dev] new import hooks & zip import In-Reply-To: <3DFA249F.20706@interet.com> Message-ID: James C. Ahlstrom wrote: > In PEP 273 under "Booting" the directory /usr/local/lib/python23.zip > (for Unix) or python23.zip in the directory of python23.dll (for > Windows) was added to sys.path so that the Python standard library > could be loaded from a zip archive. This is missing in your patch. True. I wasn't sure if there was consensus about the neccessity of this. > I am not sure whether this was deliberate. I'm not against this at all, I've just been ignoring the issue ;-) > But if you want to add > it, it requires patches to site.py, getpath.c, getpathp.c, pythonrun.c, > main.c and sysmodule.c. This moves changes to sys.path out of > sysmodule.c and into getpath[p].c where they belong. It might belong there, but I don't see an immediate reason why a default zip entry on sys.path demands this reorganization as well. > You may want to use my patches for this even if you use your own > import.c instead of mine. I'll go read them more closely, thanks. Just From jeremy@alum.mit.edu Fri Dec 13 19:13:46 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Fri, 13 Dec 2002 14:13:46 -0500 Subject: [Python-Dev] warnings compiling tkinter Message-ID: <15866.12650.879757.730361@slothrop.zope.com> Should I be concerned about these warnings? Perhaps my version of Tcl/Tk is too old? jeremy gcc -g -Wall -Wstrict-prototypes -fPIC -DWITH_APPINIT=1 -DWITH_BLT=1 -I/usr/X11R6/include -I. -I/usr/home/jeremy/src/python/dist/src/./Include -I/usr/local/include -I/usr/home/jeremy/src/python/dist/src/Include -I/usr/home/jeremy/src/python/dist/src/build-pydebug -c /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c -o /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.o /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c: In function `SetVar': /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1486: warning: passing arg 7 of `var_invoke' from incompatible pointer type /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1496: warning: passing arg 8 of `var_invoke' from incompatible pointer type /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c: In function `GetVar': /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1537: warning: passing arg 6 of `var_invoke' from incompatible pointer type /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1537: warning: passing arg 7 of `var_invoke' from incompatible pointer type /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c: In function `UnsetVar': /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1565: warning: passing arg 6 of `var_invoke2' from incompatible pointer type /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1565: warning: passing arg 7 of `var_invoke2' from incompatible pointer type gcc -shared /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.o /usr/home/jeremy/src/python/dist/src/Modules/tkappinit.o -L/usr/X11R6/lib -L/usr/local/lib -lBLT -ltk8.3 -ltcl8.3 -lX11 -o build/lib.linux-i686-2.3/_tkinter.so From martin@v.loewis.de Fri Dec 13 21:14:49 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 13 Dec 2002 22:14:49 +0100 Subject: [Python-Dev] warnings compiling tkinter In-Reply-To: <15866.12650.879757.730361@slothrop.zope.com> References: <15866.12650.879757.730361@slothrop.zope.com> Message-ID: Jeremy Hylton writes: > Should I be concerned about these warnings? Perhaps my version of > Tcl/Tk is too old? They are probably const-correctness issues. The functions whose addresses we are taking now expect const arguments, but didn't before. Can you try the following patch? If that won't help, please report the declarations of those functions in your Tcl version. Regards, Martin Index: Modules/_tkinter.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/_tkinter.c,v retrieving revision 1.137 diff -u -r1.137 _tkinter.c --- Modules/_tkinter.c 12 Dec 2002 19:05:48 -0000 1.137 +++ Modules/_tkinter.c 13 Dec 2002 21:09:27 -0000 @@ -57,6 +57,12 @@ #include #endif +/* For Tcl 8.2 and 8.3, CONST* is not defined. */ +#ifndef CONST84_RETURN +#define CONST84_RETURN +#define CONST +#endif + #define TKMAJORMINOR (TK_MAJOR_VERSION*1000 + TK_MINOR_VERSION) #if TKMAJORMINOR < 8002 @@ -1334,9 +1340,9 @@ TCL_DECLARE_MUTEX(var_mutex) -typedef const char* (*EventFunc1)(Tcl_Interp*, const char*, int); -typedef const char* (*EventFunc2)(Tcl_Interp*, const char*, const char*, int); -typedef const char* (*EventFunc3)(Tcl_Interp*, const char*, const char*, const char*, int); +typedef CONST84_RETURN char* (*EventFunc1)(Tcl_Interp*, CONST char*, int); +typedef CONST84_RETURN char* (*EventFunc2)(Tcl_Interp*, CONST char*, CONST char*, int); +typedef CONST84_RETURN char* (*EventFunc3)(Tcl_Interp*, CONST char*, CONST char*, const char*, int); typedef struct VarEvent { Tcl_Event ev; /* must be first */ TkappObject *self; @@ -1456,9 +1462,9 @@ static PyObject* var_invoke2(PyObject *_self, char* arg1, char* arg2, char* arg3, int flags, - int (*func1)(Tcl_Interp*, const char*, int), - int (*func2)(Tcl_Interp*, const char*, const char*, int), - int (*func3)(Tcl_Interp*, const char*, const char*, const char*, int)) + int (*func1)(Tcl_Interp*, CONST char*, int), + int (*func2)(Tcl_Interp*, CONST char*, CONST char*, int), + int (*func3)(Tcl_Interp*, CONST char*, CONST char*, CONST char*, int)) { return var_invoke(_self, arg1, arg2, arg3, flags, (EventFunc1)func1, (EventFunc2)func2, From arigo@tunes.org Fri Dec 13 23:46:47 2002 From: arigo@tunes.org (Armin Rigo) Date: Fri, 13 Dec 2002 15:46:47 -0800 (PST) Subject: [Python-Dev] os.path.commonprefix In-Reply-To: <15861.9426.521685.951820@montanaro.dyndns.org>; from skip@pobox.com on Mon, Dec 09, 2002 at 05:18:42PM -0600 References: <20021207160218.1528F49BA@bespin.org> <15858.8941.920830.141997@montanaro.dyndns.org> <20021209224815.8F7D94B52@bespin.org> <15861.9426.521685.951820@montanaro.dyndns.org> Message-ID: <20021213234647.439FF2029@bespin.org> Hello Skip, On Mon, Dec 09, 2002 at 05:18:42PM -0600, Skip Montanaro wrote: > How about just adding commonpathprefix? As I mentioned before, there's an > implementation in Tools/scripts/trace.py. Yes, providing a semi-broken version for compatibility is something, but providing the expected behavior too is much better (and clearly marks the cases where the previous version must not be used, by providing an alternative). The trace.py implementation seems broken because it directly uses os.sep instead of functions of os.path, ignoring os.altsep and possibly other platform-specific issues. > The docs for the function state, "Note that this may return invalid paths > because it works a character at a time." I suspect that's sufficient. Indeed. Armin From goodger@python.org Sat Dec 14 01:26:35 2002 From: goodger@python.org (David Goodger) Date: Fri, 13 Dec 2002 20:26:35 -0500 Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects abstract.c,2.93.6.7,2.93.6.8 Message-ID: This backport checkin (Oct 5 2002) changed the exception text between 2.2.1: >>> int(None) Traceback (most recent call last): File "", line 1, in ? TypeError: object can't be converted to int and 2.2.2 (last line): >>> int(None) Traceback (most recent call last): File "", line 1, in ? TypeError: int() argument must be a string or a number Should this change have been backported to 2.2.2 from the 2.3 codebase? Patrick O'Brien brought this to my attention. He found a test in the Docutils test suite that broke under 2.2.2. I've parameterized the broken test to work on old & new Python versions, which I would have had to do for Python 2.3 anyway. While minor, I think this was an enhancement and not a bug fix. Reference: discussion in http://www.python.org/sf/563740 -- David Goodger Open-source projects: - Python Docutils: http://docutils.sourceforge.net/ (includes reStructuredText: http://docutils.sf.net/rst.html) - The Go Tools Project: http://gotools.sourceforge.net/ From xscottg@yahoo.com Sat Dec 14 03:29:52 2002 From: xscottg@yahoo.com (Scott Gilbert) Date: Fri, 13 Dec 2002 19:29:52 -0800 (PST) Subject: [Python-Dev] PEP 298 In-Reply-To: <200212121910.gBCJAo222634@odiug.zope.com> Message-ID: <20021214032952.63245.qmail@web40105.mail.yahoo.com> Guido van Rossum wrote: > There are doubts about its usefulness. (AKA YAGNI.) Would you also > provide implementations for common object types? Otherwise there > wouldn't be a point, right? Thomas Heller wrote: > If you call YAGNI, it's ok for me, that's why I'm asking. Better save > the time in this case... This might be a YAGNI, since everyone can get by with there own private strategy, but it's still a good idea. Right now I have code that makes very specific checks on what type of PyObject is passed in to a function to see whether it is "safe enough" to release the GIL. For instance, the current implementation of strings seems to be safe enough for this, but most other buffer types aren't. This is a bummer since arrays aren't really "safe enough", and that's the kind of data we (people at my company) usually want to pass to a C extension. Right now I tell them they have to do an array.tostring() call first. That's ugly because it makes a copy of a potentially huge chunk of memory. It would be much better if an object could promise not to realloc or free your pointer after the GIL is released by implementing this interface. Martin v. Löwis wrote: > This is flawed. According to the PEP, an object which has > acquirelockedbuffer called will live forever: > > # Retrieving a buffer from an object puts this object in a locked > # state during which the buffer may not be freed, resized, or > # reallocated. The wording may not be specific enough, but the idea is that you can lock a buffer object and should (must) unlock it later. Between the lock and unlock, you are promised that your pointer won't be reallocated. You could release the GIL so other threads can run concurrently on multi processor hardware. You could pass the pointer to an asynchronous I/O routine. All sorts of good stuff. Martin v. Löwis wrote: > If it is just the interface, I'm -1. You don't need a PEP to define an > interface in Python - just establish an interface in the types you > care about. There are several types I care about which are out of my control: string, array, mmap, Numeric.array. If it not getting into 2.3 implies that Thomas Heller doesn't want to work on it, could you guys put it in the "abandoned" category instead of "rejected" or "retracted". It really is a good idea. __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com From martin@v.loewis.de Sat Dec 14 07:56:30 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 14 Dec 2002 08:56:30 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <20021214032952.63245.qmail@web40105.mail.yahoo.com> References: <20021214032952.63245.qmail@web40105.mail.yahoo.com> Message-ID: Scott Gilbert writes: > > # Retrieving a buffer from an object puts this object in a locked > > # state during which the buffer may not be freed, resized, or > > # reallocated. > > The wording may not be specific enough, but the idea is that you can lock a > buffer object and should (must) unlock it later. That is the flaw. It shouldn't tell me what I have to do, but instead it should tell me what happens if I don't. If I don't unlock the buffer, I would expect that it stays locked. Now it turns out that the buffer can actually go away while being locked. > > If it is just the interface, I'm -1. You don't need a PEP to define an > > interface in Python - just establish an interface in the types you > > care about. > > There are several types I care about which are out of my control: string, > array, mmap, Numeric.array. I see. So this is more than just the interface: your requirement is that these types also implement it. Of course, Numeric.array is somewhat out of control of the PEP. If that is so, I'm -0: I still would like to see it become useful on the Python level, i.e. the buffer object should somehow expose it. Regards, Martin From xscottg@yahoo.com Sat Dec 14 10:35:39 2002 From: xscottg@yahoo.com (Scott Gilbert) Date: Sat, 14 Dec 2002 02:35:39 -0800 (PST) Subject: [Python-Dev] PEP 298 In-Reply-To: Message-ID: <20021214103539.9712.qmail@web40107.mail.yahoo.com> --- "Martin v. Löwis" wrote: > > > > # Retrieving a buffer from an object puts this object in a locked > > > # state during which the buffer may not be freed, resized, or > > > # reallocated. > > > > The wording may not be specific enough, but the idea is that you can > > lock a buffer object and should (must) unlock it later. > > That is the flaw. It shouldn't tell me what I have to do, but instead > it should tell me what happens if I don't. If I don't unlock the buffer, > I would expect that it stays locked. Now it turns out that the buffer > can actually go away while being locked. > It's not a flaw, and that's an odd criticism. There aren't many functions in the Python API that are documented in terms of what happens when you misuse them. Think of it as more like knowing when you have to call Py_INCREF and Py_DECREF. If you want to keep a pointer to an object, you call Py_INCREF. If you want to keep a pointer to a buffer after releasing the GIL, you call PyObject_AcquireLockedWriteBuffer(...) The normal use case would be: void* ptr; size_t len; PyObject_AcquireLockedBuffer(obj, &ptr, &len); Py_BEGIN_ALLOW_THREADS do_work_on(ptr); Py_END_ALLOW_THREADS PyObject_ReleaseLockedBuffer(obj); If you think it's an issue, the act of locking the buffer could increase the refcount too... > > > > There are several types I care about which are out of my control: > > string, array, mmap, Numeric.array. > > I see. So this is more than just the interface: your requirement is > that these types also implement it. Of course, Numeric.array is > somewhat out of control of the PEP. > Sure, the interface is only useful if people implement it. If Python defines the interface then we get interoperability instead of each package inventing it's own scheme. It's not unreasonable to assume that people would migrate to this over time. They're using the current buffer interface now, and many people are aware of it's shortcomings. > > If that is so, I'm -0: I still would like to see it become useful on > the Python level, i.e. the buffer object should somehow expose it. > How would you use this at the Python level? I don't see how it is useful outside of a C extension. I think it would be like exposing Py_INCREF and Py_DECREF as Python builtins, but maybe you have a use case that I'm not aware of. Cheers, -Scott __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com From mal@lemburg.com Sat Dec 14 11:03:57 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 14 Dec 2002 12:03:57 +0100 Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects abstract.c,2.93.6.7,2.93.6.8 References: Message-ID: <3DFB101D.6010306@lemburg.com> David Goodger wrote: > This backport checkin (Oct 5 2002) changed the exception text between > 2.2.1: > > >>> int(None) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: object can't be converted to int > > and 2.2.2 (last line): > > >>> int(None) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: int() argument must be a string or a number > > Should this change have been backported to 2.2.2 from the 2.3 > codebase? > > Patrick O'Brien brought this to my attention. He found a test in the > Docutils test suite that broke under 2.2.2. How can a test break when you change the error message text ? I'd say that the test was broken: you should never rely on a particular message text since these can and do change rather often. > I've parameterized the > broken test to work on old & new Python versions, which I would have > had to do for Python 2.3 anyway. While minor, I think this was an > enhancement and not a bug fix. > > Reference: discussion in http://www.python.org/sf/563740 > -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From =?iso-8859-1?Q?Martin_v._L=F6wis?= Sat Dec 14 11:43:49 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Sat, 14 Dec 2002 12:43:49 +0100 Subject: [Python-Dev] PEP 298 References: <20021214103539.9712.qmail@web40107.mail.yahoo.com> Message-ID: <000701c2a366$122d3720$df2ae8d9@mira> > It's not a flaw, and that's an odd criticism. There aren't many functions > in the Python API that are documented in terms of what happens when you > misuse them. There sure are. Looking at the sys module: - exit: Most systems require it to be in the range 0-127, and produce undefined results otherwise. - _getframe: If that is deeper than the call stack, ValueError is raised. - setdefaultencoding: If name does not match any available encoding, LookupError is raised. and so on. > Think of it as more like knowing when you have to call Py_INCREF and > Py_DECREF. If you want to keep a pointer to an object, you call Py_INCREF. Precisely, and this is my complaint. It ought to be like incref/decref, but isn't. If I do incref without decref, I know what happens: the object stays alive. If I do acquire without release, I would *expect* that the the buffer stays acquired, but this is not necessarily the case. > If you think it's an issue, the act of locking the buffer could increase > the refcount too... I know something is flawed. Whether this is fixed by changing the spec, or by changing the implementation, I don't know. If the spec means "the pointer is not valid if the reference count of the underlying object goes to zero", it should say so. If the spec means "if the reference count goes to zero while the object is locked, Python aborts", it should say so. If the spec means "the buffer will not be released while it is acquired", it should say so. From reading the spec, I cannot tell which one it is. From reading the implementation, I can tell "it depends on the type". I thought that "depends on the type" is what this API was meant to eliminate. > How would you use this at the Python level? I don't see how it is useful > outside of a C extension. I want to pass buffers to fcntl, and want fcntl to change the buffer in-place. Currently, fcntl will need to keep the interpreter lock for that. For that to work, I need to create a locked buffer, and I need to find out its address. I can create a buffer using the buffer builtin, but it won't be a locked one, and I need to find out its address, but I can't. Regards, Martin From goodger@python.org Sat Dec 14 14:51:37 2002 From: goodger@python.org (David Goodger) Date: Sat, 14 Dec 2002 09:51:37 -0500 Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects abstract.c,2.93.6.7,2.93.6.8 In-Reply-To: <3DFB101D.6010306@lemburg.com> Message-ID: M.-A. Lemburg wrote: > How can a test break when you change the error message text ? > I'd say that the test was broken: you should never > rely on a particular message text since these can > and do change rather often. When Docutils reports errors, it sometimes uses the exception text returned by Python. The test that broke is one of those; it tests for an expected exception. I'm not questioning the validity of the change, just the backport. I'm fine with the change in 2.3, but I question whether the changed behavior should have been back backported to 2.2.2. I don't see the old text as a bug; the new text is improved, true, but that says "enhancement" to me, not "bugfix". My post was a question that goes to backport policy. -- David Goodger Open-source projects: - Python Docutils: http://docutils.sourceforge.net/ (includes reStructuredText: http://docutils.sf.net/rst.html) - The Go Tools Project: http://gotools.sourceforge.net/ From pedronis@bluewin.ch Sat Dec 14 15:29:27 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 14 Dec 2002 16:29:27 +0100 Subject: [Python-Dev] known obvious thing or bug (rexec)? Message-ID: <015b01c2a385$97010700$6d94fea9@newmexico> latest CVS Python 2.3a0 (#1, Dec 14 2002, 07:19:32) [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. consider >>> import rexec >>> r=rexec.RExec() >>> r.r_exec("class C: pass\nC.a=0") Traceback (most recent call last): File "", line 1, in ? File "/home/users/p/pe/pedronis/python/dist/src/Lib/rexec.py", line 309, in r_ exec exec code in m.__dict__ File "", line 2, in ? RuntimeError: classes are read-only in restricted mode vs. >>> r.r_exec("class C(object): pass\nC.a=0") >>> regards From python@rcn.com Sat Dec 14 16:52:39 2002 From: python@rcn.com (Raymond Hettinger) Date: Sat, 14 Dec 2002 11:52:39 -0500 Subject: [Python-Dev] [Python-checkins] python/dist/src/Objectsabstract.c,2.93.6.7,2.93.6.8 References: Message-ID: <002701c2a391$36de02e0$125ffea9@oemcomputer> > When Docutils reports errors, it sometimes uses the exception text returned > by Python. The test that broke is one of those; it tests for an expected > exception. I'm not questioning the validity of the change, just the > backport. I'm fine with the change in 2.3, but I question whether the > changed behavior should have been back backported to 2.2.2. I don't see the > old text as a bug; the new text is improved, true, but that says > "enhancement" to me, not "bugfix". My post was a question that goes to > backport policy. This particular rewording of the exception text was done for an instructor who found that the existing message was routinely confusing his students using Python. It must be a point of view thing. I think that usability problems are a type of bug. We often backport clarifications to doc strings, error messages, and the LaTex documentation. Using the same logic, GvR approved my backporting changes to IDLE which were throwing novice users into coniptions (like having RunScript enabled on the interactive screen). I think the idea behind the no enhancement rule is to only backport things learned by experience from the previously published version so that Py2.2.x becomes increasingly perfected. A genuine new feature has an experimental quality in that it would not have gotten exercised in the previously published version. From xscottg@yahoo.com Sat Dec 14 18:04:43 2002 From: xscottg@yahoo.com (Scott Gilbert) Date: Sat, 14 Dec 2002 10:04:43 -0800 (PST) Subject: [Python-Dev] PEP 298 In-Reply-To: <000701c2a366$122d3720$df2ae8d9@mira> Message-ID: <20021214180443.7819.qmail@web40104.mail.yahoo.com> --- "Martin_v._Löwis" wrote: > > > If you think it's an issue, the act of locking the buffer could > > increase the refcount too... > > I know something is flawed. Whether this is fixed by changing the spec, > or by changing the implementation, I don't know. > > If the spec means "the pointer is not valid if the reference count of > the underlying object goes to zero", it should say so. If the spec means > "if the reference count goes to zero while the object is locked, Python > aborts", it should say so. If the spec means "the buffer will not be > released while it is acquired", it should say so. From reading the spec, > I cannot tell which one it is. From reading the implementation, I can > tell "it depends on the type". I thought that "depends on the type" is > what this API was meant to eliminate. > The object should definitely be kept alive while someone has the buffer locked. I think it should be fixed by saying that PyObject_AcquireLockedReadBuffer and PyObject_AcquireLockedWriteBuffer will also do a Py_INCREF and that PyObject_ReleaseLockedBuffer will do a Py_DECREF. If there is no disagreement, and Thomas Heller doesn't mind, I'll submit the changes to the PEP... Would this be sufficient to keep the PEP alive (possibly in the "abandoned" state if Thomas Heller doesn't want to work on it for 2.4)? > > > How would you use this at the Python level? I don't see how it is > > useful outside of a C extension. > > I want to pass buffers to fcntl, and want fcntl to change the buffer > in-place. Currently, fcntl will need to keep the interpreter lock for > that. > > For that to work, I need to create a locked buffer, and I need to find > out its address. I can create a buffer using the buffer builtin, but it > won't be a locked one, and I need to find out its address, but I can't. > It sounds like you want to get your modifiable pointer as a Python integer. Then you want to pass that integer as the last argument of fcntl. I think it would be better if fcntl allowed the last argument to be a modifiable buffer instead of just an "int or a string" as the docs indicate. I think the fcntl C implementation could use the the locked buffer interface when it is in place. Then you could use the array module (or any other modifiable buffer type object) for your argument. BTW, the array object currently has the buffer_info() method which gives you that integer. I don't think this is the right long term solution, but it might get you past a short term hump. Cheers, -Scott __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com From jmiller@stsci.edu Sat Dec 14 18:45:55 2002 From: jmiller@stsci.edu (Todd Miller) Date: Sat, 14 Dec 2002 13:45:55 -0500 Subject: [Python-Dev] PEP-298: buffer interface widening too much? Message-ID: <3DFB7C63.4040401@stsci.edu> I've been following the PEP-298 discussion a little, because I fear eventually IAGNI. But when I look at the proposed changes, I can't help but get the feeling that the "too wide" buffer interface just got wider. By "too wide", I am referring to the extraneous segmentation and character buffer aspects of the existing interface. Adding three more calls for the locking problem bugs me. Doesn't anything ever get simpler? Looking at the proposed changes, the only thing that really seems needed for the bufferprocs is the release call and a flag bit. Assuming the release call is not implemented, the acquired buffer is never locked or unlocked, and we're no worse off than we are now: it's the BO's problem. Assuming the release call is present, the acquired buffer should *always* be locked and later released. The existing wrapper functions (plus one for release) can hide/implement this, making it straight forward to provide correct extensions: just implement a locking extension. It seems to me that the existing (3!) calls for acquiring the buffer pointer can do just fine. Except... In the case of existing extensions, there will certainly be no call to the release function. So for an extension which is slow to update to the revised protocol, there are negative consequences: buffers which are never unlocked. But so what? These consequences seem less dire (if more probable) than the existing consequences of "moving buffers." It would be a bigger problem if PEP-298 wound up fiddling with reference counts. In that case, un-released buffers would also become memory leaks. But is it really necessary to mess with the reference counts? I'm having a hard time imagining a situation in which the reference count for a buffer goes to zero while it is locked which is not a bug somewhere else. What am I missing? Another possibility is to implement a "release buffer pointer" wrapper function which *does nothing* for a Python release 2.3 and change nothing else. Let extensions which use buffers adopt usage of the wrapper and the locking protocol, with no immediate benefit. Then implement locking for real in Python 2.4. That way, the number of non-conforming extensions can be minimized, but the buffer API can be kept simple. So, in the end, what I'm wondering is whether there's a simpler middle ground between dropping the PEP and implementing it as it stands now. Are there compromises to be found between guaranteeing correctness, backwards compatability, and growing complexity? Is there another trade between backwards compatability and complexity which is tolerable for the short term with both simplicity and correctness available in the longer term? Todd From =?iso-8859-1?Q?Martin_v._L=F6wis?= Sat Dec 14 18:52:01 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Sat, 14 Dec 2002 19:52:01 +0100 Subject: [Python-Dev] PEP 298 References: <20021214180443.7819.qmail@web40104.mail.yahoo.com> Message-ID: <012001c2a3a1$e3a308d0$df2ae8d9@mira> > It sounds like you want to get your modifiable pointer as a Python integer. > Then you want to pass that integer as the last argument of fcntl. I think > it would be better if fcntl allowed the last argument to be a modifiable > buffer instead of just an "int or a string" as the docs indicate. This is desirable, yes, and a patch is being implemented for that, see #555817. However, it is not sufficient: The /dev/poll interface on Solaris requires that you must pass a structure that contains addresses to some storage. > BTW, the array object currently has the buffer_info() method which gives > you that integer. I don't think this is the right long term solution, but > it might get you past a short term hump. Yes, Thomas just explained this to me, as well. I would have never expected it to be at the array object, instead, I think it rightfully belongs to the buffer objects. Having it at the buffer object also allows you get addresses of other locked buffers - it would be unfortunate if we have the locked buffer interface, but now you must copy string objects to array objects, just to get their address. This is just as bad as having to copy array objects to string objects right now to avoid relocations to the buffer. Regards, Martin From =?iso-8859-1?Q?Martin_v._L=F6wis?= Sat Dec 14 19:16:13 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Sat, 14 Dec 2002 20:16:13 +0100 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFB7C63.4040401@stsci.edu> Message-ID: <015201c2a3a5$44fb33c0$df2ae8d9@mira> > I've been following the PEP-298 discussion a little, because I fear > eventually IAGNI. But when I look at the proposed changes, I can't > help but get the feeling that the "too wide" buffer interface just got > wider. By "too wide", I am referring to the extraneous segmentation > and character buffer aspects of the existing interface. Adding three > more calls for the locking problem bugs me. Doesn't anything ever get > simpler? The locked buffer interface is simpler than the original buffer interface. It does not support segmentation. > It seems to me that the existing (3!) calls for > acquiring the buffer pointer can do just fine. Except... > > In the case of existing extensions, there will certainly be no call to > the release function. This is a minor problem. The major problem is that the existing protocol does not guarantee locking. So if you find an object that supports the buffer interface, you still don't know how long you can use the pointer you've received. It might become invalid after calling nearly any Python API function. The PEP authors believe that this can't be retroactively fixed by declaring the existing interfaces to be locked, and I tend to agree: changing the docs doesn't change the code "out there". > But is it really necessary to mess with the reference > counts? I'm having a hard time imagining a situation in which the > reference count for a buffer goes to zero while it is locked which is > not a bug somewhere else. What am I missing? Not sure what you are referring to here. If it is the PEP: it doesn't tell whether reference counts should be increased. If you are referring to my messages: I'm not saying that the reference counts should be increased. I'm saying that the PEP does not specify what the interaction of locking and reference counting is. > So, in the end, what I'm wondering is whether there's a simpler middle > ground between dropping the PEP and implementing it as it stands now. There certainly is. The advantage of the PEP process is that there always is a specific proposal to discuss. We only have this proposal at the moment, so we can't discuss anything else. If you want to make a different proposal, please propose specific wording (either as a new PEP, or as a modification of the existing one). I can't picture precisely what it is that you are proposing, nor what its consequences would be. Regards, Martin From aahz@pythoncraft.com Sat Dec 14 19:42:15 2002 From: aahz@pythoncraft.com (Aahz) Date: Sat, 14 Dec 2002 14:42:15 -0500 Subject: [Python-Dev] Deprecate doctest? In-Reply-To: <3DFB101D.6010306@lemburg.com> References: <3DFB101D.6010306@lemburg.com> Message-ID: <20021214194215.GA9536@panix.com> On Sat, Dec 14, 2002, M.-A. Lemburg wrote: > David Goodger wrote: >> >>This backport checkin (Oct 5 2002) changed the exception text between >>2.2.1: >> >> >>> int(None) >> Traceback (most recent call last): >> File "", line 1, in ? >> TypeError: object can't be converted to int >> >>and 2.2.2 (last line): >> >> >>> int(None) >> Traceback (most recent call last): >> File "", line 1, in ? >> TypeError: int() argument must be a string or a number >> >>Should this change have been backported to 2.2.2 from the 2.3 >>codebase? >> >>Patrick O'Brien brought this to my attention. He found a test in the >>Docutils test suite that broke under 2.2.2. > > How can a test break when you change the error message text ? I'd > say that the test was broken: you should never rely on a particular > message text since these can and do change rather often. That's tricky, given that doctest is in the standard library, and that's exactly what doctest does. Are you advocating deprecating doctest? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From theller@python.net Sat Dec 14 21:27:30 2002 From: theller@python.net (Thomas Heller) Date: 14 Dec 2002 22:27:30 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <20021214180443.7819.qmail@web40104.mail.yahoo.com> References: <20021214180443.7819.qmail@web40104.mail.yahoo.com> Message-ID: Scott Gilbert writes: > --- "Martin_v._Löwis" wrote: > > > > > If you think it's an issue, the act of locking the buffer could > > > increase the refcount too... > > > > I know something is flawed. Whether this is fixed by changing the spec, > > or by changing the implementation, I don't know. The implementation follows the PEP, of course, so the PEP must be changed first. Currently the uploaded patch serves as a backup store for the changes I've done so far. > > > > If the spec means "the pointer is not valid if the reference count of > > the underlying object goes to zero", it should say so. If the spec means > > "if the reference count goes to zero while the object is locked, Python > > aborts", it should say so. If the spec means "the buffer will not be > > released while it is acquired", it should say so. From reading the spec, > > I cannot tell which one it is. From reading the implementation, I can > > tell "it depends on the type". I thought that "depends on the type" is > > what this API was meant to eliminate. > > > > The object should definitely be kept alive while someone has the buffer > locked. > > I think it should be fixed by saying that PyObject_AcquireLockedReadBuffer > and PyObject_AcquireLockedWriteBuffer will also do a Py_INCREF and that > PyObject_ReleaseLockedBuffer will do a Py_DECREF. > > If there is no disagreement, and Thomas Heller doesn't mind, I'll submit > the changes to the PEP... Would this be sufficient to keep the PEP alive > (possibly in the "abandoned" state if Thomas Heller doesn't want to work on > it for 2.4)? I will change the PEP according to Martin's 'complaints' and Scott's suggestions probably on Monday, but feel free to do it yourself if you want, Scott. If I understand the PEP process correctly, the PEP is alive until there is a definite pronouncement on it. > > > How would you use this at the Python level? I don't see how it is > > > useful outside of a C extension. > > > > I want to pass buffers to fcntl, and want fcntl to change the buffer > > in-place. Currently, fcntl will need to keep the interpreter lock for > > that. > > > > For that to work, I need to create a locked buffer, and I need to find > > out its address. I can create a buffer using the buffer builtin, but it > > won't be a locked one, and I need to find out its address, but I can't. > > A 'locked buffer' object could be built if the PEP is implemented, should it be part of the PEP? But if you expose the buffer address to Python as an integer, you're living dangerous again. Thomas From martin@v.loewis.de Sat Dec 14 21:45:00 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 14 Dec 2002 22:45:00 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: <20021214180443.7819.qmail@web40104.mail.yahoo.com> Message-ID: Thomas Heller writes: > If I understand the PEP process correctly, the PEP is alive until > there is a definite pronouncement on it. That is certainly the case. > A 'locked buffer' object could be built if the PEP is implemented, > should it be part of the PEP? But if you expose the buffer address > to Python as an integer, you're living dangerous again. Why is that? Why does it add a danger that would not be there without it. It would be dangerous if you could create a buffer object by passing an arbitrary integer, but I'm not proposing such a thing. Regards, Martin From jmiller@stsci.edu Sat Dec 14 22:44:31 2002 From: jmiller@stsci.edu (Todd Miller) Date: Sat, 14 Dec 2002 17:44:31 -0500 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira> Message-ID: <3DFBB44F.9070106@stsci.edu> Martin v. Löwis wrote: >>I've been following the PEP-298 discussion a little, because I fear >>eventually IAGNI. But when I look at the proposed changes, I can't >>help but get the feeling that the "too wide" buffer interface just got >>wider. By "too wide", I am referring to the extraneous segmentation >>and character buffer aspects of the existing interface. Adding three >>more calls for the locking problem bugs me. Doesn't anything ever get >>simpler? >> >> > >The locked buffer interface is simpler than the original buffer interface. >It does not support segmentation. > > I understand that, it just seemed to me that the complexity of the PEP-298 solution is rising out of proportion with the problem: no locking. > > >>It seems to me that the existing (3!) calls for >>acquiring the buffer pointer can do just fine. Except... >> >>In the case of existing extensions, there will certainly be no call to >>the release function. >> >> > >This is a minor problem. The major problem is that the existing protocol >does not guarantee locking. So if you find an object that supports the buffer >interface, you still don't know how long you can use the pointer >you've received. It might become invalid after calling nearly any Python >API function. > I'm suggesting that we should change the protocol to guarantee locking. The consequences of doing that appear acceptable to me, and I was hoping to stir up some discussion on why they're *not*, or to simplify the PEP enough that the problem gets fixed rather than ignored. > >The PEP authors believe that this can't be retroactively fixed >by declaring the existing interfaces to be locked, and I tend to agree: >changing the docs doesn't change the code "out there". > > I was trying to look at this from the perspective of what *really* breaks if we change the protocol to include locking implicitly. If the consequences of "obsolescent extension code not calling the buffer release" are unnecessary exceptions, that might be better than the current situation with no locking at all, where the consequences are a segfault or worse. I'm also arguing that anywhere there are "consequences", the current application is already in jeopardy because it must be (a) using the buffer protocol and (b) attempting to modify the buffer address while the buffer is in use. So, either the application already has serious issues, or changing the protocol and the subsequent "dangling locks" won't hurt it at all. Here I am assuming that locking does not affect refcount, nor block deallocation. (I'm also assuming that an object refcount cannot drop to zero during an extension function call w/o a bug somewhere else. Help me out if I'm wrong here.) > > >>But is it really necessary to mess with the reference >>counts? I'm having a hard time imagining a situation in which the >>reference count for a buffer goes to zero while it is locked which is >>not a bug somewhere else. What am I missing? >> >> > >Not sure what you are referring to here. If it is the PEP: it doesn't >tell whether reference counts should be increased. If you are referring >to my messages: I'm not saying that the reference counts should be increased. >I'm saying that the PEP does not specify what the interaction of locking >and reference counting is. > > I was referring to the recent python-dev messages between yourself and Scott Gilbert. > > >>So, in the end, what I'm wondering is whether there's a simpler middle >>ground between dropping the PEP and implementing it as it stands now. >> >> > >There certainly is. The advantage of the PEP process is that there always >is a specific proposal to discuss. We only have this proposal at the moment, >so we can't discuss anything else. If you want to make a different proposal, >please propose specific wording (either as a new PEP, or as a modification >of the existing one). I can't picture precisely what it is that you are >proposing, nor what its consequences would be. > Point taken. What I'm really proposing is this: 1. Add a buffer lock release call to the bufferprocs struct and an indicator flag bit. Thomas's call is fine already. The indicator bit could also remain the same, but the new meaining would simply be that a "release" routine is supported. The locking buffer pointer access calls should be dropped. 2. Advise objects with support the buffer API to modify their *current* buffer pointer fetch routines to implement locking if they need it. They must also fill in the release function pointer and set the bit to indicate they have it. They need to modify their "mutator methods" to check the lock. 3. Add a buffer release wrapper function the C-API to hide the nasty details of checking for and calling the release function. If there is none, the release wrapper, is a no-op. Thomas's existing release wrapper is fine. The locking buffer pointer access calls should be dropped. 4. Advise extensions to treat all objects as implementing only a locking interface. Get the locked pointer, and release it later. If the object chooses to ignore support for locking, its the object's problem. But there should be only *one* way to write an extension. 5. Advise objects supporting the buffer API *not* to mess with refcounts during buffer protocol get/release. This prevents probable (certain) memory leaks at the cost of improbable segfaults from obsolescent extensions. Any extension that this is a problem for should be happy to have a solution and be willing to update to support locking. I see no *new* risk here. 6. Advise objects which are asked to change a locked buffer to raise an exception rather than blocking. 7. Advise objects which are asked to deallocate while locked to deallocate normally. > >Regards, >Martin > > Todd From theller@python.net Sat Dec 14 23:52:03 2002 From: theller@python.net (Thomas Heller) Date: 15 Dec 2002 00:52:03 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: References: <20021214180443.7819.qmail@web40104.mail.yahoo.com> Message-ID: <7kecrwf0.fsf@python.net> > > A 'locked buffer' object could be built if the PEP is implemented, > > should it be part of the PEP? But if you expose the buffer address > > to Python as an integer, you're living dangerous again. > > Why is that? Why does it add a danger that would not be there without > it. Because you would pass the integer to a function expecting a pointer, store it in a structure as a pointer, or something like that - what else? The danger is in the function, of course, but the integer address probably encourages to implement those functions. Thomas From mmclay@comcast.net Sun Dec 15 02:06:17 2002 From: mmclay@comcast.net (Michael McLay) Date: Sat, 14 Dec 2002 21:06:17 -0500 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python Message-ID: <200212142106.01812.mclay@nist.gov> sf/653938I is a patch to integrate the fixedpoint module, that was created by Tim Peters, into Python as a first class object. That is, the patch adds this new number type so that it has direct syntax support like float, int, long, str., etc. within the interpreter. I use Tim's module to implement the type. This limits the patch to a small bit of code that adds the a syntax interface to this module. Syntax recognition for the new number format required a change to Parser/tokenizer.c and to Python/compile.c. This patch allows the new decimal type (I renamed the FixedPoint type to decimal in the fixedpoint.py file because the name is shorter and it is sufficient to distinguish the type from a binary float.) to be entered as a literal that is similar to a long, int, or float. The new syntax works as follows: >>> 12.00d 12.00d >>> .012d 0.012d >>> 1d 1.d >>> str(1.003d) '1.003' As you can see from the example, the default precision for decimal literals are determined by the precision entered expressed in the literal. The patch also adds glue to Python/bltinmodule.c to create a builtin decimal() function. This builtin decimal (Formally FixedPoint) function invokes the constructor of the Python implementation of the decimal class defined in the fixedpoint module. The implementation follows the familiar pattern for adding special builtin functions, much like the implementation of apply and abs. It does not follow the typical method for adding built in types. That would require a more invasive patch. The builtin decimal function works as follows: >>> decimal(1.333) 1.33d >>> decimal(1.333,4) 1.3330d >>> decimal(400,2) 400.00d The semantics of the precision parameter may be incorrect because it is only addressing the precision of the fraction part of the number. Shouldn't it reflect the number of significant digits? If that were the case then the following would have been the result: >>> decimal(400,2) 4.0e2d This problem is more noticable when exponents are large relative to the number of significant digits. For instance: >>> 40.e3d 40000.d The result should have been 40.e3d, not 40000. This implies more precision than is declared by the constant. As it currently is implemented the type pads with zeros on the integer side of the decimal point, whic implies more accuracy in the number than is true. I ran into this problem when I tried to implement the automatic conversion of string number values to decimal representation for numbers in 'e' format. >>> 3.03e5d 303000.00d The representation is incorrect. It should have returned 3.03e5d but it padded the result with bogus zeros. For this first cut at a new decimal type I am primarily interested in investigating the semantics of the new native decimal number type in Python. This type will be useful to bankers, accountants, and newbies. The backend implementation could be replaced with a C implementation without affecting the Python language semantics. The approach used in this patch makes it very easy to experiment with changes to the semantics because all of the experimentation can be done by changing the code in Lib/fixedpoint.py. The compiled wrapper doesn't have to be modified and recompiled. Unit testing so far just reuses the _test function in the fixedpoint.py module. Since I am not sure which way to go with the semantic interpretation of precision I decided to post the patch before making a significant change to fixedpoint.py. Some feedback on the interpretation of precision would be appreciated. Documentation: The following needs to be added to section 2.1 of the library reference manual: decimal(value, precision) Convert a string or a number to fixed point decimal number. If the argument is a string, it must contain a possibly signed decimal or floating point number, possibly embedded in whitespace. The precision parameter will be ignored on string values. The precision will be set based on the number of significant digits. Otherwise, the argument may be a plain or long integer or a floating point number, and a decimal number with the same value ( the precision attribute will set the precision of the fraction roundoff) is returned. Section 3.1.1 of the tutorial will add a brief description of decimal number usage after the description of floating point: There is full support for floating point; operators with mixed type operands convert the integer operand to floating point: >>> 3 * 3.75 / 1.5 7.5 >>> 7.0 / 2 3.5 Python also supports fixed point decimal numbers. These numbers do not suffer from the somewhat random roundoff errors that can occur with binary floating point numbers. A decimal number is created by adding a 'd' or 'D' suffix to a number: >>> 3.3d + 3d 6.3d >>> 3.3d + 3.03 6.3d >>> 3.3d + decimal(3.03,3) 6.330d >>> decimal(1.1, 16) 1.1000000000000001d >>> The builtin decimal constructor provides a means for converting float and int types to decimal types. Since floats are approximations of decimal floating point numbers there are often roundoff errors introduced by using floats. (The 1 in the last place of the conversion of 1.1 with 16 digits is a binary round off error.) For this reason the decimal function requires the specification of precision when converting a float to a decimal. This allows the significant digits of the number to be specified. (Accountants and bankers love this because it allows them to balance the books without pennies being lost due to the use of binary numbers.) >>> 3.3d/2 1.6d Note that in the example above the expression 3.3d/2 returned 1.6d. The rounding scheme for Python decimals uses "banking" rounding rules. With floating point numbers the result would have been as follows: >>> 3.3/2 1.6499999999999999 So as you can see the banking rules round off the .04999999999 portion of the number and calls it a an even 1.6. From martin@v.loewis.de Sun Dec 15 07:30:24 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 15 Dec 2002 08:30:24 +0100 Subject: [Python-Dev] PEP-298: buffer interface widening too much? In-Reply-To: <3DFBB44F.9070106@stsci.edu> References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira> <3DFBB44F.9070106@stsci.edu> Message-ID: Todd Miller writes: > Point taken. What I'm really proposing is this: Thanks for detailing this proposal. I now agree with your assumption that missing release calls might cause problems; I'd expect that they indeed will cause problems in many applications. The main issue is the interaction with ParseTuple, the 's', 'w', and 't' converters invoke a buffer operation, after which (under your proposal) the object is locked, there is no corresponding release call, and no place to put such a call. So, once you have passed an array to some C function expecting a char*, you cannot extend the array anymore. Regards, Martin From martin@v.loewis.de Sun Dec 15 07:33:06 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 15 Dec 2002 08:33:06 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <7kecrwf0.fsf@python.net> References: <20021214180443.7819.qmail@web40104.mail.yahoo.com> <7kecrwf0.fsf@python.net> Message-ID: Thomas Heller writes: > > Why is that? Why does it add a danger that would not be there without > > it. > > Because you would pass the integer to a function expecting a pointer, > store it in a structure as a pointer, or something like that - what > else? The danger is in the function, of course, but the integer > address probably encourages to implement those functions. But the functions are alreay there and implemented; as I said, I'm concerned with the fcntl/ioctl family of functions. If "I can get the address" would be encouraging to write functions that take arbitrary integers as pointers, we would already have tons of such functions, since the id() builtin also gives you an address. Regards, Martin From brett@python.org Sun Dec 15 09:00:33 2002 From: brett@python.org (Brett Cannon) Date: Sun, 15 Dec 2002 01:00:33 -0800 (PST) Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212142106.01812.mclay@nist.gov> References: <200212142106.01812.mclay@nist.gov> Message-ID: [Michael McLay] > sf/653938I is a patch to integrate the fixedpoint module, that was created by > Tim Peters, into Python as a first class object. That is, the patch adds this > new number type so that it has direct syntax support like float, int, long, > str., etc. within the interpreter. > Someone will correct me if I am wrong, but making a new decimal type that includes a new suffix for numbers is going to require a PEP. Your email, though, already seems like a rather good start for one and should not require much more work. It would be easier to get the module itself sans any Python core changes accepted initially to gauge usage and interest by the Python community before pushing for syntax integration. This is what is be done with the current rational implementation. I personally would like to see the module included but currently say no to any integration into the core syntax. -Brett From martin@v.loewis.de Sun Dec 15 09:13:48 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 15 Dec 2002 10:13:48 +0100 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: References: <200212142106.01812.mclay@nist.gov> Message-ID: Brett Cannon writes: > Someone will correct me if I am wrong, but making a new decimal type that > includes a new suffix for numbers is going to require a PEP. Your > email, though, already seems like a rather good start for one and should > not require much more work. That is certainly the case (both the need for a PEP, and that the message is a good starting point). While the technical aspects of the proposed changes look fine, I'm missing a rationale: why is having decimal literals in the language a good thing? [this is not to say they aren't, but it may not be obvious] > It would be easier to get the module itself sans any Python core changes > accepted initially to gauge usage and interest by the Python community > before pushing for syntax integration. This is what is be done with the > current rational implementation. It also appears that a decimal type feature in the language will/should interact with a rational type. I don't know whether the current rational type proposals suggests introduction of new literals, but I feel that the only rational literals you'll ever need are decimals (and we all know that floating-point literals are decimal rationals, too). So it might be that we *only* get the decimal notation from this proposal, and not a proper decimal type. Of course, my CORBA affinity tells me that a decimal type would be useful as well :-) Regards, Martin From bsder@mail.allcaps.org Sun Dec 15 09:54:56 2002 From: bsder@mail.allcaps.org (bsder@mail.allcaps.org) Date: Sun, 15 Dec 2002 01:54:56 -0800 (PST) Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212142106.01812.mclay@nist.gov> Message-ID: On Sat, 14 Dec 2002, Michael McLay wrote: > sf/653938I is a patch to integrate the fixedpoint module, that was created by -1 Two reasons: 1) I don't see a PEP about this. If I am wrong about this, please point me at the PEP so that I can review it. 2) There are official/semi-official standards for decimal floating-point arithmetic. (IEEE 854 + ANSI X3.274 + ECMA 334) The fixedpoint module seems to comply with these standards only at the most basic levels. Again, if I am wrong, please point me at the relevant docs so that I can review them. -a From fredrik@pythonware.com Sun Dec 15 10:21:15 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sun, 15 Dec 2002 11:21:15 +0100 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python References: Message-ID: <008f01c2a423$b7798400$ced241d5@hagrid> > From: > > -1 > > 1) I don't see a PEP about this. If I am wrong about this, please point > me at the PEP so that I can review it. > > 2) There are official/semi-official standards for decimal floating-point > arithmetic. (IEEE 854 + ANSI X3.274 + ECMA 334) The fixedpoint module > seems to comply with these standards only at the most basic levels. > Again, if I am wrong, please point me at the relevant docs so that I can > review them. > > -a arrogant anonymous python-dev voters appealing to authority? interesting concept. From skip@manatee.mojam.com Sun Dec 15 13:00:16 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 15 Dec 2002 07:00:16 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200212151300.gBFD0Gta017461@manatee.mojam.com> Bug/Patch Summary ----------------- 321 open / 3121 total bugs (+4) 93 open / 1834 total patches (-1) New Bugs -------- email test external dependencies (2002-12-08) http://python.org/sf/650441 pydoc ignores $PAGER if TERM=='dumb' (2002-12-09) http://python.org/sf/651124 Review libshelve.tex when possible (2002-12-09) http://python.org/sf/651149 Bozo getstate w/ slots overrides base (2002-12-10) http://python.org/sf/651701 dynload_next needs better errors (2002-12-12) http://python.org/sf/652590 extra __builtin__ stuff not documented (2002-12-12) http://python.org/sf/652749 bad documentation for the "type" builtin (2002-12-12) http://python.org/sf/652888 py_compile does not return error code (2002-12-13) http://python.org/sf/653301 PyLong_AsVoidPtr()/PyLong_FromVoidPtr() (2002-12-13) http://python.org/sf/653542 Investigate weak linking (2002-12-14) http://python.org/sf/653871 New Patches ----------- tarfile module implementation (2002-12-09) http://python.org/sf/651082 New import hooks + Import from Zip files (2002-12-12) http://python.org/sf/652586 PEP 298 implementation (2002-12-12) http://python.org/sf/652857 Generator form of os.path.walk (2002-12-12) http://python.org/sf/652980 Adds a builtin decimal type (FixedPoint) (2002-12-14) http://python.org/sf/653938 Closed Bugs ----------- Fix the METH_CLASS implementation (2002-04-25) http://python.org/sf/548651 Tix Checklist getselection getstatus bug (2002-10-10) http://python.org/sf/621511 textwrap fails on unicode using defaults (2002-10-13) http://python.org/sf/622831 inconsistent results of leading whitespace in textwrap input (2002-10-13) http://python.org/sf/622849 xmlrpc can't do proxied HTTP (2002-12-04) http://python.org/sf/648658 Closed Patches -------------- 'switch'/'case'/'else' statement (2001-11-12) http://python.org/sf/481118 splitext performances improvement (2002-03-29) http://python.org/sf/536661 Add check for setrlimit() support (2002-05-03) http://python.org/sf/551960 Detect BLT on more systems (2002-10-26) http://python.org/sf/629126 tuple arg for issubclass (2002-12-06) http://python.org/sf/649608 avoid redefinition of pthread functions (2002-12-08) http://python.org/sf/650415 From jmiller@stsci.edu Sun Dec 15 14:37:15 2002 From: jmiller@stsci.edu (Todd Miller) Date: Sun, 15 Dec 2002 09:37:15 -0500 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira> <3DFBB44F.9070106@stsci.edu> Message-ID: <3DFC939B.2090609@stsci.edu> Martin v. Löwis wrote: >Todd Miller writes: > > > >>Point taken. What I'm really proposing is this: >> >> > >Thanks for detailing this proposal. I now agree with your assumption >that missing release calls might cause problems; I'd expect that they >indeed will cause problems in many applications. > Thanks for reading it. > >The main issue is the interaction with ParseTuple, the 's', 'w', and >'t' converters invoke a buffer operation, after which (under your >proposal) the object is locked, there is no corresponding release >call, and no place to put such a call. > I've used this aspect of PyArg_ParseTuple, but forgot about it. I was thinking that you're right, the original object is "gone", we're stuck. But... it's still (in all probability) lying around in the arg tuple, so some sort of PyArg_ReleaseBuffers is possible. It could even be a near-carbon-copy of the PyArg_ParseTuple which "lost" the object to begin with, to make the book keeping easy: PyArg_ParseTuple(args, "iw#Os", &someInt, &writeableptr, &writeablesize, &someobject, &readableptr) becomes: PyArg_ReleaseBuffers(args, "iw#Os") at exit time. The parse spec defines which parts of the tuple to release, and which not to release. Anything which is not w,s, or t gets skipped. w,s,t are released. With this call, it becomes possible to "fix" the PyArg_ParseTuple case. Incidentally, does PEP-298 solve this now? (I think the answer is no. PEP-298 is just using a different approach to "phased introduction" by leaving the original problem fully exposed in backwards compatability mode. Or am I missing something here too?) > >So, once you have passed an array to some C function expecting a >char*, you cannot extend the array anymore. > Well, this does sound bad, but I was assuming the core and library would be "fixed" at the first introduction of locking, and the only problem would be with non-conforming extensions. I now think that "lock violations" might be better handled using the warning framework, defaulting to a one time warning. Todd From =?iso-8859-1?Q?Martin_v._L=F6wis?= Sun Dec 15 15:47:26 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Sun, 15 Dec 2002 16:47:26 +0100 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira> <3DFBB44F.9070106@stsci.edu> <3DFC939B.2090609@stsci.edu> Message-ID: <00c501c2a451$44cae8d0$d435e8d9@mira> > I've used this aspect of PyArg_ParseTuple, but forgot about it. I > was thinking that you're right, the original object is "gone", we're > stuck. But... it's still (in all probability) lying around in the arg > tuple, so some sort of PyArg_ReleaseBuffers is possible. That looks clumsy, and also too easy to forget unless done systematically, in which case it will add a considerable performance cost. I think the locked buffer interface is much simpler in comparison. If Jack ever manages to design an alternative parse tuple interface, I assume it will arrange to automatically release the buffer when the function returns (or, more precisely, when the buffer is deallocated. In fact, my PEP 286 could address this issue: the locked buffers would be associated with the enhanced tuple, so that releasing the tuple would unlock the buffers. This *still* has the potential for breaking code, since now the buffers are locked while the function runs (i.e. callbacks couldn't expand the array anymore); it is likely that such breakage would uncover a bug in the locking assumptions of the existing code. Your attempt to fix your proposal indicates one of the strengths of the PEP process: most initial proposals have flaws, and, unless completely spelled out, readers will typically miss the flaws on initial reading. The weakness, of course, is that a PEP might be sitting around for a long time, and nobody would be reviewing it. > Incidentally, does PEP-298 solve this now? Depends on what "this" is. Existing code will continue to work as before: No locking in ParseTuple happens, as a result, the arguments are unlocked when the function returns; this is desirable. Since the buffers aren't locked, relocation may happen. If such relocation is ever observed, people would have to replace the "s" parser with an "O" parser, and perform the locking themselves. So, yes, PEP-298 addresses this - although modification of existing code is necessary to make use of this feature (as is in your fixed proposal). > Well, this does sound bad, but I was assuming the core and library > would be "fixed" at the first introduction of locking, and the only > problem would be with non-conforming extensions. I now think that > "lock violations" might be better handled using the warning framework, > defaulting to a one time warning. I think we should strive for a solution that keeps the function parameters locked while the function runs, and unlocks them automatically at function return time. This shows another flaw in the wording of PEP 298: It implies without saying that the expectation is that you should have as many release calls as you have lock calls. Regards, Martin From mmclay@comcast.net Sun Dec 15 16:02:30 2002 From: mmclay@comcast.net (Michael McLay) Date: Sun, 15 Dec 2002 11:02:30 -0500 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: References: <200212142106.01812.mclay@nist.gov> Message-ID: <200212151102.30779.mmclay@comcast.net> On Sunday 15 December 2002 04:00 am, Brett Cannon wrote: > [Michael McLay] > > > sf/653938I is a patch to integrate the fixedpoint module, that was > > created by Tim Peters, into Python as a first class object. That is, the > > patch adds this new number type so that it has direct syntax support like > > float, int, long, str., etc. within the interpreter. > > Someone will correct me if I am wrong, but making a new decimal type that > includes a new suffix for numbers is going to require a PEP. Your > email, though, already seems like a rather good start for one and should > not require much more work. Yes, there should be a PEP. I posted this as a work in progress because I had a question on the semantics and also to get a general sense of what questions a PEP might need to answer. > > It would be easier to get the module itself sans any Python core changes > accepted initially to gauge usage and interest by the Python community > before pushing for syntax integration. This is what is be done with the > current rational implementation. The module has existed for quite some time and is available on sourceforge at . I made some changes to the module and I am interested in opinions on the change in the name of the class. > I personally would like to see the module included but currently say no to > any integration into the core syntax. The syntax change is primarily to make Python more appealing to professions that require the semantics of fixed point decimal numbers. Ideally Python would only have decimal numbers. The use of binary floating point is a performance enhancement and since the language historically has used binary numbers for float the switch to using decimal numbers would break backward compatibllity. In order to add decimal numbers as an alternative the syntax must distinguish floats from decimal. The format I proposed is already used to distinguish number types is used for complex and long numbers. The syntax change would allow decimal numbers to be used without the clutter of a constructor and a quoted string. For example: msql.execute ("INSERT INTO bankAccount VALUES ('%s','%s','%s','%s', (,'Pay Rent' ,1256,FixedPoint("1345.23") ,'1999-03-30',NULL)") would be written as follows: msql.execute ("INSERT INTO bankAccount VALUES ('%s','%s','%s','%s', (,'Pay Rent' ,1256,1345.23d ,'1999-03-30',NULL)") In this case the percentage of clutter is pretty minimal, but if you want to use Python as a serious tool for interactive calculations the clutter becomes annoynig. For instance: >>> def CalcAverage(list): ... total = 0d ... for i in list: ... total += i ... return total/len(list) ... >>> CalcAverage([FixedPoint("123.40"),FixedPoint("222.20"),FixedPoint("300.33"),FixedPoint("400.32")]) FixedPoint("261.56") Compare that to what would be required if the syntax were built into the language: >>> CalcAverage([123.40d, 222.20d, 300.33d, 400.32d]) 261.56d >>> From oren-py-d@hishome.net Sun Dec 15 16:38:14 2002 From: oren-py-d@hishome.net (Oren Tirosh) Date: Sun, 15 Dec 2002 11:38:14 -0500 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212151102.30779.mmclay@comcast.net> References: <200212142106.01812.mclay@nist.gov> <200212151102.30779.mmclay@comcast.net> Message-ID: <20021215163814.GA42886@hishome.net> On Sun, Dec 15, 2002 at 11:02:30AM -0500, Michael McLay wrote: > msql.execute ("INSERT INTO bankAccount > VALUES ('%s','%s','%s','%s', (,'Pay Rent' > ,1256,1345.23d ,'1999-03-30',NULL)") I hope no financial software handling my money actually writes such literals into the code. The above statement should not appear anywhere outside a tutorial. A decimal literal might be handy when using the interactive Python prompt as a financial calculator, though. Is that enough justification for including it into the language? Oren From bbum@codefab.com Sun Dec 15 17:58:28 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Sun, 15 Dec 2002 12:58:28 -0500 Subject: [Python-Dev] xmlrpc can't do proxied HTTP (2002-12-04) In-Reply-To: <20021215170005.4432.76076.Mailman@mail.python.org> Message-ID: On Sunday, Dec 15, 2002, at 12:00 US/Eastern, python-dev-request@python.org wrote: > xmlrpc can't do proxied HTTP (2002-12-04) > http://python.org/sf/648658 The above showed up in the 'closed' list. I clicked through to see what was going on, but I received an 'invalid artifact ID' error. ? b.bum From jmiller@stsci.edu Sun Dec 15 18:33:33 2002 From: jmiller@stsci.edu (Todd Miller) Date: Sun, 15 Dec 2002 13:33:33 -0500 Subject: [Python-Dev] PEP-298: buffer interface widening too much? Message-ID: <3DFCCAFD.6020303@stsci.edu> Martin v. Löwis wrote: >>I've used this aspect of PyArg_ParseTuple, but forgot about it. I >>was thinking that you're right, the original object is "gone", we're >>stuck. But... it's still (in all probability) lying around in the arg >>tuple, so some sort of PyArg_ReleaseBuffers is possible. >> >> > >That looks clumsy, > Well, yes, especially in light of PEP-286. :) >and also too easy to forget unless done systematically, > I really liked the idea of doing the buffer release behind the scenes, as suggested by PEP-286 and below. >in which case it will add a considerable performance cost. I think the > But won't *any* systematic solution have a performance cost? Argtuples don't sound "free". I'm really playing devil's advocate here, because I like the concept of PEP-286. >locked buffer interface is much simpler in comparison. > I disagree here. I didn't see PEP-298 address PyArg_ParseTuple at all, so there's no real basis for comparison on this point. I'll grant you that PyArg_ReleaseBuffers is probably a lame solution to a problem better solved by PEP-286; that said, because use of PyArg_ReleaseBuffers is explicit, there is no problem distributing the performance penalty to places where there is a corresponding benefit. I was not suggesting dropping the existing release call; merely adding a layer which could make sense out of PyArg_ParseTuple specs. > >If Jack ever manages to design an alternative parse tuple interface, I >assume it will arrange to automatically release the buffer when the function >returns (or, more precisely, when the buffer is deallocated. > >In fact, my PEP 286 could address this issue: the locked buffers would be >associated with the enhanced tuple, so that releasing the tuple would unlock >the buffers. > That sounds pretty cool. >This *still* has the potential for breaking code, since now >the buffers are locked while the function runs (i.e. callbacks couldn't >expand the array anymore); it is likely that such breakage would uncover >a bug in the locking assumptions of the existing code. > > I agree. But the means exist to work around such problem (release and re-aquire), and I would guess they are even rarer than the existing "buffer safety" issues. >Your attempt to fix your proposal indicates one of the strengths of the PEP >process: most initial proposals have flaws, and, unless completely spelled >out, readers will typically miss the flaws on initial reading. The weakness, >of course, is that a PEP might be sitting around for a long time, and nobody >would be reviewing it. > > > >>Incidentally, does PEP-298 solve this now? >> >> > >Depends on what "this" is. Existing code will continue to work as before: >No locking in ParseTuple happens, as a result, the arguments are unlocked >when the function returns; this is desirable. Since the buffers aren't >locked, relocation may happen. If such relocation is ever observed, people >would have to replace the "s" parser with an "O" parser, and perform the >locking themselves. > I'm just saying that we don't need 2 extra "locking calls" to achieve this. If the default behavior for the existing Python objects (array, mmap, ...) is a onetime warning when a lock is abused, where's the issue? And with the warning framework, the one time warning could be mutated into an exception or supressed by a variety of means. My thinking is that *all* buffer pointers should be "logically" locked. Objects that don't need locks don't need to change. All extensions should be written and/or updated to handle objects which require locking. Extensions which aren't updated will trigger warnings which could be supressed or converted into exceptions. For me, that's simpler than saying "correct extensions" lock the buffer, but "obsolete ones" may still segfault w/o warning. Perhaps saying that "a warning is enough" is too limiting, but this is a fringe issue, so I think maybe it is. > >So, yes, PEP-298 addresses this - although modification of existing code is >necessary to make use of this feature (as is in your fixed proposal). > > > >>Well, this does sound bad, but I was assuming the core and library >>would be "fixed" at the first introduction of locking, and the only >>problem would be with non-conforming extensions. I now think that >>"lock violations" might be better handled using the warning framework, >>defaulting to a one time warning. >> >> > >I think we should strive for a solution that keeps the function parameters >locked while the function runs, and unlocks them automatically at function >return time. > I like this idea, but wonder if it won't inflict a performance penalty on all function calls. > >This shows another flaw in the wording of PEP 298: It implies without saying >that the expectation is that you should have as many release calls as you >have lock calls. > I was certainly assuming this was the case. > >Regards, >Martin > > > Regards, Todd From =?iso-8859-1?Q?Martin_v._L=F6wis?= Sun Dec 15 18:43:12 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Sun, 15 Dec 2002 19:43:12 +0100 Subject: [Python-Dev] xmlrpc can't do proxied HTTP (2002-12-04) References: Message-ID: <016d01c2a469$d2b82500$d435e8d9@mira> > The above showed up in the 'closed' list. I clicked through to see > what was going on, but I received an 'invalid artifact ID' error. > > ? That's a buglet in both Skip's cronjob, and my SF redirector. It got moved to feature requests, and is now at http://sourceforge.net/tracker/?group_id=5470&atid=355470&func=detail&aid=648658 Regards, Martin From =?iso-8859-1?Q?Martin_v._L=F6wis?= Sun Dec 15 19:09:23 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Sun, 15 Dec 2002 20:09:23 +0100 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFCCAFD.6020303@stsci.edu> Message-ID: <017301c2a46d$7ade6160$d435e8d9@mira> > I really liked the idea of doing the buffer release behind the scenes, > as suggested by PEP-286 and below. Alas, this has a fundamental flaw also, that's why I haven't advanced it, yet. Jack found the flaw while reviewing it, but I did not manage to integrate that into the PEP, and now I forgot what it was :-( It should be in the mailing list archives, though. > But won't *any* systematic solution have a performance cost? Argtuples > don't sound "free". I'm really playing devil's advocate here, because > I like the concept of PEP-286. Without any numbers to back up this claim: I believe the largest cost from Py*Parse comes from the need to process the printf-style string, and the corresponding va_arg processing, perhap along with the many type checks. That's why I protest any unnecessary ParseTuple call, and object any functions in this class. In comparison, the PEP 286 approach should a) only produce costs if there is actually any buffer management, and b) have a management processing cost (for building data structures and iterating over them) roughly the same size as the number of resources managed. > I disagree here. I didn't see PEP-298 address PyArg_ParseTuple at all, > so there's no real basis for comparison on this point. I'll grant > you that PyArg_ReleaseBuffers is probably a lame solution to a problem > better solved by PEP-286; that said, because use of > PyArg_ReleaseBuffers is explicit, there is no problem distributing the > performance penalty to places where there is a corresponding benefit. I > was not suggesting dropping the existing release call; merely adding a > layer which could make sense out of PyArg_ParseTuple specs. I understand that. But please compare if (!PyArg_ParseTuple(args,"sss",&string1, &string2, &string3)) return NULL; if(!do_some_stuff(string1)){ PyArg_Release(args,"sss",string1,string2,string3); return NULL; } if(!do_more_stuff(string2)){ PyArg_Release(args,"sss",string1,string2,string3); return NULL; } compute_results(string3); PyArg_Release(args,"sss",string1,string2,string3); return result; to if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3)) return NULL; if(!do_some_stuff(string1)){ return NULL; } if(!do_more_stuff(string2)){ return NULL; } PyBuffer_Lock(obj3, &string3); compute_results(string3); PyBuffer_Unlock(obj3); return result; In general, you won't have to lock everything, so you can avoid a lot of clutter if you use just the PEP-298 interfaces, together with an "O" parser. > If the default behavior for the existing Python objects (array, mmap, > ...) is a onetime warning when a lock is abused, where's the issue? It's unclear. I would expect that every application that uses arrays will trigger the warning. When you get the warning, you will have no clue what you did wrong, and most likely, the error is in the Python core. Who is going to fix the hundreds and hundreds of functions that use the "s" parser, and who is not going to be terribly bored by this, knowing that 99% of them will never see an array. > My thinking is that *all* buffer pointers should be "logically" locked. That is the intent of the PEP, indeed. For backwards compatibility, it can't change the existing interface, but needs to create a new one. After some time, the core could require that objects support it, much more later, it could remove support for the old interface. > Objects that don't need locks don't need to change. All extensions > should be written and/or updated to handle objects which require > locking. Extensions which aren't updated will trigger warnings which > could be supressed or converted into exceptions. The problem is that you are talking about virtually *all* extensions. I bet you won't find a single extension that doesn't use an "s" parser. > For me, that's simpler than saying "correct extensions" lock the buffer, > but "obsolete ones" may still segfault w/o warning. But, with your warning, this would still happen? > I like this idea, but wonder if it won't inflict a performance penalty > on all function calls. I hope it won't. There is no implementation of enhanced argument tuples, yet (and well may not appear for Python 2.3); but if this is done, a PyArg_ParseTuple call that doesn't allocate any resources will leave a NULL pointer in the slot for the resources - on deallocation, all you pay is the null pointer test. Regards, Martin From lists@morpheus.demon.co.uk Sun Dec 15 18:03:48 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sun, 15 Dec 2002 18:03:48 +0000 Subject: [Python-Dev] Deprecate doctest? References: <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> Message-ID: Aahz writes: >> How can a test break when you change the error message text ? I'd >> say that the test was broken: you should never rely on a particular >> message text since these can and do change rather often. > > That's tricky, given that doctest is in the standard library, and > that's exactly what doctest does. Are you advocating deprecating > doctest? You can write the doctest to avoid the dependency on the exact test. As a (none too realistic) example, >>> 1/0 Traceback (most recent call last): File "", line 1, in ? ZeroDivisionError: integer division or modulo by zero is a bad test, because it depends on an exact text. (This one probably isn't "bad" in practice, as I can't imagine the text changing very often). On the other hand, >>> try: ... 1/0 ... except ZeroDivisionError: ... print "Divide by zero" ... Divide by zero is a good test, because it protects against such a dependency. This is a fairly well-known type of issue with doctest, which usually shows up in the form of reliance on a particular order of printing a dictionary. See section 5.2.7 of the manual (Warnings). Paul. -- This signature intentionally left blank From jmiller@stsci.edu Sun Dec 15 22:07:29 2002 From: jmiller@stsci.edu (Todd Miller) Date: Sun, 15 Dec 2002 17:07:29 -0500 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira> Message-ID: <3DFCFD21.2090707@stsci.edu> Martin v. Löwis wrote: >I understand that. But please compare > > if (!PyArg_ParseTuple(args,"sss",&string1, &string2, &string3)) > return NULL; > if(!do_some_stuff(string1)){ > PyArg_Release(args,"sss",string1,string2,string3); > return NULL; > } > if(!do_more_stuff(string2)){ > PyArg_Release(args,"sss",string1,string2,string3); > return NULL; > } > compute_results(string3); > > PyArg_Release(args,"sss",string1,string2,string3); > return result; > >to > >if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3)) > return NULL; > if(!do_some_stuff(string1)){ > return NULL; > } > if(!do_more_stuff(string2)){ > return NULL; > } > PyBuffer_Lock(obj3, &string3); > compute_results(string3); > PyBuffer_Unlock(obj3); > > return result > >In general, you won't have to lock everything, so you can avoid >a lot of clutter if you use just the PEP-298 interfaces, together >with an "O" parser. > > Well, my initial proposal would have read more like: PyObject *result = NULL; if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3)) goto fail_or_exit; if(!do_some_stuff(string1)) goto fail_or_exit; if(!do_more_stuff(string2)) goto fail_or_exit; result = compute_results(string3); fail_or_exit: PyBuffer_Unlock(obj3); return result; I'll concede that I'm not 100% comfortable with exit-gotos and that my implicit locking approach makes them necessary. >>If the default behavior for the existing Python objects (array, mmap, >>...) is a onetime warning when a lock is abused, where's the issue? >> >> > >It's unclear. I would expect that every application that uses arrays will >trigger the warning. When you get the warning, you will have no clue what >you did wrong, and most likely, the error is in the Python core. Who is > True. Isolating the missing release call is hard. Harder still will be isolating missing locks. >going to fix the hundreds and hundreds of functions that use the "s" parser, >and who is not going to be terribly bored by this, knowing that 99% of them >will never see an array. > > I guess that's the beauty of PEP-286... I now see that (or something like it) as the only possibility for really correcting the buffer problem. > > >>My thinking is that *all* buffer pointers should be "logically" locked. >> >> > >That is the intent of the PEP, indeed. For backwards compatibility, it >can't change the existing interface, but needs to create a new one. After >some time, the core could require that objects support it, much more later, >it could remove support for the old interface. > > I think that kind of backwards compatability just paves the wave for extensions to stay broken and complicates the object structures as well. I guess PEP-298 does create the possibility for creating correct extensions, I just hate the thought of 5 getpointer calls where there are now 3 and should be 2. >> Objects that don't need locks don't need to change. All extensions >>should be written and/or updated to handle objects which require >>locking. Extensions which aren't updated will trigger warnings which >>could be supressed or converted into exceptions. >> >> > >The problem is that you are talking about virtually *all* extensions. >I bet you won't find a single extension that doesn't use an "s" parser. > Most of our (numarray) extensions don't use "s" parsers, but I'll accept that it's not feasible to update all extensions. >>For me, that's simpler than saying "correct extensions" lock the buffer, >> but "obsolete ones" may still segfault w/o warning. >> >> > >But, with your warning, this would still happen? > > Yes. But it's a (minor) improvement because you have an immediate clue why the segfault occurred. > > >>I like this idea, but wonder if it won't inflict a performance penalty >>on all function calls. >> >> > >I hope it won't. There is no implementation of enhanced argument tuples, yet >(and well may not appear for Python 2.3); but if this is done, a >PyArg_ParseTuple call that doesn't allocate any resources will leave a NULL >pointer in the slot for the resources - on deallocation, all you pay is the >null pointer test. > > Sounds good to me. I hope you implement it! If PEP-286 were here now, a simplified PEP-298 would be possible. Without PEP-286, I doubt the buffer locking problem will ever be adequately fixed and life is going to get more complicated. Fortunately for me, it's all still theoretical... >Regards, >Martin > > > Regards, Todd From brett@python.org Sun Dec 15 22:32:17 2002 From: brett@python.org (Brett Cannon) Date: Sun, 15 Dec 2002 14:32:17 -0800 (PST) Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212151102.30779.mmclay@comcast.net> References: <200212142106.01812.mclay@nist.gov> <200212151102.30779.mmclay@comcast.net> Message-ID: [Michael McLay] > On Sunday 15 December 2002 04:00 am, Brett Cannon wrote: > > [Michael McLay] > > > > Someone will correct me if I am wrong, but making a new decimal type that > > includes a new suffix for numbers is going to require a PEP. Your > > email, though, already seems like a rather good start for one and should > > not require much more work. > > Yes, there should be a PEP. I posted this as a work in progress because I had > a question on the semantics and also to get a general sense of what questions > a PEP might need to answer. > > OK. But please do a PEP. PEPs do not need to have all the answers to all questions. You an have a "Considerations" section where you list these questions with the pros and cons of possible decisions. It will also help to get more people to read it (i.e., people other than python-dev). > > It would be easier to get the module itself sans any Python core changes > > accepted initially to gauge usage and interest by the Python community > > before pushing for syntax integration. This is what is be done with the > > current rational implementation. > > The module has existed for quite some time and is available on sourceforge at > . I made some changes to the module and I > am interested in opinions on the change in the name of the class. > It is still going to take someone to champion the module to get BDFL pronouncment from Guido autonomously or get enough people to rally behind to put pressure on Guido to let it in. What I would like to know is if Tim ever pushed to get the module in, and if not, why? Since he is the original author I assume he felt some reason to write it (although it quite easily be "I was bored" ) and yet didn't feel like trying to get it into the stdlib. > > The syntax change is primarily to make Python more appealing to professions > that require the semantics of fixed point decimal numbers. I understand that, but who knows how many of those types of users are out there (current and potentiol). Going so far as to integrate it into Python requires proof that there is a demand. Just saying it will help make Python appealling to one specific group is not enough. If it was Python would be diluted with a ton of ideas that appeal to only a specific group. > compatibllity. In order to add decimal numbers as an alternative the syntax > must distinguish floats from decimal. The format I proposed is already used > to distinguish number types is used for complex and long numbers. > Understood and if it is decided to add decimals as a built-in I think the way you have proposed works very well. My advice is to champion getting the module into the language. As of right now that is the key getting anything to move forward. -Brett From martin@v.loewis.de Sun Dec 15 23:57:02 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 16 Dec 2002 00:57:02 +0100 Subject: [Python-Dev] PEP-298: buffer interface widening too much? In-Reply-To: <3DFCFD21.2090707@stsci.edu> References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira> <3DFCFD21.2090707@stsci.edu> Message-ID: Todd Miller writes: > I think that kind of backwards compatability just paves the wave for > extensions to stay broken and complicates the object structures as > well. I guess PEP-298 does create the possibility for creating correct > extensions, I just hate the thought of 5 getpointer calls where there > are now 3 and should be 2. I think this is the core of your objection. However, notice that all types implementing have to be modified in either proposal, and that adding the two getpointer functions should be trivial in most cases: - if the type has read/write memory, implement them both the same way; - if the type has read-only memory, set the write pointer to NULL - make the getpointer function a wrapper around the existing getpointer, requesting only a single segment and simultaneously locking the buffer. The overhead implementation-wise is really minimal; all the hard work will come from the locking code - see Thomas' patch to the string object for an example. We could even provide a Generic function which delegates the new get functions to the existing ones, accessing segment zero. > Sounds good to me. I hope you implement it! If PEP-286 were here > now, a simplified PEP-298 would be possible. I now recall what the issue is: you can't assume that modifying the argument tuple object to record resources is feasible, as this breaks apply (where in every case the user allocates a tuple, and where this is a plain old tuple). So I'm now envisioning a strategy where a thread-local (i.e. thread-state associated) stack is maintained - on argument tuple allocation, a frame is put on this stack, and all resources go into that frame, on tuple decrefing, the frame is cleared. There would be three kinds of resources: objects, void*, and locked buffers. For efficiency, this stack and its frames would be a flat memory block. In any case, I'm still -0 on PEP 298 as it stands: it should deal with much more aspects of buffers (and I now consider argument parsing to be one of them), when those things are added, I'll support the PEP - your objection that this adds too much API is of no concern to me. Regards, Martin From tim.one@comcast.net Mon Dec 16 06:48:01 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 16 Dec 2002 01:48:01 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212142106.01812.mclay@nist.gov> Message-ID: [Michael McLay] > ... > The semantics of the precision parameter may be incorrect because > it is only addressing the precision of the fraction part of the > number. That's why the module was called FixedPoint. > Shouldn't it reflect the number of significant digits? That would be more appropriate to a floating point scheme. Aahz has been working on a Python implementation of Mike Cowlishaw's hybrid proposal: http://www2.hursley.ibm.com/decimal/ for about, oh, fifteen years now . > ... > >>> 3.03e5d > 303000.00d > > The representation is incorrect. It should have returned 3.03e5d > but it padded the result with bogus zeros. It seems you don't really want a fixed-point type. The FixedPoint constructor avoided this rat hole by making the user be explicit about the fixed number of fractional digits (or accept the default 2). > ... > Python also supports fixed point decimal numbers. These numbers > do not suffer from the somewhat random roundoff errors that can occur > with binary floating point numbers. Yes they do. In fact, roundoff error in decimal arithmetic is actually a little *worse* than in binary arithmetic. The saving grace of decimal arithmetic has to do with representation error, not with roundoff error: decimal numbers are exactly representable in a decimal arithmetic system, but "almost all" decimal numbers can't be represented exactly in a binary arithmetic system. In addition, under FixedPoint's unusual "unbounded before the decimal point, fixed # of digits after, and when mixing precision the result gets the larger of the input precisions" rules, all of the following are always exact operations: to_decimal_string from_decimal_string + - * mod divmod Take away the "unbounded before the decimal point" and/or the "result inherits the larger input precision" rules and the arithmetic operations suffer roundoff errors, just as bad (actually a little worse) as binary fixed-#-of-bits fp suffers. > ... >>> 3.3d + decimal(3.03,3) > 6.330d > >>> decimal(1.1, 16) > 1.1000000000000001d > >>> > The builtin decimal constructor provides a means for converting > float and int types to decimal types. Since floats are approximations > of decimal floating point numbers there are often roundoff errors > introduced by using floats. > (The 1 in the last place of the conversion of 1.1 with 16 digits > is a binary round off error.) That's really representation error. Until the distinction becomes clear, you'll keep believing that decimal arithmetic is somehow better . [martin@v.loewis.de] > ... > It also appears that a decimal type feature in the language > will/should interact with a rational type. Possibly. This is easy for FixedPoint, becauase internally they're actually unbounded rationals in unnormalized form, with the denominator constrained to be an exact power of 10. [bsder@mail.allcaps.org] > ... > 2) There are official/semi-official standards for decimal floating- > point arithmetic. (IEEE 854 + ANSI X3.274 + ECMA 334) The > fixedpoint module seems to comply with these standards only at the > most basic levels. To the extent that it complies with them at all, that's an accident. See above: the overwhelming design goal of FixedPoint was to eliminate rounding error entirely in the basic arithmetic operations. That can't be achieved for division, but can be (and was) for the rest. Division settled for as-if-to-infinite-precision single-rounding to-nearest/even, which is the least error-prone way to do it when you *have* to lose information. [Brett Cannon] > ... > It is still going to take someone to champion the module to get BDFL > pronouncment from Guido autonomously or get enough people to rally > behind to put pressure on Guido to let it in. Guido already agreed to fold FixedPoint into 2.3, provided someone other than me do the work . Doug Fort has done a great job on it (including LaTeX docs, Fred!), and we're just missing someone with the time/desire to do final integration. That's independent of Michael's patch, of course. The FixedPoint module is just another library module. > What I would like to know is if Tim ever pushed to get the module in, > and if not, why? I've never pushed for any of my modules to get in. If a module attracts enough users, it eventually becomes clear that it should go in, else it becomes clear that it should remain out. FixedPoint has had a small but indecently grateful and growing audience for several years now. > Since he is the original author I assume he felt some reason > to write it (although it quite easily be "I was bored" ) and yet > didn't feel like trying to get it into the stdlib. At the time I wrote it, numerically incorrect "fixed point" / "BCD" / "decimal" classes were common on c.l.py. Simulating decimal arithmetic correctly isn't difficult if you've got the right background for it, but most people who want it don't have the right background to implement it. OTOH, I had a crushingly strong background to implement it, but didn't want it for myself. So after years of pointing out errors in flawed modules of this kind, it dawned on me that I could save more time overall by writing a correct implementation, as surprise-free as numerically possible. That's all there was to it, and you haven't seen a bad decimal module on c.l.py ever since . If Python were to grow a native decimal type, I'd rather move to Cowlishaw's design, because it has the best shot at becoming a standard influential enough to sway HW design (decimal fp in HW wouldn't be notably more expensive or slower than binary fp in HW, given current chip capacities; this was a much bigger deal when chip real estate was scarce). >> The syntax change is primarily to make Python more appealing to >> professions that require the semantics of fixed point decimal numbers. > I understand that, but who knows how many of those types of users are > out there (current and potentiol). "Do applications actually use decimal data?" from Cowlishaw's FAQ should go a way toward answering that: http://www2.hursley.ibm.com/decimal/decifaq.html From brett@python.org Mon Dec 16 07:07:04 2002 From: brett@python.org (Brett Cannon) Date: Sun, 15 Dec 2002 23:07:04 -0800 (PST) Subject: [Python-Dev] summary for 2002-12-01 through 2002-12-15 Message-ID: Yes, it's that time again! You guys (are there any ladies on python-dev?) have about a day to let me know about my mistakes before I send this out to the rest of the world. Oh, I am leaving off the FixedPoint discussion on this summary on purpose; I feel that it will be all clarified soon and so I am going to wait to summarize it after that has occurred. And Martin, your name is correctly spelled (complete with emulaut!) in the original and should hopefully show up when I send it out to c.l.p. Enjoy. +++++++++++++++++++++++++++++++++++++++++++++++++++++ python-dev Summary for 2002-12-01 through 2002-12-15 +++++++++++++++++++++++++++++++++++++++++++++++++++++ ====================== Summary Announcements ====================== Aahz suggested that I try writing the summary from a third-person perspective so as to remove ambiguity if anyone ever quotes something from a summary which was written in first-person. I am giving it a go in this summary although there is not much here that is affected. There was an error in the last summary where I gave Guido's position on something incorrectly (summary of dict evaluation order). It has been fixed. I am going to fix errors like that in old summaries with an errata note in the text version. It won't show up in the HTML version, though. And go to PyCon_. =) .. _PyCon: http://www.python.org/pycon/ =================================== `New universal import mechanism`__ =================================== __ http://mail.python.org/pipermail/python-dev/2002-December/030602.html Splinter threads: - `New and Improved Import Hooks `__ - `Re: New and Improved Import Hooks `__ - `Re[2]: [Python-Dev] New and Improved Import Hooks `__ - `New Import Hook & New Zip Importer `__ - `(no subject) `__ - `Another approach for the import mechanism `__ - `Re: Another approach for the import mechanism `__ - `New Import Hook & New Zip Importer `__ - `Import: where's the PEP? `__ - `zipimport & import hooks `__ - `VFS `__ - `Zip imports, PEP 273, and the .zip extension `__ - `Import compromise `__ - `zipimport, round 3 (or would that be 37?) `__ - `Complexity of import.c `__ - `zipimport, round 4 `__ - `new import hooks & zip import `__ In case you can't figure it out from the title of all of the splnter threads, there was an immense discussion about `PEP 273`_ and getting an import mechanism for zipped modules. To give a little back-story, PEP 273 proposes allowing modules to be put into a zip file for easy distribution. You could then just add the zip file to ```sys.path`_`` , PYTHONPATH_ , etc. and importing from the zip file would be handled properly. It was scheduled to be included in Python 2.3 thanks to Paul Moore keeping James Ahlstrom's patch synced with CVS. All was right with the world, more or less. But then Wiktor Sadowski commented on a checkin for `import.c`_ and how he would like a better way to write custom imports. This led to Just van Rossum to write a new import mechanism. Basically Just was unhappy with how much fiddling the PEP 273 patch was doing to import.c. So, just like any self-respecting programmer would do, he came up with his own solution. =) Most of the discussion revolved around a few central points. One was whether this was even worth the effort. Since the PEP 273 patch worked, why should Just bother reinventing the wheel? The whole reason Just decided to come up with another implementation was a major point. The other was that he felt he could come up with a more general way of allowing custom importers *and* have zip imports a part of the core. There was also a discussion of whether Gordon McMillan's `iu.py`_ wasn't a good solution; it was already well-tested and coded. But Just wanted a certain functionality that iu.py didn't offer... Just wanted to be able to put arbitrary objects in ``sys.path``. This would allow the object to be called to figure out whether it had the module being requested or not. Taking the zip importer as an example, there would be a zipimporter instance in ``sys.path`` for each zip file in the path (once it was discovered it was a zip file; bootstrapping has been carefully handled). The issue of backward-compatibility was instantly brought up. It has been assumed that everything in ``sys.path`` was a string and starting to include zipimporter objects would break that. The idea of making zipimporters (and any subsequent importers in the stdlib) subclasses of strings was brought up and basically accepted, although Guido viewed it as an ugly hack. But in the end, Just's version acts like iu.py. The basic algorithm for handling zip files is:: def find_module(name, path): if isbuiltin(name): return builtin_filedescr if isfrozen(name): return frozen_filedescr if path is None: path = sys.path for p in sys.path: # I think Just meant to say ``path`` here try: v = zipimporter(p) except ZipImportError: pass else: w = v.find_module(name) if w is not None: return w ...handle builtin file system import... It is easily generalized to work for any registered importers. There is also the idea of adding something called a "metapath" that iu.py has. It would allow one to override any import by catching it before trying ``sys.path``. One other discussion was over this was going to need a PEP. Just has written up an explanation, but as to whether a PEP is required has not been decided. Just's implementation is now a patch at http://www.python.org/sf/652586 . .. _PEP 273: http://www.python.org/peps/pep-0273.html .. _sys.path: http://www.python.org/doc/current/lib/module-sys.html .. _PYTHONPATH: http://www.python.org/doc/current/tut/node8.html .. _import.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Python/compile.c .. _iu.py: http://www.mcmillan-inc.com/iu.html =================== `Mac OSX issues`__ =================== __ http://mail.python.org/pipermail/python-dev/2002-December/030604.html Continuation of http://mail.python.org/pipermail/python-dev/2002-November/030525.html Guido got a hold of a OS X box and ran the test suite on it and found four tests that didn't pass. One was for ``locale`` (which Guido didn't care about), another was the ``test_largefile`` test, another was for ``re``, and the last one was for ``socket``. The ``test_largefile`` suite seeks on a new file to a huge position. On OS X and Windows 2000 this actually adds to the file up to the seek point so as to make it a complete file. Windows 9x, on the other hand, just writes to that offset regardless of whether it is within the bounds of the file. Because it creates such a huge file it takes forever to complete and thus has been made an optional test on OS X. The ``re`` tests were failing for a lack of space on the stack. to allow the tests to pass you need to execute the statement ``ulimit -s 2000`` which ups the stack size to approximately 2 Mb. This has sense been added to the regression testing framework to be done for you if needed. ``socket`` was failing on ``.getsockbyaddr()`` when trying it for the machine's own name. Barry Warsaw discovered that the test passed if you set hostnames by using NetInfo. ============================================================== `RE: how to kill process on Windows started with os.spawn?`__ ============================================================== __ http://mail.python.org/pipermail/python-dev/2002-December/030595.html Skip Montanaro wanted to be able to kill a Windows process externally. He suggested implemented a ``os.kill()`` function that would kill a process in a platform-independent way. Problem is that Windows wants the process hook which is different from what POSIX wants. Another issue is that the Windows docs say that you should not kill processes externally. It seems the idea has been dropped. ========================================================= `extension module .o files wind up in the wrong place`__ ========================================================= __ http://mail.python.org/pipermail/python-dev/2002-December/030644.html Skip Montanaro discovered that if you built Python in a subdirectory of the source directory (e.g., source in ``source/`` and you are building in ``source/build/``) the .o files are put in the wrong place (they use ``source/`` as the root directory to place things instead of ``source/build/``). Skip resolved his issue by doing a ``make distclean`` and then running ``../configure``. ========================================= `Make universal newlines non-optional`__ ========================================= __ http://mail.python.org/pipermail/python-dev/2002-December/030692.html Martin v. Lwis asked if anyone mind not making universal newline support a compile option. He volunteered to remove the ``#ifdef`` statements and such to get this done. Jack Jansen (who wrote the universal newline code) pointed out that if it was required there could be issues on certain platforms. Martin had no issue with this since this would force the platform maintainers to fix the code for it to work. In the end Martin suggested leaving it an option in 2.3 and removing it as an option and requiring it in 2.4. ============================================ `Re: [Patches] Patch for xmlrpc encoding`__ ============================================ __ http://mail.python.org/pipermail/python-dev/2002-December/030984.html A discussion over a patch for ``xmlrpclib`` ended up on python-dev. The relevancy of it was that the statement ``sys.{get,set}defaultencoding`` is only useful for coercion of Unicode to byte strings. It was also stated that ASCII should always be assumed to be the default encoding. Also, never put any non-ASCII text in a string; that is what Unicode objects are for. ========================== `PEP 242 Numeric kinds`__ ========================== __ http://mail.python.org/pipermail/python-dev/2002-December/030859.html Paul DuBois emailed the list mentioning how a group of people were supposed to get together to decide the fate of `PEP 242`_ in June 2002; that didn't happen. So Paul suggested making the suggested "making it a separate deliverable on the Numeric_ site and withdrawing the PEP". Everyone who responded pretty much said to add the proposed module. A final outcome has not been decided as to whether Paul will rescind the PEP or push it forward. If you have an opinion on the matter, let it be known. .. _PEP 242: http://www.python.org/peps/pep-0242.html .. _Numeric: http://www.pfdubois.com/numpy/ ================== `We got leaks!`__ ================== __ http://mail.python.org/pipermail/python-dev/2002-December/030933.html Tim Peters discovered that the new datetime_ type was leaking memory. The C version was slowly leaking about 10 references per loop through the testing suite. But the Python version was leaking 37 reference per loop through. Tim did his reference checking using this chunk of code:: def test_main(): import gc r = unittest.TextTestRunner(stream=sys.stdout, verbosity=2) s = test_suite() while True: r.run(s) gc.collect() print gc.garbage # this is always an empty list print '*' * 10, 'total refs:', sys.gettotalrefcount() This substituted for the ``test_main()`` for the testing suite printed out the reference numbers rather nicely. Well, it didn't take the weekend to solve. Kevin Jacobs stepped up to help. The first thing he found was that it was not occurring in a debug build (which is why when you are testing new code you should **always** run it in a debug build). Then Kevin discovered the issue came up when executing a ``__hash__()`` call on anything. Tim then realized a missing decrement in ``slot_tp_hash()`` was the culprit. Then Kevin found another culprit in ``cPickle_``. That was also patched (and everything has been flagged to be backported to the 2.2 branch). The reason this thread is being mentioned is because it 1) was all solved in 5 hours and 40 minutes which is pretty cool and 2) using the chunk of code that Tim provided to stare at the reference count is a good way to look for memory leaks in your code. .. _datetime: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/nondist/sandbox/datetime/ .. _cPickle: http://www.python.org/doc/current/lib/module-cPickle.html ========================= `os.path.commonprefix`__ ========================= __ http://mail.python.org/pipermail/python-dev/2002-December/030947.html Armin Rigo noticed how ``os.path.commonprefix()`` works character-by-character and not by path separators as one might expect (although the documentation does warn about its char-by-char nature). Skip Montanaro suggested ``os.path.commonpathprefix()`` be added. He mentioned that `Tools/scripts/trace.py`_ had such functionality. As of this writing no one has taken the initiative to add the function. .. _Tools/scripts/trace.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/scripts/trace.py ======================================================== `__getstate__() not inherited when __slots__ present`__ ======================================================== __ http://mail.python.org/pipermail/python-dev/2002-December/030980.html Greg Ward noticed that "If a class and its superclass both define __slots__, it appears that __getstate__() is not inherited from the superclass." He wasn't sure if this was a bug or a feature. Well, it's a feature; since there is no guarantee that the superclass's ``__getstate__()`` will know about the subclass's ``__slots__``, it is not automatically inherited. Kevin Jacobs provided the following snippet of code to deal with this "feature":: def __getstate__(self): state = getattr(self, '__dict__', {}).copy() for obj in type(self).mro(): for name in getattr(obj,'__slots__',()): if hasattr(self, name): state[name] = getattr(self, name) return state def __setstate__(self, state): for key,value in state.items(): setattr(self, key, value) ==================== `Tarfile support`__ ==================== __ http://mail.python.org/pipermail/python-dev/2002-December/031086.html Gustavo Niemeyer asked for some help in reviewing the patch ( http://www.python.org/sf/651082 ) that adds Lars Gustabel's tarfile module to the stdlib. The question of the license came up. Lars said he would be willing to change the license over to the `Python Software Foundation`_ . An initial license hand-off agreement is available at http://www.python.org/psf/psf-contributor-agreement.html . .. _Python Software Foundation: http://www.python.org/psf/ ============================== `Tcl, Tkinter, and threads`__ ============================== __ http://mail.python.org/pipermail/python-dev/2002-December/031107.html Martin v. Lwis has now modified ```_tkinter`_`` as so to be able to be used when Tcl has been compiled with thread support. Special thanks goes out to Martin for taking over maintenance of the module. .. __tkinter: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/_tkinter.c ============ `PEP 298`__ ============ __ http://mail.python.org/pipermail/python-dev/2002-December/031109.html Splinter threads: - `PEP-298: buffer interface widening too much? `__ - `PEP-298: buffer interface widening too much? `__ Thomas Heller wanted to resolve whether to move forward with an implementation for `PEP 298`_ . This led to a discussion over the specifics of the PEP. Martin v. Lwis wnated some more specific wording from the PEP. Todd Miller then stepped in to say he didn't like how much of an interface was being tacked on. Martin commented that his `PEP 286`_ would help in this whole situation, which Todd ended up agreeing with. .. _PEP 298: http://www.python.org/peps/pep-0298.html .. _PEP 286: http://www.python.org/peps/pep-0286.html From mwh@python.net Mon Dec 16 11:20:54 2002 From: mwh@python.net (Michael Hudson) Date: 16 Dec 2002 11:20:54 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/test test_descr.py,1.169,1.170 In-Reply-To: gvanrossum@users.sourceforge.net's message of "Fri, 13 Dec 2002 09:49:47 -0800" References: Message-ID: <2mfzsyciqx.fsf@starship.python.net> gvanrossum@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib/test > In directory sc8-pr-cvs1:/tmp/cvs-serv7366/Lib/test > > Modified Files: > test_descr.py > Log Message: > Tighten the tests for assignment to __bases__: disallow empty tuple. Oh, I knew about that, but didn't think it was worth bothering with. Sorry. Cheers, M. -- That being done, all you have to do next is call free() slightly less often than malloc(). You may want to examine the Solaris system libraries for a particularly ambitious implementation of this technique. -- Eric O'Dell, comp.lang.dylan (& x-posts) From Jack.Jansen@cwi.nl Mon Dec 16 13:15:07 2002 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 16 Dec 2002 14:15:07 +0100 Subject: [Python-Dev] PEP 298 In-Reply-To: <200212131548.gBDFmrm09751@odiug.zope.com> Message-ID: <659DE131-10F8-11D7-BB61-0030655234CE@cwi.nl> On Friday, Dec 13, 2002, at 16:48 Europe/Amsterdam, Guido van Rossum wrote: >> The PyArg_Parse replacement I would like to do (for which I have had, >> unfortunately, absolutely no support up to now, and which I don't feel >> up to designing all by myself) could fix this with PEP298: it would >> use >> the locked buffer interface if available on the object, and release >> the >> locked buffer after your method returns. > > I can't give you much more than moral support. :-( > > It sounds like it would be a Big Change though; are you sure you're up > to this for Python 2.3? No, I don't think it'll happen before that time, there's oodles of MacOSX stuff I still want to do, and that'll keep me off the street for the time being. But if we have the process underway before 2.3 I hope I may be able to convince you that we should keep PyArg_Parse semantics as they are. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From guido@python.org Mon Dec 16 13:30:41 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 08:30:41 -0500 Subject: [Python-Dev] PEP 298 In-Reply-To: Your message of "Mon, 16 Dec 2002 14:15:07 +0100." <659DE131-10F8-11D7-BB61-0030655234CE@cwi.nl> References: <659DE131-10F8-11D7-BB61-0030655234CE@cwi.nl> Message-ID: <200212161330.gBGDUfi06503@pcp02138704pcs.reston01.va.comcast.net> > But if we have the process underway before 2.3 I hope I may be able > to convince you that we should keep PyArg_Parse semantics as they > are. Twist my arm! :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From jmiller@stsci.edu Mon Dec 16 14:28:28 2002 From: jmiller@stsci.edu (Todd Miller) Date: Mon, 16 Dec 2002 09:28:28 -0500 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira> <3DFCFD21.2090707@stsci.edu> Message-ID: <3DFDE30C.7080009@stsci.edu> Martin v. Löwis wrote: >Todd Miller writes: > > > >>I think that kind of backwards compatability just paves the wave for >>extensions to stay broken and complicates the object structures as >>well. I guess PEP-298 does create the possibility for creating correct >>extensions, I just hate the thought of 5 getpointer calls where there >>are now 3 and should be 2. >> >> > >I think this is the core of your objection. However, notice that all > You're right I think, but I also percieve the complexity of extension writing going up as well. Perhaps it's a false perception which can be dispelled by documentation stating that new extensions should *always* use the locking interface, with the old interface preserved for the sole purpose of backwards compatability. Without some thought, the following was not clear: the need for locking is a property of specific types, not specific extension functions. Therefore the existence of two get pointer varieties implies an ambguity which is not really there: should I use the locking or non-locking variety? I think the answer is always: use the locking variety. >types implementing have to be modified in either proposal, and that >adding the two getpointer functions should be trivial in most cases: >- if the type has read/write memory, implement them both the same > way; >- if the type has read-only memory, set the write pointer to NULL >- make the getpointer function a wrapper around the existing > getpointer, requesting only a single segment and simultaneously > locking the buffer. > >The overhead implementation-wise is really minimal; all the hard work >will come from the locking code - see Thomas' patch to the string > > Thanks for the pointer. >object for an example. We could even provide a Generic function which >delegates the new get functions to the existing ones, accessing >segment zero. > > Wouldn't this require more information about the lock implementation? As PEP-298 stands, I thought that was left up to the buffered type. It might make sense to change the locking getpointer fields into lock, and release into unlock. Then compose the primitive operations in the wrapper as you suggested. That wouldn't fix the size_t problem though. > > >>Sounds good to me. I hope you implement it! If PEP-286 were here >>now, a simplified PEP-298 would be possible. >> >> > >I now recall what the issue is: you can't assume that modifying the >argument tuple object to record resources is feasible, as this breaks >apply (where in every case the user allocates a tuple, and where this >is a plain old tuple). > > Is the efficiency of apply() so critical that a conversion is ruled out? >So I'm now envisioning a strategy where a thread-local >(i.e. thread-state associated) stack is maintained - on argument tuple >allocation, a frame is put on this stack, and all resources go into >that frame, on tuple decrefing, the frame is cleared. There would be >three kinds of resources: objects, void*, and locked buffers. For >efficiency, this stack and its frames would be a flat memory block. > > Way over my head here... I guess the point is that arg tuples aren't the only point of attack. >In any case, I'm still -0 on PEP 298 as it stands: it should deal with >much more aspects of buffers (and I now consider argument parsing to >be one of them), when those things are added, I'll support the PEP - >your objection that this adds too much API is of no concern to me. > > Well, thanks for discussing it anyway. I had a feeling this conversation would end this way before I ever opened my mouth, but I wanted to change the balance between saying something and saying *nothing*. >Regards, >Martin > Regards, Todd > > From pje@telecommunity.com Mon Dec 16 15:15:01 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Mon, 16 Dec 2002 10:15:01 -0500 Subject: [Python-Dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <3DFDCE88.1020709@zope.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> Message-ID: <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> (CC'd to python-dev because of possible impact on zipimport implementation/API) At 08:00 AM 12/16/02 -0500, Jim Fulton wrote: >Each zope package __init__ file would have a bit of boilerplate code >that would include all zope directories in sys.path: > > # Boilerplate code that combines all zope directories in sys.path > import sys, os > __path__ = [os.path.join(p, 'zope') for p in sys.path] > >Jim That should probably read: __path__ = [os.path.join(p, 'zope') for p in sys.path if isinstance(p,StringTypes)] or else it could break in 2.3 if somebody's putting non-strings on sys.path. I don't remember exactly how the python-dev discussion turned out, but I kind of got the impression that it was going to be allowable in 2.3. Also, I'm pretty sure it will not give the expected/desired results in conjunction with zipfile imports. For these and related reasons, I would suggest that a slightly different snippet be used instead (the names are lousy, improvements welcome): from namespace_packages import makeNamespacePath __path__ = makeNamespacePath('zope') And then make whatever carries "namespace_packages" be something that's distributed separately from Zope, preferably part of the standard library or distutils (and backported where possible). If we can make it so that the __init__ code never *has* to change, that will minimize the likelihood of incompatible __init__.py's floating around. Meanwhile, a module that actually *knows* something about what can be on sys.path (and how to interpret it) can do so. In order to support something like this, importer objects would need to be able to return something that's the moral equivalent of "os.path.join" on an existing sys.path entry; either another importer object or a string that can be handled by one. At this point, checking for a '.zip' extension at the end of a string isn't going to cut it any more. From guido@python.org Mon Dec 16 15:47:14 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 10:47:14 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> Message-ID: <200212161547.gBGFlEB01302@odiug.zope.com> > (CC'd to python-dev because of possible impact on zipimport > implementation/API) > > At 08:00 AM 12/16/02 -0500, Jim Fulton wrote: > >Each zope package __init__ file would have a bit of boilerplate code > >that would include all zope directories in sys.path: > > > > # Boilerplate code that combines all zope directories in sys.path > > import sys, os > > __path__ = [os.path.join(p, 'zope') for p in sys.path] > > > >Jim > > That should probably read: > > __path__ = [os.path.join(p, 'zope') for p in sys.path if > isinstance(p,StringTypes)] > > or else it could break in 2.3 if somebody's putting non-strings on > sys.path. I don't remember exactly how the python-dev discussion > turned out, but I kind of got the impression that it was going to be > allowable in 2.3. Correct. While the standard setup will still have only strings, we'll support non-strings on sys.path and some users may want to use this. > Also, I'm pretty sure it will not give the expected/desired results > in conjunction with zipfile imports. I think it will, actually; if /path/foo.zip is a zip file on sys.path, /path/foo.zip/blah references a directory inside that zip file. > For these and related reasons, I would suggest that a slightly > different snippet be used instead (the names are lousy, improvements > welcome): > > from namespace_packages import makeNamespacePath > __path__ = makeNamespacePath('zope') > > And then make whatever carries "namespace_packages" be something > that's distributed separately from Zope, preferably part of the > standard library or distutils (and backported where possible). If > we can make it so that the __init__ code never *has* to change, that > will minimize the likelihood of incompatible __init__.py's floating > around. Meanwhile, a module that actually *knows* something about > what can be on sys.path (and how to interpret it) can do so. This is not a bad idea: put the logic to add stuff to a package's __path__ according to a certain policy in the standard library. (Of course, getting it into Python 2.2 would require another release, 2.2.3 -- which may happen anyway.) > In order to support something like this, importer objects would need > to be able to return something that's the moral equivalent of > "os.path.join" on an existing sys.path entry; either another > importer object or a string that can be handled by one. At this > point, checking for a '.zip' extension at the end of a string isn't > going to cut it any more. As I said above, I think it will. Time's running short (we're close to releasing Python 2.3 alpha 1); we should decide on a name and a specific policy ASAP. I propose the following: from pkgutil import extended_path __path__ = extended_path(__path__, __name__) Where pkgutil.py could be something like this: import sys.path def extended_path(path, name): """Extend a package's path. (XXX more.)""" path = path[:] for dir in sys.path: if isinstance(dir, (str, unicode)): dir = os.path.join(dir, name) if dir not in path and os.path.isdir(dir): path.append(dir) return path --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Dec 16 16:02:47 2002 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 16 Dec 2002 11:02:47 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> Message-ID: <15869.63783.430525.295170@gargle.gargle.HOWL> >>>>> "PJE" == Phillip J Eby writes: PJE> That should probably read: | __path__ = [os.path.join(p, 'zope') for p in sys.path if | isinstance(p,StringTypes)] PJE> or else it could break in 2.3 if somebody's putting PJE> non-strings on sys.path. I don't remember exactly how the PJE> python-dev discussion turned out, but I kind of got the PJE> impression that it was going to be allowable in 2.3. Also, PJE> I'm pretty sure it will not give the expected/desired results PJE> in conjunction with zipfile imports. PJE> For these and related reasons, I would suggest that a PJE> slightly different snippet be used instead (the names are PJE> lousy, improvements welcome): | from namespace_packages import makeNamespacePath | __path__ = makeNamespacePath('zope') Random, harried thought: maybe we can use .pth files to extend the __path__? IOW, something like the above would search sys.path for zope.pth files, and do the equivalent to what site.py already does. -Barry From guido@python.org Mon Dec 16 16:07:27 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:07:27 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 11:02:47 EST." <15869.63783.430525.295170@gargle.gargle.HOWL> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <15869.63783.430525.295170@gargle.gargle.HOWL> Message-ID: <200212161607.gBGG7Rs01448@odiug.zope.com> > Random, harried thought: maybe we can use .pth files to extend the > __path__? IOW, something like the above would search sys.path for > zope.pth files, and do the equivalent to what site.py already does. That would be a useful addition to the pkgutil.py module that I just proposed. --Guido van Rossum (home page: http://www.python.org/~guido/) From Paul.Moore@atosorigin.com Mon Dec 16 16:12:40 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 16 Dec 2002 16:12:40 -0000 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) Message-ID: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> Guido said: > Time's running short (we're close to releasing Python 2.3 alpha 1); > we should decide on a name and a specific policy ASAP. I propose the > following: >=20 > from pkgutil import extended_path > __path__ =3D extended_path(__path__, __name__) >=20 > Where pkgutil.py could be something like this: >=20 > import sys.path >=20 > def extended_path(path, name): > """Extend a package's path. (XXX more.)""" > path =3D path[:] > for dir in sys.path: > if isinstance(dir, (str, unicode)): > dir =3D os.path.join(dir, name) > if dir not in path and os.path.isdir(dir): > path.append(dir) > return path Are you proposing this for the standard library? It certainly sounds reasonable, but the documentation should make clear that it still fails for certain types of sys.path entry. Specifically, - string subclasses will get converted into basic strings - strings which don't have a path structure wil break (and of course, objects other than string subclasses don't get "extended"). I don't think that either of these is even remotely a showstopper, but they do need to be documented - specifically because as a module, this code can be imported by anybody, and so the constraints on the user should be made clear. In the long term, I think the two most likely things to appear on sys.path, apart from directory names, are - "Filesystem namespace extensions" like the zipfile support. That is, things which act like virtual directories. - raw importer objects Frankly, just dumping a raw imported onto sys.path (or a package path) is so much easier than implementing a string naming scheme, that unless there's an overwhelmingly natural string representation (virtual directory or URL are the only two that come to mind) it's simpler not to bother. So saying that module paths should only contain pathlike strings, or importer objects, is probably reasonable. I wouldn't recommend enforcing this, just documenting it as the normal convention. Modules can document that they don't work with applications that violate this convention, and relentless experimentalists still have the freedom to break the recommendation (at their own risk!) if they want to see what happens... Paul. From barry@zope.com Mon Dec 16 16:16:19 2002 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 16 Dec 2002 11:16:19 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <15869.63783.430525.295170@gargle.gargle.HOWL> <200212161607.gBGG7Rs01448@odiug.zope.com> Message-ID: <15869.64595.601832.869032@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: >> Random, harried thought: maybe we can use .pth files to extend >> the __path__? IOW, something like the above would search >> sys.path for zope.pth files, and do the equivalent to what >> site.py already does. GvR> That would be a useful addition to the pkgutil.py module that GvR> I just proposed. The nice thing is if we can localize things to pkgutil.py, we wouldn't /have/ to release Python 2.2.3 to get backported support. Just release pkgutil.py as its own distutils package which Zope3 could "setup.py install" first thing. It lands in site-packages for older Python's to use, but when it gets released with Python 2.3, it'll be ignored in favor of the standard library version. -Barry From guido@python.org Mon Dec 16 16:22:13 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:22:13 -0500 Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects abstract.c,2.93.6.7,2.93.6.8 In-Reply-To: Your message of "Fri, 13 Dec 2002 20:26:35 EST." References: Message-ID: <200212161622.gBGGMD401543@odiug.zope.com> > This backport checkin (Oct 5 2002) changed the exception text between > 2.2.1: > > >>> int(None) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: object can't be converted to int > > and 2.2.2 (last line): > > >>> int(None) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: int() argument must be a string or a number > > Should this change have been backported to 2.2.2 from the 2.3 > codebase? Clarifications of error messages is acceptable for bugfix releases. Some error messages need more clarification than others, and you can argue about this particular one, but I don't think you can rely on all error messages remaining the same within bugfix releases. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 16 16:24:01 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:24:01 -0500 Subject: [Python-Dev] Deprecate doctest? In-Reply-To: Your message of "Sat, 14 Dec 2002 14:42:15 EST." <20021214194215.GA9536@panix.com> References: <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> Message-ID: <200212161624.gBGGO2j01562@odiug.zope.com> > That's tricky, given that doctest is in the standard library, and that's > exactly what doctest does. Are you advocating deprecating doctest? You gotta be kidding. --Guido van Rossum (home page: http://www.python.org/~guido/) From pje@telecommunity.com Mon Dec 16 16:23:47 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Mon, 16 Dec 2002 11:23:47 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <200212161547.gBGFlEB01302@odiug.zope.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> Message-ID: <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> (Summary: good plan for name and place, issues exist w/sketched implementation) At 10:47 AM 12/16/02 -0500, Guido van Rossum wrote: >Time's running short (we're close to releasing Python 2.3 alpha 1); >we should decide on a name and a specific policy ASAP. I propose the >following: > > from pkgutil import extended_path > __path__ = extended_path(__path__, __name__) How would that work for say, 'zope.app'? >Where pkgutil.py could be something like this: > > import sys.path > > def extended_path(path, name): > """Extend a package's path. (XXX more.)""" > path = path[:] > for dir in sys.path: > if isinstance(dir, (str, unicode)): > dir = os.path.join(dir, name) > if dir not in path and os.path.isdir(dir): > path.append(dir) > return path > >--Guido van Rossum (home page: http://www.python.org/~guido/) The 'isdir()' part isn't going to work on zipfiles... unless os.path is going to have zipfile support in 2.3. :) From guido@python.org Mon Dec 16 16:27:52 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:27:52 -0500 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Your message of "Sat, 14 Dec 2002 21:06:17 EST." <200212142106.01812.mclay@nist.gov> References: <200212142106.01812.mclay@nist.gov> Message-ID: <200212161627.gBGGRq801621@odiug.zope.com> > sf/653938I is a patch to integrate the fixedpoint module, that was > created by Tim Peters, into Python as a first class object. That is, > the patch adds this new number type so that it has direct syntax > support like float, int, long, str., etc. within the interpreter. (a) I don't want any (more) change to the syntax for Python 2.3. (b) Like the sets module, proposals for new datatypes should first be prototyped as standard library modules before being promoted to built-in status (with their own literals). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 16 16:31:15 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:31:15 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: Your message of "Sat, 14 Dec 2002 16:29:27 +0100." <015b01c2a385$97010700$6d94fea9@newmexico> References: <015b01c2a385$97010700$6d94fea9@newmexico> Message-ID: <200212161631.gBGGVFB01638@odiug.zope.com> > latest CVS > > Python 2.3a0 (#1, Dec 14 2002, 07:19:32) > [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > consider > > >>> import rexec > >>> r=rexec.RExec() > >>> r.r_exec("class C: pass\nC.a=0") > Traceback (most recent call last): > File "", line 1, in ? > File "/home/users/p/pe/pedronis/python/dist/src/Lib/rexec.py", line 309, in > r_ > exec > exec code in m.__dict__ > File "", line 2, in ? > RuntimeError: classes are read-only in restricted mode > > vs. > > >>> r.r_exec("class C(object): pass\nC.a=0") > >>> Hm... Do you see any way to break out of restricted execution mode using this? I suppose a fix would be simple enough, but I'm more and more inclined to simply rip out rexec from the distribution -- it's never going to be safe, and I doubt it's very useful as long as it's not safe. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Dec 16 16:38:13 2002 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 16 Dec 2002 11:38:13 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <15869.63783.430525.295170@gargle.gargle.HOWL> <200212161607.gBGG7Rs01448@odiug.zope.com> Message-ID: <15870.373.513177.215927@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: >> Random, harried thought: maybe we can use .pth files to extend >> the __path__? IOW, something like the above would search >> sys.path for zope.pth files, and do the equivalent to what >> site.py already does. GvR> That would be a useful addition to the pkgutil.py module that GvR> I just proposed. BTW, in the spirit of TOOWTDI, here's a very quick hack... -Barry -------------------- snip snip -------------------- # Use like this: # # from pkgutil import extendpath # __path__ = extendpath(__path__, __name__) import os import sys from site import makepath def _read(basedir, fp): dirs = {} # Stolen largely from site.py for line in fp: if line.startswith('#'): continue if line.startswith('import'): exec line continue if line.endswith('\n'): line = line[:-1] dir, dircase = makepath(basedir, line) if os.path.exists(dir): dirs[dir] = True return dirs def extendpath(path, name): path = dict(zip(sys.path, [True for x in sys.path])) for dir in sys.path: if isinstance(dir, (str, unicode)): pth = os.path.join(dir, name + os.extsep + 'pth') try: fp = open(pth) except IOError: continue try: path.update(_read(dir, fp)) finally: fp.close() return path.keys() From guido@python.org Mon Dec 16 16:42:34 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:42:34 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 16:12:40 GMT." <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> Message-ID: <200212161642.gBGGgYJ03474@odiug.zope.com> > > Time's running short (we're close to releasing Python 2.3 alpha 1); > > we should decide on a name and a specific policy ASAP. I propose the > > following: > > > > from pkgutil import extended_path > > __path__ = extended_path(__path__, __name__) > > > > Where pkgutil.py could be something like this: > > > > import sys.path > > > > def extended_path(path, name): (BTW I now think extend_path() is a better and simpler name for this.) > > """Extend a package's path. (XXX more.)""" > > path = path[:] > > for dir in sys.path: > > if isinstance(dir, (str, unicode)): > > dir = os.path.join(dir, name) > > if dir not in path and os.path.isdir(dir): > > path.append(dir) > > return path > > Are you proposing this for the standard library? Yes. > It certainly sounds reasonable, but the documentation should make > clear that it still fails for certain types of sys.path entry. > > Specifically, > > - string subclasses will get converted into basic strings I find string subclasses an ugly hack that should not be used. The meta hook (or whatever it's called now) should do this. > - strings which don't have a path structure wil break We can fix that by skipping anything for which os.path.isdir(dir) [for the original value of dir] isn't true. > (and of course, objects other than string subclasses don't get > "extended"). We can use some kind of extension hook to deal with those. The important point is that the idiom placed in __init__.py can remain the same (the two lines quoted above) while the implementation of pkgutil.extend_path() can evolve or be modified (even monkey-patched if you're desperate) as the import hooks evolve. The main reason for putting this in the stdlib is to avoid the need to avoid the boilerplate code in everybody's __init__.py. > I don't think that either of these is even remotely a > showstopper, but they do need to be documented - specifically > because as a module, this code can be imported by anybody, and so > the constraints on the user should be made clear. Of course. Though with the current state of documentation, I give less priority to documenting every little detail of this code than to getting it in the code base (with *some* documentation, for sure). > In the long term, I think the two most likely things to appear on > sys.path, apart from directory names, are > > - "Filesystem namespace extensions" like the zipfile support. > That is, things which act like virtual directories. > - raw importer objects > > Frankly, just dumping a raw imported onto sys.path (or a package > path) is so much easier than implementing a string naming > scheme, that unless there's an overwhelmingly natural string > representation (virtual directory or URL are the only two that > come to mind) it's simpler not to bother. OTOH I expect that most things you'd want to place on sys.path *do* have a natural string representation. After all, if something doesn't have a name, it's hard to talk about. And if it has a name, it can be placed in PYTHONPATH (with the minor quibble that URLs use ':' which -- at least on Unix -- is also used as the separator in *PATH variables. :-( ) > So saying that module paths should only contain pathlike strings, > or importer objects, is probably reasonable. I wouldn't recommend > enforcing this, just documenting it as the normal convention. > Modules can document that they don't work with applications that > violate this convention, and relentless experimentalists still > have the freedom to break the recommendation (at their own risk!) > if they want to see what happens... Maybe importer objects should have a standard API that lets you ask for an extended object. --Guido van Rossum (home page: http://www.python.org/~guido/) From mclay@nist.gov Mon Dec 16 16:45:34 2002 From: mclay@nist.gov (Michael McLay) Date: Mon, 16 Dec 2002 11:45:34 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: References: Message-ID: <200212161145.21295.mmclay@comcast.net> On Monday 16 December 2002 01:48 am, Tim Peters wrote: > [Michael McLay] > > > ... > > The semantics of the precision parameter may be incorrect because > > it is only addressing the precision of the fraction part of the > > number. > > That's why the module was called FixedPoint. Fixing the decimal place would make sense for applications having to do with money. That is quite different than how numbers are treated in scientific calculations. That is were I borrowed the idea of significant digits. Accountants tend to think that all digits are significant, down to the last penny. > > Shouldn't it reflect the number of significant digits? > > That would be more appropriate to a floating point scheme. Aahz has been > working on a Python implementation of Mike Cowlishaw's hybrid proposal: > > http://www2.hursley.ibm.com/decimal/ > > for about, oh, fifteen years now . Wow, so Aahz anticipated the creation of Python and started early. This IS a dedicated community. > > ... > > > > >>> 3.03e5d > > > > 303000.00d > > > > The representation is incorrect. It should have returned 3.03e5d > > but it padded the result with bogus zeros. > > It seems you don't really want a fixed-point type. The FixedPoint > constructor avoided this rat hole by making the user be explicit about the > fixed number of fractional digits (or accept the default 2). My goal is to have a new numerical type added to Python that satisfies the requirements for accountants and newbies. Your description of fixed point seems to meet this goal. My confusion was raised by the 'e' notation seeming to add precision to numbers (e.g. 1.0e5 expands to 100000.0.) This would be incorrect for reporting significant digits in scientific papers, but the rules for accounting math are different. This was simply my confusion. FixedPoint appears to be sufficient for bankers and accountants. ---snip Tim's explanation of roundoff error, thanks for the clarification--- > That's really representation error. Until the distinction becomes clear, > you'll keep believing that decimal arithmetic is somehow better . I get the distinction now, thanks. ---snip discussion of rationales and division errors--- > [Brett Cannon] > > > ... > > It is still going to take someone to champion the module to get BDFL > > pronouncment from Guido autonomously or get enough people to rally > > behind to put pressure on Guido to let it in. > > Guido already agreed to fold FixedPoint into 2.3, provided someone other > than me do the work . Doug Fort has done a great job on it > (including LaTeX docs, Fred!), and we're just missing someone with the > time/desire to do final integration. > > That's independent of Michael's patch, of course. The FixedPoint module is > just another library module. -- snip Tim's comments on when his stuff gets added to Python -- > If Python were to grow a native decimal type, I'd rather move to > Cowlishaw's design, because it has the best shot at becoming a standard > influential enough to sway HW design (decimal fp in HW wouldn't be notably > more expensive or slower than binary fp in HW, given current chip > capacities; this was a much bigger deal when chip real estate was scarce). This is a very good long term goal. So for Python3k the standard numerical type will be Cowlishaw's decimal numbers. To bad the decNumber library from IBM is not open source. In the short term what do you advise be done with the patch to add a fixed point literal and a builtin fixed point conversion function? I could change the name from decimal() to FixedPoint(), but the capital letters in the name scream "ugly". This isn't consistent with the short names used by the other numerical conversion functions (int, float, hex, oct...). Would fixed() be an acceptable name? This also suggests the suffix should be changed to 'f' and 'F' instead of 'd' and 'D'. > >> The syntax change is primarily to make Python more appealing to > >> professions that require the semantics of fixed point decimal numbers. > > > > I understand that, but who knows how many of those types of users are > > out there (current and potentiol). > > "Do applications actually use decimal data?" from Cowlishaw's FAQ should go > a way toward answering that: > > http://www2.hursley.ibm.com/decimal/decifaq.html According to this reference the most frequent numerical column type used in commercial databases is decimal (55%). The float type is used just 1.4% of the time with the rest being split between Integer and SmallInt. The data was taken from an analysis of applications used by Airline systems, Banking, Financial Analysis, Insurance, Inventory control, Management reporting, Marketing services, Order entry, Order processing, Pharmaceutical applications, and Retail sales. Is it safe to conclude that Python will be be much more appealing to a huge segment of the IT market if grows a native decimal number type? From barry@python.org Mon Dec 16 16:47:39 2002 From: barry@python.org (Barry A. Warsaw) Date: Mon, 16 Dec 2002 11:47:39 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> <200212161642.gBGGgYJ03474@odiug.zope.com> Message-ID: <15870.939.16954.763349@gargle.gargle.HOWL> GvR> (BTW I now think extend_path() is a better and simpler name GvR> for this.) Or even extendpath() > -Barry From guido@python.org Mon Dec 16 16:54:27 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 11:54:27 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> Message-ID: <200212161654.gBGGsRe03537@odiug.zope.com> > At 10:47 AM 12/16/02 -0500, Guido van Rossum wrote: > > >Time's running short (we're close to releasing Python 2.3 alpha 1); > >we should decide on a name and a specific policy ASAP. I propose the > >following: > > > > from pkgutil import extended_path > > __path__ = extended_path(__path__, __name__) > > How would that work for say, 'zope.app'? Good point. Back to the drawing board; I'll have to turn dots in the name into os.sep. > >Where pkgutil.py could be something like this: > > > > import sys.path > > > > def extended_path(path, name): > > """Extend a package's path. (XXX more.)""" > > path = path[:] > > for dir in sys.path: > > if isinstance(dir, (str, unicode)): > > dir = os.path.join(dir, name) > > if dir not in path and os.path.isdir(dir): > > path.append(dir) > > return path > > The 'isdir()' part isn't going to work on zipfiles... unless os.path is > going to have zipfile support in 2.3. :) I tried putting Zope3 in a zipfile, and it's hopeless; lots and lots of code expects to find data files (at least page templates and zcml files; probably also style sheets and other stuff that I don't even know about) by taking a module's __file__, stripping the filename to get a directory, and concatenating the directory with the name of a known data file. Plus there's the issue of extension modules. So I think the isdir() not working for zip files can be considered a feature. :-) BTW, here's a more recent draft, incorporating a search for .pth files and with a different place for the isdir(): ## pkgutil.py ################################################################# """Utilities to support packages.""" import os import sys def extend_path(path, name): """Extend a package's path. Intended use is to place the following code in a package's __init__.py: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) This will add to the package's __path__ all subdirectories of directories on sys.path named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories. """ path = path[:] for dir in sys.path: if isinstance(dir, (str, unicode)) and os.path.isdir(dir): subdir = os.path.join(dir, name) if subdir not in path and os.path.isdir(subdir): path.append(subdir) pthfile = os.path.join(dir, name + os.extsep + "pth") if os.path.isfile(pthfile): try: f = open(pthfile) except IOError, msg: sys.stderr.write("Can't open %s: %s\n" % (pthfile, msg)) else: for line in f: line = line.strip() if not line or line.startswith('#'): continue path.append(line) # Don't check for existence! f.close() return path ############################################################################### --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Mon Dec 16 16:57:36 2002 From: mwh@python.net (Michael Hudson) Date: 16 Dec 2002 16:57:36 +0000 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Michael McLay's message of "Mon, 16 Dec 2002 11:45:34 -0500" References: <200212161145.21295.mmclay@comcast.net> Message-ID: <2m8yypdhq7.fsf@starship.python.net> Michael McLay writes: > On Monday 16 December 2002 01:48 am, Tim Peters wrote: > > [Michael McLay] > > > > > ... > > > The semantics of the precision parameter may be incorrect because > > > it is only addressing the precision of the fraction part of the > > > number. > > > > That's why the module was called FixedPoint. > > Fixing the decimal place would make sense for applications having to do with > money. That is quite different than how numbers are treated in scientific > calculations. That is were I borrowed the idea of significant digits. > Accountants tend to think that all digits are significant, down to the last > penny. I'm a little confused; you're borrowing terms from scientific papers to explain things to accountants? [...] > This is a very good long term goal. So for Python3k the standard numerical > type will be Cowlishaw's decimal numbers. *I* would hope not; there are reasons to prefer binary fp to decimal fp if you're not a money man. Cheers, M. -- ... with these conditions cam the realisation that ... nothing turned a perfectly normal healthy individual into a great political or military leader better than irreversible brain damage. -- The Hitch-Hikers Guide to the Galaxy, Episode 11 From guido@python.org Mon Dec 16 17:00:02 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 12:00:02 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 11:47:39 EST." <15870.939.16954.763349@gargle.gargle.HOWL> References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> <200212161642.gBGGgYJ03474@odiug.zope.com> <15870.939.16954.763349@gargle.gargle.HOWL> Message-ID: <200212161700.gBGH02S03607@odiug.zope.com> > GvR> (BTW I now think extend_path() is a better and simpler name > GvR> for this.) > > Or even extendpath() > Jim doesn't like sequences of lowercase words strung together without using either camelcase or underscores. In this particular case I agree. --Guido van Rossum (home page: http://www.python.org/~guido/) From pje@telecommunity.com Mon Dec 16 17:02:28 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Mon, 16 Dec 2002 12:02:28 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <200212161654.gBGGsRe03537@odiug.zope.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> Message-ID: <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> At 11:54 AM 12/16/02 -0500, Guido van Rossum wrote: >I tried putting Zope3 in a zipfile, and it's hopeless; lots and lots >of code expects to find data files (at least page templates and zcml >files; probably also style sheets and other stuff that I don't even >know about) by taking a module's __file__, stripping the filename to >get a directory, and concatenating the directory with the name of a >known data file. Plus there's the issue of extension modules. > >So I think the isdir() not working for zip files can be considered a >feature. :-) Alternately, now that there's a "pkgutils" to put it in, why not add a "fileNearModule()" function that returns a file handle opened for reading on the basis of a module name and a file name? I currently use something roughly like: def fileNearModule(moduleName,filename): basedir = os.path.dirname(sys.modules[moduleName].__file__) return open(os.path.join(basedir,filename), 'r') Of course, a zipfile version would need some work, and clearly the zipfile version wouldn't be a candidate for alpha 1. Anyway, if we could establish an API for some of these sorts of things in pkgutils, then they could be the official correct way to get at data files and make zipfiles work for a lot more things long term. From Paul.Moore@atosorigin.com Mon Dec 16 17:05:28 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 16 Dec 2002 17:05:28 -0000 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) Message-ID: <16E1010E4581B049ABC51D4975CEDB88619988@UKDCX001.uk.int.atosorigin.com> From: Guido van Rossum [mailto:guido@python.org] > I find string subclasses an ugly hack that should not be used. Agreed. > The meta hook (or whatever it's called now) should do this. Why bother? > > - strings which don't have a path structure wil break > > We can fix that by skipping anything for which > os.path.isdir(dir) [for the original value of dir] isn't true. Agreed. This makes extend_path "do the right thing" for anything that can currently be on a path, and punt to an extension hook for anything else. That sounds reasonable (but see below). > > (and of course, objects other than string subclasses don't > > get "extended"). > > We can use some kind of extension hook to deal with those. It depends where you envisage that hook. I'd be unhappy with requiring something in the import hook implementation - two reasons, first because the current hook mechanism is nice and simple, and hence easy to document and use, and second because people simply won't implement the path-extension hook, and nearly everything will work fine - the breakage only occurs when the hook is used in conjunction with pkgutil. At that point, saying "I told you so" isn't much help... I've seen this sort of thing happen too often before to be comfortable with it. A simple protocol, with an obscure little requirement which is to support a relatively rare usage. People *never* implement that correctly, and the result is annoying breakages. > The important point is that the idiom placed in __init__.py > can remain the same (the two lines quoted above) while the > implementation of pkgutil.extend_path() can evolve or be > modified (even monkey-patched if you're desperate) as the > import hooks evolve. I'd say the extension mechanism should be external to both the import hook and pkgutil. It then becomes the user's responsibility, when they combine a particular hook with the use of pkgutil, to write the glue. pkgutil.register_hook(whatever) Hook authors can, of course, implement a convenience method, to be used as pkgutil.register_hook(ihook.pkgutil_hook) but if they fail to, at least the user isn't left up the creek without the proverbial paddle :-) (Hope that made sense - I was hand-waving a bit too much...) > > I don't think that either of these is even remotely a > > showstopper, but they do need to be documented - specifically > > because as a module, this code can be imported by anybody, > > and so the constraints on the user should be made clear. > > Of course. Though with the current state of documentation, > I give less priority to documenting every little detail of > this code than to getting it in the code base (with *some* > documentation, for sure). At the moment, I'm playing with the hook mechanism, with the intent of being able to provide some of the documentation. At the very least, I'd like to be able to do a tutorial-style "how to write a simple import hook" document. But yes, a lot of this area is only minimally documented right now, so we don't need to aim for complete perfection just yet :-) > OTOH I expect that most things you'd want to place on sys.path > *do* have a natural string representation. After all, if > something doesn't have a name, it's hard to talk about. And if > it has a name, it can be placed in PYTHONPATH (with the minor > quibble that URLs use ':' which -- at least on Unix -- is also > used as the separator in *PATH variables. :-( ) Hmm. Arguable. I take your point, but I'm looking at it from a different point of view. A "place to store modules" could be something as complex as a database (TCL Starkits use MetaKit). I *could* define a string representation encoding the table name, columns to use, etc etc, but why bother? Just use a database importer object directly. You can't put it on PYTHONPATH, but you can easily use something like this in a packaged application context. Let's wait and see what real use brings on this one. > Maybe importer objects should have a standard API that lets you > ask for an extended object. This way lies a VFS implementation. I'd say we should write a general import hook based on a VFS implementation, rather than the other way around. And let's leave a full VFS alone for now :-) Paul. From barry@zope.com Mon Dec 16 17:08:56 2002 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 16 Dec 2002 12:08:56 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <200212161654.gBGGsRe03537@odiug.zope.com> Message-ID: <15870.2216.65988.322683@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> BTW, here's a more recent draft, incorporating a search for GvR> .pth files and with a different place for the isdir(): Minor nit: .pth semantics should be identical to site.py .pth semantics. I.e. lines that start with "import" should be exec'd. -Barry From guido@python.org Mon Dec 16 17:18:35 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 12:18:35 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 12:02:28 EST." <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> Message-ID: <200212161718.gBGHIZL03752@odiug.zope.com> > Alternately, now that there's a "pkgutils" to put it in, why not add a > "fileNearModule()" function that returns a file handle opened for reading > on the basis of a module name and a file name? I currently use something > roughly like: > > def fileNearModule(moduleName,filename): > basedir = os.path.dirname(sys.modules[moduleName].__file__) > return open(os.path.join(basedir,filename), 'r') > > Of course, a zipfile version would need some work, and clearly the zipfile > version wouldn't be a candidate for alpha 1. > > Anyway, if we could establish an API for some of these sorts of things in > pkgutils, then they could be the official correct way to get at data files > and make zipfiles work for a lot more things long term. I think Zope3 ought to defines its own API for this, then we can decide later on how to do this. I've seen several places where what you propose isn't enough: when I tried putting everything in a zipfile, I ended up giving up when I realized that there are places that use os.stat() on filenames thus constructed. Another issue is that, at least for zipfiles, and probably in most realistic cases, the underlying implementation makes it easy to give you the entire blob of data, and trying to pretend it's a file just require wrapping it in a StringIO instance -- maybe it makes more sense to change your fileNearModule() API to return the blob of data too. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Mon Dec 16 17:36:42 2002 From: aahz@pythoncraft.com (Aahz) Date: Mon, 16 Dec 2002 12:36:42 -0500 Subject: [Python-Dev] Deprecate doctest? In-Reply-To: <200212161624.gBGGO2j01562@odiug.zope.com> References: <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> <200212161624.gBGGO2j01562@odiug.zope.com> Message-ID: <20021216173642.GA10468@panix.com> On Mon, Dec 16, 2002, Guido van Rossum wrote: >Aahz: >> >> That's tricky, given that doctest is in the standard library, and that's >> exactly what doctest does. Are you advocating deprecating doctest? > > You gotta be kidding. Of *course* I was, but I couldn't figure out what else MAL meant. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From guido@python.org Mon Dec 16 17:47:34 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 12:47:34 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 12:08:56 EST." <15870.2216.65988.322683@gargle.gargle.HOWL> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <200212161654.gBGGsRe03537@odiug.zope.com> <15870.2216.65988.322683@gargle.gargle.HOWL> Message-ID: <200212161747.gBGHlYK03859@odiug.zope.com> > Minor nit: .pth semantics should be identical to site.py .pth > semantics. I.e. lines that start with "import" should be exec'd. Barry & I had an offline discussion about this. Some conclusions: - pth files for sys.path and pth-ish files for packages should probably have different extensions; otherwise e.g. paths named in zope.pth would be added both to sys.path and to zope.__path__, which is a bad idea. I propose using pkg as the extension for package.s - Nobody remembers why there is a special case in site.py for lines in the .pth file that start with 'import', except that Martin von Loewis once wanted this. I propose not to duplicate this functionality in .pkg files. - We need to look both for subdirectories and for .pkg files -- different styles of administering a system call for both approaches. - We should only accept subdirectories that have an __init__.py. - I'm punting for now on an extension to cope with finding things in other importers. Current draft, incorporating all these changes: ###################################################################### """Utilities to support packages.""" import os import sys def extend_path(path, name): """Extend a package's path. Intended use is to place the following code in a package's __init__.py: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) This will add to the package's __path__ all subdirectories of directories on sys.path named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories. It also looks for *.pkg files beginning with __name__. This feature is similar to *.pth files, except it doesn't special-case lines starting with 'import'. """ pname = os.path.join(*name.split('.')) # Reconstitute as relative path sname = os.extsep.join(name.split('.')) # Just in case os.extsep != '.' sname_pkg = sname + os.extsep + "pkg" init_py = "__init__" + os.extsep + "py" path = path[:] for dir in sys.path: if isinstance(dir, (str, unicode)) and os.path.isdir(dir): subdir = os.path.join(dir, pname) # XXX This may still add duplicate entries to path on # case-insensitive filesystems initfile = os.path.join(subdir, init_py) if subdir not in path and os.path.isfile(initfile): path.append(subdir) # XXX Is this the right thing for subpackages like # zope.app? It looks for a file named "zope.app.pkg" pkgfile = os.path.join(dir, sname_pkg) if os.path.isfile(pkgfile): try: f = open(pkgfile) except IOError, msg: sys.stderr.write("Can't open %s: %s\n" % (pkgfile, msg)) else: for line in f: line = line.rstrip('\n') if not line or line.startswith('#'): continue path.append(line) # Don't check for existence! f.close() return path ###################################################################### --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@zope.com Mon Dec 16 17:59:31 2002 From: jim@zope.com (Jim Fulton) Date: Mon, 16 Dec 2002 12:59:31 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> Message-ID: <3DFE1483.6010703@zope.com> Phillip J. Eby wrote: > (Summary: good plan for name and place, issues exist w/sketched > implementation) > > At 10:47 AM 12/16/02 -0500, Guido van Rossum wrote: > >> Time's running short (we're close to releasing Python 2.3 alpha 1); >> we should decide on a name and a specific policy ASAP. I propose the >> following: >> >> from pkgutil import extended_path >> __path__ = extended_path(__path__, __name__) > > > How would that work for say, 'zope.app'? Since zope.app isn't a namespace package, it doesn't need to. But I suppose that the question stands for zodb.storages. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim@zope.com Mon Dec 16 18:04:00 2002 From: jim@zope.com (Jim Fulton) Date: Mon, 16 Dec 2002 13:04:00 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com> Message-ID: <3DFE1590.50908@zope.com> Guido van Rossum wrote: >>Alternately, now that there's a "pkgutils" to put it in, why not add a >>"fileNearModule()" function that returns a file handle opened for reading >>on the basis of a module name and a file name? I currently use something >>roughly like: >> >>def fileNearModule(moduleName,filename): >> basedir = os.path.dirname(sys.modules[moduleName].__file__) >> return open(os.path.join(basedir,filename), 'r') >> >>Of course, a zipfile version would need some work, and clearly the zipfile >>version wouldn't be a candidate for alpha 1. >> >>Anyway, if we could establish an API for some of these sorts of things in >>pkgutils, then they could be the official correct way to get at data files >>and make zipfiles work for a lot more things long term. >> > > I think Zope3 ought to defines its own API for this, then we can > decide later on how to do this. > > I've seen several places where what you propose isn't enough: when I > tried putting everything in a zipfile, I ended up giving up when I > realized that there are places that use os.stat() on filenames thus > constructed. This is to ease development. In development mode, we check if the data on disk has changed and reread it if it has. This wouldn't be applicable to zip files. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From fredrik@pythonware.com Mon Dec 16 18:06:32 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 16 Dec 2002 19:06:32 +0100 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> <200212161642.gBGGgYJ03474@odiug.zope.com> <15870.939.16954.763349@gargle.gargle.HOWL> <200212161700.gBGH02S03607@odiug.zope.com> Message-ID: <00d601c2a52d$e2c9c2e0$ced241d5@hagrid> Guido van Rossum wrote: > > Or even extendpath() > > > Jim doesn't like sequences of lowercase words strung together without > using either camelcase or underscores. isn't Jim using Python? >>> dir("") ['capitalize', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'replace', 'rfind', 'rindex', 'rjust', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper'] >>> dir(os.path) [... 'abspath', 'basename', 'commonprefix', 'dirname', 'exists', 'expanduser', 'expandvars', 'getatime ', 'getmtime', 'getsize', 'isabs', 'isdir', 'isfile', 'islink', 'ismount', 'join', 'normcase', 'normpath', 'os', 'split', 'splitdrive', 'splitext', 'splitunc', 'stat', 'walk'] >>> dir({}) [..., 'clear', 'copy', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values'] >>> dir(sys) [..., '_getframe', 'api_version', 'argv', 'builtin_module_names', 'byteorder', 'copyright', 'displayhook', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix', 'executable', 'exit', 'getdefaultencoding', 'getdlopenflags', 'getrecursionlimit', 'getrefcount', 'hexversion', 'last_traceback', 'last_type', 'last_value', 'maxint', 'maxunicode', 'modules', 'path', 'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', 'version', 'version_info', 'warnoptions'] if I didn't know better, I'd surely think that sequences of lowercase words strung together without underscores is the standard way to name functions and methods in Python... From guido@python.org Mon Dec 16 18:08:38 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 13:08:38 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 13:04:00 EST." <3DFE1590.50908@zope.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com> <3DFE1590.50908@zope.com> Message-ID: <200212161808.gBGI8cP04069@odiug.zope.com> > > I've seen several places where what you propose isn't enough: when I > > tried putting everything in a zipfile, I ended up giving up when I > > realized that there are places that use os.stat() on filenames thus > > constructed. > > This is to ease development. In development mode, we check if the > data on disk has changed and reread it if it has. This wouldn't be > applicable to zip files. How do we turn this off? I think it would be a good exercise to see if we could load Zope3 from a zip file. (Though I've got to say that the ~3 seconds startup time on my 2 year old Linux box is pretty speedy, so I'm not sure we could or need to make it much faster than that.) --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@zope.com Mon Dec 16 18:24:56 2002 From: jim@zope.com (Jim Fulton) Date: Mon, 16 Dec 2002 13:24:56 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com> <3DFE1590.50908@zope.com> <200212161808.gBGI8cP04069@odiug.zope.com> Message-ID: <3DFE1A78.3080004@zope.com> Guido van Rossum wrote: >>>I've seen several places where what you propose isn't enough: when I >>>tried putting everything in a zipfile, I ended up giving up when I >>>realized that there are places that use os.stat() on filenames thus >>>constructed. >>> >>This is to ease development. In development mode, we check if the >>data on disk has changed and reread it if it has. This wouldn't be >>applicable to zip files. >> > > How do we turn this off? In the zconfig file. ;) There isn't a way to turn it off yet. > I think it would be a good exercise to see > if we could load Zope3 from a zip file. (Though I've got to say that > the ~3 seconds startup time on my 2 year old Linux box is pretty > speedy, so I'm not sure we could or need to make it much faster than > that.) I presume zip files would make mac starup a lot faster, but that wouldn't work for development. It would also help startup when reading from CD, or so the theory goes. Having said all of that, I have a hard time getting excited about zip file import myself. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From guido@python.org Mon Dec 16 18:31:01 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 13:31:01 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 13:24:56 EST." <3DFE1A78.3080004@zope.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com> <3DFE1590.50908@zope.com> <200212161808.gBGI8cP04069@odiug.zope.com> <3DFE1A78.3080004@zope.com> Message-ID: <200212161831.gBGIV1i04265@odiug.zope.com> > I presume zip files would make mac starup a lot faster, but that > wouldn't work for development. Except if you can update the zipfile is under 25 seconds. ;-) > It would also help startup when reading from CD, or so the theory > goes. Having said all of that, I have a hard time getting excited > about zip file import myself. Agreed. From =?iso-8859-1?Q?Martin_v._L=F6wis?= Mon Dec 16 18:33:32 2002 From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=) Date: Mon, 16 Dec 2002 19:33:32 +0100 Subject: [Python-Dev] PEP-298: buffer interface widening too much? References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira> <3DFCFD21.2090707@stsci.edu> <3DFDE30C.7080009@stsci.edu> Message-ID: <017d01c2a531$a34083d0$9319e8d9@mira> > >We could even provide a Generic function which > >delegates the new get functions to the existing ones, accessing > >segment zero. > > > > > Wouldn't this require more information about the lock implementation? The generic function would just invoke the unlocked routine, and thus it would only be applicable for types that don't require locking (as the underlying memory can't move, anyway). If there must be a true lock count in the implementation, you couldn't use the generic wrapper. I don't know how useful that would be, as I don't know whether the typical buffer-supporting object has movable memory or fixed memory; I'd expect that immutable object typically have fixed memory, and mutable objects typically have relocatable memory. So only immutable types could use the wrapper. > As PEP-298 stands, I thought that was left up to the buffered type. > It might make sense to change the locking getpointer fields into lock, > and release into unlock. Then compose the primitive operations in the > wrapper as you suggested. That wouldn't fix the size_t problem though. Ah, right - this would be another restriction for using the generic wrapper. Perhaps then there is really no point in having it. > >I now recall what the issue is: you can't assume that modifying the > >argument tuple object to record resources is feasible, as this breaks > >apply (where in every case the user allocates a tuple, and where this > >is a plain old tuple). > > > > > Is the efficiency of apply() so critical that a conversion is ruled out? I would think so, yes: allocating objects is one of the most costly operation in Python. It is more efficient now that we have pymalloc, but still. There is also the in-C calls to functions that use Py_BuildValue, then PyObject_Call; returning an enhanced tuple from BuildValue might break existing code, yet copying the tuples inside PyObject_Call (if needed) would add a price there, too. So I'm now leaning towards not requiring a new type. > Way over my head here... I guess the point is that arg tuples aren't > the only point of attack. No, they really are the issue. However, I cannot guarantee that ParseTuple is always called with enhanced tuples, so the correctness this is meant to provide isn't really provided. Regards, Martin From just@letterror.com Mon Dec 16 19:14:36 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 16 Dec 2002 20:14:36 +0100 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <200212161718.gBGHIZL03752@odiug.zope.com> Message-ID: FWIW, I think it's generally a bad idea to depend on module.__file__ and pkg.__path__ to find data files. Playing tricks with the contents of pkg.__path__ apparently has its uses, but I think that (in general) it's a bad idea as well. module.__file__ is mostly an introspective aid, and pkg.__path__ should IMO be seen as merely an implementation detail. module.__file__: in a frozen module it will be set to "". Expect __file__ to be a path to a file and you're screwed. To the lower levels of the import mechanism, the *existence* of pkg.__path__ is all that's looked at: "hey, it's a package". Then there's freeze again: a frozen package has a __path__ variable, but it's not a list, it's a *string*. Only when an import goes through a sys.path item, __path__ is (more or less) guaranteed to be a list. The sys.meta_path import hook mechanism in my patch (the idea is stolen from Gordon McMillan) acts on the same level as builtin module imports and frozen module imports: it doesn't need sys.path. So it doesn't need any meaningful object as pkg.__path__ either. I just uploaded a new version of the patch; it now contains a test_importhooks.py script, which has a sys.meta_path test case which actually sets pkg.__path__ to None. Works like a charm. Here's a (slightly modified) comment from the test script: Depending on the kind of importer, there are different levels of freedom of what you can use as pkg.__path__. Importer object on sys.meta_path: it can use anything it pleases (even None), as long as a __path__ variable is set. Importer object on sys.path: pkg.__path__ must be a list; it's most logical to use an importer object as the only item. Could be the same importer instance that imported the package itself. A hook on sys.path_hooks: pkg.__path__ must be a list and its only item should be a string that the hook can handle itself. These are just guidelines: a set of hooks could in theory deliberately set pgk.__path__ up so submodule imports be handled by an entirely different importer. Not sure how useful that would be... Just-my-2-eurocents. From niemeyer@conectiva.com Mon Dec 16 19:19:36 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Mon, 16 Dec 2002 17:19:36 -0200 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <200212161631.gBGGVFB01638@odiug.zope.com> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> Message-ID: <20021216171936.A10545@ibook.distro.conectiva> > Hm... Do you see any way to break out of restricted execution mode > using this? I suppose a fix would be simple enough, but I'm more and > more inclined to simply rip out rexec from the distribution -- it's > never going to be safe, and I doubt it's very useful as long as it's > not safe. Do you mean the rexec.py module, or all the restricted features? I would like to see the whole scheme working someday. I'm not sure how safe it will ever be, but the problems I've seen so far are due to some language change introduced. I belive that once the language features are mature, these problems will be reduced. As a side note, I'll have a look at that bug, if nobody has done so yet. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From theller@python.net Mon Dec 16 19:22:01 2002 From: theller@python.net (Thomas Heller) Date: 16 Dec 2002 20:22:01 +0100 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: References: Message-ID: Just van Rossum writes: > FWIW, I think it's generally a bad idea to depend on module.__file__ and > pkg.__path__ to find data files. What would you suggest to do instead? There's now even such a case in the standard library: distutils.command.bdist_wininst has to find it's data-file 'wininst.exe'. Thomas From guido@python.org Mon Dec 16 19:22:27 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 14:22:27 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 17:05:28 GMT." <16E1010E4581B049ABC51D4975CEDB88619988@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619988@UKDCX001.uk.int.atosorigin.com> Message-ID: <200212161922.gBGJMRe05832@odiug.zope.com> > > > (and of course, objects other than string subclasses don't > > > get "extended"). > > > > We can use some kind of extension hook to deal with those. > > It depends where you envisage that hook. I'd be unhappy with > requiring something in the import hook implementation - two > reasons, first because the current hook mechanism is nice and > simple, and hence easy to document and use, and second because > people simply won't implement the path-extension hook, and nearly > everything will work fine - the breakage only occurs when the > hook is used in conjunction with pkgutil. At that point, saying > "I told you so" isn't much help... > > I've seen this sort of thing happen too often before to be > comfortable with it. A simple protocol, with an obscure little > requirement which is to support a relatively rare usage. People > *never* implement that correctly, and the result is annoying > breakages. Only annoying if you need the rare usage. It turns out that running from a zipfile isn't a priority for Zope3 anyway, so I'll call this a YAGNI. > > The important point is that the idiom placed in __init__.py > > can remain the same (the two lines quoted above) while the > > implementation of pkgutil.extend_path() can evolve or be > > modified (even monkey-patched if you're desperate) as the > > import hooks evolve. > > I'd say the extension mechanism should be external to both > the import hook and pkgutil. It then becomes the user's > responsibility, when they combine a particular hook with the > use of pkgutil, to write the glue. > > pkgutil.register_hook(whatever) > > Hook authors can, of course, implement a convenience method, > to be used as > > pkgutil.register_hook(ihook.pkgutil_hook) > > but if they fail to, at least the user isn't left up the creek > without the proverbial paddle :-) > > (Hope that made sense - I was hand-waving a bit too much...) Yes, except you failed to specify what pkgutil.register_hook(whatever) is supposed to do, and you failed to specify the signature for 'whatever'... IOW, please think more about the implementation and provide a patch to the latest pkgutil.py (I posted it here earlier). > > > I don't think that either of these is even remotely a > > > showstopper, but they do need to be documented - specifically > > > because as a module, this code can be imported by anybody, > > > and so the constraints on the user should be made clear. > > > > Of course. Though with the current state of documentation, > > I give less priority to documenting every little detail of > > this code than to getting it in the code base (with *some* > > documentation, for sure). > > At the moment, I'm playing with the hook mechanism, with the > intent of being able to provide some of the documentation. At > the very least, I'd like to be able to do a tutorial-style "how > to write a simple import hook" document. > > But yes, a lot of this area is only minimally documented right > now, so we don't need to aim for complete perfection just yet :-) You're talking about Just's import hooks, right? > > OTOH I expect that most things you'd want to place on sys.path > > *do* have a natural string representation. After all, if > > something doesn't have a name, it's hard to talk about. And if > > it has a name, it can be placed in PYTHONPATH (with the minor > > quibble that URLs use ':' which -- at least on Unix -- is also > > used as the separator in *PATH variables. :-( ) > > Hmm. Arguable. I take your point, but I'm looking at it from a > different point of view. A "place to store modules" could be > something as complex as a database (TCL Starkits use MetaKit). > I *could* define a string representation encoding the table name, > columns to use, etc etc, but why bother? Just use a database > importer object directly. You can't put it on PYTHONPATH, but > you can easily use something like this in a packaged application > context. A packaged application knows whether it needs to support pkgutil or not. So it can take whatever measures necessary (including overriding pkgutil.py) to make it work. > Let's wait and see what real use brings on this one. Yes please. > > Maybe importer objects should have a standard API that lets you > > ask for an extended object. > > This way lies a VFS implementation. I'm not sure what you mean by that. > I'd say we should write a general import hook based on a VFS > implementation, rather than the other way around. And let's leave a > full VFS alone for now :-) The old ihooks was trying to do something like a vfs (it has a bunch of APIs that provide hooks into all filesystem operations). But it wasn't a big success... --Guido van Rossum (home page: http://www.python.org/~guido/) From shane@zope.com Mon Dec 16 19:37:42 2002 From: shane@zope.com (Shane Hathaway) Date: Mon, 16 Dec 2002 14:37:42 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: Message-ID: <3DFE2B86.3060508@zope.com> Thomas Heller wrote: > Just van Rossum writes: > > >>FWIW, I think it's generally a bad idea to depend on module.__file__ and >>pkg.__path__ to find data files. > > What would you suggest to do instead? There's now even such a case > in the standard library: distutils.command.bdist_wininst has > to find it's data-file 'wininst.exe'. I'd like to point out Java's solution to this. http://java.sun.com/j2se/1.4.1/docs/api/java/lang/ClassLoader.html#getResource(java.lang.String) Since Python doesn't have the complication of multiple class loaders, getResource() (or whatever name it gets in Python) and its peers could be module-level functions somewhere (such as pkgutils). Shane From guido@python.org Mon Dec 16 19:41:55 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 14:41:55 -0500 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: Your message of "Mon, 16 Dec 2002 20:14:36 +0100." References: Message-ID: <200212161941.gBGJftc05924@odiug.zope.com> [Please try to limit the width of your lines to 72 characters or so.] > FWIW, I think it's generally a bad idea to depend on module.__file__ > and pkg.__path__ to find data files. Of course you think this, because you just recently invented something that would break this assumption. :-) The problem is that Zope3 needs *some* way of storing data and config files related to specific packages, and there isn't really any other convenient and conventional way except storing it in a package directory and using the package's __file__ to find it. Python's own test package uses this, and so does the pdb module for its help file (the latter actually scans sys.path for a file pdb.doc). > Playing tricks with the contents of pkg.__path__ apparently has its > uses, It's the *defined* API for adding to the set of locations from which submodules and subpackages of a package are to be loaded. You can't just call that "playing tricks." > but I think that (in general) it's a bad idea as > well. module.__file__ is mostly an introspective aid, and > pkg.__path__ should IMO be seen as merely an implementation detail. Not true, see above -- it's a defined API for a specific purpose. That's less so for __file__, but absent a well-defined alternative API, __file__ should be supported. I'm not saying that __file__ should always point to a file -- obviously it can't, if we want import from zip files. But there's considerable existing code that uses it in certain ways and we shouldn't break that unless necessary. > module.__file__: in a frozen module it will be set to > "". Expect __file__ to be a path to a file and you're > screwed. Of course. If you freeze code, you have to make sure that it will be able to find its data files in some other way. But not all code has to be freezable. > To the lower levels of the import mechanism, the *existence* of > pkg.__path__ is all that's looked at: "hey, it's a package". Then > there's freeze again: a frozen package has a __path__ variable, but > it's not a list, it's a *string*. Only when an import goes through a > sys.path item, __path__ is (more or less) guaranteed to be a list. Yeah, frozen code can't work with pkgutil as it currently stands. But I think pkgutil is pretty meaningless for frozen code: pkgutil is a mechanism to allow extensions of a package to be installed in different directories. A frozen application should gather all code belonging to a package in a single directory, and freeze that; then all we need to ensure is that calling pkgutil.extend_path() doesn't bomb out, and that's easily done by adding if not isinstance(path, list): return path to the start of extend_path(). > The sys.meta_path import hook mechanism in my patch (the idea is > stolen from Gordon McMillan) acts on the same level as builtin > module imports and frozen module imports: it doesn't need > sys.path. So it doesn't need any meaningful object as pkg.__path__ > either. I just uploaded a new version of the patch; it now contains > a test_importhooks.py script, which has a sys.meta_path test case > which actually sets pkg.__path__ to None. Works like a charm. Here's > a (slightly modified) comment from the test script: Does the metahook also apply to submodules (or subpackage) of packages? I'd expect not. (I haven't had the time to review your latest patches yet, in part due to this thread. :-) Surely you shouldn't be looking for builtin submodules of a package. > Depending on the kind of importer, there are different > levels of freedom of what you can use as pkg.__path__. > > Importer object on sys.meta_path: > it can use anything it pleases (even None), as long > as a __path__ variable is set. But I can imagine that some metahooks would like to look inside the __path__ list for more hints on where to find the submodules of the package. Of course that's up to the metahook. > Importer object on sys.path: > pkg.__path__ must be a list; it's most logical to use > an importer object as the only item. Could be the same > importer instance that imported the package itself. Multiple items would make sense too. > A hook on sys.path_hooks: > pkg.__path__ must be a list and its only item should > be a string that the hook can handle itself. IMO it should allow multiple strings too! > These are just guidelines: a set of hooks could in theory > deliberately set pgk.__path__ up so submodule imports be > handled by an entirely different importer. Not sure how > useful that would be... Very useful, for not-yet-imagined cases. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Dec 16 19:47:34 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 14:47:34 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: Your message of "Mon, 16 Dec 2002 17:19:36 -0200." <20021216171936.A10545@ibook.distro.conectiva> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> Message-ID: <200212161947.gBGJlYA05956@odiug.zope.com> > > Hm... Do you see any way to break out of restricted execution mode > > using this? I suppose a fix would be simple enough, but I'm more and > > more inclined to simply rip out rexec from the distribution -- it's > > never going to be safe, and I doubt it's very useful as long as it's > > not safe. > > Do you mean the rexec.py module, or all the restricted features? Both. Unless we spend several orders of more effort on reviewing and testing these "security" features, we're running the serious risk that someone naively believes that they are secure, uses them to protect real data, and their site gets broken into. Then who is responsible? Even if no real data is lost, the more we advertise this as secure, the more egg we have on our face when someone finds a hole. The history of Java's security features shows that even in systems that have had infinitely more scrutiny, security holes still show up -- a language implementation is simply too complex to be bug-free. > I would like to see the whole scheme working someday. I'm not sure > how safe it will ever be, but the problems I've seen so far are due > to some language change introduced. I belive that once the language > features are mature, these problems will be reduced. I wish you well, but I recommend that you start a separate project "secure Python". I don't think that the core will ever slow down its evolution to a pace where security issues can be fixed faster than they are generated by new code. > As a side note, I'll have a look at that bug, if nobody has done so yet. Please do. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Mon Dec 16 19:59:41 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 16 Dec 2002 20:59:41 +0100 Subject: [Python-Dev] known obvious thing or bug (rexec)? References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> Message-ID: <002801c2a53d$ac854b40$6d94fea9@newmexico> From: "Guido van Rossum" > > latest CVS > > > > Python 2.3a0 (#1, Dec 14 2002, 07:19:32) > > [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > > > consider > > > > >>> import rexec > > >>> r=rexec.RExec() > > >>> r.r_exec("class C: pass\nC.a=0") > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/home/users/p/pe/pedronis/python/dist/src/Lib/rexec.py", line 309, in > > r_ > > exec > > exec code in m.__dict__ > > File "", line 2, in ? > > RuntimeError: classes are read-only in restricted mode > > > > vs. > > > > >>> r.r_exec("class C(object): pass\nC.a=0") > > >>> > > Hm... Do you see any way to break out of restricted execution mode > using this? I suppose a fix would be simple enough, but I'm more and > more inclined to simply rip out rexec from the distribution -- it's > never going to be safe, and I doubt it's very useful as long as it's > not safe. basically all the checks (inaccessible __dict__, readonly __class__ etc) done for old-style classes are not done for new-style classes, things get also complicated because of metaclasses (one could enforce things in type but ... ) I agree (but I have said that in past) the best thing is to deprecate/rip out rexec. I have looked a bit at zope.security, it seems bases on a better model, maybe if it's successful it could be incorparated in the std lib in the future ... regards. From guido@python.org Mon Dec 16 20:09:39 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 15:09:39 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: Your message of "Mon, 16 Dec 2002 20:59:41 +0100." <002801c2a53d$ac854b40$6d94fea9@newmexico> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <002801c2a53d$ac854b40$6d94fea9@newmexico> Message-ID: <200212162009.gBGK9dI06095@odiug.zope.com> > I agree (but I have said that in past) the best thing is to > deprecate/rip out rexec. I have looked a bit at zope.security, it > seems bases on a better model, maybe if it's successful it could be > incorparated in the std lib in the future ... But of course to some extent Zope's security depends on some of the features of Python's security (like being able to override __builtins__). Though especially the Zope3 approach, using security proxies implemented in C, seems quite safe: *no* attribute can be accessed without confirmation from the security checker. In this situation, type(x) will returned a security proxy for the type of x; type is replaced with a security-aware version when executing untrusted code. I note that we're finding it convenient to let __str__ and __repr__ bypass the security proxy though. This means that str() of a security-proxy-wrapped container will call str() on the unwrapped items inside the container. Of course, this simply means that __str__ and __repr__ should not reveal sensitive information; not a big burden IMO. Disclaimer: I know shit about security. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From sholden@holdenweb.com Mon Dec 16 20:12:24 2002 From: sholden@holdenweb.com (Steve Holden) Date: Mon, 16 Dec 2002 15:12:24 -0500 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python References: <200212142106.01812.mclay@nist.gov> <200212161627.gBGGRq801621@odiug.zope.com> Message-ID: <010a01c2a53f$75281900$6b01a8c0@holdenweb.com> [Micheal McLay] > > sf/653938I is a patch to integrate the fixedpoint module, that was > > created by Tim Peters, into Python as a first class object. That is, > > the patch adds this new number type so that it has direct syntax > > support like float, int, long, str., etc. within the interpreter. [Guido] > > (a) I don't want any (more) change to the syntax for Python 2.3. > > (b) Like the sets module, proposals for new datatypes should first be > prototyped as standard library modules before being promoted to > built-in status (with their own literals). > Speaking of which, if sets *were* to become built-ins, would the notation { member1, member2, member3, ... } fly as a constructor literal? I don't know enough ab out the parser's operations to determine how easy disambiguation from a dictionary constructor might be, but it's certainly a desirably parallel notation with mathematical sets. all-a-long-way-from-2.3-ly y'rs - steve ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Bring your musical instrument to PyCon! http://www.python.org/pycon/ ----------------------------------------------------------------------- From guido@python.org Mon Dec 16 20:18:49 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 15:18:49 -0500 Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Your message of "Mon, 16 Dec 2002 15:12:24 EST." <010a01c2a53f$75281900$6b01a8c0@holdenweb.com> References: <200212142106.01812.mclay@nist.gov> <200212161627.gBGGRq801621@odiug.zope.com> <010a01c2a53f$75281900$6b01a8c0@holdenweb.com> Message-ID: <200212162018.gBGKIos06623@odiug.zope.com> > Speaking of which, if sets *were* to become built-ins, would the > notation > > { member1, member2, member3, ... } > > fly as a constructor literal? I don't know enough ab out the > parser's operations to determine how easy disambiguation from a > dictionary constructor might be, but it's certainly a desirably > parallel notation with mathematical sets. That's what PEP 218 proposes, yes (it even proposes "set comprehensions"). The parser has no problem with this, except that {} would be ambiguous. The PEP proposes to solve that by using {-} to write an empty set, but not everyone likes that. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Dec 16 20:57:54 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 16 Dec 2002 15:57:54 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <200212162009.gBGK9dI06095@odiug.zope.com> (Guido van Rossum's message of "Mon, 16 Dec 2002 15:09:39 -0500") References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <002801c2a53d$ac854b40$6d94fea9@newmexico> <200212162009.gBGK9dI06095@odiug.zope.com> Message-ID: Guido van Rossum writes: > Disclaimer: I know shit about security. :-( At least you're ahead of some of us. I don't know shit about security. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From niemeyer@conectiva.com Mon Dec 16 21:02:02 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Mon, 16 Dec 2002 19:02:02 -0200 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <200212161947.gBGJlYA05956@odiug.zope.com> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> Message-ID: <20021216190202.A11682@ibook.distro.conectiva> > > Do you mean the rexec.py module, or all the restricted features? > > Both. Unless we spend several orders of more effort on reviewing and > testing these "security" features, we're running the serious risk that > someone naively believes that they are secure, uses them to protect > real data, and their site gets broken into. Then who is responsible? Perhaps the main features could be left in the core, without any real application being offered as part of the "batteries". This way, a secure environment could evolve in parallel, without forking the Python development itself. > Even if no real data is lost, the more we advertise this as secure, > the more egg we have on our face when someone finds a hole. The > history of Java's security features shows that even in systems that > have had infinitely more scrutiny, security holes still show up -- > a language implementation is simply too complex to be bug-free. Agreed. > > I would like to see the whole scheme working someday. I'm not sure > > how safe it will ever be, but the problems I've seen so far are due > > to some language change introduced. I belive that once the language > > features are mature, these problems will be reduced. > > I wish you well, but I recommend that you start a separate project > "secure Python". I don't think that the core will ever slow down its > evolution to a pace where security issues can be fixed faster than > they are generated by new code. I was talking about major changes like type/class unification. I belive that major changes like this won't happen often, and that's the kind of change that affected the restricted execution so far. One idea I was wondering is to provide a "hack me" machine somewhere on the web, including a hall-of-fame for those who break the system somehow. Hummm.. I must find some infrastructure to do that. Anyway, that could help us find some obvious problems quickly, but I understand it shouldn't affect the general decision of how to support restricted execution in Python. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From bac@OCF.Berkeley.EDU Mon Dec 16 21:05:38 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 16 Dec 2002 13:05:38 -0800 (PST) Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <002801c2a53d$ac854b40$6d94fea9@newmexico> <200212162009.gBGK9dI06095@odiug.zope.com> Message-ID: [David Abrahams] > Guido van Rossum writes: > > > Disclaimer: I know shit about security. :-( > > At least you're ahead of some of us. I don't know shit about > security. > Now if I quote you two, should I censor it for the possible little children who will read this naughty language? =) trying-to-not-swear-like-on-the-soparnos-ly y'rs, Brett From guido@python.org Mon Dec 16 21:37:46 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 16:37:46 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: Your message of "Mon, 16 Dec 2002 19:02:02 -0200." <20021216190202.A11682@ibook.distro.conectiva> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> Message-ID: <200212162137.gBGLbkl07474@odiug.zope.com> > Perhaps the main features could be left in the core, without any real > application being offered as part of the "batteries". This way, a secure > environment could evolve in parallel, without forking the Python > development itself. Of course that assumes that the main features in the core are secure. Samuele's observation that restricted code can modify a new-style class passed in belies that. > I was talking about major changes like type/class unification. I belive > that major changes like this won't happen often, and that's the kind of > change that affected the restricted execution so far. As far as you know. Every change is a potential security hole. > One idea I was wondering is to provide a "hack me" machine somewhere on > the web, including a hall-of-fame for those who break the system > somehow. Hummm.. I must find some infrastructure to do that. Anyway, > that could help us find some obvious problems quickly, but I understand > it shouldn't affect the general decision of how to support restricted > execution in Python. Better put it in a serious chroot jail. I think this has been tried and broken into before. --Guido van Rossum (home page: http://www.python.org/~guido/) From niemeyer@conectiva.com Mon Dec 16 22:03:10 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Mon, 16 Dec 2002 20:03:10 -0200 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <200212162137.gBGLbkl07474@odiug.zope.com> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <200212162137.gBGLbkl07474@odiug.zope.com> Message-ID: <20021216200309.B10940@ibook.distro.conectiva> > > Perhaps the main features could be left in the core, without any real > > application being offered as part of the "batteries". This way, a secure > > environment could evolve in parallel, without forking the Python > > development itself. > > Of course that assumes that the main features in the core are secure. > Samuele's observation that restricted code can modify a new-style > class passed in belies that. How that would affect Python itself, if no rexec module was offered? He wouldn't even notice that this problem was there, unless he was using the "python-rexec" external module. Even then, he would have reported this as a bug in "python-rexec", not in Python itself. > > I was talking about major changes like type/class unification. I belive > > that major changes like this won't happen often, and that's the kind of > > change that affected the restricted execution so far. > > As far as you know. Every change is a potential security hole. I meant that the problems I have seen so far were caused by major changes. I agree that it's hard to predict what change is going to break it. > Better put it in a serious chroot jail. Indeed. It'd be good if that machine was completely dedicated to this purpose. > I think this has been tried and broken into before. That's the goal! To get into the hall-of-fame, one would have to tell how he got in. We can also send an email to someone just before executing the code, so that even anonymous hackers contribute with the idea. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From jeremy@alum.mit.edu Mon Dec 16 22:06:29 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Mon, 16 Dec 2002 17:06:29 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <20021216190202.A11682@ibook.distro.conectiva> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> Message-ID: <15870.20069.298805.666542@slothrop.zope.com> >>>>> "GN" == Gustavo Niemeyer writes: GN> One idea I was wondering is to provide a "hack me" machine GN> somewhere on the web, including a hall-of-fame for those who GN> break the system somehow. Hummm.. I must find some GN> infrastructure to do that. Anyway, that could help us find some GN> obvious problems quickly, but I understand it shouldn't affect GN> the general decision of how to support restricted execution in GN> Python. Unless there's some reason to believe this would be hard, what's the point? Jeremy From pobrien@orbtech.com Mon Dec 16 22:09:03 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Mon, 16 Dec 2002 16:09:03 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent Message-ID: <200212161609.03192.pobrien@orbtech.com> --------------Boundary-00=_37G8NHCY5EHYZUUF98V5 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8bit I'd like to call attention to a bug report I just submitted to SF concerning a potentially serious problem with the c-version of pickle. In short, I think I discovered a situation where cPickle is not doing the right thing in terms of storing a reference to a previously pickled class instance (using Pickler, of course). It looks like its doing a put of a new instance, then a get of the original instances dictionary, rather than a get of the original instance. On the other hand, pickle does the right thing. I think this is a pretty serious problem, but I'm out of my league when it comes to debugging the c-version of pickle. I'm also basing my assessment on a minimal understanding of the pickle source code combined with looking at two pickle files, one created using pickle, the other using cPickle, with no other changes to the program that produced them. I understand that cosmetic differences are to be expected. But the difference I see is more substantial. I attached the two pickle files to the SF bug report, but I can't tell if they got through so I'm attaching them here as well. If anyone can help track this down, I'd really appreciate it. If more detail is needed, or a simple code example would help, I'll be happy to do that. Thanks. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- --------------Boundary-00=_37G8NHCY5EHYZUUF98V5 Content-Type: text/plain; charset="us-ascii"; name="cpickle.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="cpickle.txt" (ccopy_reg _reconstructor p1 (cpypersyst.clock Clock p2 c__builtin__ object p3 NtRp4 (dp5 S'_ispaused' p6 I1 sS'_time' p7 F1039910904.277091 sb(i__main__ Bank p8 (dp9 S'_nextAccountNumber' p10 I3 sS'_accounts' p11 (dp12 I1 (iaccount Account (dp13 S'balance' p14 I0 sS'number' p15 I1 sbsI2 (iaccount Account (dp16 g14 I0 sg15 I2 sbssbtp17 .g1 (ccommands DepositOrWithdrawal p18 g3 NtRp19 (dp20 g7 F1039910911.252528 sS'acnt' p21 (iaccount Account p22 g13 bsS'amount' p23 I555 sb.g1 (g18 g3 NtRp24 (dp25 g7 F1039910918.068189 sg21 g22 sg23 I555 sb. --------------Boundary-00=_37G8NHCY5EHYZUUF98V5 Content-Type: text/plain; charset="us-ascii"; name="pickle.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pickle.txt" (ccopy_reg _reconstructor p0 (cpypersyst.clock Clock p1 c__builtin__ object p2 Ntp3 R(dp4 S'_ispaused' p5 I1 sS'_time' p6 F1039911604.515402 sbp7 (i__main__ Bank p8 (dp9 S'_nextAccountNumber' p10 I3 sS'_accounts' p11 (dp12 I1 (iaccount Account p13 (dp14 S'balance' p15 I0 sS'number' p16 I1 sbsI2 (iaccount Account p17 (dp18 g15 I0 sg16 I2 sbssbtp19 .g0 (ccommands DepositOrWithdrawal p20 g2 Ntp21 R(dp22 g6 F1039911613.244611 sS'acnt' p23 g13 sS'amount' p24 I555 sbp25 .g0 (g20 g2 Ntp26 R(dp27 g6 F1039911618.384868 sg23 g13 sg24 I555 sbp28 . --------------Boundary-00=_37G8NHCY5EHYZUUF98V5-- From guido@python.org Mon Dec 16 22:14:49 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 17:14:49 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: Your message of "Mon, 16 Dec 2002 20:03:10 -0200." <20021216200309.B10940@ibook.distro.conectiva> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <200212162137.gBGLbkl07474@odiug.zope.com> <20021216200309.B10940@ibook.distro.conectiva> Message-ID: <200212162214.gBGMEnn10102@odiug.zope.com> > > Of course that assumes that the main features in the core are secure. > > Samuele's observation that restricted code can modify a new-style > > class passed in belies that. > > How that would affect Python itself, if no rexec module was offered? He > wouldn't even notice that this problem was there, unless he was using > the "python-rexec" external module. Even then, he would have reported > this as a bug in "python-rexec", not in Python itself. I was assuming that even if you're not using the rexec module, you're using *some* way to execute untrusted Python code. If that untrusted Python code is given access to a new-style class, the hole is as open as ever. (Fortunately you can't modify built-in types, otherwise the hole would be much bigger and much harder to avoid.) > That's the goal! To get into the hall-of-fame, one would have to tell > how he got in. We can also send an email to someone just before > executing the code, so that even anonymous hackers contribute with the > idea. I don't expect you'll ever reach the point where it'll be wise to advertise this as safe. I certainly won't. --Guido van Rossum (home page: http://www.python.org/~guido/) From niemeyer@conectiva.com Mon Dec 16 22:18:14 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Mon, 16 Dec 2002 20:18:14 -0200 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <15870.20069.298805.666542@slothrop.zope.com> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com> Message-ID: <20021216201814.C10940@ibook.distro.conectiva> > Unless there's some reason to believe this would be hard, what's the > point? The point is to catch the security flaws, and fix them. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From bac@OCF.Berkeley.EDU Mon Dec 16 22:28:02 2002 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 16 Dec 2002 14:28:02 -0800 (PST) Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <200212161631.gBGGVFB01638@odiug.zope.com> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> Message-ID: [Guido van Rossum] > using this? I suppose a fix would be simple enough, but I'm more and > more inclined to simply rip out rexec from the distribution -- it's > never going to be safe, and I doubt it's very useful as long as it's > not safe. > Obviously Guido wants to just ditch rexec with the expectation it will never work well enough for it to be legitimately called "safe". Gustavo, on the other hand, seems rather determined to get the thing to work. Can we just move it out of circulation to the nondist/ directory? It seems Gustavo does not want to move it to a completely separate project (maybe I am misreading his emails, though). But if moving it there is not reasonable, then I am voting for completely removing it. The code will still be in older versions if someone decides to pick it up and work on it as a separate project. And perhaps it could be started from scratch for Python 3 when unification is done and there are less things to have to worry about. -Brett From jeremy@alum.mit.edu (Jeremy Hylton) Mon Dec 16 22:24:29 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) (Jeremy Hylton) Date: Mon, 16 Dec 2002 17:24:29 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <20021216201814.C10940@ibook.distro.conectiva> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com> <20021216201814.C10940@ibook.distro.conectiva> Message-ID: <15870.21149.166400.97302@slothrop.zope.com> >>>>> "GN" == Gustavo Niemeyer writes: >> Unless there's some reason to believe this would be hard, what's >> the point? GN> The point is to catch the security flaws, and fix them. If it isn't hard to hack or if there are a lot of flaws, there isn't much incentive for someone to attack it. Jeremy From greg@cosc.canterbury.ac.nz Mon Dec 16 22:28:26 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Dec 2002 11:28:26 +1300 (NZDT) Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212161145.21295.mmclay@comcast.net> Message-ID: <200212162228.gBGMSQu07742@oma.cosc.canterbury.ac.nz> Michael McLay : > I could change the name from decimal() to FixedPoint(), but the > capital letters in the name scream "ugly". Worse, it would completely lose the connotation that it is a *decimal* number type, which surely is the main point of it in the first place! If it's somehow important that the name contain at least one capital letter, how about Decimal, or perhaps FixedDecimal if it's felt necessary to distinguish it from some potential floating decimal type. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Mon Dec 16 22:34:56 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 16 Dec 2002 17:34:56 -0500 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: Your message of "Mon, 16 Dec 2002 20:18:14 -0200." <20021216201814.C10940@ibook.distro.conectiva> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com> <20021216201814.C10940@ibook.distro.conectiva> Message-ID: <200212162234.gBGMYuq10231@odiug.zope.com> > > Unless there's some reason to believe this would be hard, what's the > > point? > > The point is to catch the security flaws, and fix them. I think that Jeremy meant that this is only a useful occupation if you expect to eventually reach a point where you expect that there aren't any security flaws left. Jeremy & I both doubt that Python will ever reach that level, meaning that the whole exercise of fixing security flaws is a waste of time (if you know you *can't* make it safe, don't waste time trying). --Guido van Rossum (home page: http://www.python.org/~guido/) From lists@morpheus.demon.co.uk Mon Dec 16 22:54:44 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Mon, 16 Dec 2002 22:54:44 +0000 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) References: <200212161941.gBGJftc05924@odiug.zope.com> Message-ID: Guido van Rossum writes: >> Playing tricks with the contents of pkg.__path__ apparently has its >> uses, > > It's the *defined* API for adding to the set of locations from which > submodules and subpackages of a package are to be loaded. You can't > just call that "playing tricks." This is entirely true. I think the key issue for import hooks is whether application code is entitled to make assumptions about what the system puts onto pkg.__path__ before __init__.py gets control. But the writer of __init__.py knows how the package is being packaged, and hence what import hook is responsible for setting up __path__. So it can be coded appropriately. The mistake here is in assuming that arbitrary code can be transparently zipped (or stored in another way, using an import hook) and still work. Some (most) code can, but not all. > Of course. If you freeze code, you have to make sure that it will be > able to find its data files in some other way. But not all code has > to be freezable. This agrees with what I just said (with "freeze" replaced by "zip" or whatever). Paul -- This signature intentionally left blank From just@letterror.com Mon Dec 16 23:29:53 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 17 Dec 2002 00:29:53 +0100 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <200212161941.gBGJftc05924@odiug.zope.com> Message-ID: Guido van Rossum wrote: > > Playing tricks with the contents of pkg.__path__ apparently has its > > uses, > > It's the *defined* API for adding to the set of locations from which > submodules and subpackages of a package are to be loaded. You can't > just call that "playing tricks." pkg.__path__ is a package-local equivalent of sys.path. But if a package didn't *originate* from sys.path, it makes little sense to demand pkg.__path__ be a list of strings. But: it could then just as well be an empty list. With my patch I think frozen package imports could even be fixed to do that. Hm, that would be real easy in fact. Will try this out. > Does the metahook also apply to submodules (or subpackage) of > packages? Of course it does. An importer must be able to handle submodules of packages it loaded, if it wants. Frozen import works that way: in fact you can see the frozen import mechanism as an imaginary importer on sys.meta_path. sys.meta_path is simply traversed before sys.path/pkg.__path__, always. A big difference between my hooks and the classic imp.find_module/load_module functions is that my hooks *always* get called with the fully qualified module name (and have to look up sys.path or pkg.__path__ by themselves if they need it), and imp.find_module must be called with a (sub)name and a path argument. The latter is fine for of sys.path-based imports but is a pain for any other, as then you somehow need to reconstruct the parent name (which actually led to freeze (ab)using __path__ to communicate the parent name). > I'd expect not. (I haven't had the time to review your latest > patches yet, in part due to this thread. :-) Oh, this aspect hasn't changed since since I first posted it on sf, but I think you spent more time in zipimport.c than in import.c ;-) > Surely you shouldn't be looking for builtin submodules of a package. Why not? (Builtin submodules won't work anyway at the moment, but for other reasons.) > > Depending on the kind of importer, there are different > > levels of freedom of what you can use as pkg.__path__. > > > > Importer object on sys.meta_path: > > it can use anything it pleases (even None), as long > > as a __path__ variable is set. > > But I can imagine that some metahooks would like to look inside the > __path__ list for more hints on where to find the submodules of the > package. Of course that's up to the metahook. Exactly. __path__ can be seen as private to an importer. It just happens to be a list of strings for the sys.path importer ;-). > > Importer object on sys.path: > > pkg.__path__ must be a list; it's most logical to use > > an importer object as the only item. Could be the same > > importer instance that imported the package itself. > > Multiple items would make sense too. Sure. > > A hook on sys.path_hooks: > > pkg.__path__ must be a list and its only item should > > be a string that the hook can handle itself. > > IMO it should allow multiple strings too! It sure does, I formulated that clumsily. If no meta_path hook picks up the import, sys.path or pkg.__path__ traversal does its work as usual. Apart from the sys.path_hooks hookability of sys.path/pkg.__path__ items, nothing was changed here. > > These are just guidelines: a set of hooks could in theory > > deliberately set pgk.__path__ up so submodule imports be > > handled by an entirely different importer. Not sure how > > useful that would be... > > Very useful, for not-yet-imagined cases. I was just pointing out that it's possible... Just From skip@pobox.com Mon Dec 16 18:25:28 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 16 Dec 2002 12:25:28 -0600 Subject: [Python-Dev] summary for 2002-12-01 through 2002-12-15 In-Reply-To: References: Message-ID: <15870.6808.471292.407695@montanaro.dyndns.org> Brett> ========================================================= Brett> `extension module .o files wind up in the wrong place`__ Brett> ========================================================= Brett> __ http://mail.python.org/pipermail/python-dev/2002-December/030644.html Brett> Skip Montanaro discovered that if you built Python in a subdirectory of Brett> the source directory (e.g., source in ``source/`` and you are building in Brett> ``source/build/``) the .o files are put in the wrong place (they use Brett> ``source/`` as the root directory to place things instead of Brett> ``source/build/``). Skip resolved his issue by doing a ``make distclean`` Brett> and then running ``../configure``. Actually, no, I haven't yet resolved this. Someone else said 'make distclean' seemed to work for them. For the time being I'm just using two CVS checkouts and building in the source tree. I need to see if a bug report needs filing... Skip From neal@metaslash.com Tue Dec 17 01:49:35 2002 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 16 Dec 2002 20:49:35 -0500 Subject: [Python-Dev] deprecating APIs Message-ID: <20021217014935.GQ16807@epoch.metaslash.com> Are we still planning to provide a mechanism to declare APIs, etc as deprecated? I have this code added to pyport.h which works for gcc: /* Py_DEPRECATED(version) * Declare a macro or function deprecated. * Usage: * extern int old_var Py_DEPRECATED(2.3); * typedef int T1 Py_DEPRECATED(2.4); * extern int x() Py_DEPRECATED(2.5); */ #if defined(__GNUC__) && (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__)) #else #define Py_DEPRECATED(VERSION_UNUSED) #endif The version number is not necessary. It is only for documentation. I don't know how to support other compilers. We could add Py_USE_DEPRECATED and/or Py_HIDE_DEPRECATED for providing/hiding access to deprecated features. #ifdef Py_USE_DEPRECATED #define Py_SOME_DEPRECATED_MACRO #endif or #ifndef Py_HIDE_DEPRECATED #define Py_SOME_DEPRECATED_MACRO #endif Unfortunately, the Py_DEPRECATED macro doesn't work with macros, so we would have to use Py_USE_DEPRECATED/Py_HIDE_DEPRECATED around deprecated macros. Neal From tim.one@comcast.net Tue Dec 17 05:34:35 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 17 Dec 2002 00:34:35 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212161145.21295.mmclay@comcast.net> Message-ID: [Michael McLay] > Fixing the decimal place would make sense for applications having > to do with money. That is quite different than how numbers are > treated in scientific calculations. Of course! FixedPoint was written for people with fixed-point applications, and especially for those of such who think a computer should be as good at decimal math as their 12-year old daughter driving a $1 hand calculator <0.500021 wink>. > That is were I borrowed the idea of significant digits. Accountants > tend to think that all digits are significant, down to the last > penny. "The last penny" is an inherently fixed-point concept: the last penny is measured in units of 0.01 regardless of how many digits appear before the decimal point. That's what FixedPoint models directly. The total number of digits is unbounded, much like the national debt, but the position of the decimal point is fixed. > ... > My goal is to have a new numerical type added to Python that > satisfies the requirements for accountants and newbies. Your > description of fixed point seems to meet this goal. That was its intent, but it is an expensive flavor of arithmetic, and not to everyone's tastes. > My confusion was raised by the 'e' notation seeming to add precision > to numbers (e.g. 1.0e5 expands to 100000.0.) Well, that didn't come from the FixedPoint module -- it always gives a number of digits after the decimal point equal to the number the user tells it to use; it doesn't try to infer anything from the form of a literal. > This would be incorrect for reporting significant digits in scientific > papers, but the rules for accounting math are different. This was > simply my confusion. FixedPoint appears to be sufficient for bankers > and accountants. I believe so, but IANAB and IANAA. > ... > This is a very good long term goal. So for Python3k the standard > numerical type will be Cowlishaw's decimal numbers. Don't know. It brings its own kinds of problems, not least of which is sloth. > To bad the decNumber library from IBM is not open source. A Java implementation is open: http://www2.hursley.ibm.com/decimalj/ > In the short term what do you advise be done with the patch to > add a fixed point literal and a builtin fixed point conversion > function? In the short term it's most important to get FixedPoint into 2.3 (under whatever name people like). One thing at a time is more than I can handle these days. > I could change the name from decimal() to FixedPoint(), but the > capital letters in the name scream "ugly". I don't care about the name. > This isn't consistent with the short names used by > the other numerical conversion functions (int, float, hex, oct...). > Would fixed() be an acceptable name? Make everyone unhappy and call it decimal_fixed_point() <0.9 wink>. I'm not sure why you might prefer fixed() to decimal(), BTW. One reason I would is that decimal() reminds of oct() and hex(), but would do something very different. > This also suggests the suffix should be changed to 'f' and > 'F' instead of 'd' and 'D'. -1. 'f' is a legit suffix for a float literal in C, and so much of Python's literal syntax is identical to C's that reusing part of it to mean something else seems a Bad Idea. >> http://www2.hursley.ibm.com/decimal/decifaq.html > According to this reference the most frequent numerical column > type used in commercial databases is decimal (55%). The float type > is used just 1.4% of the time with the rest being split between > Integer and SmallInt. > ... > Is it safe to conclude that Python will be be much more appealing > to a huge segment of the IT market if grows a native decimal number > type? I've never worked in the IT market and don't know what drives them. From email received in response to FixedPoint, all I've gleaned is that (1) they want arithmetic that exactly mimics whatever idiosyncratic schemes are used by a huge variety of software systems I never heard of; and, (2) they're unable to explain what those schemes are well enough to mimic them. Most people in this business have no idea what "the rules" are, they just know that their butt is on the line if they deviate from what The Company has been seeing for the last 20 years. So, e.g., you get requests for flatly insane rounding rules, like "if the fraction is over 0.4, round away from 0, else round toward it". Most of the time it turns out that isn't really the rule they need, either, it's just the first one they generalized from one example, after failing to find docs. FixedPoint is extremely careful to give the best possible numeric result in all cases. I do know that isn't really what the IT market cares about -- compatability with legacy systems-- themselves incompatible with each other --is much more important to them. OTOH, giving the best possible result is, as the IEEE-754 fp standard demonstrated with force in practice, explainable, reproducible, and ultimately compelling. That it's also expensive is just icing on the cake . [Michael McLay] >> I could change the name from decimal() to FixedPoint(), but the >> capital letters in the name scream "ugly". [Greg Ewing] > Worse, it would completely lose the connotation that it is a *decimal* > number type, which surely is the main point of it in the first place! Only to a compsci type. The intended audience never heard of binary fixed-point -- fixed-point means decimal to them as surely as New Zealand means anarchy to an American . > If it's somehow important that the name contain at least one capital > letter, It isn't. From martin@v.loewis.de Tue Dec 17 07:12:52 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 17 Dec 2002 08:12:52 +0100 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212161609.03192.pobrien@orbtech.com> References: <200212161609.03192.pobrien@orbtech.com> Message-ID: "Patrick K. O'Brien" writes: > I'd like to call attention to a bug report I just submitted to SF > concerning a potentially serious problem with the c-version of pickle. [...] > I think this is a pretty serious problem, but I'm out of my league when > it comes to debugging the c-version of pickle. I hear such statements quite often recently, but I'm not impressed. You have the full source, so there is no principle problem that would make it impossible to investigate the problem yourself. It might take you more time to investigate this than it may take somebody else, but I'm sure you have heard about the printf function; if not, you should read some book first. Understanding software that does not work right is *always* difficult, and time-consuming, because it usually looks right - atleast that's what the author of the software thought when he wrote it. It might be that you may not have the time to investigate this, but then, that might hold for everybody else. Regards, Martin From mmclay@comcast.net Tue Dec 17 11:39:48 2002 From: mmclay@comcast.net (Michael McLay) Date: Tue, 17 Dec 2002 06:39:48 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: References: Message-ID: <200212170639.33487.mclay@nist.gov> On Tuesday 17 December 2002 12:34 am, Tim Peters wrote: > [Michael McLay] > > > Fixing the decimal place would make sense for applications having > > to do with money. That is quite different than how numbers are > > treated in scientific calculations. > > Of course! FixedPoint was written for people with fixed-point > applications, and especially for those of such who think a computer should > be as good at decimal math as their 12-year old daughter driving a $1 hand > calculator <0.500021 wink>. > > > That is were I borrowed the idea of significant digits. Accountants > > tend to think that all digits are significant, down to the last > > penny. > > "The last penny" is an inherently fixed-point concept: the last penny is > measured in units of 0.01 regardless of how many digits appear before the > decimal point. That's what FixedPoint models directly. The total number > of digits is unbounded, much like the national debt, but the position of > the decimal point is fixed. It's always good to plan ahead for the government's budget. Thanks. > > My goal is to have a new numerical type added to Python that > > satisfies the requirements for accountants and newbies. Your > > description of fixed point seems to meet this goal. > > That was its intent, but it is an expensive flavor of arithmetic, and not > to everyone's tastes. Yes, but since it does what most people expect, it will make Python more appealing to the average Joe. Now that I have a better understanding of Cowlishaw's decimal numbers I can see how they would be a better solution, but for the short term having numbers that behave like most calculators would be a reasonable start. I am concerned about one thing. Will the fixed point implementation you have implemented be backwards compatible with an implementation of Cowlishaw's decimal numbers? > > My confusion was raised by the 'e' notation seeming to add precision > > to numbers (e.g. 1.0e5 expands to 100000.0.) > > Well, that didn't come from the FixedPoint module -- it always gives a > number of digits after the decimal point equal to the number the user tells > it to use; it doesn't try to infer anything from the form of a literal. Most of the time this is probably the right thing to do. Bit I think it might be a good idea to infer the precision from string literals if no precision is explicitly defined. As it is currently implemented the behavior is: > fixed("12.3") 12.30 The addition of '0' is unexpected. I can see the need to make specify precision when the value being converted is a floating point number, but if it is a string then the number is a literal value specified by the user. They should get exactly what they write. That would be: > fixed("12.3") fixed(12.3,1) > fixed("12.305") fixed(12.305,3) Maybe this is too subtle of a rule, but without this rule it will be difficult to implement a literal that doesn't require the explicit definition of a level of precision. So, looking forward the rule for adding precision should be a default to 2 decimal places for floats and integers, but it should be a literal interpretation of the value passed to the constructor as a string. For conversions for floats the results would be: > fixed(12.3) fixed(12.30,2) > fixed(12.305) fixed(12.30,2) > fixed(12.3,3) fixed(12.300,3) > fixed(12.305,1) fixed(12.3,1) > fixed(12) fixed(12.00,2) (Perhaps it should be a default to a precision of 0 if the value passed is an integer.) Ideally the proposal for adding literals would be added to 2.3, but Guido vetoed that, and the literals would derive the precision from the number of trailing zeros that were written: > 12.3d 12.3 > 12.300d 12.300 If the rule for defining the default precision doesn't make the exception for a literal string then it will be difficult to add a syntactic literal that doesn't require the explicit definition of the level of precision. This would end up with an awkward syntax, such as: > 12.3:1d 12.3 > 12.30:3d 12.300 > > This would be incorrect for reporting significant digits in scientific > > papers, but the rules for accounting math are different. This was > > simply my confusion. FixedPoint appears to be sufficient for bankers > > and accountants. > > I believe so, but IANAB and IANAA. And neither am I. But considering the inconsistencies in their current expectations I think you have defined a class that will have few objections. > > ... > > This is a very good long term goal. So for Python3k the standard > > numerical type will be Cowlishaw's decimal numbers. > > Don't know. It brings its own kinds of problems, not least of which is > sloth. Has someone accused you of sloth? Hardly. I'm sure I'll irritate Guido by suggesting the addition of a second parser to Python, but through this mechanism I think it will be possible to make a natural and relatively painless transition to using Cowlishaw's numbers as the default number type in Python 3k > > To bad the decNumber library from IBM is not open source. > > A Java implementation is open: > > http://www2.hursley.ibm.com/decimalj/ That's great for jython, but it will require more work for the C Python implementation. > > In the short term what do you advise be done with the patch to > > add a fixed point literal and a builtin fixed point conversion > > function? > > In the short term it's most important to get FixedPoint into 2.3 (under > whatever name people like). One thing at a time is more than I can handle > these days. Typing "FixedPoint" instead of "fixed" will be quite irritating, but I'll be happy with either name if the class is included in the standard distribution. > > I could change the name from decimal() to FixedPoint(), but the > > capital letters in the name scream "ugly". > > I don't care about the name. I'd rather have a literal added to the language and a builtin "fixed" as the name, but Guido has vetoed the literal. Again. I'll be happy with anything that adds the capabilities to the standard distribution. > > This isn't consistent with the short names used by > > the other numerical conversion functions (int, float, hex, oct...). > > Would fixed() be an acceptable name? > > Make everyone unhappy and call it decimal_fixed_point() <0.9 wink>. I'm > not sure why you might prefer fixed() to decimal(), BTW. One reason I > would is that decimal() reminds of oct() and hex(), but would do something > very different. Good idea, calling it decimal_fixed_point should maximize the irritation of end users. This will lead to more pressure to add a literal to the language. I thought it might be best to use "fixed" for fixed point in case there was some incompatibility with the eventual inclusion of Cowlishaw's decimal numbers. The intention was to reserve "decimal" for the more generalized form of decimal numbers. If your FixedPoint implementation is 100% compatible then using "decimal" would be best. Being able to extend FixedPoint semantics to include floating point decimals in the future would be the best option. > > This also suggests the suffix should be changed to 'f' and > > 'F' instead of 'd' and 'D'. > > -1. 'f' is a legit suffix for a float literal in C, and so much of > Python's literal syntax is identical to C's that reusing part of it to mean > something else seems a Bad Idea. I didn't like the change for the same reason, but I made the change so that 'd' and 'D' could be reseved for Cowlishaw's decimal numbers. If they are compatible then I withdraw the change from my original proposal. > >> http://www2.hursley.ibm.com/decimal/decifaq.html > > > > According to this reference the most frequent numerical column > > type used in commercial databases is decimal (55%). The float type > > is used just 1.4% of the time with the rest being split between > > Integer and SmallInt. > > ... > > Is it safe to conclude that Python will be be much more appealing > > to a huge segment of the IT market if grows a native decimal number > > type? > > I've never worked in the IT market and don't know what drives them. From > email received in response to FixedPoint, all I've gleaned is that (1) they > want arithmetic that exactly mimics whatever idiosyncratic schemes are used > by a huge variety of software systems I never heard of; and, (2) they're > unable to explain what those schemes are well enough to mimic them. Most > people in this business have no idea what "the rules" are, they just know > that their butt is on the line if they deviate from what The Company has > been seeing for the last 20 years. So, e.g., you get requests for flatly > insane rounding rules, like "if the fraction is over 0.4, round away from > 0, else round toward it". Most of the time it turns out that isn't really > the rule they need, either, it's just the first one they generalized from > one example, after failing to find docs. > > FixedPoint is extremely careful to give the best possible numeric result in > all cases. I do know that isn't really what the IT market cares about -- > compatability with legacy systems-- themselves incompatible with each > other --is much more important to them. OTOH, giving the best possible > result is, as the IEEE-754 fp standard demonstrated with force in practice, > explainable, reproducible, and ultimately compelling. That it's also > expensive is just icing on the cake . I think you have done a fine job of providing what they need, regardless of what the end users think they want. It will be easy enough to provide conversion capabilities to the broken semantics of the sundry implementation of decimal numbers that exist. > [Michael McLay] > > >> I could change the name from decimal() to FixedPoint(), but the > >> capital letters in the name scream "ugly". > > [Greg Ewing] > > > Worse, it would completely lose the connotation that it is a *decimal* > > number type, which surely is the main point of it in the first place! > > Only to a compsci type. The intended audience never heard of binary > fixed-point -- fixed-point means decimal to them as surely as New Zealand > means anarchy to an American . Let's go easy on the kiwis. They don't always get their due. It looks like "Mad" Pearse [1] may have flown before the Wright bothers, but since he was a perfectionist he did not try to claim the title. [1] http://slashdot.org/article.pl?sid=02/12/16/0338254&mode=thread&tid=126 From mwh@python.net Tue Dec 17 12:39:46 2002 From: mwh@python.net (Michael Hudson) Date: 17 Dec 2002 12:39:46 +0000 Subject: [Python-Dev] summary for 2002-12-01 through 2002-12-15 In-Reply-To: Skip Montanaro's message of "Mon, 16 Dec 2002 12:25:28 -0600" References: <15870.6808.471292.407695@montanaro.dyndns.org> Message-ID: <2my96obyzx.fsf@starship.python.net> Skip Montanaro writes: > Actually, no, I haven't yet resolved this. Someone else said 'make > distclean' seemed to work for them. For the time being I'm just using two > CVS checkouts and building in the source tree. I need to see if a bug > report needs filing... I reopened the bug (#599248) whose fix caused the problem. Last news was that AMK was "thinking about it". I guess we could just back it out until such time as someone comes up with a better fix. Cheers, M. -- US elections For those of you fearing that the rest of the world might be making fun of the US because of this: Rest assured, we are. -- http://www.advogato.org/person/jameson/diary.html?start=12 From pobrien@orbtech.com Tue Dec 17 15:12:11 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Tue, 17 Dec 2002 09:12:11 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: References: <200212161609.03192.pobrien@orbtech.com> Message-ID: <200212170912.11638.pobrien@orbtech.com> On Tuesday 17 December 2002 01:12 am, Martin v. Löwis wrote: > "Patrick K. O'Brien" writes: > > I'd like to call attention to a bug report I just submitted to SF > > concerning a potentially serious problem with the c-version of > > pickle. > > [...] > > > I think this is a pretty serious problem, but I'm out of my league > > when it comes to debugging the c-version of pickle. > > I hear such statements quite often recently, but I'm not impressed. You don't hear statements like that from me. And impressing people on this list wasn't my goal. So I'm at a complete loss as to what I've done to deserve this kind of response from you. > You have the full source, so there is no principle problem that would > make it impossible to investigate the problem yourself. It might take > you more time to investigate this than it may take somebody else, but > I'm sure you have heard about the printf function; if not, you should > read some book first. And you don't think my asking for some help on this list is better use of two people's time? One who has an example that fails, and one who knows the cPickle implementation well enough to help track down the cause of failure. I've already spent quite a lot of time tracking the problem down to the point that I have. And it is trivial for me to fix my own code so that the problem goes away. But the underlying problem isn't with my code. It is with cPickle. And I'm concerned about just walking away from this because I think it is a serious problem for anyone who wants to correctly unpickle objects that were pickled using cPickle. Fortunately, Guido has responded to my SF bug report and I'm trying to reduce things to a simple example that illustrates the failure. I spent several hours on this last night, and I'll likely spend several hours on it today. And that's just getting us to a simple example that reproduces the problem. Frankly, I don't have the time nor the interest to get back up to speed on C. (I've read lots of C books and dabbled in it, but that was years ago.) > Understanding software that does not work right is *always* > difficult, and time-consuming, because it usually looks right - > atleast that's what the author of the software thought when he wrote > it. Sure. And this is the first time I've seen cPickle do the wrong thing. Considering that the app that found this bug is an example of a banking app that logs account transactions as pickles, I think you might appreciate that it would be a good thing if those pickles maintained proper references to those accounts and could restore transactions appropriately, rather than creating new account instances that weren't part of the bank. (Say goodbye to your money on a system recovery.) > It might be that you may not have the time to investigate this, but > then, that might hold for everybody else. And what harm did I do by asking if anyone had time to help, rather than letting this issue drop? Sorry if this isn't the kind of response you expected back from me. But I'm already irritated with the amount of time I've spent on this. Your reply really rubbed me the wrong way. If I didn't care about Python I sure as hell wouldn't be spending my time tracking this problem down. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From martin@v.loewis.de Tue Dec 17 16:28:15 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 17 Dec 2002 17:28:15 +0100 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212170912.11638.pobrien@orbtech.com> References: <200212161609.03192.pobrien@orbtech.com> <200212170912.11638.pobrien@orbtech.com> Message-ID: "Patrick K. O'Brien" writes: > > I hear such statements quite often recently, but I'm not impressed. > > You don't hear statements like that from me. And impressing people on > this list wasn't my goal. So I'm at a complete loss as to what I've > done to deserve this kind of response from you. Your original message sounded much like a support question. I was trying to say (perhaps in impolite words, for which I apologize) that posting support requests is not appropriate for this mailing list. > And you don't think my asking for some help on this list is better use > of two people's time? One who has an example that fails, and one who > knows the cPickle implementation well enough to help track down the > cause of failure. Valuing one person's time against the time of another person is always a difficult issue - unless one person pays the other for the time. > It is with cPickle. And I'm concerned about just walking away from this > because I think it is a serious problem for anyone who wants to > correctly unpickle objects that were pickled using cPickle. And it is a good thing that you have submitted a bug report to sourceforge. > Sure. And this is the first time I've seen cPickle do the wrong thing. > Considering that the app that found this bug is an example of a banking > app that logs account transactions as pickles, I think you might > appreciate that it would be a good thing if those pickles maintained > proper references to those accounts and could restore transactions > appropriately, rather than creating new account instances that weren't > part of the bank. I certainly agree that it is desirable to fix this bug, just like it is desirable to fix all the other bugs. > And what harm did I do by asking if anyone had time to help, rather than > letting this issue drop? Sorry if this isn't the kind of response you > expected back from me. But I'm already irritated with the amount of > time I've spent on this. Your reply really rubbed me the wrong way. If > I didn't care about Python I sure as hell wouldn't be spending my time > tracking this problem down. It was the (perceived) attitude "my bug report is more important than the other 320 open bug reports" that rubbed me the wrong way. I do appreciate that you took the time to report the bug. This is all volunteer's time: users voluntarily take the time to report bugs, Python developers voluntarily take the time to investigate and fix them. The perceived seriousness of a problem is but one factor for a contributor to determine whether to work on the problem. Regards, Martin From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_0052_01C2A532.3C12AD40 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable [Guido van Rossum] >The problem is that Zope3 needs *some* way of storing data and config >files related to specific packages, and there isn't really any other >convenient and conventional way except storing it in a package >directory and using the package's __file__ to find it. __importer__ , the zip specific alternative to __file__ could be = provided to solve the problem. in an __init__.py: if __file__: ............................ elif __importer__: f=3D___importer__.open()#to speedup and to get direct access to = the binary #without ___importer__.open() should work as well = =20 data1=3D___importer__.get_data("ModuleName\\index.html") data2=3D___importer__.get_data("ModuleName\\help.html") if data1: .......... f.close() It would require some minor changes to Just's zipimport.c + new function in import.c PyImport_ExecCodeModuleEx2(char *name, PyObject *co, char = *pathname,PyObject *importer) where instead of setting __file__: if (PyDict_SetItemString(d, "__importer__", importer) !=3D 0) ........................................................ As for zipimport.c: new methods: open() for speedup : ........ char* fname=3DPyString_AsString(self->archive); self->arcfile=3DPyFile_FromString(fname,"rb"); ....... return self->arcfile; #PyObject*arcfile new field(zipimporter struct) .................................................................. get_data(data_path)#modified load_module=20 .................................................................. in load_module: mod =3D PyImport_ExecCodeModuleEx2(fullname, code, modpath,self); instead of=20 mod =3D PyImport_ExecCodeModuleEx(fullname, code, modpath); Tested;) Regards Wiktor http://www.wiktorsadowski.com ------=_NextPart_000_0052_01C2A532.3C12AD40 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
[Guido van Rossum]
>The problem is that Zope3 needs *some* = way of=20 storing data and config
>files related to specific packages, and = there=20 isn't really any other
>convenient and conventional way except = storing it=20 in a package
>directory and using the package's __file__ to find = it.
 
__importer__ , the zip specific alternative to __file__ could=20 be provided to solve the problem.
 
in an __init__.py:
 
if __file__:
    = ............................
elif=20 __importer__:
      = f=3D___importer__.open()#to=20 speedup and to get direct access to the binary
      #without  ___importer_= _.open() =20 should work as=20 well           &nb= sp;           &nbs= p;           
&= nbsp;    =20 data1=3D___importer__.get_data("ModuleName\\index.html")
  &= nbsp;  =20 data2=3D___importer__.get_data("ModuleName\\help.html")
  &n= bsp;  =20 if data1:
     =20 ..........
      f.close()
 
It would require some minor changes to Just's zipimport.c
+ = new =20 function in import.c
PyImport_ExecCodeModuleEx2(char *name, PyObject = *co,=20 char *pathname,PyObject *importer)
where instead of setting = __file__:
if=20 (PyDict_SetItemString(d, "__importer__", importer) !=3D=20 0)
........................................................
 
As for zipimport.c:
new methods:
open() for speedup=20 :
........
        char*=20 fname=3DPyString_AsString(self->archive);
 self->arcfile=3D= PyFile_FromString(fname,"rb");
 .......
 return=20 self->arcfile; #PyObject*arcfile new field(zipimporter=20 struct)
..............................................................= ....
get_data(data_path)#modified=20 load_module=20
..................................................................in=20 load_module:
mod =3D PyImport_ExecCodeModuleEx2(fullname, code,=20 modpath,self);
instead of
mod =3D = PyImport_ExecCodeModuleEx(fullname, code,=20 modpath);
 
Tested;)
 
Regards
Wiktor
http://www.wiktorsadowski.com<= /DIV>
 

 
------=_NextPart_000_0052_01C2A532.3C12AD40-- From niemeyer@conectiva.com Tue Dec 17 17:34:23 2002 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Tue, 17 Dec 2002 15:34:23 -0200 Subject: [Python-Dev] known obvious thing or bug (rexec)? In-Reply-To: <200212162234.gBGMYuq10231@odiug.zope.com> References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com> <20021216201814.C10940@ibook.distro.conectiva> <200212162234.gBGMYuq10231@odiug.zope.com> Message-ID: <20021217153423.A17719@ibook.distro.conectiva> > I think that Jeremy meant that this is only a useful occupation if you > expect to eventually reach a point where you expect that there aren't > any security flaws left. Jeremy & I both doubt that Python will ever > reach that level, meaning that the whole exercise of fixing security > flaws is a waste of time (if you know you *can't* make it safe, don't > waste time trying). Ok.. I'll take your suggestion. Thank you. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From just@letterror.com Tue Dec 17 19:54:05 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 17 Dec 2002 20:54:05 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <005501c2a529$db51ab80$0201010a@wiktor> Message-ID: Wiktor Sadowski wrote: > [Guido van Rossum] > >The problem is that Zope3 needs *some* way of storing data and config > >files related to specific packages, and there isn't really any other > >convenient and conventional way except storing it in a package > >directory and using the package's __file__ to find it. > > __importer__ , the zip specific alternative to __file__ could be > provided to solve the problem. Hm, I like this! > in an __init__.py: > > if __file__: > ............................ > elif __importer__: > f=___importer__.open()#to speedup and to get direct access to the binary > #without ___importer__.open() should work as well > data1=___importer__.get_data("ModuleName\\index.html") > data2=___importer__.get_data("ModuleName\\help.html") > if data1: > .......... > f.close() Or: try: __importer__ except NameError: else: x = __importer__.get_data(...) > It would require some minor changes to Just's zipimport.c > + new function in import.c > PyImport_ExecCodeModuleEx2(char *name, PyObject *co, char *pathname,PyObject *importer) > where instead of setting __file__: > if (PyDict_SetItemString(d, "__importer__", importer) != 0) > ......................................................... I'd say it should set __importer__ in *addition* to __file__. > As for zipimport.c: > new methods: > open() for speedup : > ......... > char* fname=PyString_AsString(self->archive); > self->arcfile=PyFile_FromString(fname,"rb"); > ....... > return self->arcfile; #PyObject*arcfile new field(zipimporter struct) > ................................................................... > get_data(data_path)#modified load_module > ................................................................... > in load_module: > mod = PyImport_ExecCodeModuleEx2(fullname, code, modpath,self); > instead of > mod = PyImport_ExecCodeModuleEx(fullname, code, modpath); > > Tested;) Ugh, you should really consider getting a decent diff tool :-/ How can *I* test if if I have no clue what exactly you've done? I can see that keeping the file object open could speed things up, but is it a polite & decent thing to do? Just From nas@python.ca Tue Dec 17 20:14:27 2002 From: nas@python.ca (Neil Schemenauer) Date: Tue, 17 Dec 2002 12:14:27 -0800 Subject: [Python-Dev] deprecating APIs In-Reply-To: <20021217014935.GQ16807@epoch.metaslash.com> References: <20021217014935.GQ16807@epoch.metaslash.com> Message-ID: <20021217201427.GA2520@glacier.arctrix.com> Neal Norwitz wrote: > Are we still planning to provide a mechanism to declare APIs, etc as > deprecated? I think it would be a good idea. Neil From Jack.Jansen@oratrix.com Tue Dec 17 21:11:40 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 17 Dec 2002 22:11:40 +0100 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <200212161747.gBGHlYK03859@odiug.zope.com> Message-ID: <22D6ACBE-1204-11D7-BC1B-000A27B19B96@oratrix.com> On maandag, dec 16, 2002, at 18:47 Europe/Amsterdam, Guido van Rossum wrote: > - Nobody remembers why there is a special case in site.py for lines in > the .pth file that start with 'import', except that Martin von > Loewis once wanted this. I propose not to duplicate this > functionality in .pkg files. If you *really* *really* want to prepend an item to sys.path [Jack sits quietly to a thundering speech as to why you shouldn't want this, studying his fingernails and various other parts of his anatomy] then the "import" hack in .pth files is the only way to do it. And if I want to release a MacPython-OSX that sits on top of Apple-supplied /usr/bin/python in MacOSX 10.2 then I really must have a way to prepend to sys.path, because there is at least one Mac module (macfs) that is braindead in Python 2.2.0. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From greg@cosc.canterbury.ac.nz Tue Dec 17 21:42:04 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 18 Dec 2002 10:42:04 +1300 (NZDT) Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Message-ID: <200212172142.gBHLg4K17376@oma.cosc.canterbury.ac.nz> Tim Peters : > under FixedPoint's unusual "unbounded > before the decimal point, fixed # of digits after, and when mixing precision > the result gets the larger of the input precisions" rules, all of the > following are always exact operations: > > + - * This is true for + and -, but not * -- you need the sum of the input precisions to avoid losing information in that case. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From Jack.Jansen@oratrix.com Tue Dec 17 21:43:23 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 17 Dec 2002 22:43:23 +0100 Subject: [Python-Dev] Strange crash in import Message-ID: <911129A1-1208-11D7-BC1B-000A27B19B96@oratrix.com> I'm getting a strange crash in import, and it turns out that there's a refcount 0 object on sys.path. Has any of the code for zip imports been checked in yet? Otherwise, does anyone have a clue what could be going on (i.e. what silly mistake did I make:-)? The situation I'm in (the stack trace is below) is that I'm in the tp_init routine of an object when a macos-error occurs. I then call PyMac_Error to format the error. PyMac_Error then needs to import a pure Python module (which will give it the pretty huge list of MacOS error codes and the corresponding strings). This crashes, because the second entry on sys.path is a refcount 0 object: #1 0x003fcf10 in find_module (realname=0x46f2cc "", path=0x13b310, buf=0xbfffeac0 "macresource.pyc", buflen=1025, p_fp=0xbfffeed0) at ../Python/import.c:987 987 if (PyUnicode_Check(v)) { (gdb) p _PyObject_Dump(path) object : ['', , '/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/plat- darwin', '/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/lib- tk', '/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/lib- dynload', '/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site- packages', '/Library/Frameworks/Python.framework/Versions/2.3/Mac/Lib', '/Library/Frameworks/Python.framework/Versions/2.3/Mac/Lib/lib- scriptpackages'] If I manually import the "macresource" module beforehand everything works as expected, and sys.path[1] isn't a refcount zero object. Here's the full stack trace: (gdb) where #0 PyType_IsSubtype (a=0x0, b=0x457d44) at ../Objects/typeobject.c:704 #1 0x003fcf10 in find_module (realname=0x46f2cc "", path=0x13b310, buf=0xbfffeac0 "macresource.pyc", buflen=1025, p_fp=0xbfffeed0) at ../Python/import.c:987 #2 0x003fe520 in import_submodule (mod=0x4522b8, subname=0xbfffefc0 "macresource", fullname=0xbfffefc0 "macresource") at ../Python/import.c:1998 #3 0x003fe034 in load_next (mod=0x4522b8, altmod=0x4522b8, p_name=0xbfffefc0, buf=0xbfffefc0 "macresource", p_buflen=0xbffff3d0) at ../Python/import.c:1863 #4 0x003ffef0 in import_module_ex (name=0x0, globals=0x457d44, locals=0x1b6, fromlist=0x13b390) at ../Python/import.c:1714 #5 0x003ff0d4 in PyImport_ImportModuleEx (name=0x145b9c "macresource", globals=0x148780, locals=0x148780, fromlist=0x13b390) at ../Python/import.c:1755 #6 0x003ce2fc in builtin___import__ (self=0x0, args=0x457d44) at ../Python/bltinmodule.c:40 #7 0x00370b98 in PyObject_CallFunction (callable=0x12efd0, format=0x0) at ../Objects/abstract.c:1675 #8 0x003feb40 in PyImport_Import (module_name=0x145b88) at ../Python/import.c:2160 #9 0x003fefb8 in PyImport_ImportModule (name=0x0) at ../Python/import.c:1686 #10 0x004108a0 in PyMac_StrError (err=-43) at ../Python/mactoolboxglue.c:107 #11 0x00410e00 in PyMac_Error (err=-43) at ../Python/mactoolboxglue.c:169 #12 0x00265af4 in myPyMac_GetFSRef (v=0x18c760, fsr=0x13e4f8) at /Users/jack/src/python/Mac/Modules/file/_Filemodule.c:2400 #13 0x00263ef8 in FSRef_tp_init (self=0x13e4f0, args=0x457d44, kwds=0x1b6) at /Users/jack/src/python/Mac/Modules/file/_Filemodule.c:1243 #14 0x003b407c in type_call (type=0x268454, args=0x14fed0, kwds=0x0) at ../Objects/typeobject.c:438 #15 0x00374028 in PyObject_Call (func=0x0, arg=0x457d44, kw=0x1b6) at ../Objects/abstract.c:1675 #16 0x003da754 in do_call (func=0x45e88c, pp_stack=0xbffff8fc, na=1, nk=4443236) at ../Python/ceval.c:3398 #17 0x003da53c in call_function (pp_stack=0xbffff8fc, oparg=4554052) at ../Python/ceval.c:3268 #18 0x003d8044 in eval_frame (f=0xa37c0) at ../Python/ceval.c:2009 #19 0x003d9354 in PyEval_EvalCodeEx (co=0x141660, globals=0x457d44, locals=0x3fcf10, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2554 #20 0x003dbfe0 in PyEval_EvalCode (co=0x0, globals=0x457d44, locals=0x1b6) at ../Python/ceval.c:478 #21 0x0040b788 in run_node (n=0x0, filename=0x457d44 "", globals=0x148780, locals=0x148780, flags=0x0) at ../Python/pythonrun.c:1089 #22 0x0040ac88 in PyRun_InteractiveOneFlags (fp=0xa390c, filename=0x141660 "", flags=0xbffffc50) at ../Python/pythonrun.c:593 #23 0x0040aa80 in PyRun_InteractiveLoopFlags (fp=0x44aaac, filename=0x137140 "", flags=0x14a4c0) at ../Python/pythonrun.c:522 #24 0x0040c604 in PyRun_AnyFileExFlags (fp=0xa0000cd4, filename=0x444fdc "", closeit=669964, flags=0xbffffc50) at ../Python/pythonrun.c:485 #25 0x00416db0 in Py_Main (argc=-1610609452, argv=0x0) at ../Modules/main.c:442 #26 0x00001b5c in _start (argc=673326145, argv=0x0, envp=0x44653c) at /SourceCache/Csu/Csu-45/crt.c:267 #27 0x000019dc in start () -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From greg@cosc.canterbury.ac.nz Tue Dec 17 21:50:20 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 18 Dec 2002 10:50:20 +1300 (NZDT) Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Message-ID: <200212172150.gBHLoKb17403@oma.cosc.canterbury.ac.nz> Tim Peters : > [Greg Ewing] > > Worse, it would completely lose the connotation that it is a *decimal* > > number type, which surely is the main point of it in the first place! > > Only to a compsci type. The intended audience never heard of binary > fixed-point -- fixed-point means decimal to them as surely as New Zealand > means anarchy to an American . But they're going to have to learn that there's something other than decimal in order to understand why they should be using this fixed thingy in the first place, instead of using floats. I don't think you'll be doing them any favours by using terminology that obscures the difference. There's a precedent for this in SQL, which has a type called DECIMAL. Now, why do you think they called it that and not something blander like NUMBER? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From Wiktor Sadowski" This is a multi-part message in MIME format. ------=_NextPart_000_0047_01C2A558.C1C03720 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0048_01C2A558.C1C03720" ------=_NextPart_001_0048_01C2A558.C1C03720 Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable [Just van Rossum] >Ugh, you should really consider getting a decent diff tool :-/ How can >*I* test if if I have no clue what exactly you've done? Well, as you already know my implementation of custom imports (import = hooks) is a bit different and I am using modified Python22 ,so it's not easy to make diffs. Zipimport.c in attached zipimport.tar.gz is to provide you with more = details. Don't try to compile it ;-) >I can see that keeping the file object open could speed things up, but >is it a polite & decent thing to do? Yes, but it would be nice to not fopen/fclose for each = __importer__.get_data. Best Wiktor http://www.wiktorsadowski.com ------=_NextPart_001_0048_01C2A558.C1C03720 Content-Type: text/html; charset="windows-1250" Content-Transfer-Encoding: quoted-printable
[Just van Rossum]
>Ugh, you should really consider getting a = decent=20 diff tool :-/ How can
>*I* test if if I have no clue what exactly = you've=20 done?
 
Well, as you already know my implementation of custom imports = (import=20 hooks) is a bit different
and I am using  modified = Python22=20 ,so it's not easy to make diffs.
 
Zipimport.c in attached zipimport.tar.gz  is to provide = you with=20 more details.
Don't try to compile it ;-)
 
>I can see that keeping the file object open could speed things = up,=20 but
>is it a polite & decent thing to do?
 
Yes, but it would be nice to not fopen/fclose for each=20 __importer__.get_data.
 
Best
Wiktor
http://www.wiktorsadowski.com<= /DIV> ------=_NextPart_001_0048_01C2A558.C1C03720-- ------=_NextPart_000_0047_01C2A558.C1C03720 Content-Type: application/x-gzip; name="zipimport.tar.gz" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="zipimport.tar.gz" H4sICNpL/j0AAHppcGltcG9ydC50YXIA7Tz9UxvHkvlVqcr/MJbLIIEAQcDOmYdTshE2d9hQQPz8 YlNbizSCPa92ld2VbWLzv19/zOfuSohccq/qKsp7lnamu6enp7unp6eX36NJNJ6kWbE++O6v+ggh njx5IuAbPt3St9h+0t3ZAYidJ0+2t3ce72xD2+MfdwC++93fn7/68zBKBvF0KEXz5Ka4TpP16+YP 39vGvMimg2Isx5cyK3Wl+VCO8lLjOMzy6zAutQ7S8SSKJbXif3kRFtFADK7DbEVMbgL5pZBJHqVJ /v5C7ImvzfXJzaDZEfidqu/m7S4jb6wIo7QyE+nlf8tBIYCXKIkKoCHCZCjy6QT7xcoGohQ3EwkA gmfjof8aTQ7Vb6JfA/L1h+8bJzfHNE7wqt/bt88wQJgNrqNPcleIDZhLWFwn4ViKdCSKa4lkhAIA Vjy8EQgkByzCG0bQdCUTmYWxwB4RJaO0jDLJYJJfdgmFobjlqWiGG/n0cmMYZQCYZjcbTcJF6h6n iATTvN21qwAA5yAeBnLFEWAzAeJoafY5zIYg5kEMpA2ypX4li6BIBy1cVCOV9u4MSBBMMAyLsKWb V0LkreNhd2D2OAUQRiFAh2C2ed6hJ8QN8uh3yY+IW35MR6NcFshBDQ9xGg6JiXE6nMayZecNw+cy HnUEszKaxjGsKPHCLYCBy8wjRfkkHHwMr6R+VKo+7DiDgVgCmRTZDTPzkHRVerJ+cS0HH1vppG3Q SPy6uSOWyivTrp3XD987mhsk8nPLW96VQZx3PJW48p4/fh7mQBg1Hhb98CpJMykAaDoG/gHSJY7m Jorwo8zFIMyUyocFLxoZ4WcphmmyXMADQKZiDMAijkZSgCCHMkOtEqC3gzjMc5mvs75XloKE1sBf 4Bv8lWrDfNaeFZMgjGPQPZpdF6XciEaiRSgP9sSbX46O2mTGRGXtmTZJ7tq1HWSU1WY2M6f9Fv6f yWKaJaJlpddW3N7apUGt+OH7stiq6rbgorAGovrRHEGssUzUo5r0g5ObXnYVnIAjlufTCeg202se P3jq8LEeECdB0OzgPBuCPksnN2dFFiVXpGKgdZ682m0A1fNe2+QxT26CwzcvTvsHLR8WGUTOQGqG aC/nHzWgMI8y9BnYcwnSTlPDP9sTr3vvTnrnr476b3iRgaV+lgVnslCjwfOXQcACh5404yk3mloP iFKRpiJOk6smjFKaZoNWtKQj2uEhslE5D4T1RSwtiQfMErAhJ7hHvQ6LwbXMNWfHxFVJvHNI8jzR RPUMxuGNuNTbDtrVGAEEf5obyONGkW4o8HVQBNwwYL9ostE5uwzCBlpNQHFW9G9H/1kRL6ej91b4 q5sXpguoAGLFbBoTeNDa20Drb+3u8mRoNuAWhzQBcAiFSAeDaZaFyYDm1USmFbdECHdz+F8LRuK+ Nu/B7969E2Gcp2L918MTBU+CnBjx8fJfZjL8uGu7329fIMRZ/6RtCH37JnpH59CkCTXMpCa7ipPV PbGNv1FLmBTD6OGAiGlao+m3ocuoMLRttx0hJKnA6cAiTkEaacZPUS7CQkhoUYxYPUFelQohWF5E cSxg61OQt9YkXsQyzFqkrNbgSryJVbHNENyM09uhZxB2MpjctGDZOwTaQe+jcDYZB1VCNWIkt/yh u1waDajAgxqTkazI3luZbF6IB7QYWjQMYalT567XZbnRg9t1UcMDg7uVjcD1OgdZOuafbNmVbWAG sJ2QnlJpCKsMPjWrk77xKws/z25w77yC9QwxJgG/MfTC0xHwIPKbfN033HXlAP5pNuHhEAiReQ2Q ikinGXDyyWy8DZ8Az/MmD17Kgn1Dq+lDNO3ylVAfGNcHNPYhuOUApwRmtN56GAWOYx4WclxCUH4b 51XdQZgRDeuwUA20DJTmgCSdJ+iVKRbPSapWVkN1zFCCKocLrVJUYsgrEGYPt8p3eq+0+Ir529m2 esc2cNe+NBsfOj2lIxb88KZbimc+pdHQD2iGkoKvakzDMQvNer8/I0KwnS6D9V2Oy+CG9DLAkx0F gKNMcrDQrgvAMPIMkDSE8nOCfezVvy+npnllEtiIHx5M0D8vKmMLVtuI3SWBJCggOAx1NEWvqp3q nYGSjpSMG0U0bNR+13pFaFNsQ5P1fWIVeYN/dXjhybYULFvWLNoMJu3ysMoRdCmW8+FUWDVjEFyJ ymytSuMPb762adX4fj3Dk5vX4WSCXLwK8/+SiiVX33g7087AXWEgRZLSbCKcy2fThNHevP9Dz07G uayl2iVwOw/LsjU+pUa3C5z0MHLSZ1kLA06rfLBQKlsxgerRCpB3jXJDsMjGsUKWUAr9LrTeRxQS dOyxeIFzSe4dS5yJ+CcT5sA7g5Cu2hFgicBOspbhdXl9ua1Veu7R4G0YT6V3MlA0xHiaFxCRFRhc h7CRFoUcrmOPd0pwz4VGZ32fw27GsEbmvsSyXDLSMscIpPAP0a2dK8fNEemQiADsR/haXeUJVvW0 U0qwRRc0yh+yjNJhjwnNOgjb3YT9IC8qCpNDW95LLUX49SZNSAaapGqiaS9mB+lEJosYAKrUfW1g hEuHj/Rj72SBU61pAWEixgF8uUEjUeo0s0tWJysMF9ETidfBzmEBsUBMQOm2RURzb9+gU2TznEDD T+HZ9J2TunPTdohiUoME7rbYjJrmQp+EF1qW+3kkLTvfHeGkZ/sjBDE8gviMoenAts7SDMndmQwa msBlBJ8meA8r1SUjVp9VK+ElR8Rs1FUXxnqoIi+butTzwjFgRm4S19fWDkva8nCfdXeWB2F1pFor ZN2AgAtbAiR+/9KN0uSLzZ5Z3Sy9kPHP2zqdqd1HV3kIbTZ8QCrZDBxaOatbDhftGatEoBQT3nP3 VeNSElkNFRaAcjktZI57Mu3HORHJ5+3EaglgzXRwgLIw0YGxNANn1c3BNQ5OBXRO7+oqJ53yj9EE KdKm9icEARsrJHAduetUjMYV71chq2dizwvogS2eUaoRM7XjaNCuRbuKcTP8iwB4Vti4K7Couc5Q mb0O3YnYNN89whAEvbefRKI2hrHoD/xTsX8fg2YbIciWilqY5dI9EA2kPLFdIrM0pJ/arrzLHsMF U6/1oPYsTmPv6fDGzUFchkPQmasoEckU71wF/Lq8KST40CGwg9NM5BeTh7DjIE01QmOQJkWUTKVN LBjzAKjdmYkGbF0geS6axus8hfRIQkJU4R0ZY8kWgWrJNb+WcNU83Jcj70JpTK3qChiofG16hwEP 1vaAVF73z18Fb3unvdOXZ8AdHLibtx0iAJzNIAA9ixDAeLKEiU3zUGy8UBeBzcEkW2o08OuW8lES l1HifWvlyvY13chXJEitrgTZ3HHy4jw4fv6f/RfnHcFbbTpy97GO0M6+g8UZai5kcYh8NzqBErJF Z7fV7NyJrPaJjkElIehZQ04QjOjs/BSk1vJST+kAoFGEAn7SdWK1NKDp37ru9w/6p6f9/aC3v3/a Pztr4XdbdBe9ERd71YIAiNoPz1sVyktMCNF47+3i3Kz1rDvM0o6N4ZAvmnaH0BrKPRSQRACviHDs AlpDyRULcH1Uk5TrMJJ+BJwyuQkYeVFtR33FLbemJ5/Zg+4Qb4KrPZmc1CGEeUBurr4PRvptKuH6 p753zDtFTed1mF/XsQcyqJuPmotd0pdYiBENYAvqwUQ7JZGks2WS1nMKm8hImkGC85ODo97LswD0 pffL0bn4JpzG572z/vm/TvodweijOLzKGbWs+WaIYe3KFln4CZyBrJMEuvyadpj09exF/AzXZXGU F2zCNQARcDaj2W5aNR7fzEQ3MGTVs1lI3VCvufXsXYa1wsB8f12zzAfotmf3qVGq5DwJtTBGm2Tp oF2uAtCKhQ8KmB2GVsAem7AGc0y4VORhYPCBIQKjzfsy7ij2VJ6cIJxyKiAVS6zwCPFuGnq5HY97 5jbBHAXWnrGrV8fBl9AIJVf6Ch1+2Zonip+ieAiNKJWOUCZLBCjFmxONFswrjNXllMWd5gBLYWqY A4lcgl6G6OgIEkSBYVUE11eKlZ6Nwui6FNNM2VqBB6anBMKflj2O0uehOZ/i5D9CYLErurSXTBPd I4cG3TnDKnT4TTfUFpgPyUBtGOUfDSZNuxZzKEu4JRzWpw7h8G/4IkB1+1eEGTZp2Rn0tpr3PheG obQwYsPTEzSkVwkwMIQAkwQPDiOKtcijhNpU6ZMzbTqVuWwJ9BZdWISVjRnn79rSMHW+tnAqul+h 3M/B4VEfmib4G4synBzCIBsskEggEIyYo7E0RK5lCGVHGgInpzBVDz4YaDqR06H8Cg9oA8hZFvro TurunOnhjLNjzvVEd2bnysTLWAN5a1vPFq1lOQbPPorTz96pGQyP+KLxwQJMRcvME7J7TLMBoMoy T/DkS/lUY5GcqFTMI0BNbv0Aak/CwlS21B4Z8DoaKaOvoEWDCkabsGguP1rf6nbz5abhaeYEYCHl R2ClI9a2tjp4uEM/dwaNbBkSSyhGOAZ6PXshxGWqwSks+xFICFOymJsFSnRj0P3Sfdzd2X6+07Wl Ns/D4VPRZ3ov0MuEMdoVGPFVEgJjaoDGaBCnOVHavY9UMC8e/q8FAv+SHdmb9DdQA2jWjPrmJNV8 iT7uqNOyZzdAeq4AEYMWQTklKyxdncqCYsPCcznqaLt8xUf2pi7K/FIhy2LZoLtcAmSHxvkqILU6 8ULs8yWM0oSt7iZpArTroiGtD54iHOBgr/RgcxjFq8MucYtA2rNVGDu7BsWocIYObUFQ8ICLgoJT XUwwxvEuBo4LcA9w7ZEXZZvFqjG2H4NkLYlVroC5i9CicLvzV3R7C1bUTvo+xkLaZvj2dwHo9sVi +wBV19Tp7GQlhWcw3VReYwVxaDtuw9480HwwI9gJt9Ebyzi2BralVqurOoGkurpkDT2sLpLZOAJf iDFEAik0ylwpUyiJa6+04yofS7ljUyqcg1OlKITiQ8+Yoxw25Az9SdnUCMz1CH/Q6re7P2qrtxk1 a/hHyFLJ5Kt8WPXYeuwx82MX2v4sxQSmWERaOio/YHTRSj0G4LJ/RJMmaLci4cqU2NnaP5vQZQBb qtFxK1zWVDf82DRJXlqA4DkeByjf31L3V3ddDrEOenGwk/t1qur0fnfmJo55w9M8Fm1zqd64SkHL JG7DQM3L1yqWeQNaXVUbq7utmzAieCuzS2g9gEM6keWCuX9mcOQ9K4ZAvtV8KOwmTzV4j+Ih8aNq zB7lHxJK/qgRvS1eb8+6iowYfurd0FFXXdZ1TjBuzxvBCA45LSzqUvF4BcPAunXxtrZKXEOWIfg9 ji6he23TxrZOO3bgHZElVYoc7WiIQAuCP0hnOBMW8NdrztI3sbfJis+iFdjCKk+j42O5TM5wpKOK mgw4LTVd8tQgbTKSJxFzzFbZIqV3iAHxnQVtli8GCMSOOUOjZqrU2OgU0rFa5LH8M7wY9AkOdeEl 5L8hpPzlTe9t7/Co9/yoz/zcmhstrzrRst32blpt+xwF+799vad8iszCzyrPvoJf9orK471havzK h02kD0Lu0N6VBxk4VFSXzjANyAHsGRVv0ZTaX2kWo4mu7ujl+KU6d6mTLljRoesJzzpbKXhcEQ1q xu266/X1rgMaLQ58q6HvPqHxVlM+p9GpBIcgmYOWLXQC4ZtGddxv02VF8JqLRM3C4t5hPCkhLFCn BYTg+OAeeqvnShoHA1/OupI4vGPZHMZh4YHxWaGEYZfA9tRO4unJCL+5lnPT29BoWLskelXb5Y0F /rNjfPvmauGDPUtw/osux/MlhNSIlPJJvEAHkBhEzuvFY/jSRoTs4OFYF/0b1qiU8ddldS0efoYA DoK1iM76KE94lVP5aYNiN1ltvfU19r1kiHWkLF+DjQxX5qC4RXp315G48+KFJeYJMcrJBmx+Dpm3 RPQAptp3324On6Pi2u5L/rZRtwsbrm3HQlbBW+P8NXdIEruiqSCRP5qh3SiQDzdA0ouDiO6O9wJM 7QAYx6Sx5ZleNYvBT1hPvLa5gzR57ytvMPVRjUaG7hl1PjN2n2kyVrEy3pi3ykU5tu6Gny0qUq/s JkjD2y5q32YjVL15oEqW32JjCJNYhJebA7zkH9gAh+KH19j2hi7EWk4hFFH8BxQV31cLoJgAx2Jb v8OuW2jE3QuxBGef0YiSYC3LqG62hUAEv+nDtyz8s2dC/NSegbY1D23z8Sy0H+ehbW0bNBDTPQOq 56rqYuBUXXDKrWlLPrTSkCBn1K2WClfJCXKCMBZ4/Qiki5SqOMIYb8XCgt8MVxaG6lY5qbIuWi+o SuV/6rCqrYmfjONA/JmuzlZ6vQAwLtMiDCUwa33UOit7SbfZbsygogahimF0JYgSoHagIVWw6Ct5 jWqlOks9VRNhzwk6oXogiOF6P4CELxaFtWxnDrdAA/9Vjd+wjr3WlBWw8x7HVfWlVANVur/4IocB d7Hxui3zd7SJvmfwwiIXf/4O9lvdZDx8nZWFm/g4BHLND9mH5OcmBBjwM8H3KktpVi8Z9CFbNrmj 31ZXBTYly07OqDUhzhk4UcA2XQQfnt/aGj/eemcthzDej5QH7y67uVf8OTEhg8sPmRuese1V1ge8 ziQo5w2LVkvLyhSse6KiN4uUH7fL4uxEDvQ8pVQWwYD32YsYoXY36iygaBVj4J4/oFfGIQUveC5K s+7QNaeCzpSpUSwdJZNp4ccCJS2tt/gZAv43/RWH8rpQlKOWhyjDv3jhjZPxTrwLvgt//1p5tbCK yN3R7r+rspwY+OsryBesH0cou9bMtFb5UhxprdYoDtL3kigevmf7d+NbczDNdkO3OU79rGMOOy3q stMyGq1mpfxiENALtYGqtdBb9cqlBNcvV7CNx8AiGrVHyy9yMC30wUcnSJxX/fFlfSjOMC1mlFLR O/bDgG7QC7cGKqXnvvxgimT9409thFLRMhXTqjeOVP10ym9Hr8MfPKBCbNXT5jnpa3DNt1kkHkHN zxaRW0DMj5f+7ID6ewE/O8lITeCB+9rtim5e3WzbvQm70HWQkFg0eDLAzLap8OU0RX2+m+tsmppB J3wtp9/f5xdg4xqwbeUO1IHT2jy5km31LbDyuvZBZzDK5Bn0v2xZAyD9smagKeB/pfW9Z6JdJYIf 5eIhVSSDxj5SOXZwM26G1PJirR9zpZhswYGbAnKlbLhNw6mXCcXG+oMrdNyZjHfKn4Fpv+65Ughc rQSu1sQimbnFsKY6m3D3j4L+u5Pj03Od+McCNANLjaU9rrSrmXXBswAdUm5a1T9QVHqD0cULjidw jx39TgtqXdSpVBWL+Pe/sM6IfmFVpfU2xZg9DXyLvdJLBd0L6io3VuA26+DovWFFHdfLKnVwCPJh Vd5u2dxKs+OtHzzZZSCTU6nnk3+dvzp+E/RODoO3/dOzw+M3uAx3mjlKimDsHqbMcQTBrZXmPD/g FRh7b2jWL1dlDPxCBWcd/u7vz9+fvz//nz7/A0HRP5IAVAAA ------=_NextPart_000_0047_01C2A558.C1C03720-- From martin@v.loewis.de Tue Dec 17 22:11:36 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 17 Dec 2002 23:11:36 +0100 Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal) In-Reply-To: <200212161747.gBGHlYK03859@odiug.zope.com> References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <200212161654.gBGGsRe03537@odiug.zope.com> <15870.2216.65988.322683@gargle.gargle.HOWL> <200212161747.gBGHlYK03859@odiug.zope.com> Message-ID: Guido van Rossum writes: > - Nobody remembers why there is a special case in site.py for lines in > the .pth file that start with 'import', except that Martin von > Loewis once wanted this. It serves the deployment of packages: you can have code that is run from site.py, by just installing a file. This is much simpler than, say, modifying sitecustomize. I specifically wanted this to register new codecs, and, to some degree, it is used for that. The Korean codecs package ships a PTH file that says import korean.aliases korean/aliases.py starts with import encodings.aliases encodings.aliases.aliases.update({ 'cp949': 'korean.cp949', 'ms949': 'korean.cp949', 'uhc': 'korean.cp949', 'euc_kr': 'korean.euc_kr', ... Adding new codecs or new aliases is very similar to adding new items to sys.path. I guess you can also use it to register new importers, if Just's future PEP is accepted. Regards, Martin From just@letterror.com Tue Dec 17 22:13:47 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 17 Dec 2002 23:13:47 +0100 Subject: [Python-Dev] Strange crash in import In-Reply-To: <911129A1-1208-11D7-BC1B-000A27B19B96@oratrix.com> Message-ID: Jack Jansen wrote: > Has any of the code for zip imports been checked in yet? Nope, nothing. I would've loved to take the blame ;-) Just From greg@cosc.canterbury.ac.nz Tue Dec 17 22:14:22 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 18 Dec 2002 11:14:22 +1300 (NZDT) Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212170639.33487.mclay@nist.gov> Message-ID: <200212172214.gBHMEMZ17581@oma.cosc.canterbury.ac.nz> Michael McLay : > Let's go easy on the kiwis. They don't always get their due. It > looks like "Mad" Pearse [1] may have flown before the Wright > bothers, but since he was a perfectionist he did not try to claim > the title. Most people seem to agree that he managed to get off the ground, which was a non-trivial achievement at the time, but it wasn't what you would call a controlled flight. The Wrights were still the first to actually make a plane go where *they* wanted it to go. :-) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From just@letterror.com Tue Dec 17 22:18:20 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 17 Dec 2002 23:18:20 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <004b01c2a550$60608460$0201010a@wiktor> Message-ID: Wiktor Sadowski wrote: > >I can see that keeping the file object open could speed things up, > >but is it a polite & decent thing to do? > > Yes, but it would be nice to not fopen/fclose for each > __importer__.get_data. It also does this for each import (I copied this behavior from the PEP 273 implementation). I experimented with keeping the open file object around, and performance hardly went up. Just PS: your computer thinks it's monday. From tim.one@comcast.net Tue Dec 17 22:40:17 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 17 Dec 2002 17:40:17 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212172142.gBHLg4K17376@oma.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > This is true for + and -, but not * -- you need the > sum of the input precisions to avoid losing information > in that case. Yup. The module docs have the full story. From tim.one@comcast.net Tue Dec 17 23:03:56 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 17 Dec 2002 18:03:56 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212172150.gBHLoKb17403@oma.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > But they're going to have to learn that there's something > other than decimal in order to understand why they should > be using this fixed thingy in the first place, instead > of using floats. I don't think you'll be doing them any > favours by using terminology that obscures the difference. We were talking about the name of a hypothetical builtin function, not about documentation terminology. If you want to presume that users are idiots as well as simply ignorant, then fixed_point_decimal() is what you want. "decimal" leaves out the vital distinction between fixed and floating semantics, which is more important to "good to the last penny" results than the distinction between binary and decimal. A short name simply can't capture all that's relevant (and I don't care). > There's a precedent for this in SQL, which has a type > called DECIMAL. Now, why do you think they called it that > and not something blander like NUMBER? You can't take any useful lessons from SQL here. Most variants of SQL have both DECIMAL and NUMERIC types, usually aliases, sometimes subtly different. From just@letterror.com Tue Dec 17 23:09:35 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 18 Dec 2002 00:09:35 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Message-ID: FWIW, http://www.python.org/sf/652586 has been enhanced with the mod.__importer__ feature. Just From greg@cosc.canterbury.ac.nz Tue Dec 17 23:41:54 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 18 Dec 2002 12:41:54 +1300 (NZDT) Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Message-ID: <200212172341.gBHNfsr18263@oma.cosc.canterbury.ac.nz> Tim Peters : > If you want to presume that users are idiots as well as simply > ignorant, then fixed_point_decimal() is what you want. > "decimal" leaves out the vital distinction between fixed and floating > semantics, which is more important to "good to the last penny" results than > the distinction between binary and decimal. I don't agree with that. The first and foremost reason why a penny-adder will get the right result with this type whereas he wouldn't with a Python float is that it's decimal and not binary. Floatness only becomes an issue when the number of dollars he's adding up becomes large enough. If it doesn't, he could do the same calculation on his floating-point-decimal calculator and get the right answer, too. So, no matter what it's called -- fixed, decimal, or even fixed_decimal, he's not going to be able to guess that he needs to use it, he's going to have to be told. Given that, there's no point in worrying about what naive users may or may not think when deciding on a name for this. And I think a name should be chosen which reflects what it is about this type that makes it useful. I agree that the non-floatness is important, but it seems to me that the decimalness is at least as important, too. You're going to have to face this issue if you ever want to add a floating decimal type as well. If the fixed decimal type is just called "fixed", what are you going to call a floating decimal? You can't just call it "float", because we've already got one of those! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From pobrien@orbtech.com Wed Dec 18 00:23:05 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Tue, 17 Dec 2002 18:23:05 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: References: <200212161609.03192.pobrien@orbtech.com> <200212170912.11638.pobrien@orbtech.com> Message-ID: <200212171823.05716.pobrien@orbtech.com> On Tuesday 17 December 2002 10:28 am, Martin v. Löwis wrote: > Your original message sounded much like a support question. I was > trying to say (perhaps in impolite words, for which I apologize) that > posting support requests is not appropriate for this mailing list. Apology accepted. No hard feelings. Sorry I rubbed you the wrong way as well. That wasn't my intention. > And it is a good thing that you have submitted a bug report to > sourceforge. Thank you for simplifying my example even more, and describing why cPickle is doing what it's doing. Of course, we're still left with the question of whether or not this should be considered a bug. I think it is, but I couldn't tell from your description on SF whether you feel the same way or not. Certainly the difference between pickle and cPickle is disturbing, is it not? > It was the (perceived) attitude "my bug report is more important than > the other 320 open bug reports" that rubbed me the wrong way. I do > appreciate that you took the time to report the bug. I honestly didn't have that attitude. Sorry if my message sounded that way. I just wanted to try to resolve this while it was fresh, that's all. And I do consider it pretty serious that pickle and cPickle are not working identically (in substance, not cosmetic differences). But I don't mean to imply that this bug is more or less important than any other. It just happens to be important to me. ;-) Thank you for your help. It is greatly appreciated. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From tim_one@email.msn.com Wed Dec 18 04:44:43 2002 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 17 Dec 2002 23:44:43 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212172341.gBHNfsr18263@oma.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > ... > You're going to have to face this issue if you ever want to add a > floating decimal type as well. If the fixed decimal type is just > called "fixed", what are you going to call a floating decimal? You > can't just call it "float", because we've already got one of those! So if the fixed decimal type is called "decimal", what are *you* going to call the new floating decimal type? Correct me if I'm missing the obvious, but it seems "float" is just as forbidden to you then. At least if the fixed-point type is called "fixed", it leaves "decimal" available for its successor. But it's all hypothetical so long as FixedPoint/fixed/decimal remains just a class in a library module. From Paul.Moore@atosorigin.com Wed Dec 18 09:37:55 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 18 Dec 2002 09:37:55 -0000 Subject: [Python-Dev] Zipping Zope3 Message-ID: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com> > FWIW, has been > enhanced with the mod.__importer__ feature.=20 ... and I've just added a comment expressing some concerns. I think we need a PEP about now, pinning down the "abstract" import hook interface. What zipimport does risks turning into a de-facto requirement, before we have any other hooks to compare with. My specific issue is with get_data, which is totally unneeded for the import protocol itself (although it does solve real, related issues). I don't think it should be a requirement of the import hook API. Documenting it as an optional interface which hooks may choose to support may be reasonable, though. [I can construct theoretical use cases where it could be hard to support get_data. But I don't like theoretical use cases any more than anyone else - I'm just suggesting that we wait for practical experience before adding anything more to what is currently a nice, clean, and simple API] Paul. PS If Just doesn't want to, I'll volunteer to write such a PEP. But be warned, if I do, I'll probably indulge my instincts and make the API as minimal as I can get away with (I *like* the iterator interface :-)) From mal@lemburg.com Wed Dec 18 09:43:17 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 18 Dec 2002 10:43:17 +0100 Subject: [Python-Dev] Deprecate doctest? References: <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> <200212161624.gBGGO2j01562@odiug.zope.com> <20021216173642.GA10468@panix.com> Message-ID: <3E004335.7090500@lemburg.com> Aahz wrote: > On Mon, Dec 16, 2002, Guido van Rossum wrote: > >>Aahz: >> >>>That's tricky, given that doctest is in the standard library, and that's >>>exactly what doctest does. Are you advocating deprecating doctest? >> >>You gotta be kidding. > > > Of *course* I was, but I couldn't figure out what else MAL meant. Inever talked about doctest; only the fact that error message strings should not be relied on since these tend to change rather often between Python releases. It is also possible that some of these cat get localized (I remember having seen Martin work on something like this in the past). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From just@letterror.com Wed Dec 18 09:58:52 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 18 Dec 2002 10:58:52 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > .... and I've just added a comment expressing some concerns. Likewise ;-) > I think we need a PEP about now, pinning down the "abstract" > import hook interface. What zipimport does risks turning into > a de-facto requirement, before we have any other hooks to > compare with. True. > My specific issue is with get_data, which is totally unneeded > for the import protocol itself (although it does solve real, > related issues). I don't think it should be a requirement of > the import hook API. Documenting it as an optional interface > which hooks may choose to support may be reasonable, though. By no means it will be a *required* part of the protocol, if it'll be part of it at all. I just quickly added it as it's simply a method that's needed for zipimporter objects. I like to look at the big issue the other way around: if we had a Virtual File System protocol it would be a snap to add importing capabilities. What we then need is Zip file support for that VFS, and the actual import hook could be completely independent from Zip file support: it would work for *any* VFS implementation as it's generic. The importer protocol should be in no way hard-wired to any file-system-like properties (and indeed it isn't). > [I can construct theoretical use cases where it could be hard > to support get_data. But I don't like theoretical use cases > any more than anyone else - I'm just suggesting that we wait > for practical experience before adding anything more to what > is currently a nice, clean, and simple API] Don't worry: I couldn't agree with you more. > PS If Just doesn't want to, I'll volunteer to write such a PEP. I've started, but I'm sure I can use some help. I'll send you my draft when I'm stuck ;-) Thanks for the offer; we could surely do this together. > But be warned, if I do, I'll probably indulge my instincts > and make the API as minimal as I can get away with (I *like* > the iterator interface :-)) The iterator interface is a great example for me, too: it's a piece of art ;-) Just From martin@v.loewis.de Wed Dec 18 10:25:16 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 18 Dec 2002 11:25:16 +0100 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212171823.05716.pobrien@orbtech.com> References: <200212161609.03192.pobrien@orbtech.com> <200212170912.11638.pobrien@orbtech.com> <200212171823.05716.pobrien@orbtech.com> Message-ID: [Jim, this is about a perceived cPickle bug where refcount 1 objects lose their identity when on load they appear in multiple dump calls] "Patrick K. O'Brien" writes: > Thank you for simplifying my example even more, and describing why > cPickle is doing what it's doing. Of course, we're still left with the > question of whether or not this should be considered a bug. I think it > is, but I couldn't tell from your description on SF whether you feel > the same way or not. Certainly the difference between pickle and > cPickle is disturbing, is it not? I was happy to discover that this is a *really* rare case, so I refrained from classifying it as a bug or non-bug. For example, you can't trigger it by using cPickle.dump[s]. It is, strictly speaking, a bug, since the documentation says that all objects will be recorded in the memo. The question is whether this is a documentation bug (i.e. the documentation is promising too much) or an implementation bug in cPickle. There clearly is a documentation bug: objects like integers are never put in the memo, and lose identity on unmarshalling. Of course, it is not in the spirit of pickle that their identity is preserved. So any correction would have to include a clarification of the documentation. I'm a bit concerned about performance implications of changing the behaviour: you'll have to put every object into the memo, even though in most cases, you'll never need to lookup any refcount 1 objects. It is not clear to my by what principle cPickle decides to use put or put2. If cPickle is changed, the distinction between put and put2 goes away. A cPickle expert might be able to tell, but I think there are none left (which is also a reason for my initial reaction - you are as much a cPickle expert as anybody else here). To eliminate the performance concerns, it might be feasible to add another flag to pickler objects, indicating whether this was called from pickle.dump[s] or pickle.Pickler, adding the performance cost only to people who use pickler objects. For that to be reasonable, one should be sure that a) the majority of users uses pickle.dump[s], and thus sees the old behaviour, and b) refcount 1 objects are frequent enough to worry about the performance hit I would appreciate if anybody could provide data on either aspect. > I honestly didn't have that attitude. Sorry if my message sounded that > way. I just wanted to try to resolve this while it was fresh, that's > all. And I do consider it pretty serious that pickle and cPickle are > not working identically (in substance, not cosmetic differences). But I > don't mean to imply that this bug is more or less important than any > other. Whether it is serious or not can be only answered once it is understood: As you can see, I still hesitate to classify it as serious. The reason is that cPickle always behaved this way, and nobody ever noticed. This tells me: a) almost nobody uses multiple dump calls to the same pickler, and b) of those who do, almost nobody worries about cross-dump identities, and c) of those who do, almost nobody ever ran into a case where a refcount 1 object occurred in a bad position of two dumps, and d) of those who did, almost nobody noticed, and e) of those who did, only a single person was worried enough to report this to the cPickle maintainers. I'd really like to know what Jim Fulton thinks about all this. Regards, Martin From aahz@pythoncraft.com Wed Dec 18 12:36:39 2002 From: aahz@pythoncraft.com (Aahz) Date: Wed, 18 Dec 2002 07:36:39 -0500 Subject: [Python-Dev] Deprecate doctest? In-Reply-To: <3E004335.7090500@lemburg.com> References: <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> <200212161624.gBGGO2j01562@odiug.zope.com> <20021216173642.GA10468@panix.com> <3E004335.7090500@lemburg.com> Message-ID: <20021218123639.GA28416@panix.com> On Wed, Dec 18, 2002, M.-A. Lemburg wrote: > Aahz wrote: >>On Mon, Dec 16, 2002, Guido van Rossum wrote: >>>Aahz: >>>> >>>>That's tricky, given that doctest is in the standard library, and that's >>>>exactly what doctest does. Are you advocating deprecating doctest? >>> >>>You gotta be kidding. >> >>Of *course* I was, but I couldn't figure out what else MAL meant. > > Inever talked about doctest; only the fact that error message > strings should not be relied on since these tend to change > rather often between Python releases. It is also possible that > some of these cat get localized (I remember having seen Martin work > on something like this in the past). I'd say a better statement is "rely only on error strings that you control; don't rely on Python's internal error strings". -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From aahz@pythoncraft.com Wed Dec 18 13:04:11 2002 From: aahz@pythoncraft.com (Aahz) Date: Wed, 18 Dec 2002 08:04:11 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212170639.33487.mclay@nist.gov> References: <200212170639.33487.mclay@nist.gov> Message-ID: <20021218130411.GA1503@panix.com> On Tue, Dec 17, 2002, Michael McLay wrote: > On Tuesday 17 December 2002 12:34 am, Tim Peters wrote: >> [Michael McLay] >>> >>> To bad the decNumber library from IBM is not open source. >> >> A Java implementation is open: >> >> http://www2.hursley.ibm.com/decimalj/ > > That's great for jython, but it will require more work for the C Python > implementation. Much of a first draft of a Python implementation is available at my Web site. Tim's jab at me ("fifteen years") is because it really needs to be rewritten from scratch and refactored, and I haven't gotten the mental traction to do it. Currently it does all the basics other than division (which is when I realized that it needed rewriting). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From guido@python.org Wed Dec 18 13:26:21 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 18 Dec 2002 08:26:21 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Your message of "Tue, 17 Dec 2002 23:44:43 EST." References: Message-ID: <200212181326.gBIDQLS18363@pcp02138704pcs.reston01.va.comcast.net> I've thought about this a bit, and I find fixedpoint or fixed or fixpoint all much better names than decimal. And fixed_point_decimal (or variations thereof) is simply too long. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Wed Dec 18 13:30:14 2002 From: mwh@python.net (Michael Hudson) Date: 18 Dec 2002 13:30:14 +0000 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src setup.py,1.126,1.127 In-Reply-To: Andrew Kuchling's message of "Tue, 17 Dec 2002 12:17:57 -0500" References: <20021217171757.GA16244@ute.mems-exchange.org> Message-ID: <2mu1hbbgk9.fsf@starship.python.net> Andrew Kuchling writes: > On Tue, Dec 17, 2002 at 08:47:20AM -0800, mwh@users.sourceforge.net wrote: > >+ # Do exactly what install_lib does but make sure correct access modes get > >+ # set on installed directories and files. All installed files with get > >+ # mode 644 unless they are a shared library in which case they will get > >+ # mode 755. All installed directories will get mode 755. > > If install_lib is getting the mode wrong, shouldn't this fix be applied to > install_lib instead of just the setup.py? Probably. Do you have the time/inclination to work out any possible backwards compatibility interactions, where it needs to go, etc.? If you do, be my guest. I don't, and neither did the guy who wrote the patch. Cheers, M. -- Imagine if every Thursday your shoes exploded if you tied them the usual way. This happens to us all the time with computers, and nobody thinks of complaining. -- Jeff Raskin From guido@python.org Wed Dec 18 13:33:23 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 18 Dec 2002 08:33:23 -0500 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Your message of "Wed, 18 Dec 2002 09:37:55 GMT." <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com> Message-ID: <200212181333.gBIDXN518460@pcp02138704pcs.reston01.va.comcast.net> > My specific issue is with get_data, which is totally unneeded > for the import protocol itself (although it does solve real, > related issues). I don't think it should be a requirement of > the import hook API. Documenting it as an optional interface > which hooks may choose to support may be reasonable, though. I like get_data(). The caller should always be prepared for it to return an error (how does it do that? raise what? or return None? that should be specified!). An importer that can't do this should implement a version that always reports an error. This is much easier on the client than calling hasattr(importer, 'get_data') first or catching AttributeError. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Dec 18 13:34:18 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 18 Dec 2002 08:34:18 -0500 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: Your message of "18 Dec 2002 11:25:16 +0100." References: <200212161609.03192.pobrien@orbtech.com> <200212170912.11638.pobrien@orbtech.com> <200212171823.05716.pobrien@orbtech.com> Message-ID: <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net> > It is, strictly speaking, a bug, since the documentation says that all > objects will be recorded in the memo. The question is whether this is > a documentation bug (i.e. the documentation is promising too much) or > an implementation bug in cPickle. Doc bug. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Wed Dec 18 13:48:15 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 18 Dec 2002 14:48:15 +0100 Subject: [Python-Dev] Zipping Zope3 References: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com> <200212181333.gBIDXN518460@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <02c501c2a69c$1e0135e0$0900a8c0@spiff> Guido van Rossum wrote: > I like get_data(). The caller should always be prepared for it to > return an error (how does it do that? raise what? or return None? that > should be specified!). An importer that can't do this should > implement a version that always reports an error. This is much easier > on the client than calling hasattr(importer, 'get_data') first or > catching AttributeError. IOError makes a certain kind of sense. on the other hand, I'm not sure that requiring the use of hasattr/try- except is that much of a pain, in practice. ::: And if we're making get_data part of the protocol, can we please have an optional get_module_names/list_modules/listdir method too? Quoting myself from an earlier post (update the examples as necessary): as I might have mentioned before, it would be really nice if custom importers supported optional "load_data" and "get_module_names" hooks. the former can be used by code that looks for data files on the path (Zope, PIL, most likely hundreds of other tools/programs): for path in sys.path: if hasattr(path, "load_data"): try: data =3D path.load_data(name) except IOError: pass else: return data # got it if isinstance(path, types.StringTypes) and = os.path.isdir(path): try: data =3D open(os.path.join(path, file), "rb").read() except IOError: pass else: return data # ... the latter can be used by code that looks for plugins using a name pattern (PIL's *ImagePlugins, for example): for path in sys.path: if hasattr(path, "get_module_names"): names =3D path.get_module_names() elif isinstance(path, types.StringTypes) and = os.path.isdir(path): names =3D os.listdir(path) else: continue for name in names: if name.endswith("ImagePlugin"): __import__(name) From pedronis@bluewin.ch Wed Dec 18 13:55:42 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 18 Dec 2002 14:55:42 +0100 Subject: [Python-Dev] Zipping Zope3 References: Message-ID: <004f01c2a69d$27f8faa0$6d94fea9@newmexico> From: "Just van Rossum" > > .... and I've just added a comment expressing some concerns. > > Likewise ;-) > > > I think we need a PEP about now, pinning down the "abstract" > > import hook interface. What zipimport does risks turning into > > a de-facto requirement, before we have any other hooks to > > compare with. > > True. > I cannot do this for the first 2.3 alpha, but between it and the beta, I will try to implement such a PEP for Jython in order not to underestimate some details. One thing is that in Jython import logic we do not have the find_module/load_module logic... regards. From just@letterror.com Wed Dec 18 14:06:40 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 18 Dec 2002 15:06:40 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <004f01c2a69d$27f8faa0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > I cannot do this for the first 2.3 alpha, but between it and the > beta, I will try to implement such a PEP for Jython in order not to > underestimate some details. One thing is that in Jython import logic > we do not have the find_module/load_module logic... That's not neccesarily a showstopper for the importer protocol: during an actual import Python calls find_module and load_module quickly after one another, so this can be captured in an import_module function which simply does: def import_module(fullname): loader = importer.find_module(fullname) if loader: return loader.load_module(fullname) return None # raise ImportError, whatever Paul Moore and I are now working on a PEP and we're making pretty good progress. If you (or anyone for that matter) want to be involved (passively or actively) before it gets posted, let me know. Just From Paul.Moore@atosorigin.com Wed Dec 18 14:07:36 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 18 Dec 2002 14:07:36 -0000 Subject: [Python-Dev] Zipping Zope3 Message-ID: <16E1010E4581B049ABC51D4975CEDB8861998C@UKDCX001.uk.int.atosorigin.com> From: Guido van Rossum [mailto:guido@python.org] >> My specific issue is with get_data, which is totally unneeded >> for the import protocol itself (although it does solve real, >> related issues). I don't think it should be a requirement of >> the import hook API. Documenting it as an optional interface >> which hooks may choose to support may be reasonable, though. > I like get_data(). The caller should always be prepared for > it to return an error (how does it do that? raise what? or > return None? that should be specified!). An importer that > can't do this should implement a version that always reports > an error. This is much easier on the client than calling > hasattr(importer, 'get_data') first or catching AttributeError. I don't have a big problem with get_data(). But currently, it's badly under-specified. You mention the question of how it returns errors (not just "can't do this" - what about "file not found"?). There's also the question of the exact format of the "filename". Should it be like a module name (dotted)? This has the advantage of matching how the rest of the importer logic works, but the disadvantage of being unnatural to the user. Should it be a pseudo-path format? This has the disadvantage of being platform specific (the zipimporter already jumps through some hoops to convert os.sep/os.altsep into forward slashes (which is what the zip format uses), and other importers may have even more work to do). Or something else? Also is it a method of the object that implements load_module, or the one that implements find_module? I could argue either. I can code things either way. But I don't have enough "real world" experience to say which is best in practice. Paul. From Paul.Moore@atosorigin.com Wed Dec 18 15:05:02 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 18 Dec 2002 15:05:02 -0000 Subject: [Python-Dev] Zipping Zope3 Message-ID: <16E1010E4581B049ABC51D4975CEDB8861998D@UKDCX001.uk.int.atosorigin.com> Fredrik Lundh said: > And if we're making get_data part of the protocol, can we > please have an optional get_module_names/list_modules/listdir > method too? This is why I'm concerned about get_data. It isn't necessarily enough - we've already had people ask about get_module_names and stat. Who knows what else could be useful? I suspect that some things simply aren't going to be noticed until the new import hooks get actually used "in the wild". At the moment, this issue is in the "Open Issues" section of the (not yet released) PEP. Paul From pedronis@bluewin.ch Wed Dec 18 15:11:34 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 18 Dec 2002 16:11:34 +0100 Subject: [Python-Dev] Zipping Zope3 References: Message-ID: <010301c2a6a7$c11dcc60$6d94fea9@newmexico> From: "Just van Rossum" > Paul Moore and I are now working on a PEP and we're making pretty good > progress. If you (or anyone for that matter) want to be involved > (passively or actively) before it gets posted, let me know. Question: how does the patch respectively PEP features interact with imp module? From pobrien@orbtech.com Wed Dec 18 15:41:30 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 18 Dec 2002 09:41:30 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net> References: <200212161609.03192.pobrien@orbtech.com> <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212180941.30127.pobrien@orbtech.com> On Wednesday 18 December 2002 07:34 am, Guido van Rossum wrote: > > It is, strictly speaking, a bug, since the documentation says that > > all objects will be recorded in the memo. The question is whether > > this is a documentation bug (i.e. the documentation is promising > > too much) or an implementation bug in cPickle. > > Doc bug. Not sure I completely agree, but I'm not trying to pick a fight, either. I just wish the following bit from the docs was true, whether pickle or cPickle were used, whether the object was first pickled inside some container and later pickled on its own, whether the instance was a new-style class or an old-style class, etc.: It is possible to make multiple calls to the dump() method of the same Pickler instance. These must then be matched to the same number of calls to the load() method of the corresponding Unpickler instance. If the same object is pickled by multiple dump() calls, the load() will all yield references to the same object. The fact that I'm no longer sure whether I'll get back a reference to a previously pickle instance or not calls into question whether pickling can be the foundation for a reliable Python persistence system. I can't help but express my disappointment. Was I really expecting too much from pickle/cPickle? In any case, I've submitted the following documentation bug report on SF: [ 655802 ] cPickle not always same as pickle -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From just@letterror.com Wed Dec 18 15:55:13 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 18 Dec 2002 16:55:13 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <010301c2a6a7$c11dcc60$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > Question: how does the patch respectively PEP features interact with > imp module? Sneak preview from the PEP (glad I already wrote that part ;-): """ imp module integration The new import hooks are not easily integrated in the existing imp.find_module() and imp.load_module() calls. It's questionable whether it's possible at all without breaking code; it is better to simply add a new function to the imp module. The meaning of the existing imp.find_module() and imp.load_module() calls changes from: "they expose the builtin import mechanism" to "they expose the basic *unhooked* builtin import mechanism". They simply won't invoke any import hooks. A new imp module function is proposed under the name "find_module2", with is used like the following pattern: loader = imp.find_module2(fullname, path) if loader is not None: loader.load_module(fullname) In the case of a "basic" import, one the imp.find_module() function would handle, the loader object would be a wrapper for the current output of imp.find_module(), and loader.load_module() would call imp.load_module() with that output. Note that this wrapper is currently not yet implemented, although a Python prototype exists in the test_importhooks.py script (the ImpWrapper class). """ It needs to be fleshed out a bit more, but this is the basic idea. Just From guido@python.org Wed Dec 18 16:00:42 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 18 Dec 2002 11:00:42 -0500 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: Your message of "Wed, 18 Dec 2002 09:41:30 CST." <200212180941.30127.pobrien@orbtech.com> References: <200212161609.03192.pobrien@orbtech.com> <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net> <200212180941.30127.pobrien@orbtech.com> Message-ID: <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> > Not sure I completely agree, but I'm not trying to pick a fight, either. > I just wish the following bit from the docs was true, whether pickle or > cPickle were used, whether the object was first pickled inside some > container and later pickled on its own, whether the instance was a > new-style class or an old-style class, etc.: > > > It is possible to make multiple calls to the dump() method of the same > Pickler instance. These must then be matched to the same number of > calls to the load() method of the corresponding Unpickler instance. If > the same object is pickled by multiple dump() calls, the load() will > all yield references to the same object. > > > The fact that I'm no longer sure whether I'll get back a reference to a > previously pickle instance or not calls into question whether pickling > can be the foundation for a reliable Python persistence system. I can't > help but express my disappointment. Was I really expecting too much > from pickle/cPickle? Probably. I have to admit that the "multiple calls to dump()" feature is not often used; maybe it should be deprecated so we don't make promises we can't fulfill. E.g. Zope never uses this. > In any case, I've submitted the following documentation bug report on > SF: > > [ 655802 ] cPickle not always same as pickle You could help by submitting a doc patch that satisfies your desire for more clear and honest docs. If you don't know LaTeX, that doesn't matter; just pretend it's plain text and somebody will fix up the markup, as long as you provide the new words. --Guido van Rossum (home page: http://www.python.org/~guido/) From pobrien@orbtech.com Wed Dec 18 16:22:52 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 18 Dec 2002 10:22:52 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> References: <200212161609.03192.pobrien@orbtech.com> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212181022.52849.pobrien@orbtech.com> On Wednesday 18 December 2002 10:00 am, Guido van Rossum wrote: > I have to admit that the "multiple calls to dump()" feature > is not often used; maybe it should be deprecated so we don't make > promises we can't fulfill. E.g. Zope never uses this. Since it works 99% of the time, I'd rather fix it and make it work 100% of the time. If it can't be fixed, the promises made in the docs should be tempered. See my next suggestion first. Here is another thought that might outway the performance concerns: for whatever reason, all instances of new-style classes (even with refcount == 1, or perhaps because they don't have a refcount == 1) *are* getting put in the cPickle memo. If the goal is to move from old-style to new-style, we're already taking a performace hit with new-style class instances. So why not pay the price with old-style as well and put them in the cPickle memo? Then the discrepancy between pickle and cPickle goes away, and this "multiple dump" feature works as advertised. Or create a second version of cPickle that does work exactly the same as pickle. It will still have a performance benefit over pickle and it won't be any slower than the regular cPickle if all your instances are of new-style classes. Just thinking out loud. > > In any case, I've submitted the following documentation bug report > > on SF: > > > > [ 655802 ] cPickle not always same as pickle > > You could help by submitting a doc patch that satisfies your > desire for more clear and honest docs. If you don't know LaTeX, that > doesn't matter; just pretend it's plain text and somebody will fix up > the markup, as long as you provide the new words. I'll add this to my todo list, assuming you don't like my suggestions above. My documentation bug report contained annotations that suggest some wording changes already, but I'm sure Fred would prefer patches with specific revisions. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From jim@zope.com Wed Dec 18 18:36:49 2002 From: jim@zope.com (Jim Fulton) Date: Wed, 18 Dec 2002 13:36:49 -0500 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent References: <200212161609.03192.pobrien@orbtech.com> <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E00C041.7020901@zope.com> Guido van Rossum wrote: >>Not sure I completely agree, but I'm not trying to pick a fight, either. >>I just wish the following bit from the docs was true, whether pickle or >>cPickle were used, whether the object was first pickled inside some >>container and later pickled on its own, whether the instance was a >>new-style class or an old-style class, etc.: >> >> >>It is possible to make multiple calls to the dump() method of the same >>Pickler instance. These must then be matched to the same number of >>calls to the load() method of the corresponding Unpickler instance. If >>the same object is pickled by multiple dump() calls, the load() will >>all yield references to the same object. >> >> >>The fact that I'm no longer sure whether I'll get back a reference to a >>previously pickle instance or not calls into question whether pickling >>can be the foundation for a reliable Python persistence system. I can't >>help but express my disappointment. Was I really expecting too much >>from pickle/cPickle? >> > > Probably. I have to admit that the "multiple calls to dump()" feature > is not often used; maybe it should be deprecated so we don't make > promises we can't fulfill. E.g. Zope never uses this. Zope does use this. Zope doesn't pickle the same object multiple times in the separate load calls, however. This may be a matter of luck. I think Patrick raises a reasonable issue, although it would require slowing things down a bit to fix it. Patrick, why do you feel you need this for a reliable persistence system? ZODB doesn't need this. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pobrien@orbtech.com Wed Dec 18 19:28:07 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 18 Dec 2002 13:28:07 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <3E00C041.7020901@zope.com> References: <200212161609.03192.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> <3E00C041.7020901@zope.com> Message-ID: <200212181328.07455.pobrien@orbtech.com> On Wednesday 18 December 2002 12:36 pm, Jim Fulton wrote: > I think Patrick raises a reasonable issue, although it would require > slowing things down a bit to fix it. > > Patrick, why do you feel you need this for a reliable persistence > system? ZODB doesn't need this. I don't necessarily *need* this, but it would be nice to have. Otherwise I have to remember to always work around it. Here is a quick rundown on my particular use case: I'm working on a very simple persistence framework, called PyPerSyst (http://sourceforge.net/projects/pypersyst), whose goal is to be able to persist any picklable object graph, support ACID requirements, and be able to safely recover from crashes. The difference between PyPerSyst and ZODB is that PyPerSyst keeps everything in RAM (which is how it keeps things simple). The basic approach is to start by pickling an object graph (which we call the System). Changes to the System are then made by sending a Transaction class instance to the persistence Engine (sort of a mediator thing). The Engine pickles the Transaction to a transaction log, then executes the transaction against the System. If there is a crash, the System is recovered from the pickle file and then all Transactions in the log are loaded and re-executed against the System. The actual pickling strategy is delegated by the Engine to one of several Storage classes. One of the storages uses a single file (look for `singlefile.py`) for the System snapshot as well as all Transactions. This storage also allows Transactions to contain direct references to objects in the System. In order for this to work properly, the object references in the Transactions must resolve to the same objects in the System when Transactions are unpickled during recovery. In our first sample app (a Banking app, no less) this didn't work properly because of the "bug" in cPickle (and recent bank account transactions would effectively disappear, even though they appeared to be recovered properly). Because this is a framework that is supposed to work with any System and any Transaction defined by the end-user, I can't easily guarantee that they won't be referencing class instances that cPickle won't want to memoize. So I either need to disallow this particular storage strategy, or have it use only pickle (though I'd like the speed of cPickle), or make sure cPickle works the same as pickle. Did any of this make sense? Did I answer your question? [The other thing I don't know is if this "bug" that we are discussing is the only example of its kind, or whether there are more to be discovered. But we can cross that bridge when we get there.] -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From guido@python.org Wed Dec 18 19:39:32 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 18 Dec 2002 14:39:32 -0500 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: Your message of "Wed, 18 Dec 2002 10:22:52 CST." <200212181022.52849.pobrien@orbtech.com> References: <200212161609.03192.pobrien@orbtech.com> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> <200212181022.52849.pobrien@orbtech.com> Message-ID: <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net> > Here is another thought that might outway the performance concerns: for > whatever reason, all instances of new-style classes (even with refcount > == 1, or perhaps because they don't have a refcount == 1) *are* getting > put in the cPickle memo. Pickling of new-style classes is currently done using a Python helper. I am still hoping that I'll be able to dig myself out of the pile of Zope work that I'm currently under, and then I'd like to work on making pickling of new-style classes a lot more efficient. So please don't use this argument to cripple pickle any further. --Guido van Rossum (home page: http://www.python.org/~guido/) From pobrien@orbtech.com Wed Dec 18 19:44:43 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 18 Dec 2002 13:44:43 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net> References: <200212161609.03192.pobrien@orbtech.com> <200212181022.52849.pobrien@orbtech.com> <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212181344.43536.pobrien@orbtech.com> On Wednesday 18 December 2002 01:39 pm, Guido van Rossum wrote: > Pickling of new-style classes is currently done using a Python > helper. I am still hoping that I'll be able to dig myself out of the > pile of Zope work that I'm currently under, and then I'd like to work > on making pickling of new-style classes a lot more efficient. So > please don't use this argument to cripple pickle any further. Okay. (ROFL) :-) -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From sholden@holdenweb.com Wed Dec 18 20:17:51 2002 From: sholden@holdenweb.com (Steve Holden) Date: Wed, 18 Dec 2002 15:17:51 -0500 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent References: <200212161609.03192.pobrien@orbtech.com> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> <200212181022.52849.pobrien@orbtech.com> <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <014b01c2a6d2$8ba255d0$6b01a8c0@holdenweb.com> > > Here is another thought that might outway the performance concerns: for > > whatever reason, all instances of new-style classes (even with refcount > > == 1, or perhaps because they don't have a refcount == 1) *are* getting > > put in the cPickle memo. > > Pickling of new-style classes is currently done using a Python helper. > I am still hoping that I'll be able to dig myself out of the pile of > Zope work that I'm currently under, and then I'd like to work on > making pickling of new-style classes a lot more efficient. So please > don't use this argument to cripple pickle any further. > I'm not a crippled pickle, I'm a crippled pickle's son and I'll be finished crippling pickles when the pickle crippling's done. Sorry. Been working too hard ... ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ Bring your musical instrument to PyCon! http://www.python.org/pycon/ ----------------------------------------------------------------------- From greg@cosc.canterbury.ac.nz Wed Dec 18 20:53:15 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 19 Dec 2002 09:53:15 +1300 (NZDT) Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: Message-ID: <200212182053.gBIKrFb27247@oma.cosc.canterbury.ac.nz> Tim Peters : > So if the fixed decimal type is called "decimal", what are *you* going to > call the new floating decimal type? I don't know, but whatever it is it somehow needs to connote both decimalness and floatness. Maybe the module should be called decimal, then we could have decimal.fixed and decimal.float. Or maybe decimal could be a built-in abstract type with class attributes fixed and float that are subclasses of it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From Jack.Jansen@oratrix.com Wed Dec 18 21:50:02 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Wed, 18 Dec 2002 22:50:02 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Message-ID: On woensdag, dec 18, 2002, at 16:55 Europe/Amsterdam, Just van Rossum wrote: > The new import hooks are not easily integrated in the existing > imp.find_module() and imp.load_module() calls. It's questionable > whether > it's possible at all without breaking code; it is better to simply add > a > new function to the imp module. The meaning of the existing > imp.find_module() and imp.load_module() calls changes from: "they > expose > the builtin import mechanism" to "they expose the basic *unhooked* > builtin import mechanism". They simply won't invoke any import hooks. A > new imp module function is proposed under the name "find_module2", with > is used like the following pattern: > > loader = imp.find_module2(fullname, path) > if loader is not None: > loader.load_module(fullname) If we want to use this for freeze-like functionality then the loader will also need a method (or methods) that at the very least return (for modules for which it makes sense) either the source code for the module or a file-like object that can be used to read the source code. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From greg@cosc.canterbury.ac.nz Wed Dec 18 21:45:00 2002 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 19 Dec 2002 10:45:00 +1300 (NZDT) Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212181022.52849.pobrien@orbtech.com> Message-ID: <200212182145.gBILj0E27411@oma.cosc.canterbury.ac.nz> "Patrick K. O'Brien" : > Or create a second version of cPickle that does work exactly the same as > pickle. Or make it an option or alternative entry point to the existing cPickle. The two cPickles would be almost exactly the same code-wise, wouldn't they? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From just@letterror.com Wed Dec 18 22:34:20 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 18 Dec 2002 23:34:20 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Message-ID: Jack Jansen wrote: > > loader = imp.find_module2(fullname, path) > > if loader is not None: > > loader.load_module(fullname) > > If we want to use this for freeze-like functionality then the loader > will also need a method (or methods) that at the very least return > (for modules for which it makes sense) either the source code for the > module or a file-like object that can be used to read the source code. Features schmeatures! I would agree, but Paul and I have decided to keep the importer protocol real simple, at least for now. Extensions to the importer protocol are definitely needed, but we'd like to concentrate on the basics first. We (as in: all of us) simply need more time to come up with a decent, coherent and simple set of (optional) extensions to the protocol. The zipimporter object now has a get_data(filename) method with which you can do what you want (in a less convenient way), and Paul questions even that addition, as we might not be able to go back once it's in the core. It could even be the subject of a PEP of its own ;-) Also: *anything* that makes zipimporter closer to a file system is going at it from the wrong direction. We *first* need a virtual file system, then an import hook that can import from it, and *then* a zip back end... Just From pobrien@orbtech.com Wed Dec 18 23:25:29 2002 From: pobrien@orbtech.com (Patrick K. O'Brien) Date: Wed, 18 Dec 2002 17:25:29 -0600 Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent In-Reply-To: <200212182145.gBILj0E27411@oma.cosc.canterbury.ac.nz> References: <200212182145.gBILj0E27411@oma.cosc.canterbury.ac.nz> Message-ID: <200212181725.29937.pobrien@orbtech.com> On Wednesday 18 December 2002 03:45 pm, Greg Ewing wrote: > "Patrick K. O'Brien" : > > Or create a second version of cPickle that does work exactly the > > same as pickle. > > Or make it an option or alternative entry point to the > existing cPickle. The two cPickles would be almost > exactly the same code-wise, wouldn't they? I agree. I was speaking in general terms. The implementation should avoid duplicating code. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." ----------------------------------------------- From neal@metaslash.com Thu Dec 19 00:44:45 2002 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 18 Dec 2002 19:44:45 -0500 Subject: [Python-Dev] test_datetime fails on alpha Message-ID: <20021219004445.GD12063@epoch.metaslash.com> I used the alpha in the SF compile to verify the bug. The problem was originally reported on the snake-farm from the test suite. nnorwitz@usf-cf-alpha-linux-1:~/python/dist/src$ ./python Python 2.3a0 (#15, Dec 18 2002, 15:52:53) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import timedelta >>> timedelta(1, 7, 31) datetime.timedelta(1, 7, 31) >>> _.days, _.seconds, _.microseconds (30064771073, 133143986183, 31) Neal From tim.one@comcast.net Thu Dec 19 01:04:40 2002 From: tim.one@comcast.net (Tim Peters) Date: Wed, 18 Dec 2002 20:04:40 -0500 Subject: [Python-Dev] test_datetime fails on alpha In-Reply-To: <20021219004445.GD12063@epoch.metaslash.com> Message-ID: [Neal Norwitz] > I used the alpha in the SF compile to verify the bug. The problem was > originally reported on the snake-farm from the test suite. > > nnorwitz@usf-cf-alpha-linux-1:~/python/dist/src$ ./python > Python 2.3a0 (#15, Dec 18 2002, 15:52:53) > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from datetime import timedelta > >>> timedelta(1, 7, 31) > datetime.timedelta(1, 7, 31) > >>> _.days, _.seconds, _.microseconds > (30064771073, 133143986183, 31) It's really no help. The *first* thing to do is to build Python with optimzation disabled. Try a debug build, if possible, and step thru delta_new(). Luckily, that's the hairiest code in the whole module . Putting the "wrong parts" in hex, that result is (0x700000001L, 0x1F00000007L, 31) which suggests nothing to me. I don't see any 32-bit assumptions in the code here, and thanks to two previous lives on 64-bit boxes they usually punch me in the eyeball if they exist. From jepler@unpythonic.net Thu Dec 19 00:30:34 2002 From: jepler@unpythonic.net (Jeff Epler) Date: Wed, 18 Dec 2002 18:30:34 -0600 Subject: [Python-Dev] test_datetime fails on alpha In-Reply-To: References: <20021219004445.GD12063@epoch.metaslash.com> Message-ID: <20021218183022.A987@unpythonic.net> On Wed, Dec 18, 2002 at 08:04:40PM -0500, Tim Peters wrote: > [Neal Norwitz] > > I used the alpha in the SF compile to verify the bug. The problem was > > originally reported on the snake-farm from the test suite. > > > > nnorwitz@usf-cf-alpha-linux-1:~/python/dist/src$ ./python > > Python 2.3a0 (#15, Dec 18 2002, 15:52:53) > > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> from datetime import timedelta > > >>> timedelta(1, 7, 31) > > datetime.timedelta(1, 7, 31) > > >>> _.days, _.seconds, _.microseconds > > (30064771073, 133143986183, 31) > > It's really no help. The *first* thing to do is to build Python with > optimzation disabled. Try a debug build, if possible, and step thru > delta_new(). Luckily, that's the hairiest code in the whole module . > > Putting the "wrong parts" in hex, that result is > > (0x700000001L, 0x1F00000007L, 31) ^ ^ (0x1F) ^^ ^^ > which suggests nothing to me. something is sure suggested to me by those numbers. It's as though successive members are 4 bytes apart, but they're being treated as 8 bytes wide, or something. how the code manages it is beyond me, though. (I don't even see a "datetime" module in my CVS tree) Jeff From neal@metaslash.com Thu Dec 19 02:34:22 2002 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 18 Dec 2002 21:34:22 -0500 Subject: [Python-Dev] test_datetime fails on alpha In-Reply-To: <20021218183022.A987@unpythonic.net> References: <20021219004445.GD12063@epoch.metaslash.com> <20021218183022.A987@unpythonic.net> Message-ID: <20021219023422.GE12063@epoch.metaslash.com> On Wed, Dec 18, 2002 at 06:30:34PM -0600, Jeff Epler wrote: > On Wed, Dec 18, 2002 at 08:04:40PM -0500, Tim Peters wrote: > > [Neal Norwitz] > > > I used the alpha in the SF compile to verify the bug. The problem was > > > originally reported on the snake-farm from the test suite. > > > > > > >>> from datetime import timedelta > > > >>> timedelta(1, 7, 31) > > > datetime.timedelta(1, 7, 31) > > > >>> _.days, _.seconds, _.microseconds > > > (30064771073, 133143986183, 31) > > > > It's really no help. The *first* thing to do is to build Python with > > optimzation disabled. Try a debug build, if possible, and step thru > > delta_new(). Luckily, that's the hairiest code in the whole module . > > > > Putting the "wrong parts" in hex, that result is > > > > (0x700000001L, 0x1F00000007L, 31) > ^ ^ (0x1F) > ^^ ^^ > > which suggests nothing to me. > > something is sure suggested to me by those numbers. It's as though > successive members are 4 bytes apart, but they're being treated as 8 bytes > wide, or something. That kept bugging me. I just checked in the fix. The problem was the accessors members for days, seconds, and microseconds were T_LONGs instead of T_INTs. Test passes on alpha & x86 for me. Neal From guido@python.org Thu Dec 19 03:54:34 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 18 Dec 2002 22:54:34 -0500 Subject: [Python-Dev] deprecating APIs In-Reply-To: Your message of "Mon, 16 Dec 2002 20:49:35 EST." <20021217014935.GQ16807@epoch.metaslash.com> References: <20021217014935.GQ16807@epoch.metaslash.com> Message-ID: <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net> > Are we still planning to provide a mechanism to declare APIs, etc as > deprecated? I think this is a neat idea. > I have this code added to pyport.h which works for gcc: > > /* Py_DEPRECATED(version) > * Declare a macro or function deprecated. > * Usage: > * extern int old_var Py_DEPRECATED(2.3); > * typedef int T1 Py_DEPRECATED(2.4); > * extern int x() Py_DEPRECATED(2.5); > */ > #if defined(__GNUC__) && (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) > #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__)) > #else > #define Py_DEPRECATED(VERSION_UNUSED) > #endif > > The version number is not necessary. It is only for documentation. > I don't know how to support other compilers. What does this do under GCC? I suppose it issues a warning when you use the deprecated API? > We could add Py_USE_DEPRECATED and/or Py_HIDE_DEPRECATED for > providing/hiding access to deprecated features. > > #ifdef Py_USE_DEPRECATED > #define Py_SOME_DEPRECATED_MACRO > #endif > > or > > #ifndef Py_HIDE_DEPRECATED > #define Py_SOME_DEPRECATED_MACRO > #endif > > Unfortunately, the Py_DEPRECATED macro doesn't work with macros, so we > would have to use Py_USE_DEPRECATED/Py_HIDE_DEPRECATED around > deprecated macros. That's OK. Just the fact that when reading the .h files one sees the DEPRECATED keyword should be a big hint... --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Thu Dec 19 05:58:37 2002 From: tim.one@comcast.net (Tim Peters) Date: Thu, 19 Dec 2002 00:58:37 -0500 Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python In-Reply-To: <200212170639.33487.mclay@nist.gov> Message-ID: [Michael McLay] > ... > be a reasonable start. I am concerned about one thing. Will the > fixed point implementation you have implemented be backwards > compatible with an implementation of Cowlishaw's decimal numbers? Nope. Any scheme that can lose information is, in general, incompatible with any other scheme that can lose information, unless they lose information in identical ways in all cases. Since Cowlishaw's is a floating scheme, and FixedPoint is a fixed scheme, they'll get different results in some information-losing cases. Examples aren't hard to find. Here's one: what's 13% of 100006.73? The true result is exactly 10000673 * 13 / 10000 = 13000.8749. Do that via fixed-points with 2 digits after the decimal point, and it's rounded down to 13000.87 (which is the best possible result if you get to keep no more than two digits after the decimal point). Use Cowlishaw's scheme with 8 digits of precision (which counts all digits -- the decimal point floats) instead, and the trailing 9 has to be tossed, and so rounds the last retained digit from 4 up to 5: 13000.875. You've got 8 digits now, but a worse result if pennies are what you need: if you round *that* to the nearest penny, to-nearest/even ("banker's rounding") says "OK, I have to toss the 5, and 7 is odd, so round it up the even 8", leaving 13000.88. The end results aren't the same, and I recall now that this is why FixedPoint was designed to lose info on multiply: when you're computing percentages to the nearest penny (be it sales tax or rebates), you can't get the best possible result in all cases if you use a scheme that rounds twice. A fixed-point scheme can get away with rounding just once (& very naturally), but a floating scheme can have "too many digits", and therefore force the user to do a second rounding step to cut the result back to the number of fractional digits they wanted. Double-rounding can't be as good as single-rounding in all cases, unless the floating scheme uses much more precision than the fixed scheme (there are tricky theorems that kick in then -- if you have "enough" excess precision, it's provable that double-rounding is always the same as single-rounding -- but not one user in a million can be expected to know this, and even if they did the application is too difficult since it effectively requires proving that you've always got enough extra precision relative to your app's specific numbers). BTW, this is why FixedPoint's "good to the last penny" has more to do with the fixed-point business than with the decimal business. Decimal arithmetic is just as prone to double-rounding errors as binary arithmetic, and floats-- whether decimal or binary --force the user into the double-rounding business when they're working with fixed-point quantitities. From fredrik@pythonware.com Thu Dec 19 07:16:29 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 19 Dec 2002 08:16:29 +0100 Subject: [Python-Dev] Zipping Zope3 References: Message-ID: <005201c2a72e$8ec4c6b0$ced241d5@hagrid> Just van Rossum wrote: > We (as in: all of us) simply need more time to come up with a decent, > coherent and simple set of (optional) extensions to the protocol. some of us has been shipping deployment tools for many years, and have a pretty good understanding of what optional extensions we and our customers have needed over the years... From just@letterror.com Thu Dec 19 09:17:33 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 19 Dec 2002 10:17:33 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <005201c2a72e$8ec4c6b0$ced241d5@hagrid> Message-ID: Fredrik Lundh wrote: > Just van Rossum wrote: > > > We (as in: all of us) simply need more time to come up with a > > decent, coherent and simple set of (optional) extensions to the > > protocol. > > some of us has been shipping deployment tools for many years, and > have a pretty good understanding of what optional extensions we and > our customers have needed over the years... Ok, replace "all of us" by "all of us except /F". Seriously, what do you want me to do? Guido asked for i.stat_file(filename) i.open_file(filename) you asked for i.get_module_names() i.load_data() Jack asked for i.get_module_code(modulename) i.get_module_source(modulename) I'm sure someone suggested i.listdir() or else someone will. Although I agree with the intentions of all of those suggestions, there's just no way we can please everyone right now. Btw. what you want is _possible_ with the current zipimporter object: there's zi.get_data(filename), and you can inspect the "files" attribute (a dict mapping filenames to index info; it's an internal attribute, so I might rename it to _files). I have enough on my hands to get the PEP done and to get the implementation and the basic design right. Design descisions about (optional) extensions to the importer protocol will have to wait. If someone wants to pick that up and make a concrete proposal, be my guest. Just From Paul.Moore@atosorigin.com Thu Dec 19 09:22:37 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Thu, 19 Dec 2002 09:22:37 -0000 Subject: [Python-Dev] Zipping Zope3 Message-ID: <16E1010E4581B049ABC51D4975CEDB88619991@UKDCX001.uk.int.atosorigin.com> From: Just van Rossum [mailto:just@letterror.com] > Btw. what you want is _possible_ with the current zipimporter > object: there's zi.get_data(filename), and you can inspect the > "files" attribute (a dict mapping filenames to index info; it's > an internal attribute, so I might rename it to _files). Just to clarify this point: you can subclass the existing zipimport.zipimporter class and add any extra methods you want. Then, by replacing zipimporter in sys.path_hooks with your subclass, you transparently get your extra functionality. I'm not saying that this is the ultimate solution, but it gives us an opportunity to experiment before committing to a specific implementation in the core. Paul. From just@letterror.com Thu Dec 19 09:34:35 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 19 Dec 2002 10:34:35 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619991@UKDCX001.uk.int.atosorigin.com> Message-ID: Moore, Paul wrote: > Just to clarify this point: you can subclass the existing > zipimport.zipimporter class and add any extra methods you want. > Then, by replacing zipimporter in sys.path_hooks with your > subclass, you transparently get your extra functionality. It think would be better (for now) to just use the zipimporter object as is, and write a helper function. Subclassing should be possible, but might give you bootstrapping problems. Just From Jack.Jansen@oratrix.com Thu Dec 19 12:56:49 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Thu, 19 Dec 2002 13:56:49 +0100 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Message-ID: <564D985C-1351-11D7-84BC-003065517236@oratrix.com> On woensdag, dec 18, 2002, at 23:34 Europe/Amsterdam, Just van Rossum wrote: > Features schmeatures! I would agree, but Paul and I have decided to > keep > the importer protocol real simple, at least for now. Agreed. > The zipimporter object now has a get_data(filename) method > with which you can do what you want (in a less convenient way), and > Paul > questions even that addition, as we might not be able to go back once > it's in the core. It could even be the subject of a PEP of its own ;-) > > Also: *anything* that makes zipimporter closer to a file system is > going > at it from the wrong direction. We *first* need a virtual file system, > then an import hook that can import from it, and *then* a zip back > end... Ok, point taken, -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From just@letterror.com Thu Dec 19 13:14:56 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 19 Dec 2002 14:14:56 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) Message-ID: PEP: XXX Title: New Import Hooks Version: $Revision:$ Last-Modified: $Date:$ Author: Just van Rossum , Paul Moore Status: Draft Type: Standards Track Content-Type: text/plain Created: Python-Version: 2.3 Post-History: Abstract This PEP proposes to add a new set of import hooks that offer better customization of the Python import mechanism. Contrary to the current __import__ hook, a new-style hook can be injected into the existing scheme, allowing for a finer grained control of how modules are found and how they are loaded. Motivation The only way to customize the import mechanism is currently to override the builtin __import__ function. However, overriding __import__ has many problems. To begin with: - An __import__ replacement needs to *fully* reimplement the entire import mechanism, or call the original __import__ before or after the custom code. - It has very complex semantics and responsibilities. - __import__ gets called even for modules that are already in sys.modules, which is almost never what you want, unless you're writing some sort of monitoring tool. The situation gets worse when you need to extend the import mechanism from C: it's currently impossible, apart from hacking Python's import.c or reimplementing much of import.c from scratch. There is a fairly long history of tools written in Python that allow extending the import mechanism in various way, based on the __import__ hook. The Standard Library includes two such tools: ihooks.py (by GvR) and imputil.py (Greg Stein), but perhaps the most famous is iu.py by Gordon McMillan, available as part of his Installer [1] package. Their usefulness is somewhat limited because they are written in Python; bootstrapping issues need to worked around as you can't load the module containing the hook with the hook itself. So if you want the entire Standard Library to be loadable from an import hook, the hook must be written in C. Use cases This section lists several existing applications that depend on import hooks. Among these, a lot of duplicate work was done that could have been saved if there had been a more flexible import hook at the time. This PEP should make life a lot easier for similar projects in the future. Extending the import mechanism is needed when you want to load modules that are stored in a non-standard way. Examples include modules that are bundled together in an archive; byte code that is not stored in a pyc formatted file; modules that are loaded from a database over a network. The work on this PEP was partly triggered by the implementation of PEP 273 [2], which adds imports from Zip archives as a builtin feature to Python. While the PEP itself was widely accepted as a must-have feature, the implementation left a few things to desire. For one thing it went through great lengths to integrate itself with import.c, adding lots of code that was either specific for Zip file imports or *not* specific to Zip imports, yet was not generally useful (or even desirable) either. Yet the PEP 273 implementation can hardly be blamed for this: it is simply extremely hard to do, given the current state of import.c. Packaging applications for end users is a typical use case for import hooks, if not *the* typical use case. Distributing lots of source or pyc files around is not always appropriate (let alone a separate Python installation), so there is a frequent desire to package all needed modules in a single file. So frequent in fact that multiple solutions have been implemented over the years. The oldest one is included with the Python source code: Freeze [3]. It puts marshalled byte code into static objects in C source code. Freeze's "import hook" is hard wired into import.c, and has a couple of issues. Later solutions include Fredrik Lundh's Squeeze [4], Gordon McMillan's Installer [1] and Thomas Heller's py2exe [5]. MacPython ships with a tool called BuildApplication. Squeeze, Installer and py2exe use an __import__ based scheme (py2exe currently uses Installer's iu.py, Squeeze used ihooks.py), MacPython has two Mac-specific import hooks hard wired into import.c, that are similar to the Freeze hook. The hooks proposed in this PEP enables us (at least in theory; it's not a short term goal) to get rid of the hard coded hooks in import.c, and would allow the __import__-based tools to get rid of most of their import.c emulation code. Before work on the design and implementation of this PEP was started, a new BuildApplication-like tool for MacOSX prompted one of the authors of this PEP (JvR) to expose the table of frozen modules to Python, in the imp module. The main reason was to be able to use the freeze import hook (avoiding fancy __import__ support), yet to also be able to supply a set of modules at runtime. This resulted in sf patch #642578 [6], which was mysteriously accepted (mostly because nobody seemed to care either way ;-). Yet it is completely superfluous when this PEP gets accepted, as it offers a much nicer and general way to do the same thing. Rationale While experimenting with alternative implementation ideas to get builtin Zip import, it was discovered that achieving this is possible with only a fairly small amount of changes to import.c. This allowed to factor out the Zip-specific stuff into a new source file, while at the same time creating a *general* new import hook scheme: the one you're reading about now. An earlier design allowed non-string objects on sys.path. Such an object would have the neccesary methods to handle an import. This has two disadvantages: 1) it breaks code that assumes all items on sys.path are strings; 2) it is not compatible with the PYTHONPATH environment variable. The latter is directly needed for Zip imports. A compromise came from Jython: allow string *subclasses* on sys.path, which would then act as importer objects. This avoids some breakage, and seems to work well for Jython (where it is used to load modules from .jar files), but it was perceived as an "ugly hack". This lead to a more elaborate scheme, (mostly copied from McMillan's iu.py) in which each in a list of candidates is asked whether it can handle the sys.path item, until one is found that can. This list of candidates is a new object in the sys module: sys.path_hooks. Traversing sys.path_hooks for each path item for each new import can be expensive, so the results are cached in another new object in the sys module: sys.path_importer_cache. It maps sys.path entries to importer objects. To minimize the impact on import.c as well as to avoid adding extra overhead, it was chosen to not add an explicit hook and importer object for the existing file system import logic (as iu.py has), but to simply fall back to the builtin logic if no hook on sys.path_hooks could handle the path item. If this is the case, a None value is stored in sys.path_importer_cache, again to avoid repeated lookups. (Later we can go further and add a real importer object for the builtin mechanism, for now, the None fallback scheme should suffice.) A question was raised: what about importers that don't need *any* entry on sys.path? (Builtin and frozen modules fall into that category.) Again, Gordon McMillan to the rescue: iu.py contains a thing he calls the "metapath". In this PEP's implementation, it's a list of importer objects that is traversed *before* sys.path. This list is yet another new object in the sys.module: sys.meta_path. Currently, this list is empty by default, and frozen and builtin module imports are done after traversing sys.meta_path, but still before sys.path. (Again, later we can add real frozen, builtin and sys.path importer objects on sys.meta_path, allowing for some extra flexibility, but this could be done as a "phase 2" project, possibly for Python 2.4. It would be the finishing touch as then *every* import would go through sys.meta_path, making it the central import dispatcher.) As a bonus, the idea from the second paragraph of this section was implemented after all: a sys.path item may *be* an importer object. This use is discouraged for general purpose code, but it's very convenient, for experimentation as well as for projects of which it's known that no component wrongly assumes that sys.path items are strings. Specification part 1: The Importer Protocol This PEP introduces a new protocol: the "Importer Protocol". It is important to understand the context in which the protocol operates, so here is a brief overview of the outer shells of the import mechanism. When an import statement is encountered, the interpreter looks up the __import__ function in the builtin name space. __import__ is then called with four arguments, amongst which are the name of the module being imported (may be a dotted name) and a reference to the current global namespace. The builtin __import__ function (known as PyImport_ImportModuleEx in import.c) will then check to see whether the module doing the import is a package by looking for a __path__ variable in the current global namespace. If it is indeed a package, it first tries to do the import relative to the package. For example if a package named "spam" does "import eggs", it will first look for a module named "spam.eggs". If that fails, the import continues as an absolute import: it will look for a module named "eggs". Dotted name imports work pretty much the same: if package "spam" does "import eggs.bacon", first "spam.eggs.bacon" is tried, and only if that fails "eggs.bacon" is tried. Deeper down in the mechanism, a dotted name import is split up by its components. For "import spam.ham", first an "import spam" is done, and only when that succeeds is "ham" imported as a submodule of "spam". The Importer Protocol operates at this level of *individual* imports. By the time an importer gets a request for "spam.ham", module "spam" has already been imported. The protocol involves two objects: an importer and a loader. An importer object has a single method: importer.find_module(fullname) This method returns a loader object if the module was found, or None if it wasn't. If find_module() raises an exception, it will be propagated to the caller, aborting the import. A loader object also has one method: loader.load_module(fullname) This method returns the loaded module. In many cases the importer and loader can be one and the same object: importer.find_module() would just return self. The 'fullname' argument of both methods is the fully qualified module name, for example "spam.eggs.ham". As explained above, when importer.find_module("spam.eggs.ham") is called, "spam.eggs" has already been imported and added to sys.modules. However, the find_module() method isn't neccesarily always called during an actual import: meta tools that analyze import dependencies (such as freeze, Installer or py2exe) don't actually load modules, so an importer shouldn't *depend* on the parent package being available in sys.modules. The load_module() method has a few responsibilities that it must fulfill *before* it runs any code: - It must create the module object. From Python this can be done via the new.module() function, the imp.new_module() function or via the module type object; from C with the PyModule_New() function or the PyImport_ModuleAdd() function. The latter also does the following step: - It must add the module to sys.modules. This is crucial because the module code may (directly or indirectly) import itself; adding it to sys.modules beforehand prevents unbounded recursion in the worst case and multiple loading in the best. - The __file__ attribute must be set. This must be a string, but it may be a dummy value, for example "". The priviledge of not having a __file__ attribute at all is reserved for builtin modules. - If it's a package, the __path__ variable must be set. This must be a list, but may be empty if __path__ has no further significance to the importer (more on this later). - It should add an __importer__ attribute to the module, set to the loader object. This is mostly for introspection, but can be used for importer-specific extra's, for example getting data associated with an importer. If the module is a Python module (as opposed to a builtin module or an dynamically loaded extension), it should execute the module's code in the module's global name space (module.__dict__). Here is a minimal pattern for a load_module() method: def load_module(self, fullname): ispkg, code = self._get_code(fullname) mod = imp.new_module(fullname) sys.modules[fullname] = mod mod.__file__ = "<%s>" % self.__class__.__name__ mod.__importer__ = self if ispkg: mod.__path__ = [] exec code in mod.__dict__ return mod Specification part 2: Registering Hooks There are two types of import hooks: Meta hooks and Path hooks. Meta hooks are called at the start of import processing, before any other import processing (so that meta hooks can override sys.path processing, or frozen modules, or even builtin modules). To register a meta hook, simply add the importer object to sys.meta_path (the list of registered meta hooks). Path hooks are called as part of sys.path (or package __path__) processing, at the point where their associated path item is encountered. A path hook can be registered in either of two ways: - By simply including an importer object directly on the path. This approach is discouraged for general purpose hooks, as existing code may not be expecting non-strings to exist on sys.path. - By registering an importer factory in sys.path_hooks. sys.path_hooks is a list of callables, which will be checked in sequence to determine if they can handle a given path item. The callable is called with one argument, the path item. The callable must raise ImportError if it is unable to handle the path item, and return an importer object if it can handle the path item. The callable is typically the class of the import hook, and hence the class __init__ method is called. (This is also the reason why it should raise ImportError: an __init__ method can't return anything.) The results of path hook checks are cached in sys.path_importer_cache, which is a dictionary mapping path entries to importer objects. The cache is checked before sys.path_hooks is scanned. If it is necessary to force a rescan of sys.path_hooks, it is possible to manually clear all or part of sys.path_importer_cache. Just like sys.path itself, the new sys variables must have specific types: sys.meta_path and sys.path_hooks must be Python lists. sys.path_importer_cache must be a Python dict. Modifying these variables in place is allowed, as is replacing them with new objects. Packages and the role of __path__ If a module has a __path__ attribute, the import mechanism will treat it as a package. The __path__ variable is used instead of sys.path when importing submodules of the package. The rules for sys.path therefore also apply to pkg.__path__. So sys.path_hooks is also consulted when pkg.__path__ is traversed and importer objects as path items are also allowed (yet, are discouraged for the same reasons as they are discouraged on sys.path, at least for general purpose code). Meta importers don't neccesarily use sys.path at all to do their work and therefore may also ignore the value of pkg.__path__. In this case it is still advised to set it to list, which can be empty. Integration with the 'imp' module The new import hooks are not easily integrated in the existing imp.find_module() and imp.load_module() calls. It's questionable whether it's possible at all without breaking code; it is better to simply add a new function to the imp module. The meaning of the existing imp.find_module() and imp.load_module() calls changes from: "they expose the builtin import mechanism" to "they expose the basic *unhooked* builtin import mechanism". They simply won't invoke any import hooks. A new imp module function is proposed under the name "find_module2", with is used like the following pattern: loader = imp.find_module2(fullname, path) if loader is not None: loader.load_module(fullname) In the case of a "basic" import, one the imp.find_module() function would handle, the loader object would be a wrapper for the current output of imp.find_module(), and loader.load_module() would call imp.load_module() with that output. Note that this wrapper is currently not yet implemented, although a Python prototype exists in the test_importhooks.py script (the ImpWrapper class) included with the patch. Open Issues The new hook method allows for the possibility of objects other than strings appearing on sys.path. Existing code is entitled to assume that sys.path only contains strings (the Python documentation states this). It is not clear if this will cause significant breakage. In particular, it is much less clear that code is entitled to assume that sys.path contains a list of *directory names* - most code which assumes that sys.path items contain strings also rely on this extra assumption, and so could be considered as broken (or at least "not robust") already. Modules often need supporting data files to do their job, particularly in the case of complex packages or full applications. Current practice is generally to locate such files via sys.path (or a package __path__ variable). This approach will not work, in general, for modules loaded via an import hook. There are a number of possible ways to address this problem: - "Don't do that". If a package needs to locate data files via its __path__, it is not suitable for loading via an import hook. The package can still be located on a directory in sys.path, as at present, so this should not be seen as a major issue. - Locate data files from a standard location, rather than relative to the module file. A relatively simple approach (which is supported by distutils) would be to locate data files based on sys.prefix (or sys.exec_prefix). For example, looking in os.path.join(sys.prefix, "data", package_name). - Import hooks could offer a standard way of getting at datafiles relative to the module file. The standard zipimport object provides a method get_data(name) which returns the content of the "file" called name, as a string. To allow modules to get at the importer object, zipimport also adds an attribute "__importer__" to the module, containing the zipimport object used to load the module. If such an approach is used, it is important that client code takes care not to break if the get_data method (or the __importer__ attribute) is not available, so it is not clear that this approach offers a general answer to the problem. Requiring loaders to set the module's __importer__ attribute means that the loader will not get thrown away once the load is complete. This increases memory usage, and stops loaders from being lightweight, "throwaway" objects. As loader objects are not required to offer any useful functionality (any such functionality, such as the zipimport get_data() method mentioned above, is optional) it is not clear that the __importer__ attribute will be helpful, in practice. On the other hand, importer objects are mostly permanent, as they live or are kept alive on sys.meta_path, sys.path_importer_cache or sys.path, so for a loader to keep a reference to the importer costs us nothing extra. Whether loaders will ever need to carry so much independent state for this to become a real issue is questionable. Implementation A C implementation is available as SourceForge patch 652586. http://www.python.org/sf/652586 References [1] Installer by Gordon McMillan http://www.mcmillan-inc.com/install1.html [2] PEP 273, Import Modules from Zip Archives, Ahlstrom http://www.python.org/peps/pep-0273.html [3] The Freeze tool Tools/freeze/ in a Python source distribution [4] Squeeze http://starship.python.net/crew/fredrik/ipa/squeeze.htm [5] py2exe by Thomas Heller http://py2exe.sourceforge.net/ [6] imp.set_frozenmodules() patch http://www.python.org/sf/642578 Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From guido@python.org Thu Dec 19 14:39:58 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 19 Dec 2002 09:39:58 -0500 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Your message of "Thu, 19 Dec 2002 08:16:29 +0100." <005201c2a72e$8ec4c6b0$ced241d5@hagrid> References: <005201c2a72e$8ec4c6b0$ced241d5@hagrid> Message-ID: <200212191439.gBJEdwo23277@odiug.zope.com> > some of us has been shipping deployment tools for many years, and > have a pretty good understanding of what optional extensions we and > our customers have needed over the years... Would you mind summarizing what you think is needed? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Dec 19 14:47:37 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 19 Dec 2002 09:47:37 -0500 Subject: [Python-Dev] Zipping Zope3 In-Reply-To: Your message of "Thu, 19 Dec 2002 13:56:49 +0100." <564D985C-1351-11D7-84BC-003065517236@oratrix.com> References: <564D985C-1351-11D7-84BC-003065517236@oratrix.com> Message-ID: <200212191447.gBJElbc23342@odiug.zope.com> > > Also: *anything* that makes zipimporter closer to a file system is > > going at it from the wrong direction. We *first* need a virtual > > file system, then an import hook that can import from it, and > > *then* a zip back end... Note that there's a design for a VFS API in ihooks.py. Worth checking out. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Dec 19 14:52:39 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 19 Dec 2002 09:52:39 -0500 Subject: [Python-Dev] Mail to fredrik@pythonware.com bounces Message-ID: <200212191452.gBJEqdn23797@odiug.zope.com> Has the effbot decided he's not interested in feedback any more? :-) --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Thu, 19 Dec 2002 09:41:46 -0500 From: Mail Delivery Subsystem To: Subject: Returned mail: see transcript for details This is a MIME-encapsulated message - --gBJEfko23279.1040308906/odiug.zope.com The original message was received at Thu, 19 Dec 2002 09:39:59 -0500 from guido@localhost ----- The following addresses had permanent fatal errors ----- (reason: 550 ... denied) ----- Transcript of session follows ----- ... while talking to mailgw.algonet.se.: >>> RCPT To: <<< 550 ... denied 550 5.1.1 ... User unknown - --gBJEfko23279.1040308906/odiug.zope.com Content-Type: message/delivery-status Reporting-MTA: dns; odiug.zope.com Arrival-Date: Thu, 19 Dec 2002 09:39:59 -0500 Final-Recipient: RFC822; fredrik@pythonware.com Action: failed Status: 5.1.1 Remote-MTA: DNS; mailgw.algonet.se Diagnostic-Code: SMTP; 550 ... denied Last-Attempt-Date: Thu, 19 Dec 2002 09:41:03 -0500 - --gBJEfko23279.1040308906/odiug.zope.com Content-Type: message/rfc822 Return-Path: Received: from guido (guido@localhost) by odiug.zope.com (8.11.0/8.11.0) with ESMTP id gBJEdwo23277; Thu, 19 Dec 2002 09:39:59 -0500 Message-Id: <200212191439.gBJEdwo23277@odiug.zope.com> To: "Fredrik Lundh" cc: "Jack Jansen" , "Samuele Pedroni" , "Moore, Paul" , python-dev@python.org Subject: Re: [Python-Dev] Zipping Zope3 In-reply-to: Your message of "Thu, 19 Dec 2002 08:16:29 +0100." <005201c2a72e$8ec4c6b0$ced241d5@hagrid> References: <005201c2a72e$8ec4c6b0$ced241d5@hagrid> From: Guido van Rossum Date: Thu, 19 Dec 2002 09:39:58 -0500 Sender: guido@odiug.zope.com > some of us has been shipping deployment tools for many years, and > have a pretty good understanding of what optional extensions we and > our customers have needed over the years... Would you mind summarizing what you think is needed? - --Guido van Rossum (home page: http://www.python.org/~guido/) - --gBJEfko23279.1040308906/odiug.zope.com-- ------- End of Forwarded Message From neal@metaslash.com Thu Dec 19 14:53:46 2002 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 19 Dec 2002 09:53:46 -0500 Subject: [Python-Dev] deprecating APIs In-Reply-To: <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net> References: <20021217014935.GQ16807@epoch.metaslash.com> <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021219145346.GG12063@epoch.metaslash.com> On Wed, Dec 18, 2002 at 10:54:34PM -0500, Guido van Rossum wrote: > > What does this do under GCC? I suppose it issues a warning when you > use the deprecated API? I modified abstract.h: - PyAPI_FUNC(int) PyNumber_Check(PyObject *o); + PyAPI_FUNC(int) PyNumber_Check(PyObject *o) Py_DEPRECATED(2.3); I used PyNumber_Check in selectmodule.c and got: Modules/selectmodule.c: In function `select_select': Modules/selectmodule.c:211: warning: `PyNumber_Check' is deprecated (declared at Include/abstract.h:534) Neal From guido@python.org Thu Dec 19 14:59:10 2002 From: guido@python.org (Guido van Rossum) Date: Thu, 19 Dec 2002 09:59:10 -0500 Subject: [Python-Dev] deprecating APIs In-Reply-To: Your message of "Thu, 19 Dec 2002 09:53:46 EST." <20021219145346.GG12063@epoch.metaslash.com> References: <20021217014935.GQ16807@epoch.metaslash.com> <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net> <20021219145346.GG12063@epoch.metaslash.com> Message-ID: <200212191459.gBJExAa25897@odiug.zope.com> > > What does this do under GCC? I suppose it issues a warning when you > > use the deprecated API? > > I modified abstract.h: > > - PyAPI_FUNC(int) PyNumber_Check(PyObject *o); > + PyAPI_FUNC(int) PyNumber_Check(PyObject *o) Py_DEPRECATED(2.3); > > I used PyNumber_Check in selectmodule.c and got: > > Modules/selectmodule.c: In function `select_select': > Modules/selectmodule.c:211: warning: `PyNumber_Check' is deprecated > (declared at Include/abstract.h:534) Neat! +1. Now it's time to make a list of APIs we want to declare deprecated. (Some have probably been deprecated for quite a while...) --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Thu Dec 19 15:46:41 2002 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 19 Dec 2002 10:46:41 -0500 Subject: [Python-Dev] deprecating APIs In-Reply-To: <200212191459.gBJExAa25897@odiug.zope.com> References: <20021217014935.GQ16807@epoch.metaslash.com> <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net> <20021219145346.GG12063@epoch.metaslash.com> <200212191459.gBJExAa25897@odiug.zope.com> Message-ID: <20021219154641.GH12063@epoch.metaslash.com> On Thu, Dec 19, 2002 at 09:59:10AM -0500, Guido van Rossum wrote: > > Now it's time to make a list of APIs we want to declare deprecated. > (Some have probably been deprecated for quite a while...) There are only 847. :-) I don't think this really helps... But for usage counts (by grep): http://www.metaslash.com/py/api-counts.txt If nothing else, it's a quick list of names to look over. Neal From just@letterror.com Thu Dec 19 18:14:55 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 19 Dec 2002 19:14:55 +0100 Subject: [Python-Dev] MacOSX build broken Message-ID: A plain ./configure ends like this, without creating a Makefile: checking for --with-universal-newlines... --without-universal-newlines is unsupported, see README The README confirms --with-universal-newlines is no longer supported. Yet I didn't specify --with-universal-newlines and the README doesn't tell me what to do either. What's up? Just From martin@v.loewis.de Thu Dec 19 19:06:36 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 19 Dec 2002 20:06:36 +0100 Subject: [Python-Dev] MacOSX build broken In-Reply-To: References: Message-ID: Just van Rossum writes: > What's up? It was a bug; it should be fixed now. Regards, Martin From brett@python.org Thu Dec 19 21:10:11 2002 From: brett@python.org (Brett Cannon) Date: Thu, 19 Dec 2002 13:10:11 -0800 (PST) Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: Message-ID: Now why couldn't you guys have finished this *before* the summary went out?!? Then I could have just said, "python-dev talked a lot about a new import system. Read the PEP if you care". =) Good PEP, all kiddding aside. Both of you did a good job covering the bases in an easy way to comprehend. -Brett P.S.: Removed cc: to peps@python.org From just@letterror.com Thu Dec 19 21:34:56 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 19 Dec 2002 22:34:56 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: Message-ID: Brett Cannon wrote: > Now why couldn't you guys have finished this *before* the summary > went out?!? Then I could have just said, "python-dev talked a lot > about a new import system. Read the PEP if you care". =) ;-) Now if only we got a PEP number... > Good PEP, all kiddding aside. Both of you did a good job covering > the bases in an easy way to comprehend. Thanks. Just From Jack.Jansen@oratrix.com Thu Dec 19 23:03:15 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Fri, 20 Dec 2002 00:03:15 +0100 Subject: [Python-Dev] Strange crash in import In-Reply-To: <911129A1-1208-11D7-BC1B-000A27B19B96@oratrix.com> Message-ID: <0DF13AB8-13A6-11D7-88F3-000A27B19B96@oratrix.com> On dinsdag, dec 17, 2002, at 22:43 Europe/Amsterdam, Jack Jansen wrote: > I'm getting a strange crash in import, and it turns out that there's a > refcount 0 object on sys.path. Just in case anyone is interested: this turned out to be my mistake, I overly enthusiastically cleared out memory beyond my own object. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From aahz@pythoncraft.com Thu Dec 19 23:24:54 2002 From: aahz@pythoncraft.com (Aahz) Date: Thu, 19 Dec 2002 18:24:54 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: Message-ID: <20021219232454.GA24288@panix.com> First of all, excellent job! I've been feeling somewhat frustrated by the number of directions this discussion has taken, and I'm glad to have something that's clear and understandable. One typo nitpick, one real question: On Thu, Dec 19, 2002, Just van Rossum wrote: > > - The __file__ attribute must be set. This must be a string, but it > may be a dummy value, for example "". The priviledge of > not having a __file__ attribute at all is reserved for builtin > modules. "privilege" > sys.path_hooks is a list of callables, which will be checked in > sequence to determine if they can handle a given path item. The > callable is called with one argument, the path item. The callable > must raise ImportError if it is unable to handle the path item, and > return an importer object if it can handle the path item. The > callable is typically the class of the import hook, and hence the > class __init__ method is called. (This is also the reason why it > should raise ImportError: an __init__ method can't return anything.) Any reason we can't require importer objects to be new-style classes, which could then use __new__? (I vaguely recall some discussion about this, but it should be documented in the PEP.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From airkjh@i-on.net Fri Dec 20 06:52:31 2002 From: airkjh@i-on.net (=?ks_c_5601-1987?B?sejA58f1?=) Date: Fri, 20 Dec 2002 15:52:31 +0900 Subject: [Python-Dev] Hi everyone!! Message-ID: This is a multi-part message in MIME format. ------_=_NextPart_001_01C2A7F4.5E05D1F6 Content-Type: text/plain; charset="ks_c_5601-1987" Content-Transfer-Encoding: quoted-printable I=A1=AFm a new member of python mailing list. I=A1=AFve developed some programs with python for 2 years ( especially, = Database programs using Oracle , MS-SQL , DB2 ) =20 I hope to be a good helper and friend to you Have a nice day =20 ------------------------------------ Kim Jae Hyuk Customer Engineer Development Division I-ON Communications TEL : 82-2-574-1080 Mail : airkjh@i-on.net ------------------------------------ =20 ------_=_NextPart_001_01C2A7F4.5E05D1F6 Content-Type: text/html; charset="ks_c_5601-1987" Content-Transfer-Encoding: quoted-printable

I=A1=AFm a new member of python mailing = list.

I=A1=AFve developed some programs with python for 2 = years ( especially, Database programs using Oracle , = MS-SQL , DB2 )

 

I hope to be a good helper and friend to = you

Have a nice day

 

------------------------------------<= o:p>

Kim Jae Hyuk

Customer Engineer

Development Division

I-ON Communications

TEL  : = 82-2-574-1080

Mail : airkjh@i-on.net

------------------------------------<= o:p>

 

------_=_NextPart_001_01C2A7F4.5E05D1F6-- From fredrik@pythonware.com Fri Dec 20 07:47:26 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 20 Dec 2002 08:47:26 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: <20021219232454.GA24288@panix.com> Message-ID: <004201c2a7fc$0c30ff10$ced241d5@hagrid> Aahz keeps confusing me: > Any reason we can't require importer objects to be new-style classes, > which could then use __new__? (I vaguely recall some discussion about > this, but it should be documented in the PEP.) any reason we *should* require importer objects to be new-style classes? if you think raising exceptions is a problem, please say so. if you have a valid argument for using return values instead of exceptions (it's not like people will be installing thousands of hooks, so it's not obvious what the problem is), is there any reason not to allow people to use *any* callable object that can return a value? (including three-line wrappers for pre-2.2 classes). From Paul.Moore@atosorigin.com Fri Dec 20 09:39:07 2002 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Fri, 20 Dec 2002 09:39:07 -0000 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D800@UKDCX001.uk.int.atosorigin.com> Aahz said: > Any reason we can't require importer objects to be new-style > classes, which could then use __new__? (I vaguely recall > some discussion about this, but it should be documented in > the PEP.) Basically, why bother? It just makes writing importers (mildly) more complex (most people know about __init__ whereas __new__ is ... err ... newer), where the existing proposal works fine. But you're right, a comment in the PEP would be good. Paul From aahz@pythoncraft.com Fri Dec 20 14:02:34 2002 From: aahz@pythoncraft.com (Aahz) Date: Fri, 20 Dec 2002 09:02:34 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: <004201c2a7fc$0c30ff10$ced241d5@hagrid> References: <20021219232454.GA24288@panix.com> <004201c2a7fc$0c30ff10$ced241d5@hagrid> Message-ID: <20021220140234.GA13980@panix.com> On Fri, Dec 20, 2002, Fredrik Lundh wrote: > > if you have a valid argument for using return values instead of > exceptions (it's not like people will be installing thousands of > hooks, so it's not obvious what the problem is), is there any reason > not to allow people to use *any* callable object that can return a > value? (including three-line wrappers for pre-2.2 classes). Good point. That should go in the PEP. (In case it wasn't clear, I was arguing neither for nor against new-style classes, I just wanted it documented in the PEP since it was discussed.) Hmmmm.... I seem to also recall seeing discussion about the callable raising something other than ImportError when it just can't handle a path element. What happened to that idea? That would allow the hook handler to distinguish between "I had an error in handling this path" and "I can't handle this path". -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com From just@letterror.com Fri Dec 20 14:08:58 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 20 Dec 2002 15:08:58 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: <20021220140234.GA13980@panix.com> Message-ID: Aahz wrote: > Hmmmm.... I seem to also recall seeing discussion about the callable > raising something other than ImportError when it just can't handle a > path element. What happened to that idea? That would allow the hook > handler to distinguish between "I had an error in handling this path" > and "I can't handle this path". I argued that it is unlikely that calling the hook would cause an import, so raising ImportError should be just fine. I don't recall getting any opposition for that. It's now specced as "must raise ImportError". I really think the issue is too minor to warrant a new exception. Just From jim@interet.com Fri Dec 20 17:08:07 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 20 Dec 2002 12:08:07 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: Message-ID: <3E034E77.8090609@interet.com> Just van Rossum wrote: > PEP: XXX > Title: New Import Hooks First, I commend Just and Paul for their work. It is not easy sloughing through Python imports. And the background on how imports work and the desired changes are valuable. I am the author of the competing PEP 273 implementation, and if Just's implementation can provide a better implementation, I won't mind. But the changes to the imp module IMHO are ill advised. I am posting an alternative here, and will post other objections separately. > Integration with the 'imp' module > > The new import hooks are not easily integrated in the existing > imp.find_module() and imp.load_module() calls. It's questionable > whether it's possible at all without breaking code; it is better to > simply add a new function to the imp module. Although I originally found the find/load dichotomy annoying, I now think it should be kept. It solves the real world problem of finding files which are not Python modules. Currently Zope and others find configuration and data files either by looping over sys.path themselves, or by looking at the base name of module.__file__ (the file name of an imported module). Both fail for zip imports. I too find the imp find/load function signatures hard to love. But I also don't want to break any code which depends on them, nor make gratuitous changes which make them useless for zip imports. I think the imp module should Just Work for zip imports. I suggest we keep imp.find_module but add another argument: find_module(name [, path [, suffixes]]) where suffixes defaults to imp.get_suffixes(). This enables it to be used to search for files with different suffixes, for example suffixes=[".conf"]. To make this work, the returned file object would have to be either a real file object, or another object with a read() method. Python has lots of precedents for accepting file-like objects. For example, sys.stdout can be replaced with any object which has a write method. The returned file-like object must have a read() method and a close() method. It could also have a stat() method if it is a zip file, because zip files record the file date, time and size. So you call file.read() for a configuration file, or pass it to imp.load_module() if it is a Python module. JimA From jim@interet.com Fri Dec 20 17:23:37 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 20 Dec 2002 12:23:37 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: <3E034E77.8090609@interet.com> Message-ID: <3E035219.6050007@interet.com> James C. Ahlstrom wrote: > to be used to search for files with different suffixes, for > example suffixes=[".conf"]. Ooops, that should be suffixes = [(".conf", "r", 0)] JimA From just@letterror.com Fri Dec 20 18:25:27 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 20 Dec 2002 19:25:27 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: <3E034E77.8090609@interet.com> Message-ID: James C. Ahlstrom wrote: > Although I originally found the find/load dichotomy annoying, > I now think it should be kept. It *is* kept, in every detail of the PEP. > It solves the real > world problem of finding files which are not Python modules. > Currently Zope and others find configuration and data files > either by looping over sys.path themselves, or by looking > at the base name of module.__file__ (the file name of an > imported module). Both fail for zip imports. That's why I introduced __importer__ and __importer__.get_data(). (The details of their specification might change, though. Oh wait, get_data() has no specification at all yet ;-) > I too find the imp find/load function signatures hard to love. But > I also don't want to break any code which depends on them, nor > make gratuitous changes which make them useless for zip imports. > I think the imp module should Just Work for zip imports. I didn't change anything there and I sure don't break any code, because there's hardly any code out there that expects imp.find_module() to work for zip files <0.5 wink>. However, for imp.find_module() to "just work" and be *useful* for imports from importer objects, it will need to return a loader object. I don't see how this can be done without breaking imp.find_module(). I thought of returning a loader object instead of a file object for "hooked" imports, but unless we add a dummy close() method to all loader objects, this will break a common idiom: file, filename, stuff = imp.find_module(...) if file: file.close() And that's just the beginning of the trouble. It's also rather hackish. So I propose to add a new function to the imp module, as described in the PEP. And no, this one doesn't solve the data file problem. I could change imp.find_module() to return *something* for hooked imports, but it won't be useful to do imports without a loader object. > I suggest we keep imp.find_module but add another argument: > > find_module(name [, path [, suffixes]]) > > where suffixes defaults to imp.get_suffixes(). This enables it > to be used to search for files with different suffixes, for > example suffixes=[".conf"]. I'm not sure if I like this: imp.find_module() is designed for finding modules, not arbitrary files. It's an interesting idea though, but this would neccesarily complicate the importer protocol, as each importer would have to deal with arbitrary suffixes. It implies file-system semantics that importer objects can't neccesarily satisfy, eg. an importer doesn't neccesarily deal with suffixes at *all*. > To make this work, the returned file object would have to be > either a real file object, or another object with a read() method. > Python has lots of precedents for accepting file-like objects. For > example, sys.stdout can be replaced with any object which has a > write method. > > The returned file-like object must have a read() method and > a close() method. It could also have a stat() method if it > is a zip file, because zip files record the file date, time > and size. (File objects don't have a stat method.) > So you call file.read() for a configuration file, or pass > it to imp.load_module() if it is a Python module. I'll think about this some more, but I'm not yet convinced. Just From theller@python.net Fri Dec 20 19:05:09 2002 From: theller@python.net (Thomas Heller) Date: 20 Dec 2002 20:05:09 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: Message-ID: > > I suggest we keep imp.find_module but add another argument: > > > > find_module(name [, path [, suffixes]]) > > > > where suffixes defaults to imp.get_suffixes(). This enables it > > to be used to search for files with different suffixes, for > > example suffixes=[".conf"]. > > I'm not sure if I like this: imp.find_module() is designed for finding > modules, not arbitrary files. It's an interesting idea though, but this > would neccesarily complicate the importer protocol, as each importer > would have to deal with arbitrary suffixes. It implies file-system > semantics that importer objects can't neccesarily satisfy, eg. an > importer doesn't neccesarily deal with suffixes at *all*. I also found this an interesting idea, first. But then, how would this (working) code have to be written with the new imp.find_module() function: def get_exe_bytes (self): # wininst.exe is in the same directory as this file directory = os.path.dirname(__file__) filename = os.path.join(directory, "wininst.exe") return open(filename, "rb").read() Thomas From just@letterror.com Fri Dec 20 19:06:38 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 20 Dec 2002 20:06:38 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: <3E034E77.8090609@interet.com> Message-ID: James C. Ahlstrom wrote: > I suggest we keep imp.find_module but add another argument: > > find_module(name [, path [, suffixes]]) > > where suffixes defaults to imp.get_suffixes(). This enables it > to be used to search for files with different suffixes, for > example suffixes=[".conf"]. One more thought: finding modules and finding data files is not the same thing. Modules can be anywhere and must be accessible to anyone, but data files belong to specific modules. Searching the entire sys.path for a data file seems way too broad; there's got to be some information available as to where the data is. Eg. I doubt it's a *feature* that pdb.doc may also be found in site-packages. Just PS: Sorry for Cc-ing peps@python.org before, it was a bad idea to crosspost the draft to python-dev and peps to begin with :-( From just@letterror.com Fri Dec 20 19:14:01 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 20 Dec 2002 20:14:01 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: Message-ID: Thomas Heller wrote: > But then, how would this (working) code have to be written with the > new imp.find_module() function: > > def get_exe_bytes (self): > # wininst.exe is in the same directory as this file > directory = os.path.dirname(__file__) > filename = os.path.join(directory, "wininst.exe") > return open(filename, "rb").read() With the importer protocol (as I imagine it now, this doesn't match zipimporter's current behavior yet) it could be written like this: def get_exe_bytes (self): if os.path.isfile(__file__): # wininst.exe is in the same directory as this file directory = os.path.dirname(__file__) filename = os.path.join(directory, "wininst.exe") return open(filename, "rb").read() else: path = __name__.replace(".", os.sep) + os.sep + \ "wininst.exe" return __importer__.get_data(path) Just PS: please take care to not Cc to peps@python.org any more in this thread, I think I've done enough harm ;-) From theller@python.net Fri Dec 20 19:46:05 2002 From: theller@python.net (Thomas Heller) Date: 20 Dec 2002 20:46:05 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: Message-ID: Just van Rossum writes: > Thomas Heller wrote: > > > But then, how would this (working) code have to be written with the > > new imp.find_module() function: > > > > def get_exe_bytes (self): > > # wininst.exe is in the same directory as this file > > directory = os.path.dirname(__file__) > > filename = os.path.join(directory, "wininst.exe") > > return open(filename, "rb").read() > > With the importer protocol (as I imagine it now, this doesn't match > zipimporter's current behavior yet) it could be written like this: > > def get_exe_bytes (self): > if os.path.isfile(__file__): > # wininst.exe is in the same directory as this file > directory = os.path.dirname(__file__) > filename = os.path.join(directory, "wininst.exe") > return open(filename, "rb").read() > else: > path = __name__.replace(".", os.sep) + os.sep + \ > "wininst.exe" > return __importer__.get_data(path) > Looks ok to me, although I would prefer to write path = os.sep.join(__name__.split(".") + ["wininst.exe"]) or path = os.path.join(*__name__.split(".") + ["wininst.exe"]) I just wanted to make sure that I don't have to construct the arguments for imp.find_module() from __file__. > > PS: please take care to not Cc to peps@python.org any more in this > thread, I think I've done enough harm ;-) Sorry for that, I noticed it as it was already too late. Thomas From jim@interet.com Fri Dec 20 20:22:22 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 20 Dec 2002 15:22:22 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: Message-ID: <3E037BFE.7020509@interet.com> Thomas Heller wrote: > But then, how would this (working) code have to be written with the > new imp.find_module() function: > > def get_exe_bytes (self): > # wininst.exe is in the same directory as this file > directory = os.path.dirname(__file__) > filename = os.path.join(directory, "wininst.exe") > return open(filename, "rb").read() It would look like this: suffix = [(".exe", "rb", 0)] file, path, descr = imp.find_module("wininst", __path__, suffix) data = file.read() file.close() return data Note that for a zip file, __file__ is something like C:/Python/lib/zarchive.zip/subdir/mymod.pyc and nothing is going to make code using __file__ Just Work. Well, maybe you could strip the last component "mymod.pyc" and use it as __path__ in a call to imp.find_module(). Not too hard, and it works with files and zip archives. But maybe not with future ftp:// imported modules. Your code knows the directory. My version would search in your package __path__ for it. But you only have one wininst.exe per package, right? And maybe putting it on package path is a feature. JimA From just@letterror.com Fri Dec 20 20:19:52 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 20 Dec 2002 21:19:52 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: Message-ID: Thomas Heller wrote: > Looks ok to me, although I would prefer to write > > path = os.sep.join(__name__.split(".") + ["wininst.exe"]) > or > path = os.path.join(*__name__.split(".") + ["wininst.exe"]) Much better. I think I'd prefer the first, mostly because os.path.join() might do more magic than needed. I think it's time to pin down the next extensions to the importer protocol. How about the next speclet: bool = loader.is_package(fullname) code = loader.get_code(fullname) str = loader.get_source(fullname) (is_package and get_code are needed for modulefinder-like tools, get_source is needed for linecache-like tools.) All three should raise ImportError if the module isn't available. get_code and get_source should return None if there's no code or source associated with the module, respectively. "No code" means it's a builtin module or an extension. "No source" means just that (eg. for zipimport it would mean "I've only got a .pyc or .pyo file for you"). If the importer does support get_source(), yet doesn't have a code object readily available, get_code() should compile the source by itself and return the code. This is to make life easier for the caller: if it only needs the code object it shouldn't need to also check get_source() and deal with that. They should be optional, yet strongly recommended for general purpose hooks. There's no need to support them for hooks that are specific to a deliverable app; for example zipimport should have them, but if Thomas writes a dedicated hook for the apps built by py2exe, he should be free to leave them out. Btw. would it be better if mod.__importer__ was named mod.__loader__? This is closer to the truth, yet I fear that __loader__ is too generic. > > PS: please take care to not Cc to peps@python.org any more in this > > thread, I think I've done enough harm ;-) > > Sorry for that, I noticed it as it was already too late. No need to be sorry, it was my fault... Just From jim@interet.com Fri Dec 20 20:41:29 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 20 Dec 2002 15:41:29 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: Message-ID: <3E038079.3040607@interet.com> Just van Rossum wrote: > One more thought: finding modules and finding data files is not the same > thing. Modules can be anywhere and must be accessible to anyone, but > data files belong to specific modules. Searching the entire sys.path for > a data file seems way too broad; there's got to be some information > available as to where the data is. Eg. I doubt it's a *feature* that > pdb.doc may also be found in site-packages. I don't think that is a problem for packages, because imp.find_module() does not accept dotted names now. To import P.M you must currently import P, and then use P.__path__ as the path in a call to imp.find_module(). Presumably the package author knows her package path, and can use that in imp.find_module(). We aren't looking for data files in Python std lib, right? Hmmm. You have a point if the data is specific to a module and not to a package. It seems we are forced to use __file__, the only way to know the module's source. The __file__ can be /usr/lib/python22/lib/os.pyc /usr/lib/python/python22.zip/os.pyc so if we strip off "os.pyc" the rest can be used as the path. Hmmm. And if __file__ is None??? JimA From theller@python.net Fri Dec 20 20:52:41 2002 From: theller@python.net (Thomas Heller) Date: 20 Dec 2002 21:52:41 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: Message-ID: Just van Rossum writes: > Thomas Heller wrote: > > > Looks ok to me, although I would prefer to write > > > > path = os.sep.join(__name__.split(".") + ["wininst.exe"]) > > or > > path = os.path.join(*__name__.split(".") + ["wininst.exe"]) > > Much better. I think I'd prefer the first, mostly because os.path.join() > might do more magic than needed. Me too. Also because I wasn't sure first if I had to write path = os.path.join(*l1 + l2) or path = os.path.join(*(l1 + l2)) > > I think it's time to pin down the next extensions to the importer > protocol. How about the next speclet: > > bool = loader.is_package(fullname) > code = loader.get_code(fullname) > str = loader.get_source(fullname) > > (is_package and get_code are needed for modulefinder-like tools, > get_source is needed for linecache-like tools.) I like this much better than Jim's proposal, although I cannot think too deep currently about this. > > All three should raise ImportError if the module isn't available. > get_code and get_source should return None if there's no code or source > associated with the module, respectively. "No code" means it's a builtin > module or an extension. "No source" means just that (eg. for zipimport > it would mean "I've only got a .pyc or .pyo file for you"). > > If the importer does support get_source(), yet doesn't have a code > object readily available, get_code() should compile the source by itself > and return the code. This is to make life easier for the caller: if it > only needs the code object it shouldn't need to also check get_source() > and deal with that. > > They should be optional, yet strongly recommended for general purpose > hooks. There's no need to support them for hooks that are specific to a > deliverable app; for example zipimport should have them, but if Thomas > writes a dedicated hook for the apps built by py2exe, he should be free > to leave them out. > > Btw. would it be better if mod.__importer__ was named mod.__loader__? Yes. > This is closer to the truth, yet I fear that __loader__ is too generic. Before I can give better comments, I have to read the PEP it seems. Thomas From jim@interet.com Fri Dec 20 21:02:09 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 20 Dec 2002 16:02:09 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: Message-ID: <3E038551.6030502@interet.com> Just van Rossum wrote: > I didn't change anything there and I sure don't break any code, because > there's hardly any code out there that expects imp.find_module() to work > for zip files <0.5 wink>. Huh? In my implementation imp.find_module() imp.load_module() work for zip files. You mean they don't work to provide access to data files, right? >> find_module(name [, path [, suffixes]]) > I'm not sure if I like this: Me either, it was just an idea to solve a prominent problem. > I'll think about this some more, but I'm not yet convinced. Me too. JimA From just@letterror.com Fri Dec 20 22:19:30 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 20 Dec 2002 23:19:30 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: <3E038551.6030502@interet.com> Message-ID: James C. Ahlstrom wrote: > > I didn't change anything there and I sure don't break any code, > > because there's hardly any code out there that expects > > imp.find_module() to work for zip files <0.5 wink>. > > Huh? In my implementation imp.find_module() imp.load_module() > work for zip files. I know, but this is only possible when the import extension is hard coded in import.c, and lifting the need for hard coded hooks in import.c was the entire point of my exercise. This comes at the cost of it not working with imp.find_module(). > You mean they don't work to provide access to data files, right? Misunderstanding, I meant what I said. In my implementation imp.find_module() *doesn't* work for zip files, but I don't think that's such a big deal as zipimport is a new thing, or rather, the new import hooks are, erm, new. You've hit a weak spot in my scheme though, and perhaps the PEP should make that clearer. Nevertheless I see no decent way to integrate the importer protocol with imp.find_module/load_module. The way I see it (as I tried to say in the PEP) is that imp.find_module/load_module interfaces the builtin import mechanism as it is in Python 2.2 and before (builtin, frozen and plain directory imports, that's it). imp.load_module() is a primitive dispatcher that switches on the filetype enum field of a filedescr struct. What is needed with the new hooks is that the load_module() of a importer/loader object be called. This is a modernization (OO-ification) of the same idea, but obviously not compatible with the current imp module's world view. Hence my proposal to add a new function called imp.find_module2(), which also takes into account that the importer protocol needs the *full* module name -- another (even more unfixable, I think) incompatibility with imp.find_module(). (Btw. the only uses in the standard library of imp.find_module or imp.load_module are in ihooks.py imputil.py, pyclbr.py and pydoc.py. It's not like it's an insanely popular API, and I stick with my claim that I didn't break anything ;-) Just From lists@morpheus.demon.co.uk Fri Dec 20 23:32:26 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Fri, 20 Dec 2002 23:32:26 +0000 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: <3E037BFE.7020509@interet.com> Message-ID: "James C. Ahlstrom" writes: > It would look like this: > > suffix = [(".exe", "rb", 0)] > file, path, descr = imp.find_module("wininst", __path__, suffix) > data = file.read() > file.close() > return data It's an interesting idea, but like others, I think that it's not going to work right in practice. The main problem is that you don't explain how to implement this version of imp.find_module in terms of the importer protocol defined in the hooks PEP. > Note that for a zip file, __file__ is something like > C:/Python/lib/zarchive.zip/subdir/mymod.pyc > and nothing is going to make code using __file__ Just Work. That's not true. Under your patch, sure that's what __file__ is. Offhand, I'm not sure what Just's zipimporter code puts in there, and I certainly wouldn't guarantee it for an arbitrary implementation of zip imports. Frozen and builtin modules don't have usable __file__ attributes, and for something like a hook loading modules from a database, there is no meaningful __file__ value. > Well, maybe you could strip the last component "mymod.pyc" and > use it as __path__ in a call to imp.find_module(). Not too > hard, and it works with files and zip archives. But maybe not > with future ftp:// imported modules. And of course only with zip archives if you make imp.find_module() support that usage. This is a circular argument. > Your code knows the directory. My version would search in your > package __path__ for it. But you only have one wininst.exe per > package, right? And maybe putting it on package path is a feature. That's what the get_data(name) method Just is proposing is supposed to address. The only difficulty with it is pinning down what the "name" argument means. At the lowest level, it's an arbitrary cookie which identifies a chunk of data. The trick is to avoid making that cookie *too* unrelated to how things work in the filesystem... Paul. -- This signature intentionally left blank From Jack.Jansen@oratrix.com Sat Dec 21 13:57:24 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Sat, 21 Dec 2002 14:57:24 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: Message-ID: <22166EBA-14EC-11D7-ACA8-000A27B19B96@oratrix.com> On vrijdag, dec 20, 2002, at 21:19 Europe/Amsterdam, Just van Rossum wrote: > Thomas Heller wrote: > >> Looks ok to me, although I would prefer to write >> >> path = os.sep.join(__name__.split(".") + ["wininst.exe"]) >> or >> path = os.path.join(*__name__.split(".") + ["wininst.exe"]) > > Much better. I think I'd prefer the first, mostly because > os.path.join() > might do more magic than needed. But that magic would actually be needed for MacOS9 pathnames. os.path.join(*['foo', 'bar']) will correctly return ':foo:bar', whereas os.sep.join will return 'foo:bar', which is wrong. Not that we should care all that much anymroe about MacOS9 pathnames, but still... -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From just@letterror.com Sat Dec 21 14:38:39 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 21 Dec 2002 15:38:39 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: Message-ID: Paul Moore wrote: > > Note that for a zip file, __file__ is something like > > C:/Python/lib/zarchive.zip/subdir/mymod.pyc > > and nothing is going to make code using __file__ Just Work. > > That's not true. Under your patch, sure that's what __file__ > is. Offhand, I'm not sure what Just's zipimporter code puts in there, It actually does the same thing as Jim's patch regarding __file__. But he's right in both cases: using __file__ is will not "Just Work" ;-) > and I certainly wouldn't guarantee it for an arbitrary implementation > of zip imports. Frozen and builtin modules don't have usable __file__ > attributes, and for something like a hook loading modules from a > database, there is no meaningful __file__ value. Indeed. [ ... ] > That's what the get_data(name) method Just is proposing is supposed to > address. The only difficulty with it is pinning down what the "name" > argument means. At the lowest level, it's an arbitrary cookie which > identifies a chunk of data. The trick is to avoid making that cookie > *too* unrelated to how things work in the filesystem... I just wrote this to Paul in private mail: The 'name' argument of i.get_data(name) should be seen as a 'cookie', meaning the importer protocol doesn't prescribe any semantics for it. However, for importer objects that have some file system-like properties (for example zipimporter) it is recommended to use os.sep as a separator character to specify a (possibly virtual) directories hierarchy. For example if the importer allows access to a module's source code via i.get_data(name), the 'name' argument should be constructed like this: name = mod.__name__.replace(".", os.sep) + ".py" Note that this is not the recommended way to retrieve source code, the (optional) method i.get_source(fullname) is more general, as it doesn't imply *any* file-system-like characteristics. But in the light of Jack's remark regarding MacOS > Much better. I think I'd prefer the first, mostly because > > os.path.join() might do more magic than needed. > > But that magic would actually be needed for MacOS9 pathnames. > os.path.join(*['foo', 'bar']) will correctly return ':foo:bar', > whereas os.sep.join will return 'foo:bar', which is wrong. "needed" and "wrong" are highly questionalble in the context of importer.get_data()... Just From nas@python.ca Sun Dec 22 02:25:52 2002 From: nas@python.ca (Neil Schemenauer) Date: Sat, 21 Dec 2002 18:25:52 -0800 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: Message-ID: <20021222022552.GA12082@glacier.arctrix.com> Just van Rossum wrote: > This PEP proposes to add a new set of import hooks that offer better > customization of the Python import mechanism. Contrary to the > current __import__ hook, a new-style hook can be injected into the > existing scheme, allowing for a finer grained control of how modules > are found and how they are loaded. I only had time to skim this PEP. Currently I am using import hooks to load PTL modules (like .py modules but they have a different extension and need a different compiler). Most of the discussion has focused on loading .py modules from places other than filesystems. I was afraid my use case would not get addressed (iu.py doesn't really work, ihooks.py does). It looks like the hooks proposed by this PEP are sufficent for my needs. This PEP addresses a real problem, IMHO. The current __import__ hook is hard to use. There is too much complicated package import logic that must be reimplemented. Neil From just@letterror.com Sun Dec 22 18:07:52 2002 From: just@letterror.com (Just van Rossum) Date: Sun, 22 Dec 2002 19:07:52 +0100 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: <20021222022552.GA12082@glacier.arctrix.com> Message-ID: Neil Schemenauer wrote: > I only had time to skim this PEP. Currently I am using import hooks > to load PTL modules (like .py modules but they have a different > extension and need a different compiler). Most of the discussion has > focused on loading .py modules from places other than filesystems. I > was afraid my use case would not get addressed (iu.py doesn't really > work, ihooks.py does). It looks like the hooks proposed by this PEP > are sufficent for my needs. Interesting use case! I wish I had known about it earlier. Can you point me to your code using ihooks, if it's available? I'm actually not so sure the new hook mechanism solves all your problems as well as it could/should. I assume these special modules simply live on sys.path, intermixed with "regular" modules? This would mean you specifically need to extend the builtin sys.path/pkg.__path__ find/load logic, which is something the PEP doesn't cater for (yet). If the special modules live in a special folder, things are easier. Also: special modules will not work when packaged in a zip file (zipimport doesn't come *close* to transparently replacing the file system). If you compile to straight .py[co] (as I assume) and put those in the zip archive, you should be fine, though. This also touches on the modulefinder-like-tools-issue. Your hook should ideally support the get_code(fullname) method, so that packaging tools will be able to deal with it more or less transparently. Obviously the hook need to be installed when running modulefinder-like-tools. (Hmm, this suggests that an external hook registry might be handy, perhaps a *.pth-like trick...) Just From nas@python.ca Mon Dec 23 06:46:12 2002 From: nas@python.ca (Neil Schemenauer) Date: Sun, 22 Dec 2002 22:46:12 -0800 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) In-Reply-To: References: <20021222022552.GA12082@glacier.arctrix.com> Message-ID: <20021223064612.GA14325@glacier.arctrix.com> [I'm on a really slow link so this won't be detailed :-(] > Interesting use case! I wish I had known about it earlier. Can you > point me to your code using ihooks, if it's available? It's part of Quixote (google knows where). The module is ptl_import.py. > I'm actually not so sure the new hook mechanism solves all your > problems as well as it could/should. I assume these special modules > simply live on sys.path, intermixed with "regular" modules? That's the way the current hooks work. It probably would be okay if certain directorys could contain only .ptl modules. I'm not sure our users would like that incompatibilty. Obviously it would be nice if the new hooks could easily support the current behavior. Neil From bbum@codefab.com Mon Dec 23 16:58:49 2002 From: bbum@codefab.com (Bill Bumgarner) Date: Mon, 23 Dec 2002 11:58:49 -0500 Subject: [Python-Dev] ANN: PyObjC 0.8 Message-ID: PyObjC version 0.8 is now available for download. See... http://pyobjc.sourceforge.net/ ... for more information. The installer package includes a Project Builder project template for easily creating new Cocoa-Python projects and a slew of examples. Alternatively, the developer can choose to develop applications without using Apple's developer tools. Full support for Interface Builder documents is included. The PyObjC Python module provides a feature complete bridge between the Python programming language and Objective-C. With PyObjC, it is possible to transparently message Objective-C objects from Python and Python objects from Objective-C. Objective-C objects can be subclassed from within Python. Support is also included for using the various functions, enumerated types, and global variables found throughout the Objective-C framework. References to python dictionaries pass into Objective-C act like a well behaved member of the NSDictionary class cluster. Likewise, Python List references behave like a member of the NSArray class cluster. Strings are transparently converted to the native type. NSArray and NSDictionary references passed into Python behave like native Python lists and dictionaries; including the ability to enumerate either using the 'in' operator and to slice NSArray and NSMutableArray instances. PyObjC fully supports creating full featured Cocoa applications written in pure Python. There are aspects of PyObjC that are more powerful than Cocoa in pure Obj-C (the ability to automatically define classes/outlets based on the contents of a NIB file, for example). PyObjC requires OS X 10.2 or greater. 10.1 support is possible and will likely happen soon-- contact me if you need 10.1 support and are willing to do a bit of grunt work to generate the appropriate files (easy to do-- just need a 10.1 development machine). The installer package is designed to work with the built-in Python provided in OS X 10.2. Source is included on the disk image and the pyobjc module works with Python 2.2 or greater as installed directly from the Python Source, with the MacPython packages, and with the Fink build of python. PyObjC also provides an awesome environment for exploring frameworks. The following transcript was copied out of a Terminal window-- it is an example of working directly with the Objective-C runtime from Python within the terminal. >>> from objc import * >>> from Foundation import * >>> b = NSBundle.bundleWithPath_("/System/Library/PrivateFrameworks/ PBXCore.framework") >>> b.principalClass() >>> NSBundle.searchPathsForSupportFilesWithSubpath_("FooBar") ( "/Volumes/Data/Users/bbum/Developer/ProjectBuilder Extras/FooBar", "/Network/Developer/ProjectBuilder Extras/FooBar", "/Developer/ProjectBuilder Extras/FooBar" ) >>> PBXProject = lookUpClass("PBXProject") >>> p = PBXProject.projectWithFile_("/Developer/Examples/AppKit/TextEdit/ TextEdit.pbproj") >>> p.targets() () >>> p.buildStyles() ( , ) The version # of 0.8 indicates that documentation is lacking, unit tests are incomplete and there may be a bug or two lurking within. In practice, the module has proven to be very stable and to allow for the development of extremely complex Cocoa projects in pure Python or a mix of Python and Objective-C. From exarkun@meson.dyndns.org Tue Dec 24 07:09:41 2002 From: exarkun@meson.dyndns.org (Jp Calderone) Date: Tue, 24 Dec 2002 02:09:41 -0500 Subject: [Python-Dev] inet_ntop/inet_pton Message-ID: <20021224070941.GA26312@meson.dyndns.org> --ZPt4rx8FFjLCG7dd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I'm curious if 2.3 is going to make these two functions available in the socket module. There seems to be some support for them (or at least an attempt at functionality on platforms that don't support them) in socketmodule.c, but I don't see a Python interface for them. Is someone working on this? Should I work up a patch myself? :) Jp -- 12:00am up 8 days, 9:45, 3 users, load average: 0.05, 0.06, 0.01 --ZPt4rx8FFjLCG7dd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (GNU/Linux) iD8DBQE+CAg1edcO2BJA+4YRAubLAKDGav4gohL771bVdjOR0wzk3kL3SgCeJH0o 0hBl0ANf1vzvWl24VHc8PjM= =RcFu -----END PGP SIGNATURE----- --ZPt4rx8FFjLCG7dd-- From guido@python.org Tue Dec 24 19:30:34 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 24 Dec 2002 14:30:34 -0500 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 Message-ID: <200212241930.gBOJUYD10795@odiug.zope.com> I expect that I'll have more time for the Python 2.3 release the rest of this year, so I will do my darndest to do an alpha release before the year is over. I've partially updated PEP 283 (the release schedule), though there are still a lot of things to be done that aren't there. I'll get to these on Thursday. I've read PEP 302, and while there are some things left to discuss, I would really like to see Just's code checked in for the alpha release, so that it gets widespread testing. Just, are you up for that? Happy holidays everybody! --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Tue Dec 24 20:48:12 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 24 Dec 2002 21:48:12 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: <200212241930.gBOJUYD10795@odiug.zope.com> Message-ID: Guido van Rossum wrote: > I've read PEP 302, and while there are some things left to discuss, I > would really like to see Just's code checked in for the alpha release, > so that it gets widespread testing. Just, are you up for that? Sure ;-) I'll wait a few days to see if there are any objections or showstoppers. I updated the patch yesterday, there was a fairly large change in zipimport.c. If it were in CVS, this would have been the log msg: general: - incorporated patch from Paul Moore that adds a default zip archive path to sys.path on Windows (it was already there on unix). Thanks Paul! import.c: - matches latest version of PEP 302 (rev 1.3), regarding the new 'path' argument of importer.find_module(fullname, path=None) zipimporter.c - removed the subdir feature, which allowed the path to the zip archive to be extended with a subdirectory. PEP 273 stated this was needed for package support (and only for that). However, with the new import hooks this is no longer true: a path item containing the plain zip archive path can also deal with submodules (find_module receives the full module name after all). Therefore a pkg.__path__ from a package loaded from a zip archive will contain the *plain* zip archive path. - as a consequence I could simplify and clean up lots of things (esp. zipimporter_init: eg. it no longer needs to check sys.path_importer_cache; yay). Getting rid of the zipimporter.prefix attribute altogether helped a lot in other places. - this change additionally enabled me to get rid of the restriction that zip paths must end in .ZIP or .zip; any extension (or even no extension) will now work. - implemented all the (optional) extensions of the Importer Protocol that the latest version of PEP 302 defines. Merry Christmas everyone! Just From pedronis@bluewin.ch Tue Dec 24 21:09:32 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 24 Dec 2002 22:09:32 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: Message-ID: <023701c2ab90$c1736240$6d94fea9@newmexico> > zipimporter.c > - removed the subdir feature, which allowed the path to the > zip archive to be extended with a subdirectory. PEP 273 > stated this was needed for package support (and only for > that). However, with the new import hooks this is no longer > true: a path item containing the plain zip archive path can > also deal with submodules (find_module receives the full > module name after all). Therefore a pkg.__path__ from a > package loaded from a zip archive will contain the *plain* > zip archive path. > - as a consequence I could simplify and clean up lots of > things (esp. zipimporter_init: eg. it no longer needs to > check sys.path_importer_cache; yay). Getting rid of the > zipimporter.prefix attribute altogether helped a lot in > other places. > - this change additionally enabled me to get rid of the > restriction that zip paths must end in .ZIP or .zip; any > extension (or even no extension) will now work. will not this break __path__ manipulations? regards. From pedronis@bluewin.ch Tue Dec 24 22:45:40 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 24 Dec 2002 23:45:40 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: <023701c2ab90$c1736240$6d94fea9@newmexico> Message-ID: <02a801c2ab9e$2f6ab520$6d94fea9@newmexico> From: "Samuele Pedroni" > > zipimporter.c > > - removed the subdir feature, which allowed the path to the > > zip archive to be extended with a subdirectory. PEP 273 > > stated this was needed for package support (and only for > > that). However, with the new import hooks this is no longer > > true: a path item containing the plain zip archive path can > > also deal with submodules (find_module receives the full > > module name after all). Therefore a pkg.__path__ from a > > package loaded from a zip archive will contain the *plain* > > zip archive path. > > - as a consequence I could simplify and clean up lots of > > things (esp. zipimporter_init: eg. it no longer needs to > > check sys.path_importer_cache; yay). Getting rid of the > > zipimporter.prefix attribute altogether helped a lot in > > other places. > > - this change additionally enabled me to get rid of the > > restriction that zip paths must end in .ZIP or .zip; any > > extension (or even no extension) will now work. > > will not this break __path__ manipulations? > further this change is backward incompatible with what is allowed by Jython 2.1, it was considered a feature to be able to put path/to/a.zip/python in sys.path, so that the a.b package would be looked up under path/to/a.zip/python/a/b. regards. From just@letterror.com Wed Dec 25 09:03:45 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 25 Dec 2002 10:03:45 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: <02a801c2ab9e$2f6ab520$6d94fea9@newmexico> Message-ID: From: "Samuele Pedroni" > > zipimporter.c > > - removed the subdir feature, which allowed the path to the > > zip archive to be extended with a subdirectory. PEP 273 > > stated this was needed for package support (and only for > > that). However, with the new import hooks this is no longer > > true: a path item containing the plain zip archive path can > > also deal with submodules (find_module receives the full > > module name after all). Therefore a pkg.__path__ from a > > package loaded from a zip archive will contain the *plain* > > zip archive path. > > - as a consequence I could simplify and clean up lots of > > things (esp. zipimporter_init: eg. it no longer needs to > > check sys.path_importer_cache; yay). Getting rid of the > > zipimporter.prefix attribute altogether helped a lot in > > other places. > > - this change additionally enabled me to get rid of the > > restriction that zip paths must end in .ZIP or .zip; any > > extension (or even no extension) will now work. > > will not this break __path__ manipulations? If the particular manipulation did work for zip files at all before, yes :-(. (It wouldn't have worked with a Zip archive that was packed by a freeze-like tool, unless the *results* of the manipulations were explicitly flattened during packaging.) > further this change is backward incompatible with what is allowed by > Jython 2.1, > > it was considered a feature to be able to put > > path/to/a.zip/python > > in sys.path, so that the a.b package would be looked up under > > path/to/a.zip/python/a/b. PEP 273 doesn't document it as such; it only says it's needed for package imports. Also, to be honest, my implementation had some issues with that usage: it would look for the plain .zip archive in sys.path_importer_cache, which would obviously not be found, causing the zip file index to be read again for every package directory. The only solution I thought of that could solve that is for the zipimporter object to *add* entries to sys.path_importer_cache itself, and I found it bad enough already that it _read_ from the cache itself in the previous version. It's a messy feature :-(. I personally don't care about this feature; it's easy enough to package the archive so that it's not needed. Regarding the __path__ manipulations: this assumes file-system properties and can't work for importers in *general* and it feels like a hack to specially allow it for Zip archives (it definitely was a hack in my implementation, therefore I'm happy to get rid of it ;-). In many other respects Zip archives also won't be able to be compatible with a real file system anyway, eg. why should __path__ manipulations work and not __file__ manipulations? (Now if we had a virtual file system with Zip file support, things would be different!) I still think that __path__ manipulations are evil, as would be module-specific sys.path manipulation. To me, sys.path is the domain of *applications*, which implies that pkg.__path__ should be left alone also (at least by the package itself). It seems Guido is going the opposite direction with pkgutil.py :-(. Just From pedronis@bluewin.ch Wed Dec 25 12:38:33 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 25 Dec 2002 13:38:33 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: Message-ID: <001801c2ac12$89f4b620$6d94fea9@newmexico> From: "Just van Rossum" > I still think that __path__ manipulations are evil, as would be > module-specific sys.path manipulation. To me, sys.path is the domain of > *applications*, which implies that pkg.__path__ should be left alone > also (at least by the package itself). It seems Guido is going the > opposite direction with pkgutil.py :-(. Indeed it seemed that there was consensus to want them (messy, hackish or whatever they may seem). Honestly non considering the internals, the new __path__ conveys _less_ information that it could. That's the crucial point in my eyes. I'm -1 on the changes. regards. From just@letterror.com Wed Dec 25 14:16:39 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 25 Dec 2002 15:16:39 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: <001801c2ac12$89f4b620$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > Indeed it seemed that there was consensus to want [__path__ > manipulations] (messy, hackish or whatever they may seem). But, as I wrote, it *can't* work with importers in general. It can be made to work with specific importers, such as zipimporter, but I don't see the point (I'll rant about that in a second ;-). > Honestly non considering the internals, the new __path__ conveys > _less_ information that it could. That's the crucial point in my > eyes. For zipimporter, yes, you can see it that way. I just think it conveys *different* information. > I'm -1 on the changes. :-( I still have to see an example of a pkg.__path__ manipulation in which the intentions couldn't be solved in a different way, for example like how os.py imports platform-specific features. If os.py would have done that by munging sys.path everybody would have been appalled. The more I think about it, the more I think we should deprecate __path__ altogether. The path import mechanism code could just as well look for "package/submodule.py" on sys.path (which would in fact make pkgutils.py redundant as the feature it currently implements would then be standard behavior). That's a bit extreme and is perhaps not even doable in a 100% b/w compatible way, yet I do believe __path__ was a design mistake. http://www.python.org/doc/essays/packages.html doesn't explain the rationale as to why this particular solution was chosen; I _suspect_ it was simply an easy solution that left much of the lower level import logic unchanged. If that's true, then being able to modify pkg.__path__ is merely a side effect of the chosen solution. I could be wrong, though... Repackaging applications for use by end users (with freeze, py2exe, etc) should be easy and straightforward. Manipulating pkg.__path__ makes this harder than neccesary and that alone should be reason enough to strongly discourage it. Just From pedronis@bluewin.ch Wed Dec 25 16:06:43 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 25 Dec 2002 17:06:43 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: Message-ID: <00ec01c2ac2f$9e2720c0$6d94fea9@newmexico> From: "Just van Rossum" > Samuele Pedroni wrote: > > > Indeed it seemed that there was consensus to want [__path__ > > manipulations] (messy, hackish or whatever they may seem). > > But, as I wrote, it *can't* work with importers in general. It can be > made to work with specific importers, such as zipimporter, but I don't > see the point (I'll rant about that in a second ;-). but such importers would better go in meta_path. Btw, reading PEP 302 it seems that when dealing with a package, first *all* importers in sys.meta_path called with fullname and __path__ of the package. For symmetry (as long as we keep __path__) I think the import mechanism would have to look for a list in __meta__path__ (or maybe just a single value in __meta_importer__): if it's absent, thing should work as described above, if it's empty, sys.meta_path should be ignored, otherwise it's content should be used instead of sys.meta_path and __path__ should be ignored up to passing it to the meta importer(s). Basically this would mean that the contents of __path__ should be interpreted/used only by the meta importer. regards From just@letterror.com Wed Dec 25 21:42:45 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 25 Dec 2002 22:42:45 +0100 Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31) In-Reply-To: <00ec01c2ac2f$9e2720c0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > From: "Just van Rossum" > > > But, as I wrote, it *can't* work with importers in general. It can > > be made to work with specific importers, such as zipimporter, but I > > don't see the point (I'll rant about that in a second ;-). > > but such importers would better go in meta_path. Yes, but that doesn't make modifying __path__ less fragile ;-) > Btw, reading PEP 302 it seems that when dealing with a package, first > *all* importers in sys.meta_path called with > > fullname and __path__ of the package. Correct. > For symmetry (as long as we keep __path__) I think the import > mechanism would have to look for a list in __meta__path__ (or maybe > just a single value in __meta_importer__): Hm, you may have a point there. However, this makes life yet a bit harder for tools that only *find* modules, and not load them (eg. modulefinder.py). This is already a problem (even without the new hooks, but less so) with __path__: currently a modulefinder (let's use that term for a generic non-loading module finding tool) has to calculate what __path__ *would* be, were it actually loaded, and pass that as the second argument to imp.find_module(). For regular file system imports this is not such a big deal, but for importers that have no notion of a file system, yet use pkg.__path__, it is. The PEP proposes an optional method for loader objects named get_path(), that would return what __path__ would be set to were the package actually loaded. I'm not all that excited about the neccessity for this... pkg.__meta_path__ would make things yet more complicated (does it need to be passed to imp.find_module as well?). What you call __meta_importer__ might fit into the scheme, perhaps just as __importer__ (and rename what the PEP now defines as __importer__ to __loader__, which seems a good idea anyway). I think iu.py does something similar: it adds a __importsub__ symbol to packages, which is a function to import submodules. However, this all leads to a narrower search path for submodules. I want it to be *wider*. Back to __path__: about the only reason I can think for having it in the first place (besides being the "I'm a package" flag) is optimization: it avoids some stat calls by restricting the lookup to a specific directory. Yet there's the desire to import submodules from other places. Eg: loading .pyc modules from a zip archive, yet have dynamically loaded extensions come from the file system. Extension modules as submodules of frozen packages is currently not possible without deeper trickery: __path__ is a string in that case, so modifying it wouldn't be possible, let alone solve the problem. Let's assume for a moment we want to deprecate __path__. (Hm, I tried -- in my mind -- to also get rid of the "I'm a package" flag, but this is not realistic as long as we have implicit relative imports.) Submodules would be found on sys.path by constructing a relative path like "package/submodule.py". Here's a sample directory structure: dir1/spam/__init__.py dir1/spam/ham.py dir2/spam/eggs.pyd dir1 and dir2 are items on sys.path. "import spam.eggs" would first load "spam" from dir1/spam/__init__.py and then load "spam.eggs" from dir2/spam/eggs.pyd. It appears as if this is more or less what pkgutils.py tries to accomplish, yet brings the control of this feature back to the domain of the application, where it belongs. It also makes life easier for modulefinders: imp.find_module() could now be made to accept fully qualified module names (only when no 'path' argument is given). It would *also* allow me to get rid of the second argument of the find_module() method of the importer protocol as well as of the proposed imp.find_module2() function. It makes a lot of things simpler... Yet one more thing this scheme enables: extension modules as submodules in a multiplatform library setup (ie. different platforms using the same library on a server). Finding and loading of submodules would be completely decoupled from the parent module; parent and child could be imported by completely unrelated importers. The latter is already a *feature* of sys.meta_path (which explains why I'm not thrilled about Samuele's proposal ;-) and I would love for this to be extended to file system imports. Just From Jack.Jansen@oratrix.com Wed Dec 25 22:00:43 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Wed, 25 Dec 2002 23:00:43 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: Message-ID: <5085E098-1854-11D7-BC72-000A27B19B96@oratrix.com> On dinsdag, dec 24, 2002, at 21:48 Europe/Amsterdam, Just van Rossum wrote: > Guido van Rossum wrote: > >> I've read PEP 302, and while there are some things left to discuss, I >> would really like to see Just's code checked in for the alpha release, >> so that it gets widespread testing. Just, are you up for that? > > Sure ;-) I'll wait a few days to see if there are any objections or > showstoppers. Please don't wait too long, as I have no reasonable way to apply the patch on MacOS9, and I can imagine that MacPython-OS9 might encounter problems, with its own hooks in import.c. If you don't want to check it in on the trunk please do so on a branch. Or did you test MacPython-OS9 interaction yourself? -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From pedronis@bluewin.ch Wed Dec 25 22:02:08 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 25 Dec 2002 23:02:08 +0100 Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31) References: Message-ID: <049201c2ac61$454bdcc0$6d94fea9@newmexico> From: "Just van Rossum" > However, this all leads to a narrower search path for submodules. I want > it to be *wider*. Basically we want the same thing, but so either 1) __path__ manipulations are supported 2) or __path__ is deprecated and we get wide-importing by default (that means the union of all relevant "directories" is considered and not just one) indeed my use-case for 1) is to get 2). Both solutions are OK, and yes the 2nd simplifies some things. [ Then supporting path/to/a.zip/pkg1 in sys.path only would be still a feature and would be easier to support or just Jython could want to support it. ] My only remaining worry, once the above is sorted out, is whether the PEP302 scheme can be in the future expanded to support ihooks like functionalities (like those used by Quixote) in a non-messy way. regards. From just@letterror.com Wed Dec 25 22:48:59 2002 From: just@letterror.com (Just van Rossum) Date: Wed, 25 Dec 2002 23:48:59 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: <5085E098-1854-11D7-BC72-000A27B19B96@oratrix.com> Message-ID: Jack Jansen wrote: > Please don't wait too long, as I have no reasonable way to apply the > patch on MacOS9, and I can imagine that MacPython-OS9 might encounter > problems, with its own hooks in import.c. If you don't want to check > it in on the trunk please do so on a branch. > > Or did you test MacPython-OS9 interaction yourself? No, but the current version of the patch doesn't do any significant path munging, so I don't expect any problems. Nevertheless I'll send you the relevant files (with Mac line endings) in a separate email. Just From just@letterror.com Wed Dec 25 23:43:44 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 26 Dec 2002 00:43:44 +0100 Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31) In-Reply-To: <049201c2ac61$454bdcc0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > Basically we want the same thing, but so either > > 1) __path__ manipulations are supported 2) or __path__ is deprecated > and we get wide-importing by default (that means the union of all > relevant "directories" is considered and not just one) > > indeed my use-case for 1) is to get 2). > > Both solutions are OK, and yes the 2nd simplifies some things. Hey, a consensus ;-) Here's a concrete proposal: - deprecate pkg.__path__ *manipulation*, starting with Python 2.3a1. - pkg.__path__ will be set as before. - if len(pkg.__path__) > 1: issue DeprecationWarning search submodules on pkg.__path__, the old way. else: search submodules on sys.path, the new way. (Fancier: keep a copy of __path__ in a separate variable, say __orgpath__, issue a warning and invoke the old logic if __orgpath__ != __path__. This is robust even against alterations of __path__[0], but seems a little overkill to me.) This should be fully b/w compatible. > [ Then supporting path/to/a.zip/pkg1 in sys.path only would be still > a feature and would be easier to support or just Jython could want to > support it. ] Or: if Python ever grows a virtual file system, this feature could be readded. I'd prefer to leave it out for now. > My only remaining worry, once the above is sorted out, is whether the > PEP302 scheme can be in the future expanded to support ihooks like > functionalities (like those used by Quixote) in a non-messy way. Right now there are two ways to go about addressing the issue: A) turn/wrap the builtin Path importer into a real object on sys.meta_path. B) create a subset of ihooks.py as a new module (written in Python), reimplementing some of the path import logic. Either one should provide an API for adding file type handlers. (I'm not so sure ihooks even provides a straightforward way for multiple independent packages to each add their own file type handlers. For example I don't see how Quixote could currently be compatible with other packages doing similar things, short of chaining __import__ hooks. A new solution should have it as a major goal to make this possible.) B could be a prototype for A, so it seems logical to initially go for B. This doesn't (IMO) need to be in first alpha, so there's more time to experiment and discuss. Just From pedronis@bluewin.ch Thu Dec 26 00:14:30 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 26 Dec 2002 01:14:30 +0100 Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31) References: Message-ID: <059601c2ac73$c329aca0$6d94fea9@newmexico> From: "Just van Rossum" > > Both solutions are OK, and yes the 2nd simplifies some things. > > Hey, a consensus ;-) between the 2 of us, yes, let's wait what the others have to say. Deprecating __path__ is maybe a small vs big change but is a subtle one. > > My only remaining worry, once the above is sorted out, is whether the > > PEP302 scheme can be in the future expanded to support ihooks like > > functionalities (like those used by Quixote) in a non-messy way. > > Right now there are two ways to go about addressing the issue: > > A) turn/wrap the builtin Path importer into a real object on > sys.meta_path. > > B) create a subset of ihooks.py as a new module (written in Python), > reimplementing some of the path import logic. > > Either one should provide an API for adding file type handlers. (I'm not > so sure ihooks even provides a straightforward way for multiple > independent packages to each add their own file type handlers. For > example I don't see how Quixote could currently be compatible with other > packages doing similar things, short of chaining __import__ hooks. A new > solution should have it as a major goal to make this possible.) > > B could be a prototype for A, so it seems logical to initially go for B. > This doesn't (IMO) need to be in first alpha, so there's more time to > experiment and discuss. my view-point: when you bring ihooks-type functionality in the picture, I see the following separation of concerns: - opaque importers: fullname -> module - other importers: * the importers allow to fish for file-like objects of some types (corresponding to file extensions in filesystem-like cases) * there is a way to register new types and way to convert file-like objects into code/modules. instead of asking directly for a module, one would ask the importer for [.__init__] and e.g. types (module,code,source (py), bytecode (pyc), plt, pltc) [module and code are special types ] the importers would return some proxies for all relevant objects, the proxies would allow to possibly "open" and "read" the object and will present an optional timestamp. The converters would be used to make code/modules out of the objects. regards. From guido@python.org Thu Dec 26 01:15:00 2002 From: guido@python.org (Guido van Rossum) Date: Wed, 25 Dec 2002 20:15:00 -0500 Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31) In-Reply-To: Your message of "Thu, 26 Dec 2002 01:14:30 +0100." <059601c2ac73$c329aca0$6d94fea9@newmexico> References: <059601c2ac73$c329aca0$6d94fea9@newmexico> Message-ID: <200212260115.gBQ1F0207353@pcp02138704pcs.reston01.va.comcast.net> > Deprecating __path__ is maybe a small vs big change but is a subtle one. Especially when we just had a discussion in Zope3-dev where the conclusion was that we wanted to extend __path__ to solve a certain problem with separate distribution of self-contained subpackages of the zope package. I'll read the whole thread tomorrow (when I'm back to work) and will comment more then. --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Thu Dec 26 10:19:45 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Thu, 26 Dec 2002 11:19:45 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: Message-ID: <3E0AD7C1.4010400@worldonline.dk> [Samuele] >>further this change is backward incompatible with what is allowed by >>Jython 2.1, >> >>it was considered a feature to be able to put >> >>path/to/a.zip/python >> >>in sys.path, so that the a.b package would be looked up under >> >>path/to/a.zip/python/a/b. [Just] > PEP 273 doesn't document it as such; it only says it's needed for > package imports. Also, to be honest, my implementation had some issues > with that usage: it would look for the plain .zip archive in > sys.path_importer_cache, which would obviously not be found, causing the > zip file index to be read again for every package directory. The only > solution I thought of that could solve that is for the zipimporter > object to *add* entries to sys.path_importer_cache itself, and I found > it bad enough already that it _read_ from the cache itself in the > previous version. It's a messy feature :-(. > > I personally don't care about this feature; it's easy enough to package > the archive so that it's not needed. That may be true for CPython but it isn't for jython where the toplevel namespace in a zip file is taken already for java classes. By far the most natural way to package a jython/java application would be to put the jython modules in the same .zip (or .jar) file as the rest of the java application and that is best achieved by putting the python modules under a seperate directory in the zip file. So Jython cares about this feature. So mush that we may have to be incompatible on this point with CPython. regards, finn From just@letterror.com Thu Dec 26 12:06:10 2002 From: just@letterror.com (Just van Rossum) Date: Thu, 26 Dec 2002 13:06:10 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: <3E0AD7C1.4010400@worldonline.dk> Message-ID: Finn Bock wrote: > That may be true for CPython but it isn't for jython where the > toplevel namespace in a zip file is taken already for java classes. > By far the most natural way to package a jython/java application > would be to put the jython modules in the same .zip (or .jar) file as > the rest of the java application and that is best achieved by putting > the python modules under a seperate directory in the zip file. > > So Jython cares about this feature. So mush that we may have to be > incompatible on this point with CPython. I wonder, though, how big a deal is this incompatibility? Can Jython load modules from .pyc files in zip files? Can CPython import modules from .jar files? In other words, can zip/jar files ever be shared between Jython and CPython? Just From pedronis@bluewin.ch Thu Dec 26 12:10:33 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 26 Dec 2002 13:10:33 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: Message-ID: <002301c2acd7$cb1b3820$6d94fea9@newmexico> From: "Just van Rossum" > I wonder, though, how big a deal is this incompatibility? Can Jython > load modules from .pyc files in zip files? Can CPython import modules > from .jar files? In other words, can zip/jar files ever be shared > between Jython and CPython? the real problem is that in order to support this, if Jython/Python keep __path__, Jython has to put different values in there wrt to Python. If __path__ goes deprecated then it's just a sys.path contents issue, so what Jython does could be considered just a small extension. regards. From jim@interet.com Fri Dec 27 15:19:59 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 27 Dec 2002 10:19:59 -0500 Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #) References: <3E037BFE.7020509@interet.com> Message-ID: <3E0C6F9F.4060407@interet.com> Paul Moore wrote: > It's an interesting idea, but like others, I think that it's not going > to work right in practice. The main problem is that you don't explain > how to implement this version of imp.find_module in terms of the > importer protocol defined in the hooks PEP. I am exploring the idea of adding the feature to imp.find/load_module keeping the existing logic. I am not using the hooks PEP because I think the hooks PEP is a bad idea. Anyway, after some thought, I think my idea of a file-like object is not good. The easiest solution is to add a get_data() to go with find_module() and load_module(). The function signature of get_data() (or maybe "get_module()") is the same as that of load_module(), and the interface is: file, pathname, desc = imp.find_module(name, path=p, suffix=s) bytes = imp.get_data(name, file, pathname, desc) JimA From guido@python.org Fri Dec 27 17:59:52 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 27 Dec 2002 12:59:52 -0500 Subject: [Python-Dev] PEP 302 and __path__ In-Reply-To: Your message of "Thu, 26 Dec 2002 13:10:33 +0100." <002301c2acd7$cb1b3820$6d94fea9@newmexico> References: <002301c2acd7$cb1b3820$6d94fea9@newmexico> Message-ID: <200212271759.gBRHxqx21233@pcp02138704pcs.reston01.va.comcast.net> I've thought a lot about __path__, and my conclusion is that I want to keep it. It was put in as a feature of packages long ago (surviving a major redesign, when other features were dropped) and IMO serves an important purpose. For example, some 3rd party packages (e.g. PMW) use __path__ to select the most recently installed version when the package is imported. I don't like the idea of widening the package search path by default -- I can see all sorts of confusion when two versions of a package are on the path, and I prefer the (default) situation where the first one found hides the second one completely. That said, there are cases where a widened search path is desirable, and this can be achieved by explicit __path__ manipulation; that's why I recently added pkgutil.py (there's a potential use of this in Zope3). I also like the idea of allowing references to directories *inside* a zip archive (e.g. /path/to/foo.zip/dir/subdir/) on sys.path and on __path__. This seems pretty natural, and Jython already uses this. Using a smidgeon of caching, it should be pretty efficient to find the zip file in the path (even if it doesn't have a .zip extension; I don't think we should require that). Of course, there is an issue when the OS's path separator isn't the same as the zipfile's path separator; I think we should use the OS's path separator throughout and translate that to the zipfile's path separator (always "/" I believe?) internally. We should also silently strip a leading "/" on the paths used in the zipfile (i.e., it shouldn't matter if the zipfile index uses /foo/bar.py or foo/bar.py -- in both cases you could refer to this as /path/to/foo.zip/foo/bar.py. For directories, a trailing slash should also be optional (for files it should be illegal). Then __file__ for a package or module loaded from a zipfile should be set to the a path as above (e.g. /path/to/foo.zip/foo/bar.py) and __path__ for a package loaded from a zipfile should be initialized to e.g. ['/path/to/foo.zip/foo']. I'm not requiring *all* importers to follow this convention. It's fine if e.g. the "freeze" importer does something else (although I wouldn't be surprised if "freeze" ends up being deprecated once we have zip import as a standard feature). But for importers that map to something reasonably close to a (read-only) hierarchical file system, it seems useful to use OS-filename-like syntax in __file__ and __path__. Now, for such importers, it would be nice if we could use such paths to extract other data from the importer as well. I think that the right API for this would be some function living in the imp module: you pass it a path and it returns the data as a string, or raises an IOError (or subclass thereof) instance if it can't find the data. Let's call this API imp.get_data(filename). We'll see how it interfaces to importer/loader objects in a minute. I also would like to propose a new API to find modules: imp.get_loader(name[, path]). This should return a module loader object, or None if the module isn't found; it should raise an exception only if something unexpected went wrong. Once we have a module loader object, loader.load_module(name[, path]) should load (and return) the requested module. The name argument in both cases should be the fully dotted module name; the path argument should be omitted or None to load a toplevel module or package (no dot in the name), and it should be the package path to load a submodule or subpackage. Note that the package path can contain multiple entries. imp.get_loader() will have to probe each in turn until it finds one that has the requested module, and then return the corresponding loader. That loader, when its load_module() method is called, may use or ignore the path passed in. I don't want to add a separate meta-path to a package; it seems overkill, especially since we don't even have a good use case for meta-path in the normal case. I'm trying to write up pseudo-code to describe the whole setup more precisely, but it's taking more time than expected, so I'll send this mail with my intentions out first. --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@python.net Fri Dec 27 21:05:34 2002 From: gward@python.net (Greg Ward) Date: Fri, 27 Dec 2002 16:05:34 -0500 Subject: [Python-Dev] smtplib docs: has_option() or has_extn()? Message-ID: <20021227210534.GA5107@cthulhu.gerg.ca> Doc/lib/libsmtplib.tex refers to a mythical has_option() method of SMTP objects. (The string 'has_option' occurs nowhere in Lib/smtplib.py. Neither does 'hasoption'.) I'm pretty sure that it should be talking about has_extn(), ie.: --- lib/libsmtplib.tex 24 Mar 2002 15:41:40 -0000 1.22 +++ lib/libsmtplib.tex 27 Dec 2002 21:03:40 -0000 @@ -138,9 +138,9 @@ Identify yourself to an ESMTP server using \samp{EHLO}. The hostname argument defaults to the fully qualified domain name of the local host. Examine the response for ESMTP option and store them for use by -\method{has_option()}. +\method{has_extn()}. -Unless you wish to use \method{has_option()} before sending +Unless you wish to use \method{has_extn()} before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by \method{sendmail()} when necessary. \end{methoddesc} This look wrong to anyone? (Or is my CVS out-of-date?) Greg -- Greg Ward http://www.gerg.ca/ "Very funny, Scotty. Now beam my *clothes* down." From jim@interet.com Fri Dec 27 20:58:12 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 27 Dec 2002 15:58:12 -0500 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: <200212241930.gBOJUYD10795@odiug.zope.com> Message-ID: <3E0CBEE4.1020902@interet.com> Let's focus on PEP 302 as a new import hook mechanism, not as a way to implement zip imports. Just's PEP 302 code can be used to implement zip imports without making his import hooks part of the sys module and thus making them a public feature of Python forever. THIS IS NOT ABOUT ZIP IMPORTS! Python already has an import hook, namely __import__. PEP 302 adds three more hooks: sys.path_hooks, sys.meta_path and sys.path_importer_hooks. The idea of four import hooks is already fishy. PEP 302 enables non-strings on sys.path, and adds two new Python objects "importer" and "loader". It changes the meaning of imp.find_module() and imp.load_module(), and adds a new imp.find_module2(). It changes the meaning of __import__. It proposes to deprecate __path__ manipulations. That is a lot of external changes. That is a lot of code written in C. I think the proper import hook design is to write Python's import mechanism in Python along the lines of Greg's imputil.py and Gordon's iu.py. Import.c would be responsible for flat non-package imports from directories and zip files, including the import of the real importer iu.py. The imp module would be extended with simple C import utilities that can be used to speed up iu. Once iu.py is imported (probably from site), all imports are satisfied using the iu module. To provide custom import hooks, the user overrides the iu module by writing Python code. For example, site.py can attempt an import of custom_iu before importing iu, and the user provides custom_iu.py, probably by copying iu.py. I am not sure of the best way to do the override, but I am sure it is done in Python code. That enables the user to create custom hooks in Python, while relying on the source iu.py and the utilities in the imp module for basic functionality. If sys.path_hooks and the other hooks described in PEP 302 are approved, they can be implemented in iu.py in Python. This design still requires C support for zip imports, but there are two implementations for that available (from JimA and Just). Other problems such as bootstrapping are easily solved. I am tired of these endless import hook discussions, which always seem to start from nifty ideas instead of from a formal solicitation of hook requirements. I don't object to useful features, but I don't think anything other than easy replacement of the Python import mechanism will bring this perennial topic to an end. JimA From guido@python.org Fri Dec 27 22:15:23 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 27 Dec 2002 17:15:23 -0500 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: Your message of "Fri, 27 Dec 2002 15:58:12 EST." <3E0CBEE4.1020902@interet.com> References: <200212241930.gBOJUYD10795@odiug.zope.com> <3E0CBEE4.1020902@interet.com> Message-ID: <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net> > Let's focus on PEP 302 as a new import hook mechanism, not > as a way to implement zip imports. Just's PEP 302 code can > be used to implement zip imports without making his import > hooks part of the sys module and thus making them a public > feature of Python forever. THIS IS NOT ABOUT ZIP IMPORTS! Agreed. It's about making it easier to do things *like* zip import. > Python already has an import hook, namely __import__. PEP > 302 adds three more hooks: sys.path_hooks, sys.meta_path and > sys.path_importer_hooks. The idea of four import hooks is > already fishy. There's no path_importer_hooks; there's path_importer_cache, but that's not a new hook, it's a cache for the path_hooks hook. So I think the PEP proposes only two new hooks. Two new hooks is arguably still too much, but the existing __import__ hook is inadequate because it requires you to reimplement too much functionality (the PEP argues this point convincingly IMO). So we need at least one new hook. Personally, I think sys.meta_path is the lesser important of the two hooks proposed by the PEP. It would be needed if you have to override the standard builtin or frozen import behavior, but you can already do that with the heavier gun of overriding __import__. Other than that, you can hook up arbitrary weird importers by placing magic-cookie strings in sys.path (e.g. of the form "") and registering a path hook that looks for the magic cookie. (Long ago I had an idea to do this for builtin and frozen imports too, so you could control the relative priority of builtin/frozen modules relative to some directories. But I never found a use case. If someone wants this, though, it's easily added to the path_hooks feature.) Magic cookie strings are backwards compatible: from inspecting code that does something with sys.path, it looks like there are many places that assume sys.path contains only strings, but almost none that assume that all those strings are valid directory names. Typical code uses a sys.path item as input to os.path.isdir() or os.path.join() -- these require strings but don't make other assumptions. In the end there's usually something that passes the result to open() or os.stat() to see if it exists. Magic cookies will cause this test to fail, but the code is prepared for such failure -- however it's not prepared for TypeError coming out of a string concatenation. > PEP 302 enables non-strings on sys.path, and adds two new > Python objects "importer" and "loader". It changes the meaning > of imp.find_module() and imp.load_module(), and adds a new > imp.find_module2(). It changes the meaning of __import__. > It proposes to deprecate __path__ manipulations. Yes, this is definitely too much. I'd like to limit this to implementing sys.path_hooks -- there should be only one way to do it. We might still want to add one new API to imp, to access the new module-finding functionality (find_module2 is a poor choice of name though). > That is a lot of external changes. That is a lot of code > written in C. Um, last I looked, most of the code written in C was specific to the zip importer; only a relatively small amount of code was added to import.c (about 10%). If we get rid of the meta_path hook, it will be less; if we drop non-string objects on sys.path, less again. > I think the proper import hook design is to write Python's > import mechanism in Python along the lines of Greg's imputil.py > and Gordon's iu.py. Import.c would be responsible for flat > non-package imports from directories and zip files, including > the import of the real importer iu.py. The imp module would be > extended with simple C import utilities that can be used to > speed up iu. Once iu.py is imported (probably from site), > all imports are satisfied using the iu module. That's a design that I have had in mind long ago, but I don't see it happening soon, because it would be a much larger overhaul of import.c. Also, there are more risks: if import.c somehow can't find iu.py, it's hosed; and I fear that it could be a significant performance risk in its first implementation (I vaguely remember that Greg did some timing tests with imputil.py that confirmed this). > To provide custom import hooks, the user overrides the iu > module by writing Python code. For example, site.py can > attempt an import of custom_iu before importing iu, and the > user provides custom_iu.py, probably by copying iu.py. > I am not sure of the best way to do the override, but I am > sure it is done in Python code. That enables the user to > create custom hooks in Python, while relying on the source > iu.py and the utilities in the imp module for basic > functionality. More machinery that's not yet designed and implemented. I'd like to get something implemented by Monday, that supports zip import and *some* hookability. A scaled-down version of Just's code seems the only realistic possibility. > If sys.path_hooks and the other hooks described in PEP 302 > are approved, they can be implemented in iu.py in Python. > > This design still requires C support for zip imports, but > there are two implementations for that available (from JimA > and Just). Other problems such as bootstrapping are easily > solved. > > I am tired of these endless import hook discussions, which > always seem to start from nifty ideas instead of from a > formal solicitation of hook requirements. I don't object to > useful features, but I don't think anything other than easy > replacement of the Python import mechanism will bring this > perennial topic to an end. I'm not so sure. In practice, hooks are used for two things: import from other media than directories (e.g. zip files), and supporting additional filename extensions that trigger special transformations. But the latter need is much less common than the former (the only real example I know of is Quixote) and it's pretty much orthogonal to it. Neither should require you to replace __import__, as they currently do. (I've got to run now -- more over the weekend as my family gives me some time off. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Dec 27 21:27:23 2002 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 27 Dec 2002 16:27:23 -0500 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 References: Message-ID: <3E0CC5BB.8030200@interet.com> Just van Rossum wrote: > From: "Samuele Pedroni" >>it was considered a feature to be able to put >>path/to/a.zip/python >>in sys.path, so that the a.b package would be looked up under >>path/to/a.zip/python/a/b. > > PEP 273 doesn't document it as such; it only says it's needed for > package imports. The cornerstone of PEP 273 is Directory Equivalence, a zipped directory must act like the directory. And my implementation does provide this feature. > Also, to be honest, my implementation had some issues > with that usage: it would look for the plain .zip archive in > sys.path_importer_cache, which would obviously not be found, causing the > zip file index to be read again for every package directory. Mine reads the zip directory only once, and caches the whole directory tree. The payoff from the heavy mods to import.c, I guess. JimA From just@letterror.com Fri Dec 27 22:33:18 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 27 Dec 2002 23:33:18 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <3E0CBEE4.1020902@interet.com> Message-ID: James C. Ahlstrom wrote: > Let's focus on PEP 302 as a new import hook mechanism, not > as a way to implement zip imports. Just's PEP 302 code can > be used to implement zip imports without making his import > hooks part of the sys module and thus making them a public > feature of Python forever. THIS IS NOT ABOUT ZIP IMPORTS! Indeed it isn't. > Python already has an import hook, namely __import__. Yes, and the PEP explains clearly what the problems with __import__ (as a hook) are. > PEP 302 adds three more hooks: sys.path_hooks, sys.meta_path and > sys.path_importer_hooks. The latter does not exist. There are *two* new hooks: sys.meta_path and sys.path_hooks. Then there is sys.path_importer_cache, which caches the results of sys.path_hooks. > The idea of four import hooks is already fishy. Well, __import__ is fishy, the new hooks > PEP 302 enables non-strings on sys.path, and adds two new > Python objects "importer" and "loader". It changes the meaning > of imp.find_module() and imp.load_module(), and adds a new > imp.find_module2(). Yes. The latter remains to be seen (it's not even implemented yet), and I see Guido just proposed an alternative. I haven't had time yet to read his full post. > It changes the meaning of __import__. Where did you get that idea? > It proposes to deprecate __path__ manipulations. It certainly does not. *I* proposed that, but not in PEP 302. Guido is against it, so that settles that. > That is a lot of external changes. That is a lot of code > written in C. Except it's not a lot of code. > I think the proper import hook design is to write Python's > import mechanism in Python along the lines of Greg's imputil.py > and Gordon's iu.py. This is what iu.py *is*. But iu.py is more: while being a close reimplementation of all the semantic details, it's a better abstraction of the Python import mechanism than import.c is. PEP 302 attempts to expose the key benefits of this better abstraction to Python. > Import.c would be responsible for flat > non-package imports from directories and zip files, including > the import of the real importer iu.py. The imp module would be > extended with simple C import utilities that can be used to > speed up iu. Once iu.py is imported (probably from site), > all imports are satisfied using the iu module. You can do that now by simply using iu.py. > To provide custom import hooks, the user overrides the iu > module by writing Python code. For example, site.py can > attempt an import of custom_iu before importing iu, and the > user provides custom_iu.py, probably by copying iu.py. > I am not sure of the best way to do the override, but I am > sure it is done in Python code. That enables the user to > create custom hooks in Python, while relying on the source > iu.py and the utilities in the imp module for basic > functionality. > > If sys.path_hooks and the other hooks described in PEP 302 > are approved, they can be implemented in iu.py in Python. This is backwards: iu.py implements a superset of PEP 302 (with different details). So you have that now. > This design still requires C support for zip imports, but > there are two implementations for that available (from JimA > and Just). Other problems such as bootstrapping are easily > solved. > > I am tired of these endless import hook discussions, which > always seem to start from nifty ideas instead of from a > formal solicitation of hook requirements. PEP 302 started from the real requirements of a real use case: zipimport. Added to that are ideas taken from an extremely well though out model that was developed to solve real problems (iu.py). > I don't object to > useful features, but I don't think anything other than easy > replacement of the Python import mechanism will bring this > perennial topic to an end. This is no improvement. You can replace the import mechanism by overriding __import__. This sucks for most purposes as the import mechanism is very complicated and contains many subtle details and pitfalls. PEP 302 allows customizing *parts* of the import mechanism without having to deal with most of these pitfalls and complications. It allows completely independend components to add hooks that will work together seamlessy. This is not true for replacing __import__. Just From just@letterror.com Fri Dec 27 22:37:50 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 27 Dec 2002 23:37:50 +0100 Subject: [Python-Dev] Python 2.3a1 release -- Dec 31 In-Reply-To: <3E0CC5BB.8030200@interet.com> Message-ID: James C. Ahlstrom wrote: > Just van Rossum wrote: > > From: "Samuele Pedroni" > >>it was considered a feature to be able to put path/to/a.zip/python > >>in sys.path, so that the a.b package would be looked up under > >>path/to/a.zip/python/a/b. > > > > PEP 273 doesn't document it as such; it only says it's needed for > > package imports. > > The cornerstone of PEP 273 is Directory Equivalence, a zipped > directory must act like the directory. And my implementation does > provide this feature. > > > Also, to be honest, my implementation had some issues with that > > usage: it would look for the plain .zip archive in > > sys.path_importer_cache, which would obviously not be found, > > causing the zip file index to be read again for every package > > directory. > > Mine reads the zip directory only once, and caches the whole > directory tree. The payoff from the heavy mods to import.c, I guess. I had a long talk with Guido yesterday, and I'm going to go back to supporting the archive.zip/subdir/ feature as well as adding a proper archive.zip/subdir/ string to __path__. I hope to do this before monday, if my parents-in-law allow me ;-) Just From just@letterror.com Fri Dec 27 22:59:48 2002 From: just@letterror.com (Just van Rossum) Date: Fri, 27 Dec 2002 23:59:48 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > Two new hooks is arguably still too much, but the existing __import__ > hook is inadequate because it requires you to reimplement too much > functionality (the PEP argues this point convincingly IMO). So we > need at least one new hook. Personally, I think sys.meta_path is the > lesser important of the two hooks proposed by the PEP. It would be > needed if you have to override the standard builtin or frozen import > behavior, but you can already do that with the heavier gun of > overriding __import__. The sys.meta_path feature adds 32 lines to import.c, it's *really* useful and easy to use. I would be *extremely* sad to see it go. (I had an immediate use for it in test_importhooks.py: after a test is run I wanted to unload the modules that were imported during the test. The ImportTracker is a meta importer and only *records* imports. It's seven lines long and helped me clean up the test script quite a bit. Not a typical use case perhaps, but to me still demonstrates the power of meta_path quite well. Doing the same with __import__ is of course possible, but is much more cumbersome and doesn't have the same semantics.) > > PEP 302 enables non-strings on sys.path, and adds two new > > Python objects "importer" and "loader". It changes the meaning > > of imp.find_module() and imp.load_module(), and adds a new > > imp.find_module2(). It changes the meaning of __import__. > > It proposes to deprecate __path__ manipulations. > > Yes, this is definitely too much. Removing the importer-object-on-sys.path feature I could live with. It's clearly not needed, and for quick experimentation sys.meta_path is your pal (another reason I don't want to see it go). It's only there because it was easy: it's 17 lines in import.c. > I'd like to limit this to > implementing sys.path_hooks -- there should be only one way to do it. There should be only one way to do it if import.c did it only one way. Overriding or emulating builtin and frozen imports *must* be possible, and sys.meta_path is the feature for it. A key aspect of sys.meta_path is that it allows to refactor import.c in a really nice way; in the future, sys.meta_path would look like this: [BuiltinImporter, FrozenImporter, PathImporter] sys.meta_path is then the central import dispatcher. PathImporter would be responsible for sys.path/pkg.__path__ handling and invoking sys.path_hooks and dealing with sys.path_import_cache. It is *key* to PEP 302, and to me simply a killer feature. Just From lalo@laranja.org Fri Dec 27 23:03:41 2002 From: lalo@laranja.org (Lalo Martins) Date: Fri, 27 Dec 2002 21:03:41 -0200 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: References: <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20021227230341.GA15340@laranja.org> On Fri, Dec 27, 2002 at 11:59:48PM +0100, Just van Rossum wrote: > > The sys.meta_path feature adds 32 lines to import.c, it's *really* > useful and easy to use. I would be *extremely* sad to see it go. > > (I had an immediate use for it in test_importhooks.py: after a test is > run I wanted to unload the modules that were imported during the test. > The ImportTracker is a meta importer and only *records* imports. It's > seven lines long and helped me clean up the test script quite a bit. Not > a typical use case perhaps, but to me still demonstrates the power of > meta_path quite well. Doing the same with __import__ is of course > possible, but is much more cumbersome and doesn't have the same > semantics.) Just my 1.5c - PyUnit has a similar use case. It is documented (well) at http://pyunit.sourceforge.net/notes/reloading.html []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From macquigg@cadence.com Sat Dec 28 00:41:33 2002 From: macquigg@cadence.com (David MacQuigg) Date: Fri, 27 Dec 2002 16:41:33 -0800 Subject: [Python-Dev] linktree.py Message-ID: <9A77785A255522459FDD2D24C94B27D19DE7B8@exmbx02sj.cadence.com> Hello, I'm new to Python, and I'm very impressed with the potential of this = language to simplify much of the software my company provides. As a = first project, I'm looking to replace some tricky scripts written in csh = for setting up our tool hierarchies. An example is a script to = construct a "mock hierarchy" out of some patch files and links to a = production hierarchy. Our hierarchies are typically two gigabytes, = thousands of files, lots of strange links, many of which are broken. So = the script needs to be fairly robust. The functions in 'shutil' won't = do it. 'copytree' works, but the cost is a huge amount of disk space. = A mock hierarchy takes not much more space than the patch files = themselves. After a brief search on the Python website, I couldn't find a suitable = utility, and decided to write my own (see below). It seems like this = could be a useful, general-purpose addition to 'shutil'. I've tested this on a large hierarchy on a Solaris workstation, and it = seems to work. There are some dark corners which worry me however, so I = thought I would get some advice from the experts. Has anything like = this already been done or is it planned? Would a function like this be = a useful addition to 'shutil'? I'm intending to continue work on it, = and will be glad to contribute it to the library. import os, sys from os.path import * def linktree(oldtree, newtree): """linktree(oldtree, newtree) -> None Leave 'oldtree' undisturbed. Recursively add links to 'newtree', = until it looks just like 'oldtree', but with the new files patched in.=20 """ names =3D os.listdir(oldtree) print 'Candidates: ', names for name in names: oldpath =3D abspath(join(oldtree, name)) newpath =3D abspath(join(newtree, name)) if not exists(newpath) and not islink(newpath): if not exists(oldpath): # Don't link to a bad link. print '*** Error ***\nPath does not exist:\n', oldpath continue print 'New link: ', newpath os.symlink(oldpath, newpath) # OK if oldpath is a good link. elif isfile(newpath) and not islink(newpath): print 'As is: ', newpath=20 pass # Leave new files as is. elif isdir(newpath) and not islink(newpath): print 'Down one level \n', newpath linktree(oldpath, newpath) # Recursive call print 'Up one level' else: print '*** Error ***\nNot a file or directory:\n', newpath - Dave ************************************************************* * * David MacQuigg * email: macquigg@cadence.com * * * Principal Product Engineer * phone: USA 520-721-4583 * * * * Analog Artist * * * * * 9320 East Mikelyn Lane * * * * Cadence Design Systems, Inc. * Tucson, Arizona 85710 * ************************************************************* *=20 From guido@python.org Sat Dec 28 01:09:09 2002 From: guido@python.org (Guido van Rossum) Date: Fri, 27 Dec 2002 20:09:09 -0500 Subject: [Python-Dev] linktree.py In-Reply-To: Your message of "Fri, 27 Dec 2002 16:41:33 PST." <9A77785A255522459FDD2D24C94B27D19DE7B8@exmbx02sj.cadence.com> References: <9A77785A255522459FDD2D24C94B27D19DE7B8@exmbx02sj.cadence.com> Message-ID: <200212280109.gBS199508938@pcp02138704pcs.reston01.va.comcast.net> Sorry, you've reached the python *developers* list. For review of your code, you're better off in comp.lang.python. If you want to contribute a patch, please check here: http://www.python.org/dev/devfaq.html#patches --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Sat Dec 28 10:24:32 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 28 Dec 2002 11:24:32 +0100 Subject: [Python-Dev] PEP 302 and __path__ In-Reply-To: <200212271759.gBRHxqx21233@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > I also would like to propose a new API to find modules: > imp.get_loader(name[, path]). Hey, that's exactly what I proposed, with a better name ;-) > This should return a module loader > object, or None if the module isn't found; it should raise an > exception only if something unexpected went wrong. Once we have a > module loader object, loader.load_module(name[, path]) should load > (and return) the requested module. loader.load_module() doesn't need the optional path argument and is indeed spec'ed without it in the PEP. This is symmetrical with imp.load_module(). Just From pedronis@bluewin.ch Sat Dec 28 12:57:31 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 28 Dec 2002 13:57:31 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: Message-ID: <003d01c2ae70$af669120$6d94fea9@newmexico> From: "Just van Rossum" > > The sys.meta_path feature adds 32 lines to import.c, it's *really* > useful and easy to use. I would be *extremely* sad to see it go. > > (I had an immediate use for it in test_importhooks.py: after a test is > run I wanted to unload the modules that were imported during the test. > The ImportTracker is a meta importer and only *records* imports. ... I agree with Guido that special cookies can be used for normal importers. I see also the point for such really _meta_ importers. But if one has many of them how the code that installs them respectively the user decide on the order of invocation. It seems that theoretically to let code programatically alter meta_path, there's a need for categories/priorities for meta importers: - loggging importers come before everything - global importers after etc Lacking this, it seems equivalent to installing __import__ hooks apart a tiny bit of convenience. > There should be only one way to do it if import.c did it only one way. > Overriding or emulating builtin and frozen imports *must* be possible, > and sys.meta_path is the feature for it. A key aspect of sys.meta_path > is that it allows to refactor import.c in a really nice way; in the > future, sys.meta_path would look like this: > > [BuiltinImporter, FrozenImporter, PathImporter] > > sys.meta_path is then the central import dispatcher. PathImporter would > be responsible for sys.path/pkg.__path__ handling and invoking > sys.path_hooks and dealing with sys.path_import_cache. It is *key* to > PEP 302, and to me simply a killer feature. This would be nice, but if the above point is not resolved and this is not concretely implemented, we risk to have meta_path used in non-intended ways or be as much problematic as __import__. Lacking precise policies and clear-cut commitment for its future uses, it's better left out (it can be added in the future). From pedronis@bluewin.ch Sat Dec 28 13:31:14 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 28 Dec 2002 14:31:14 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: <003d01c2ae70$af669120$6d94fea9@newmexico> Message-ID: <000d01c2ae75$65030aa0$6d94fea9@newmexico> > > Lacking this, it seems equivalent to installing __import__ hooks apart a tiny > bit of convenience. > ok, not irrelevant detail, with meta_path hooks can be removed, still where to put a hook in meta_path is problematic. From just@letterror.com Sat Dec 28 13:37:34 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 28 Dec 2002 14:37:34 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <003d01c2ae70$af669120$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > I agree with Guido that special cookies can be used for normal > importers. I find this rather ugly, and would only do that if it's crucial that the pathless importer should be invoked somewhere in the middle of sys.path. And I don't have a use case for that. There are plenty of use cases for sys.meta_path. > I see also the point for such really _meta_ importers. But if one has > many of them how the code that installs them respectively the user > decide on the order of invocation. I have no idea what you mean by this. > It seems that theoretically to let code programatically alter > meta_path, there's a need for categories/priorities for meta > importers: > > - loggging importers come before everything - global importers after > etc > > Lacking this, it seems equivalent to installing __import__ hooks > apart a tiny bit of convenience. I disagree that it's only a "tiny bit" of convenience: - PEP-302-style importers don't have to deal with the "is this module already loaded" issue, it will only be invoked for new imports. - They don't have to deal with complete dotted name imports, they only get invoked for the head of a dotted name. The rules for dotted name imports are very subtle and are all taken care of by the surrounding code, so this really is a big win. > > There should be only one way to do it if import.c did it only one > > way. Overriding or emulating builtin and frozen imports *must* be > > possible, and sys.meta_path is the feature for it. A key aspect of > > sys.meta_path is that it allows to refactor import.c in a really > > nice way; in the future, sys.meta_path would look like this: > > > > [BuiltinImporter, FrozenImporter, PathImporter] > > > > sys.meta_path is then the central import dispatcher. PathImporter > > would be responsible for sys.path/pkg.__path__ handling and > > invoking sys.path_hooks and dealing with sys.path_import_cache. It > > is *key* to PEP 302, and to me simply a killer feature. > > This would be nice, but if the above point is not resolved Which point? > and this > is not concretely implemented, we risk to have meta_path used in > non-intended ways or be as much problematic as __import__. The new import hooks operate at *such* a different level that __import__ that I honestly don't see what the problem is. > Lacking precise policies and clear-cut commitment for its future > uses, it's better left out (it can be added in the future). I disagree. Better be able to *use* it and establish conventions based on actual usage than leave it out and fantasize about what it *could* do. PEP 302 should replace 99% of the use cases from __import__ and hard-coded hooks in import.c. Leaving out sys.meta_path would reduce this number significantly and would be a really dumb decision. Just From just@letterror.com Sat Dec 28 13:39:01 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 28 Dec 2002 14:39:01 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <000d01c2ae75$65030aa0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > > Lacking this, it seems equivalent to installing __import__ hooks > > apart a tiny bit of convenience. > > ok, not irrelevant detail, with meta_path hooks can be removed, still > where to put a hook in meta_path is problematic. I have no idea what you're talking about :-( Just From guido@python.org Sat Dec 28 13:48:24 2002 From: guido@python.org (Guido van Rossum) Date: Sat, 28 Dec 2002 08:48:24 -0500 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: Your message of "Sat, 28 Dec 2002 14:37:34 +0100." References: Message-ID: <200212281348.gBSDmOH26908@pcp02138704pcs.reston01.va.comcast.net> > Samuele Pedroni wrote: > > > I agree with Guido that special cookies can be used for normal > > importers. [Just] > I find this rather ugly, and would only do that if it's crucial that the > pathless importer should be invoked somewhere in the middle of sys.path. > And I don't have a use case for that. There are plenty of use cases for > sys.meta_path. Sure, but pleading for cookies is that they allow easier user control through PYTHONPATH, and/or through sys.path manipulation; cookies allow you to control precisely the import order. It seems they are indeed different features. Once we have path_hooks, cookies are a simple consequence; meta_path is an orthogonal feature. I agree with Samuele's concern that if you have multiple meta_path importers, ordering them may become an issue. On the other hand, ordering of entries in sys.path is a bit of a black art, but in 99% of the cases, insert-in-front or append-at-end seem suffiecient. I expect that the same will be true for sys.meta_path. (Though it would be a bit more convenient if indeed it included the built-in, frozen and path importers -- maybe we can get that done in alpha 2.) I agree with Just that either of these hooks is *much* more convenient than overriding __import__ -- because with __import__, you have to reimplement the whole look-in-sys.path-try-relative-try-absolute-import-parent-packages routine, unless you have a truly trivial use (like printing a log message and calling the real __import__). --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Sat Dec 28 14:06:56 2002 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 28 Dec 2002 15:06:56 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: <200212281348.gBSDmOH26908@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <006f01c2ae7a$61ffea80$6d94fea9@newmexico> From: "Guido van Rossum" > I agree with Samuele's concern that if you have multiple meta_path > importers, ordering them may become an issue. On the other hand, > ordering of entries in sys.path is a bit of a black art, but in 99% of > the cases, insert-in-front or append-at-end seem suffiecient. I > expect that the same will be true for sys.meta_path. the difference is that sys.path is really a deployment env issue and there are PYTHONPATH, site.py etc ... For meta_path it seems there are programmatical uses (testing frameworks etc), maybe a priority mechanism is overkill, maybe it would be useful. From mwh@python.net Sat Dec 28 14:53:17 2002 From: mwh@python.net (Michael Hudson) Date: 28 Dec 2002 14:53:17 +0000 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: Just van Rossum's message of "Sat, 28 Dec 2002 14:37:34 +0100" References: Message-ID: <2m8yya6vqa.fsf@starship.python.net> Just van Rossum writes: > I find this rather ugly, and would only do that if it's crucial that the > pathless importer should be invoked somewhere in the middle of sys.path. > And I don't have a use case for that. There are plenty of use cases for > sys.meta_path. I've not really been following this discussion with utmost care, but the name "meta_path" doesn't move me. Wouldn't sys.importers be better name? Or maybe I'm misunderstanding it's intent -- which is still an argument for a better name. Cheers, M. -- Also, remember to put the galaxy back when you've finished, or an angry mob of astronomers will come round and kneecap you with a small telescope for littering. -- Simon Tatham, ucam.chat, from Owen Dunn's review of the year From esr@thyrsus.com Sat Dec 28 16:08:12 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Sat, 28 Dec 2002 11:08:12 -0500 Subject: [Python-Dev] What's the magic formula for regression-testing a single module? Message-ID: <20021228160812.GA10268@thyrsus.com> What's the magic formula for regression-testing a single library module? I found a bug in ConfigParser -- the readfp() and write() functions are not perfect inverses in some situations. I think I have a fix, and want to verify correctness. -- Eric S. Raymond From neal@metaslash.com Sat Dec 28 16:25:22 2002 From: neal@metaslash.com (Neal Norwitz) Date: Sat, 28 Dec 2002 11:25:22 -0500 Subject: [Python-Dev] What's the magic formula for regression-testing a single module? In-Reply-To: <20021228160812.GA10268@thyrsus.com> References: <20021228160812.GA10268@thyrsus.com> Message-ID: <20021228162522.GY12063@epoch.metaslash.com> On Sat, Dec 28, 2002 at 11:08:12AM -0500, Eric S. Raymond wrote: > What's the magic formula for regression-testing a single library module? I typically do: ./python -E -tt ./Lib/test/regrtest.py test_strptime Replace test_strptime with the test(s) you want, presumably test_cfgparser. Neal From just@letterror.com Sat Dec 28 17:17:21 2002 From: just@letterror.com (Just van Rossum) Date: Sat, 28 Dec 2002 18:17:21 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <2m8yya6vqa.fsf@starship.python.net> Message-ID: Michael Hudson wrote: > I've not really been following this discussion with utmost care, but > the name "meta_path" doesn't move me. Wouldn't sys.importers be > better name? Or maybe I'm misunderstanding it's intent -- which is > still an argument for a better name. I don't like the name much either, but it _does_ fit quite well as the idea is that sys.path importing gets invoked through an item on meta_path. Perhaps this sketch helps a little: sys.meta_path = [ BuiltinImporter(), # handles builtin modules FrozenImporter(), # handles frozen modules PathImporter(), # handles sys.path ] def find_module(fullname, path=None): for i in sys.meta_path: l = i.find_module(fullname, path) if l is not None: return l return None class PathImporter: def find_module(self, fullname, path=None): if path is None: path = sys.path for p in path: i = get_path_importer(p) # deals with sys.path_hooks & cache if i is not None: l = i.find_module(fullname) if l is not None: return l retun None This is how you should look at it, even though it doesn't match the implementation (yet). Right now you should imagine those three objects on meta_path as an implicit extension of sys.meta_path, they don't yet physically exist. This should change in the near future, perhaps even with 2.3a2. Just From python@rcn.com Sat Dec 28 17:56:56 2002 From: python@rcn.com (python@rcn.com) Date: Sat, 28 Dec 2002 12:56:56 -0500 Subject: [Python-Dev] Mesenne Twister copyright notice Message-ID: During his review of the Mersenne Twister code (in python/nondist/sandbox/twister), Martin raised a question for python-dev about the best way to handle the copyright notice in the original code at http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html . I incorporated the copyright notice into the head of _random.c, but the notice seems to indicate that it should also be in the documentation (either in librandom.tex or random.py). I don't think having multiple copies of the copyright notice or having clutter in the LaTeX docs is desirable. Inclusion by reference or footnote may be sufficient. Do you guys have any thoughts on the best way to resolve this one? It is my only unanswered review comment for the module before loading it for the alpha release. Raymond Hettinger From python@rcn.com Sat Dec 28 17:53:41 2002 From: python@rcn.com (Raymond Hettinger) Date: Sat, 28 Dec 2002 12:53:41 -0500 Subject: [Python-Dev] Mersenne Twister copyright References: Message-ID: <005b01c2ae9a$a19b47a0$125ffea9@oemcomputer> During his review of the Mersenne Twister code (in python/nondist/sandbox/twister), Martin raised a question for python-dev about the best way to handle the copyright notice in the original code at http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html . I incorporated the copyright notice into the head of _random.c, but the notice seems to indicate that it should also be in the documentation (either in librandom.tex or random.py). I don't think having multiple copies of the copyright notice or having clutter in the LaTeX docs is desirable. Inclusion by reference or footnote may be sufficient. Do you guys have any thoughts on the best way to resolve this one? It is my only unanswered review comment for the module before loading it for the alpha release. Raymond Hettinger From mwh@python.net Sat Dec 28 17:52:43 2002 From: mwh@python.net (Michael Hudson) Date: 28 Dec 2002 17:52:43 +0000 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: Just van Rossum's message of "Sat, 28 Dec 2002 18:17:21 +0100" References: Message-ID: <2misxeav4k.fsf@starship.python.net> Just van Rossum writes: > Michael Hudson wrote: > > > I've not really been following this discussion with utmost care, but > > the name "meta_path" doesn't move me. Wouldn't sys.importers be > > better name? Or maybe I'm misunderstanding it's intent -- which is > > still an argument for a better name. > > I don't like the name much either, but it _does_ fit quite well as the > idea is that sys.path importing gets invoked through an item on > meta_path. Perhaps this sketch helps a little: > > > sys.meta_path = [ > BuiltinImporter(), # handles builtin modules > FrozenImporter(), # handles frozen modules > PathImporter(), # handles sys.path > ] [...] Right, that's what I thought was roughly the plan -- but I still don't like the name. In this world, sys.path is sort-of subsidiary to sys.meta_path -- which jars, to me at least. It makes sense with a bit of historical knowledge, but not without, IMHO. Of course, these mild gripes shouldn't delay the real work -- but names are important. Cheers, M. -- Remember - if all you have is an axe, every problem looks like hours of fun. -- Frossie -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From mal@lemburg.com Sat Dec 28 18:19:25 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 28 Dec 2002 19:19:25 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice References: Message-ID: <3E0DEB2D.9040609@lemburg.com> python@rcn.com wrote: > During his review of the Mersenne Twister code (in python/nondist/sandbox/twister), Martin raised a question for python-dev about the best way to handle the copyright notice in the original code at http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html . > > I incorporated the copyright notice into the head of _random.c, but the notice seems to indicate that it should also be in the documentation (either in librandom.tex or random.py). I don't think having multiple copies of the copyright notice or having clutter in the LaTeX docs is desirable. Inclusion by reference or footnote may be sufficient. > > Do you guys have any thoughts on the best way to resolve this one? It is my only unanswered review comment for the module before loading it for the alpha release. The usual way to deal with this is to include a .txt/.doc file in the module directory where the module is placed. Examples are the LICENSE.txt file and the various .doc files we already have in the std lib directory. I'd suggest to include a twister.txt file together with the module which then includes the copyright notice. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Sat Dec 28 22:10:21 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 28 Dec 2002 23:10:21 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0DEB2D.9040609@lemburg.com> References: <3E0DEB2D.9040609@lemburg.com> Message-ID: <3E0E214D.4060407@v.loewis.de> M.-A. Lemburg wrote: > The usual way to deal with this is to include a .txt/.doc file in the > module directory where the module is placed. Examples are the LICENSE.txt > file and the various .doc files we already have in the std lib directory. Can you give a few examples? I could not find a file LICENSE.txt (you probably meant LICENSE), and no .txt or .doc file I checked was relevant to licensing. > I'd suggest to include a twister.txt file together with the module > which then includes the copyright notice. My concern is that people who produce binary distributions will have to include this text file in their distribution as well. People who embed Python (in a way that includes _random.c) will have to include the text file in their applications. Do we really want this? Regards, Martin From tim.one@comcast.net Sat Dec 28 22:26:52 2002 From: tim.one@comcast.net (Tim Peters) Date: Sat, 28 Dec 2002 17:26:52 -0500 Subject: [Python-Dev] Mersenne Twister copyright In-Reply-To: <005b01c2ae9a$a19b47a0$125ffea9@oemcomputer> Message-ID: I'd like to see a LICENSES.txt file added to the distribution, into which we copy the licenses for all the external software we *may* ship with a Python distribution. On Windows that's quite a lot (zlib, bz2, Tcl/Tk, OpenSSL, ...). I'm not even thinking about the legalities here, it's just good manners to acknowledge other peoples' work. It's also good manners for people building on our work to pass on the licenses too, and most licenses formally require that they do so. > ... > Do you guys have any thoughts on the best way to resolve this > one? The issue isn't unique to you, so don't feel you have to solve it. > It is my only unanswered review comment for the module before > loading it for the alpha release. I expect to make a few hours to dig into the sandbox code tonight, but don't let that influence you . From mal@lemburg.com Sat Dec 28 22:58:24 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 28 Dec 2002 23:58:24 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice References: <3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de> Message-ID: <3E0E2C90.6060505@lemburg.com> Martin v. L=F6wis wrote: > M.-A. Lemburg wrote: >=20 >> The usual way to deal with this is to include a .txt/.doc file in the >> module directory where the module is placed. Examples are the LICENSE.= txt >> file and the various .doc files we already have in the std lib directo= ry. >=20 >=20 > Can you give a few examples? I could not find a file LICENSE.txt (you=20 > probably meant LICENSE), and no .txt or .doc file I checked was relevan= t=20 > to licensing. Have a look at the /usr/local/lib/python2.2/ dir. The LICENSE.txt file has the Python copyright notice. profile.doc has the license for the profiler. >> I'd suggest to include a twister.txt file together with the module >> which then includes the copyright notice. >=20 > My concern is that people who produce binary distributions will have to= =20 > include this text file in their distribution as well. People who embed=20 > Python (in a way that includes _random.c) will have to include the text= =20 > file in their applications. >=20 > Do we really want this? If that's what the licenses want, there's no way to drive around this I guess. Note that people redistributing the code can also take these .txt files and add them to their application documentation to meet the license requirements. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Sat Dec 28 23:13:08 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 29 Dec 2002 00:13:08 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0E2C90.6060505@lemburg.com> References: <3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de> <3E0E2C90.6060505@lemburg.com> Message-ID: <3E0E3004.9000900@v.loewis.de> M.-A. Lemburg wrote: > Have a look at the /usr/local/lib/python2.2/ dir. The LICENSE.txt > file has the Python copyright notice. Ah, I see. It is renamed to LICENSE.txt when installed. > profile.doc has the license for the profiler. Right. It appears that ActivePython is breaking InfoSeeks copyright, by not including this file in their distribution... > If that's what the licenses want, there's no way to drive around > this I guess. There are certainly various options: a) we could refuse to incorporate the code in question, or postpone that until an alternative implementation comes along that does not have such requirements, or b) we could try to contact the authors of the code in question, to find out whether they would accept other means of acknowledgement (e.g. mentioning their names in the ACKS file, and putting some text into the copied source - as Raymond already did) It is certainly the case that we haven't been very careful with such things in the past (in particular in binary distributions), but that is IMO no excuse to be more careful now. Regards, Martin From Jack.Jansen@oratrix.com Sat Dec 28 23:58:49 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Sun, 29 Dec 2002 00:58:49 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0E3004.9000900@v.loewis.de> Message-ID: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com> On zondag, dec 29, 2002, at 00:13 Europe/Amsterdam, Martin v. L=F6wis=20 wrote: > a) we could refuse to incorporate the code in question, or postpone=20 > that until an alternative implementation comes along that does not=20 > have such requirements, or This sounds like overkill. > > b) we could try to contact the authors of the code in question, to=20 > find out whether they would accept other means of acknowledgement=20 > (e.g. mentioning their names in the ACKS file, and putting some text=20= > into the copied source - as Raymond already did) This is probably a good idea. Alternatively, there's another option: c) Add a special directory (Misc/Licenses comes to mind, copy it to=20 lib/python2.3/config/Licenses on install) that has the text files for=20 the various packages that require their license to be in a file of its=20= own. We could even get all fancy, and install only those licenses for=20 which the corresponding code is actually used. > It is certainly the case that we haven't been very careful with such=20= > things in the past (in particular in binary distributions), but that=20= > is IMO no excuse to be more careful now. MacPython has always been fairly careful about this, and I wouldn't be=20= surprised if the same holds for PythonWin. -- - Jack Jansen =20 http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma=20 Goldman - From martin@v.loewis.de Sun Dec 29 00:24:12 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 29 Dec 2002 01:24:12 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com> References: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com> Message-ID: <3E0E40AC.6040906@v.loewis.de> Jack Jansen wrote: > MacPython has always been fairly careful about this, and I wouldn't be > surprised if the same holds for PythonWin. As Tim points out, the Windows distribution incorporates a number of add-on libraries with their own copying conditions, which aren't always followed. For example, the db 1.85 package which was the basis of bsddb on Windows still has the original Berkeley advertising clause (which was waived now); it also has the requirement that The Regents of the University of California are acknowledged and their license terms be included in binary-only distributions; I don't think this requirement was followed. The same holds for other copies of the BSD license, e.g. the one in getaddrinfo.c requires that the WIDE project is acknowledged (I'm guilty of incorporating that one into Python). pcre-int.h says that the origin of PCRE must not be misrepresented by omission [of Philip Hazel's copyright]. regexpr.c says that Tatu Ylonen's text must appear in all copies. xmlrpclib.py says that Secret Lab's copyright text must be mentioned in documentation. And so on. Does MacPython acknowledge all these people and organizations, in the documentation? Regards, Martin From mal@lemburg.com Sun Dec 29 12:33:41 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 29 Dec 2002 13:33:41 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice References: <3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de> <3E0E2C90.6060505@lemburg.com> <3E0E3004.9000900@v.loewis.de> Message-ID: <3E0EEBA5.1020803@lemburg.com> Martin v. L=F6wis wrote: > M.-A. Lemburg wrote: >=20 >> Have a look at the /usr/local/lib/python2.2/ dir. The LICENSE.txt >> file has the Python copyright notice. >=20 > Ah, I see. It is renamed to LICENSE.txt when installed. >=20 >> profile.doc has the license for the profiler. > =20 > Right. It appears that ActivePython is breaking InfoSeeks copyright, by= =20 > not including this file in their distribution... The license is also copied in the documentation for that module, so I believe everything is OK. >> If that's what the licenses want, there's no way to drive around >> this I guess.=20 >=20 > There are certainly various options: >=20 > a) we could refuse to incorporate the code in question, or postpone tha= t=20 > until an alternative implementation comes along that does not have such= =20 > requirements, or -1 Adding a few license text files to the module directory or a separate section of the Python documentation shouldn't keep us from including the code. > b) we could try to contact the authors of the code in question, to find= =20 > out whether they would accept other means of acknowledgement (e.g.=20 > mentioning their names in the ACKS file, and putting some text into the= =20 > copied source - as Raymond already did) -0 While this would certainly be a much better solution, I don't think it is required. Instead, we should come up with a way which satisfies the licenses. Since most licenses require a copy of the text to be included in the documentation, I think the easiest and most reliable way of achieving this is to add a new licenses section to the documentation and Python's LICENSE file itself (Latex allows including text from other files in the docs, so adding new licenses to the LICENSE file would suffice if LICENSE is included verbatim in the Latex docs). > It is certainly the case that we haven't been very careful with such=20 > things in the past (in particular in binary distributions), but that is= =20 > IMO no excuse to be more careful now. Agreed. Since you have already scanned much of the code, perhaps you could summarize your license findings in a new sandbox file until we have come up with a solution that satisfies everyone ?! --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Sun Dec 29 12:43:43 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 29 Dec 2002 13:43:43 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0EEBA5.1020803@lemburg.com> References: <3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de> <3E0E2C90.6060505@lemburg.com> <3E0E3004.9000900@v.loewis.de> <3E0EEBA5.1020803@lemburg.com> Message-ID: <3E0EEDFF.7020402@v.loewis.de> M.-A. Lemburg wrote: > Since you have already scanned much of the code, perhaps you > could summarize your license findings in a new sandbox file > until we have come up with a solution that satisfies everyone ?! > That scan was not exhaustive; if anybody would collect this information in a systematic manner, that would be much appreciated. Regards, Martin From skip@manatee.mojam.com Sun Dec 29 13:00:18 2002 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 29 Dec 2002 07:00:18 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200212291300.gBTD0Irc016025@manatee.mojam.com> Bug/Patch Summary ----------------- 326 open / 3145 total bugs (+4) 104 open / 1855 total patches (+11) New Bugs -------- email: huge address lines blow stack (2002-12-15) http://python.org/sf/654362 make errors better like cgi (2002-12-16) http://python.org/sf/654558 asyncore.py and "handle_expt" (2002-12-16) http://python.org/sf/654766 doctest and exception messages (2002-12-16) http://python.org/sf/654783 datetime docs need review, LaTeX (2002-12-16) http://python.org/sf/654846 Slightly modify locals() doc? (2002-12-17) http://python.org/sf/655271 cPickle not always same as pickle (2002-12-18) http://python.org/sf/655802 binascii.a2b_base64 with non base64 data (2002-12-19) http://python.org/sf/656392 gettext.py crash on bogus preamble (2002-12-24) http://python.org/sf/658233 Accept None for time.ctime() and friends (2002-12-24) http://python.org/sf/658254 cPickle does relative import (2002-12-26) http://python.org/sf/658693 asyncore connect() and winsock errors (2002-12-26) http://python.org/sf/658749 no docs for HTMLParser.handle_pi (2002-12-27) http://python.org/sf/659188 'realpath' function missing from os.path (2002-12-27) http://python.org/sf/659228 optparse store_true uses 1 and 0 (2002-12-28) http://python.org/sf/659604 New Patches ----------- gzip shouldn't ValueError on corruptfile (2002-12-15) http://python.org/sf/654421 Fix bug in IE/CGI [bug 427345] (2002-12-16) http://python.org/sf/654910 commands.py for Windows (2002-12-17) http://python.org/sf/655421 Add warnings to unsafe Cookie classes (2002-12-18) http://python.org/sf/655760 /dev/ptmx support for ptys (cygwin) (2002-12-19) http://python.org/sf/656590 Documentation support for PEP 301 (2002-12-23) http://python.org/sf/658093 PEP 301 implementation (2002-12-23) http://python.org/sf/658094 Mersenne Twister (2002-12-24) http://python.org/sf/658251 skips.txt for regrtest.py (2002-12-24) http://python.org/sf/658316 Add inet_pton and inet_ntop to socket (2002-12-24) http://python.org/sf/658327 email: minimal header encoding (2002-12-24) http://python.org/sf/658407 Fix for bug 494589 (2002-12-25) http://python.org/sf/658599 regex fixes for _strptime (2002-12-26) http://python.org/sf/658820 posixpath missing getctime (2002-12-27) http://python.org/sf/658927 Use PyArg_UnpackTuple where possible (2002-12-28) http://python.org/sf/659536 Closed Bugs ----------- Python 2.1 installs shared libs with mode 0700 (2001-05-17) http://python.org/sf/425007 Left to right (2001-08-06) http://python.org/sf/448679 unreliable file.read() error handling (2002-02-23) http://python.org/sf/521782 lib-dynload/*.so wrong permissions (2002-07-17) http://python.org/sf/583206 property missing doc'd __name__ attr (2002-09-22) http://python.org/sf/612969 missing DECREF's in embedding example (2002-10-21) http://python.org/sf/626275 ftplib bails out on empty responses (2002-11-11) http://python.org/sf/636685 Undocumented side effect of eval (2002-11-20) http://python.org/sf/641111 property() builtin not documented (2002-11-23) http://python.org/sf/642742 .extend() doc, .__doc__ too restrictive (2002-11-29) http://python.org/sf/645777 Closed Patches -------------- install lib-dynload .so files mode 555 (2002-10-26) http://python.org/sf/629278 Fix for major rexec bugs (2002-11-11) http://python.org/sf/636769 Allow keyword args in dict.update() (2002-11-25) http://python.org/sf/643565 Set Next Statement for Python debuggers (2002-11-25) http://python.org/sf/643835 From max@malva.ua Sun Dec 29 13:20:38 2002 From: max@malva.ua (Max Ischenko) Date: Sun, 29 Dec 2002 15:20:38 +0200 Subject: [Python-Dev] Encoding file for KOI8-U Message-ID: <20021229132038.GA8795@malva.ua> --zYM0uCDKw75PZbzx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I wonder, whether koi8-u.py would be included in Python 2.3? If not, I could submit one. Though I'm not the author, it seems correct. -- Bst rgrds, M.A.X.: Mechanical Artificial Xenomorph. --zYM0uCDKw75PZbzx Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="koi8_u.py" """ Python Character Mapping Codec is modified version of 'koi8_r.py'. This character scheme compliant to RFC2319 Written by Marc-Andre Lemburg (mal@lemburg.com). Modified by Maxim Dzumanenko . (c) Copyright CNRI, All Rights Reserved. NO WARRANTY. (c) Copyright 2000 Guido van Rossum. """#" import codecs ### Codec APIs class Codec(codecs.Codec): def encode(self,input,errors='strict'): return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): return codecs.charmap_decode(input,errors,decoding_map) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): pass ### encodings module API def getregentry(): return (Codec().encode,Codec().decode,StreamReader,StreamWriter) ### Decoding Map decoding_map = codecs.make_identity_dict(range(256)) decoding_map.update({ 0x0080: 0x2500, # BOX DRAWINGS LIGHT HORIZONTAL 0x0081: 0x2502, # BOX DRAWINGS LIGHT VERTICAL 0x0082: 0x250C, # BOX DRAWINGS LIGHT DOWN AND RIGHT 0x0083: 0x2510, # BOX DRAWINGS LIGHT DOWN AND LEFT 0x0084: 0x2514, # BOX DRAWINGS LIGHT UP AND RIGHT 0x0085: 0x2518, # BOX DRAWINGS LIGHT UP AND LEFT 0x0086: 0x251C, # BOX DRAWINGS LIGHT VERTICAL AND RIGHT 0x0087: 0x2524, # BOX DRAWINGS LIGHT VERTICAL AND LEFT 0x0088: 0x252C, # BOX DRAWINGS LIGHT DOWN AND HORIZONTAL 0x0089: 0x2534, # BOX DRAWINGS LIGHT UP AND HORIZONTAL 0x008a: 0x253C, # BOX DRAWINGS LIGHT VERTICAL AND HORIZONTAL 0x008b: 0x2580, # UPPER HALF BLOCK 0x008c: 0x2584, # LOWER HALF BLOCK 0x008d: 0x2588, # FULL BLOCK 0x008e: 0x258C, # LEFT HALF BLOCK 0x008f: 0x2590, # RIGHT HALF BLOCK 0x0090: 0x2591, # LIGHT SHADE 0x0091: 0x2592, # MEDIUM SHADE 0x0092: 0x2593, # DARK SHADE 0x0093: 0x2320, # TOP HALF INTEGRAL 0x0094: 0x25A0, # BLACK SQUARE 0x0095: 0x2219, # BULLET OPERATOR 0x0096: 0x221A, # SQUARE ROOT 0x0097: 0x2248, # ALMOST EQUAL TO 0x0098: 0x2264, # LESS THAN OR EQUAL TO 0x0099: 0x2265, # GREATER THAN OR EQUAL TO 0x009a: 0x00A0, # NO-BREAK SPACE 0x009b: 0x2321, # BOTTOM HALF INTEGRAL 0x009c: 0x00B0, # DEGREE SIGN 0x009d: 0x00B2, # SUPERSCRIPT DIGIT TWO 0x009e: 0x00B7, # MIDDLE DOT 0x009f: 0x00F7, # DIVISION SIGN 0x00a0: 0x2550, # BOX DRAWINGS DOUBLE HORIZONTAL 0x00a1: 0x2551, # BOX DRAWINGS DOUBLE VERTICAL 0x00a2: 0x2552, # BOX DRAWINGS DOWN SINGLE AND RIGHT DOUBLE 0x00a3: 0x0451, # CYRILLIC SMALL LETTER IO 0x00a4: 0x0454, # CYRILLIC SMALL LETTER UKRAINIAN IE 0x00a5: 0x2554, # BOX DRAWINGS DOUBLE DOWN AND RIGHT 0x00a6: 0x0456, # CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I 0x00a7: 0x0457, # CYRILLIC SMALL LETTER YI (Ukrainian) 0x00a8: 0x2557, # BOX DRAWINGS DOUBLE DOWN AND LEFT 0x00a9: 0x2558, # BOX DRAWINGS UP SINGLE AND RIGHT DOUBLE 0x00aa: 0x2559, # BOX DRAWINGS UP DOUBLE AND RIGHT SINGLE 0x00ab: 0x255A, # BOX DRAWINGS DOUBLE UP AND RIGHT 0x00ac: 0x255B, # BOX DRAWINGS UP SINGLE AND LEFT DOUBLE 0x00ad: 0x0491, # CYRILLIC SMALL LETTER GHE WITH UPTURN 0x00ae: 0x255D, # BOX DRAWINGS DOUBLE UP AND LEFT 0x00af: 0x255E, # BOX DRAWINGS VERTICAL SINGLE AND RIGHT DOUBLE 0x00b0: 0x255F, # BOX DRAWINGS VERTICAL DOUBLE AND RIGHT SINGLE 0x00b1: 0x2560, # BOX DRAWINGS DOUBLE VERTICAL AND RIGHT 0x00b2: 0x2561, # BOX DRAWINGS VERTICAL SINGLE AND LEFT DOUBLE 0x00b3: 0x0401, # CYRILLIC CAPITAL LETTER IO 0x00b4: 0x0404, # CYRILLIC CAPITAL LETTER UKRAINIAN IE 0x00b5: 0x2563, # DOUBLE VERTICAL AND LEFT 0x00b6: 0x0406, # CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I 0x00b7: 0x0407, # CYRILLIC CAPITAL LETTER YI (Ukrainian) 0x00b8: 0x2566, # BOX DRAWINGS DOUBLE DOWN AND HORIZONTAL 0x00b9: 0x2567, # BOX DRAWINGS UP SINGLE AND HORIZONTAL DOUBLE 0x00ba: 0x2568, # BOX DRAWINGS UP DOUBLE AND HORIZONTAL SINGLE 0x00bb: 0x2569, # BOX DRAWINGS DOUBLE UP AND HORIZONTAL 0x00bc: 0x256A, # BOX DRAWINGS VERTICAL SINGLE AND HORIZONTAL DOUBLE 0x00bd: 0x0490, # CYRILLIC CAPITAL LETTER GHE WITH UPTURN 0x00be: 0x256C, # BOX DRAWINGS DOUBLE VERTICAL AND HORIZONTAL 0x00bf: 0x00A9, # COPYRIGHT SIGN 0x00c0: 0x044E, # CYRILLIC SMALL LETTER YU 0x00c1: 0x0430, # CYRILLIC SMALL LETTER A 0x00c2: 0x0431, # CYRILLIC SMALL LETTER BE 0x00c3: 0x0446, # CYRILLIC SMALL LETTER TSE 0x00c4: 0x0434, # CYRILLIC SMALL LETTER DE 0x00c5: 0x0435, # CYRILLIC SMALL LETTER IE 0x00c6: 0x0444, # CYRILLIC SMALL LETTER EF 0x00c7: 0x0433, # CYRILLIC SMALL LETTER GHE 0x00c8: 0x0445, # CYRILLIC SMALL LETTER HA 0x00c9: 0x0438, # CYRILLIC SMALL LETTER I 0x00ca: 0x0439, # CYRILLIC SMALL LETTER SHORT I 0x00cb: 0x043A, # CYRILLIC SMALL LETTER KA 0x00cc: 0x043B, # CYRILLIC SMALL LETTER EL 0x00cd: 0x043C, # CYRILLIC SMALL LETTER EM 0x00ce: 0x043D, # CYRILLIC SMALL LETTER EN 0x00cf: 0x043E, # CYRILLIC SMALL LETTER O 0x00d0: 0x043F, # CYRILLIC SMALL LETTER PE 0x00d1: 0x044F, # CYRILLIC SMALL LETTER YA 0x00d2: 0x0440, # CYRILLIC SMALL LETTER ER 0x00d3: 0x0441, # CYRILLIC SMALL LETTER ES 0x00d4: 0x0442, # CYRILLIC SMALL LETTER TE 0x00d5: 0x0443, # CYRILLIC SMALL LETTER U 0x00d6: 0x0436, # CYRILLIC SMALL LETTER ZHE 0x00d7: 0x0432, # CYRILLIC SMALL LETTER VE 0x00d8: 0x044C, # CYRILLIC SMALL LETTER SOFT SIGN 0x00d9: 0x044B, # CYRILLIC SMALL LETTER YERU 0x00da: 0x0437, # CYRILLIC SMALL LETTER ZE 0x00db: 0x0448, # CYRILLIC SMALL LETTER SHA 0x00dc: 0x044D, # CYRILLIC SMALL LETTER E 0x00dd: 0x0449, # CYRILLIC SMALL LETTER SHCHA 0x00de: 0x0447, # CYRILLIC SMALL LETTER CHE 0x00df: 0x044A, # CYRILLIC SMALL LETTER HARD SIGN 0x00e0: 0x042E, # CYRILLIC CAPITAL LETTER YU 0x00e1: 0x0410, # CYRILLIC CAPITAL LETTER A 0x00e2: 0x0411, # CYRILLIC CAPITAL LETTER BE 0x00e3: 0x0426, # CYRILLIC CAPITAL LETTER TSE 0x00e4: 0x0414, # CYRILLIC CAPITAL LETTER DE 0x00e5: 0x0415, # CYRILLIC CAPITAL LETTER IE 0x00e6: 0x0424, # CYRILLIC CAPITAL LETTER EF 0x00e7: 0x0413, # CYRILLIC CAPITAL LETTER GHE 0x00e8: 0x0425, # CYRILLIC CAPITAL LETTER HA 0x00e9: 0x0418, # CYRILLIC CAPITAL LETTER I 0x00ea: 0x0419, # CYRILLIC CAPITAL LETTER SHORT I 0x00eb: 0x041A, # CYRILLIC CAPITAL LETTER KA 0x00ec: 0x041B, # CYRILLIC CAPITAL LETTER EL 0x00ed: 0x041C, # CYRILLIC CAPITAL LETTER EM 0x00ee: 0x041D, # CYRILLIC CAPITAL LETTER EN 0x00ef: 0x041E, # CYRILLIC CAPITAL LETTER O 0x00f0: 0x041F, # CYRILLIC CAPITAL LETTER PE 0x00f1: 0x042F, # CYRILLIC CAPITAL LETTER YA 0x00f2: 0x0420, # CYRILLIC CAPITAL LETTER ER 0x00f3: 0x0421, # CYRILLIC CAPITAL LETTER ES 0x00f4: 0x0422, # CYRILLIC CAPITAL LETTER TE 0x00f5: 0x0423, # CYRILLIC CAPITAL LETTER U 0x00f6: 0x0416, # CYRILLIC CAPITAL LETTER ZHE 0x00f7: 0x0412, # CYRILLIC CAPITAL LETTER VE 0x00f8: 0x042C, # CYRILLIC CAPITAL LETTER SOFT SIGN 0x00f9: 0x042B, # CYRILLIC CAPITAL LETTER YERU 0x00fa: 0x0417, # CYRILLIC CAPITAL LETTER ZE 0x00fb: 0x0428, # CYRILLIC CAPITAL LETTER SHA 0x00fc: 0x042D, # CYRILLIC CAPITAL LETTER E 0x00fd: 0x0429, # CYRILLIC CAPITAL LETTER SHCHA 0x00fe: 0x0427, # CYRILLIC CAPITAL LETTER CHE 0x00ff: 0x042A, # CYRILLIC CAPITAL LETTER HARD SIGN }) ### Encoding Map encoding_map = codecs.make_encoding_map(decoding_map) --zYM0uCDKw75PZbzx-- From mal@lemburg.com Sun Dec 29 14:40:48 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 29 Dec 2002 15:40:48 +0100 Subject: [Python-Dev] Encoding file for KOI8-U References: <20021229132038.GA8795@malva.ua> Message-ID: <3E0F0970.6060104@lemburg.com> Max Ischenko wrote: > I wonder, whether koi8-u.py would be included in Python 2.3? > If not, I could submit one. Such a codec is already included in Python CVS. > Though I'm not the author, it seems correct. Maxim submitted this codec a while ago. It has a few bugs. The version in CVS is compliant to the RFC and also faster: """ Python Character Mapping Codec for KOI8U. This character scheme is compliant to RFC2319 Written by Marc-Andre Lemburg (mal@lemburg.com). Modified by Maxim Dzumanenko . (c) Copyright 2002, Python Software Foundation. """#" import codecs, koi8_r ### Codec APIs class Codec(codecs.Codec): def encode(self,input,errors='strict'): return codecs.charmap_encode(input,errors,encoding_map) def decode(self,input,errors='strict'): return codecs.charmap_decode(input,errors,decoding_map) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): pass ### encodings module API def getregentry(): return (Codec().encode,Codec().decode,StreamReader,StreamWriter) ### Decoding Map decoding_map = koi8_r.decoding_map.copy() decoding_map.update({ 0x00a4: 0x0454, # CYRILLIC SMALL LETTER UKRAINIAN IE 0x00a6: 0x0456, # CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I 0x00a7: 0x0457, # CYRILLIC SMALL LETTER YI (UKRAINIAN) 0x00ad: 0x0491, # CYRILLIC SMALL LETTER UKRAINIAN GHE WITH UPTURN 0x00b4: 0x0403, # CYRILLIC CAPITAL LETTER UKRAINIAN IE 0x00b6: 0x0406, # CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I 0x00b7: 0x0407, # CYRILLIC CAPITAL LETTER YI (UKRAINIAN) 0x00bd: 0x0490, # CYRILLIC CAPITAL LETTER UKRAINIAN GHE WITH UPTURN }) ### Encoding Map encoding_map = codecs.make_encoding_map(decoding_map) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From python@rcn.com Sun Dec 29 17:18:34 2002 From: python@rcn.com (Raymond Hettinger) Date: Sun, 29 Dec 2002 12:18:34 -0500 Subject: [Python-Dev] Mesenne Twister copyright notice References: <3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de> <3E0E2C90.6060505@lemburg.com> <3E0E3004.9000900@v.loewis.de> <3E0EEBA5.1020803@lemburg.com> Message-ID: <008001c2af5e$52bb7a00$7311a044@oemcomputer> [MAL] > Instead, we should come up with a way which satisfies the licenses. > Since most licenses require a copy of the text to be included in > the documentation, I think the easiest and most reliable way of > achieving this is to add a new licenses section to the documentation > and Python's LICENSE file itself (Latex allows including text from > other files in the docs, so adding new licenses to the LICENSE file > would suffice if LICENSE is included verbatim in the Latex docs). This seems to be the most straight-forward way to go. If everyone agrees, I'll start the ball rolling by adding the verbatim text for the MersenneTwister code. Raymond Hettinger From Jack.Jansen@oratrix.com Sun Dec 29 21:39:10 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Sun, 29 Dec 2002 22:39:10 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0E40AC.6040906@v.loewis.de> Message-ID: On zondag, dec 29, 2002, at 01:24 Europe/Amsterdam, Martin v. L=F6wis=20 wrote: > > Does MacPython acknowledge all these people and organizations, in the=20= > documentation? Oops, you're right: MacPython acknowledges all the stuff that *I*=20 added, but of course I never thought about the things in the core:-( On the other thread: will including the copyright=20 notices/acknowledgements in the documentation be good enough if we=20 don't always ship the documentation? Or should we put this in a special=20= section that is always shipped, even with binary installers that don't=20= include documentation? -- - Jack Jansen =20 http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma=20 Goldman - From martin@v.loewis.de Sun Dec 29 21:56:40 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 29 Dec 2002 22:56:40 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: References: Message-ID: <3E0F6F98.1000404@v.loewis.de> Jack Jansen wrote: > On the other thread: will including the copyright > notices/acknowledgements in the documentation be good enough if we don't > always ship the documentation? This is exactly my concern. It is essentially for people producing binary distributions to decide how they want this to be organized. > Or should we put this in a special > section that is always shipped, even with binary installers that don't > include documentation? For the specific case in question, I think MAL's suggestion of adding yet another file is sufficient - maintainers of binary distributions then need to collect all those text files and distribute them. Complying with the various licenses is really something that the packagers most solve for themselves - the source distribution can only prepare things in a way that makes complying simple. Of course, if the license says that you must have certain words in the documentation of a binary distribution, then you really must have those words somewhere: either in text files that you "declare" to be documentation, or (probably more to the spirit of these clauses) in a place where users of the binary distribution won't overlook them. My feeling is that this can get quite annoying both for people producing binary distributions, as well as for people using them. So I would rather negotiate terms with the authors, or consider rejection of contributions that have such clauses. Of course, if nobody but me is annoyed by such clauses, I won't fight windmills. Regards, Martin From Jack.Jansen@oratrix.com Sun Dec 29 23:03:09 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Mon, 30 Dec 2002 00:03:09 +0100 Subject: [Python-Dev] Moving Mac/Lib to Lib/plat-mac before 2.3a1? Message-ID: Oops, there something that's been on my todo list for quite a while but that I haven't taken any action on (at least I can put part of the blame on Just, it was his idea and he didn't take any action either:-): moving Mac/Lib to Lib/plat-mac. The main reason for doing this is that most (if not all) of those modules are useable in unix-Python on MacOSX too, and moreover it would make the MacPython-OS9 tree look a little more like the normal Python tree. site.py would get one extra line with a special case (if sys.platform==darwin we not only add plat-darwin to the path but also plat-mac). Is it a good idea to try and get this in before 2.3a1? Is it a better idea to try and get this in between 2.3a1 and the next release (2.3a2? 2.3b1?)? And on the technical implementation: I think I would ask the SF maintainers to *copy* Mac/Lib to Lib/plat-mac, and then manually delete Mac/Lib on the trunk and Lib/plat-mac on the 2.2 maintenance branch. Is that a good idea? A similar case can be made for other Mac subdirs (Mac/Tools->Tools/Mac, Mac/scripts->Tools/Mac/scripts), but that is less essential and can probably wait until later. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From skip@pobox.com Sun Dec 29 23:23:07 2002 From: skip@pobox.com (Skip Montanaro) Date: Sun, 29 Dec 2002 17:23:07 -0600 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: References: <3E0E40AC.6040906@v.loewis.de> Message-ID: <15887.33755.211400.175809@montanaro.dyndns.org> Jack> On the other thread: will including the copyright Jack> notices/acknowledgements in the documentation be good enough if we Jack> don't always ship the documentation? Is there any reason the copyright builtin can't be embellished to spit out all the necessary copyrights? Skip From lalo@laranja.org Mon Dec 30 00:09:09 2002 From: lalo@laranja.org (Lalo Martins) Date: Sun, 29 Dec 2002 22:09:09 -0200 Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library) Message-ID: <20021230000909.GC15340@laranja.org> PAX - the "Pythonic API for XML" - is a XML-handling library. Like with DOM, you're supposed to first parse the whole stream into the internal representation, and then operate on it. What distinguishes PAX is that its implementation is python-oriented; you iterate trough child nodes via the normal ways (for, getitem, and in 2.2+, an iterator), the attributes are dictionary-based, etc. I used it to successfully reimplement TAL and PageTemplates (used in Zope). This reimplementation (AltPT) is quite stable and in production users in a few dozen servers around the world. PAX also includes a powerful python-oriented transform engine, which I used in Alt-TAL. The code has been tested on Python 2.1, 2.2 and recent CVS checkouts. It not only works, but transparently uses recent interesting features when available - notably, enumerate(). Most of the code is in public domain, and of course I make no objections to switching to PSL. In fact the package was designed all along with the intention of offering it to the standard library (and I like to think that this led me to cleaner design and a readable implementation). It is in active maintenence; if not much development happened in the last few months, it's because it reached some maturity. In fact I just released 1.0alpha1 before sending this mail (15k tar.bz2) PAX has a small site at my very-slow (due to bad connectivity) Zope server at http://python.laranja.org/pax/ - this site, btw, runs on AltPT. The releases can be found there, and the cvs is (for hystorical reasons) at :pserver:anonymous@cvs.sf.net:/cvsroot/flyingcircus module miscfiles/python/pax (because it was originally developed as I needed an XML library for my "flyingcircus" pda project). Should I write a PEP? Or a patch (silly, it would be a quite big patch)? []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From guido@python.org Mon Dec 30 00:17:36 2002 From: guido@python.org (Guido van Rossum) Date: Sun, 29 Dec 2002 19:17:36 -0500 Subject: [Python-Dev] Moving Mac/Lib to Lib/plat-mac before 2.3a1? In-Reply-To: Your message of "Mon, 30 Dec 2002 00:03:09 +0100." References: Message-ID: <200212300017.gBU0Ha012546@pcp02138704pcs.reston01.va.comcast.net> > Oops, there something that's been on my todo list for quite a while but > that I haven't taken any action on (at least I can put part of the > blame on Just, it was his idea and he didn't take any action either:-): > moving Mac/Lib to Lib/plat-mac. The main reason for doing this is that > most (if not all) of those modules are useable in unix-Python on MacOSX > too, and moreover it would make the MacPython-OS9 tree look a little > more like the normal Python tree. site.py would get one extra line with > a special case (if sys.platform==darwin we not only add plat-darwin to > the path but also plat-mac). > > Is it a good idea to try and get this in before 2.3a1? Is it a better > idea to try and get this in between 2.3a1 and the next release (2.3a2? > 2.3b1?)? Either way works for me -- I have no problem with doing it now. > And on the technical implementation: I think I would ask the SF > maintainers to *copy* Mac/Lib to Lib/plat-mac, and then manually delete > Mac/Lib on the trunk and Lib/plat-mac on the 2.2 maintenance branch. Is > that a good idea? It's sort of history falsification -- and you'd have to delete it on the 2.1 branch too (we might issue 2.1.4). I'm also a bit worried that the SF folks don't respond in time for the release. Are you sure you need the CVS history in the new location? > A similar case can be made for other Mac subdirs (Mac/Tools->Tools/Mac, > Mac/scripts->Tools/Mac/scripts), but that is less essential and can > probably wait until later. OK. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Dec 30 01:29:19 2002 From: dave@boost-consulting.com (David Abrahams) Date: Sun, 29 Dec 2002 20:29:19 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL Message-ID: Recently an issue has come up on the C++-sig which I think merits a little attention here. To boil it down, the situation looks like this: Shared library Q uses threading but not Python. It supplies a an interface by which users can supply callback functions. Some of these callbacks will be invoked directly in response to external calls into Q; others will be invoked on threads started by calls into Q. Python extension module A calls shared library Q, but doesn't use its callback interface. It works fine by itself. Python extension module B calls shared library Q and uses Q's callback interface. Because some of the callbacks need to use the Python API, and *might* be invoked by threads, they must all acquire the GIL. Because they also might be invoked by direct calls into Q, B must always release the GIL before calling anything in Q. Problem: using B while A is loaded breaks A: because B has installed callbacks in Q that acquire the GIL, A must also release the GIL before calling into Q. Notice that the author of A may have had no reason to believe anyone would install Python callbacks in Q! It seems to me that for situations like these, where a function may or may not be called on Python's main thread, it would be helpful if Python supplied a "recursive mutex" GIL acquisition/release pair, for which acquisition and release on the main thread would simply bump a counter. Is this something that was considered and rejected? TIA, Dave -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From pyth@devel.trillke.net Mon Dec 30 01:43:06 2002 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 30 Dec 2002 02:43:06 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <15887.33755.211400.175809@montanaro.dyndns.org>; from skip@pobox.com on Sun, Dec 29, 2002 at 05:23:07PM -0600 References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> Message-ID: <20021230024306.A2841@prim.han.de> Skip Montanaro wrote: > > Jack> On the other thread: will including the copyright > Jack> notices/acknowledgements in the documentation be good enough if we > Jack> don't always ship the documentation? > > Is there any reason the copyright builtin can't be embellished to spit out > all the necessary copyrights? great idea. let's implement drm.py. That should finally convince the companies in the content industry to embrace python. "of course, we have DRM builtin into python." almost-as-good-as-region-codes-ly y's holger From nhodgson@bigpond.net.au Mon Dec 30 02:12:07 2002 From: nhodgson@bigpond.net.au (Neil Hodgson) Date: Mon, 30 Dec 2002 13:12:07 +1100 Subject: [Python-Dev] Mesenne Twister copyright notice References: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com> <3E0E40AC.6040906@v.loewis.de> Message-ID: <01d401c2afa8$da7d0c70$3da48490@neil> Martin v. Löwis: > pcre-int.h says that the origin of PCRE must not be misrepresented by > omission [of Philip Hazel's copyright]. Discussion of this requirement for Scintilla (where I won't accept contributions with attribution requirements) led to this from Philip Hazel: % Yes. The licence says you must acknowledge PCRE in any software you % distribute that includes it. So Scintilla should acknowledge it. % However, the licence does not say that you have to impose this condition % on anybody who subsequently distrubutes other software that includes % your software. At least, that's one way I can read the licence. % % I'll put a sentence in if I can find some wording that says that % reasonably concisely. However, the current PCRE license still reads to me to require attribution no matter how deeply embedded it is. http://www.pcre.org/license.txt Neil From barry@python.org Mon Dec 30 02:22:36 2002 From: barry@python.org (Barry A. Warsaw) Date: Sun, 29 Dec 2002 21:22:36 -0500 Subject: [Python-Dev] Moving Mac/Lib to Lib/plat-mac before 2.3a1? References: Message-ID: <15887.44524.384976.546553@gargle.gargle.HOWL> >>>>> "JJ" == Jack Jansen writes: JJ> And on the technical implementation: I think I would ask the JJ> SF maintainers to *copy* Mac/Lib to Lib/plat-mac, and then JJ> manually delete Mac/Lib on the trunk and Lib/plat-mac on the JJ> 2.2 maintenance branch. Is that a good idea? My suggestion would be to grab the a copy of the cvs repository, do the re-organization yourself, and send SF a tarball of just the new directories and instructions for where you want them to unpack it. Then cvsrm all the individual files and let cvs up -P zap the old directories. -Barry From neal@metaslash.com Mon Dec 30 03:04:06 2002 From: neal@metaslash.com (Neal Norwitz) Date: Sun, 29 Dec 2002 22:04:06 -0500 Subject: [Python-Dev] test_pyclbr failling Message-ID: <20021230030406.GB12063@epoch.metaslash.com> test_pyclbr is failing due to the changes in random: test_pyclbr l1=['Random'] l2=['CoreGenerator'] ignore=('_verify',) class= test test_pyclbr failed -- Traceback (most recent call last): File "/home/neal/build/python/dist/src/Lib/test/test_pyclbr.py", line 146, in test_others cm('random', ignore=('_verify',)) # deleted File "/home/neal/build/python/dist/src/Lib/test/test_pyclbr.py", line 97, in checkModule self.assertListEq(real_bases, pyclbr_bases, ignore) File "/home/neal/build/python/dist/src/Lib/test/test_pyclbr.py", line 33, in assertListEq self.fail("%r missing" % item) File "/home/neal/build/python/dist/src/Lib/unittest.py", line 260, in fail raise self.failureException, msg AssertionError: 'Random' missing Neal From aahz@pythoncraft.com Mon Dec 30 03:45:52 2002 From: aahz@pythoncraft.com (Aahz) Date: Sun, 29 Dec 2002 22:45:52 -0500 Subject: [Python-Dev] GC at exit? Message-ID: <20021230034551.GA18622@panix.com> [Originally posted to comp.lang.python with no response; asking here before filing a bug report] Is garbage collection supposed to run when Python exits? The following program does not print any output, unless I uncomment the gc.collect() (or add a for loop that forces GC after creating the cycle): import gc class A: pass class B: def __del__(self): print "Bye-bye!" a = A() b = A() a.b = b b.a = a a.x = B() del a del b #gc.collect() -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From tim.one@comcast.net Mon Dec 30 04:45:56 2002 From: tim.one@comcast.net (Tim Peters) Date: Sun, 29 Dec 2002 23:45:56 -0500 Subject: [Python-Dev] GC at exit? In-Reply-To: <20021230034551.GA18622@panix.com> Message-ID: [Aahz] > [Originally posted to comp.lang.python with no response; asking here > before filing a bug report] > > Is garbage collection supposed to run when Python exits? The language reference manual doesn't promise that any garbage will be collected, ever. So, no, "supposed to" doesn't apply. From mmclay@comcast.net Mon Dec 30 04:49:42 2002 From: mmclay@comcast.net (Michael McLay) Date: Sun, 29 Dec 2002 23:49:42 -0500 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <15887.33755.211400.175809@montanaro.dyndns.org> References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> Message-ID: <200212292349.29240.mclay@nist.gov> On Sunday 29 December 2002 06:23 pm, Skip Montanaro wrote: > Jack> On the other thread: will including the copyright > Jack> notices/acknowledgements in the documentation be good enough if > we Jack> don't always ship the documentation? > > Is there any reason the copyright builtin can't be embellished to spit out > all the necessary copyrights? A requirement could be added to the PEP 2 that would require a copyright disclosure be included in the package/module so that the name "copyright" would retrieve the relevant copyright for that package/module. A copyright that did not require disclosure would return None. (BTW, PEP two is titled "Procedure for Adding New Modules". Should that be renamed "Procedure for Adding New Modules and Packages"?) >>> import mycoolpackage >>> mycoolpackage.copyright 'All rights to the use of mycoolpackage are hereby granted to the PSF. \n\n enjoy - authors of mycoolpackage\n' >>> The copyright builtin could be extended to add a search of all imported libraries to retrieve all of the copyrights that are used in an application. From skip@pobox.com Mon Dec 30 05:00:08 2002 From: skip@pobox.com (Skip Montanaro) Date: Sun, 29 Dec 2002 23:00:08 -0600 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <200212292349.29240.mclay@nist.gov> References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> <200212292349.29240.mclay@nist.gov> Message-ID: <15887.53976.949262.429193@montanaro.dyndns.org> Michael> A requirement could be added to the PEP 2 that would require a Michael> copyright disclosure be included in the package/module so that Michael> the name "copyright" would retrieve the relevant copyright for Michael> that package/module. Excellent idea. I would name it "__copyright__" though. Skip From jeremy@alum.mit.edu Mon Dec 30 05:04:36 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Mon, 30 Dec 2002 00:04:36 -0500 Subject: [Python-Dev] test_pyclbr failling In-Reply-To: <20021230030406.GB12063@epoch.metaslash.com> References: <20021230030406.GB12063@epoch.metaslash.com> Message-ID: <15887.54244.866974.808863@slothrop.zope.com> It would be helpful if the pyclbr tests could be re-written so that they don't depend on an arbitrary set of unrelated modules. While people should run the full test suite, it seems common to run a limited set of tests when you're changing a single module and test_pyclbr always gets skipped. (I also wonder how useful this pyclbr module is for modern Python, since almost every module in test_pyclbr that has been modified in the last year has caused tests to fail.) Jeremy From python@rcn.com Mon Dec 30 06:00:26 2002 From: python@rcn.com (Raymond Hettinger) Date: Mon, 30 Dec 2002 01:00:26 -0500 Subject: [Python-Dev] Mesenne Twister copyright notice References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> <200212292349.29240.mclay@nist.gov> Message-ID: <008201c2afc8$c0659d00$125ffea9@oemcomputer> > A requirement could be added to the PEP 2 that would require a copyright > disclosure be included in the package/module so that the name "copyright" > would retrieve the relevant copyright for that package/module. A copyright > that did not require disclosure would return None. Though it makes the copyright accessible in a binary distribution, I don't think it fulfills the license requirements of being in printed in the documentation. So far, the simplest thing that gets the job done and minimizes the PITA factor is to have a section in the docs for licenses, paste them in a verbatim block, and make a link to them from the module docs. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From skip@pobox.com Mon Dec 30 06:11:37 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 30 Dec 2002 00:11:37 -0600 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <008201c2afc8$c0659d00$125ffea9@oemcomputer> References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> <200212292349.29240.mclay@nist.gov> <008201c2afc8$c0659d00$125ffea9@oemcomputer> Message-ID: <15887.58265.41467.93079@montanaro.dyndns.org> >> A requirement could be added to the PEP 2 that would require a >> copyright disclosure be included in the package/module so that the >> name "copyright" would retrieve the relevant copyright for that >> package/module. A copyright that did not require disclosure would >> return None. Raymond> Though it makes the copyright accessible in a binary Raymond> distribution, I don't think it fulfills the license Raymond> requirements of being in printed in the documentation. Ah, but now you have a "machine" that will generate the bit of documentation you desire as part of the install process (binary or source). Skip From mmclay@comcast.net Mon Dec 30 06:54:40 2002 From: mmclay@comcast.net (Michael McLay) Date: Mon, 30 Dec 2002 01:54:40 -0500 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <15887.58265.41467.93079@montanaro.dyndns.org> References: <3E0E40AC.6040906@v.loewis.de> <008201c2afc8$c0659d00$125ffea9@oemcomputer> <15887.58265.41467.93079@montanaro.dyndns.org> Message-ID: <200212300154.40233.mmclay@comcast.net> On Monday 30 December 2002 01:11 am, Skip Montanaro wrote: > >> A requirement could be added to the PEP 2 that would require a > >> copyright disclosure be included in the package/module so that the > >> name "copyright" would retrieve the relevant copyright for that > >> package/module. A copyright that did not require disclosure would > >> return None. > > Raymond> Though it makes the copyright accessible in a binary > Raymond> distribution, I don't think it fulfills the license > Raymond> requirements of being in printed in the documentation. > > Ah, but now you have a "machine" that will generate the bit of > documentation you desire as part of the install process (binary or source). +1 A script to generate the section in the documentation would keep the binary distribution in sync with the paper docs. From martin@v.loewis.de Mon Dec 30 07:15:31 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Dec 2002 08:15:31 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: Message-ID: <3E0FF293.80103@v.loewis.de> David Abrahams wrote: > Python extension module B calls shared library Q and uses Q's callback > interface. Because some of the callbacks need to use the Python API, > and *might* be invoked by threads, they must all acquire the GIL. Wrong. If the code in B that calls Q does not allow threads, the callbacks don't need to reacquire the GIL. > Problem: using B while A is loaded breaks A: because B has installed > callbacks in Q that acquire the GIL, A must also release the GIL > before calling into Q. Can you please explain what a callback is? Can the callbacks occur in the context of a different thread, i.e. different from the one that has installed the callback? If it is a true callback (i.e. Q will call B back while being called from B), then this won't interfere at all with A. Regards, Martin From martin@v.loewis.de Mon Dec 30 07:20:33 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Dec 2002 08:20:33 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <15887.58265.41467.93079@montanaro.dyndns.org> References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> <200212292349.29240.mclay@nist.gov> <008201c2afc8$c0659d00$125ffea9@oemcomputer> <15887.58265.41467.93079@montanaro.dyndns.org> Message-ID: <3E0FF3C1.1020307@v.loewis.de> Skip Montanaro wrote: > Ah, but now you have a "machine" that will generate the bit of documentation > you desire as part of the install process (binary or source). The question still is what is easiest for people producing Python binaries. Do they want to paste files from the CVS to fulfill the requirements, or would they prefer to print a variable in Python? I agree with Raymond that collecting the statements alone in copyright won't help, since it does not meet the licensing criteria of some of the licenses. Regards, Martin From martin@v.loewis.de Mon Dec 30 07:24:19 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Dec 2002 08:24:19 +0100 Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library) In-Reply-To: <20021230000909.GC15340@laranja.org> References: <20021230000909.GC15340@laranja.org> Message-ID: <3E0FF4A3.7040602@v.loewis.de> Lalo Martins wrote: > Should I write a PEP? Or a patch (silly, it would be a quite big patch)? I think a library PEP would be needed. However, I would encourage you first to collect a user community for this code, and consider inclusion in the Python core only after a few years. One vehicle for potentially wider distribution of your code would be inclusion into PyXML, which I would help to perform if desired. Regards, Martin From fredrik@pythonware.com Mon Dec 30 10:25:27 2002 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 30 Dec 2002 11:25:27 +0100 Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library) References: <20021230000909.GC15340@laranja.org> Message-ID: <006f01c2afed$c6f15590$0900a8c0@spiff> Lalo Martins wrote: > PAX - the "Pythonic API for XML" - is a XML-handling library. >=20 > Like with DOM, you're supposed to first parse the whole stream into = the > internal representation, and then operate on it. >=20 > What distinguishes PAX is that its implementation is python-oriented; = you > iterate trough child nodes via the normal ways (for, getitem, and in = 2.2+, > an iterator), the attributes are dictionary-based, etc. how is this different from ElementTrees or all the other light-weight DOM variants out there? how much memory does it use, compared to minidom and the pyxml tools? how fast is it (compared to the same tools? compared to more optimized tools such as libxml and henry spencer's stuff?) (I think the right approach would be to add it to the xml-sig = distribution first, and move it into the core only when you have more than a dozen users...) From mwh@python.net Mon Dec 30 10:58:01 2002 From: mwh@python.net (Michael Hudson) Date: 30 Dec 2002 10:58:01 +0000 Subject: [Python-Dev] GC at exit? In-Reply-To: Aahz's message of "Sun, 29 Dec 2002 22:45:52 -0500" References: <20021230034551.GA18622@panix.com> Message-ID: <2m3cof3hae.fsf@starship.python.net> Aahz writes: > [Originally posted to comp.lang.python with no response; asking here > before filing a bug report] > > Is garbage collection supposed to run when Python exits? The following > program does not print any output, unless I uncomment the gc.collect() > (or add a for loop that forces GC after creating the cycle): > > import gc > > class A: > pass > > class B: > def __del__(self): > print "Bye-bye!" > > a = A() > b = A() > a.b = b > b.a = a > a.x = B() > > del a > del b > > #gc.collect() One possibility is that the __del__ method *is* being run, but after sys.stdout & so on has been torn down. python -v can help, maybe? Cheers, M. -- Unfortunately, nigh the whole world is now duped into thinking that silly fill-in forms on web pages is the way to do user interfaces. -- Erik Naggum, comp.lang.lisp From mwh@python.net Mon Dec 30 12:41:28 2002 From: mwh@python.net (Michael Hudson) Date: 30 Dec 2002 12:41:28 +0000 Subject: [Python-Dev] Fix for bug 597919 (compiler package vs SET_LINENO) Message-ID: <2m1y3zd6h3.fsf@starship.python.net> I've finally come up with a patch I'm 90% happy with for this problem, got my iBook's modem talking to my ISP right and created a diff. Unfortunately, SF's tracker isn't behaving and I'm about to disappear from the internet for a day or so and 2.3a1 is due tomorrow, so it's here: http://starship.python.net/crew/mwh/hacks/compiler-fixes-3.diff There's one outstanding problem with the compiler package to do with code of the form (unary operator)constant which is shown up by test_grammar, but that aside I can compile the test suite and the standard library using the patched compiler package and it runs fine. I'd really like to see this patch in 2.3a1. I'll fix the -literal thing for 2.3a2, but I bet noone notices it in the mean time :) Cheers, M. -- at any rate, I'm satisfied that not only do they know which end of the pointy thing to hold, but where to poke it for maximum effect. -- Eric The Read, asr, on google.com From guido@python.org Mon Dec 30 13:05:10 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 30 Dec 2002 08:05:10 -0500 Subject: [Python-Dev] test_pyclbr failling In-Reply-To: Your message of "Mon, 30 Dec 2002 00:04:36 EST." <15887.54244.866974.808863@slothrop.zope.com> References: <20021230030406.GB12063@epoch.metaslash.com> <15887.54244.866974.808863@slothrop.zope.com> Message-ID: <200212301305.gBUD5Ar14517@pcp02138704pcs.reston01.va.comcast.net> > It would be helpful if the pyclbr tests could be re-written so that > they don't depend on an arbitrary set of unrelated modules. While > people should run the full test suite, it seems common to run a > limited set of tests when you're changing a single module and > test_pyclbr always gets skipped. I disagree. I maintain pyclbr.py (having mostly rewritten its main loop last August and another major update more recently) and usually a failure is now a symptom of a bug in pyclbr. > (I also wonder how useful this pyclbr module is for modern Python, > since almost every module in test_pyclbr that has been modified in the > last year has caused tests to fail.) That was true before my most recent rewrite. After it, I was able to add back a bunch of tests that were commented out, and vastly reduce the number of exceptions for other tests. It's still not perfect, but a lot more so. pyclbr is used by IDLE's class and path browser commands, so it can't be deprecated. I'll look into the most recent random failure. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Dec 30 13:47:36 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 08:47:36 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: <3E0FF293.80103@v.loewis.de> ("Martin v. =?iso-8859-1?q?L=F6wis"'s?= message of "Mon, 30 Dec 2002 08:15:31 +0100") References: <3E0FF293.80103@v.loewis.de> Message-ID: "Martin v. L=F6wis" writes: > David Abrahams wrote: >> Python extension module B calls shared library Q and uses Q's callback >> interface. Because some of the callbacks need to use the Python API, >> and *might* be invoked by threads, they must all acquire the GIL. > > Wrong. If the code in B that calls Q does not allow threads, the > callbacks don't need to reacquire the GIL. I think you must be misunderstanding me. These callbacks might be invoked on threads that are not the Python main thread.=20=20 http://www.python.org/doc/current/api/threads.html says: "the rule exists that only the thread that has acquired the global interpreter lock may operate on Python objects or call Python/C API functions" I am taking that statement at face value and assuming it means what it says. Note that these threads were not started by Python's thread API. >> Problem: using B while A is loaded breaks A: because B has installed >> callbacks in Q that acquire the GIL, A must also release the GIL >> before calling into Q. > > Can you please explain what a callback is?=20 I'm trying to leave out irrelevant details, but a callback happens to be a virtual function in a C++ class instance in this case. These callbacks implement behaviors of base classes supplied by the library, Qt. For the purposes of this discussion, it might just as well be a 'C' language pointer-to-function, though: void (*)() > Can the callbacks occur in > the context of a different thread, i.e. different from the one that > has installed the callback? I think the answer is yes, but I'm trying to leave out the irrelevant, and I'm pretty sure that it doesn't matter one whit which thread installs the callback. What matters, AFAICT, is that the callback might be invoked on a thread that's not Python's main thread, thus must acquire the GIL, so if Python's main thread wants to call something in Q that might invoke one of these callbacks, it must release the GIL. > If it is a true callback (i.e. Q will call B back while being called > from B), then this won't interfere at all with A. Maybe we have different ideas of what "callback" means. In your terms, it is not a "true callback". --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From lalo@laranja.org Mon Dec 30 13:57:18 2002 From: lalo@laranja.org (Lalo Martins) Date: Mon, 30 Dec 2002 11:57:18 -0200 Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library) In-Reply-To: <006f01c2afed$c6f15590$0900a8c0@spiff> References: <20021230000909.GC15340@laranja.org> <006f01c2afed$c6f15590$0900a8c0@spiff> Message-ID: <20021230135718.GF15340@laranja.org> On Mon, Dec 30, 2002 at 11:25:27AM +0100, Fredrik Lundh wrote: > (I think the right approach would be to add it to the xml-sig distribution > first, and move it into the core only when you have more than a dozen > users...) This seems reasonable. Perhaps I was over-hasted by the threat of 2.3a ;-) I will subscribe to the XML-sig asap. []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From skip@pobox.com Mon Dec 30 14:32:25 2002 From: skip@pobox.com (Skip Montanaro) Date: Mon, 30 Dec 2002 08:32:25 -0600 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0FF3C1.1020307@v.loewis.de> References: <3E0E40AC.6040906@v.loewis.de> <15887.33755.211400.175809@montanaro.dyndns.org> <200212292349.29240.mclay@nist.gov> <008201c2afc8$c0659d00$125ffea9@oemcomputer> <15887.58265.41467.93079@montanaro.dyndns.org> <3E0FF3C1.1020307@v.loewis.de> Message-ID: <15888.22777.811400.882470@montanaro.dyndns.org> >>>>> "Martin" =3D=3D Martin v L=F6wis writes: Martin> Skip Montanaro wrote: >> Ah, but now you have a "machine" that will generate the bit of d= ocumentation >> you desire as part of the install process (binary or source). Martin> The question still is what is easiest for people producing Martin> Python binaries. Do they want to paste files from the CVS t= o Martin> fulfill the requirements, or would they prefer to print a Martin> variable in Python? You simply have an automated post-install or binary distro build step w= hich does the little find-the-copyrights dance and dumps the result to somet= hing like LICENSES.txt or LICENSES.tex or LICENSES.hlp in the appropriate directory and either declare it to be documentation or incorporate it i= nto the documentation delivered with the distribution. People distributing Python with another package will have to do this so= rt of thing themselves, but that's not our concern. Skip From theller@python.net Mon Dec 30 15:38:08 2002 From: theller@python.net (Thomas Heller) Date: 30 Dec 2002 16:38:08 +0100 Subject: [Python-Dev] 'unhashable type' and new style classes Message-ID: <3cofplen.fsf@python.net> This code class X: def __cmp__(self, other): return 1 c1 = X() d = {} d[c1] = None raises a TypeError, unhashable type, since the class does not define a __hash__ method. This error is no longer raised when X derives from object. Bug (or feature)? Thomas From aahz@pythoncraft.com Mon Dec 30 14:59:12 2002 From: aahz@pythoncraft.com (Aahz) Date: Mon, 30 Dec 2002 09:59:12 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: Message-ID: <20021230145912.GA8481@panix.com> On Sun, Dec 29, 2002, David Abrahams wrote: > > Python extension module B calls shared library Q and uses Q's callback > interface. Because some of the callbacks need to use the Python API, > and *might* be invoked by threads, they must all acquire the GIL. > Because they also might be invoked by direct calls into Q, B must > always release the GIL before calling anything in Q. So you're saying that the callback functions in B acquire the GIL? > Problem: using B while A is loaded breaks A: because B has installed > callbacks in Q that acquire the GIL, A must also release the GIL > before calling into Q. Why? The callbacks in B will simply hang until they acquire the GIL. I think this thread might be better handled on c.l.py, at least until it's understood well enough to be clear whether something does need to change in Python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are three kinds of lies: Lies, Damn Lies, and Statistics." --Disraeli From dave@boost-consulting.com Mon Dec 30 15:15:22 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 10:15:22 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: <20021230145912.GA8481@panix.com> (Aahz's message of "Mon, 30 Dec 2002 09:59:12 -0500") References: <20021230145912.GA8481@panix.com> Message-ID: Aahz writes: > On Sun, Dec 29, 2002, David Abrahams wrote: >> >> Python extension module B calls shared library Q and uses Q's callback >> interface. Because some of the callbacks need to use the Python API, >> and *might* be invoked by threads, they must all acquire the GIL. >> Because they also might be invoked by direct calls into Q, B must >> always release the GIL before calling anything in Q. > > So you're saying that the callback functions in B acquire the GIL? Yes. >> Problem: using B while A is loaded breaks A: because B has installed >> callbacks in Q that acquire the GIL, A must also release the GIL >> before calling into Q. > > Why? The callbacks in B will simply hang until they acquire the GIL. If A doesn't release the GIL, one of its direct calls into Q from Python may invoke a callback in B, which tries to acquire the lock when it is already held. This is a no-no. I realize that the docs for PyEval_AcquireLock() say: "If this thread already has the lock, a deadlock ensues", but the behavior we're seeing is consistent with a scenario where trying to acquire an already-held is a no-op and releasing it is unconditional. Eventually the GIL release in B's callback takes effect and when A returns to Python there is no thread state. > I think this thread might be better handled on c.l.py, at least > until it's understood well enough to be clear whether something does > need to change in Python. I'm pretty certain I understand what's going on. If it would be better for you to take this to c.l.py, I'm happy to do so, but AFAICT there _is_ a Python core issue here: there's no way to find out whether you've already got the GIL**, so if you _might_ have to acquire it, you must always acquire it. **If I'm wrong about that, please let me know. It isn't obvious from the documentation. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From guido@python.org Mon Dec 30 15:27:45 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 30 Dec 2002 10:27:45 -0500 Subject: [Python-Dev] 'unhashable type' and new style classes In-Reply-To: Your message of "30 Dec 2002 16:38:08 +0100." <3cofplen.fsf@python.net> References: <3cofplen.fsf@python.net> Message-ID: <200212301527.gBUFRjT19772@odiug.zope.com> > This code > > class X: > def __cmp__(self, other): > return 1 > > c1 = X() > > d = {} > d[c1] = None > > raises a TypeError, unhashable type, since the class does > not define a __hash__ method. > > This error is no longer raised when X derives from object. > Bug (or feature)? It's something in between. :-( I've been struggling with this since 2.2 and not found a good solution. Built-in mutable types like list and dict deal with this by having an explicit tp_hash slot that raises an exception. Can you add a SF entry and assign it to me? Any insight you might have in the matter would be appreciated. I don't think I'll get to this before the 2.3a1 release. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Dec 30 15:52:27 2002 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 30 Dec 2002 10:52:27 -0500 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: <200212241930.gBOJUYD10795@odiug.zope.com> <3E0CBEE4.1020902@interet.com> <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E106BBB.4020909@interet.com> Guido van Rossum wrote: >>Python already has an import hook, namely __import__. PEP >>302 adds three more hooks: sys.path_hooks, sys.meta_path and >>sys.path_importer_hooks. The idea of four import hooks is >>already fishy. > > There's no path_importer_hooks; there's path_importer_cache, Ooops, sorry. I meant sys.path_importer_cache. > but > that's not a new hook, it's a cache for the path_hooks hook. So I > think the PEP proposes only two new hooks. I disagree. It is publicly available in sys, and the PEP specifies it can be replaced or modified. And if you want to add a hook, it is easier to put it into sys.path_importer_cache than to add it to the list in sys.path_hooks. If you add it to sys.path_hooks, you must clear its entry in sys.path_importer_cache too. >>That is a lot of external changes. That is a lot of code >>written in C. > > Um, last I looked, most of the code written in C was specific to the > zip importer; only a relatively small amount of code was added to > import.c (about 10%). I agree 90% of the C would be needed anyway. >>I think the proper import hook design is to write Python's >>import mechanism in Python along the lines of Greg's imputil.py >>and Gordon's iu.py. > That's a design that I have had in mind long ago, but I don't see it > happening soon, because it would be a much larger overhaul of > import.c. Is there some reason you need hooks right away? A current application? > I'm not so sure. In practice, hooks are used for two things: import > from other media than directories (e.g. zip files), and supporting > additional filename extensions that trigger special transformations. The PEP 302 hooks do not support the second case very well when the files with special extensions are in zip archives. Adding the hook replaces the zip hook, as all PEP 302 hooks are alternatives to each other. JimA From jim@interet.com Mon Dec 30 16:16:31 2002 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 30 Dec 2002 11:16:31 -0500 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: Message-ID: <3E10715F.6040003@interet.com> Just van Rossum wrote: >>It changes the meaning of __import__. > > Where did you get that idea? I believe the base (unreplaced) __import__ function does not find hooked imports. It follows that __import__ will not find modules in zip archives. > This is backwards: iu.py implements a superset of PEP 302 (with > different details). So you have that now. Which is why I think we should fix iu.py instead of add more public import hooks. It is OK with me to use your import hooks as an internal feature to implement zip imports. > PEP 302 allows customizing *parts* of the import mechanism > without having to deal with most of these pitfalls and complications. It > allows completely independend components to add hooks that will work > together seamlessy. This is not true for replacing __import__. I don't believe PEP 302 provides hooks that work together. It provides only one hook for each component of sys.path, and replaces any hook that was already there. JimA From theller@python.net Mon Dec 30 17:21:37 2002 From: theller@python.net (Thomas Heller) Date: 30 Dec 2002 18:21:37 +0100 Subject: [Python-Dev] 'unhashable type' and new style classes In-Reply-To: <200212301527.gBUFRjT19772@odiug.zope.com> References: <3cofplen.fsf@python.net> <200212301527.gBUFRjT19772@odiug.zope.com> Message-ID: Guido van Rossum writes: > > This code > > > > class X: > > def __cmp__(self, other): > > return 1 > > > > c1 = X() > > > > d = {} > > d[c1] = None > > > > raises a TypeError, unhashable type, since the class does > > not define a __hash__ method. > > > > This error is no longer raised when X derives from object. > > Bug (or feature)? > > It's something in between. :-( > > I've been struggling with this since 2.2 and not found a good > solution. Built-in mutable types like list and dict deal with this by > having an explicit tp_hash slot that raises an exception. Why doesn't the default implementation raise a TypeError (or be NULL), and the immutable types override this? > > Can you add a SF entry and assign it to me? Any insight you might > have in the matter would be appreciated. Sure. Thomas From just@letterror.com Mon Dec 30 16:29:52 2002 From: just@letterror.com (Just van Rossum) Date: Mon, 30 Dec 2002 17:29:52 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <3E10715F.6040003@interet.com> Message-ID: James C. Ahlstrom wrote: > >>It changes the meaning of __import__. > > > > Where did you get that idea? > > I believe the base (unreplaced) __import__ function does not find > hooked imports. It follows that __import__ will not find modules in > zip archives. Not true. (It couldn't be true, simply because any import statement *physically* goes through __import__.) It _is_ true that zipimports won't neccesarily work when using existing __import__ replacements. How big a deal that is, I honestly don't know. > > This is backwards: iu.py implements a superset of PEP 302 (with > > different details). So you have that now. > > Which is why I think we should fix iu.py instead of > add more public import hooks. It is OK with me to > use your import hooks as an internal feature to > implement zip imports. And I see *nothing* against making them public. They solve real problems for real applications (besides zipimport) as well. > > PEP 302 allows customizing *parts* of the import mechanism without > > having to deal with most of these pitfalls and complications. It > > allows completely independend components to add hooks that will > > work together seamlessy. This is not true for replacing __import__. > > I don't believe PEP 302 provides hooks that work together. It > provides only one hook for each component of sys.path, and replaces > any hook that was already there. And __import__ *does* provide this, how? Sure, PEP 302 is not a full replacement of (say) ihooks.py, but it's nevertheless a vast improvement over raw __import__ hooks. Just From guido@python.org Mon Dec 30 16:30:10 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 30 Dec 2002 11:30:10 -0500 Subject: [Python-Dev] 'unhashable type' and new style classes In-Reply-To: Your message of "30 Dec 2002 18:21:37 +0100." References: <3cofplen.fsf@python.net> <200212301527.gBUFRjT19772@odiug.zope.com> Message-ID: <200212301630.gBUGUAE03901@odiug.zope.com> > > > This code > > > > > > class X: > > > def __cmp__(self, other): > > > return 1 > > > > > > c1 = X() > > > > > > d = {} > > > d[c1] = None > > > > > > raises a TypeError, unhashable type, since the class does > > > not define a __hash__ method. > > > > > > This error is no longer raised when X derives from object. > > > Bug (or feature)? > > > > It's something in between. :-( > > > > I've been struggling with this since 2.2 and not found a good > > solution. Built-in mutable types like list and dict deal with this by > > having an explicit tp_hash slot that raises an exception. > > Why doesn't the default implementation raise a TypeError (or be NULL), > and the immutable types override this? The default implementation of __hash__ must match the default implementation of __cmp__ (and rich comparisons, __eq__ etc.). So the default implementation cannot raise an exception, because objects are defined to be immutable by default. (Maybe this was a mistake, but it's not so easy to change without causing backwards incompatibilities.) > > Can you add a SF entry and assign it to me? Any insight you might > > have in the matter would be appreciated. > > Sure. Thanks. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Mon Dec 30 16:40:02 2002 From: tim@zope.com (Tim Peters) Date: Mon, 30 Dec 2002 11:40:02 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: Message-ID: [David Abrahams] > ... > but AFAICT there _is_ a Python core issue here: there's no way to find > out whether you've already got the GIL**, so if you _might_ have to acquire > it, you must always acquire it. > > **If I'm wrong about that, please let me know. It isn't obvious from > the documentation. It's true -- you can't know whether you have the GIL, unless you code up another layer of your own machinery to keep track of who has the GIL. Mark Hammond faces this issue (in all its multifacted glories) in the Win32 extensions, and built some C++ classes there to help him out. It's difficult at best, and last I looked (several years ago) I wasn't convinced Mark's approach was 109% reliable. The worst it gets in the Python core is in posixmodule.c's _PyPclose, which needs to build a new interpreter state from scratch, in order to build a new thread state from scratch, in order to acquire the GIL, in order to call some Py API functions. This isn't as hard as it *can* get, because in that function we know the thread executing does not hold the GIL. The now-defunct Thread-SIG used to have bouts of angst over all this. I think we eventually figured out a better approach there, but it required real work to implement, and nobody had time for that. The only that's changed since then is that nobody remembers the better approach anymore <0.9 wink>. happy-new-year-ly y'rs - tim From martin@v.loewis.de Mon Dec 30 17:01:31 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 18:01:31 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <3E0FF293.80103@v.loewis.de> Message-ID: David Abrahams writes: > > Wrong. If the code in B that calls Q does not allow threads, the > > callbacks don't need to reacquire the GIL. > > I think you must be misunderstanding me. These callbacks might be > invoked on threads that are not the Python main thread. Which thread is the main thread is not relevant at all. What matters is whether the callback is invoked in the context of the thread that installed it (i.e. as a result of calling a function in B). > "the rule exists that only the thread that has acquired the global > interpreter lock may operate on Python objects or call Python/C API > functions" > > I am taking that statement at face value and assuming it means what it > says. And rightly so. Notice that it does not use the term "main thread". > > Can you please explain what a callback is? > > I'm trying to leave out irrelevant details, but a callback happens to > be a virtual function in a C++ class instance in this case. These > callbacks implement behaviors of base classes supplied by the library, > Qt. Ok, now I know how a callback is installed (by redefining a virtual method). The other question is: How is it invoked? I.e. who invokes it, and why? I suppose the immediate answer is "the library Q". However, that library does not invoke it with out a trigger: What is that trigger? > What matters, AFAICT, is that the callback might be invoked on a > thread that's not Python's main thread, thus must acquire the GIL, > so if Python's main thread wants to call something in Q that might > invoke one of these callbacks, it must release the GIL. Which thread is the main thread is completely irrelevant. If you have something like class MyB(B.Base): def overridden(self): print "This overrides a virtual function" def doit(): b = MyB() B.do_the_real_work(b) # will call Q, which will call the callback, # which will call overridden then there is no need for the C++ code in B to release the GIL, nor is there a need for B to reacquire the GIL in the callback. This is independent of whether doit is called in the main thread, or in the context of any other thread: at the point where do_the_real_work is called, the current thread holds the GIL. If it keeps the GIL, then the GIL will be held at the point when the callback occurs, in the context of the thread where the callback occurs. > Maybe we have different ideas of what "callback" means. In your > terms, it is not a "true callback". Then I'm still curious as to what triggers invocation of that virtual function. Regards, Martin From martin@v.loewis.de Mon Dec 30 17:04:20 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 18:04:20 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <20021230145912.GA8481@panix.com> Message-ID: David Abrahams writes: > > So you're saying that the callback functions in B acquire the GIL? > > Yes. Ok. What would break if they wouldn't? Regards, Martin From dave@boost-consulting.com Mon Dec 30 17:26:34 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 12:26:34 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 18:01:31 +0100") References: <3E0FF293.80103@v.loewis.de> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> > Wrong. If the code in B that calls Q does not allow threads, the >> > callbacks don't need to reacquire the GIL. >>=20 >> I think you must be misunderstanding me. These callbacks might be >> invoked on threads that are not the Python main thread.=20=20 > > Which thread is the main thread is not relevant at all. Sorry, I got "main" confused with "current". My point is that the callback may be invoked by threads that don't hold the GIL. > What matters is whether the callback is invoked in the context of > the thread that installed it (i.e. as a result of calling a function > in B). I still don't see how the thread that installed it has any bearing. Imagine there's a global function pointer variable somewhere. Some thread comes along and makes it point to some function f ("installs the callback"). Now there are various ways that callback can be called. Some other thread may pick up the global variable at any time and invoke the function it points to. Why does it matter which thread set the variable? >> > Can you please explain what a callback is?=20 >>=20 >> I'm trying to leave out irrelevant details, but a callback happens to >> be a virtual function in a C++ class instance in this case. These >> callbacks implement behaviors of base classes supplied by the library, >> Qt.=20=20 > > Ok, now I know how a callback is installed (by redefining a virtual > method).=20 Technically, redefining a virtual function by itself doesn't do anything. You have to make an instance of the class which redefines that function available to the library somehow. But you knew that. > The other question is: How is it invoked? I.e. who invokes > it, and why? > > I suppose the immediate answer is "the library Q". However, that > library does not invoke it with out a trigger: What is that trigger? There are several ways, IIUC. It may be invoked in response to direct calls into Q's API (which may be made from a Python extension module). It may also be invoked by some thread that Q has launched. >> What matters, AFAICT, is that the callback might be invoked on a >> thread that's not Python's main thread, thus must acquire the GIL, >> so if Python's main thread wants to call something in Q that might >> invoke one of these callbacks, it must release the GIL. > > Which thread is the main thread is completely irrelevant. If you have > something like > > class MyB(B.Base): > def overridden(self): > print "This overrides a virtual function" > > def doit(): > b =3D MyB() > B.do_the_real_work(b) # will call Q, which will call the callback, > # which will call overridden > > then there is no need for the C++ code in B to release the GIL, nor > is there a need for B to reacquire the GIL in the callback. This is > independent of whether doit is called in the main thread, or in the > context of any other thread: at the point where do_the_real_work is > called, the current thread holds the GIL. Yes, we understand that problem. I had this exact discussion with the designer of B. He explained that the problem is that Q might also invoke the virtual function on a thread that is not holding the GIL. > If it keeps the GIL, then the GIL will be held at the point when the > callback occurs, in the context of the thread where the callback > occurs. Yes. >> Maybe we have different ideas of what "callback" means. In your >> terms, it is not a "true callback". > > Then I'm still curious as to what triggers invocation of that virtual > function. Q, either directly via its API, or in some thread it started. --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From dave@boost-consulting.com Mon Dec 30 17:27:25 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 12:27:25 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 18:04:20 +0100") References: <20021230145912.GA8481@panix.com> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> > So you're saying that the callback functions in B acquire the GIL? >>=20 >> Yes. > > Ok. What would break if they wouldn't? All of the places where Q invokes the callback on arbitrary threads it has started, but which don't hold the GIL. --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From dave@boost-consulting.com Mon Dec 30 17:29:01 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 12:29:01 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: <20021230145912.GA8481@panix.com> (Aahz's message of "Mon, 30 Dec 2002 09:59:12 -0500") References: <20021230145912.GA8481@panix.com> Message-ID: Aahz writes: > I think this thread might be better handled on c.l.py, at least until > it's understood well enough to be clear whether something does need to > change in Python. BTW, I tried to comply with your request. Currently the crossover posting which I Bcc'd to python-dev is being held for moderator approval: python-dev-admin@python.org writes: > Your mail to 'Python-Dev' with the subject > > Extension modules, Threading, and the GIL > > Is being held until the list moderator can review it for approval. > > The reason it is being held: > > Message has implicit destination > > Either the message will get posted to the list, or you will receive > notification of the moderator's decision. > -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From martin@v.loewis.de Mon Dec 30 17:58:06 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 18:58:06 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <20021230145912.GA8481@panix.com> Message-ID: David Abrahams writes: > > Ok. What would break if they wouldn't? > > All of the places where Q invokes the callback on arbitrary threads it > has started, but which don't hold the GIL. So Q creates new threads which perform callbacks? But Q also performs the callbacks when invoked from A? Sounds like a bug in Q to me... Regards, Martin From martin@v.loewis.de Mon Dec 30 18:11:02 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 19:11:02 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <3E0FF293.80103@v.loewis.de> Message-ID: David Abrahams writes: > > What matters is whether the callback is invoked in the context of > > the thread that installed it (i.e. as a result of calling a function > > in B). > > I still don't see how the thread that installed it has any bearing. If the callbacks would only occur while Q is being called from B, and in the context of the thread which also hosts that call, then B should not release the lock. Then the callback would not need to reacquire it. > There are several ways, IIUC. It may be invoked in response to direct > calls into Q's API (which may be made from a Python extension module). > It may also be invoked by some thread that Q has launched. I find this surprising. Is this *any* API of Q that can trigger the callbacks, or a dispatch_one_event call that can do so? > Yes, we understand that problem. I had this exact discussion with the > designer of B. He explained that the problem is that Q might also > invoke the virtual function on a thread that is not holding the GIL. I would more deeply question such a statement. It sounds like this library uses the callback for event processing of some kind, in which case the callbacks are only invoked if events are processed. It should be possible to describe more precisely under what specific conditions event processing occurs. > > Then I'm still curious as to what triggers invocation of that virtual > > function. > > Q, either directly via its API, or in some thread it started. Then calling this API must release the GIL. Regards, Martin From dave@boost-consulting.com Mon Dec 30 18:14:16 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 13:14:16 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 18:58:06 +0100") References: <20021230145912.GA8481@panix.com> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> > Ok. What would break if they wouldn't? >>=20 >> All of the places where Q invokes the callback on arbitrary threads it >> has started, but which don't hold the GIL. > > So Q creates new threads which perform callbacks? But Q also performs > the callbacks when invoked from A? Sounds like a bug in Q to me... Why do you say that? Q doesn't know anything about Python or its constraints. Why should it be prohibited from invoking these callbacks in whatever way it deems appropriate for its problem domain? I know, I know, all library authors should design with Python in mind , but seriously, Q =3D=3D Qt, a library that's used extensively and successfully by thousands. It's conceivable that this is a serious design flaw in Qt, but I'm inclined to disbelieve that. I think rather that this is a library design which doesn't interoperate well with Python's constraints on GIL manipulation. In this case, as Python is intended to be good for general interoperability, it seems like Python ought to budge if possible. --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From mal@lemburg.com Mon Dec 30 18:14:10 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 30 Dec 2002 19:14:10 +0100 Subject: [Python-Dev] Mesenne Twister copyright notice In-Reply-To: <3E0F6F98.1000404@v.loewis.de> References: <3E0F6F98.1000404@v.loewis.de> Message-ID: <3E108CF2.6010206@lemburg.com> Martin v. L=F6wis wrote: > Jack Jansen wrote: >=20 >> On the other thread: will including the copyright=20 >> notices/acknowledgements in the documentation be good enough if we=20 >> don't always ship the documentation?=20 >=20 >=20 > This is exactly my concern. It is essentially for people producing=20 > binary distributions to decide how they want this to be organized. >=20 >> Or should we put this in a special section that is always shipped,=20 >> even with binary installers that don't include documentation? >=20 >=20 > For the specific case in question, I think MAL's suggestion of adding=20 > yet another file is sufficient - maintainers of binary distributions=20 > then need to collect all those text files and distribute them. Complyin= g=20 > with the various licenses is really something that the packagers most=20 > solve for themselves - the source distribution can only prepare things=20 > in a way that makes complying simple. In a later mail I suggested adding these licenses to the standard Python license file. This should probably be done as new section explaining to which code the license applies and where the code was obtained from. Then binary distributions would only have to include this one file (which they have to include anyway, since the Python license requires this). --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From martin@v.loewis.de Mon Dec 30 18:30:39 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 19:30:39 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <20021230145912.GA8481@panix.com> Message-ID: David Abrahams writes: > > So Q creates new threads which perform callbacks? But Q also performs > > the callbacks when invoked from A? Sounds like a bug in Q to me... > > Why do you say that? Q doesn't know anything about Python or its > constraints. Why should it be prohibited from invoking these > callbacks in whatever way it deems appropriate for its problem domain? It is moderately evil for a library to create threads "under the hoods", IMO; in some domains, that might be a reasonable thing to do, provided there is a way for the application author to manage the threads on a higher level (e.g. by limiting the total number of threads that the library can create simultaneously). If a library is creating new threads and invokes application code in these threads, threading should follow a threading model. That threading model has to be described, so that every application author can rely on certain features. The threading model is part of the library interface, just like the API. It appears that Q has no threading model. That is truly evil. In some cases, combining libraries with different threading models just won't work. For example, I recently found that Tcl's appartment threading model isn't really compatible with Python's GIL. It is possible to achieve interworking to some respect, but there are limitations that just can't be overcome (e.g. you just cannot invoke event dispatching in a thread that hasn't originally create the Tcl interpreter). If the threading model of Q is unknown or undefined, you cannot expect any kind of interworking. > I think rather that this is a library design which doesn't > interoperate well with Python's constraints on GIL manipulation. It seems to me that there is no design in the library, and this is the cause for the interoperability problem (or, perhaps, you just haven't presented the design). > In this case, as Python is intended to be good for general > interoperability, it seems like Python ought to budge if possible. As Tim explains, this might not be possible. Regards, Martin From dave@boost-consulting.com Mon Dec 30 18:49:04 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 13:49:04 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 19:30:39 +0100") References: <20021230145912.GA8481@panix.com> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> > So Q creates new threads which perform callbacks? But Q also performs >> > the callbacks when invoked from A? Sounds like a bug in Q to me... >>=20 >> Why do you say that? Q doesn't know anything about Python or its >> constraints. Why should it be prohibited from invoking these >> callbacks in whatever way it deems appropriate for its problem >> domain? > > It is moderately evil for a library to create threads "under the > hoods", IMO; in some domains, that might be a reasonable thing to > do, provided there is a way for the application author to manage the > threads on a higher level (e.g. by limiting the total number of > threads that the library can create simultaneously). I am not intimately familiar with Qt; these threads may not in fact be created "under the hood". Whether they are or not is IMO irrelevant to the problem we're having, because it's not how the thread is started that matters. > If a library is creating new threads and invokes application code in > these threads, threading should follow a threading model. That > threading model has to be described, so that every application > author can rely on certain features. The threading model is part of > the library interface, just like the API. > > It appears that Q has no threading model. That is truly evil. Though I am not intimately familiar with Qt, I can assure you that it *does* have a threading model. > In some cases, combining libraries with different threading models > just won't work. For example, I recently found that Tcl's appartment > threading model isn't really compatible with Python's GIL. It is > possible to achieve interworking to some respect, but there are > limitations that just can't be overcome (e.g. you just cannot invoke > event dispatching in a thread that hasn't originally create the Tcl > interpreter). > > If the threading model of Q is unknown or undefined, you cannot expect > any kind of interworking. > >> I think rather that this is a library design which doesn't >> interoperate well with Python's constraints on GIL manipulation. > > It seems to me that there is no design in the library I think that judgement is at best premature. > and this is the cause for the interoperability problem (or, > perhaps, you just haven't presented the design). No, I haven't. I'm not very familiar with it myself. I'm just relating information I've got from the author of PyQt, who is very familiar with it. >> In this case, as Python is intended to be good for general >> interoperability, it seems like Python ought to budge if possible. > > As Tim explains, this might not be possible. It sounded to me like Tim explained that it is possible but unimplemented. --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From dave@boost-consulting.com Mon Dec 30 18:54:33 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 13:54:33 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 19:11:02 +0100") References: <3E0FF293.80103@v.loewis.de> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> > What matters is whether the callback is invoked in the context of >> > the thread that installed it (i.e. as a result of calling a function >> > in B). >>=20 >> I still don't see how the thread that installed it has any bearing. > > If the callbacks would only occur while Q is being called from B, > and in the context of the thread which also hosts that call, then B > should not release the lock. Then the callback would not need to > reacquire it. AFAICT, that still has nothing to do with which thread installs the callback. It has everything to do with which threads may invoke the callback, and whether they already hold the GIL. >> There are several ways, IIUC. It may be invoked in response to >> direct calls into Q's API (which may be made from a Python >> extension module). It may also be invoked by some thread that Q >> has launched. > > I find this surprising. Is this *any* API of Q that can trigger the > callbacks, or a dispatch_one_event call that can do so? I don't know; I'll have to defer to Phil for this answer. >> Yes, we understand that problem. I had this exact discussion with >> the designer of B. He explained that the problem is that Q might >> also invoke the virtual function on a thread that is not holding >> the GIL. > > I would more deeply question such a statement. It sounds like this > library uses the callback for event processing of some kind, in > which case the callbacks are only invoked if events are > processed. It should be possible to describe more precisely under > what specific conditions event processing occurs. Probably. I can't do it, though. >> > Then I'm still curious as to what triggers invocation of that virtual >> > function. >>=20 >> Q, either directly via its API, or in some thread it started. > > Then calling this API must release the GIL. Yes, that is clearly the case under present conditions. The point is that this is merely by virtue of the fact that some other extension module may install a GIL-acquiring callback in Q. --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From martin@v.loewis.de Mon Dec 30 19:05:27 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 20:05:27 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <20021230145912.GA8481@panix.com> Message-ID: David Abrahams writes: > I am not intimately familiar with Qt Ah, so Q is Qt :-) Care to reveal what A and B are, and which the callback is that caused a deadlock? > It sounded to me like Tim explained that it is possible but > unimplemented. Trying to channel Tim: All experts for this stuff have tried and failed; Mark Hammond has a sort-of solution which Tim believes to be strictly-speaking incorrect. While people may have thought they have a solution, it is troubling that nobody can remember what the solution is. Maybe the email signature just had not enough space to write it down :-) Regards, Martin From dave@boost-consulting.com Mon Dec 30 19:19:55 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 14:19:55 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 20:05:27 +0100") References: <20021230145912.GA8481@panix.com> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> I am not intimately familiar with Qt > > Ah, so Q is Qt :-)=20 I actually revealed that several messages back. > Care to reveal what A and B are A is some extension module written by one of my users, that calls Qt but doesn't install any callbacks. B is some extension module which uses PyQt, thus installs callbacks which invoke Python. > and which the callback is that caused a deadlock? There was no deadlock, as I've said. The symptom is that Python complains at some point that there's no thread state. It goes away if A releases the GIL before calling into Qt, and reacquires the GIL afterwards. I speculate that the callback releases the GIL when it is finished, so that when A returns to Python there is no current thread. By that time, the callback has completed, so it's hard to know for sure which one it was. >> It sounded to me like Tim explained that it is possible but >> unimplemented. > > Trying to channel Tim: All experts for this stuff have tried and > failed; Mark Hammond has a sort-of solution which Tim believes to be > strictly-speaking incorrect. While people may have thought they have > a solution, it is troubling that nobody can remember what the > solution is. Maybe the email signature just had not enough space to > write it down :-) Hmm, it seems as though a mutex-protected record of which thread is currently holding the GIL should be enough to handle it. There must be subtle details which complicate the problem, e.g. that there's no portable/reliable way to identify the current thread (?) --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From guido@python.org Mon Dec 30 21:12:00 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 30 Dec 2002 16:12:00 -0500 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: Your message of "Mon, 30 Dec 2002 17:29:52 +0100." References: Message-ID: <200212302112.gBULC0S04413@odiug.zope.com> I should mention that I'm about to approve Just's additions to import.c and very close to approving his zipimport.c, so you can expect his checkins soon. This means that sys.meta_path, sys.path_hooks, and sys.path_importer_cache are new ways of hooking import. We can continue the discussion after 2.3a1 is released -- API changes are okay during alpha testing. I should note that a package's __path__ continues to exist with semantics pretty close to what they were; I am striving for Jython compatibility here. --Guido van Rossum (home page: http://www.python.org/~guido/) From martin@v.loewis.de Mon Dec 30 21:29:14 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 30 Dec 2002 22:29:14 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <20021230145912.GA8481@panix.com> Message-ID: David Abrahams writes: > There was no deadlock, as I've said. Yes, you only said this was a no-no, in http://mail.python.org/pipermail/python-dev/2002-December/031449.html I inferred from that (apparently incorrectly) that the no-no is a deadlock. > The symptom is that Python complains at some point that there's no > thread state. It goes away if A releases the GIL before calling > into Qt, and reacquires the GIL afterwards. Now I'm confused. In http://mail.python.org/pipermail/python-dev/2002-December/031424.html you said "A must also release the GIL" and then "the author of A may have had no reason to believe anyone would install Python callbacks in Q". From that I inferred that A does *not* release the GIL (as the author had no reason to). Now you are saying that A releases the GIL. Which one it is? > I speculate that the callback releases the GIL when it is finished, > so that when A returns to Python there is no current thread. That would be a bug in the callback. If there was a thread state when it was called, there should be a thread state when it returns. > Hmm, it seems as though a mutex-protected record of which thread is > currently holding the GIL should be enough to handle it. It depends on what "it" is, here. This one? Q: Is there a way to find out whether the current thread holds the GIL? If so, a mutex-protected record might work, but also might be expensive. Regards, Martin From Jack.Jansen@oratrix.com Mon Dec 30 22:11:46 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Mon, 30 Dec 2002 23:11:46 +0100 Subject: [Python-Dev] 'unhashable type' and new style classes In-Reply-To: <200212301630.gBUGUAE03901@odiug.zope.com> Message-ID: On maandag, dec 30, 2002, at 17:30 Europe/Amsterdam, Guido van Rossum wrote: > The default implementation of __hash__ must match the default > implementation of __cmp__ (and rich comparisons, __eq__ etc.). So the > default implementation cannot raise an exception, because objects are > defined to be immutable by default. (Maybe this was a mistake, but > it's not so easy to change without causing backwards > incompatibilities.) Is it an option to enforce that these methods are overridden together, then? I.e. if you override any of them you must override all of them, otherwise you get a warning? -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From guido@python.org Mon Dec 30 22:26:50 2002 From: guido@python.org (Guido van Rossum) Date: Mon, 30 Dec 2002 17:26:50 -0500 Subject: [Python-Dev] 'unhashable type' and new style classes In-Reply-To: Your message of "Mon, 30 Dec 2002 23:11:46 +0100." References: Message-ID: <200212302226.gBUMQon06368@odiug.zope.com> > On maandag, dec 30, 2002, at 17:30 Europe/Amsterdam, Guido van Rossum > wrote: > > The default implementation of __hash__ must match the default > > implementation of __cmp__ (and rich comparisons, __eq__ etc.). So the > > default implementation cannot raise an exception, because objects are > > defined to be immutable by default. (Maybe this was a mistake, but > > it's not so easy to change without causing backwards > > incompatibilities.) > > Is it an option to enforce that these methods are overridden together, > then? I.e. if you override any of them you must override all of them, > otherwise you get a warning? That's nice, but I'm not sure I can see right away how to do the warning. Also, there are cases where a base class defines both properly, and then a subclass overrides one but not the other. Is that worth a warning or not? That depends on how they change it! --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@oratrix.com Mon Dec 30 23:15:03 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 31 Dec 2002 00:15:03 +0100 Subject: [Python-Dev] 'unhashable type' and new style classes In-Reply-To: <200212302226.gBUMQon06368@odiug.zope.com> Message-ID: <86977B19-1C4C-11D7-B0FB-000A27B19B96@oratrix.com> On maandag, dec 30, 2002, at 23:26 Europe/Amsterdam, Guido van Rossum wrote: >> On maandag, dec 30, 2002, at 17:30 Europe/Amsterdam, Guido van Rossum >> wrote: >>> The default implementation of __hash__ must match the default >>> implementation of __cmp__ (and rich comparisons, __eq__ etc.). So >>> the >>> default implementation cannot raise an exception, because objects are >>> defined to be immutable by default. (Maybe this was a mistake, but >>> it's not so easy to change without causing backwards >>> incompatibilities.) >> >> Is it an option to enforce that these methods are overridden together, >> then? I.e. if you override any of them you must override all of them, >> otherwise you get a warning? > > That's nice, but I'm not sure I can see right away how to do the > warning. I assume there's some equivalent of PyType_Ready, or not? Wouldn't that be the right spot? > Also, there are cases where a base class defines both > properly, and then a subclass overrides one but not the other. Is > that worth a warning or not? That depends on how they change it! It's easy enough to add dummy methods that just call the same method in the base class. But still: this whole warning idea could be overkill, that's for you to decide:-) -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From neal@metaslash.com Mon Dec 30 23:41:39 2002 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 30 Dec 2002 18:41:39 -0500 Subject: [Python-Dev] bugs in stdlib Message-ID: <20021230234139.GE12063@epoch.metaslash.com> Here's some problems I found in the stdlib. I already fixed one problem in ConfigParser, but I'm not sure it's correct. Lib/cgitb.py:202: No global (path) found Lib/cgitb.py:204: No global (path) found Lib/ConfigParser.py:581: Invalid arguments to (__init__), got 1, expected 4 Lib/imaplib.py:444: No global (root) found Lib/pprint.py:118: Invalid arguments to (format), got 3, expected 4 Lib/pprint.py:121: Invalid arguments to (format), got 3, expected 4 Neal From DavidA@ActiveState.com Tue Dec 31 00:31:38 2002 From: DavidA@ActiveState.com (David Ascher) Date: Mon, 30 Dec 2002 16:31:38 -0800 Subject: [Python-Dev] Mesenne Twister copyright notice References: <3E0F6F98.1000404@v.loewis.de> <3E108CF2.6010206@lemburg.com> Message-ID: <3E10E56A.5020404@ActiveState.com> M.-A. Lemburg wrote: > In a later mail I suggested adding these licenses to the standard > Python license file. This should probably be done as new section > explaining to which code the license applies and where the code > was obtained from. > > Then binary distributions would only have to include this one > file (which they have to include anyway, since the Python license > requires this). This would suit us (ActiveState) just fine. In general, we don't mind doing whatever is necessary to give credit where credit is due (as well as comply with the license in general, of course) -- but the simpler it is, the more likely it is to actually be carried through -- the more automatable the better as well. --david From amk@amk.ca Tue Dec 31 01:19:33 2002 From: amk@amk.ca (A.M. Kuchling) Date: Mon, 30 Dec 2002 20:19:33 -0500 Subject: [Python-Dev] Don't touch whatsnew23.tex Message-ID: <200212310119.gBV1JXw16395@nyman.amk.ca> I'm in the middle of updating the "What's New" document for 2.3a1. Please don't make any checkins to it; if you want to make a change, please e-mail me a patch instead. (Should be finished later this evening, unless someone adds more features first...) --amk (www.amk.ca) Thank you for letting me borrow your objects. -- Ute Lemper in concert, March 13, 1997 From aahz@pythoncraft.com Tue Dec 31 00:57:36 2002 From: aahz@pythoncraft.com (Aahz) Date: Mon, 30 Dec 2002 19:57:36 -0500 Subject: [Python-Dev] GC at exit? In-Reply-To: References: <20021230034551.GA18622@panix.com> Message-ID: <20021231005736.GA15066@panix.com> On Sun, Dec 29, 2002, Tim Peters wrote: > [Aahz] >> >> Is garbage collection supposed to run when Python exits? > > The language reference manual doesn't promise that any garbage will be > collected, ever. So, no, "supposed to" doesn't apply. Thanks. I'll tell people to run gc.collect() at the end of their applications if they care. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are three kinds of lies: Lies, Damn Lies, and Statistics." --Disraeli From esr@thyrsus.com Tue Dec 31 01:43:55 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Mon, 30 Dec 2002 20:43:55 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <20021230234139.GE12063@epoch.metaslash.com> References: <20021230234139.GE12063@epoch.metaslash.com> Message-ID: <20021231014355.GA14992@thyrsus.com> Neal Norwitz : > Here's some problems I found in the stdlib. > I already fixed one problem in ConfigParser, but I'm not sure it's correct. I've got another fix and some enhancements in the works for ConfigParser. Passes all regression tests, but I'm field-testing before I commit. Just because I'm nervous about patch collisions, what code did your patch affect? -- Eric S. Raymond From dave@boost-consulting.com Tue Dec 31 04:11:06 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 23:11:06 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "30 Dec 2002 22:29:14 +0100") References: <20021230145912.GA8481@panix.com> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> There was no deadlock, as I've said.=20=20 > > Yes, you only said this was a no-no, in > > http://mail.python.org/pipermail/python-dev/2002-December/031449.html > > I inferred from that (apparently incorrectly) that the no-no is a > deadlock. In that very message, I also wrote: I realize that the docs for PyEval_AcquireLock() say: "If this thread already has the lock, a deadlock ensues", but the behavior we're seeing is consistent with a scenario where trying to acquire an already-held is a no-op and releasing it is unconditional. Eventually the GIL release in B's callback takes effect and when A returns to Python there is no thread state. >> The symptom is that Python complains at some point that there's no >> thread state. It goes away if A releases the GIL before calling >> into Qt, and reacquires the GIL afterwards. > > Now I'm confused. In > > http://mail.python.org/pipermail/python-dev/2002-December/031424.html > > you said "A must also release the GIL"=20 Yes, that's the inevitable conclusion. > and then "the author of A may have had no reason to believe anyone > would install Python callbacks in Q". From that I inferred that A > does *not* release the GIL (as the author had no reason to). Yes, not releasing the GIL in A was fine until B came along and installed callbacks in Q which acquire the GIL. > > Now you are saying that A releases the GIL. Which one it is? No, I am not saying A releases the GIL. I am saying that A that must release the GIL if it is to work properly in the presence of B. A is currently broken in the presence of B. The addition of B to the system places a new constraint on A. >> I speculate that the callback releases the GIL when it is finished, >> so that when A returns to Python there is no current thread. > > That would be a bug in the callback.=20 Not if it has previously acquired the GIL. > If there was a thread state when it was called, there should be a > thread state when it returns. Yes, the whole problem is that there's no way to know whether there's a thread state. >> Hmm, it seems as though a mutex-protected record of which thread is >> currently holding the GIL should be enough to handle it.=20 > > It depends on what "it" is, here. This one? > > Q: Is there a way to find out whether the current thread holds the GIL? > > If so, a mutex-protected record might work, but also might be > expensive. Yes. I assume that acquiring the GIL already needs to do synchronization, though. --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From martin@v.loewis.de Tue Dec 31 10:49:19 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 31 Dec 2002 11:49:19 +0100 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: References: <20021230145912.GA8481@panix.com> Message-ID: David Abrahams writes: > >> The symptom is that Python complains at some point that there's no > >> thread state. It goes away if A releases the GIL before calling > >> into Qt, and reacquires the GIL afterwards. [...] > No, I am not saying A releases the GIL. "...there is no thread state. It [the thread state] goes away if A releases the GIL ..." >From that I inferred that A releases the GIL, since you said that there is no thread state. Rereading your message, I now see that you meant "It [the problem] goes away". So I now understand that you reported that there is no deadlock, and that A does not release the GIL, and that Python reports that there is no thread state "when A returns to Python". You also report that B acquires the GIL. I can't understand why this happens. How does B acquire the GIL? Assuming that B uses PyEval_AcquireThread/PyEval_ReleaseThread, I would expect that a) there is a deadlock if this happens in a context of a call to A, since the GIL is already held, and (if, for some reason, locks are recursive on this platform), b) the code if (PyThreadState_Swap(tstate) != NULL) Py_FatalError( "PyEval_AcquireThread: non-NULL old thread state"); should trigger, as there is an old thread state. So I infer that B does not use PyEval_AcquireThread/PyEval_ReleaseThread. What else does it use? > >> I speculate that the callback releases the GIL when it is finished, > >> so that when A returns to Python there is no current thread. > > > > That would be a bug in the callback. > > Not if it has previously acquired the GIL. Even then, but I refrain from speculating how B's code to release the lock actually looks. > > If there was a thread state when it was called, there should be a > > thread state when it returns. > > Yes, the whole problem is that there's no way to know whether there's > a thread state. Wrong. If B acquires the GIL, B must use some thread state to do so. It must install that thread state through PyThreadState_Swap, directly or indirectly. That will return the old thread state, or NULL. > > If so, a mutex-protected record might work, but also might be > > expensive. > > Yes. I assume that acquiring the GIL already needs to do > synchronization, though. Sure. But with that proposed change, you have not only the GIL lock call (which is a single sem_wait call on Posix, and an InterlockedCompareExchange call on Win32). You also get a mutex call, and a call to find out the current thread. Regards, Martin From neal@metaslash.com Tue Dec 31 12:19:03 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 31 Dec 2002 07:19:03 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <20021231014355.GA14992@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> Message-ID: <20021231121903.GK12063@epoch.metaslash.com> On Mon, Dec 30, 2002 at 08:43:55PM -0500, Eric S. Raymond wrote: > > I've got another fix and some enhancements in the works for > ConfigParser. Passes all regression tests, but I'm field-testing > before I commit. Just because I'm nervous about patch collisions, > what code did your patch affect? I added class InterpolationSyntaxError(Error). Fred quickly fixed it :-), added a bunch of docstrings, fixed the constructor call I noted, and added some tests. Neal From dave@boost-consulting.com Tue Dec 31 12:24:31 2002 From: dave@boost-consulting.com (David Abrahams) Date: Tue, 31 Dec 2002 07:24:31 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL In-Reply-To: (martin@v.loewis.de's message of "31 Dec 2002 11:49:19 +0100") References: <20021230145912.GA8481@panix.com> Message-ID: martin@v.loewis.de (Martin v. L=F6wis) writes: > David Abrahams writes: > >> >> The symptom is that Python complains at some point that there's no >> >> thread state. It goes away if A releases the GIL before calling >> >> into Qt, and reacquires the GIL afterwards. > [...] >> No, I am not saying A releases the GIL.=20=20 > > "...there is no thread state. It [the thread state] goes away if A > releases the GIL ..." > > From that I inferred that A releases the GIL, since you said that > there is no thread state. Rereading your message, I now see that you > meant "It [the problem] goes away". Right. > So I now understand that you reported that there is no deadlock, and > that A does not release the GIL, and that Python reports that there is > no thread state "when A returns to Python". You also report that B > acquires the GIL. > > I can't understand why this happens. How does B acquire the GIL? > > Assuming that B uses PyEval_AcquireThread/PyEval_ReleaseThread, I > would expect that=20 > a) there is a deadlock if this happens in a context of a call to > A, since the GIL is already held, and (if, for some reason, > locks are recursive on this platform),=20 > b) the code > > if (PyThreadState_Swap(tstate) !=3D NULL) > Py_FatalError( > "PyEval_AcquireThread: non-NULL old thread state"); > > should trigger, as there is an old thread state. > > So I infer that B does not use > PyEval_AcquireThread/PyEval_ReleaseThread. What else does it use? Looking at the SIP sources, it appears to be using PyEval_SaveThread/PyEval_RestoreThread, but I'd have to ask Phil to weigh in on this one to know for sure. Here's a stack backtrace reported by my user. You can ignore the oddness of frame #4; the SIP author is patching Python's instance method table, but has convinced me that what he's doing is harmless (it's still evil, of course ). Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1024 (LWP 5948)] PyErr_SetObject (exception=3D0x8108a8c, value=3D0x81ab208) at Python/erro= rs.c:39 (gdb) bt #0 PyErr_SetObject (exception=3D0x8108a8c, value=3D0x81ab208) at Python/errors.c:39 #1 0x08087ac7 in PyErr_Format (exception=3D0x8108a8c,=20 format=3D0x80df620 "%.50s instance has no attribute '%.400s'") at Python/errors.c:408 #2 0x080b0467 in instance_getattr1 (inst=3D0x82c5654, name=3D0x8154558) at Objects/classobject.c:678 #3 0x080b3e35 in instance_getattr (inst=3D0x82c5654, name=3D0x8154558) at Objects/classobject.c:715 #4 0x40cd2a43 in instanceGetAttr () from /usr/local/lib/python2.2/site-packages/libsip.so #5 0x08056794 in PyObject_GetAttr (v=3D0x82c5654, name=3D0x8154558) at Objects/object.c:1108 #6 0x0807705e in eval_frame (f=3D0x811a974) at Python/ceval.c:1784 #7 0x0807866e in PyEval_EvalCodeEx (co=3D0x8161de0, globals=3D0x81139b4,= =20 locals=3D0x81139b4, args=3D0x0, argcount=3D0, kws=3D0x0, kwcount=3D0,= defs=3D0x0,=20 defcount=3D0, closure=3D0x0) at Python/ceval.c:2595 #8 0x0807a700 in PyEval_EvalCode (co=3D0x8161de0, globals=3D0x81139b4,=20 locals=3D0x81139b4) at Python/ceval.c:481 #9 0x080950b1 in run_node (n=3D0x81263b8,=20 filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p= y",=20 globals=3D0x81139b4, locals=3D0x81139b4, flags=3D0xbffffac4) at Python/pythonrun.c:1079 #10 0x08095062 in run_err_node (n=3D0x81263b8,=20 filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p= y",=20 globals=3D0x81139b4, locals=3D0x81139b4, flags=3D0xbffffac4) at Python/pythonrun.c:1066 #11 0x08094ccb in PyRun_FileExFlags (fp=3D0x8104038,=20 filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p= y",=20 start=3D257, globals=3D0x81139b4, locals=3D0x81139b4, closeit=3D1,=20 flags=3D0xbffffac4) at Python/pythonrun.c:1057 #12 0x080938b1 in PyRun_SimpleFileExFlags (fp=3D0x8104038,=20 filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p= y",=20 closeit=3D1, flags=3D0xbffffac4) at Python/pythonrun.c:685 #13 0x0809481f in PyRun_AnyFileExFlags (fp=3D0x8104038,=20 filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p= y",=20 closeit=3D1, flags=3D0xbffffac4) at Python/pythonrun.c:495 #14 0x08053632 in Py_Main (argc=3D2, argv=3D0xbffffb54) at Modules/main.c= :364 #15 0x08052ee6 in main (argc=3D2, argv=3D0xbffffb54) at Modules/python.c:= 10 #16 0x40088627 in __libc_start_main (main=3D0x8052ed0
, argc=3D2,=20 ubp_av=3D0xbffffb54, init=3D0x80522d4 <_init>, fini=3D0x80cf610 <_fin= i>,=20 rtld_fini=3D0x4000dcd4 <_dl_fini>, stack_end=3D0xbffffb4c) at ../sysdeps/generic/libc-start.c:129 (gdb)=20 On the line of the error oldtype =3D tstate->curexc_type; (gdb) p tstate $1 =3D (PyThreadState *) 0x0 (gdb)=20 >> > If there was a thread state when it was called, there should be a >> > thread state when it returns. >>=20 >> Yes, the whole problem is that there's no way to know whether there's >> a thread state. > > Wrong. If B acquires the GIL, B must use some thread state to do > so. It must install that thread state through PyThreadState_Swap, > directly or indirectly. That will return the old thread state, or > NULL. Let me rephrase: the whole problem is that there's no way to know if you have the interpreter lock. You can't call PyThreadState_Swap to find out if there's a thread state if you don't have the interpreter lock. You can't acquire the lock if you already have it. >> > If so, a mutex-protected record might work, but also might be >> > expensive. >>=20 >> Yes. I assume that acquiring the GIL already needs to do >> synchronization, though. > > Sure. But with that proposed change, you have not only the GIL lock > call (which is a single sem_wait call on Posix, and an > InterlockedCompareExchange call on Win32). You also get a mutex > call, and a call to find out the current thread. There you go, it's a harder problem than I thought ;-) --=20 David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution From mal@lemburg.com Tue Dec 31 13:08:19 2002 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 31 Dec 2002 14:08:19 +0100 Subject: [Python-Dev] Third-Party Licenses (Mesenne Twister copyright notice) In-Reply-To: <3E10E56A.5020404@ActiveState.com> References: <3E0F6F98.1000404@v.loewis.de> <3E108CF2.6010206@lemburg.com> <3E10E56A.5020404@ActiveState.com> Message-ID: <3E1196C3.70805@lemburg.com> David Ascher wrote: > M.-A. Lemburg wrote: > >> In a later mail I suggested adding these licenses to the standard >> Python license file. This should probably be done as new section >> explaining to which code the license applies and where the code >> was obtained from. >> >> Then binary distributions would only have to include this one >> file (which they have to include anyway, since the Python license >> requires this). > > > This would suit us (ActiveState) just fine. In general, we don't mind > doing whatever is necessary to give credit where credit is due (as well > as comply with the license in general, of course) -- but the simpler it > is, the more likely it is to actually be carried through -- the more > automatable the better as well. So what's the general feeling ? Should we take this approach ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Tue Dec 31 14:02:33 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 09:02:33 -0500 Subject: [Python-Dev] Python 2.3a1 release status Message-ID: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net> I'm planning for a release by 6pm EST today. I'm happy with the changes that've been checked in overnight and the quality control checks that have been made. I'm waiting for one more big change to land: an important API change to datetime.c that was already implemented in the .py version in the sandbox. I expect this around lunchtime (EST) today. Once that change is committed, I'll create a release branch, declare a checkin moratorium for people outside PythonLabs, and start working on the final stretch of the release process (see PEP 101 if you want to know how much work that is). If you know of other pending changes that haven't been committed, either check them in (only if they're stable, tested and documented, of course!) or let me know ASAP. Jack Jansen: I don't expect that you're quite ready with the Mac tree. Or are you? Let me know what you want me to say about the Mac in the announcement and on the web. The draft text for the website is already visible at http://www.python.org/2.3/. If you think you can use this to get some short-term gain on the stock market, please share. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Tue Dec 31 14:09:37 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 31 Dec 2002 15:09:37 +0100 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C In-Reply-To: <3E106BBB.4020909@interet.com> Message-ID: [JvR] > > but [sys.path_importer_cache is] not a new hook, it's a cache for > > the path_hooks hook. So I think the PEP proposes only two new > > hooks. [James C. Ahlstrom] > I disagree. It is publicly available in sys, and the PEP specifies > it can be replaced or modified. And if you want to add a hook, it > is easier to put it into sys.path_importer_cache than to add it > to the list in sys.path_hooks. If you add it to sys.path_hooks, > you must clear its entry in sys.path_importer_cache too. This is all only half true. sys.path_importer_cache is exposed so you can *clear* it in case you install a path hook that should take over sys.path items that may already have a handler. To only partially clear it is advanced usage. By no means you're supposed to *add* to sys.path_importer_cache explicitly, as someone after you may clear it again, and then you're hosed if there's no proper path hook. I also disagree that adding an importer to sys.path_importer_cache would always be *easier* than doing it the right way by adding a hook to sys.path_hooks. If would be somewhat easier for "cookie" path items, but I think in 99% of these cases a path-less importer on sys.meta_path would work just as well. I guess I should start on some proper documentation... Just From martin@v.loewis.de Tue Dec 31 14:46:51 2002 From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) Date: 31 Dec 2002 15:46:51 +0100 Subject: [Python-Dev] Python 2.3a1 release status In-Reply-To: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net> References: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > I'm planning for a release by 6pm EST today. It appears that IDLEfork will not be merged for 2.3a1. Is there a plan to merge it for 2.3? Regards, Martin From esr@thyrsus.com Tue Dec 31 14:56:39 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 09:56:39 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <20021231121903.GK12063@epoch.metaslash.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> Message-ID: <20021231145639.GA22240@thyrsus.com> Neal Norwitz : > On Mon, Dec 30, 2002 at 08:43:55PM -0500, Eric S. Raymond wrote: > > I've got another fix and some enhancements in the works for > > ConfigParser. Passes all regression tests, but I'm field-testing > > before I commit. Just because I'm nervous about patch collisions, > > what code did your patch affect? > > I added class InterpolationSyntaxError(Error). Fred quickly fixed it > :-), added a bunch of docstrings, fixed the constructor call I noted, > and added some tests. Doesn't sound like there should be a problem. Most of my changes are to the _read logic -- I also added a bunch of regression tests for the new features, but I did it in such a way that collision seems unlikely. I'll check them in today or tomorrow and make sure everything is copacetic. -- Eric S. Raymond From guido@python.org Tue Dec 31 15:18:31 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 10:18:31 -0500 Subject: [Python-Dev] Python 2.3a1 release status In-Reply-To: Your message of "31 Dec 2002 15:46:51 +0100." References: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200212311518.gBVFIV914092@odiug.zope.com> > > I'm planning for a release by 6pm EST today. > > It appears that IDLEfork will not be merged for 2.3a1. Is there a plan > to merge it for 2.3? Yes, I hope they'll be ready for 2.3a2. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Dec 31 15:22:18 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 10:22:18 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: Your message of "Tue, 31 Dec 2002 09:56:39 EST." <20021231145639.GA22240@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> Message-ID: <200212311522.gBVFMIY14143@odiug.zope.com> > Neal Norwitz : > > On Mon, Dec 30, 2002 at 08:43:55PM -0500, Eric S. Raymond wrote: > > > I've got another fix and some enhancements in the works for > > > ConfigParser. Passes all regression tests, but I'm field-testing > > > before I commit. Just because I'm nervous about patch collisions, > > > what code did your patch affect? > > > > I added class InterpolationSyntaxError(Error). Fred quickly fixed it > > :-), added a bunch of docstrings, fixed the constructor call I noted, > > and added some tests. > > Doesn't sound like there should be a problem. Most of my changes are > to the _read logic -- I also added a bunch of regression tests for > the new features, but I did it in such a way that collision seems > unlikely. I'll check them in today or tomorrow and make sure > everything is copacetic. Eric, can you put off your checkin until after I've created the 2.3a1 release branch? --Guido van Rossum (home page: http://www.python.org/~guido/) From esr@snark.thyrsus.com Tue Dec 31 15:27:58 2002 From: esr@snark.thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 10:27:58 -0500 Subject: [Python-Dev] ConfigParser changes Message-ID: <200212311527.gBVFRwS9022537@snark.thyrsus.com> I merged in my changes and regression-tested everything. Looks like we're OK, all tests passed including my new ones. I also checked in a remnant doc change for some nntplib enhancements I did a month or so ago. This flushes my patch queue. -- Eric S. Raymond "Government is not reason, it is not eloquence, it is force; like fire, a troublesome servant and a fearful master. Never for a moment should it be left to irresponsible action." -- George Washington, in a speech of January 7, 1790 From esr@thyrsus.com Tue Dec 31 15:35:38 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 10:35:38 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <200212311522.gBVFMIY14143@odiug.zope.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> Message-ID: <20021231153538.GA22602@thyrsus.com> Guido van Rossum : > Eric, can you put off your checkin until after I've created the 2.3a1 > release branch? Ouch. Our mails crossed. I just checked them in. You can back them out if you're feeling really paranoid -- but all the unit tests pass. -- Eric S. Raymond From guido@python.org Tue Dec 31 15:50:45 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 10:50:45 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: Your message of "Tue, 31 Dec 2002 10:35:38 EST." <20021231153538.GA22602@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> Message-ID: <200212311550.gBVFojY09030@odiug.zope.com> > Guido van Rossum : > > Eric, can you put off your checkin until after I've created the 2.3a1 > > release branch? > > Ouch. Our mails crossed. I just checked them in. You can back them out > if you're feeling really paranoid -- but all the unit tests pass. I find it hard to believe that our mails crossed. The first time I asked you to wait until after the release was last night: | Subject: Re: [Python-Dev] bugs in stdlib | From: Guido van Rossum | To: esr@thyrsus.com | Date: Mon, 30 Dec 2002 20:58:25 -0500 | | > I've got another fix and some enhancements in the works for | > ConfigParser. Passes all regression tests, but I'm field-testing | > before I commit. Just because I'm nervous about patch collisions, | > what code did your patch affect? | | CVS is your friend, Eric. :-) | | Can you put off checking in your changes until after the 2.3a1 | release tomorrow? I'm a bit nervous about last-minute changes to | ConfigParser, based on the history of that module. | | --Guido van Rossum (home page: http://www.python.org/~guido/) If you saw that and chose to ignore my advice, I would've appreciated a reply at least. I'll leave the decision to revert your checkin to Fred, who knows the code (and has pending changes to it of his own, which now cause conflicts). --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Tue Dec 31 15:58:48 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 31 Dec 2002 16:58:48 +0100 Subject: [Python-Dev] Python 2.3a1 release status In-Reply-To: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > If you know of other pending changes that haven't been committed, > either check them in (only if they're stable, tested and documented, > of course!) or let me know ASAP. One thing I completely forgot to persue because of PEP 302 was the plan to move Tools/freeze/modulefinder.py to Lib/. I would love for this to be in 2.3a1, but nor Thomas Heller nor I have written the needed documentation for it yet. If you allow it to go into 2.3a1, I promise to submit documentation well before 2.3a2. There's also a patch from me for modulefinder.py sitting on sf which would be nice to make it in at the same time: http://www.python.org/sf/643711 It vastly reduces the number of false warnings about missing submodules. Additionally it adds a new API for getting info about missing modules which distinguishes between "surely missing" and "maybe missing". Just From guido@python.org Tue Dec 31 16:03:38 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 11:03:38 -0500 Subject: [Python-Dev] Python 2.3a1 release status In-Reply-To: Your message of "Tue, 31 Dec 2002 16:58:48 +0100." References: Message-ID: <200212311603.gBVG3cH16115@odiug.zope.com> > One thing I completely forgot to persue because of PEP 302 was the plan > to move Tools/freeze/modulefinder.py to Lib/. I would love for this to > be in 2.3a1, but nor Thomas Heller nor I have written the needed > documentation for it yet. If you allow it to go into 2.3a1, I promise to > submit documentation well before 2.3a2. Fine. At least add a note to Misc/NEWS, and make sure that the freeze tool still works. > There's also a patch from me for modulefinder.py sitting on sf which > would be nice to make it in at the same time: > http://www.python.org/sf/643711 > It vastly reduces the number of false warnings about missing submodules. > Additionally it adds a new API for getting info about missing modules > which distinguishes between "surely missing" and "maybe missing". Ditto. It looks like it's going to be at least a couple more hours before datetime.c is ready anyway. --Guido van Rossum (home page: http://www.python.org/~guido/) From esr@thyrsus.com Tue Dec 31 16:03:47 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 11:03:47 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <200212311550.gBVFojY09030@odiug.zope.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> Message-ID: <20021231160347.GA22796@thyrsus.com> Guido van Rossum : > If you saw that and chose to ignore my advice, I would've appreciated > a reply at least. I did not. I would have heeded that advice. -- Eric S. Raymond From lalo@laranja.org Tue Dec 31 16:14:45 2002 From: lalo@laranja.org (Lalo Martins) Date: Tue, 31 Dec 2002 14:14:45 -0200 Subject: [Python-Dev] sstruct Message-ID: <20021231161445.GC31681@laranja.org> While idly browsing around, I came across Just's module sstruct. Any good reason this is not in the library? It is a kind of cross between struct and ctypes, only a few years more mature. With new-style classes it could probably gain some nifty features. http://starship.python.net/~just/code/sstruct.py []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From martin@v.loewis.de Tue Dec 31 16:30:01 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 31 Dec 2002 17:30:01 +0100 Subject: [Python-Dev] sstruct In-Reply-To: <20021231161445.GC31681@laranja.org> References: <20021231161445.GC31681@laranja.org> Message-ID: <3E11C609.5000206@v.loewis.de> Lalo Martins wrote: > While idly browsing around, I came across Just's module sstruct. Any good > reason this is not in the library? I can think of two reasons: 1. Nobody has proposed to include it, yet. 2. It is not obvious that it is useful. On the latter point: What application is finding it so useful that they incorporate it in their distribution? Regards, Martin From fdrake@acm.org Tue Dec 31 16:31:47 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 31 Dec 2002 11:31:47 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <200212311550.gBVFojY09030@odiug.zope.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> Message-ID: <15889.50803.219337.935244@grendel.zope.com> Guido van Rossum writes: > I'll leave the decision to revert your checkin to Fred, who knows the > code (and has pending changes to it of his own, which now cause > conflicts). The conflict was minor; Eric managed to add a blank line at the end of the file, and I'd made adjustments to the exception being raised there. Regarding Eric's changes, I'd like to see them discussed before being made part of the library; the interface seems quite ad hoc, and it's not clear that they are the right way to go about adding sequence support. What I've seen for sequences in INI files in the past is something like this (working from memory): [SomeSection] NumPathEntries=3 PathEntry1=c:\path\one PathEntry2=c:\path\two PathEntry3=c:\path\three (I'm not advocating that as the "right" way to do it, I'm just noting that what I've seen in practice is very different from the solution Eric implemented.) The value of having __str__() return a parsable INI file is highly questionable; the motivation should be explained and discussed. The issue of ordering sections and options in the same way as the input seems unimportant as well; comments are lost anyway. Being able to surgically edit a config file using the ConfigParser module is simply not a supported use case. (I wrote a module once that did support surgical editing, and it was a royal pain! Long lost in a disk crash.) I'll be reverting the change shortly. Eric, I would like to encourage you to pursue this functionality via discussion on python-dev with the patch submitted via SourceForge. The functional extensions are not unwelcome by any means. NB: For a very different approach to handling configuration, there's something in Zope's CVS called "ZConfig". Of particular interest is the development branch that includes support for configuration schema: http://cvs.zope.org/Packages/ZConfig/?only_with_tag=zconfig-schema-devel-branch BTW, I'm back on python-dev now, but it's still a good idea to CC things that need my specific attention. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From esr@thyrsus.com Tue Dec 31 16:24:06 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 11:24:06 -0500 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <20021231160347.GA22796@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <20021231160347.GA22796@thyrsus.com> Message-ID: <20021231162406.GA22850@thyrsus.com> Eric S. Raymond : > Guido van Rossum : > > If you saw that and chose to ignore my advice, I would've appreciated > > a reply at least. > > I did not. I would have heeded that advice. On reflection, more explanation seems called for... I've been flooded in a particularly repellent way recently. Somebody is using my From address for spam mail about quack remedies for prostate trouble, somebody so illiterate that the Subject line read "urnination flow". As a result, I've been getting several *thousand* bounce messages a day from sites rightly rejecting this crud, and hitting the 'd' key in rapid-fire mode. I may have inadvertently flushed your mail, dammit, which is not something I would ever do deliberately. I subjunctively apologize. You may not be alone, I fear. I almost blew past mail from Dennis Ritchie last night, just barely caught myself in time. I'm wondering now what other important bits I lost... :-( :-( :-( I do not regard the ConfigParser changes as at all urgent -- the bug is a minor one. So I won't be offended if Fred chooses to back them out. -- Eric S. Raymond From skip@pobox.com Tue Dec 31 16:44:55 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 31 Dec 2002 10:44:55 -0600 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <20021231162406.GA22850@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <20021231160347.GA22796@thyrsus.com> <20021231162406.GA22850@thyrsus.com> Message-ID: <15889.51591.384682.949437@montanaro.dyndns.org> Eric> I've been flooded in a particularly repellent way recently. Eric> Somebody is using my From address for spam mail about quack Eric> remedies for prostate trouble, somebody so illiterate that the Eric> Subject line read "urnination flow". As a result, I've been Eric> getting several *thousand* bounce messages a day from sites Eric> rightly rejecting this crud, and hitting the 'd' key in rapid-fire Eric> mode. Sounds like you need to install spambayes or bogofilter. ;-) Skip From barry@python.org Tue Dec 31 16:51:54 2002 From: barry@python.org (Barry A. Warsaw) Date: Tue, 31 Dec 2002 11:51:54 -0500 Subject: [Python-Dev] bugs in stdlib References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <20021231160347.GA22796@thyrsus.com> <20021231162406.GA22850@thyrsus.com> Message-ID: <15889.52010.949501.208196@gargle.gargle.HOWL> >>>>> "ESR" == Eric S Raymond writes: ESR> I've been flooded in a particularly repellent way recently. ESR> Somebody is using my From address for spam mail about quack ESR> remedies for prostate trouble, somebody so illiterate that ESR> the Subject line read "urnination flow". Welcome to the world of Klez. Someone who has you in their address book is infected. -Barry From skip@pobox.com Tue Dec 31 17:04:24 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 31 Dec 2002 11:04:24 -0600 Subject: [Python-Dev] Patch 640843 - split OPT into two parts Message-ID: <15889.52760.92971.951623@montanaro.dyndns.org> I took a break from my fiddling with the OPT make/configure variable when I discovered the problems with out-of-tree builds. Andrew got that fixed a couple days ago, so I revisited my changes. I'd like to check them in, but should probably wait until Guido bundles 2.3a1. Changes are to these files: configure.in (and configure, of course) setup.py Makefile.pre.in Misc/NEWS There are some structural changes in configure.in that make it worth a careful look-see. In particular, I hope I haven't bolluxed up Martin's SCO changes from September (I can't test on that platform). Here's the new text from Misc/NEWS: - On systems which build using the configure script, compiler flags which used to be lumped together using the OPT flag have been split into two groups, OPT and BASECFLAGS. OPT is meant to carry just optimization- and debug-related flags like "-g" and "-O3". BASECFLAGS is meant to carry compiler flags that are required to get a clean compile. On some platforms (many Linux flavors in particular) BASECFLAGS will be empty by default. On others, such as MacOSX and SCO, it will contain required flags. This change allows people building Python to override OPT without fear of clobbering compiler flags which are required to get a clean build. Everything passes on both MacOSX and Mandrake 8.1 from CVS as of a few minutes ago. I uploaded a new context diff to SF so people can eyeball it, though I'll wait for Guido to say "check it in". There's one itty-bitty change to setup.py that's unrelated. It adds the fink directories /sw/lib and /sw/include to the relevant search lists when building on Darwin. Skip From esr@thyrsus.com Tue Dec 31 16:57:01 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 11:57:01 -0500 Subject: [Python-Dev] ConfigParser changes In-Reply-To: <15889.50803.219337.935244@grendel.zope.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> Message-ID: <20021231165701.GA22958@thyrsus.com> Fred L. Drake, Jr. : > Regarding Eric's changes, I'd like to see them discussed before being > made part of the library; the interface seems quite ad hoc, and it's > not clear that they are the right way to go about adding sequence > support. Do you have the same reaction to the multiline string support? And do you want the line-continuation bugfix? > The value of having __str__() return a parsable INI file is highly > questionable; the motivation should be explained and discussed. Cleanliness. I believe in invertible representations. Whenever I write a class, I like to have its str() or repr() emit something parseable. If nothing else, this is useful for debugging. This is just general principles, I don't (unlike my other changes) have a use case for it. > The issue of ordering sections and options in the same way as the input > seems unimportant as well; comments are lost anyway. Being able to > surgically edit a config file using the ConfigParser module is simply > not a supported use case. It is now. That was the point of the enhancements. > I'll be reverting the change shortly. Eric, I would like to encourage > you to pursue this functionality via discussion on python-dev with the > patch submitted via SourceForge. The functional extensions are not > unwelcome by any means. OK. Then let the discussion begin. Here are the issues: 1. There is a minor bug in the way continuations are handled that, in some circumstances, can mean the write() representation is not invertible. 2. Currently ConfigParser cannot support multiline strings -- that is, values that are multiline and have embedded whitespace. I need this capability. I added support for this through a class member that lets you declare string quotes. Is there some objection to supporting this value type, or just to the magic-class-member interface? 3. Currently ConfigParser cannot support structured values. Again, I need this capability (for association lists of coordinates and names, as it happens). The syntax I have implemented is a configurable list bracket character surrounding comma-separated lists. The alternative Fred suggests is, I think, extremely ugly. If anyone has a more natural suggestion than my proposal, I'm willing to hear it. 4. I *do* in fact want to support `surgical' alteration of .INI files. I have a use case for this in a configuration editor I'm writing for FreeCiv. More generally, when writing code to support invertible representations, I think it is good citizenship to perturb the original data as little as possible. Thus I regard the fact that the present code permutes options and sections into an arbitrary order dependent on the hash table implementation as a design bug, and actually a fairly serious sort of thoughtlessness. -- Eric S. Raymond From guido@python.org Tue Dec 31 17:13:56 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 12:13:56 -0500 Subject: [Python-Dev] ConfigParser changes In-Reply-To: Your message of "Tue, 31 Dec 2002 11:57:01 EST." <20021231165701.GA22958@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> Message-ID: <200212311713.gBVHDvJ24099@odiug.zope.com> > Cleanliness. I believe in invertible representations. Whenever I > write a class, I like to have its str() or repr() emit something > parseable. If nothing else, this is useful for debugging. This is > just general principles, I don't (unlike my other changes) have a > use case for it. I don't like classes whose str() or repr() can be many lines in a typical case. Since this is read from a file, there should be (and is) a method to write it to a file; if there's no use case for converting the whole shebang to a string, don't provide that, and especially not as str() or repr(). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Dec 31 17:26:26 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 12:26:26 -0500 Subject: [Python-Dev] Patch 640843 - split OPT into two parts In-Reply-To: Your message of "Tue, 31 Dec 2002 11:04:24 CST." <15889.52760.92971.951623@montanaro.dyndns.org> References: <15889.52760.92971.951623@montanaro.dyndns.org> Message-ID: <200212311726.gBVHQQs24193@odiug.zope.com> > I took a break from my fiddling with the OPT make/configure variable when I > discovered the problems with out-of-tree builds. Andrew got that fixed a > couple days ago, so I revisited my changes. I'd like to check them in, but > should probably wait until Guido bundles 2.3a1. Changes are to these files: > > configure.in (and configure, of course) > setup.py > Makefile.pre.in > Misc/NEWS > > There are some structural changes in configure.in that make it worth a > careful look-see. In particular, I hope I haven't bolluxed up Martin's SCO > changes from September (I can't test on that platform). > > Here's the new text from Misc/NEWS: > > - On systems which build using the configure script, compiler flags which > used to be lumped together using the OPT flag have been split into two > groups, OPT and BASECFLAGS. OPT is meant to carry just optimization- and > debug-related flags like "-g" and "-O3". BASECFLAGS is meant to carry > compiler flags that are required to get a clean compile. On some > platforms (many Linux flavors in particular) BASECFLAGS will be empty by > default. On others, such as MacOSX and SCO, it will contain required > flags. This change allows people building Python to override OPT without > fear of clobbering compiler flags which are required to get a clean build. > > Everything passes on both MacOSX and Mandrake 8.1 from CVS as of a few > minutes ago. > > I uploaded a new context diff to SF so people can eyeball it, though I'll > wait for Guido to say "check it in". There's one itty-bitty change to > setup.py that's unrelated. It adds the fink directories /sw/lib and > /sw/include to the relevant search lists when building on Darwin. Are you sure you want to risk this right before the release? If it breaks some platform's build, we're hosed. OTOH if you want to risk it, I'm not going to stop you -- I agree that in principle the change is a desirable one. --Guido van Rossum (home page: http://www.python.org/~guido/) From python@rcn.com Tue Dec 31 17:32:18 2002 From: python@rcn.com (Raymond Hettinger) Date: Tue, 31 Dec 2002 12:32:18 -0500 Subject: [Python-Dev] test_coercion Message-ID: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer> Can anyone reproduce this test failure: C:\py23\Lib\test>python_d regrtest.py test_coercion test_coercion test test_coercion produced unexpected output: ********************************************************************** *** mismatch between line 523 of expected output and line 523 of actual output: - [1] *= => [1] + [1] *= ... exceptions.TypeError *** mismatch between line 535 of expected output and line 535 of actual output: - [1] *= => [1, 1] + [1] *= ... exceptions.TypeError *********************************************************************** 1 test failed: test_coercion [12762 refs] From esr@thyrsus.com Tue Dec 31 17:26:42 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 12:26:42 -0500 Subject: [Python-Dev] ConfigParser changes In-Reply-To: <200212311713.gBVHDvJ24099@odiug.zope.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <200212311713.gBVHDvJ24099@odiug.zope.com> Message-ID: <20021231172642.GC22958@thyrsus.com> Guido van Rossum : > I don't like classes whose str() or repr() can be many lines in a > typical case. Since this is read from a file, there should be (and > is) a method to write it to a file; if there's no use case for > converting the whole shebang to a string, don't provide that, and > especially not as str() or repr(). Noted for the futuee, thanks. -- Eric S. Raymond From fdrake@acm.org Tue Dec 31 17:36:15 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 31 Dec 2002 12:36:15 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <20021231165701.GA22958@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> Message-ID: <15889.54671.243224.307879@grendel.zope.com> Eric S. Raymond writes: > Do you have the same reaction to the multiline string support? Yes. It should definately be discussed. > And do you want the line-continuation bugfix? That would be nice to have; I should have responded to that point separately. Feel free to commit a minimal patch to fix that, and add a regression test to Lib/test/test_cfgparser.py. > > The value of having __str__() return a parsable INI file is highly > > questionable; the motivation should be explained and discussed. > > Cleanliness. I believe in invertible representations. Whenever I write > a class, I like to have its str() or repr() emit something parseable. > If nothing else, this is useful for debugging. This is just general > principles, I don't (unlike my other changes) have a use case for it. I don't think it's clear that using __str__() for this is valuable. Creating a potentially large string in memory may not be the best approach; I actually like the write() method better; the caller can decide to use a StringIO if that's what makes sense, or toss things to a real file if needed. If others like this, I won't object. It should be a separate patch from the other changes, though. > > The issue of ordering sections and options in the same way as the input > > seems unimportant as well; comments are lost anyway. Being able to > > surgically edit a config file using the ConfigParser module is simply > > not a supported use case. > > It is now. That was the point of the enhancements. Perhaps I missed it; did you preserve comments as well? > OK. Then let the discussion begin. Here are the issues: Excellent! Thanks. > 1. There is a minor bug in the way continuations are handled that, in > some circumstances, can mean the write() representation is not invertible. Per above, that can be fixed at any time. Bugs are written to be squashed. ;-) > 2. Currently ConfigParser cannot support multiline strings -- that > is, values that are multiline and have embedded whitespace. I need > this capability. I added support for this through a class member > that lets you declare string quotes. Is there some objection to > supporting this value type, or just to the magic-class-member > interface? I think the interface should be discussed; your implementation may be sufficient, but after working with pyexpat as much as I have, I've developed a strong dislike for attributes that have substantial side effects. If we're going to add a special syntax for multi-line values, we may need to think about encoding special values, text encoding, and Unicode. And that's never easy to get concensus on. ;-) > 3. Currently ConfigParser cannot support structured values. Again, I > need this capability (for association lists of coordinates and names, > as it happens). The syntax I have implemented is a configurable list > bracket character surrounding comma-separated lists. The alternative > Fred suggests is, I think, extremely ugly. If anyone has a more > natural suggestion than my proposal, I'm willing to hear it. I think "suggests" is too strong a word; I was only citing precedence, not advocating that approach. I think it's pretty ugly as well. > 4. I *do* in fact want to support `surgical' alteration of .INI files. > I have a use case for this in a configuration editor I'm writing for > FreeCiv. More generally, when writing code to support invertible > representations, I think it is good citizenship to perturb the > original data as little as possible. Thus I regard the fact that I think there's a distinction between reading a configuration and editing it: The original code never wrote a file back out. That was something you added in revision 1.19. > the present code permutes options and sections into an arbitrary > order dependent on the hash table implementation as a design bug, and > actually a fairly serious sort of thoughtlessness. Only if you write the file back out. ;-) Seriously, I have no objection to supporting surgical editing. I do think that separate classes should be used for read-only uses and read-write uses; a read-only ConfigParser-thingie should remain very lightweight, and surgical editing just isn't that. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From bellman+pep-divmod@lysator.liu.se Tue Dec 31 17:38:37 2002 From: bellman+pep-divmod@lysator.liu.se (Thomas Bellman) Date: Tue, 31 Dec 2002 18:38:37 +0100 (MET) Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors Message-ID: <200212311738.SAA02244@proton.lysator.liu.se> I humbly submit this PEP for your dissection. As I am not subscribed to python-dev, please make sure that your comments are CC:ed to , so I can see them. I have also posted this to comp.lang.python. I have written an implementation also, but I need to check it some more to see if I've got the reference counting correct before I dare post it. :-) PEP: 303 Title: Extend divmod() for Multiple Divisors Version: $Revision: 1.2 $ Last-Modified: $Date: 2002/12/31 16:02:49 $ Author: Thomas Bellman Status: Draft Type: Standards Track Content-Type: text/plain Created: 31-Dec-2002 Python-Version: 2.3 Post-History: Abstract This PEP describes an extension to the built-in divmod() function, allowing it to take multiple divisors, chaining several calls to divmod() into one. Specification The built-in divmod() function would be changed to accept multiple divisors, changing its signature from divmod(dividend, divisor) to divmod(dividend, *divisors). The dividend is divided by the last divisor, giving a quotient and a remainder. The quotient is then divided by the second to last divisor, giving a new quotient and remainder. This is repeated until all divisors have been used, and divmod() then returns a tuple consisting of the quotient from the last step, and the remainders from all the steps. A Python implementation of the new divmod() behaviour could look like: def divmod(dividend, *divisors): modulos = () q = dividend while divisors: q,r = q.__divmod__(divisors[-1]) modulos = (r,) + modulos divisors = divisors[:-1] return (q,) + modulos Motivation Occasionally one wants to perform a chain of divmod() operations, calling divmod() on the quotient from the previous step, with varying divisors. The most common case is probably converting a number of seconds into weeks, days, hours, minutes and seconds. This would today be written as: def secs_to_wdhms(seconds): m,s = divmod(seconds, 60) h,m = divmod(m, 60) d,h = divmod(h, 24) w,d = divmod(d, 7) return (w,d,h,m,s) This is tedious and easy to get wrong each time you need it. If instead the divmod() built-in is changed according the proposal, the code for converting seconds to weeks, days, hours, minutes and seconds then become def secs_to_wdhms(seconds): w,d,h,m,s = divmod(seconds, 7, 24, 60, 60) return (w,d,h,m,s) which is easier to type, easier to type correctly, and easier to read. Other applications are: - Astronomical angles (declination is measured in degrees, minutes and seconds, right ascension is measured in hours, minutes and seconds). - Old British currency (1 pound = 20 shilling, 1 shilling = 12 pence) - Anglo-Saxon length units: 1 mile = 1760 yards, 1 yard = 3 feet, 1 foot = 12 inches. - Anglo-Saxon weight units: 1 long ton = 160 stone, 1 stone = 14 pounds, 1 pound = 16 ounce, 1 ounce = 16 dram - British volumes: 1 gallon = 4 quart, 1 quart = 2 pint, 1 pint = 20 fluid ounces Rationale The idea comes from APL, which has an operator that does this. (I don't remember what the operator looks like, and it would probably be impossible to render in ASCII anyway.) The APL operator takes a list as its second operand, while this PEP proposes that each divisor should be a separate argument to the divmod() function. This is mainly because it is expected that the most common uses will have the divisors as constants right in the call (as the 7, 24, 60, 60 above), and adding a set of parentheses or brackets would just clutter the call. Requiring an explicit sequence as the second argument to divmod() would seriously break backwards compatibility. Making divmod() check its second argument for being a sequence is deemed to be too ugly to contemplate. And in the case where one *does* have a sequence that is computed other-where, it is easy enough to write divmod(x, *divs) instead. Requiring at least one divisor, i.e rejecting divmod(x), has been considered, but no good reason to do so has come to mind, and is thus allowed in the name of generality. Calling divmod() with no divisors should still return a tuple (of one element). Code that calls divmod() with a varying number of divisors, and thus gets a return value with an "unknown" number of elements, would otherwise have to special case that case. Code that *knows* it is calling divmod() with no divisors is considered to be too silly to warrant a special case. Processing the divisors in the other direction, i.e dividing with the first divisor first, instead of dividing with the last divisor first, has been considered. However, the result comes with the most significant part first and the least significant part last (think of the chained divmod as a way of splitting a number into "digits", with varying weights), and it is reasonable to specify the divisors (weights) in the same order as the result. The inverse operation: def inverse_divmod(seq, *factors): product = seq[0] for x,y in zip(factors, seq[1:]): product = product * x + y return product could also be useful. However, writing seconds = (((((w * 7) + d) * 24 + h) * 60 + m) * 60 + s) is less cumbersome both to write and to read than the chained divmods. It is therefore deemed to be less important, and its introduction can be deferred to its own PEP. Also, such a function needs a good name, and the PEP author has not managed to come up with one yet. Calling divmod("spam") does not raise an error, despite strings supporting neither division nor modulo. However, unless we know the other object too, we can't determine whether divmod() would work or not, and thus it seems silly to forbid it. Backwards Compatibility Any module that replaces the divmod() function in the __builtin__ module, may cause other modules using the new syntax to break. It is expected that this is very uncommon. Code that expects a TypeError exception when calling divmod() with anything but two arguments will break. This is also expected to be very uncommon. No other issues regarding backwards compatibility are known. Reference Implementation Not finished yet, but it seems a rather straightforward new implementation of the function builtin_divmod() in Python/bltinmodule.c Copyright This document has been placed in the public domain. -- Thomas Bellman, Lysator Computer Club, Linköping University, Sweden "Adde parvum parvo magnus acervus erit" ! bellman @ lysator.liu.se (From The Mythical Man-Month) ! Make Love -- Nicht Wahr! From Jack.Jansen@oratrix.com Tue Dec 31 17:38:49 2002 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 31 Dec 2002 18:38:49 +0100 Subject: [Python-Dev] Python 2.3a1 release status In-Reply-To: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On dinsdag, dec 31, 2002, at 15:02 Europe/Amsterdam, Guido van Rossum wrote: > Jack Jansen: I don't expect that you're quite ready with the Mac tree. > Or are you? Let me know what you want me to say about the Mac in the > announcement and on the web. Sorry for not telling you earlier, but all the Mac stuff (the unix-Mac-stuff, that is) has been ready since last night. MacPython-OS9 will take another week (or more), so I'd like to get control of the release branch again after the unix/win release has been done. The main bit of Mac news is that all the old MacPython tools and modules now work (if you do a framework build). I plan to do a binary release later (not for a1, probably starting with the beta's) and that will be called MacPython-OSX or MacPython for short (MacPython-OS9 will be the new name for the old MacPython). -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From just@letterror.com Tue Dec 31 17:46:28 2002 From: just@letterror.com (Just van Rossum) Date: Tue, 31 Dec 2002 18:46:28 +0100 Subject: [Python-Dev] sstruct In-Reply-To: <3E11C609.5000206@v.loewis.de> Message-ID: "Martin v. L=F6wis" wrote: > Lalo Martins wrote: > > While idly browsing around, I came across Just's module sstruct.=20 > > Any good reason this is not in the library? =20 >=20 > I can think of two reasons: > 1. Nobody has proposed to include it, yet. Including its author ;-) > 2. It is not obvious that it is useful. >=20 > On the latter point: What application is finding it so useful that > they incorporate it in their distribution? It's incredibly useful in my TrueType/OpenType conversion package (http://fonttools.sf.net/) where I convert lots of arbitrary and largely unrelated struct-like data formats to and from Python objects. Having the field names in the format string and automatically using these for dict keys or object attribute names was a major win in maintainability and readability of the code, and it being only a thin layer on top of the struct module makes it pretty fast. That said, I was never convinced of a usability so wide as to propose it for the std lib, and the amount of feedback I got over the years (it's pretty old!) didn't change that in a positive direction. The version on sf might be slightly newer than the one on starship (I don't even know!), but even on sf it hasn't changed since I checked it in there, three years ago: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/fonttools/fonttools/Lib/ sstruct.py Just From skip@pobox.com Tue Dec 31 18:11:16 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 31 Dec 2002 12:11:16 -0600 Subject: [Python-Dev] Patch 640843 - split OPT into two parts In-Reply-To: <200212311726.gBVHQQs24193@odiug.zope.com> References: <15889.52760.92971.951623@montanaro.dyndns.org> <200212311726.gBVHQQs24193@odiug.zope.com> Message-ID: <15889.56772.30091.47277@montanaro.dyndns.org> >> I'd like to check them in, but should probably wait until Guido >> bundles 2.3a1. ... Guido> Are you sure you want to risk this right before the release? If Guido> it breaks some platform's build, we're hosed. That's why I said I should probably wait until 2.3a1 is out the door. ;-) Skip From guido@python.org Tue Dec 31 18:22:19 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 13:22:19 -0500 Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors In-Reply-To: Your message of "Tue, 31 Dec 2002 18:38:37 +0100." <200212311738.SAA02244@proton.lysator.liu.se> References: <200212311738.SAA02244@proton.lysator.liu.se> Message-ID: <200212311822.gBVIMJM24712@odiug.zope.com> On the one hand I find this a nice backwards-compatible extension of divmod(). On the other hand I've coded this use case with separate divmod() calls and never had any trouble getting it right. So I think this doesn't address a real need, and it conflicts with the need to avoid unnecessary growth of the language. Explaining it (especially why it works backward) takes some effort, and I don't want to do that to all the book authors etc. --Guido van Rossum (home page: http://www.python.org/~guido/) From lalo@laranja.org Tue Dec 31 18:44:57 2002 From: lalo@laranja.org (Lalo Martins) Date: Tue, 31 Dec 2002 16:44:57 -0200 Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors In-Reply-To: <200212311738.SAA02244@proton.lysator.liu.se> References: <200212311738.SAA02244@proton.lysator.liu.se> Message-ID: <20021231184457.GE31681@laranja.org> I'm +0 on this (reasons below), but since Guido already said no: I find this function pretty useful, as it serves to collapse ugly code, and I hate ugly python code. However, the name sucks. With your changes, 'divmod' doesn't describe it anymore. So I suggest you come up with a better name and offer it as a contribution to, say, the math module. In that case I would prefer to have the reverse function too. (The fact that using it with 2 args is equal to calling divmod is at most a funny fact. As Guido says, growing the library = good; growing the language = questionable) []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From guido@python.org Tue Dec 31 18:39:17 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 13:39:17 -0500 Subject: [Python-Dev] test_coercion In-Reply-To: Your message of "Tue, 31 Dec 2002 12:32:18 EST." <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer> References: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer> Message-ID: <200212311839.gBVIdHm31228@odiug.zope.com> > Can anyone reproduce this test failure: > > C:\py23\Lib\test>python_d regrtest.py test_coercion > test_coercion > test test_coercion produced unexpected output: > ********************************************************************** > *** mismatch between line 523 of expected output and line 523 of actual output: > - [1] *= => [1] > + [1] *= ... exceptions.TypeError > *** mismatch between line 535 of expected output and line 535 of actual output: > - [1] *= => [1, 1] > + [1] *= ... exceptions.TypeError > *********************************************************************** > 1 test failed: > test_coercion > [12762 refs] It doesn't fail for me with current CVS on Linux and Windows 98. I tried the debug and the regular build. --Guido van Rossum (home page: http://www.python.org/~guido/) From lalo@laranja.org Tue Dec 31 18:31:17 2002 From: lalo@laranja.org (Lalo Martins) Date: Tue, 31 Dec 2002 16:31:17 -0200 Subject: [Python-Dev] sstruct In-Reply-To: References: <3E11C609.5000206@v.loewis.de> Message-ID: <20021231183117.GD31681@laranja.org> On Tue, Dec 31, 2002 at 06:46:28PM +0100, Just van Rossum wrote: > "Martin v. Löwis" wrote: > > 1. Nobody has proposed to include it, yet. > > Including its author ;-) That is why I posted to a forum where I knew said author would read me ;-) > > 2. It is not obvious that it is useful. > > > > On the latter point: What application is finding it so useful that > > they incorporate it in their distribution? Heck, I would have used it instead of struct in all places I ever used struct, if I knew about it before :-) (But the only one of these who is public is the Python client library of the PicoGUI toolkit. It has a heavy dependency on struct, which could be made much more readable with sstruct - actually this is why sstruct got my attention in the first place.) I guess it's essentialy Just's fault for not publicizing his cool stuff better... Perhaps it is not a good time for it to go into 2.3, but I think this is the kind of thing that belongs in the library. It does the same job as struct, but with a more readable (and arguably, pythonic) API. I can volunteer to write the docs and tests, if needed. []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From esr@thyrsus.com Tue Dec 31 18:31:02 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 13:31:02 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <15889.54671.243224.307879@grendel.zope.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> Message-ID: <20021231183102.GA23564@thyrsus.com> Fred L. Drake, Jr. : > That would be nice to have; I should have responded to that point > separately. Feel free to commit a minimal patch to fix that, and add > a regression test to Lib/test/test_cfgparser.py. On my to-do list. > I don't think it's clear that using __str__() for this is valuable. > Creating a potentially large string in memory may not be the best > approach; I actually like the write() method better; the caller can > decide to use a StringIO if that's what makes sense, or toss things to > a real file if needed. Guido has already made this point. As a matter of taste, I like the serialization operation to be separate from I/O. I think that's cleaner, and in this case the in-memory string is unlikely to get large. But I don't actually need this change, so I won't argue for it against opposition. > Perhaps I missed it; did you preserve comments as well? No. But comments are fair game to be discarded, since they're not part of the data content. In my use case they're not important. > I think the interface should be discussed; your implementation may be > sufficient, but after working with pyexpat as much as I have, I've > developed a strong dislike for attributes that have substantial side > effects. A fair point. I would be happy to change the interface away from a magic member to a real method. Likewise for the structured-value stuff. > If we're going to add a special syntax for multi-line values, we may > need to think about encoding special values, text encoding, and > Unicode. And that's never easy to get concensus on. ;-) Then we shouldn't try. Let's not let the ideal be the enemy of the good here; the multiline-literal syntax is othorgonal to how we handle string encoding, and internationalization can be layered on it later. > > 3. Currently ConfigParser cannot support structured values. Again, I > > need this capability (for association lists of coordinates and names, > > as it happens). The syntax I have implemented is a configurable list > > bracket character surrounding comma-separated lists. The alternative > > Fred suggests is, I think, extremely ugly. If anyone has a more > > natural suggestion than my proposal, I'm willing to hear it. > > I think "suggests" is too strong a word; I was only citing precedence, > not advocating that approach. I think it's pretty ugly as well. I'm taking suggestions on how to do it right. I don't think there's any obviously better way than what I've implemented, except maybe for making the trigger a real method rather than a magic member. > > 4. I *do* in fact want to support `surgical' alteration of .INI files. > > I have a use case for this in a configuration editor I'm writing for > > FreeCiv. More generally, when writing code to support invertible > > representations, I think it is good citizenship to perturb the > > original data as little as possible. Thus I regard the fact that > > I think there's a distinction between reading a configuration and > editing it: The original code never wrote a file back out. That was > something you added in revision 1.19. Right, because I needed it. > Seriously, I have no objection to supporting surgical editing. I do > think that separate classes should be used for read-only uses and > read-write uses; a read-only ConfigParser-thingie should remain very > lightweight, and surgical editing just isn't that. I had this conversation with Guido a couple years back. I too would have been happier with two classes, one read-only and one that supports writing back out. Here is whatt Guido said and I replied: --------------------------------------------------------------------------- Guido van Rossum : > One possible suggestion (just for your contemplation): if you're only > adding methods, would it make sense to make a subclass, so that from > the class one uses it is clear whether one intends to modify the > options or just to read them? That would enable future optimizations. Yes, and I even know what I'd call the class: ConfigEditor. Unfortunately this plan it trips on a historical fact; one (1) of the write methods, add_section(), already existed in ConfigParser. Moving it to a ConfigEditor subclass would break backward compatibility. And I think it would be just too ugly to move all but one of the write methods into ConfigEditor and leave add_section() in the base class. If not for this problem I would be more than happy to do as you suggest. --------------------------------------------------------------------------- Guido dropped the thread, which I took to mean that he saw the force of the objection. But I'd *still* like to do as he suggested. If the consensus is that its OK to deprecate add_section() and write() in the base class, I'll write ConfigEditor and move all my surgery stuff over there in a heartbeat. -- Eric S. Raymond From DavidA@ActiveState.com Tue Dec 31 18:38:38 2002 From: DavidA@ActiveState.com (David Ascher) Date: Tue, 31 Dec 2002 10:38:38 -0800 Subject: [Python-Dev] bugs in stdlib In-Reply-To: <20021231162406.GA22850@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <20021231160347.GA22796@thyrsus.com> <20021231162406.GA22850@thyrsus.com> Message-ID: <3E11E42E.8080506@ActiveState.com> Eric S. Raymond wrote: > Eric S. Raymond : > >>Guido van Rossum : >> >>>If you saw that and chose to ignore my advice, I would've appreciated >>>a reply at least. >> >>I did not. I would have heeded that advice. > > > On reflection, more explanation seems called for... > > I've been flooded in a particularly repellent way recently. Somebody > is using my From address for spam mail about quack remedies for > prostate trouble, somebody so illiterate that the Subject line read > "urnination flow". We've been seeing misspellings as deliberate ways to get around word-list filters. It's probably not illiteracy as much as wickedness. From guido@python.org Tue Dec 31 18:44:18 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 13:44:18 -0500 Subject: [Python-Dev] Creating 2.3a1 branch Message-ID: <200212311844.gBVIiID05084@odiug.zope.com> I'm creating the 2.3a1 release branch. Checkins from now on won't make it into 2.3a1 unless you mail me ASAP for special dispensation. (Tim is tracking down a leak that may get this.) The branch point is tagged with r23a1-fork; the branch itself will be called r23a1-branch. Thanks everybody! Release later this afternoon. --Guido van Rossum (home page: http://www.python.org/~guido/) From nas@python.ca Tue Dec 31 19:04:23 2002 From: nas@python.ca (Neil Schemenauer) Date: Tue, 31 Dec 2002 11:04:23 -0800 Subject: [Python-Dev] test_coercion In-Reply-To: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer> References: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer> Message-ID: <20021231190423.GA749@glacier.arctrix.com> Raymond Hettinger wrote: > Can anyone reproduce this test failure: Not I. That behavior change was introduced recently by me. The main change was abstract.c 2.113. Are you sure python_d is up-to-date? Neil From fdrake@acm.org Tue Dec 31 19:01:54 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 31 Dec 2002 14:01:54 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <20021231183102.GA23564@thyrsus.com> References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> Message-ID: <15889.59810.427468.150879@grendel.zope.com> Eric S. Raymond writes: > No. But comments are fair game to be discarded, since they're not part > of the data content. In my use case they're not important. I don't think they're fair game if we want a surgical implementation. Neither are blank lines. > > If we're going to add a special syntax for multi-line values, we may > > need to think about encoding special values, text encoding, and > > Unicode. And that's never easy to get concensus on. ;-) > > Then we shouldn't try. Let's not let the ideal be the enemy of the good > here; the multiline-literal syntax is othorgonal to how we handle > string encoding, and internationalization can be layered on it later. The question of exactly where to draw the line has bit us so many times with ConfigParser that I'm quite wary. I'll leave it for others to determine whether I'm too much so. > > I think "suggests" is too strong a word; I was only citing precedence, > > not advocating that approach. I think it's pretty ugly as well. > > I'm taking suggestions on how to do it right. I don't think there's any > obviously better way than what I've implemented, except maybe for making the > trigger a real method rather than a magic member. I'm not suggesting that there's a better way, only a precedent. > I had this conversation with Guido a couple years back. I too would have > been happier with two classes, one read-only and one that supports writing > back out. Here is whatt Guido said and I replied: [long quotation elided] > Guido dropped the thread, which I took to mean that he saw the force > of the objection. But I'd *still* like to do as he suggested. > > If the consensus is that its OK to deprecate add_section() and write() > in the base class, I'll write ConfigEditor and move all my surgery > stuff over there in a heartbeat. I think add_section() is useful for modifying an in-memory configuration (perhaps loaded from multiple sources, other than INI files). The implementation would be different than for your proposed ConfigEditor, but that's OK. write() may also be useful for other reasons, and probably isn't worth deprecating. Maintenance of those methods is not a good reason to create a separate ConfigEditor class. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From ark@research.att.com Tue Dec 31 19:21:45 2002 From: ark@research.att.com (Andrew Koenig) Date: 31 Dec 2002 14:21:45 -0500 Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors In-Reply-To: <20021231184457.GE31681@laranja.org> References: <200212311738.SAA02244@proton.lysator.liu.se> <20021231184457.GE31681@laranja.org> Message-ID: Lalo> I'm +0 on this (reasons below), but since Guido already said no: Lalo> I find this function pretty useful, as it serves to collapse Lalo> ugly code, and I hate ugly python code. However, the name Lalo> sucks. With your changes, 'divmod' doesn't describe it anymore. Lalo> So I suggest you come up with a better name and offer it as a Lalo> contribution to, say, the math module. In that case I would Lalo> prefer to have the reverse function too. If it helps, APL uses the names "base" and "represent" for these two functions. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From esr@thyrsus.com Tue Dec 31 19:23:32 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 14:23:32 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <15889.59810.427468.150879@grendel.zope.com> References: <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> <15889.59810.427468.150879@grendel.zope.com> Message-ID: <20021231192332.GA23881@thyrsus.com> Fred L. Drake, Jr. : > I don't think they're fair game if we want a surgical implementation. > Neither are blank lines. That depends on whether the surgery is at the data level or the representation level. The former covers my use case. -- Eric S. Raymond From fdrake@acm.org Tue Dec 31 19:35:33 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 31 Dec 2002 14:35:33 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <20021231192332.GA23881@thyrsus.com> References: <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> <15889.59810.427468.150879@grendel.zope.com> <20021231192332.GA23881@thyrsus.com> Message-ID: <15889.61829.425684.299710@grendel.zope.com> >Fred L. Drake, Jr. : > > I don't think they're fair game if we want a surgical implementation. > > Neither are blank lines. Eric S. Raymond writes: > > That depends on whether the surgery is at the data level or the > representation level. The former covers my use case. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim@multitalents.net Tue Dec 31 19:35:48 2002 From: tim@multitalents.net (Tim Rice) Date: Tue, 31 Dec 2002 11:35:48 -0800 (PST) Subject: [Python-Dev] Patch 640843 - split OPT into two parts In-Reply-To: <15889.52760.92971.951623@montanaro.dyndns.org> References: <15889.52760.92971.951623@montanaro.dyndns.org> Message-ID: On Tue, 31 Dec 2002, Skip Montanaro wrote: > There are some structural changes in configure.in that make it worth a > careful look-see. In particular, I hope I haven't bolluxed up Martin's SCO > changes from September (I can't test on that platform). I wouldn't worry about breaking SCO. It's been broken since the decision to define _XOPEN_SOURCE. -- Tim Rice Multitalents (707) 887-1469 tim@multitalents.net From fdrake@acm.org Tue Dec 31 19:37:34 2002 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 31 Dec 2002 14:37:34 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <20021231192332.GA23881@thyrsus.com> References: <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> <15889.59810.427468.150879@grendel.zope.com> <20021231192332.GA23881@thyrsus.com> Message-ID: <15889.61950.238264.559700@grendel.zope.com> (Sorry for the empty reply; fingers were *way* too quick!) I wrote: > I don't think they're fair game if we want a surgical implementation. > Neither are blank lines. Eric S. Raymond writes: > That depends on whether the surgery is at the data level or the > representation level. The former covers my use case. At the abstract data level, the order is fair game as well. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From martin@v.loewis.de Tue Dec 31 19:38:15 2002 From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 31 Dec 2002 20:38:15 +0100 Subject: [Python-Dev] sstruct In-Reply-To: <20021231183117.GD31681@laranja.org> References: <3E11C609.5000206@v.loewis.de> <20021231183117.GD31681@laranja.org> Message-ID: <3E11F227.9060904@v.loewis.de> >>>On the latter point: What application is finding it so useful that >>>they incorporate it in their distribution? > > > Heck, I would have used it instead of struct in all places I ever used > struct, if I knew about it before :-) Can you explain why? Isn't it much slower than struct? Regards, Martin From tim@zope.com Tue Dec 31 19:40:06 2002 From: tim@zope.com (Tim Peters) Date: Tue, 31 Dec 2002 14:40:06 -0500 Subject: [Python-Dev] New leak Message-ID: "A leak" in the datetime test turned out to be in the scaffolding that prints the current refcount. Here's a whittled test: import sys lastrc = 0 while True: thisrc = sys.gettotalrefcount() if 1: # change 1 to 0 and there's no leak x = '*' * 10 # This leaks; ditto tuple times int, or list times int print >> sys.stderr, 'total refs:', thisrc, 'delta:', thisrc - lastrc lastrc = thisrc Run that in a debug build, and it leaks one reference per loop trip. From esr@thyrsus.com Tue Dec 31 20:10:38 2002 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 31 Dec 2002 15:10:38 -0500 Subject: [Python-Dev] Re: ConfigParser changes In-Reply-To: <15889.61950.238264.559700@grendel.zope.com> References: <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> <15889.59810.427468.150879@grendel.zope.com> <20021231192332.GA23881@thyrsus.com> <15889.61950.238264.559700@grendel.zope.com> Message-ID: <20021231201038.GA24063@thyrsus.com> Fred L. Drake, Jr. : > Eric S. Raymond writes: > > That depends on whether the surgery is at the data level or the > > representation level. The former covers my use case. > > At the abstract data level, the order is fair game as well. My use case calls that assummption into some question. There are attribute groups within sections where I think the meaning is order-dependent. -- Eric S. Raymond From lists@morpheus.demon.co.uk Tue Dec 31 20:24:45 2002 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Tue, 31 Dec 2002 20:24:45 +0000 Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C References: <3E106BBB.4020909@interet.com> Message-ID: Just van Rossum writes: > This is all only half true. sys.path_importer_cache is exposed so > you can *clear* it in case you install a path hook that should take > over sys.path items that may already have a handler. To only > partially clear it is advanced usage. By no means you're supposed to > *add* to sys.path_importer_cache explicitly, as someone after you > may clear it again, and then you're hosed if there's no proper path > hook. Whoo. I was thinking about how I'd simulate the (now unavailable) feature of adding importers directly to sys.path. It took me virtually no time at all to come up with importer = MyImporter(...) imp_id = str(id(importer)) # A cookie sys.path.append(imp_id) sys.path_importer_cache[imp_id] = importer The only way I could come up with of doing the same thing without messing with sys.path_importer_cache was by implementing my own equivalent - basically id_cache = {} def id_factory(id): try: return id_cache[id] except KeyError: raise ImportError sys.path_hooks.append(id_factory) importer = MyImporter(...) imp_id = str(id(importer)) # A cookie sys.path.append(imp_id) id_cache[imp_id] = importer I can't think of any reason why this is *intrinsically* better than the first way other than because someone may clear the cache on you. And it's certainly not as intuitive. > I also disagree that adding an importer to sys.path_importer_cache > would always be *easier* than doing it the right way by adding a > hook to sys.path_hooks. Hmm. See above. OK, so that's generic code, and specific cases may always be easier to code "the right way". > If would be somewhat easier for "cookie" path items, but I think in > 99% of these cases a path-less importer on sys.meta_path would work > just as well. But sys.meta_path happens *before* sys.path. It's entirely likely that a cookie item might want to run *after* sys.path. For that we need "stage 2", where the real sys.path importer is a proper item on sys.meta_path. At the moment, whether sys.meta_path should run before or after sys.path is a judgement call - which usage is more common? In the longer term, the normal insert(0) vs append() approach gives both options. Disclaimer: As usual, this is all theory. Real use cases take precedence. Paul. PS Jim's comments have now raised two (as far as I can recall) issues with the proposed hook mechanism which should be documented in the "Open Issues" section of the PEP: 1. sys.meta_path allows hooks to be added *before* sys.path processing, but there is no equivalent facility for hooks *after* sys.path (response: either wait for "phase 2" or use a cookie at the end of sys.path). 2. There is no easy way of "stacking" hooks (response: there isn't one now, either, so that's not a critical issue). -- This signature intentionally left blank From lalo@laranja.org Tue Dec 31 21:11:48 2002 From: lalo@laranja.org (Lalo Martins) Date: Tue, 31 Dec 2002 19:11:48 -0200 Subject: [Python-Dev] sstruct In-Reply-To: <3E11F227.9060904@v.loewis.de> References: <3E11C609.5000206@v.loewis.de> <20021231183117.GD31681@laranja.org> <3E11F227.9060904@v.loewis.de> Message-ID: <20021231211148.GL31681@laranja.org> On Tue, Dec 31, 2002 at 08:38:15PM +0100, "Martin v. Löwis" wrote: > >Heck, I would have used it instead of struct in all places I ever used > >struct, if I knew about it before :-) > > Can you explain why? Isn't it much slower than struct? I haven't yet tested its speed, but Just claims it's just a bit slower. However, in a choice between clarity and speed I almost always, scratch that, always choose clarity. Then *if* I have performance problems I go back and speed up where I can, as long as I don't compromise clarity beyond some reasonable arbitrary limit. The struct module, however, falls a lot beyond this limit, and if I had a choice I wouldn't even have used it at all. (Short answer: Mu.) []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From tim.one@comcast.net Tue Dec 31 22:14:17 2002 From: tim.one@comcast.net (Tim Peters) Date: Tue, 31 Dec 2002 17:14:17 -0500 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Modules _randommodule.c,1.1,1.2 In-Reply-To: Message-ID: [nnorwitz@users.sourceforge.net] > Modified Files: > _randommodule.c > Log Message: > Try to get compilation working for cygwin > > Index: _randommodule.c And then, for example, before: > ! PyObject_GenericGetAttr, /*tp_getattro*/ and after: > ! 0, /*tp_getattro*/ followed by: > + Random_Type.tp_getattro = PyObject_GenericGetAttr; > + Random_Type.tp_alloc = PyType_GenericAlloc; > + Random_Type.tp_free = _PyObject_Del; in the module init function. Please don't make this kind of change -- it makes the code so much harder to follow. If this is needed for Cygwin, then, e.g., do #define DEFERRED(x) 0 /* some boxes can't resolve addresses at compile-time */ and make the "after" line DEFERRED(PyObject_GenericGetAttr), /*tp_getattro*/ IOW, the type slots should be readable on their own, as a static unit. From neal@metaslash.com Tue Dec 31 22:28:23 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 31 Dec 2002 17:28:23 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Modules _randommodule.c,1.1,1.2 In-Reply-To: References: Message-ID: <20021231222823.GB20404@epoch.metaslash.com> On Tue, Dec 31, 2002 at 05:14:17PM -0500, Tim Peters wrote: > > And then, for example, before: > > > ! PyObject_GenericGetAttr, /*tp_getattro*/ > > and after: > > > ! 0, /*tp_getattro*/ > > followed by: > > > + Random_Type.tp_getattro = PyObject_GenericGetAttr; > > + Random_Type.tp_alloc = PyType_GenericAlloc; > > + Random_Type.tp_free = _PyObject_Del; > > in the module init function. > > Please don't make this kind of change -- it makes the code so much harder to > follow. If this is needed for Cygwin, then, e.g., do > > #define DEFERRED(x) 0 /* some boxes can't resolve addresses at compile-time > */ > > and make the "after" line > > DEFERRED(PyObject_GenericGetAttr), /*tp_getattro*/ > > IOW, the type slots should be readable on their own, as a static unit. I agree, this makes sense. The only problem is that the functions/methods will have to be specified twice. It would be nice if we only had to specify each once. Any ideas? There are several places this has already been done (and at least one more coming). Do you want a global Py_DEFERRED() or some other name? Neal From guido@python.org Tue Dec 31 23:35:07 2002 From: guido@python.org (Guido van Rossum) Date: Tue, 31 Dec 2002 18:35:07 -0500 Subject: [Python-Dev] Python 2.3a1 released Message-ID: <200212312335.gBVNZ7p15291@pcp02138704pcs.reston01.va.comcast.net> I'm happy to announce the successful release of Python 2.3a1 tonight, on Dec 31 (in *some* timezones :-). Go pick it up from its home page: http://www.python.org/2.3/ This is an alpha release: if you have Python code that's important (to you), please test it thoroughly with this release, but don't use the release for production runs. Please report any problems to the SourceForge bug tracker: http://sourceforge.net/bugs/?group_id=5470 What's new? The language isn't changing much this time. Instead, we have lots of new or improved standard library modules: bsddb, bz2, datetime, heapq, logging, optparse, ossaudiodev, random (Mersenne Twister algorithm), sets, socket (added timeouts), textwrap, and zipimport. That's right, you can now import modules from zip files. More elaborate news is on the website: http://www.python.org/2.3/highlights.html (brief) http://www.python.org/doc/2.3a1/whatsnew/ (extensive) http://www.python.org/2.3/NEWS.html (exhausting) --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Dec 30 16:40:58 2002 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 30 Dec 2002 11:40:58 -0500 Subject: [Python-Dev] Extension modules, Threading, and the GIL References: <20021230145912.GA8481@panix.com> Gcc: nnfolder+archive:sent Message-ID: The following message is a courtesy copy of an article that has been posted to comp.lang.python as well. David Abrahams writes: > Aahz writes: >> I think this thread might be better handled on c.l.py, at least >> until it's understood well enough to be clear whether something does >> need to change in Python. > > I'm pretty certain I understand what's going on. If it would be > better for you to take this to c.l.py, I'm happy to do so, but AFAICT > there _is_ a Python core issue here: there's no way to find out > whether you've already got the GIL**, so if you _might_ have to acquire > it, you must always acquire it. > > **If I'm wrong about that, please let me know. It isn't obvious from > the documentation. This is the continuation of a thread originated on python-dev; I was asked to re-raise it here on python-list until the issue is better understood. The original posting was here: http://aspn.activestate.com/ASPN/Mail/Message/python-dev/1482879 The he essence of the problem is that there's no way to write code that uses the Python 'C' API and which has no knowledge of whether it is running on Python's main thread when it is entered. The two respondents were left with some questions; you can read those, and my responses, in the thread at the bottom of the page referenced above. Thanks for your attention, Dave -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution