From skip@mojam.com (Skip Montanaro) Sat Sep 4 22:26:37 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Sat, 4 Sep 1999 16:26:37 -0500 Subject: [Python-Dev] Catching "return" and "return expr" at compile time Message-ID: <199909042126.QAA03363@dolphin.mojam.com> --Multipart_Sat_Sep__4_16:26:37_1999-1 Content-Type: text/plain; charset=US-ASCII Attached is a context diff against the latest version of Python/compile.c that checks at compile time for functions that both return expressions or execute return statements with no expression (or equivalently, fall off the end of the function). I figured I'd post it here to get a little friendly feedback and bug discovery before shooting it off to c.l.py. I modified compile.c instead of some preexisting PyLint script because I don't know what's popular out there. On the other hand, I'm sure most people who would be interested in this sort of thing have access to the C source... The basic idea is that each straight line chunk of code is terminated one of four ways: 1. return with no expression 2. return an expression 3. raise an exception 4. fall off the end of the chunk Falling off the end of the function is obviously treated like return with no expression. (This is, after all, what motivated me to do this. ;-) This information is recorded in a new bit vector added to the struct compiling object that's carried around during the compilation. Compound statements simply aggregate the bit vectors for their various clauses in ways appropriate to their semantics. At the end of a function's compilation, the set of return bits computed up to that point tells you whether or not to spit out a warning. Note that it does nothing to recognize constant expressions. The following function will generate a warning: def f(): i = 0 while 1: i = i + 1 if i > 10: return i even though the only way to return from the function is the return statement. To get the above to shut up the compiler you'd have to do something like class CantGetHere: pass def f(): i = 0 while 1: i = i + 1 if i > 10: return i raise CantGetHere Raise statements are treated as a valid way to "return" from a function. Registering them as separate styles of returns serves effectively to turn off the "no return" bit for a block of code. Raise is compatible with either form of return, though they aren't compatible with each other. The code is run whenever a module is compiled. I didn't bother to add a new flag to the python interpreter to enable/disable warnings during compilation, though a -w switch for Python has been mentioned before. I ran the modified byte code compiler over a silly test module as well as executing ./python Lib/test/regrtest.py ./python Lib/compileall.py It uncovered some interesting programming practices and one item I think is an actual bug. In Lib/gzip.py, GzipFile._read returns EOFError at one point instead of raising it. At other points in the method it raises EOFError. There are no other return statements in the function. (I haven't taken the time/had the nerve to run it against my own Python code yet. ;-) I'm firmly of the opinion that more subtle bugs exist in the way people write functions that return and raise values than in the code that calls those functions, contrary to a few vocal folks on c.l.py who may believe otherwise. Enjoy, Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... --Multipart_Sat_Sep__4_16:26:37_1999-1 Content-Type: application/octet-stream Content-Disposition: attachment; filename="compile.diffs" Content-Transfer-Encoding: quoted-printable Index: compile.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /projects/cvsroot/python/dist/src/Python/compile.c,v retrieving revision 2.97 diff -c -r2.97 compile.c *** compile.c 1999/01/28 15:08:09 2.97 --- compile.c 1999/09/04 20:48:05 *************** *** 320,325 **** --- 320,330 ---- int c_stacklevel; /* Current stack level */ int c_maxstacklevel; /* Maximum stack level */ int c_firstlineno; + #define R_NORET 1 /* no return */ + #define R_EXPR 2 /* return expression */ + #define R_NONE 4 /* return */ + #define R_RAISE 8 /* raise */ + int c_rettype; /* how a function is exited */ PyObject *c_lnotab; /* Table mapping address to line number */ int c_last_addr, c_last_line, c_lnotab_next; #ifdef PRIVATE_NAME_MANGLING *************** *** 2053,2061 **** if (NCH(n) < 2) { com_addoparg(c, LOAD_CONST, com_addconst(c, Py_None)); com_push(c, 1); } ! else com_node(c, CHILD(n, 1)); com_addbyte(c, RETURN_VALUE); com_pop(c, 1); } --- 2058,2069 ---- if (NCH(n) < 2) { com_addoparg(c, LOAD_CONST, com_addconst(c, Py_None)); com_push(c, 1); + c->c_rettype =3D R_NONE; } ! else { com_node(c, CHILD(n, 1)); + c->c_rettype =3D R_EXPR; + } com_addbyte(c, RETURN_VALUE); com_pop(c, 1); } *************** *** 2078,2083 **** --- 2086,2092 ---- i =3D NCH(n)/2; com_addoparg(c, RAISE_VARARGS, i); com_pop(c, i); + c->c_rettype =3D R_RAISE; } = static void *************** *** 2299,2310 **** --- 2308,2342 ---- } = static void + com_set_rettype(c, in_rettype, stmt_rettype) + struct compiling *c; + int in_rettype; + int stmt_rettype; + { + /* merge the accumulated return type coming into the statement + with the return type generated by the statement */ + if (!(in_rettype & R_NORET)) + /* all branches before this statement led to a return + of some type, so the return possibilities of this + statement are irrelevant */ + c->c_rettype =3D in_rettype; + else + /* there was at least one path leading to this statement + that did not return - merge the return possibilities + coming in with the return possibilities from this + statement, which might include R_NORET */ + c->c_rettype =3D (in_rettype & ~R_NORET) | stmt_rettype; + } + = + static void com_if_stmt(c, n) struct compiling *c; node *n; { int i; int anchor =3D 0; + int in_rettype =3D c->c_rettype; + int if_rettype =3D 0; REQ(n, if_stmt); /*'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite] */ for (i =3D 0; i+3 < NCH(n); i+=3D4) { *************** *** 2318,2331 **** com_addfwref(c, JUMP_IF_FALSE, &a); com_addbyte(c, POP_TOP); com_pop(c, 1); com_node(c, CHILD(n, i+3)); com_addfwref(c, JUMP_FORWARD, &anchor); com_backpatch(c, a); /* We jump here with an extra entry which we now pop */ com_addbyte(c, POP_TOP); } ! if (i+2 < NCH(n)) com_node(c, CHILD(n, i+2)); if (anchor) com_backpatch(c, anchor); } --- 2350,2373 ---- com_addfwref(c, JUMP_IF_FALSE, &a); com_addbyte(c, POP_TOP); com_pop(c, 1); + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, i+3)); + if_rettype |=3D c->c_rettype; com_addfwref(c, JUMP_FORWARD, &anchor); com_backpatch(c, a); /* We jump here with an extra entry which we now pop */ com_addbyte(c, POP_TOP); } ! if (i+2 < NCH(n)) { ! c->c_rettype =3D R_NORET; com_node(c, CHILD(n, i+2)); + if_rettype |=3D c->c_rettype; + } + else + /* no else clause - have to consider that we might + get here without returning */ + if_rettype |=3D R_NORET; + com_set_rettype(c, in_rettype, if_rettype); if (anchor) com_backpatch(c, anchor); } *************** *** 2338,2343 **** --- 2380,2387 ---- int break_anchor =3D 0; int anchor =3D 0; int save_begin =3D c->c_begin; + int in_rettype =3D c->c_rettype; + int while_rettype =3D 0; REQ(n, while_stmt); /* 'while' test ':' suite ['else' ':' suite] */ com_addfwref(c, SETUP_LOOP, &break_anchor); block_push(c, SETUP_LOOP); *************** *** 2348,2354 **** --- 2392,2400 ---- com_addbyte(c, POP_TOP); com_pop(c, 1); c->c_loops++; + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 3)); + while_rettype =3D c->c_rettype; c->c_loops--; com_addoparg(c, JUMP_ABSOLUTE, c->c_begin); c->c_begin =3D save_begin; *************** *** 2357,2364 **** com_addbyte(c, POP_TOP); com_addbyte(c, POP_BLOCK); block_pop(c, SETUP_LOOP); ! if (NCH(n) > 4) com_node(c, CHILD(n, 6)); com_backpatch(c, break_anchor); } = --- 2403,2418 ---- com_addbyte(c, POP_TOP); com_addbyte(c, POP_BLOCK); block_pop(c, SETUP_LOOP); ! if (NCH(n) > 4) { ! c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 6)); + while_rettype |=3D c->c_rettype; + } + else + /* no else clause - have to consider that we might + never execute the loop body */ + while_rettype |=3D R_NORET; + com_set_rettype(c, in_rettype, while_rettype); com_backpatch(c, break_anchor); } = *************** *** 2371,2376 **** --- 2425,2432 ---- int break_anchor =3D 0; int anchor =3D 0; int save_begin =3D c->c_begin; + int in_rettype =3D c->c_rettype; + int for_rettype =3D 0; REQ(n, for_stmt); /* 'for' exprlist 'in' exprlist ':' suite ['else' ':' suite] */ com_addfwref(c, SETUP_LOOP, &break_anchor); *************** *** 2388,2394 **** --- 2444,2452 ---- com_push(c, 1); com_assign(c, CHILD(n, 1), OP_ASSIGN); c->c_loops++; + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 5)); + for_rettype =3D c->c_rettype; c->c_loops--; com_addoparg(c, JUMP_ABSOLUTE, c->c_begin); c->c_begin =3D save_begin; *************** *** 2396,2403 **** com_pop(c, 2); /* FOR_LOOP has popped these */ com_addbyte(c, POP_BLOCK); block_pop(c, SETUP_LOOP); ! if (NCH(n) > 8) com_node(c, CHILD(n, 8)); com_backpatch(c, break_anchor); } = --- 2454,2469 ---- com_pop(c, 2); /* FOR_LOOP has popped these */ com_addbyte(c, POP_BLOCK); block_pop(c, SETUP_LOOP); ! if (NCH(n) > 8) { ! c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 8)); + for_rettype |=3D c->c_rettype; + } + else + /* no else clause - have to consider that we might + never execute the loop body */ + for_rettype |=3D R_NORET; + com_set_rettype(c, in_rettype, for_rettype); com_backpatch(c, break_anchor); } = *************** *** 2477,2486 **** --- 2543,2557 ---- int else_anchor =3D 0; int i; node *ch; + int in_rettype =3D c->c_rettype; + int try_rettype =3D 0; + int tryclause_rettype =3D 0; = com_addfwref(c, SETUP_EXCEPT, &except_anchor); block_push(c, SETUP_EXCEPT); + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 2)); + tryclause_rettype =3D c->c_rettype; com_addbyte(c, POP_BLOCK); block_pop(c, SETUP_EXCEPT); com_addfwref(c, JUMP_FORWARD, &else_anchor); *************** *** 2517,2523 **** --- 2588,2596 ---- } com_addbyte(c, POP_TOP); com_pop(c, 1); + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, i+2)); + try_rettype |=3D c->c_rettype; com_addfwref(c, JUMP_FORWARD, &end_anchor); if (except_anchor) { com_backpatch(c, except_anchor); *************** *** 2533,2540 **** anything. */ com_addbyte(c, END_FINALLY); com_backpatch(c, else_anchor); ! if (i < NCH(n)) com_node(c, CHILD(n, i+2)); com_backpatch(c, end_anchor); } = --- 2606,2622 ---- anything. */ com_addbyte(c, END_FINALLY); com_backpatch(c, else_anchor); ! if (i < NCH(n)) { ! /* pick up with the return stuff where the try clause ended */ ! c->c_rettype =3D tryclause_rettype; com_node(c, CHILD(n, i+2)); + try_rettype |=3D c->c_rettype; + } + else + /* no else clause - just merge try clause return info + with return info from all the except clauses */ + try_rettype |=3D tryclause_rettype; + com_set_rettype(c, in_rettype, try_rettype); com_backpatch(c, end_anchor); } = *************** *** 2545,2554 **** --- 2627,2640 ---- { int finally_anchor =3D 0; node *ch; + int in_rettype =3D c->c_rettype; + int try_rettype =3D 0; = com_addfwref(c, SETUP_FINALLY, &finally_anchor); block_push(c, SETUP_FINALLY); + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 2)); + try_rettype =3D c->c_rettype; com_addbyte(c, POP_BLOCK); block_pop(c, SETUP_FINALLY); block_push(c, END_FINALLY); *************** *** 2561,2570 **** --- 2647,2661 ---- com_backpatch(c, finally_anchor); ch =3D CHILD(n, NCH(n)-1); com_addoparg(c, SET_LINENO, ch->n_lineno); + /* shouldn't start with R_NORET, because it's essentially + the tail end of the try clause */ + c->c_rettype =3D 0; com_node(c, ch); + try_rettype |=3D c->c_rettype; com_addbyte(c, END_FINALLY); block_pop(c, END_FINALLY); com_pop(c, 3); /* Matches the com_push above */ + com_set_rettype(c, in_rettype, try_rettype); } = static void *************** *** 2886,2891 **** --- 2977,2984 ---- "'break' outside loop"); } com_addbyte(c, BREAK_LOOP); + /* else clause wouldn't get executed, so R_NORET is possible */ + c->c_rettype =3D R_NORET; break; case continue_stmt: com_continue_stmt(c, n); *************** *** 3156,3167 **** --- 3249,3267 ---- if (TYPE(ch) =3D=3D varargslist) com_arglist(c, ch); c->c_infunction =3D 1; + c->c_rettype =3D R_NORET; com_node(c, CHILD(n, 4)); c->c_infunction =3D 0; com_addoparg(c, LOAD_CONST, com_addconst(c, Py_None)); com_push(c, 1); com_addbyte(c, RETURN_VALUE); com_pop(c, 1); + com_set_rettype(c, c->c_rettype, R_NONE); + if ((c->c_rettype & R_NONE) && (c->c_rettype & R_EXPR)) { + PySys_WriteStderr("warning: File \"%s\", line %d: %s", + c->c_filename, c->c_lineno, c->c_name); + PySys_WriteStderr(" has \"return\" and \"return expr\"\n"); + } } = static void --Multipart_Sat_Sep__4_16:26:37_1999-1-- From guido@CNRI.Reston.VA.US Tue Sep 7 16:20:47 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 07 Sep 1999 11:20:47 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Sat, 04 Sep 1999 16:26:37 CDT." <199909042126.QAA03363@dolphin.mojam.com> References: <199909042126.QAA03363@dolphin.mojam.com> Message-ID: <199909071521.LAA18017@eric.cnri.reston.va.us> This is a valuable service! Even though I'm sure that it will cause some pain for people who were used to this programming style... I'm not sure I like the fact that you can't turn it off -- traditionally, Python has had a "no warnings" policy. That has been diluted a bit (python -t prints warnings) but so far it has been the default. I'm wondering if we should introduce a general '-w' flag to turn on warnings like this (which would subsume -t)? Or perhaps there should be a -W flag ("no warnings") and warnings should be the default? There are also platform problems, e.g. on the Mac, stderr doesn't always exist, and on Windows, it doesn't exist if pythonw.exe is used... --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@cnri.reston.va.us Tue Sep 7 17:13:27 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 12:13:27 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909071521.LAA18017@eric.cnri.reston.va.us>; from Guido van Rossum on Tue, Sep 07, 1999 at 11:20:47AM -0400 References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> Message-ID: <19990907121326.B729@cnri.reston.va.us> On 07 September 1999, Guido van Rossum said: > This is a valuable service! Even though I'm sure that it will cause > some pain for people who were used to this programming style... > > I'm not sure I like the fact that you can't turn it off -- > traditionally, Python has had a "no warnings" policy. That has been > diluted a bit (python -t prints warnings) but so far it has been the > default. > > I'm wondering if we should introduce a general '-w' flag to turn on > warnings like this (which would subsume -t)? Or perhaps there should > be a -W flag ("no warnings") and warnings should be the default? Yes yes yes! While adding "-w" is a long way from having a comprehensive set of compile-time warnings in place, it at least means that someone is *thinking* about it. Also, I would suggest that there should be some standard internal mechanism for reporting errors rather than just calling 'PySys_WriteStderr()'. Something as simple as this would probably do the trick: void Py_Warning (char *filename, int line, char *msg) { if (on_a_platform_where_stderr_means_something) PySys_WriteStderr ("warning: file %s, line %d: %s", filename, line, msg); else do_whatever_it_takes_for_this_platform(); } Well, you get the idea. I make no claim that this is an appropriate name for this function, nor do I have anything to say about where it should live. It should also be smart about unknown filename or line number (eg. skip filename if filename == NULL, skip line number if line == -1). Oh, and of course we'll need to add a global variable $^W so that programmers can turn run-time warnings on and off as needed. *duck* Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time warning system in addition to the compile-time warnings of -t and Skip's patch.) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gstein@lyra.org Tue Sep 7 18:04:14 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 07 Sep 1999 10:04:14 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> Message-ID: <37D5458E.384B77ED@lyra.org> Greg Ward wrote: > > On 07 September 1999, Guido van Rossum said: > > This is a valuable service! Even though I'm sure that it will cause > > some pain for people who were used to this programming style... > > > > I'm not sure I like the fact that you can't turn it off -- > > traditionally, Python has had a "no warnings" policy. That has been > > diluted a bit (python -t prints warnings) but so far it has been the > > default. > > > > I'm wondering if we should introduce a general '-w' flag to turn on > > warnings like this (which would subsume -t)? Or perhaps there should > > be a -W flag ("no warnings") and warnings should be the default? > > Yes yes yes! While adding "-w" is a long way from having a > comprehensive set of compile-time warnings in place, it at least means > that someone is *thinking* about it. I would recommend no warnings by default, and -Wfeature to add specific types of warnings. This pattern follows that used by gcc (well, gcc has *some* warnings by default). Rather than invent a new set of switches, I'd rather steal an existing semantic :-) > Also, I would suggest that there should be some standard internal > mechanism for reporting errors rather than just calling > 'PySys_WriteStderr()'. Something as simple as this would probably do Why? Why not just use PySys_WriteStdErr() as your requested function? It can easily determine "oops. no stderr. let's do something else." >... > Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time > warning system in addition to the compile-time warnings of -t and Skip's > patch.) There is no such thing as run-time vs compile-time warnings. You always have a compiler at run-time, and it can be used at any time. Therefore, you just have "(compilation) warnings" (I could imagine that people will come up with other kinds of warnings once the feature is provided). I would suggest sys.warnings be a dictionary. python -Wbad-return -Wlines-per-func=50 >>> print sys.warnings {'bad-return': None, 'lines-per-func': '50'} Cheers, -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Tue Sep 7 18:33:53 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 13:33:53 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <37D5458E.384B77ED@lyra.org>; from Greg Stein on Tue, Sep 07, 1999 at 10:04:14AM -0700 References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> Message-ID: <19990907133353.A1235@cnri.reston.va.us> On 07 September 1999, Greg Stein said: > > Also, I would suggest that there should be some standard internal > > mechanism for reporting errors rather than just calling > > 'PySys_WriteStderr()'. Something as simple as this would probably do > > Why? Why not just use PySys_WriteStdErr() as your requested function? It > can easily determine "oops. no stderr. let's do something else." Hmm, that makes sense for the "what's the local equivalent of stderr?" determination. Probably that actually belongs in mywrite() (the static function in Python/sysmodule.c that PySys_WriteStdout() and PySys_WriteStderr() both invoke), so that the same thing can be done for stdout and stderr. However, I still think a separate function for printing source-code-based warnings is a good idea. This is mainly so that the association from (filename, line_number, message) to "warning: file %s, line %s: %s" % (filename, line_number, message) is done in *one* place, rather than everywhere a warning message is generated. For instance, platforms that don't have stderr, but instead pop up a window with all your compile-time warnings nicely formatted, could take advantage of knowing the filename and line number separately to nicely format those warnings. (Of course, this argues *against* putting the "what's the local equivalent of stderr?" determination in the low-level mywrite() function... arg...) > > Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time > > warning system in addition to the compile-time warnings of -t and Skip's > > patch.) > > There is no such thing as run-time vs compile-time warnings. You always > have a compiler at run-time, and it can be used at any time. Therefore, > you just have "(compilation) warnings" (I could imagine that people will > come up with other kinds of warnings once the feature is provided). Well, currently that's true, since currently Python's only warning is the tab warning from -t -- clearly a compile-time warning. (Is this true? I'm no expert on the internals, but I've certainly not seen any other warnings from Python, and I've included plenty of bugs in my code -- umm, just seeing if it would catch them, yeah that's it...) However, one could certainly envision a world where Python issues runtime warnings. If my time machine were working, I'd zip back and suggest to Guido that mistakes with the % operator should issue warnings rather than raising exceptions. (Ignore the language philosophy issue and presume this would be worthwhile.) There are probably other situations where, ignoring past history and language philosophy, it would make sense to issue a warning and march on ahead rather than blowing up immediately. Sometimes Python has a bit of an itchy trigger finger for that ol' TypeError... Anyways, the focus should probably be on compile-time warnings: I can't think of any major runtime errors offhand that Python currently does *nothing* about, so there's not a great need to go scattering the code with runtime warnings. But they *are* a theoretical possibility, and there *is* a difference with compile-time warnings. > I would suggest sys.warnings be a dictionary. > > python -Wbad-return -Wlines-per-func=50 > > >>> print sys.warnings > {'bad-return': None, 'lines-per-func': '50'} Makes sense -- true to warn (possibly giving some extra meaning to "truth", as in this example), and false to not warn. Or maybe None to not warn, not-None to warn. Of course, if there are only compile-time warnings, then modifying sys.warnings will only affect future imports, execs, evals, etc. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skip@mojam.com (Skip Montanaro) Tue Sep 7 18:41:27 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 7 Sep 1999 12:41:27 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <19990907133353.A1235@cnri.reston.va.us> References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <14293.19862.328236.265600@dolphin.mojam.com> [ ...Lot's of stuff about details of warning implementation snipped... ] That's why I only provided the code to check for inconsistent use of returns, not the flag to turn it on and off! In a message I accidentally sent only to Guido and myself on the subject, I outlined my take on things, which really does about exhaust my knowledge/interest on how/when to enable warnings: Guido> I'm not sure I like the fact that you can't turn it off -- Guido> traditionally, Python has had a "no warnings" policy. That has Guido> been diluted a bit (python -t prints warnings) but so far it has Guido> been the default. The only reason for not being able to turn it off was that would require introducing some sort of -w flag, which wasn't the point of the exercise. We can have the -w/-t/-W discussion now. I haven't any particular opinion on the best way to do it, although I would much prefer it be a run-time as opposed to compile-time option. One other issue might be whether or not to ignore an existing .pyc file and always recompile .py's if warnings are enabled. Of course, we're still all adults here (I think), so perhaps it's sufficient to remind people in the docs to delete the desired .pyc files before running with warnings enabled. Guido> I'm wondering if we should introduce a general '-w' flag to turn Guido> on warnings like this (which would subsume -t)? Or perhaps there Guido> should be a -W flag ("no warnings") and warnings should be the Guido> default? -w sounds fine to me. Guido> There are also platform problems, e.g. on the Mac, stderr doesn't Guido> always exist, and on Windows, it doesn't exist if pythonw.exe is Guido> used... Perhaps on those platforms a file could be opened in a standard location to catch stderr (I presume you can detect the lack of stderr at run-time?). While that would force some (more) Unix conventions on programmers on those platforms, it would also provide more cross-platform uniformity. Skip From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Tue Sep 7 19:07:01 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Tue, 7 Sep 1999 14:07:01 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <14293.21573.647149.941018@anthem.cnri.reston.va.us> >>>>> "GW" == Greg Ward writes: GW> However, one could certainly envision a world where Python GW> issues runtime warnings. If my time machine were working, I'd GW> zip back and suggest to Guido that mistakes with the % GW> operator should issue warnings rather than raising exceptions. GW> (Ignore the language philosophy issue and presume this would GW> be worthwhile.) Moderately off-topic, but since you brought it up, here's what I use in Mailman (since site-admins can make mistakes editing their templates, which contains %(keys)s... we'd like to make Mailman more robust so it doesn't totally crap out when that happens). We (hopefully) always interpolate with a SafeDict instead of a raw Python dictionary. -Barry class SafeDict(UserDict): """Dictionary which returns a default value for unknown keys. This is used in maketext so that editing templates is a bit more robust. """ def __init__(self, d): # optional initial dictionary is a Python 1.5.2-ism. Do it this way # for portability UserDict.__init__(self) self.update(d) def __getitem__(self, key): try: return self.data[key] except KeyError: if type(key) == StringType: return '%('+key+')s' else: return '' % `key` From tim_one@email.msn.com Tue Sep 7 19:26:30 1999 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 7 Sep 1999 14:26:30 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909071521.LAA18017@eric.cnri.reston.va.us> Message-ID: <002401bef95e$81963720$3f2d153f@tim> [Guido] > ... > I'm wondering if we should introduce a general '-w' flag to turn on > warnings like this (which would subsume -t)? Or perhaps there should > be a -W flag ("no warnings") and warnings should be the default? The latter, if for no other reason than that new users should get bludgeoned into good practice from their first day. If something's serious enough to trigger a warning, and you insist on doing it anyway, then you should at least know enough about Python to be able to find the -W switch . Note that in response to 1,379 distinct complaints about insane Perl semantics, TomC's stock answer is that every serious Perl programmer runs with -w and "use strict". He's right! Every serious Perl programmer does. Perl picked the wrong default, letting naive programmers hang themselves 1,379 distinct ways by default. Besides, warning by default will enhance your enviable reputation as a ruthless dictator opposed to freedom and creativity . > There are also platform problems, e.g. on the Mac, stderr doesn't > always exist, and on Windows, it doesn't exist if pythonw.exe is > used... But this is already a problem for, e.g., reporting fatal syntax errors, yes? That is, -w/-W isn't creating a new problem here, it's making the lack of a solution to an old problem more evident. all's-for-the-best-in-this-best-of-all-possible-worlds-ly y'rs - tim From gward@cnri.reston.va.us Tue Sep 7 19:47:40 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 14:47:40 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <002401bef95e$81963720$3f2d153f@tim>; from Tim Peters on Tue, Sep 07, 1999 at 02:26:30PM -0400 References: <199909071521.LAA18017@eric.cnri.reston.va.us> <002401bef95e$81963720$3f2d153f@tim> Message-ID: <19990907144739.B1235@cnri.reston.va.us> On 07 September 1999, Tim Peters said: > The latter, if for no other reason than that new users should get bludgeoned > into good practice from their first day. If something's serious enough to > trigger a warning, and you insist on doing it anyway, then you should at > least know enough about Python to be able to find the -W switch . > > Note that in response to 1,379 distinct complaints about insane Perl > semantics, TomC's stock answer is that every serious Perl programmer runs > with -w and "use strict". He's right! Every serious Perl programmer does. > Perl picked the wrong default, letting naive programmers hang themselves > 1,379 distinct ways by default. I agree, but I'm only willing to do so publicly because Tim has. So does the Perl documentation (ie. Tom C., I assume); from "man perl": DIAGNOSTICS The -w switch produces some lovely diagnostics. [...] Did we mention that you should definitely consider using the -w switch? BUGS The -w switch is not mandatory. D'you think that's a hint? Obviously, there *must* be a way to turn off warnings, so we can continue to run our crufty, bug-ridden old code without too many problems. Greg S.'s suggestion for being able to customize *which* warnings are printed is also important. Much hair was pulled when Perl 5.004 was released with a whole bunch of new warning messages -- lots of people had to go back and "fix" working code, or remove the -w switch from production scripts to clean up the mess on their stderr, etc. I suspect most of those people (myself included) were enlightened by the new warnings, but annoyed by having to go and fix what wasn't necessarily broken. A lot of people now recommend using -w only when developing, and removing it for production use, simply because of the risk of new warning messages when you upgrade Perl. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mhammond@skippinet.com.au Wed Sep 8 00:01:25 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 8 Sep 1999 09:01:25 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14293.19862.328236.265600@dolphin.mojam.com> Message-ID: <000e01bef984$e975e6d0$0801a8c0@bobcat> > Perhaps on those platforms a file could be opened in a > standard location to > catch stderr (I presume you can detect the lack of stderr at > run-time?). Not really - there are 2 scenarios here. pythonw.exe, for example, always has a valid stdout handle - it just goes nowhere. When Python is embedded in certain COM servers (such as ASP), the stdout handle is invalid - operations on it will fail (perversely, this also means a single "print" statement in your Python code can raise an exception and make your code fail - and seeing as print statements are the debugging state of the art at the moment, this is less than ideal - but I digress) So Im not sure we can check this reasonably at runtime - invalid handles are easy, but valid handles that go nowhere useful (as in pythonw.exe, and therefore the majority of cases we care about) is obviously difficult. OTOH, pythonw.exe doesnt print tracebacks either. Although not ideal, people arent loudly complaining about this - they know to develop and debug using python.exe. As the warnings we are discussing are compile time warnings, we could simply document that they should run "compileall" over their scripts to generate the warnings before attempting to embed it in some sort of wierd system. On my third hand, I would _really_ like to see this in a lint tool rather than in the core. I realize there is no such tool at the moment, but IMO is where we should be heading. Skip's return statement warnings are fine and a nice addition, but in my experience account for a trivial number of my errors. Stuff like warning about a variable name used only once, for example, will probably never get into core Python but in my opinion is far more valuable. So adding this "-w" switch is fine, but still doesnt give us the framework we need to actually create a truly useful package of warnings for the Python developer. [And I am slowly and painfully starting work in this - a lint tool based on the Python parser module. Dont hold your breath though :-] Mark. From guido@CNRI.Reston.VA.US Wed Sep 8 00:05:19 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 07 Sep 1999 19:05:19 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Wed, 08 Sep 1999 09:01:25 +1000." <000e01bef984$e975e6d0$0801a8c0@bobcat> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <199909072305.TAA20414@eric.cnri.reston.va.us> > > Perhaps on those platforms a file could be opened in a > > standard location to > > catch stderr (I presume you can detect the lack of stderr at > > run-time?). > > Not really - there are 2 scenarios here. pythonw.exe, for example, always > has a valid stdout handle - it just goes nowhere. When Python is embedded > in certain COM servers (such as ASP), the stdout handle is invalid - > operations on it will fail (perversely, this also means a single "print" > statement in your Python code can raise an exception and make your code > fail - and seeing as print statements are the debugging state of the art at > the moment, this is less than ideal - but I digress) > > So Im not sure we can check this reasonably at runtime - invalid handles > are easy, but valid handles that go nowhere useful (as in pythonw.exe, and > therefore the majority of cases we care about) is obviously difficult. > > OTOH, pythonw.exe doesnt print tracebacks either. Although not ideal, > people arent loudly complaining about this - they know to develop and debug > using python.exe. As the warnings we are discussing are compile time > warnings, we could simply document that they should run "compileall" over > their scripts to generate the warnings before attempting to embed it in > some sort of wierd system. Hmm... Perhaps pythonw.exe could use freopen() to point stdout and stderr to a log file in a temp directory? The wizards will know where to look... > On my third hand, I would _really_ like to see this in a lint tool rather > than in the core. I realize there is no such tool at the moment, but IMO > is where we should be heading. Skip's return statement warnings are fine > and a nice addition, but in my experience account for a trivial number of > my errors. Stuff like warning about a variable name used only once, for > example, will probably never get into core Python but in my opinion is far > more valuable. So adding this "-w" switch is fine, but still doesnt give > us the framework we need to actually create a truly useful package of > warnings for the Python developer. > > [And I am slowly and painfully starting work in this - a lint tool based on > the Python parser module. Dont hold your breath though :-] Eventually, I also plan to have some kind of lint in IDLE. If the CP4E money comes, I'll start working on that for earnest... --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein@lyra.org Wed Sep 8 12:53:46 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 04:53:46 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <37D64E4A.4A8A206C@lyra.org> Greg Ward wrote: > ... > > I would suggest sys.warnings be a dictionary. > > > > python -Wbad-return -Wlines-per-func=50 > > > > >>> print sys.warnings > > {'bad-return': None, 'lines-per-func': '50'} > > Makes sense -- true to warn (possibly giving some extra meaning to > "truth", as in this example), and false to not warn. Or maybe None to > not warn, not-None to warn. Of course, if there are only compile-time > warnings, then modifying sys.warnings will only affect future imports, > execs, evals, etc. Actually, I had intended *presence* in the dictionary to mean "enabled." I don't think we'd want to pre-populate the dict with all possible flags ahead of time, then check for each of them on the command line. (startup time!) However, if we "simply" parsed the command line, extracted all -W options and dropped them into the dict, then we're set. Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim@digicool.com Wed Sep 8 13:41:18 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 08 Sep 1999 08:41:18 -0400 Subject: [Python-Dev] More Python command-line features Message-ID: <37D6596E.A6B80783@digicool.com> While we're talking about Python command-line features, I have a couple of requests that would make life alot easier for us. 1. I'd like the options given before the script name to be accessable to Python scripts. For example, in: python -O foo.py bar spam I'd like to have a sys variable that told me that the arguments ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] (aka sys.argv). This is needed if I want to fork/exec (or span or whatever) Python with the same option. 2. I'd like to be able to supply environment variables to Python on the command line, as in: python -O -e PYTHONHOME=/Zope foo.py bar spam This would be very helpful in environment-variable-challenged environments like windows 9x, and might be handy elswhere as well. If people agree that these would be good ideas, I'd ve happy to supply a patch. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward@cnri.reston.va.us Wed Sep 8 13:48:49 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 8 Sep 1999 08:48:49 -0400 Subject: [Python-Dev] More Python command-line features In-Reply-To: <37D6596E.A6B80783@digicool.com>; from Jim Fulton on Wed, Sep 08, 1999 at 08:41:18AM -0400 References: <37D6596E.A6B80783@digicool.com> Message-ID: <19990908084848.A18815@cnri.reston.va.us> On 08 September 1999, Jim Fulton said: > > While we're talking about Python command-line features, I have a couple > of requests that would make life alot easier for us. > > 1. I'd like the options given before the script name to > be accessable to Python scripts. For example, in: > > python -O foo.py bar spam > > I'd like to have a sys variable that told me that the arguments > ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] > (aka sys.argv). This is needed if I want to fork/exec (or span or > whatever) Python with the same option. Regarding -O, it would also be nice to have a higher-level way to find out what optimization level the current interpreter is running under. Currently, as I understand it, there's no way to predict whether py_compile will generate .pyc or .pyo files, which is a minor annoyance in the Distutils installation code. However, if Jim's mythical sys variable is guaranteed to canonicalize Python's command-line options so that something like "-O" in sys.python_argv would always answer this question, I'd be satisfied. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <37D6596E.A6B80783@digicool.com> <19990908084848.A18815@cnri.reston.va.us> Message-ID: <14294.26144.574453.888184@weyr.cnri.reston.va.us> Greg Ward writes: > Regarding -O, it would also be nice to have a higher-level way to find > out what optimization level the current interpreter is running under. > Currently, as I understand it, there's no way to predict whether I agree. In fact, I'd even be willing to add support to compile the parse-trees produced by the parser module with or without optimization (pick your favorite level); that would be easy with the current implementation. I didn't add this earlier because Guido objected, saying that the internal optimization flag could change. That would require that the parser module implementation change accordingly. I don't think that would be a huge problem, other than for my having to send Guido a pre-forma gripe that he'd given me more work to do two days before a release. ;-) So, should .compile() accept an optional optimization level, with the default being to use the "current" setting? -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From skip@mojam.com (Skip Montanaro) Wed Sep 8 15:54:20 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 8 Sep 1999 09:54:20 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000e01bef984$e975e6d0$0801a8c0@bobcat> References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <14294.30407.419817.523625@dolphin.mojam.com> Mark> On my third hand, I would _really_ like to see this in a lint tool Mark> rather than in the core. I realize there is no such tool at the Mark> moment, but IMO is where we should be heading. Skip's return Mark> statement warnings are fine and a nice addition, but in my Mark> experience account for a trivial number of my errors. Stuff like Mark> warning about a variable name used only once, for example, will Mark> probably never get into core Python but in my opinion is far more Mark> valuable. So adding this "-w" switch is fine, but still doesnt Mark> give us the framework we need to actually create a truly useful Mark> package of warnings for the Python developer. I'm not sure the stuff I wrote belongs in the core either, certainly not in C code. As I mentioned when I posted it though, I wasn't sure where a PyLint type program already existed that I could simply graft onto. I've fiddled around enough with the compile.c code in the past couple of years that I understand it fairly well already. I do have some Python code that does peephole optimization on Python bytecode. I could have put it in there (it already divides functions into basic blocks), but again, not many people have it laying about to play with. Can we start/settle on a Python-based source code framework for this sort of thing? Ideally, I'd like to see a framework that brings the parser module's output up to a level where mere mortals like me can reason about Python code. Skip From skip@mojam.com (Skip Montanaro) Wed Sep 8 15:58:13 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 8 Sep 1999 09:58:13 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909072305.TAA20414@eric.cnri.reston.va.us> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> <199909072305.TAA20414@eric.cnri.reston.va.us> Message-ID: <14294.30895.37706.568973@dolphin.mojam.com> Guido> Eventually, I also plan to have some kind of lint in IDLE. If Guido> the CP4E money comes, I'll start working on that for earnest... Speaking of which, just where *is* IDLE? I get the Python source via CVS, but I'll be damned if I have anything called idle.py or IDLE.py or even anything that matches "*idle*" or "*IDLE*" glob patterns. I just executed "cvs update -A ." from the top of my tree and checked again. Still nothing. Is it a separate module from the main Python source? Thx, Skip From Fred L. Drake, Jr." References: <000e01bef984$e975e6d0$0801a8c0@bobcat> <199909072305.TAA20414@eric.cnri.reston.va.us> <14294.30895.37706.568973@dolphin.mojam.com> Message-ID: <14294.31497.168088.467694@weyr.cnri.reston.va.us> Skip Montanaro writes: > Speaking of which, just where *is* IDLE? I get the Python source via CVS, > but I'll be damned if I have anything called idle.py or IDLE.py or even Look in Tools/idle/ -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From Fred L. Drake, Jr." References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> <14294.30407.419817.523625@dolphin.mojam.com> Message-ID: <14294.31767.411418.442754@weyr.cnri.reston.va.us> Skip Montanaro writes: > thing? Ideally, I'd like to see a framework that brings the parser module's > output up to a level where mere mortals like me can reason about Python This was exactly what I wanted to prevent when I created the parser module! ;-) I think a wrapper that simplifies the parse tree wouldn't be too hard to do; you simply have to be sure that the simplified version can be re-elaborated to pass back to the byte-code compiler via parser.sequence2ast().compile(). Otherwise you can't modify the tree without loosing line number information, which would be nice to keep around! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal@lemburg.com Wed Sep 8 16:08:02 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 08 Sep 1999 17:08:02 +0200 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909071521.LAA18017@eric.cnri.reston.va.us> <002401bef95e$81963720$3f2d153f@tim> <19990907144739.B1235@cnri.reston.va.us> Message-ID: <37D67BD2.72A43DF5@lemburg.com> Greg Ward wrote: > > On 07 September 1999, Tim Peters said: > > The latter, if for no other reason than that new users should get bludgeoned > > into good practice from their first day. If something's serious enough to > > trigger a warning, and you insist on doing it anyway, then you should at > > least know enough about Python to be able to find the -W switch . > > > > Note that in response to 1,379 distinct complaints about insane Perl > > semantics, TomC's stock answer is that every serious Perl programmer runs > > with -w and "use strict". He's right! Every serious Perl programmer does. > > Perl picked the wrong default, letting naive programmers hang themselves > > 1,379 distinct ways by default. > > I agree, but I'm only willing to do so publicly because Tim has. So > does the Perl documentation (ie. Tom C., I assume); from "man perl": > > DIAGNOSTICS > The -w switch produces some lovely diagnostics. > [...] > Did we mention that you should definitely consider using the > -w switch? > > BUGS > The -w switch is not mandatory. > > D'you think that's a hint? > > Obviously, there *must* be a way to turn off warnings, so we can > continue to run our crufty, bug-ridden old code without too many > problems. > > Greg S.'s suggestion for being able to customize *which* warnings are > printed is also important. Much hair was pulled when Perl 5.004 was > released with a whole bunch of new warning messages -- lots of people > had to go back and "fix" working code, or remove the -w switch from > production scripts to clean up the mess on their stderr, etc. I suspect > most of those people (myself included) were enlightened by the new > warnings, but annoyed by having to go and fix what wasn't necessarily > broken. A lot of people now recommend using -w only when developing, > and removing it for production use, simply because of the risk of new > warning messages when you upgrade Perl. I'd suggest to use the -W [=] kind of command line option interface for warnings and to also add an environment variable to customize the standard settings, e.g. PYTHONWARNINGS. About enabling warning per default: you should consider the fact that much code out there will probably produce such warnings, even if it is perfectly valid (e.g. consider Skip's example with while 1:...). Enabling it is definitely not a good idea for production code -- it is during the development step. Since production code is likely to run using -O, I suggest disabling warnings when -O is used and enabling them otherwise. Also, I'd like to second GregS' idea with the sys.warnings dict. Together with a generic -W = interface this would be great for adding customized warnings to Python scripts (i.e. not only the ones that the interpreter itself produces). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 123 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Wed Sep 8 15:57:59 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 08 Sep 1999 16:57:59 +0200 Subject: [Python-Dev] Accessing internal flag values References: <37D6596E.A6B80783@digicool.com> <19990908084848.A18815@cnri.reston.va.us> Message-ID: <37D67977.1882CFDC@lemburg.com> Greg Ward wrote: > > On 08 September 1999, Jim Fulton said: > > > > While we're talking about Python command-line features, I have a couple > > of requests that would make life alot easier for us. > > > > 1. I'd like the options given before the script name to > > be accessable to Python scripts. For example, in: > > > > python -O foo.py bar spam > > > > I'd like to have a sys variable that told me that the arguments > > ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] > > (aka sys.argv). This is needed if I want to fork/exec (or span or > > whatever) Python with the same option. > > Regarding -O, it would also be nice to have a higher-level way to find > out what optimization level the current interpreter is running under. > Currently, as I understand it, there's no way to predict whether > py_compile will generate .pyc or .pyo files, which is a minor annoyance > in the Distutils installation code. Check out mxTools (from my Python Pages). It has a function which lets you control the value of the optimization flag: """ The following functions are installed as add-ons to the builtin sys module. sys.verbosity([level]) If level is given, the value of the interpreter's verbosity flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to e.g. enable verborse lookup output to stderr for import statements even when the interpreter was not invoked with '-v' or '-vv' switch or to force verbosity to be switched off. sys.debugging([level]) If level is given, the value of the interpreter's debugging flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to check whether the interpreter was called with '-d' flag or not. Some extensions use this flag to enable/disable debugging log output (e.g. all the mx Extensions). sys.optimization([level]) If level is given, the value of the interpreter's optimization flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to e.g. compile Python scripts in optimized mode even though the interpreter was not started with -O. """ -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 123 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein@lyra.org Wed Sep 8 20:20:32 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 12:20:32 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> <14294.30407.419817.523625@dolphin.mojam.com> <14294.31767.411418.442754@weyr.cnri.reston.va.us> Message-ID: <37D6B700.42F07735@lyra.org> Fred L. Drake, Jr. wrote: > > Skip Montanaro writes: > > thing? Ideally, I'd like to see a framework that brings the parser module's > > output up to a level where mere mortals like me can reason about Python > > This was exactly what I wanted to prevent when I created the parser > module! ;-) > I think a wrapper that simplifies the parse tree wouldn't be too > hard to do; you simply have to be sure that the simplified version can > be re-elaborated to pass back to the byte-code compiler via > parser.sequence2ast().compile(). Otherwise you can't modify the > tree without loosing line number information, which would be nice to > keep around! This has already been done. Grab the Python2C distribution from http://www.mudlib.org/~rassilon/p2c/. There is a module named "transformer.py" which does just what you're thinking -- it converts Python's deeply-nested trees into something human-readable. Each of the resulting node types are doc'd at the top of the module. It is also over a couple years old, so it has had some decent debugging/stabilization. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond@skippinet.com.au Wed Sep 8 23:03:07 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 9 Sep 1999 08:03:07 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30407.419817.523625@dolphin.mojam.com> Message-ID: <002801befa45$efed8240$0801a8c0@bobcat> > Can we start/settle on a Python-based source code framework > for this sort of > thing? Ideally, I'd like to see a framework that brings the > parser module's > output up to a level where mere mortals like me can reason > about Python > code. Actually, I struggled with this a _lot_, then found that P2C has a module called "transform" which flattens the parse tree down to something I can understand (which is good :-) I could simply attach it, but it is grafted to P2C IIRC. If there is interest I will rip it out... Mark. From gstein@lyra.org Wed Sep 8 23:11:35 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 15:11:35 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <002801befa45$efed8240$0801a8c0@bobcat> Message-ID: <37D6DF17.6CA3AB94@lyra.org> Mark Hammond wrote: >... > Actually, I struggled with this a _lot_, then found that P2C has a module > called "transform" which flattens the parse tree down to something I can > understand (which is good :-) > > I could simply attach it, but it is grafted to P2C IIRC. If there is > interest I will rip it out... transformer.py operates quite independently. It is a relatively large module, though (33k), so I would recommend people just grab the P2C distribution from http://www.mudlib.org/~rassilon/p2c/. I have started to put together a page at http://www.lyra.org/greg/python/ that includes the various modules that I've "published". I'll get transformer over there in the next day or two. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond@skippinet.com.au Wed Sep 8 23:29:47 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 9 Sep 1999 08:29:47 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <37D6DF17.6CA3AB94@lyra.org> Message-ID: <002c01befa49$a9b74730$0801a8c0@bobcat> Oops - for some reason I thought we were on Python-help where Greg doesnt hang out... > transformer.py operates quite independently. It does - however, the only simple way to see what it does is to use P2C. All I had in mind was a simple "if __name__=='__main__': block to demonstrate its output. Mark. From tim_one@email.msn.com Fri Sep 10 07:16:20 1999 From: tim_one@email.msn.com (Tim Peters) Date: Fri, 10 Sep 1999 02:16:20 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <000001befb54$0095e240$a52d153f@tim> [Mark Hammond] > ... > On my third hand, I would _really_ like to see this in a lint tool > rather than in the core. I realize there is no such tool at the > moment, but IMO is where we should be heading. Following the lead taken by other modern languages, like javalint, c++lint, perllint, dylanlint and even vblint ? C lint was a hack needed due to the combination of bad language design choices and poor compilers, but C compilers are smarter now than lint ever was. Who still uses lint? It's dead, and it's not missed. > Skip's return statement warnings are fine and a nice addition, but in > my experience account for a trivial number of my errors. Stuff like > warning about a variable name used only once, for example, will probably > never get into core Python but in my opinion is far more valuable. The notion that a valuable idea will never get into the core is disturbing. I don't really care how it's implemented, but a *visibly* separate "checking" tool is bad UI, one that even the C community left behind with relief. > So adding this "-w" switch is fine, but still doesnt give us the framework > we need to actually create a truly useful package of warnings for the > Python developer. No, but adding the "-W" switch does give us the means by which (perhaps the illusion of) "a" smarter compiler can be invoked & controlled. > [And I am slowly and painfully starting work in this - a lint tool based > on the Python parser module. Dont hold your breath though :-] Aaron W has had a capable pylint tool for a couple years, & it remains virtually unknown; and, far as I can tell, Aaron reciprocated the lack of interest by dropping active development. So why was C lint successful in its day while every crack at pylint flops (btw, yours will too <0.5 wink>)? I think it's two sides of the same coin: C lint found dozens of deadly problems that infested almost all C code (remember the pre-prototype days?). Versions of pylint offer very little beyond pointing out unique vrbl names, perhaps indentation checking, and ...? I'm drawing a blank here. I suppose they should strive to give better msgs for runaway triple-quoted strings. What else? Skip's "return" checker, and far as I can tell then we're already at the point of diminishing returns. My claim is that pylints don't get used both because they're a separate step, and because the odds of them catching something interesting are comparatively tiny. Python simply doesn't have many errors that *can* be caught at compile-time. It's like me firing up the spell-checker at this point to verify "compile-time" -- the expected payoff is negative. There's little enough useful a pylint could do that a mod to add those few smarts to the core would be a fraction of the size & effort of yet another separate tool. Better, in the core, it would actually do people some good because it would actually get used. Which specific problems do you expect your lint tool to uncover? Perhaps there's a world of mechanically-checkable Python errors I haven't yet bumped into. you-windows-guys-write-strange-code-ly y'rs - tim From da@ski.org Fri Sep 10 07:28:07 1999 From: da@ski.org (David Ascher) Date: Thu, 9 Sep 1999 23:28:07 -0700 (Pacific Daylight Time) Subject: [Python-Dev] FYI: Techniques for scientific C++ Message-ID: In case you haven't seen enough C++ papers, you might find this paper interesting. It's a good way to learn more about templates without trying to actually use them, and it's a very good way to decide to stay away from high-power C++ techniques like expression templates. Yeeagh! (I especially like the program shown in section 1.11.1). --david Date: Thu, 26 Aug 1999 21:59:06 -0500 (EST) From: Todd Veldhuizen Subject: OON: Techniques for scientific C++ I've updated and expanded my slides on "Techniques for Scientific C++" into a document. It's available in HTML and Postscript from: HTML: http://extreme.indiana.edu/~tveldhui/papers/techniques/ PS: http://extreme.indiana.edu/~tveldhui/papers/techniques/techniques.ps Here's the table of contents. 1.1: About this document 1.2: C++ Compilers 1.2.1: Placating sub-standard compilers 1.2.2: The compiler landscape 1.2.3: C++-specific optimization 1.3: Compile times 1.3.1: Headers 1.3.2: Prelinking 1.3.3: The program database approach -- Visual Age C++ 1.3.4: Quadratic/Cubic template algorithms 1.4: Static Polymorphism 1.4.1: Are virtual functions evil? 1.4.2: Solution A: simple engines 1.4.3: Solution B: the Barton and Nackman Trick 1.5: Callback inlining techniques 1.5.1: Callbacks: the typical C++ approach 1.5.1.1: Expression templates 1.5.1.2: STL-style function objects 1.5.1.3: Pointer-to-function as a template parameter 1.6: Managing code bloat 1.6.1: Avoid kitchen-sink template parameters 1.6.2: Put function bodies outside template classes 1.6.3: Inlining levels 1.7: Containers 1.7.1: STL-style containers 1.7.2: Data/View containers 1.8: Aliasing and restrict 1.9: Traits 1.9.1: An example: average() 1.9.2: Type promotion example 1.10: Expression templates 1.10.1: Performance implications of pairwise evaluation 1.10.2: Recursive templates 1.10.3: Expression templates: building parse trees 1.10.4: A minimal implementation 1.10.5: Refinements 1.10.6: Pointers to more information 1.10.7: References 1.11: Template metaprograms 1.11.1: Template metaprograms: some history 1.11.2: The need for specialized algorithms 1.11.3: Using template metaprograms to specialize algorithms 1.12: Comma overloading 1.12.1: An example 1.13: Interfacing with Fortran codes 1.14: Some thoughts on performance tuning 1.14.1: General suggestions 1.14.2: Know what your compiler can do 1.14.3: Data structures and algorithms 1.14.4: Efficient use of the memory hierarchy Cheers, Todd - -- Todd Veldhuizen tveldhui@acm.org Indiana Univ. Comp. Sci. http://extreme.indiana.edu/~tveldhui/ - --------------------- Object Oriented Numerics List -------------------------- From mhammond@skippinet.com.au Fri Sep 10 08:36:39 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 10 Sep 1999 17:36:39 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000001befb54$0095e240$a52d153f@tim> Message-ID: <003a01befb5f$39f25400$0801a8c0@bobcat> [Tim laments the death of lint for C :-] > The notion that a valuable idea will never get into the core > is disturbing. Agreed. I based my assesment simply on my perception of what is likely to happen, not my opinion of what _should_ happen. I do agree that it is far far preferable for Python itself to be capable of issuing these warnings, and if Guido feels that is the best direction then it would be very cool. Only Guido can state if he would support such efforts, and probably should right about now (the funk soul brother - sorry - just got the Fat Boy Slim CD, and its going around in my head :-) > Aaron W has had a capable pylint tool for a couple years, & it remains > virtually unknown; and, far as I can tell, Aaron reciprocated > the lack of > interest by dropping active development. Which tends to be the biggest problem with it. A number of people have tried to use it, but often get stymied by the lack of 1.5.?isms - ie, "raise" (ie re-raise) and "assert". It bombs at these statements, and there is some real magic I didnt want to understand. Aaron agrees that a parser module based one would be better. But your original point still remains - I agree having Python do this is a better solution all round. > (remember the pre-prototype days?). Versions of pylint offer > very little > beyond pointing out unique vrbl names, perhaps indentation > checking, and > ...? I'm drawing a blank here. I suppose they should strive > to give better > msgs for runaway triple-quoted strings. What else? Skip's "return" > checker, and far as I can tell then we're already at the point of > diminishing returns. Agreed. However, all of these would be very valuable and account for the vast majority of my errors. > you-windows-guys-write-strange-code-ly y'rs - tim Only cos we use a strange OS Mark. From guido@CNRI.Reston.VA.US Fri Sep 10 15:05:23 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 10 Sep 1999 10:05:23 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Fri, 10 Sep 1999 17:36:39 +1000." <003a01befb5f$39f25400$0801a8c0@bobcat> References: <003a01befb5f$39f25400$0801a8c0@bobcat> Message-ID: <199909101405.KAA26303@eric.cnri.reston.va.us> > Agreed. I based my assesment simply on my perception of what is likely to > happen, not my opinion of what _should_ happen. I do agree that it is far > far preferable for Python itself to be capable of issuing these warnings, > and if Guido feels that is the best direction then it would be very cool. > Only Guido can state if he would support such efforts, and probably should > right about now (the funk soul brother - sorry - just got the Fat Boy Slim > CD, and its going around in my head :-) I agree it should happen, and Tim's argument about keeping lint and compiler together is a convincing one. What stands in the way? (a) There's so much else to do... (b) *Someone* needs to design a good framework for spitting out warnings, and for giving the programmer a way to control which warnings to ignore. I've seen plenty of good suggestions here; now somebody should simply go off and come up with a proposal too good to refuse. (c) I have a different agenda with CP4E -- I think it would be great if the editor could do syntax checking and beyond *while you type*, like the spell checker in MS Word. (I *like* Word's spell checker, even though I hate the style checker [too stupid], and I gladly put up with the spell checker's spurious complaints -- it's easy to turn off, easy to ignore, and it finds lots of good stuff.) Because the editor has to deal with incomplete and sometimes ungrammatical things, and because it has to work in real time (staying out of the way when you're frantically typing, catching up when your fingers take a rest), it needs a different kind of parser. But that's another project, and once the Python core has a warning framework in place, I'm sure we'll find more things that are worth warning about. I'm not always in agreement with Tim Peters when he says that Python is so dynamic that it's impossible to check for certain errors. It may be impossible to say *for sure* that something is an error, but there sure are lots of situations where you're doing something that's *likely* to be an error. E.g. if the compiler sees len(1), and there's no local or global variable named len, it *could* be the case that the user has set up a parallel universe where the len() built-in accepts an integer argument, but more likely it's an honest mistake, and I would like to get a warning about it. The hard part here is to issue this warning for len(x) where x is some variable or expression that is likely to have a non-sequence value (barring alternate universes); this might require global analysis that's hard or expensive enough that we can't put it in the core (yet). This may be seen as an argument for a separate lint... --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@mojam.com (Skip Montanaro) Fri Sep 10 15:16:52 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 10 Sep 1999 09:16:52 -0500 (CDT) Subject: [Python-Dev] Where does lint-like functionality belong? Message-ID: <14297.4820.321654.848624@dolphin.mojam.com> Okay, if Tim's assertions are correct (and I have no reason to suspect them), a separate pylint will be doomed, so the only reasonable thing to do is to place it in the core where it will be rarin' to go all the time. Perl's experience with -w seems to suggest that it's best to always enable whatever warnings you can as well. (More and more I see people using gcc's -Wall flag as well.) Now, my return consistency stuff was easy enough to write in C for two reasons. One, I'm fairly comfortable with the compile.c code. Two, adding my checks required no extra memory management overhead. Consider a few other checks you might conceivably add to the byte code compiler: * tab nanny stuff (already enabled with -t, right?) * variables set but not used * variables used before set If all of this sort of stuff is added to the compiler proper, I predict a couple major problems will surface: * The complexity of the code will increase significantly, making it harder to maintain and enhance * Fewer and fewer people will be able to approach the code, making it less likely that new checks are added * Future extensions like pluggable virtual machines will be harder to add because their byte code compilers will be harder to integrate into the core In addition, more global checks probably won't be possible (reasoning about code across module boundaries for instance) because the compiler's view of the world is fairly narrow. I think lint-like tools should be implemented in Python (possibly with the support of an extension module for performance-critical sections) which is then called from the compiler proper under appropriate circumstances (warnings enabled, all necessary modules importable, etc). I believe the code would be much more easily maintained and extended. You'd be able to swap in a new byte code compiler without risking the loss of your checking code. Skip From mal@lemburg.com Fri Sep 10 20:06:28 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 10 Sep 1999 21:06:28 +0200 Subject: [Python-Dev] Relative Package Imports Message-ID: <37D956B4.3E79C2C1@lemburg.com> This is a multi-part message in MIME format. --------------60F2EF1167E1920659766E3D Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi everybody, I've spent the last two hours trying to get relative package imports to work because I need them for my extension packages which will soon all move under a new top-level package name to overcome the conflicts with PIL and Zope. Here are the results... Demo Package Structure: (see the attached demopkg.zip) [a] [b] bc.py ab.py With the attached patch you can do the following: # Pretty useless... import a.b.__.ab # From inside bc.py: from __ import ab # At top-level (also useless, but shows how this situation is handled): import __.sys # __ is bound to None since we are at top-level; sys is still # being loaded though. Of course, common usage will be of the form: form __.__ import submodule_at_higher_level Please tell me what you think and give it a try. It's a first try and may have some design errors. Especially the way head and tail are treated in Python/import.c:import_module_ex may cause trouble -- I need help here. Note: The patch is against the CVS version. If you run Python in verbose mode, the patch will produce some verbose output of what it's doing. Enjoy, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 113 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ --------------60F2EF1167E1920659766E3D Content-Type: text/plain; charset=us-ascii; name="parentimport.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="parentimport.patch" --- /home/lemburg/orig/Python/Python/import.c Fri Apr 9 19:00:51 1999 +++ Python/import.c Fri Sep 10 20:51:02 1999 @@ -1572,10 +1572,14 @@ load_next(mod, altmod, p_name, buf, p_bu char *dot = strchr(name, '.'); int len; char *p; PyObject *result; + if (Py_VerboseFlag) + printf("# load_next: (1) name='%s', buf='%.*s'\n", + name,*p_buflen,buf); + if (dot == NULL) { *p_name = NULL; len = strlen(name); } else { @@ -1586,10 +1590,39 @@ load_next(mod, altmod, p_name, buf, p_bu PyErr_SetString(PyExc_ValueError, "Empty module name"); return NULL; } + /* Handle "__" indicator telling the import mechanism to + continue the search one level higher in the package + hierarchy */ + if (strncmp(name,"__",len) == 0) { + PyObject *modules = PyImport_GetModuleDict(); + + /* Strip the final dotted name from buf */ + dot = strrchr(buf, '.'); + if (dot == NULL) + *p_buflen = 0; + else + *p_buflen = dot - buf; + buf[*p_buflen] = '\0'; + + /* Fetch the parent module or revert to a top-level search */ + if (*p_buflen > 0) { + mod = PyDict_GetItemString(modules,buf); + if (mod == NULL) { + PyErr_SetString(PyExc_SystemError, + "Parent module missing"); + return NULL; + } + } + else + mod = Py_None; + Py_INCREF(mod); + return mod; + } + p = buf + *p_buflen; if (p != buf) *p++ = '.'; if (p+len-buf >= MAXPATHLEN) { PyErr_SetString(PyExc_ValueError, @@ -1597,10 +1630,14 @@ load_next(mod, altmod, p_name, buf, p_bu return NULL; } strncpy(p, name, len); p[len] = '\0'; *p_buflen = p+len-buf; + + if (Py_VerboseFlag) + printf("# load_next: (2) modname='%s', fullname=buf='%s'\n", + p,buf); result = import_submodule(mod, p, buf); if (result == Py_None && altmod != mod) { Py_DECREF(result); /* Here, altmod must be None and mod must not be None */ --------------60F2EF1167E1920659766E3D Content-Type: application/x-zip-compressed; name="demopkg.zip" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="demopkg.zip" UEsDBAoAAgAAAASgKiejNvGqNwAAADcAAAANABUAYS9fX2luaXRfXy5weVVUCQADJ0fZN01H 2TdVeAQA9AFkAApwcmludCAnSW5pdCBtb2R1bGUgJXMsIHBhdGg9JXMnICUgKF9fbmFtZV9f LF9fcGF0aF9fKQpQSwMECgACAAAALaAqJ+7ynLElAAAAJQAAAAcAFQBhL2FiLnB5VVQJAAN2 R9k3hEfZN1V4BAD0AWQACnByaW50ICdJbml0IG1vZHVsZSAlcycgJSAoX19uYW1lX18pClBL AwQKAAIAAAAHoConozbxqjcAAAA3AAAADwAVAGEvYi9fX2luaXRfXy5weVVUCQADLkfZN09H 2TdVeAQA9AFkAApwcmludCAnSW5pdCBtb2R1bGUgJXMsIHBhdGg9JXMnICUgKF9fbmFtZV9f LF9fcGF0aF9fKQpQSwMEFAACAAgAFqYqJzw5S0GaAAAAAgEAAAkAFQBhL2IvYmMucHlVVAkA A4tS2TeTUtk3VXgEAPQBZABljkEKgzAQRfc5xYcitpscoOABuumqpcsQddSAyUgSld6+amMR uprFf2/+F4M3LiK/ORNhuR57QhZyZDgr5bQlpS5CJOjh38a1MHZgH8HNLigldYmigF6vlDIX idmSn/8ihI7HvobjGZ2eCLqqKARETk82+Ysn94Q7e6t7jEG3hHnzS7qKxrNdoH3OoecZ1pl/ OWZCS/HYsUgfUEsBAhYDCgACAAAABKAqJ6M28ao3AAAANwAAAA0ADQAAAAAAAQAAAKSBAAAA AGEvX19pbml0X18ucHlVVAUAAydH2TdVeAAAUEsBAhYDCgACAAAALaAqJ+7ynLElAAAAJQAA AAcADQAAAAAAAQAAAKSBdwAAAGEvYWIucHlVVAUAA3ZH2TdVeAAAUEsBAhYDCgACAAAAB6Aq J6M28ao3AAAANwAAAA8ADQAAAAAAAQAAAKSB1gAAAGEvYi9fX2luaXRfXy5weVVUBQADLkfZ N1V4AABQSwECFgMUAAIACAAWpionPDlLQZoAAAACAQAACQANAAAAAAABAAAApIFPAQAAYS9i L2JjLnB5VVQFAAOLUtk3VXgAAFBLBQYAAAAABAAEABgBAAAlAgAAAAA= --------------60F2EF1167E1920659766E3D-- From guido@CNRI.Reston.VA.US Fri Sep 10 20:16:28 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 10 Sep 1999 15:16:28 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Fri, 10 Sep 1999 21:06:28 +0200." <37D956B4.3E79C2C1@lemburg.com> References: <37D956B4.3E79C2C1@lemburg.com> Message-ID: <199909101916.PAA27287@eric.cnri.reston.va.us> > I've spent the last two hours trying to get relative package > imports to work because I need them for my extension packages > which will soon all move under a new top-level package name > to overcome the conflicts with PIL and Zope. I'd much rather use absolute package names for anything that's not in the same directory as the current module. --Guido van Rossum (home page: http://www.python.org/~guido/) From da@ski.org Sat Sep 11 05:06:54 1999 From: da@ski.org (David Ascher) Date: Fri, 10 Sep 1999 21:06:54 -0700 (Pacific Daylight Time) Subject: [Python-Dev] jpythonc: unreachable Python becomes unreachable Java (fwd) Message-ID: Greg Ward notes on the JPython list: > Looks like I've found a definite bug in jpythonc -- unreachable Python > code compiles to unreachable Java code. Here's a simple example: > [..] > Yikes... does this mean that jpythonc will have to analyze Python code > for reachability? Is this even possible? Guess the answer for now is, > "Don't do that!". This is another sort of warning that a checker could do in the core which could catch some errors. Not sure that it makes sense to try and find all the cases that an arbitrary Java compiler could find, but presumably Greg's original code which led him to the jpythonc bug find had a plain-old python bug in it? Did Aaron's pylint find any nonreachable code? --david From mal@lemburg.com Sat Sep 11 11:29:48 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 11 Sep 1999 12:29:48 +0200 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> Message-ID: <37DA2F1C.7ED64960@lemburg.com> Guido van Rossum wrote: > > > I've spent the last two hours trying to get relative package > > imports to work because I need them for my extension packages > > which will soon all move under a new top-level package name > > to overcome the conflicts with PIL and Zope. > > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. Of course, you could do everything with absolute names, but then the package author would dictate the complete absolute path which is not always desirable since it can cause name collisions such as DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) and Mark's win32 stuff. As more and more packages appear, we run into this problem more and more often. Relative imports would greatly reduce the risk these collisions, because packages could be written self-contained meaning that they can reach their internal modules via relative paths only and don't have to know about the absolute paths by which they are reachable from other parts of the Python universe. I could then make all my packages self-contained and distribute them in two forms without having to change a single line for the added support: 1. under the new 'mx' package, e.g. mx.DateTime 2. for backward compatibility under 'DateTime' Another major advantage is that I could take any other self-contained package and install it under absolute paths of my choice, e.g. put Zope under org.zope.core, Python under org.python.core etc., without harming their functionality or having to dive deep into their import structures to fix them manually. To further enhance this mechanism I would like to have an alias mechanism in import, pickle et al. so that changes in the package structures become manageable without user intervention: pickles are a major problem whenever import structures change because they store absolute module names. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 111 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From tim_one@email.msn.com Sat Sep 11 21:29:58 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 11 Sep 1999 16:29:58 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DA2F1C.7ED64960@lemburg.com> Message-ID: <000901befc94$6a463d20$6aa2143f@tim> [Guido] > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. [M.-A. Lemburg] > Of course, you could do everything with absolute names, but then > the package author would dictate the complete absolute path which > is not always desirable since it can cause name collisions such > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > released) and Mark's win32 stuff. > > As more and more packages appear, we run into this problem more > and more often. I never understand package complaints. Maybe it's the imprecision of the language people use, or maybe because it's because people don't give fully fleshed-out examples. Whatever, in the end, I never have any idea what the complaint is, or in what way the solution is "solving" anything. In the above, "absolute" doesn't appear to mean "absolute" in any OS-sensible sense. So what does it really mean? Does it mean the same things to Guido and MAL? In MAL's hint of examples, I don't see any problem. If mxDateTime unpacks itself into a directory named DateTime, then *of course* it's going to collide with other packages that want to do likewise. Install it into mxDateTime instead, and take "absolute" to mean "any module that wants an mxDateTime service and does not itself live directly in mxDateTime/ must import the desired module via a path beginning 'mxDateTime.'", and everything looks straightforward to me (and that outcome makes me infer that this is thus probably what Guido has in mind too). Similarly for win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be good to have an explicit way to say "std library" -- "Lib." comes to mind). > I could then make all my packages self-contained and > distribute them in two forms without having to change > a single line for the added support: > > 1. under the new 'mx' package, e.g. mx.DateTime > 2. for backward compatibility under 'DateTime' Ah, so that's what this is about. I vote screw #2. Releasing it that way was a mistake. Better to live with the brief & finite pain of repairing it than complicating Python to cover up for it. > Another major advantage is that I could take any other > self-contained package and install it under absolute paths > of my choice, e.g. put Zope under org.zope.core, Python under > org.python.core etc., without harming their functionality > or having to dive deep into their import structures to fix > them manually. I view that not as an advantage but as harmful complication. Zope etc add great value to a Python installation, and when I write a killer app full of "import zope.this" and "import zope.that", I don't want the possibility that it's not going to work on my client's machine just because their sysadmin installed Zope into some silly site-specific path named after his soon-to-be ex-girlfriend . I don't want a way to work around him doing that, either: I don't want him to be able to screw me to begin with. > To further enhance this mechanism I would like to have an > alias mechanism in import, pickle et al. so that changes > in the package structures become manageable without user > intervention: pickles are a major problem whenever import > structures change because they store absolute module names. This is a different issue, and may have merits of its own. WRT the relative import scheme, its advantage seems to lie in providing a way to partially recover from the damage the new scheme causes <0.5 wink>. As is, the package name used by a release is part of its published interface. You can't change it without causing pain, any more than you can make incompatible changes to public class methods or input-output behavior. In return, package clients are uniform, simple and portable, making life easiest for the people who know least. The burden is on package authors to choose names wisely, and that's where the burden should be. if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs - tim From tim_one@email.msn.com Sun Sep 12 01:41:16 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 11 Sep 1999 20:41:16 -0400 Subject: [Python-Dev] FYI: Techniques for scientific C++ In-Reply-To: Message-ID: <000101befcb7$85be2b80$4c2d153f@tim> [David Ascher, concerning http://extreme.indiana.edu/~tveldhui/papers/techniques/] ] > In case you haven't seen enough C++ papers, you might find this paper > interesting. It's a good way to learn more about templates without trying > to actually use them, and it's a very good way to decide to stay away from > high-power C++ techniques like expression templates. Yeeagh! Didn't these people get to play with m4 in their youth? Ah, this *is* their youth . As for so much else, this is stuff they could have been doing with Lisp/Scheme compile-time macros 15 years ago, except the latter are much clearer, simpler and easier to use. It's not "a trick" in Scheme, it's a fundamental approach, and the full power of the language is available at every step. If this kind of thing still holds a perverse attraction, check out http://www.fftw.org/ These guys produce what are usually the fastest FFT algorithms in the world across platforms, via a dynamic programming approach that generates algorithm fragments and times them on the target platform, eventually "growing" superior platform-specific FFT code for each vector size. Until C++ templates can do file I/O at compile-time (to take advantage of timings from previous runs), they won't even get close . computation-at-compile-time-is-a-powerful-thing-but- then-so-is-a-skunk's-odor-ly y'rs - tim From gmcm@hypernet.com Sun Sep 12 03:41:44 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Sat, 11 Sep 1999 22:41:44 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DA2F1C.7ED64960@lemburg.com> Message-ID: <1275018545-12250772@hypernet.com> M.-A. Lemburg wrote: > Guido van Rossum wrote: > > > > I'd much rather use absolute package names for anything that's not > > in the same directory as the current module. > > Of course, you could do everything with absolute names, but then > the package author would dictate the complete absolute path which is > not always desirable since it can cause name collisions such as > DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) > and Mark's win32 stuff. I can see your point (although I also believe that authors - Guido excepted - should come up with collision-free names, probably in a Java-ish scheme). But I strongly believe that import.c should be left alone, maybe even to die. There are too many people doing import hooks to make fiddling with its behavior safe. I'm also a strong proponent of Greg's imputil.py scheme, which makes it a breeze to do import hooks. And my experience disproves the notion that the import mechanism needs to be in C. If you don't believe me, try the ZlibArchive stuff (which is cross platform) from my Win32 installer stuff. You can pack the standard library into one 475K file, and get a perceptible performance boost. OTOH, I could see doing a framework of packages, in which case relative imports might be handy. This seems to work fine: def relimp(nm): rpth = string.split(nm, '/') tpth = string.split(__name__, '.')[:-1] for node in rpth: if node == '..': del tpth[-1] else: tpth.append(node) #print `tpth` return __import__(string.join(tpth, '.')) b = relimp('../packageA2.b') > As more and more packages appear, we run into this problem more > and more often. Relative imports would greatly reduce the > risk these collisions, because packages could be written > self-contained meaning that they can reach their internal > modules via relative paths only and don't have to know about > the absolute paths by which they are reachable from other parts > of the Python universe. > > I could then make all my packages self-contained and > distribute them in two forms without having to change > a single line for the added support: > > 1. under the new 'mx' package, e.g. mx.DateTime > 2. for backward compatibility under 'DateTime' > > Another major advantage is that I could take any other > self-contained package and install it under absolute paths > of my choice, e.g. put Zope under org.zope.core, Python under > org.python.core etc., without harming their functionality > or having to dive deep into their import structures to fix > them manually. > > To further enhance this mechanism I would like to have an > alias mechanism in import, pickle et al. so that changes > in the package structures become manageable without user > intervention: pickles are a major problem whenever import > structures change because they store absolute module names. > > -- > Marc-Andre Lemburg > ______________________________________________________________________ > Y2000: 111 days left Business: > http://www.lemburg.com/ Python Pages: > http://www.lemburg.com/python/ > > > > _______________________________________________ > Python-Dev maillist - Python-Dev@python.org > http://www.python.org/mailman/listinfo/python-dev - Gordon From tim_one@email.msn.com Sun Sep 12 10:18:16 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sun, 12 Sep 1999 05:18:16 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909101405.KAA26303@eric.cnri.reston.va.us> Message-ID: <000001befcff$bf654e20$492d153f@tim> [MarkH] > ... > I based my assesment simply on my perception of what is likely to > happen, not my opinion of what _should_ happen. I based mine on what Guido was waiting for someone to say . We worry too much about disagreeing here; different opinions are great! Guido will squash the ones he can't stand anyway. [about Aaron's pylint's lack of 1.5.2 smarts] > ... > Aaron agrees that a parser module based one would be better. You can't beat a real parse, no argument there. Luckily, the compiler parses too. [Guido] > What stands in the way? > > (a) There's so much else to do... How did Perl manage to attract 150 people with nothing to do except hack on Perl internals? "Wow, that code's such a mess I bet even *I* could get something into it" <0.6 wink>. > (b) *Someone* needs to design a good framework for spitting out > warnings, and for giving the programmer a way to control which > warnings to ignore. I've seen plenty of good suggestions here; now > somebody should simply go off and come up with a proposal too good to > refuse. The response has been ... absent. Anyone doing this? I liked JimF's push to make cmd-line options available to Python programs too. Somehow they seem related to me. > (c) I have a different agenda with CP4E -- I think it would be great > if the editor could do syntax checking and beyond *while you type*, > like the spell checker in MS Word. (I *like* Word's spell checker, > even though I hate the style checker [too stupid], and I gladly put up > with the spell checker's spurious complaints -- it's easy to turn off, > easy to ignore, and it finds lots of good stuff.) > > Because the editor has to deal with incomplete and sometimes > ungrammatical things, and because it has to work in real time (staying > out of the way when you're frantically typing, catching up when your > fingers take a rest), it needs a different kind of parser. Different from what? Python's own parser for sure. IDLE has at least two distinct parsers of its own that have nothing in common with Python's parser and little in common with each other. Using the horrid tricks in PyParse.py, it may even be possible to write the kind of parser you need in Python and still have it run fast enough. For parsing-on-the-fly from random positions, take my word for it and Barry's as insurance : the single most frequent question you need to have a fast and reliable answer for is "is this character in a string?". Unfortunately, turns out that's the hardest question to answer too. The next one is "am I on a continuation line, and if so where's the start?". Given rapid & bulletproof ways to answer those, the rest is pretty easy. > But that's another project, and once the Python core has a warning > framework in place, I'm sure we'll find more things that are worth > warning about. That was frequently predicted for various pylint projects too . > I'm not always in agreement with Tim Peters when he says that Python > is so dynamic that it's impossible to check for certain errors. It > may be impossible to say *for sure* that something is an error, but > there sure are lots of situations where you're doing something that's > *likely* to be an error. We have no disagreement there. What a compiler does must match the advertised semantics of the language-- or its own advertised deviations from those --without compromise. A warning system has no such constraint; to the contrary, in the case of a semantic mess like Perl, most of its value is in pointing out *legal* constructs that are unlikely to work the way you intended. > E.g. if the compiler sees len(1), and there's no local or global > variable named len, it *could* be the case that the user has set up a > parallel universe where the len() built-in accepts an integer > argument, but more likely it's an honest mistake, and I would like to > get a warning about it. Me too. More: I'd also like to get a warning for *having* a local or global variable named len! Masking the builtin names is simply bad practice, and is also usually an honest mistake. BTW, I was surprised that the most frequent gotcha among new Python users at Dragon turned out to be exactly that: dropping a "len" or a "str" or whatever (I believe len, str and list were most common) into their previously working code-- because they just learned about that builtin --and getting runtime errors as a result. That is, they already had a local var of that name, and forgot. Then they were irked that Python didn't nag them from the start (with a msg they understood, of course). > The hard part here is to issue this warning for len(x) where x is some > variable or expression that is likely to have a non-sequence value > (barring alternate universes); this might require global analysis > that's hard or expensive enough that we can't put it in the core > (yet). This may be seen as an argument for a separate lint... Curiously, Haskell is statically type-safe but doesn't require declarations of any kind -- it does global analysis, and has a 100% reliable type inference engine (the language was, of course, designed to make this true). Yet I don't think I've ever seen a Haskell program on the web that didn't explicitly declare the type of every global anyway. I think this is natural, too: while it's a PITA to declare the type of every stinking local that lives for two lines and then vanishes, the types of non-local names aren't self-evident: type decls really help for them. So if anyone is thinking of doing the kind of global analysis Guido mentions here, and is capable of doing it , I'd much rather they put their effort into optional static type decls for Python2. Many of the same questions need to be answered either way (like "what's a type?", and "how do we spell a type?" -- the global analysis warnings won't do any good if you can't communicate the substance of an error ), and optional decls are likely to have bigger bang for the buck. [Skip Montanaro] > ... > Perl's experience with -w seems to suggest that it's best to always > enable whatever warnings you can as well. While that's my position, I don't want to oversell the Perl experience. That language allows so many goofy constructs, and does so many wild guesses at runtime, that Perl is flatly unusable without -w for non-trivial programs. Not true of Python, although the kinds of warnings people have suggested so far certainly do seem worthy of loud complaint by default. > (More and more I see people using gcc's -Wall flag as well.) If you have to write portable C++ code, and don't enable every warning you can get on every compiler you have, and don't also turn on "treat warnings as errors", non-portable code will sneak into the project rapidly. That's my experience, over & over. gcc catches stuff MS doesn't, and vice versa, and MetroWerks yet another blob, and platform-specific cruft *still* gets in. It's irksome. > Now, my return consistency stuff was easy enough to write in C for two > reasons. One, I'm fairly comfortable with the compile.c code. I don't anticipate dozens of people submitting new warning code. It would be unprecendented if even two of us decided this was our thing. If would be almost unprecendented if even one of us followed up on it <0.6 wink>. > Two, adding my checks required no extra memory management overhead. Really good global analysis likely requires again as much C code as already exists. Luckily, I don't think putting in some warnings requires that all conceivable warnings be implemented at once . For stuff that complex, I'd rather make it optional and write it in Python; I don't believe any law prevents the compiler from running Python code. > Consider a few other checks you might conceivably add to the byte code > compiler: > > * tab nanny stuff (already enabled with -t, right?) Very timidly, yes . Doesn't complain by default, and you need -tt to make it an error. Only catches 4 vs 8 tab size ambiguities, but that's good enough for almost everyone almost all the time. > * variables set but not used > * variables used before set These would be wonderful. The Perl/pylint "gripe about names unique in a module" is a cheap approximation that gets a surprising percentage of the benefit for the cost of a dict and an exception list. > If all of this sort of stuff is added to the compiler proper, I predict a > couple major problems will surface: > > * The complexity of the code will increase significantly, making it > harder to maintain and enhance The complexity of the existing code should be almost entirely unaffected, because non-trivial semantic analysis belongs in a new subsystem with its own code. > * Fewer and fewer people will be able to approach the code, making it > less likely that new checks are added As opposed to what? The status quo, with no checks at all? Somehow, facing the prospect of *some* checks doesn't frighten me away . Besides, I don't buy the premise: if someone takes this on as their project, worrying that they'll decline to add new valuable checks is like MarkH worrying that I wouldn't finish adding full support for stinking tabs to the common IDLE/PythonWin editing components. People take pride in their hackery. > * Future extensions like pluggable virtual machines will be harder > to add because their byte code compilers will be harder to integrate > into the core If you're picturing adding this stuff sprayed throughout the guts of the existing com_xxx routines, we've got different pictures in mind. Semantic analysis is usually a pass between parsing and code generation, transforming the parse tree and complaining about source it thinks is fishy. If done in any conventional way, it has no crosstalk at all with either the parsing work that precedes it or the code generation that follows it. It's a pipe stage between them, whose output is of the same type as its input. That is, it's a "pluggable component" in its own right, and doesn't even need to be run. So potential for interference just isn't there. At present, Python is very unusual both in: 1) Having no identifiable semantic pass at all, parsing directly to byte code, and enforcing its few semantic constraints (like "'continue' not properly in loop") lumped in with both of those. and 2) Having one trivial optimization pass-- 76 lines of code instead of the usual 76,000 --running after the byte code has been generated. However, the sole transformation made here (distinguishing local from non-local names) is much more properly viewed as being a part of semantic analysis than as being "an optimization". It's deducing trivial info about what names *mean* (i.e., basic semantics), and is called optimization here only because Python didn't do it at first. So relating this to a traditional compiler, I'd say that "optimize()" is truly Python's semantic analysis pass, and all that comes before it is the parsing pass -- a parsing pass with output in a form that's unfortunately clumsy for further semantic analysis, but so it goes. The byte code is such a direct reflection of the parse tree that there's really little fundamental difference between them. So for minimal disruption, I'd move "optimize" into a new module and call it the semantic analysis pass, and it would work with the byte code. Just as now, you wouldn't *need* to call it at all. Unlike now, the parsing pass probably needs to save away some more info (e.g., I don't *think* it keeps track of what all is in a module now in any usable way). For Python2, I hope Guido adopts a more traditional structure (i.e., parsing produces a parse tree, codegen produces bytecode from a parse tree, and other tree->tree transformers can be plugged in between them). Almost all compilers follow this structure, and not because compiler writers are unimaginative droids . Compile-time for Python isn't anywhere near being a problem now, even on my creaky old 166MHz machine; I suspect the current structure reflects worry about that on much older & slower machines. Some of the most useful Perl msgs need to be in com_xxx, though, or even earlier. The most glaring example is runaway triple-quoted strings. Python's "invalid token" pointing at the end of the file is maddeningly unhelpful; Perl says it looks like you have a runaway string, and gives the line number it thinks it may have started on. That guess is usually correct, or points you to what you *thought* was the end of a different string. Either way your recovery work is slashed. (Of course IDLE is even better: the whole tail of the file changes to "string color", and you just need to look up until the color changes!) > In addition, more global checks probably won't be possible (reasoning about > code across module boundaries for instance) because the compiler's view of > the world is fairly narrow. As above, I don't think there's enough now even to analyze one module in isolation. > I think lint-like tools should be implemented in Python (possibly with the > support of an extension module for performance-critical sections) which is > then called from the compiler proper under appropriate circumstances > (warnings enabled, all necessary modules importable, etc). I have no objection to that. I do object to the barely conceivable getting in the way of the plainly achievable, though -- the pylint tools out there now, just like your return consistency checker, do a real service already without any global analysis. Moving that much into the core (implemented in Python if possible, recoded in C if not) gets a large chunk of the potential benefit for maybe 5% of the eventual work. It's nice that Guido is suddenly keen on global analysis too, but I don't see him volunteering to do any work either . > I believe the code would be much more easily maintained and extended. If it's written in Python, of course. > You'd be able to swap in a new byte code compiler without risking the > loss of your checking code. I never understood this one; even if there *were* a competing byte code compiler out there <0.1 wink>, keeping as much as possible out of com_xxx should render it a non-issue. If I'm missing your point and there is some fundamental conflict here, fine, then it's another basis on which bytecode compilers will compete. more-concerned-about-things-that-exist-than-things-that-don't-ly y'rs - tim From iporres@abo.fi Sun Sep 12 10:54:36 1999 From: iporres@abo.fi (Ivan Porres) Date: Sun, 12 Sep 1999 12:54:36 +0300 Subject: [Python-Dev] Memory usage in Python Message-ID: <19990912125436.B735@marchica.abo.fi> Dear all, I've been reading this list for a while but this is the first time I post a message. I am interested on using Python as a scripting language for small embedded systems. One of the limiting factors on a embedded system is the memory. Memory cost money, uses space and drains power. We have released a big patch to selectively disable some features of a Python interpreter at compile time. With the patch, it is possible to disable, i.e., complex numbers or file objects, in order to reduce the memory foot print of the Python interpreter. The patch is available at http://www.abo.fi/~iporres/python Now, I am interested on studying and reducing the memory consumption of Python programs. That is all the memory allocated by the interpreter in order to run a python program. I would like to know if somebody else has been working on the same direction. Is there any document explaining how Python uses the memory? I would be nice to know which structures are most used, their size, how often are accessed, etc... I have the impression that most people is concerned about optimising Python for speed, but I still think that some Python users will also benefit from size optimisations. Ivan -- Ivan Porres Paltor Turku Centre for Computer Science Åbo Akademi, Department of Computer Science Phone: +358-2-2154033 Lemminkäinengatan 14A FIN-20520 Turku - Finland http://www.abo.fi/~iporres From skip@mojam.com (Skip Montanaro) Sun Sep 12 15:20:14 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Sun, 12 Sep 1999 09:20:14 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000001befcff$bf654e20$492d153f@tim> References: <199909101405.KAA26303@eric.cnri.reston.va.us> <000001befcff$bf654e20$492d153f@tim> Message-ID: <14299.46750.214399.735453@dolphin.mojam.com> Skip> * Future extensions like pluggable virtual machines will be harder Skip> to add because their byte code compilers will be harder to Skip> integrate into the core Tim> If you're picturing adding this stuff sprayed throughout the guts Tim> of the existing com_xxx routines, we've got different pictures in Tim> mind. This was precisely my example, because that's the way I implemented the return warning stuff, by modifying the com_xxx routines. I believe that's the wrong way to go in the long run, and I see by the rest of your message you feel the same way as well. To the greatest extent possible, I think this stuff should be implemented in Python. (We may disagree on that point.) Being able to plug in new parse tree analysis/transformation modules between parse tree creation and code generation could at least be controlled from Python. Skip P.S. Something I just noticed: Since the node typedef (node.h) and the macros that manipulate nodes are shared across multiple files shouldn't they be named something slightly less likely to clash with other packages? From mal@lemburg.com Mon Sep 13 11:05:46 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:05:46 +0200 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> Message-ID: <37DCCC7A.545B44E6@lemburg.com> Tim Peters wrote: > > [Guido] > > I'd much rather use absolute package names for anything that's not in > > the same directory as the current module. > > [M.-A. Lemburg] > > Of course, you could do everything with absolute names, but then > > the package author would dictate the complete absolute path which > > is not always desirable since it can cause name collisions such > > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > > released) and Mark's win32 stuff. > > > > As more and more packages appear, we run into this problem more > > and more often. > > I never understand package complaints. Maybe it's the imprecision of the > language people use, or maybe because it's because people don't give fully > fleshed-out examples. Whatever, in the end, I never have any idea what the > complaint is, or in what way the solution is "solving" anything. My original post contained an example package using relative imports. The example uses intra-package imports across subpackage levels which currently is only possible using absolute module names (see below for a definition). Note that I did not want to start a discussion about absolute vs. relative names (I believe everybody agrees that realtive file names are a Good Thing). The 'import __.module' thing is not new: ni.py had support for this and my patch simply adds it back to the implementation. Think of the '__' as an additional feature that authors can use at their own will. They don't *have* to, just as you don't have to use relative file paths in your config files. It's just a convenience that some authors may want to use in their code. > In the above, "absolute" doesn't appear to mean "absolute" in any > OS-sensible sense. So what does it really mean? Does it mean the same > things to Guido and MAL? "Absolute" means fully qualified name, i.e. the complete path to the modules from the top-level root via all subpackage down to the module name itself, e.g. TextTools.Constants.TagTable. > In MAL's hint of examples, I don't see any problem. If mxDateTime unpacks > itself into a directory named DateTime, then *of course* it's going to > collide with other packages that want to do likewise. Install it into > mxDateTime instead, and take "absolute" to mean "any module that wants an > mxDateTime service and does not itself live directly in mxDateTime/ must > import the desired module via a path beginning 'mxDateTime.'", and > everything looks straightforward to me (and that outcome makes me infer that > this is thus probably what Guido has in mind too). This is what I intend to do: move all my stuff under a 'mx' package and then continue reinventing (faster, different, doesn't-slip-when-wet) wheels ;-). BTW, the queue thingie is just a proof of concept fun project. > Similarly for > win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be > good to have an explicit way to say "std library" -- "Lib." comes to mind). I'd prefer 'org.python.core.' or 'stdlib.' or just simply 'python.' but that's a different issue. > > I could then make all my packages self-contained and > > distribute them in two forms without having to change > > a single line for the added support: > > > > 1. under the new 'mx' package, e.g. mx.DateTime > > 2. for backward compatibility under 'DateTime' > > Ah, so that's what this is about. I vote screw #2. Releasing it that way > was a mistake. Not until Zope went Open Source ;-) ... > Better to live with the brief & finite pain of repairing it > than complicating Python to cover up for it. Screwing #2 is not possible unless I want all those already stored DateTime pickles to fail loading... ok, I could probably provide some kind of compatibility package which then redirects the import to mx.DateTime. > > Another major advantage is that I could take any other > > self-contained package and install it under absolute paths > > of my choice, e.g. put Zope under org.zope.core, Python under > > org.python.core etc., without harming their functionality > > or having to dive deep into their import structures to fix > > them manually. > > I view that not as an advantage but as harmful complication. Zope etc add > great value to a Python installation, and when I write a killer app full of > "import zope.this" and "import zope.that", I don't want the possibility that > it's not going to work on my client's machine just because their sysadmin > installed Zope into some silly site-specific path named after his soon-to-be > ex-girlfriend . I don't want a way to work around him doing that, > either: I don't want him to be able to screw me to begin with. Zope is not a package AFAIK (and probably never will be due to the pickle complications), so 'import zope.this' won't work anyways unless you add a Zope wrapping package of your own -- and this will only work for Zope modules not relying on other Zope modules unless they use relative imports. > > To further enhance this mechanism I would like to have an > > alias mechanism in import, pickle et al. so that changes > > in the package structures become manageable without user > > intervention: pickles are a major problem whenever import > > structures change because they store absolute module names. > > This is a different issue, and may have merits of its own. WRT the relative > import scheme, its advantage seems to lie in providing a way to partially > recover from the damage the new scheme causes <0.5 wink>. I'm not proposing a new scheme... only a convenience for package authors. > As is, the package name used by a release is part of its published > interface. You can't change it without causing pain, any more than you can > make incompatible changes to public class methods or input-output behavior. > In return, package clients are uniform, simple and portable, making life > easiest for the people who know least. The burden is on package authors to > choose names wisely, and that's where the burden should be. Sure, but we are heading into these kind of problems just now and have no proper solution at hand. Simply coping out by blaiming the package authors is not the right way to handle the situation; referring them to some global name registry (the one at NIST) isn't either. As the Zope example shows, it's not easy to move from half-packaged to full packaged (relative imports will ease this move, though) due to external references using absolute object type/class names. Changes in the package structure produce the same problems. It's not so much the burden of the programmer having to adjust his code to the new layout I'm talking about here: it's the user with all his data stored in object databases that worries me. > if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs - tim Not if just one of them decides to make it top-level... grabbing the name until eternity ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Mon Sep 13 11:17:31 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:17:31 +0200 Subject: [Python-Dev] Relative Package Imports References: <1275018545-12250772@hypernet.com> Message-ID: <37DCCF3B.2F1E962D@lemburg.com> Gordon McMillan wrote: > > M.-A. Lemburg wrote: > > Guido van Rossum wrote: > > > > > > I'd much rather use absolute package names for anything that's not > > > in the same directory as the current module. > > > > Of course, you could do everything with absolute names, but then > > the package author would dictate the complete absolute path which is > > not always desirable since it can cause name collisions such as > > DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) > > and Mark's win32 stuff. > > I can see your point (although I also believe that authors - Guido > excepted - should come up with collision-free names, probably in a > Java-ish scheme). Agreed. > But I strongly believe that import.c should be left alone, maybe > even to die. There are too many people doing import hooks to > make fiddling with its behavior safe. > > I'm also a strong proponent of Greg's imputil.py scheme, which > makes it a breeze to do import hooks. And my experience > disproves the notion that the import mechanism needs to be in C. If > you don't believe me, try the ZlibArchive stuff (which is cross > platform) from my Win32 installer stuff. You can pack the standard > library into one 475K file, and get a perceptible performance boost. You're probably right in saying that we don't need the code in C. I just wanted to avoid yet another import hook being incompatible with all the other existing hooks. Perhaps we should restart the import discussion all over and come up with a more flexbile 100% compatible framework based on Greg's imputil scheme. Then I could add my hook for the relative imports and be happy ;-) > OTOH, I could see doing a framework of packages, in which case > relative imports might be handy. This seems to work fine: > > def relimp(nm): > rpth = string.split(nm, '/') > tpth = string.split(__name__, '.')[:-1] > for node in rpth: > if node == '..': > del tpth[-1] > else: > tpth.append(node) > #print `tpth` > return __import__(string.join(tpth, '.')) > > b = relimp('../packageA2.b') This is pretty much how my patch works... except that I use the ni.py style '__' pseudo package instead of '../'. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Mon Sep 13 11:27:04 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:27:04 +0200 Subject: [Python-Dev] Memory usage in Python References: <19990912125436.B735@marchica.abo.fi> Message-ID: <37DCD178.15676B9F@lemburg.com> Ivan Porres wrote: > > Dear all, > > I've been reading this list for a while but this is the first time I > post a message. I am interested on using Python as a scripting > language for small embedded systems. One of the limiting factors on a > embedded system is the memory. Memory cost money, uses space and > drains power. > > We have released a big patch to selectively disable some features of a > Python interpreter at compile time. With the patch, it is possible to > disable, i.e., complex numbers or file objects, in order to reduce > the memory foot print of the Python interpreter. The patch is > available at http://www.abo.fi/~iporres/python > > Now, I am interested on studying and reducing the memory consumption of > Python programs. That is all the memory allocated by the interpreter > in order to run a python program. I would like to know if somebody > else has been working on the same direction. Is there any document > explaining how Python uses the memory? I would be nice to know which > structures are most used, their size, how often are accessed, > etc... You should have a look at Vladimir Marangozov's pymalloc (hope this URL still works): http://sirac.inrialpes.fr/~marangoz/python/ It has some logging facilities which give detailed reports about the storage sizes etc. > I have the impression that most people is concerned about optimising > Python for speed, but I still think that some Python users will > also benefit from size optimisations. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim@digicool.com Mon Sep 13 13:44:56 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 08:44:56 -0400 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> Message-ID: <37DCF1C8.27CF657F@digicool.com> Guido van Rossum wrote: > > > I've spent the last two hours trying to get relative package > > imports to work because I need them for my extension packages > > which will soon all move under a new top-level package name > > to overcome the conflicts with PIL and Zope. > > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. I'll second Marc-Andre here. A significant headache occurs when you have a package that has sub-packages. Sub-packages need to be able to reference other sub-packages within the same package without knowing where the containing package is installed. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From guido@CNRI.Reston.VA.US Mon Sep 13 14:09:59 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 13 Sep 1999 09:09:59 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 08:44:56 EDT." <37DCF1C8.27CF657F@digicool.com> References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> Message-ID: <199909131309.JAA29239@eric.cnri.reston.va.us> [MAL] > > > I've spent the last two hours trying to get relative package > > > imports to work because I need them for my extension packages > > > which will soon all move under a new top-level package name > > > to overcome the conflicts with PIL and Zope. [GvR] > > I'd much rather use absolute package names for anything that's not in > > the same directory as the current module. [JimF] > I'll second Marc-Andre here. > > A significant headache occurs when you have a package > that has sub-packages. Sub-packages need to be able to > reference other sub-packages within the same package without > knowing where the containing package is installed. You never need to know where it is installed. When I said absolute package name I meant package name (e.g. zope.foo.bar.subpack) not filename. As Tim has argued, the ability to change the name of the toplevel here is a liability, not a feature. (On reduced bandwidth here, hence not so subtle.) --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Mon Sep 13 14:14:56 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Mon, 13 Sep 1999 23:14:56 +1000 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DCF1C8.27CF657F@digicool.com> Message-ID: <000d01befde9$f9c70690$0801a8c0@bobcat> [Jim writes] > I'll second Marc-Andre here. > > A significant headache occurs when you have a package > that has sub-packages. Sub-packages need to be able to > reference other sub-packages within the same package without > knowing where the containing package is installed. I certainly dont want to know _where_ the package is installed, but do want to know the package name. A package name, IMO, is like a module name - once named, you are largely stuck with it. I for one use packages quite extensively, but have never lamented the loss of __ from the ni days. If someone, for example, wanted to change the package name of "win32com" or "pywin", then I would expect lots of code to break, in the same way as changing the name any standard module would break code. I see Marc's issue with pickle, but I dont see the problem with absolute package names any more than I see a problem with explicitely naming modules in an import statement... Mark. From jim@digicool.com Mon Sep 13 14:32:25 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:32:25 -0400 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> Message-ID: <37DCFCE9.A00DB3E0@digicool.com> Tim Peters wrote: > (snip) > As is, the package name used by a release is part of its published > interface. You can't change it without causing pain, any more than you can > make incompatible changes to public class methods or input-output behavior. > In return, package clients are uniform, simple and portable, making life > easiest for the people who know least. The burden is on package authors to > choose names wisely, and that's where the burden should be. Not all packages are part of the external interface. In fact, all Zope names are essentially internal, since Zope is an application. The issue is not so much access to access from outside as it is access between packages within Zope. Further, the current support for relative imports allows a package to be moved into another package without breaking the pulic interface wrt the containing package. Here's an example that I hope will be motivating: Suppose Marc-Andre has a package mx with subpackages DateTime and stringtools. If mx was installed in the Python path then a module in the mx.DateTime package could get at stringtools like: import mx.stringtools So far, so good. Zope has a notion of products which are *self contained* packages that are sub-packages of the Products package. So, suppose someone wants to write a NiftyDB product, which is a Zope product that provides access to an external database. Now the author of the NiftyDB product wants to use the mx package. The mx package is not a standard part of Zope, or of Python, so they simpley include it in the NiftyDB product directory. Becase relative imports are allowed in the current import scheme, they can use mx as usual. A NiftyDB module can import DateTime as follows: import mx.DateTime So even though mx is istalled as a sub-package, the public interface is unchanged, at least wrt the containing package. Unfortunately, the internal import of stringtools in the DateTime package: import mx.stringtools will fail, because mx is no longer a top-level module. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@digicool.com Mon Sep 13 14:32:39 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:32:39 -0400 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> Message-ID: <37DCFCF7.DACEF21D@digicool.com> "M.-A. Lemburg" wrote: > > Tim Peters wrote: > > > > [Guido] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > > > > [M.-A. Lemburg] > > > Of course, you could do everything with absolute names, but then > > > the package author would dictate the complete absolute path which > > > is not always desirable since it can cause name collisions such > > > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > > > released) and Mark's win32 stuff. > > > > > > As more and more packages appear, we run into this problem more > > > and more often. > > > > I never understand package complaints. Maybe it's the imprecision of the > > language people use, or maybe because it's because people don't give fully > > fleshed-out examples. Whatever, in the end, I never have any idea what the > > complaint is, or in what way the solution is "solving" anything. > > My original post contained an example package using relative > imports. The example uses intra-package imports across subpackage > levels which currently is only possible using absolute module > names (see below for a definition). I agree that this is a valid complaint about the current scheme. > Note that I did not want to start a discussion about absolute vs. > relative names (I believe everybody agrees that realtive file names > are a Good Thing). The 'import __.module' thing is not new: ni.py > had support for this and my patch simply adds it back to the > implementation. And, in fact, the current scheme does let you use relative paths to go down. (snip) > > > I could then make all my packages self-contained and > > > distribute them in two forms without having to change > > > a single line for the added support: > > > > > > 1. under the new 'mx' package, e.g. mx.DateTime > > > 2. for backward compatibility under 'DateTime' > > > > Ah, so that's what this is about. I vote screw #2. Releasing it that way > > was a mistake. > > Not until Zope went Open Source ;-) ... > > > Better to live with the brief & finite pain of repairing it > > than complicating Python to cover up for it. > > Screwing #2 is not possible unless I want all those already > stored DateTime pickles to fail loading... ok, I could probably > provide some kind of compatibility package which then redirects > the import to mx.DateTime. Regardless of how you spell the import, the pickles *must* reflect the absolute path. Otherwise, pickleability depends on where you unpickle. If you rearrange packages, or rename modules, there is a pickling issue. This is clearly a problem in need of a better solution. (snip) > Zope is not a package AFAIK Nope. If I had it to do over it would be. It will be eventually. (While Zope is not a package, it makes extensive use of packges. Zope has a total of about 350 modules and packages, of which about 44 are in the top-level namespace.) > (and probably never will be due to the pickle complications), The pickling issues are solvable in a number of ways, although it is a bit painfull. I considered fixing the package layout in Zope 2, but I ran out of time. :) > so 'import zope.this' won't work anyways > unless you add a Zope wrapping package of your own -- and this will > only work for Zope modules not relying on other Zope modules unless > they use relative imports. This is a good example. It should be possible to make Zope package- portable but it isn't, at least not without writing import hooks, which make my head hurt alot more than meta-classes. ;) > > > To further enhance this mechanism I would like to have an > > > alias mechanism in import, pickle et al. so that changes > > > in the package structures become manageable without user > > > intervention: pickles are a major problem whenever import > > > structures change because they store absolute module names. > > > > This is a different issue, and may have merits of its own. WRT the relative > > import scheme, its advantage seems to lie in providing a way to partially > > recover from the damage the new scheme causes <0.5 wink>. > > I'm not proposing a new scheme... only a convenience for package > authors. I think that this is an important convenience that is probably not appreciated until you write a complex package structure. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@digicool.com Mon Sep 13 14:35:21 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:35:21 -0400 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> <199909131309.JAA29239@eric.cnri.reston.va.us> Message-ID: <37DCFD99.F48FC65E@digicool.com> Guido van Rossum wrote: > > [MAL] > > > > I've spent the last two hours trying to get relative package > > > > imports to work because I need them for my extension packages > > > > which will soon all move under a new top-level package name > > > > to overcome the conflicts with PIL and Zope. > [GvR] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > [JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. > > You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. I wasn't refering to file path location either, but to package name-space location. See the example I just posted in reply to Tim. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From guido@CNRI.Reston.VA.US Mon Sep 13 14:40:10 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 13 Sep 1999 09:40:10 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 12:05:46 +0200." <37DCCC7A.545B44E6@lemburg.com> References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> Message-ID: <199909131340.JAA29340@eric.cnri.reston.va.us> > Note that I did not want to start a discussion about absolute vs. > relative names (I believe everybody agrees that realtive file names > are a Good Thing). The 'import __.module' thing is not new: ni.py > had support for this and my patch simply adds it back to the > implementation. It was left out of the 1.5 package implementation on purpose, and I will fight proposals to get it back. It is an ugly hack that deserves to die. So there! > "Absolute" means fully qualified name, i.e. the complete path to > the modules from the top-level root via all subpackage down to the > module name itself, e.g. TextTools.Constants.TagTable. No disagreement there. BTW, The compatibility issues with class references in object databases need to be resolved in a different way -- package renamings are but one of the problems here. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Sep 13 14:57:32 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 15:57:32 +0200 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> <199909131309.JAA29239@eric.cnri.reston.va.us> Message-ID: <37DD02CC.26B20BDD@lemburg.com> Guido van Rossum wrote: > > [MAL] > > > > I've spent the last two hours trying to get relative package > > > > imports to work because I need them for my extension packages > > > > which will soon all move under a new top-level package name > > > > to overcome the conflicts with PIL and Zope. > [GvR] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > [JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. > > You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. I think Jim meant the absolute package name just like you and I did. I don't really understand why you and Tim oppose to relative imports... Tim's argument about sysadmins installing packages under fantasy names isn't really all that realistic, IMHO, even though it would work. The ability to host a subpackage inside another package is a very attractive feature for large package based applications: you don't have to edit a single line in the subpackage's source code and can install/update the subpackage without harming its functionality within the system. BTW: The patch I posted does not add a circular reference to the modules as the old ni.py implementation (sometimes) did. Since the __ handling is done dynamically and on import only there is no need to add a module attribute __ pointing to the parent module. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Mon Sep 13 15:08:36 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 16:08:36 +0200 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> <199909131340.JAA29340@eric.cnri.reston.va.us> Message-ID: <37DD0564.7361B789@lemburg.com> Guido van Rossum wrote: > > > Note that I did not want to start a discussion about absolute vs. > > relative names (I believe everybody agrees that realtive file names > > are a Good Thing). The 'import __.module' thing is not new: ni.py > > had support for this and my patch simply adds it back to the > > implementation. > > It was left out of the 1.5 package implementation on purpose, and I > will fight proposals to get it back. It is an ugly hack that deserves > to die. So there! I agree that it's ugly to include the __ attribute in the module namespace due to the possible circular reference (parent->module, module->parent), but the patch I sent doesn't do this... or was "ugly" referring to the two underscores looking strange ? Could you elaborate a bit on the reasons for dropping __ support ? Not-really-looking-for-a-fight-here--ly, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm@hypernet.com Mon Sep 13 15:35:16 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 13 Sep 1999 10:35:16 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909131309.JAA29239@eric.cnri.reston.va.us> References: Your message of "Mon, 13 Sep 1999 08:44:56 EDT." <37DCF1C8.27CF657F@digicool.com> Message-ID: <1274889332-20022765@hypernet.com> [relative package imports] 0000,7F00,0000[JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. [GvR] 0000,7F00,0000> You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. In between. I can see relative packages as *one* way of handling certain problems. Consider: zope.Win32 zope.Unix with both of these having alternate implementations of subpackages foo and bar. Then for (the current) foo.a to get to (the current) bar.b, using a relative import seems a natural. This can, of course, be done in pure Python. So can doing things in zope.__init__.py that make the appropriate implementations of foo and bar appear to be zope.foo and zope.bar. On any criteria I can think of, this would be a superior solution. (*) What I am against is further complicating the already over complicated built in import mechanism. (*) such as a zope.__init__.py that looks like this: import sys if sys.platform[:3] == 'win': nm = __name__ + '.Win32' else: nm = __name__ + '.Unix' new = __import__(nm) sys.modules[__name__] = sys.modules[nm] - Gordon From gmcm@hypernet.com Mon Sep 13 15:56:12 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 13 Sep 1999 10:56:12 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DCFCE9.A00DB3E0@digicool.com> Message-ID: <1274888077-20098343@hypernet.com> [Jim Fulton] > Here's an example that I hope will be motivating: > > Suppose Marc-Andre has a package mx with subpackages DateTime > and stringtools. ... > Zope has a notion of products which are *self contained* packages that > are sub-packages of the Products package. So, suppose someone wants > to write a NiftyDB product, which is a Zope product that provides > access to an external database. Now the author of the NiftyDB product > wants to use the mx package. The mx package is not a standard part of > Zope, or of Python, so they simpley include it in the NiftyDB product > directory. This would motivate me to strangle the author of NiftyDB. He should say "This requires Marc-Andre's mx package", and put error handling on the import. Otherwise I might end up with a gazillion separate copies of the mx package installed on my system and no way to clean it up. Your notion of "self contained packages" is flawed. Reducto ad adsurdum, it leads to downloading the OS with every app you install. (Crap, I hope I haven't given Sun any new ideas...). - Gordon From jim@digicool.com Mon Sep 13 16:16:03 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 11:16:03 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> Message-ID: <37DD1533.24A05F69@digicool.com> Gordon McMillan wrote: > > [Jim Fulton] > > Here's an example that I hope will be motivating: > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > and stringtools. > ... > > Zope has a notion of products which are *self contained* packages that > > are sub-packages of the Products package. So, suppose someone wants > > to write a NiftyDB product, which is a Zope product that provides > > access to an external database. Now the author of the NiftyDB product > > wants to use the mx package. The mx package is not a standard part of > > Zope, or of Python, so they simpley include it in the NiftyDB product > > directory. > > This would motivate me to strangle the author of NiftyDB. He > should say "This requires Marc-Andre's mx package", and put error > handling on the import. This is unacceptable. People should not have to hack a global namespace to get a local feature. > Otherwise I might end up with a gazillion > separate copies of the mx package installed on my system and no > way to clean it up. Or you might not. In any case, the ability to have localized references should be there. Actually, Zope does provide a place to share packages that multiple products use. This involves putting shared packages in a sub-package of the 'Shared' package. For example, Digital Creations shared packages go in Shared.DC. Putting mx in Shared.ACME runs into the same problem. Although the product author can: import Shared.ACME.mx.DateTime the import of mx.stringtools in DateTime will still fail. Now, maybe ACME will hack mx to change the internal imports: import Shared.ACME.mx.stringutils but their hacks will get broken when we make Zope a package and Shared becomes a sub-package of Zope. > Your notion of "self contained packages" is flawed. Reducto ad > adsurdum, it leads to downloading the OS with every app you > install. (Crap, I hope I haven't given Sun any new ideas...). I'm sorry, but relying on existing Python installations for commercial applications is just not practical. In fact, Zope binary distributions contain their own Python distributions because relying on preinstalled Python distributions is just way too much of a support burden and a hassle for out customers. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@digicool.com Mon Sep 13 16:21:15 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 11:21:15 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> Message-ID: <37DD166B.746B19AD@digicool.com> Gordon McMillan wrote: > > [relative package imports] > [JimF] > > > I'll second Marc-Andre here. > > > > > > A significant headache occurs when you have a package > > > that has sub-packages. Sub-packages need to be able to > > > reference other sub-packages within the same package without > > > knowing where the containing package is installed. > [GvR] > > You never need to know where it is installed. When I said absolute > > package name I meant package name (e.g. zope.foo.bar.subpack) not > > filename. As Tim has argued, the ability to change the name of the > > toplevel here is a liability, not a feature. > > In between. I can see relative packages as *one* way of handling > certain problems. Consider: > > zope.Win32 > zope.Unix > > with both of these having alternate implementations of subpackages foo > and bar. Then for (the current) foo.a to get to (the current) bar.b, > using a relative import seems a natural. I wouldn't use relative imports for this. > This can, of course, be done in pure Python. So can doing things in > zope.__init__.py that make the appropriate implementations of foo and > bar appear to be zope.foo and zope.bar. On any criteria I can think of, > this would be a superior solution. (*) I agree. This is not a good motivating example for relative imports. > What I am against is further complicating the already over complicated > built in import mechanism. The current scheme doesn't seem all that complicated to me. I wouldn't be in favor of making it more complicated if there wasn't a good reason. I think that, in working on the Zope framework, I've found some pretty good reasons for relative imports. > > (*) such as a zope.__init__.py that looks like this: > > import sys > if sys.platform[:3] == 'win': > nm = __name__ + '.Win32' > else: > nm = __name__ + '.Unix' > new = __import__(nm) > sys.modules[__name__] = sys.modules[nm] Right, this would be silly. This is not a good example. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal@lemburg.com Mon Sep 13 16:27:31 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 17:27:31 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> Message-ID: <37DD17E3.74FEB43F@lemburg.com> Gordon McMillan wrote: > > [Jim Fulton] > > Here's an example that I hope will be motivating: > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > and stringtools. > ... > > Zope has a notion of products which are *self contained* packages that > > are sub-packages of the Products package. So, suppose someone wants > > to write a NiftyDB product, which is a Zope product that provides > > access to an external database. Now the author of the NiftyDB product > > wants to use the mx package. The mx package is not a standard part of > > Zope, or of Python, so they simpley include it in the NiftyDB product > > directory. > > This would motivate me to strangle the author of NiftyDB. He > should say "This requires Marc-Andre's mx package", and put error > handling on the import. Otherwise I might end up with a gazillion > separate copies of the mx package installed on my system and no > way to clean it up. The mx package installed in product dir won't be visible to outside scripts: it is self-contained and only works in the context of Zope. There's nothing bad about it if you take the POV of a typical Zope user who doesn't know about the internals too much. OTOH, if the user does have the skill to install packages himself, he could install the mx package top-level and then leave it out of the NiftyDB product dir. Both setups will work without any source code change... that's the real advantage. Currently, only the top-level installation of the mx package will work and this of course introduces other problems like version control, e.g. the NiftyDB product (relying on mx 1.9) may not support version 2.0 of mx which the sysadmin installed at top-level. There would be no way (apart from patching NiftyDB) to get back to a working setup. > Your notion of "self contained packages" is flawed. Reducto ad > adsurdum, it leads to downloading the OS with every app you > install. (Crap, I hope I haven't given Sun any new ideas...). While you do of course have a point in that packages should not be loaded twice (a smarter import mechanism written in Python could handle these situations along with the relative import), you should consider that Zope is an application. It could go the WinXX way and simply hack around in the OS or wrap everything it needs up in the package itself. Both have their pros and cons, both are possible using relative imports (I'm starting to feel like one of those marketing guys ;). BTW, I really don't care much about this feature being coded in Python or C. All I would like is to have it enabled and available per default, so that writing self-contained packages becomes a standard in the Python world. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jcw@equi4.com Mon Sep 13 17:23:35 1999 From: jcw@equi4.com (Jean-Claude Wippler) Date: Mon, 13 Sep 1999 18:23:35 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> Message-ID: <37DD2507.8A7AD0D1@equi4.com> "M.-A. Lemburg" wrote: > Gordon McMillan wrote: > > Jim Fulton wrote: [deeply nested snippety snip] > > > > This would motivate me to strangle the author of NiftyDB. He > > should say "This requires Marc-Andre's mx package", [...] Yes. > The mx package installed in product dir won't be visible to outside > scripts: it is self-contained and only works in the context of Zope. But why shouldn't it be visible? The only approach which will not self-destruct IMO, is to segment on source-of-origin. Your package in your namespace, Gordon's in his, and Jim's in yet another. Given that there is no ordering relationship, that means three areas next to each other. If Jim takes M-A's package and modifies it, then that would be a reason to put M-A's-modified-by-Jim-package within Jim's area. > Currently, only the top-level installation of the mx package > will work and this of course introduces other problems like > version control, e.g. the NiftyDB product (relying on mx 1.9) > may not support version 2.0 of mx which the sysadmin installed > at top-level. There would be no way (apart from patching NiftyDB) > to get back to a working setup. Which points to a solution in a completely different direction: version numbers in package directories, like GNU and Tcl do, to name two. > [...] have it enabled and available per default, so that writing > self-contained packages becomes a standard in the Python world. A noble goal. Now this may be a silly question, but what's the issue here? I'm used to writing things which sort of look like this: import myGoobledygook utils = myGoobledygook print utils.fun(123) Couldn't this be applied here as well: mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") In other words: figure out a way to get at the proper modules, then use an alias in your own code to stay away from naming/access dependencies? And if a parent wants to tell a submodule how to find it, can't it just set a variable at module level in that submodule, to reach it? Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, DNS or LDAP servers, FS mount points - it's all the same issue. -- Jean-Claude From jim@digicool.com Mon Sep 13 17:56:42 1999 From: jim@digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 12:56:42 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> Message-ID: <37DD2CCA.B5C29881@digicool.com> Jean-Claude Wippler wrote: > (snip) > > The mx package installed in product dir won't be visible to outside > > scripts: it is self-contained and only works in the context of Zope. > > But why shouldn't it be visible? Because visibility has a cost. Making mx a top-level name means that someone else can't make it a top-level name. This is why packages are a good idea. If the need is local, the definition shouldn't be made global *just* to get around a limitation in spelling. > The only approach which will not > self-destruct IMO, is to segment on source-of-origin. Your package in > your namespace, Gordon's in his, and Jim's in yet another. Given that > there is no ordering relationship, that means three areas next to each > other. It must be possible to package things together regardless of point of origin. Otherwise, you have a big obstical to reuse. BTW, I think that there is ample evidence that relative imports will not cause anything to self destruct. > If Jim takes M-A's package and modifies it, then that would be a > reason to put M-A's-modified-by-Jim-package within Jim's area. But Jim wants to use M-A's package as a black box. I don't want to hack all of his imports due to a packaging detail. (snip) > Now this may be a silly question, but what's the issue here? I'm used > to writing things which sort of look like this: > import myGoobledygook > utils = myGoobledygook > print utils.fun(123) > Couldn't this be applied here as well: > mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") > In other words: figure out a way to get at the proper modules, then use > an alias in your own code to stay away from naming/access dependencies? > > And if a parent wants to tell a submodule how to find it, can't it just > set a variable at module level in that submodule, to reach it? > > Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, > DNS or LDAP servers, FS mount points - it's all the same issue. Yes, these are options, as are import hooks. While these are possible alternatives, they are far less attractive than simply making it possible to spell relative imports. Why? Well, for one, they require a lot of cooperation among packages that might be written by totally different authors. I might come up with a handy relative import function, but how do I get M-A to use it? Well, I'm sure I could get him to use it, but how about all of the other people whose packages I want to reuse. No problem, we'll come up with a standard mechansism, maybe even a standard library module. Of course, someone is bound to realize sooner or later that it's silly to have a standard relative import library module *and* a standard import statement and add the feature to standard import. :) Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal@lemburg.com Mon Sep 13 19:21:08 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 20:21:08 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> Message-ID: <37DD4094.1BDCE45F@lemburg.com> Jean-Claude Wippler wrote: > > "M.-A. Lemburg" wrote: > > Gordon McMillan wrote: > > > Jim Fulton wrote: > [deeply nested snippety snip] > > > > > > This would motivate me to strangle the author of NiftyDB. He > > > should say "This requires Marc-Andre's mx package", [...] > > Yes. Of course ;-) ... but that's not the point. Commercial products have to come with all batteries included, not just some of them. But you can't include everything known in the Python universe just to avoid conflicts when someone writes a cool plugin for your package. > > The mx package installed in product dir won't be visible to outside > > scripts: it is self-contained and only works in the context of Zope. > > But why shouldn't it be visible? Because it would probably affect other apps using the same package. Just think of what happens to Win9x if you constantly update the DLLs... > The only approach which will not > self-destruct IMO, is to segment on source-of-origin. Your package in > your namespace, Gordon's in his, and Jim's in yet another. Given that > there is no ordering relationship, that means three areas next to each > other. If Jim takes M-A's package and modifies it, then that would be a > reason to put M-A's-modified-by-Jim-package within Jim's area. > > > Currently, only the top-level installation of the mx package > > will work and this of course introduces other problems like > > version control, e.g. the NiftyDB product (relying on mx 1.9) > > may not support version 2.0 of mx which the sysadmin installed > > at top-level. There would be no way (apart from patching NiftyDB) > > to get back to a working setup. > > Which points to a solution in a completely different direction: version > numbers in package directories, like GNU and Tcl do, to name two. A different topic -- an interesting one too :-) How does Java handle this problem ? > > [...] have it enabled and available per default, so that writing > > self-contained packages becomes a standard in the Python world. > > A noble goal. And one which is only reachable by getting whatever mechanism is needed to make it work into the standard distribution -- not only a hook for relative imports but the real thing ! > Now this may be a silly question, but what's the issue here? I'm used > to writing things which sort of look like this: > import myGoobledygook > utils = myGoobledygook > print utils.fun(123) > Couldn't this be applied here as well: > mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") > In other words: figure out a way to get at the proper modules, then use > an alias in your own code to stay away from naming/access dependencies? > > And if a parent wants to tell a submodule how to find it, can't it just > set a variable at module level in that submodule, to reach it? > > Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, > DNS or LDAP servers, FS mount points - it's all the same issue. True, but please remember that we are not talking about adding some esoteric module storage to the standard mechanism (the one reachable via the "import" statement). Relative import only adds a logical notion to imports that opens up new possibilities which are not easily available using other techniques. BTW, would you want to miss relative file name paths ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim@interet.com Mon Sep 13 21:52:57 1999 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 13 Sep 1999 16:52:57 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> Message-ID: <37DD6429.E4F6BF92@interet.com> Jim Fulton wrote: > > Gordon McMillan wrote: > > > > [Jim Fulton] > > > Here's an example that I hope will be motivating: > > > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > > and stringtools. > > ... > > > Zope has a notion of products which are *self contained* packages that > > > are sub-packages of the Products package. So, suppose someone wants > > > to write a NiftyDB product, which is a Zope product that provides > > > access to an external database. Now the author of the NiftyDB product > > > wants to use the mx package. The mx package is not a standard part of > > > Zope, or of Python, so they simpley include it in the NiftyDB product > > > directory. First, I am not all that opposed to having a notion of ".." available in the import statement. If we can write zope.dir1.mod1 which is a relative import going down, maybe we can write ../dir2/mod2 or something spelled differently. But I think there would still be problems. We would be relying on all package authors to use ".." or "__" within their package. But it is more natural to write zope.this.that or mx.this.that everywhere, and that is what people have in fact done. Probably there would be bugs and at least annoyance. And it seems unfortunate to need this feature. A proper package scheme should really isolate package internals from the outside world, and if this isn't happening we need a new scheme. I also don't think there is any escape from needing standard package names, at least the head name "zope" for example. Of course all this currently works if (in Jim's example) both "zope" and "mx" are top-level names. Then the ".." is not necessary. But, as Jim points out: >> But why shouldn't it be visible? >Because visibility has a cost. Making mx a top-level name >means that someone else can't make it a top-level name. >This is why packages are a good idea. The global shared nature of PYTHONPATH and its name space makes it difficult to guarantee that all required packages are going to be present in a complicated installation like zope. And if anyone else installs another Python package, it can easily break the first installation. One solution is for an application to establish its own PYTHONPATH which can not be altered. If this points to "zopedir" then the installer can freely install mx to the directory zopedir/mx and be confident that another mx installation is not damaged, nor used. The logical extension is to place each package into its own file using a scheme like Gordon McMillan is using. If the package contents is obtained by seeking from the END of the file, then multiple package files can be concattenated with cat package1 package2 zope mx >> bigpackage and a large installation like zope can be shipped with its own "bigpackage" library which is essentially a normal PYTHONPATH archive with everything above the head directory names thrown away. The Python library is included under "Lib" just as it is now. The bigpackage library implies its own PYTHONPATH of "Lib;." If the Python library files are placed in the root, the implied PYTHONPATH is ".". Jim Ahlstrom From jcw@equi4.com Mon Sep 13 22:55:43 1999 From: jcw@equi4.com (Jean-Claude Wippler) Date: Mon, 13 Sep 1999 23:55:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> <37DD4094.1BDCE45F@lemburg.com> Message-ID: <37DD72DF.D5904CC@equi4.com> Marc-Andre, > BTW, would you want to miss relative file name paths ? Well, that thought did cross my mind while posting that reply... I'm not sure. Yes, they are convenient, but "current working directory" is not always a pleasant concept (yet more global state - think of tree walks and the inconvenience of having to get, alter, work, restore it - when using chdir to implement this). I don't use .. as often as might be expected. I use "cd ~/something" a lot more. While that may look relative at another level, it really is not (as "echo ~" shows). It does illustrate how nearly equivalent the two approaches are. DOS/Windows has always had a broken relative path: the current drive. And the "hash/bang" headers of Unix seem to always use absolute paths. Some of this may seem to point to the need for relative paths. But I think it hides a more fundamental issue: you need to find out context. Once you do, relativeness no longer matters (obviously). A system which has a current directory and ".." is equivalent to one which has no such thing and passes a "starting directory" in the environment, say. I think a system with less global state is more modular -> preferable. Another angle: I have built many types of tree structures, still do. Less and less of those contain a "parent" link. Instead of storing a parent you can just as easily keep state while descending into children. - Unix file info does not maintain a parent directory, it's redundant. - Directories do, and file system mount points are messy because of it. - Afaik, it's considered bad style to use "../header.h" in C includes. - Upward pointers can introduce cycles (no not here, as has been said). - In C++, member objects rarely need pointers to the enclosing object. It is not for a module to know where it "is" in a hierarchy, it is for a parent to (sometimes) provide that reference when it uses the module. If you want want access to a module called mx, and there may be many of them - then you ask another module to locate it for you. One which is able to choose / decide for you. Proximity may be a good discriminator, but the decision of proximity was not taken by you - and you won't know what rule is best. So you ask an "importer" (which could well always be the standard Python mechanism). Given the task to write such as beast, I'd probably want to implement the following module search as default: - look for the module first in the parent (i.e. as sibling) - move one level up, look again, etc - all the way to the top That's just one way to do it - proximity is not *always* what you want. So if Zope chooses a different packaging style, let it override import. But please don't build ".." into your modules, it doesn't belong there. Sorry for all the handwaving generalities. The issues are the same IMO, whether relativeness is provided or context, and therefore the outcome of this discussion will never be conclusive - they both work. -- Jean-Claude From tim_one@email.msn.com Tue Sep 14 08:02:24 1999 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 14 Sep 1999 03:02:24 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DD0564.7361B789@lemburg.com> Message-ID: <000001befe7f$191104c0$fe2d153f@tim> [MAL] > I agree that it's ugly to include the __ attribute in the module > namespace due to the possible circular reference (parent->module, > module->parent), but the patch I sent doesn't do this... or was > "ugly" referring to the two underscores looking strange ? > > Could you elaborate a bit on the reasons for dropping __ support ? There are two sections on why __ was dropped in http://www.python.org/doc/essays/packages.html They don't refer to circularity, but to "limited use", "poor readability" and "awkwardness". A deeper reason may be hiding in the essay's "most packages will have a relative shallow substructure": this is Guido , the man who invented two-level scoping, and class inheritance without a "super" hook back to the (anonymous) parent. For all Python's dynamicism, it very much favors shallow, static name hierarchies. I don't think it's coincidence that Python's own source code is in a two-level directory structure either! The only #include with a ".." is in grammar.h, and there it's in a comment : #include "bitset.h" /* Sigh... */ So if we cut to the core here, I'd bet Guido doesn't object so much to relative imports as to the idea that anyone would go off and create a package structure so fractally convoluted that relative imports are strongly more attractive than naming the target package in full. Or maybe Guido doesn't care about that at all. I do regardless. I know Python's restrictions can grate, but in all, and in my repeated experience, they force you to rethink complicated designs and refactor them into simpler schemes that fit what Python is best at spelling. Nesting packages 8 deep is clumsy now? Damn straight, and I'm thankful for that: the clumsier it is, the less gratuitous inherited complexity I'll have to deal with in my future lives <0.5 wink>. Things that came up in this thread that are worth fixing include: + Problems with persistent class references (incl. pickles). + Dealing with incompatible versions of packages. If someone wants to embed a copy of (say) mxDateTime in their own package, the only excuses are that they're afraid of overwriting the user's existing mxDateTime installation (if any), and/or of having some future installation of something else overwrite mxDateTime with an incompatible version. Those are bad problems, but package embedding is no solution. You have a much better approach to that already via the DateTime.__version__ string! "Something like that" needs to be formalized and imposed on all public packages. at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim PS: > Just think of what happens to Win9x if you constantly update the DLLs... As a matter of personal experience, it gets much stabler! The older DLLs get replaced by less-buggy newer ones, thanks to version numbers, rules, and installers that finally play by the rules. The mean time between crashes when I installed Win95 a few years ago was about an hour; now it's at least days and possibly weeks (don't know -- never leave the puter on that long). When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. From mal@lemburg.com Tue Sep 14 09:32:38 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 10:32:38 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> Message-ID: <37DE0825.10D2BC6F@lemburg.com> James C. Ahlstrom wrote: > > First, I am not all that opposed to having a notion of ".." > available in the import statement. If we can write zope.dir1.mod1 > which is a relative import going down, maybe we can write > ../dir2/mod2 or something spelled differently. But I think > there would still be problems. > > We would be relying on all package authors to use ".." > or "__" within their package. But it is more natural to write > zope.this.that or mx.this.that everywhere, and that is what > people have in fact done. Probably there would be bugs and > at least annoyance. Think of it as an opportunity: it makes intra-package imports very simple without hard-wiring paths to submodules into the package. In the end, having written a self-contained package is a "feature" of that package. Other authors can then say: ok, I can plugin your package if it's self-contained. -- I have a feeling that this discussion is loosing its grip on reality a bit. Jim and I are not argueing to change Python's face, internal structure or look&feel. The '__' thing doesn't harm anyone, doesn't break any functionality or code. It is even now already possible using import hooks and has been implemented before in ni.py using just that technology. All that we are talking about is reviving relative imports so that its usage is possible *without* adding yet another hook. This is not about the pros and cons of deeply nested packages and neither about advantages of shallow structures. Its only about adding semantics to the string '__' used in import statements. Nothing more. People are not forced to use it. Besides, if used it will be a package internal technique and not be visible to users of that package. Now why would anyone want to make life harder for package authors ? Give the people something to play with so that they have more fun at what their doing for the benefit of the community ! -- > And it seems unfortunate to need this feature. A proper > package scheme should really isolate package internals from > the outside world, and if this isn't happening we need a new scheme. > > I also don't think there is any escape from needing standard > package names, at least the head name "zope" for example. > > Of course all this currently works if (in Jim's example) both > "zope" and "mx" are top-level names. Then the ".." is not > necessary. But, as Jim points out: > > >> But why shouldn't it be visible? > > >Because visibility has a cost. Making mx a top-level name > >means that someone else can't make it a top-level name. > >This is why packages are a good idea. > > The global shared nature of PYTHONPATH and its name space makes > it difficult to guarantee that all required packages are going > to be present in a complicated installation like zope. And if > anyone else installs another Python package, it can easily break > the first installation. > > One solution is for an application to establish its own > PYTHONPATH which can not be altered. If this points to "zopedir" > then the installer can freely install mx to the directory > zopedir/mx and be confident that another mx installation > is not damaged, nor used. > > The logical extension is to place each package into its own > file using a scheme like Gordon McMillan is using. If the > package contents is obtained by seeking from the END of the > file, then multiple package files can be concattenated with > cat package1 package2 zope mx >> bigpackage > and a large installation like zope can be shipped with its > own "bigpackage" library which is essentially a normal > PYTHONPATH archive with everything above the head directory > names thrown away. The Python library is included under > "Lib" just as it is now. The bigpackage library implies > its own PYTHONPATH of "Lib;." If the Python library files are > placed in the root, the implied PYTHONPATH is ".". This is a possibility and in fact I'm using such a setup in my current application. Still it doesn't avoid possible conflicts due to being top-level, e.g. the user could install an extension which relies on a specific top-level name already taken by the hosting package, e.g. the host defines a DateTime package and the extension comes with its own DateTime package. This is the current situation with Zope and mxODBC (which needs mxDateTime). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Tue Sep 14 09:49:52 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 10:49:52 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> <37DD4094.1BDCE45F@lemburg.com> <37DD72DF.D5904CC@equi4.com> Message-ID: <37DE0C30.4B0521B@lemburg.com> Jean-Claude Wippler wrote: > > Marc-Andre, > > > BTW, would you want to miss relative file name paths ? > > Well, that thought did cross my mind while posting that reply... > > I'm not sure. Yes, they are convenient, but "current working directory" > is not always a pleasant concept (yet more global state - think of tree > walks and the inconvenience of having to get, alter, work, restore it - > when using chdir to implement this). I don't use .. as often as might > be expected. I use "cd ~/something" a lot more. While that may look > relative at another level, it really is not (as "echo ~" shows). It > does illustrate how nearly equivalent the two approaches are. > > DOS/Windows has always had a broken relative path: the current drive. > > And the "hash/bang" headers of Unix seem to always use absolute paths. ...and people have turned to /usr/bin/env to get around this necessity :-) > Some of this may seem to point to the need for relative paths. But I > think it hides a more fundamental issue: you need to find out context. > Once you do, relativeness no longer matters (obviously). A system which > has a current directory and ".." is equivalent to one which has no such > thing and passes a "starting directory" in the environment, say. > > I think a system with less global state is more modular -> preferable. Agreed. I would much prefer imputil.py to become a standard -- then I could add my hook into its chain and wouldn't have to argue for using relative imports ;-) But then, how would I tell if someone else already has integrated such a hook in the chain ? I sure wouldn't want to add another one just to make sure... If the standard mechanism already knows about '__' then I wouldn't have to worry. > Another angle: I have built many types of tree structures, still do. > Less and less of those contain a "parent" link. Instead of storing a > parent you can just as easily keep state while descending into children. > > - Unix file info does not maintain a parent directory, it's redundant. > - Directories do, and file system mount points are messy because of it. > - Afaik, it's considered bad style to use "../header.h" in C includes. > - Upward pointers can introduce cycles (no not here, as has been said). > - In C++, member objects rarely need pointers to the enclosing object. Note that my patch does exactly this: it resolves the __ while descending into the package structure -- without any explicit back reference stored in the modules. > It is not for a module to know where it "is" in a hierarchy, it is for a > parent to (sometimes) provide that reference when it uses the module. Right. > If you want want access to a module called mx, and there may be many of > them - then you ask another module to locate it for you. One which is > able to choose / decide for you. Proximity may be a good discriminator, > but the decision of proximity was not taken by you - and you won't know > what rule is best. So you ask an "importer" (which could well always be > the standard Python mechanism). Given the task to write such as beast, > I'd probably want to implement the following module search as default: > - look for the module first in the parent (i.e. as sibling) > - move one level up, look again, etc > - all the way to the top > That's just one way to do it - proximity is not *always* what you want. Now this is an interesting idea... looks like acquisition at the module import level. Right now the scheme is as follows: - look for the module in the import module's dir - look for the module on the PYTHONPATH You would just add a third lookup step in between the two... it would cause many more module lookups though. > So if Zope chooses a different packaging style, let it override import. > But please don't build ".." into your modules, it doesn't belong there. > > Sorry for all the handwaving generalities. The issues are the same IMO, > whether relativeness is provided or context, and therefore the outcome > of this discussion will never be conclusive - they both work. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Tue Sep 14 10:21:45 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 11:21:45 +0200 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <37DE13A9.4150AF0E@lemburg.com> Tim Peters wrote: > > [MAL] > > I agree that it's ugly to include the __ attribute in the module > > namespace due to the possible circular reference (parent->module, > > module->parent), but the patch I sent doesn't do this... or was > > "ugly" referring to the two underscores looking strange ? > > > > Could you elaborate a bit on the reasons for dropping __ support ? > > There are two sections on why __ was dropped in > > http://www.python.org/doc/essays/packages.html > > They don't refer to circularity, but to "limited use", "poor readability" > and "awkwardness". Ok, so the two underscores look strange... that's arguable. "Limited use" is not: there clearly is a use for relative imports. Please remember that we are talking about package internals here -- not user interfaces of that package. > A deeper reason may be hiding in the essay's "most > packages will have a relative shallow substructure": this is Guido , > the man who invented two-level scoping, and class inheritance without a > "super" hook back to the (anonymous) parent. For all Python's dynamicism, > it very much favors shallow, static name hierarchies. I don't think it's > coincidence that Python's own source code is in a two-level directory > structure either! The only #include with a ".." is in grammar.h, and there > it's in a comment : > > #include "bitset.h" /* Sigh... */ > > So if we cut to the core here, I'd bet Guido doesn't object so much to > relative imports as to the idea that anyone would go off and create a > package structure so fractally convoluted that relative imports are strongly > more attractive than naming the target package in full. This is not the intention of relative imports. Its all about making packages relocatable. And that's a Good Thing IMHO. > Or maybe Guido doesn't care about that at all. I do regardless. I know > Python's restrictions can grate, but in all, and in my repeated experience, > they force you to rethink complicated designs and refactor them into simpler > schemes that fit what Python is best at spelling. Nesting packages 8 deep > is clumsy now? Damn straight, and I'm thankful for that: the clumsier it > is, the less gratuitous inherited complexity I'll have to deal with in my > future lives <0.5 wink>. Tim, I promise you not use 8 levels deep ;-) I usually create my packages using at most two levels (moving to at most three levels in future mx package releases) and even better: the common package interfaces are all available at the top-most level while internal stuff hides in higher levels. You don't have to think much about the internal structure of the package while you use it. That's something I have to worry about and the advantages of relative imports show up in such a context which is why I posted the patch. > Things that came up in this thread that are worth fixing include: > > + Problems with persistent class references (incl. pickles). > > + Dealing with incompatible versions of packages. If someone wants to embed > a copy of (say) mxDateTime in their own package, the only excuses are that > they're afraid of overwriting the user's existing mxDateTime installation > (if any), and/or of having some future installation of something else > overwrite mxDateTime with an incompatible version. Those are bad problems, > but package embedding is no solution. You have a much better approach to > that already via the DateTime.__version__ string! "Something like that" > needs to be formalized and imposed on all public packages. So the distutils will have to check for already installed package versions and warn about conflicts. Right, that's something which definitely needs to become a standard (too ;). > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim > > PS: > > > Just think of what happens to Win9x if you constantly update the DLLs... > > As a matter of personal experience, it gets much stabler! The older DLLs > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > installers that finally play by the rules. The mean time between crashes > when I installed Win95 a few years ago was about an hour; now it's at least > days and possibly weeks (don't know -- never leave the puter on that long). > When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. My experience is that intra-DLL references simply don't match anymore and cause the system to become instable. Also, some weird installers don't care about the version numbers and install older versions at their will. The outcome is a complete version mess. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm@hypernet.com Tue Sep 14 13:21:26 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Tue, 14 Sep 1999 08:21:26 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE0C30.4B0521B@lemburg.com> Message-ID: <1274810963-24736486@hypernet.com> M.-A. Lemburg wrote: 0000,7F00,0000> Agreed. I would much prefer imputil.py to become a standard -- then I > could add my hook into its chain and wouldn't have to argue for using > relative imports ;-) Hear hear! 0000,7F00,0000> But then, how would I tell if someone else 0000,7F00,0000> already has integrated such a hook in the chain ? I sure wouldn't want > to add another one just to make sure... Actually, imputil was designed to be used either generally or specifically. In other words, a package can install a link in the chain just for its contents. The hook can pass on the request as soon as it realizes that the request is not "his". The overhead involved is probably insignificantly different from having complex tests in a single generalized hook. - Gordon From jim@digicool.com Tue Sep 14 13:20:53 1999 From: jim@digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 12:20:53 +0000 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <37DE3DA4.3C7C14A8@digicool.com> Tim Peters wrote: > > [MAL] > > I agree that it's ugly to include the __ attribute in the module > > namespace due to the possible circular reference (parent->module, > > module->parent), but the patch I sent doesn't do this... or was > > "ugly" referring to the two underscores looking strange ? > > > > Could you elaborate a bit on the reasons for dropping __ support ? > > There are two sections on why __ was dropped in > > http://www.python.org/doc/essays/packages.html > > They don't refer to circularity, but to "limited use", "poor readability" > and "awkwardness". Which are all quite subjective. > A deeper reason may be hiding in the essay's "most > packages will have a relative shallow substructure": this is Guido , > the man who invented two-level scoping, and class inheritance without a > "super" hook back to the (anonymous) parent. For all Python's dynamicism, > it very much favors shallow, static name hierarchies. I don't think it's > coincidence that Python's own source code is in a two-level directory > structure either! The only #include with a ".." is in grammar.h, and there > it's in a comment : > > #include "bitset.h" /* Sigh... */ > > So if we cut to the core here, I'd bet Guido doesn't object so much to > relative imports as to the idea that anyone would go off and create a > package structure so fractally convoluted that relative imports are strongly > more attractive than naming the target package in full. > > Or maybe Guido doesn't care about that at all. I do regardless. I know > Python's restrictions can grate, but in all, and in my repeated experience, > they force you to rethink complicated designs and refactor them into simpler > schemes that fit what Python is best at spelling. Nesting packages 8 deep > is clumsy now? Damn straight, and I'm thankful for that: the clumsier it > is, the less gratuitous inherited complexity I'll have to deal with in my > future lives <0.5 wink>. Nobody has suggested building 8-level package hierarchies. In fact, the example I gave occured when a two-level package hierarchy was used in a two-level hierarchy. I don't think the Package structure of Zope is flawed *except* for the fact that it is one level too *shallow*. The ability to do relative imports would be very helpful for the work we're doing. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@interet.com Tue Sep 14 14:30:38 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 09:30:38 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> Message-ID: <37DE4DFE.FFD199ED@interet.com> "M.-A. Lemburg" wrote: > > Still it doesn't avoid possible conflicts > due to being top-level, e.g. the user could install an extension > which relies on a specific top-level name already taken by the hosting > package, e.g. the host defines a DateTime package and the > extension comes with its own DateTime package. This is the current > situation with Zope and mxODBC (which needs mxDateTime). Yes, the name conflict at the global level is exactly the problem. And to repeat, I don't really mind if "__" is added to imports, although I don't like the spelling. But... Perhaps a better solution is a multiple global name space. Suppose there is a command line option (or other mechanism such as a special file in sys.executable) which specifies sys.path for the application. To be specific, suppose zope is installed in the "zopehome" directory and the zope package lives in zopehome/zope. And suppose zope needs mx, so it installs it in zopehome/mx. This really is an obvious structure, and anyone could see that replacing mx/* would upgrade the mx package. The sys.path would be zopehome plus perhaps some subdirectories of zopehome. Doesn't this solve the problem? Why not? And don't complain about wasted disk space. The smallest disk you can buy today is 4 gigs and costs $150. I find the PYTHONPATH mechanism totally unreliable for commercial programs anyway. It is a global object, and an installation of a second Python program can break the first one. I don't think there is any solution to this other than specify sys.path on a per-application basis. If this is false, what is the other solution? Jim Ahlstrom From jim@digicool.com Tue Sep 14 13:42:10 1999 From: jim@digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 12:42:10 +0000 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> Message-ID: <37DE42A2.E27F7E55@digicool.com> "M.-A. Lemburg" wrote: > > James C. Ahlstrom wrote: > > > > First, I am not all that opposed to having a notion of ".." > > available in the import statement. If we can write zope.dir1.mod1 > > which is a relative import going down, maybe we can write > > ../dir2/mod2 or something spelled differently. But I think > > there would still be problems. > > > > We would be relying on all package authors to use ".." > > or "__" within their package. But it is more natural to write > > zope.this.that or mx.this.that everywhere, and that is what > > people have in fact done. Probably there would be bugs and > > at least annoyance. > > Think of it as an opportunity: it makes intra-package imports > very simple without hard-wiring paths to submodules > into the package. > > In the end, having written a self-contained package is a "feature" > of that package. Other authors can then say: ok, I can > plugin your package if it's self-contained. Well said. It really should be possible to provide *self-contained* package hierarchies. > I have a feeling that this discussion is loosing its grip > on reality a bit. Jim and I are not argueing to change > Python's face, internal structure or look&feel. The '__' thing > doesn't harm anyone, doesn't break any functionality or > code. It is even now already possible using import hooks and > has been implemented before in ni.py using just that technology. > > All that we are talking about is reviving relative imports > so that its usage is possible *without* adding yet another > hook. This is not about the pros and cons of deeply nested > packages and neither about advantages of shallow structures. > Its only about adding semantics to the string '__' used > in import statements. Nothing more. People are not forced > to use it. Besides, if used it will be a package internal > technique and not be visible to users of that package. > > Now why would anyone want to make life harder for package > authors ? I don't think nearly enough Python code has been packagized. Packages are critcal for large projects and for reuse of code within large projects. There are at least two people in this discussion that are trying hard to make use of packages and have observed a significant problem. If we really want to encourage developers of reusable Python software to use packages, then I think it would be wise to pay attention to people who are actively using packages and have encountered problems. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@interet.com Tue Sep 14 14:46:23 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 09:46:23 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE42A2.E27F7E55@digicool.com> Message-ID: <37DE51AF.67BB9CC0@interet.com> Jim Fulton wrote: > > "M.-A. Lemburg" wrote: > > In the end, having written a self-contained package is a "feature" > > of that package. Other authors can then say: ok, I can > > plugin your package if it's self-contained. > > Well said. It really should be possible to provide > *self-contained* package hierarchies. > There are at > least two people in this discussion that are trying hard to > make use of packages and have observed a significant problem. OK, just to make sure I understand this. You are saying that a ".." operation is needed in imports so that a package author with a package which looks like this: mypackage mypackage/sub1/mod1.py mypackage/sub2/mod2.py can write "import __.sub2.mod2" instead of "import mypackage.sub2.mod2" within the file mod1.py. Right? Jim Ahlstrom From jim@digicool.com Tue Sep 14 14:24:10 1999 From: jim@digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 13:24:10 +0000 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE42A2.E27F7E55@digicool.com> <37DE51AF.67BB9CC0@interet.com> Message-ID: <37DE4C7A.A553A8F2@digicool.com> "James C. Ahlstrom" wrote: > > Jim Fulton wrote: > > > > "M.-A. Lemburg" wrote: > > > In the end, having written a self-contained package is a "feature" > > > of that package. Other authors can then say: ok, I can > > > plugin your package if it's self-contained. > > > > Well said. It really should be possible to provide > > *self-contained* package hierarchies. > > > There are at > > least two people in this discussion that are trying hard to > > make use of packages and have observed a significant problem. > > OK, just to make sure I understand this. You are saying that > a ".." operation is needed in imports so that a package author > with a package which looks like this: > mypackage > mypackage/sub1/mod1.py > mypackage/sub2/mod2.py > > can write "import __.sub2.mod2" instead of "import mypackage.sub2.mod2" > within the file mod1.py. Right? Yup. Note that sys.modules would still have the absolute paths for keys. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward@cnri.reston.va.us Tue Sep 14 15:17:01 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 14 Sep 1999 10:17:01 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <000001befe7f$191104c0$fe2d153f@tim>; from Tim Peters on Tue, Sep 14, 1999 at 03:02:24AM -0400 References: <37DD0564.7361B789@lemburg.com> <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <19990914101700.A4401@cnri.reston.va.us> On 14 September 1999, Tim Peters said: > + Dealing with incompatible versions of packages. If someone wants to embed > a copy of (say) mxDateTime in their own package, the only excuses are that > they're afraid of overwriting the user's existing mxDateTime installation > (if any), and/or of having some future installation of something else > overwrite mxDateTime with an incompatible version. Those are bad problems, > but package embedding is no solution. You have a much better approach to > that already via the DateTime.__version__ string! "Something like that" > needs to be formalized and imposed on all public packages. > > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim Been there, tried that, bought the flame war. I made the mistake of kicking off the Distutils SIG back in Decemver with a proposal for a standard version numbering scheme for Python module distributions. See http://www.python.org/pipermail/distutils-sig/1998-December/000016.html for the kick-off of that "heated discussion". ;-) FWIW, if I was posting that message today, I would s/must/should/ and that's about it. And I would stress that these are "Proposed Recommended Guidelines" rather than "Things You Must Do or Distutils Will Slam the Door in Your Face", which is what it must have sounded like based on some of the responses. Folks: please don't reheat this flamewar on python-dev -- if you're interested in it, it will undoubtedly come up again on distutils-sig in due course! Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal@lemburg.com Tue Sep 14 15:24:06 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 16:24:06 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> Message-ID: <37DE5A86.4CD04049@lemburg.com> James C. Ahlstrom wrote: > > "M.-A. Lemburg" wrote: > > > > Still it doesn't avoid possible conflicts > > due to being top-level, e.g. the user could install an extension > > which relies on a specific top-level name already taken by the hosting > > package, e.g. the host defines a DateTime package and the > > extension comes with its own DateTime package. This is the current > > situation with Zope and mxODBC (which needs mxDateTime). > > Yes, the name conflict at the global level is exactly the problem. > And to repeat, I don't really mind if "__" is added to imports, > although I don't like the spelling. But... Doesn't look nice, but what other syntax would look better ? It will have to use identifiers and thus is restricted to [a-zA-Z0-9_]+. > Perhaps a better solution is a multiple global name space. Suppose > there is a command line option (or other mechanism such as a special > file in sys.executable) which specifies sys.path for the application. > To be specific, suppose zope is installed in the "zopehome" > directory and the zope package lives in zopehome/zope. And suppose > zope needs mx, so it installs it in zopehome/mx. This really is an > obvious structure, and anyone could see that replacing mx/* would > upgrade the mx package. The sys.path would be zopehome plus perhaps > some subdirectories of zopehome. Doesn't this solve the problem? > Why not? And don't complain about wasted disk space. The smallest > disk you can buy today is 4 gigs and costs $150. This would solve the problem for Zope (but only after I have restructured my packages to all go under the mx package -- which is what started this thread; it wouldn't be needed using relative imports). But what about other installations or tools like mxCrypto which plug into existing packages (Andrew's crypto package) ? Today such plugins only work side-by-side, ie. you have to install one part which sits in the plugin slot and one part which gets installed somewhere in a top-level PYTHONPATH dir. The plugin slot part then imports the Real Thing from outside the host package (this is how mxCrypto integrates itself with AMK's lib). > I find the PYTHONPATH mechanism totally unreliable for commercial > programs anyway. It is a global object, and an installation of > a second Python program can break the first one. I don't think > there is any solution to this other than specify sys.path on a > per-application basis. If this is false, what is the other > solution? The solution I'm using (and Zope is too) is to wrap a complete Python installation into the product, all setup to do the right thing. It makes the archive a little heavier... which is actually not so bad from a marketing POV. Bizarre, but people seem to want all those megs. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim@digicool.com Tue Sep 14 15:13:13 1999 From: jim@digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 14:13:13 +0000 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> <37DE13A9.4150AF0E@lemburg.com> Message-ID: <37DE57F9.9B3BFDC4@digicool.com> "M.-A. Lemburg" wrote: > > Tim Peters wrote: (snip) > > As a matter of personal experience, it gets much stabler! The older DLLs > > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > > installers that finally play by the rules. The mean time between crashes > > when I installed Win95 a few years ago was about an hour; now it's at least > > days and possibly weeks (don't know -- never leave the puter on that long). > > When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. > > My experience is that intra-DLL references simply don't match anymore > and cause the system to become instable. Also, some weird installers > don't care about the version numbers and install older versions at > their will. The outcome is a complete version mess. This has been my experince too. I cringe anytime I see some installer stuff DLLs in my system areas. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From skip@mojam.com (Skip Montanaro) Tue Sep 14 16:34:48 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 14 Sep 1999 10:34:48 -0500 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda Message-ID: <199909141534.KAA03648@dolphin.mojam.com> I haven't been paying real close attention to the relative package import thread on python-dev, but some way or other I strongly believe something has to be done to improve the situation I now find myself in or more and more people are going to start getting bitten by the same sort of problem. Allow me to 'splain. I use a web server based on Zope's ZServer (which is itself based on medusa) sitting behind an Apache web server as a long-running process to handle what many people traditionally used CGI for. It's a big performance boost over CGI. Unfortunately, I have recently been experiencing frequent server hangs. So far I've been unable to figure out what the cause is. I do notice occasional tracebacks like: Unhandled exception in thread: Traceback (innermost last): File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send _push_mode = 0 File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer self.ac_in_buffer = '' File "/usr/lib/python1.5/asynchat.py", line 253, in pop # this could maybe be made faster with a computed regex? IndexError: list index out of range and also notice that the server can pile up a huge number of connections in the ESTABLISHED state, at which point the whole mess grinds to a halt with not much computation or network traffic happening. A separate shell script uses netstat to detect when a large number of sockets have piled up and restarts the server. Brutal, but crudely effective. (When in doubt, treat the symptoms...) Today it just dawned on me looking at the above traceback that ZServer is getting the wrong version of asynchat.py (and presumably of asyncore.py as well). It's using the version delivered with Python 1.5.2 distribution instead of the version that was delivered with the version of Zope I'm using (1.something). So I started looking around at the versions and dates of various copies of asynchat.py. Here's what I found: source version number:date owner Python 1.5.2 1.2:1999/06/18 guido Python CVS 1.2:1999/06/18 guido Zope 1.? 1.7:1999/04/09 amos Zope 2.0 1.9:1999/07/19 amos Medusa 990902 2.24:1999/07/07 rushing What's apparently been happening is that people have picked up asyncore and asynchat at various time and stuck them in their own CVS repositories without somehow freezing the Id string of the version they originally got from Sam Rushing. It's not clear what the differences are until you compare the actual files. It turns out that the Zope 2.0 and Medusa versions have no content differences, only wildly different version numbers. The Medusa and Python CVS versions only have one difference: if index > 0: # don't bother reporting the empty string (source of subtle bugs) self.collect_incoming_data (self.ac_in_buffer[:index]) which *may* be what's causing my problems (note the IndexError in my traceback). Somewhere along the way I think we need to apply more restraint to our packaging and numbering of Python modules. I don't know what form that restraint will eventually take, but at this point I'm willing to replace Greg Ward's s/must/should/ with s/should/must/. a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Tue Sep 14 17:19:34 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Tue, 14 Sep 1999 12:19:34 -0400 (EDT) Subject: Path hacking [Long] (was Re: [Python-Dev] Relative Package Imports) References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> Message-ID: <14302.30102.360791.488002@anthem.cnri.reston.va.us> Finally, something I can relate to. Although I have a goal of packagizing everything I write these days, I haven't experienced any of the problems that lead others to suggest relative imports. The most complicated app that I hack on (continuously) is Mailman, which has a main package and several subpackages off the main one. I always use absolute paths in my import statements, so I don't see what the fuss is about. But I'm perfectly willing to admit that I don't have enough experience. However... >>>>> "JCA" == James C Ahlstrom writes: JCA> I find the PYTHONPATH mechanism totally unreliable for JCA> commercial programs anyway. It is a global object, and an JCA> installation of a second Python program can break the first JCA> one. I don't think there is any solution to this other than JCA> specify sys.path on a per-application basis. If this is JCA> false, what is the other solution? I completely agree with JimA here. It's been a pain with the Knowbot stuff, a pain with Mailman, and a pain with other packages that I've installed for shared use within CNRI. The .pth files solve part of the problem nicely. They let me install, say PIL or PCT in a shared location, for access by all the Python users at my site, without the users having to individually hack their dot-files, etc. But this doesn't work so well for apps like Mailman or the Knowbot stuff because we can't expect that the person installing those applications will be able to install a .pth file in the right place. Also, .pth files don't let you tightly control sys.path, e.g. you can only add paths, not delete or reorder them. Plus you have a global naming problem. Mailman's top level package is called "Mailman", so I can be fairly confident that I'm not going to collide, but it means that I have an extra directory level within my install that contains all the core importable modules. I don't think that's a big deal, but it's a convention that other packaged app writers should follow. The problem is getting Mailman's (or the Knowbots') top level directory on sys.path, and in exactly controlling the contents of sys.path. Our first approach with Knowbots was to do direct sys.path.insert()s, which is quite ugly and error prone. Plus if you're adding many paths, or adding and deleting, that's a lot of gibberish at the top of your entry level executables. And now let's say that you have a dozen or two dozen entry level executables that all have to perform the same sys.path magic. That's a lot of cutting-and-pasting (and /highly/ error prone patching when directory structures change). It's a lose. So for Knowbots we wrote a small module called pathhack that all entry level executables imported. pathhack was good because it put all that sys.path munging nonsense in one place so it was manageable from a s/w engineering standpoint. But it sucked because those executables had to /find/ pathhack.py! Bootstrap lossage (we've actually gone back to sys.path.insert). With Mailman, I could solve that problem because I added a configure/make phase. This let me write a module template called paths.py.in which configure flippered into paths.py containing path hackage based on --prefix. The next trick was that "make install" copied that paths.py file into all the subdirectories that had top level entry points into the Mailman system (e.g. the bin directory, the cron directory, the cgi directory). So now, an executable need only do import paths import Mailman.Utils import Mailman.Logging.Utils and absolute paths work like a charm. I can even provide a `pythonlib' directory that contains newer versions of standard modules that have fixes for folks running older Pythons. Thus I do from Mailman.pythonlib import rfc822 and the rest of my code uses my special rfc822 module with no changes. I'm very happy with how this works for Mailman, however we can't use the same approach (or let's say Guido doesn't want to use this approach) for the Knowbots stuff because there /is/ no "make install" step. You just unpack it and go. But it still has to play lots of games searching the file system for various things. What I've been thinking is that Python needs a registry . JPython's already got such a beast, and it integrates with Java's system properties, so that things like the PYTHONPATH equivalent are set in the registry and immediately available. But it's not very flexible, and you still need an install step in order to bootstrap the locating of the registry. I think we can do a little bit better. Python already knows how to find it's sys module. We can add an object into sys, call it sys.registry, which would contain things like sys.path definitions, and all sorts of other application specific keys. This object would be tied to a file (or files) which might be human readable, a marshal/pickle (or both). Bootstrap location of this file(s) is an issue, but see below. This would let you do things like the following at the beginning of every top level executable: import sys sys.application = 'zope' sys.registry.setpath(sys.application+'.pythonpath') I'm sure all kinds of lengthy discussion will now ensue about the exact interface of the registry object, but I'll make just a few observations: - There should be a system wide registry and a user specific registry. This let's an admin install shared applications easily, but also lets individual users have their own overrides. - The system-wide registry can be located in say sys.prefix/lib/python/site-packages. The user registry would reside somewhere in $HOME. This could all be platform specific so that on Windows, maybe the Python registry is integrated with the Windows registry, while in JPython it would be integrated with the standard JPython registry mechanism. - You should be able to specify registry entries on the command line. - There needs to be defined rules for resolving registry keys b/w system, user, and command line specifications. JPython has some experience here (although there have been requests to change JPython's lookup order), and at the very least, JPython and CPython should be as consistent as possible (CPython won't have to merge in Java's system properties). - The sys.registry object should be read/writable. This would let an install script do something like: import sys sys.registry.lock() sys.registry.put('zope.pythonpath', '@prefix@:@prefix@/matools:@prefix@/pythonlib') sys.registry.write() sys.registry.unlock() which would write either the global system registry or the local user registry, depending on permissions (or maybe that's spelled explicitly in the API). - In a sense you're pushing the namespace issue up a level into the registry, but at least this is a domain we can completely control from Python; it abstracts away the file system, and I don't think there's any way to avoid requiring conventions and cooperation for registry key naming. I also don't think it'll be a big problem in practice. When I packagize and re-release my Zarathustra's Ocular Python Experience virtual reality system, I'll try to think of a non-colliding top level package name. - (oh darn, I know I had more points, but Guido just popped in and I lost my train of thought). Well, this has gone on long enough so I might as well let you guys shoot this idea all to hell. Let me close by saying that while I think the Windows registry is a mess, I also think that it might be useful for Python. Does it solve the same problem that the relative imports is trying to solve? I dunno, but that's why I changed the Subject: line above. :) -Barry From gmcm@hypernet.com Tue Sep 14 17:47:25 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Tue, 14 Sep 1999 12:47:25 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE5A86.4CD04049@lemburg.com> Message-ID: <1274795004-25696413@hypernet.com> [MA Lemburg] > The solution I'm using (and Zope is too) is to wrap a > complete Python installation into the product, all setup to > do the right thing. It makes the archive a little heavier... > which is actually not so bad from a marketing POV. Bizarre, but > people seem to want all those megs. This is fine if you're distributing an app - it's a black box, it's standalone, no dependencies outside the app. Cool. I do that too. I use imputil to serve modules out of archives - no need to tweak the modules at all. In the upcoming version (of my installer) sys.path has exactly one entry, and the only python that needs to be outside an archive is exceptions.pyc. (I repeat here: while the pieces of my installer that handle binary dependencies are Win32 only, the parts that handle pure Python are cross-platform.) But both you and Jim are aiming for this non-existant in- between space - in some respects you want a "normal" Python installation, but you want to black-box your turf. I'm sorry, but if you're going to fit into a Python installation, you should make public your dependencies. Yes, installation becomes more complicated, but you're dealing with *Python users*. They're at least as likely to get screwed by having multiple different copies of the same thing around as you are in having dependencies outside of your personal control. If you also want to distribute as a standalone, freeze/squeeze/ whatever it. Just make sure that what you distribute this way won't interfere (or be confounded by) any existing Python installations. - Gordon From gstein@lyra.org Tue Sep 14 18:23:36 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 10:23:36 -0700 (PDT) Subject: [Python-Dev] revised import code (was: Relative Package Imports) In-Reply-To: <37DCCF3B.2F1E962D@lemburg.com> Message-ID: On Mon, 13 Sep 1999, M.-A. Lemburg wrote: > Gordon McMillan wrote: > > But I strongly believe that import.c should be left alone, maybe > > even to die. There are too many people doing import hooks to > > make fiddling with its behavior safe. > > > > I'm also a strong proponent of Greg's imputil.py scheme, which > > makes it a breeze to do import hooks. And my experience > > disproves the notion that the import mechanism needs to be in C. If > > you don't believe me, try the ZlibArchive stuff (which is cross > > platform) from my Win32 installer stuff. You can pack the standard > > library into one 475K file, and get a perceptible performance boost. > > You're probably right in saying that we don't need the code in C. > I just wanted to avoid yet another import hook being incompatible > with all the other existing hooks. > > Perhaps we should restart the import discussion all over and > come up with a more flexbile 100% compatible framework based > on Greg's imputil scheme. Then I could add my hook for the relative > imports and be happy ;-) I'll code up some patches to strip down import.c and replace with imputil.py and some bootstrap/support C code -- as soon as 1.6 development opens up. Guido? Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim@interet.com Tue Sep 14 18:36:03 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:36:03 -0400 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> <37DE13A9.4150AF0E@lemburg.com> <37DE57F9.9B3BFDC4@digicool.com> Message-ID: <37DE8783.2B8D6455@interet.com> Jim Fulton wrote: > > "M.-A. Lemburg" wrote: > > > > Tim Peters wrote: > (snip) > > > As a matter of personal experience, it gets much stabler! The older DLLs > > > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > > > > My experience is that intra-DLL references simply don't match anymore > > and cause the system to become instable. Also, some weird installers > > don't care about the version numbers and install older versions at > > their will. The outcome is a complete version mess. > > This has been my experince too. I cringe anytime I see some > installer stuff DLLs in my system areas. My experience has been the same as Tim's. You should generally upgrade DLL's as a matter of course, because your system will be more stable. Installers are required to check DLL versions before replacing them. Not doing this is a major sin, and reason enough to stop dealing with a vendor. Jim Ahlstrom From jim@interet.com Tue Sep 14 18:49:41 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:49:41 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <37DE5A86.4CD04049@lemburg.com> Message-ID: <37DE8AB5.DDB8C432@interet.com> "M.-A. Lemburg" wrote: > Doesn't look nice, but what other syntax would look better ? It will > have to use identifiers and thus is restricted to [a-zA-Z0-9_]+. Maybe "__up__" or "__up" following the convention that double-under names are special? Just a thought... > But what about other installations or tools like mxCrypto > which plug into existing packages (Andrew's crypto package) ? Today > such plugins only work side-by-side, ie. you have to install one > part which sits in the plugin slot and one part which gets installed > somewhere in a top-level PYTHONPATH dir. The plugin slot part then > imports the Real Thing from outside the host package (this is how > mxCrypto integrates itself with AMK's lib). I am not familiar with this. Dosen't this work?: zopehome/part_in_top_level_slot.py zopemome/plugin_slot/part_in_plugin_slot.py > The solution I'm using (and Zope is too) is to wrap a > complete Python installation into the product, all setup to > do the right thing. It makes the archive a little heavier... > which is actually not so bad from a marketing POV. Bizarre, > but people seem to want all those megs. For the record, I distribute a hacked main program which sets its own sys.path. I think your and zope's solution may break if another Python program is installed, and such program changes PYTHONPATH and/or the registry. That is why I never change PYTHONPATH nor the registry. Jim Ahlstrom From jim@interet.com Tue Sep 14 18:59:03 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:59:03 -0400 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <37DE8CE7.F211F6FA@interet.com> Skip Montanaro wrote: > So I started looking around at the versions and dates of various copies of > asynchat.py. >... > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have Yes, I have had this happen too. I am unwilling and unable to risk this sort of problem at a customer's site. So I ship a complete app with no external dependencies. Also crude but effective. Jim Ahlstrom From gstein@lyra.org Tue Sep 14 19:55:47 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 11:55:47 -0700 (PDT) Subject: [Python-Dev] relative package imports, version , numbering, yadda yadda In-Reply-To: <37DE8CE7.F211F6FA@interet.com> Message-ID: On Tue, 14 Sep 1999, James C. Ahlstrom wrote: > Skip Montanaro wrote: > > So I started looking around at the versions and dates of various copies of > > asynchat.py. > >... > > What's apparently been happening is that people have picked up asyncore and > > asynchat at various time and stuck them in their own CVS repositories > > without somehow freezing the Id string of the version they originally got > > from Sam Rushing. It's not clear what the differences are until you compare > > the actual files. It turns out that the Zope 2.0 and Medusa versions have > > Yes, I have had this happen too. I am unwilling and unable to risk > this sort of problem at a customer's site. So I ship a complete > app with no external dependencies. Also crude but effective. We did the same with the Python-based apps/servers at Microsoft. Merchant Server was a big frozen app (based on the non-sensical requirement to hide the fact that Python was used). In Site Server 2.0 and 3.0, we used a mini-install -- just the Lib files we needed plus our extensions. In the Site Server (non-frozen) case, we did use the registry, but built the interpreter with a custom "version". SS20 and SS30. In the registry, this meant we used Python/PythonCore/SS20 (I think that's the layout). Worked great, no complaints. Oh, and the DLLs we put into the system directory (pretty necessary for COM) were named SS20.DLL to prevent conflicts. I'm not sure who said it, but I agree with the following statement: * ship your app as a complete black box, or ship your app with dependencies on modules/packages [at the top level] This monkeying around with "mx.foo" working where mx is at the top level or is embedded is just scary. That said, personally, I would just do something like the following at the startup of my app: ZopeImporter("zopedir").install() [where ZopeImporter is a imputil.Importer subclass] The importer would just Do The Right Thing for all imports, and only defer to the Python library for things that weren't shipped with Zope (the empty set?) Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim@interet.com Tue Sep 14 20:11:54 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 15:11:54 -0400 Subject: Path hacking [Long] (was Re: [Python-Dev] Relative Package Imports) References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <14302.30102.360791.488002@anthem.cnri.reston.va.us> Message-ID: <37DE9DFA.269967B5@interet.com> "Barry A. Warsaw" wrote: > > So for Knowbots we wrote a small module called pathhack that all entry > level executables imported. pathhack was good because it put all that > sys.path munging nonsense in one place so it was manageable from a s/w > engineering standpoint. But it sucked because those executables had > to /find/ pathhack.py! Bootstrap lossage (we've actually gone back to > sys.path.insert). Yes, exactly the problem I had, bootstraping the import of pathhack. Actually it gets worse because Python imports exceptions.py, site.py and sitecustomize.py during Py_Initialize(), so if you are having a really bad day, you might pick up the wrong version of those. AFAIK, the only way to solve that currently is to use freeze to build pathhack into the binary executable. That is what I do anyway. But it is not an ideal solution. > What I've been thinking is that Python needs a registry . Yikes! As you say, Window's registry is a mess. > [Lots of good ideas omitted...] > - The system-wide registry can be located in say > sys.prefix/lib/python/site-packages. The user registry > would reside somewhere in $HOME. This could all be platform > specific so that on Windows, maybe the Python registry is integrated > with the Windows registry, while in JPython it would be integrated > with the standard JPython registry mechanism. Python already has three directories it knows about: sys.executable is the directory of the interpreter binary, sys.dllfullpath could be the directory of the interpreter as a shared library (I have a patch for this), and there is the directory of the main Python program as given on the command line. Perhaps we can put the registry in one of these directories. That would be consistent on all platforms. > - You should be able to specify registry entries on the command line. This is vital because I am worried about a bad registry. > - There needs to be defined rules for resolving registry keys b/w > system, user, and command line specifications. JPython has some > experience here (although there have been requests to change I am not sure a full registry is required. Once you can control sys.path and can get an accurate import of sitecustomize.py, you can do everything else there. Maybe just a command line option is enough. But I will think about it... Jim Ahlstrom From jim@interet.com Tue Sep 14 20:37:11 1999 From: jim@interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 15:37:11 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274795004-25696413@hypernet.com> Message-ID: <37DEA3E7.389CE@interet.com> Gordon McMillan wrote: > > But both you and Jim are aiming for this non-existant in- > between space - in some respects you want a "normal" > Python installation, but you want to black-box your turf. I don't really have much choice here. For some purposes, PYTHONPATH is better than a black-box. It is better for development because it avoids creating the black-box. And I can look at and modify Python app and library files directly. But for a customer install I need a bullet-proof dumb-simple set of required state which I can explain to customer support staff. Thus the need for both. > I'm sorry, but if you're going to fit into a Python installation, > you should make public your dependencies. Yes, installation > becomes more complicated, but you're dealing with *Python > users*. They're at least as likely to get screwed by having > multiple different copies of the same thing around as you are > in having dependencies outside of your personal control. I can't fit into a Python installation because Python installations do not support commercial software concerns. I am not dealing with Python users, my customers are office workers. They don't know what Python is, have no other Python apps, and if they did, my install wouldn't hurt them anyway. > If you also want to distribute as a standalone, freeze/squeeze/ > whatever it. Just make sure that what you distribute this > way won't interfere (or be confounded by) any existing Python > installations. Yes, exactly my goal. Jim Ahlstrom From guido@CNRI.Reston.VA.US Tue Sep 14 20:57:51 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 15:57:51 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Tue, 14 Sep 1999 12:19:34 EDT." <14302.30102.360791.488002@anthem.cnri.reston.va.us> References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <14302.30102.360791.488002@anthem.cnri.reston.va.us> Message-ID: <199909141957.PAA15470@eric.cnri.reston.va.us> I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@CNRI.Reston.VA.US Tue Sep 14 21:09:07 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 16:09:07 -0400 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda In-Reply-To: Your message of "Tue, 14 Sep 1999 10:34:48 CDT." <199909141534.KAA03648@dolphin.mojam.com> References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <199909142009.QAA15534@eric.cnri.reston.va.us> > So I started looking around at the versions and dates of various copies of > asynchat.py. Here's what I found: > > source version number:date owner > Python 1.5.2 1.2:1999/06/18 guido > Python CVS 1.2:1999/06/18 guido > Zope 1.? 1.7:1999/04/09 amos > Zope 2.0 1.9:1999/07/19 amos > Medusa 990902 2.24:1999/07/07 rushing > > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have > no content differences, only wildly different version numbers. The Medusa > and Python CVS versions only have one difference: > > if index > 0: > # don't bother reporting the empty string (source of subtle bugs) > self.collect_incoming_data (self.ac_in_buffer[:index]) > > which *may* be what's causing my problems (note the IndexError in my > traceback). My bad. I despise putting RCS identifiers in code I release, but I don't always freeze them before incorporating other people's code in my CVS tree. I will fix this if I can and I promise to try not to repeat this mistake in the future. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Tue Sep 14 21:57:19 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 22:57:19 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274795004-25696413@hypernet.com> Message-ID: <37DEB6AF.53974AAB@lemburg.com> Gordon McMillan wrote: > > [MA Lemburg] > > The solution I'm using (and Zope is too) is to wrap a > > complete Python installation into the product, all setup to > > do the right thing. It makes the archive a little heavier... > > which is actually not so bad from a marketing POV. Bizarre, but > > people seem to want all those megs. > > This is fine if you're distributing an app - it's a black box, it's > standalone, no dependencies outside the app. Cool. I do that > too. I use imputil to serve modules out of archives - no need to > tweak the modules at all. In the upcoming version (of my > installer) sys.path has exactly one entry, and the only python > that needs to be outside an archive is exceptions.pyc. > > (I repeat here: while the pieces of my installer that handle > binary dependencies are Win32 only, the parts that handle > pure Python are cross-platform.) > > But both you and Jim are aiming for this non-existant in- > between space - in some respects you want a "normal" > Python installation, but you want to black-box your turf. Actually, I want my stuff to be as flexible as possible. Of course I'm using it in my product too, but that's a different story. I have the unsatisfying impression that this discussion doesn't lead anywhere. I can't really understand all the evil sides Tim and Guido seem to see in their views of relative imports. For people like Jim and me, who have quite some experience in doing Python packages, this small (! hey, it's only about 20 lines of code !) additional feature could be of great use. I'm still waiting for some ultimate argument that blows relative imports away. The arguments put forward so far have all been in the category "evil", "ugly", "I don't like it". Hey, we can do better than that... Note that if this doesn't work out, Jim and I could always agree on some standard import hook that we'd both use, but that's exactly what we would very much like to avoid in favour of some established standard hard-wired into the Python distribution. > I'm sorry, but if you're going to fit into a Python installation, > you should make public your dependencies. Yes, installation > becomes more complicated, but you're dealing with *Python > users*. They're at least as likely to get screwed by having > multiple different copies of the same thing around as you are > in having dependencies outside of your personal control. I am probably going to ship all the mx* stuff in one big package -- not as separate packages anymore. That way I can write code which depends on other parts of the mx universe more easily. > If you also want to distribute as a standalone, freeze/squeeze/ > whatever it. Just make sure that what you distribute this > way won't interfere (or be confounded by) any existing Python > installations. It won't interfere with anything since the Python interpreter I use is configured *not* too look in any standard places for extensions and the like. Plus it's configured to run faster, but that's a different story ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido@CNRI.Reston.VA.US Tue Sep 14 22:09:53 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 17:09:53 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 11:21:15 EDT." <37DD166B.746B19AD@digicool.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> Message-ID: <199909142109.RAA15752@eric.cnri.reston.va.us> Jim Fulton wrote: > I wouldn't be in favor of making it more complicated if there wasn't > a good reason. I think that, in working on the Zope framework, > I've found some pretty good reasons for relative imports. And in a later message: > I don't think the Package structure of Zope is flawed *except* > for the fact that it is one level too *shallow*. The ability to > do relative imports would be very helpful for the work we're doing. But I haven't seen explained what it is that Zope is doing where relative packages would be helpful. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@CNRI.Reston.VA.US Tue Sep 14 23:47:18 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 18:47:18 -0400 Subject: [Python-Dev] License cleanup Message-ID: <199909142247.SAA16609@eric.cnri.reston.va.us> After more than four years of living with an out-of-dated license for Python, CNRI has finally agreed to clean up Python's copyright status. I expect that this won't have any real effect before Python 1.6 is released, but I am required to start preparing for the transition now. We will use a new license (a clone of the JPython license) and we will require that all contributors explicitly allow us the use of their contribution: either a few email paragraphs in an email message, or a longer form with a wet signature, depending on the size of the contribution. I believe the text of the license and forms we use is quite uncontroversial; these very same words have been used for JPython for quite a while. The words are all on the web: http://www.python.org/1.5/pylicense.html [proposed license] http://www.python.org/1.5/bugrelease.html [email release] http://www.python.org/1.5/wetsign.html [wet signature release] If you are reading python-dev but you never contributed any code to Python, you can stop reading now. If you *did* contribute code to Python, however, I'd love it if you saved me some work and filled out the wet signature form and mailed it to me at the given address. If you need help jogging your memory what your contributions were, send me email; I can try grepping the CVS files for your name. If you believe that special circumstances exist that make it impossible or difficult for you to sign the form, please send me email, and we'll discuss the matter. If you contributed something and I don't hear from you, you will eventually hear from me again -- but I hope I can save myself the hassle of writing each of you through this mass mailing. Thanks in advance! --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein@lyra.org Wed Sep 15 00:08:32 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 16:08:32 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: Actually, I don't like them all that much :-( [I don't recall any specific discussion about it, but I may have missed it and/or simply because I've never used JPython.] The BSD-ish license that Python has always used is much more preferable. I dislike the regulation of the "Python" name, the requirement to prominently discuss modifications made, and the revocation clause. I might find other items, but that is from a quick read using Lynx on a tiny monitor... Heck, how could people like PPSI, PythonWare, or D.C. truely like that license? Each of those companies uses "Python" significantly in their marketing and their business. I can certainly state that PPSI will never do anything in an official capacity to recognize that license. [there is a separate issue of whether "Python" can be trademarked, but the license does use the term "trade name" which could easily be argued to include the term "Python" and thus subject the name to the license.] > If you *did* contribute code to Python, however, I'd love it if you > saved me some work and filled out the wet signature form and mailed it > to me at the given address. No problem. Future contributions and agreemend to abide by that license are a different issue. It doesn't have the "feels good" feeling that the old license does. I'm not sure that bodes well, and it doesn't sit well with me at the moment. Regards, -g -- Greg Stein, http://www.lyra.org/ From guido@CNRI.Reston.VA.US Wed Sep 15 00:31:47 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 19:31:47 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Tue, 14 Sep 1999 16:08:32 PDT." References: Message-ID: <199909142331.TAA16699@eric.cnri.reston.va.us> > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. I might > find other items, but that is from a quick read using Lynx on a tiny > monitor... Hm... We may have to review the regulation of the Python name. This made sense in the context of the previous uses of this license (JPython and Grail) but Python is a different thing -- the name Python stands for more than just the implementation. I'll discuss this with CNRI's legal team. I don't see how the other things you mention can be much of a problem (most Open Source licenses have a revocation clause these days, I think, and I don't see how discussing the modifications made can be a problem with open source users). > Heck, how could people like PPSI, PythonWare, or D.C. truely like that > license? Each of those companies uses "Python" significantly in their > marketing and their business. I can certainly state that PPSI will never > do anything in an official capacity to recognize that license. How can you say that without consulting with the board? And I am *on* that board! I despise your attitude. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Wed Sep 15 00:44:54 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Tue, 14 Sep 1999 19:44:54 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <14302.56822.478320.862970@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Heck, how could people like PPSI, PythonWare, or D.C. truely GS> like that license? Each of those companies uses "Python" GS> significantly in their marketing and their business. Data point: I know that there are a number of companies that have embedded JPython in their commercial products. So far I've had zero complaints from them on the JPython license. -Barry From gstein@lyra.org Wed Sep 15 01:11:56 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:11:56 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142331.TAA16699@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > Hm... We may have to review the regulation of the Python name. This > made sense in the context of the previous uses of this license > (JPython and Grail) but Python is a different thing -- the name Python > stands for more than just the implementation. I'll discuss this with > CNRI's legal team. Cool. > I don't see how the other things you mention can be much of a problem > (most Open Source licenses have a revocation clause these days, I > think, and I don't see how discussing the modifications made can be a > problem with open source users). I'll do some more reading. As I said: that was my first cut. The revocation clause doesn't sit well with me. Maybe other OSS packages have it, but I believe that is usually because the license was developed by a company and its legal team. I don't think the GPL, BSD, MPL, and Apache licenses have revocation clauses, and I consider those to be the "most open" types of licenses (MPL less so). The Python 1.5 license is just as open, more so than most. > > Heck, how could people like PPSI, PythonWare, or D.C. truely like that > > license? Each of those companies uses "Python" significantly in their > > marketing and their business. I can certainly state that PPSI will never > > do anything in an official capacity to recognize that license. > > How can you say that without consulting with the board? And I am *on* > that board! I despise your attitude. Because the President (me) runs the day-to-day operation and direction of the company. The Board advises. The Board typically has other duties such as replacing me :-), handling stock issues, etc, but the Board is typically not involved with most issues. This is standard practice for corporate organization. Therefore, I *can* make that choice, and even do it unilaterally if I wanted to be an ass about it. Will I refuse to listen to the board or the shareholders or the employees? Of course I'll listen. [further PPSI issues should be taken offline] Regardless: it boils down to the "Python" requirement in that license. PPSI simply cannot operate under that license. If it gets dropped, then cool. -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Wed Sep 15 01:16:10 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:16:10 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Barry A. Warsaw wrote: > >>>>> "GS" == Greg Stein writes: > GS> Heck, how could people like PPSI, PythonWare, or D.C. truely > GS> like that license? Each of those companies uses "Python" > GS> significantly in their marketing and their business. > > Data point: I know that there are a number of companies that have > embedded JPython in their commercial products. So far I've had zero > complaints from them on the JPython license. Are they using it in their marketing, or simply as an underlying driving force for their products? If they *are* using it in their marketing, then they have exposed themselves to a liability. According to the license that they are using, they are not allowed to use JPython in their marketing. If they do, then they are in breach of the license and it could be terminated on them. Their products could no longer include JPython and they'd be SOL. I would be interested to hear from somebody using JPython and marketing it and how they interpreted that license. Possibly I'm missing something, but that language seems pretty darn clear to me. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Wed Sep 15 01:24:25 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:24:25 -0700 (PDT) Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DEB6AF.53974AAB@lemburg.com> Message-ID: On Tue, 14 Sep 1999, M.-A. Lemburg wrote: >... > I have the unsatisfying impression that this discussion doesn't > lead anywhere. I can't really understand all the evil sides Tim and > Guido seem to see in their views of relative imports. For people > like Jim and me, who have quite some experience in doing > Python packages, this small (! hey, it's only about 20 lines of > code !) additional feature could be of great use. $| was probably a small addition to Perl, but I don't think you could then argue that it was a good thing to do. Small doesn't mean good. Any change "could be of great use" to *somebody*, but does it make it good for Python as a whole? Changes imply doc, maintenance, future compatibility, etc. > I'm still waiting for some ultimate argument that blows > relative imports away. The arguments put forward so far have > all been in the category "evil", "ugly", "I don't like it". > Hey, we can do better than that... People seem to be disagreeing with your stated requirement. In other words, they're saying that you shouldn't be attempting to make your package "portable across the module-import-space." I tend to agree. Fix it at one location. If an app doesn't like that, then they can individually compensate. I don't believe that Python's standard machinery is required to handle this particular notion of packaging. > Note that if this doesn't work out, Jim and I could always agree on > some standard import hook that we'd both use, but that's > exactly what we would very much like to avoid in favour of some > established standard hard-wired into the Python distribution. No reason the standard hook could not go into imputil.py. I believe people mostly care that it doesn't become *standard* or *default* behavior -- that it only is available when explicitly requested by an app. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond@skippinet.com.au Wed Sep 15 01:49:35 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 15 Sep 1999 10:49:35 +1000 Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <002e01beff14$306e5b00$0801a8c0@bobcat> > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: Obviously IANAL. However, this language does make me feel less comfortable than the existing one. The ability to terminate would appear an issue - it would seem to take a braver CEO to base their technology on Python with this hanging over them. Sure, it may rarely be invoked, but I certainly wouldnt want to fight it in court if it was. If I was writing in C, I could worst-case grudgingly accept needing to change compilers - but I dont have that luxury for Python. If my license was terminated, I have nowhere else to turn. It is a real shame when lawyers get so involved. Obviously Guido has no say in this, but IMO the ideal scenario would be to use the exsting language, but simply change the names and dates. Im guessing this would be unacceptable to CNRI. Being NAL, I suppose I have no choice other than to trust this licence. However, Im not looking forward to showing this licence to people as they are deciding if Python is the appropriate technology choice - to date, there has never been an issue - all they need to is not remove any copyright notice from the code (which is not actually seen in most apps) and add the copyright notice to the documentation. This new one seems much scarier to me.. Just my $200.00 worth (remember, we are talking lawyers fees here :-) I dont have a real concern as I dont understand the legal implications; just a slight uneasiness about it all...Not being controversial for the sake of it, just airing my possibly il-informed opinion - no opinions were solicitied, but that has never stopped me before :-) Of course, I will be sending my "wet" signature on the form. Im not sure what to put in the "contribution description" - maybe just "various small changes to the Windows port"?? I can't say Ive added entire modules, but my name appears against a number of small patches to a fairly large set of files... Mark. From tim_one@email.msn.com Wed Sep 15 06:19:48 1999 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 01:19:48 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <19990914101700.A4401@cnri.reston.va.us> Message-ID: <000301beff39$ee3015a0$612d153f@tim> [Tim, speaks of the devil ...] > "Something like that" [MAL's __version__ string] needs to be formalized > and imposed on all public packages. > > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim [... and Greg Ward of his legions appears!] > Been there, tried that, bought the flame war. I made the mistake of > kicking off the Distutils SIG back in Decemver with a proposal for a > standard version numbering scheme for Python module distributions. See > > http://www.python.org/pipermail/distutils-sig/1998-December/000016.html > > for the kick-off of that "heated discussion". ;-) Greg, if you call that a flame war, your credentials as an ex-Perl'er are in serious doubt . Except for the cowboy contingent, most participants were moving swiftly to consensus! > FWIW, if I was posting that message today, I would s/must/should/ and > that's about it. No, it's "must" or it's useless. What wasn't brought up in that thread is that the Distutil "version number" is an artficial construct created for the primary benefit of Distutil tools -- it needn't have anything whatsoever to do with whatever silly string the developer wants to *display* as being their "version number". It's instead a coordinate in an abstract but rigidly defined Distutil space, specifically designed to make programmatic navigation of that space reliable in a shared and uniform way. If a developer chooses, users need never be exposed to it. I'd use the x.y.z Distutil version number directly to keep my own life simpler, but if someone else wants to display a GUID followed by a 3-letter country code and the number of nanoseconds since the birth of Mohammed, fine -- they still have to map that to Distutil VN space internally or write their own stinkin' disttools. You may have went overboard on the *semantics* of the Distutil VN, though: its only real meaning is in what Distutil tools *do* with it. Fight this battle again. Without a uniform way for an installer to *know* when it's thought safe to replace a package with another version of that package, Python installations will never move beyond the similar hell of Windows 3.1. even-herds-of-cats-wear-collars-ly y'rs - tim From tim_one@email.msn.com Wed Sep 15 06:40:35 1999 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 01:40:35 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <000501beff3c$d58a8be0$612d153f@tim> I dislike the new license. Selling Python at work wasn't easy, but the short & straightforward CWI license went a *long* way toward convincing the suits there was little to worry about. The new license has several blobs of lawyer-speak that ensure the next battle will be much harder -- the prospect of license revocation, some fuzzy concept of derivative works, and vague "prominent display" requirements? Boston lawyers charge Really Big Bux to guess what that gibberish might mean in Virginia. The only bright side is that we now get explicit rights to "perform" and "display" Python . > If you believe that special circumstances exist that make it > impossible or difficult for you to sign the form, please send me > email, and we'll discuss the matter. It would less hassle for me if you took all my contributions out <0.9 wink>. i'll-sign-but-it's-really-really-really-depressing-ly y'rs - tim From tim_one@email.msn.com Wed Sep 15 07:42:02 1999 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 02:42:02 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE0825.10D2BC6F@lemburg.com> Message-ID: <000701beff45$6b51c5a0$612d153f@tim> Don't tell, but this is getting repetitive. One more batch of repeats from me, and I'm done: + The notion that Python packages aren't self-contained now is wrong: a package module can import other modules in the package without qualification. This is good design. + Where self-containment breaks down is in going across *sub*-packages. I was surprised Guido pretended to cater to them, since Python (wisely, IMO) stayed away from sub-modules from the start. Hierarchy is a great tool for managing complexity, but it also introduces complexity of its own. And when a flat package space suffices, hierarchy introduces artificial complications. + I agree with Gordon that, if anything, the default "import" behavior is already too complicated. By the time you wade thru PYTHONPATH, and PYTHONSTARTUP, and .pth files, and people mucking with sys.path, and site config files, and symlinks under Unix, and packages mucking with their own __path__ attributes, running Python with -v is the only hope of figuring out why nothing ever works <0.6 wink>. + All this dynamicism is in support of a concept that's actually static: the structure of an installation. As Barry laments, you can't always know the structure in advance of installation, but the solution is (I believe) along the lines he suggests: invent a way to register/query the structure once & for all after it is known. + JimF and MAL's shared notion that they're the only ones making significant use of packages is myopic. Dragon also uses packages, and heavily, although products using them have not been announced and I can't talk about them yet. Anything I'm within 10 miles of doesn't use *any* import hooks, or play any other tricks on the system. Instead I urge people to write their imports as explicitly, long-windedly and straightforwardly as possible (with binding to a convenient local alias name on the following line). Do we embed sub-packages? Sure. Do modules move around in the hierarchy over time? Of course. So how do we deal with this supposed nightmare? Because all imports are relentlessly straightforward (even intra-package imports are fully qualified), and none are hiding in dynamic functions, we get into Emacs and do a global search-and-replace. Takes 30 seconds, and the new structure is plain as day to everyone reading the code. low-tech-it's-not-just-for-breakfast-anymore-ly y'rs - tim From ping@lfw.org Wed Sep 15 08:31:29 1999 From: ping@lfw.org (ping@lfw.org) Date: Wed, 15 Sep 1999 00:31:29 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: > > http://www.python.org/1.5/pylicense.html [proposed license] > http://www.python.org/1.5/bugrelease.html [email release] > http://www.python.org/1.5/wetsign.html [wet signature release] Hi all. I'm sorry i haven't contributed anything to the relative-import and python-path discussions of late, but that's because so far i haven't had any ideas that have crossed my threshold of being sufficiently insightful to propose. I will follow the discussion with much interest. I'm afraid i have to say that the revocation clause makes me pretty uncomfortable. I know that it says CNRI will revoke only on a "material breach", but i still have a nasty suspicion that it sounds frightening enough to scare many people away. I don't think we want that. I suppose Greg's other points of contention are valid too but it's really the revocation that bugs me the most. -- ?!ng From fredrik@pythonware.com Wed Sep 15 08:42:28 1999 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 15 Sep 1999 09:42:28 +0200 Subject: [Python-Dev] License cleanup References: Message-ID: <008701beff4d$dcebfb10$f29b12c2@secret.pythonware.com> > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. same here. reading the new one made me feel very uneasy, but I cannot really say much about it before I've discussed it with people who know more about this... just a few small notes: the BSD-ish license used up to now has been a major selling argument for Python, while this one seems to really push the bounds of what qualifies as an open source license... (it also seems to imply that Python is a trademark, which is, as far as I can tell, is not true at this time. and archive corporation/seagate already owns the trademark wrt. software). the worst thing is that we will have to run this by our lawyers before we can decide whether to continue contributing to 1.6 development :-( From mal@lemburg.com Wed Sep 15 10:24:43 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 11:24:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <000701beff45$6b51c5a0$612d153f@tim> Message-ID: <37DF65DB.46F0191A@lemburg.com> Tim Peters wrote: > > + The notion that Python packages aren't self-contained now is wrong: a > package module can import other modules in the package without > qualification. This is good design. Agreed. And we would like to take that notion one step further -- without breaking the two-scope resolution Python uses now. The '__' tags are just place holders which could just as well be handled by a simple Python fuction (like the one Gordon posted) using perfectly valid technology. Sticking this technology into the import statement is really only a more elegant approach. Nothing more. The whole thing already works using the current Python implementation. > + JimF and MAL's shared notion that they're the only ones making significant > use of packages is myopic. Oh, c'mon Tim, we never did say that. We only mentioned having used package for quite a while. This includes having seen all the different pitfalls they have to offer. > Dragon also uses packages, and heavily, although > products using them have not been announced and I can't talk about them yet. > Anything I'm within 10 miles of doesn't use *any* import hooks, or play any > other tricks on the system. Instead I urge people to write their imports as > explicitly, long-windedly and straightforwardly as possible (with binding to > a convenient local alias name on the following line). Do we embed > sub-packages? Sure. Do modules move around in the hierarchy over time? Of > course. So how do we deal with this supposed nightmare? Because all > imports are relentlessly straightforward (even intra-package imports are > fully qualified), and none are hiding in dynamic functions, we get into > Emacs and do a global search-and-replace. Takes 30 seconds, and the new > structure is plain as day to everyone reading the code. This is perfectly ok if you're just using your own code, but it gets a mess when third-party packages are involved simply because you can't be sure they don't use import hooks, local imports or hacks to __path__ etc. If you still want them to be useable, you'll have to go down and dive into their structure. This takes time, is error prone and not necessarily fun. Also, I don't know how "explicitly, long-windedly and straightforwardly" writing module imports is any different from using relative imports. An example: mx .DateTime .ODBC.Windows In ODBC.Windows I would write: import __.__.DateTime which is just as explicit as writing import mx.DateTime except that the information about the top-level hierarchy is *not* included in the import information. Anyway, I'm not too inclined continuing this discussion anymore. I would never have thought that such a nifty little patch would have stirred up so much negative response. I'll just hack up my very own import mechanism using imputil and be done. So there! -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Wed Sep 15 11:05:27 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 12:05:27 +0200 Subject: [Python-Dev] License cleanup References: <008701beff4d$dcebfb10$f29b12c2@secret.pythonware.com> Message-ID: <37DF6F67.59FCEA6D@lemburg.com> Fredrik Lundh wrote: > > > The BSD-ish license that Python has always used is much more preferable. I > > dislike the regulation of the "Python" name, the requirement to > > prominently discuss modifications made, and the revocation clause. > > same here. reading the new one made me feel very > uneasy, but I cannot really say much about it before > I've discussed it with people who know more about > this... Dito. Some comments: """ 4.Licensee may not use CNRI trademarks or trade name, including Python or CNRI, in a trademark sense to endorse or promote products or services of Licensee, or any third party. Licensee may use the mark Python in connection with Licensee's derivative versions that are based on or incorporate the Software, but only in the form "Python-based ___________________," or equivalent. """ Say I want to sell Python 1.6 training, how would I promote this ? Since I'm not producing a derivative work, I guess I couldn't use the name 'Python' at all... hmm, I could probably try Pyth*n ;-) """ 3.In the event Licensee prepares a derivative work that is based on or incorporates the Software or any part thereof, and wants to make the derivative work available to the public as provided herein, then Licensee hereby agrees to indicate in any such work, in a prominently visible way, the nature of the modifications made to CNRI's Software. """ How explicit would that indication have to be ? E.g. do I have to provide a patch or would a simple run-down of new features suffice ? Needless to say, I would not be able to sell products based on Python 1.6 with the revocation clause in the license. In the end, I'd probably have to negotiate a separate license with CNRI not having this clause. Anything else would be unacceptable in a commercial setting. Is this intended ? And finally in the "Python Contribution Agreement": """ Licensee confirms to CNRI that, to the best of Licensee's knowledge and belief, the Contribution is free of any claims of parties other than Licensee under copyright, patent or other rights or interests ("claims"). """ Best knowledge and belief do not guard against law suit. Why doesn't this text protect the contributor in some way against charges forwarded by CNRI to the contributor ? (Note that the disclaimer in the Python License is not valid everywhere.) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From bwarsaw@python.org Wed Sep 15 12:56:48 1999 From: bwarsaw@python.org (Barry A. Warsaw) Date: Wed, 15 Sep 1999 07:56:48 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: <14303.35200.979238.868146@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Are they using it in their marketing, or simply as an GS> underlying driving force for their products? I'm not sure that JPython is much of a marketing advantage right now, so AFAIK none of them are actively promoting their use of JPython in their product. However, my reading of the second half of item 4 would allow them to say something like "You can even extend your flapjabs using our keen JPython-based scripting capabilities". GS> If they *are* using it in their marketing, then they have GS> exposed themselves to a liability. According to the license GS> that they are using, they are not allowed to use JPython in GS> their marketing. If they do, then they are in breach of the GS> license and it could be terminated on them. Their products GS> could no longer include JPython and they'd be SOL. I hope that wouldn't really be the case, but it's an interesting point, so I'm sure we'll bring it up. -Barry From jim@interet.com Wed Sep 15 13:20:54 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 08:20:54 -0400 Subject: [Python-Dev] License cleanup References: Message-ID: <37DF8F26.37BC155E@interet.com> Greg Stein wrote: > > On Tue, 14 Sep 1999, Guido van Rossum wrote: > > I believe the text of the license and forms we use is quite > > uncontroversial; these very same words have been used for JPython for > > quite a while. The words are all on the web: Oh boy, this is really going to cause trouble. Where's my flame suit... > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. I might > find other items, but that is from a quick read using Lynx on a tiny > monitor... I guess I am used to reading license agreements, and I am not very worried about the new one. Before we all get upset, lets remember that Guido works in a large company with lots of lawyers, and he trapped between a group of Internet geeks (hey, I like Internet geeks) and his buracracy. And remember that lawyers respond better to specific proposals for language changes than philosophical discussion. First off, the license is not revokable. It is only revokable on breach. If a license can not be revoked on breach it doesn't really mean anything. This is totally standard. Suppose someone else claims to own Python and starts selling "The True Standard Python" for $100. Suppose they change the standard library names so software only runs on their version. CNRI should be able to revoke their license to use Python. This is something we would all want CNRI to do. The protection of the Python name is a necessity. That is really all CNRI has, since the license gives away use of the software itself. If CNRI doesn't own "Python" then it can't object when someone else claims they own it. Don't we want them to object? The license doesn't say you can't use "Python", it sayes you can't use it in a trademark sense. I think that means you can say "I am teaching a course on Python, which is CNRI's software" but not "I am teaching a course on my Python, all rights reserved". Actually this is a little unclear, perhaps (4) could be made a little clearer. Paragraph (3) is a little troublesome. I seems to mean that if you ship a modified Python, you must say it is modified. I presume it doesn't mean that you must describe your own code in the event it incorporates Python. Really, we need to know what CNRI wants us to do here. On the contributions side (wetsign.html) it says you are contributing software free of third party claims "to the best of your knowledge and belief" not "represents and warrants" which is different. CNRI really has to be told that as far as you know, you didn't steal the software you are contributing. This is reasonable. Actually I might like to see a warranty disclaimer "NO WARRANTIES etc." like the license paragraph (5) and (6). I am not sure I need it since the contribution is free, but I usually ship free software with a disclaimer for "fitness for any particular purpose etc.". This is a pretty weak license agreement. Remember that if it is too weak, it prevents CNRI from defending Python against others who would claim they own it or who claim they are the true source of the language design (paranoia department: Microsoft's Python++). We want CNRI to defend Python, right? Jim Ahlstrom From gward@cnri.reston.va.us Wed Sep 15 13:30:32 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 15 Sep 1999 08:30:32 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <14302.56822.478320.862970@anthem.cnri.reston.va.us>; from Barry A. Warsaw on Tue, Sep 14, 1999 at 07:44:54PM -0400 References: <199909142247.SAA16609@eric.cnri.reston.va.us> <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: <19990915083032.A17433@cnri.reston.va.us> On 14 September 1999, Barry A. Warsaw said: > Data point: I know that there are a number of companies that have > embedded JPython in their commercial products. So far I've had zero > complaints from them on the JPython license. Just thought I should join the tide of opposition: heck, I *work* for CNRI and I still don't like the license. I didn't say much about the new JPython license because a) I trust Barry's judgement, b) it was certainly an improvement over the old JPython license, and c) I wasn't especially worried about one part of CNRI (Guido's group) taking JPython away from another part (the group that Andrew and I are on). However, that doesn't change the fact that the "new" license is a nasty piece of legalistic gibberish. Making it the license for Python 1.6 would be a major setback -- while it was better than the old JPython license, it's a damn sight worse than the old Python license. I have zero sympathy for the legal beagles here with their narrow corporatist viewpoint; trying to treat Python as just another potential piece of intellectual property is wrong-headed in the extreme. The free software world simply does not work that way. BTW, I suspect that the companies embedding JPython haven't minded the license because they come from the Java world, a world that seems to me to be dominated by corporate pin-headed thinking. The idea of community, openness, and sharing is utterly alien to these suit-wearing, smarmy Java frat-boy types, so JPython's licensing terms were probably a breath of fresh air to them. ("What? No $100,000 source license fee? Wow!") ("But wait Chip -- it's not BUZZWORD COMPLIANT! I can't find enough TLAs!!!") Hmmm, enough flaming Java weenies. Please, don't anybody take the last paragraphy too seriously or personally... Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fredrik@pythonware.com Wed Sep 15 13:39:30 1999 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 15 Sep 1999 14:39:30 +0200 Subject: [Python-Dev] License cleanup References: <37DF8F26.37BC155E@interet.com> Message-ID: <003e01beff77$5c87fe40$f29b12c2@secret.pythonware.com> > The license doesn't say you can't use "Python", it sayes you > can't use it in a trademark sense. quick check: which of these uses "Python" in a trademark sense, and thus violates the license: pythonware? professional python services? pythonworks? programming python? python training? python powered? the viper python implementation? python imaging library? wxpython? pythonwin? etc. all of them? none of them? From mal@lemburg.com Wed Sep 15 14:09:27 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 15:09:27 +0200 Subject: [Python-Dev] License cleanup References: <37DF8F26.37BC155E@interet.com> <003e01beff77$5c87fe40$f29b12c2@secret.pythonware.com> Message-ID: <37DF9A87.55AF6279@lemburg.com> Fredrik Lundh wrote: > > > The license doesn't say you can't use "Python", it sayes you > > can't use it in a trademark sense. > > quick check: which of these uses "Python" in a trademark > sense, and thus violates the license: > > pythonware? professional python services? pythonworks? > programming python? python training? python powered? > the viper python implementation? python imaging library? > wxpython? pythonwin? etc. > > all of them? none of them? Using a word in a trademark sense usually simply means using it in corporate relationships (at least that's how it works in Germany). If you are a company and talk about, write about or otherwise use the word in a commercial context then you are using the word in a trademark sense. There are several ways to declare a trademark, e.g. there are word marks, logo marks, sound marks, color marks etc. (don't know if these are the right translations). A word mark, for example, refers to a specific spelling of the word regardeless of the font, style or color. Logo marks refer to a specific design including font, style and color. Note that a trademark owner can still give you permission to use the mark in any decent way without paying fees or royalties. So even if CNRI does own the mark, they could still make it usable by others. In fact, if done right, this is a Good Thing. The answer to your question depends on what kind mark CNRI owns. [There currently is a very strong movement in Germany against people who are applying what they learned from domain grabbing to trademarks. Prominent examples include "WWW" and "Webspace". Even the color violet is trademarked (by a company producing chocolate)] -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim@interet.com Wed Sep 15 16:43:13 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 11:43:13 -0400 Subject: [Python-Dev] Re: Path hacking Message-ID: <37DFBE91.C93816F1@interet.com> "Guido van Rossum" wrote: > --> The solution: Ah, finally a specific proposal... > Suppose your application (as a whole, not the individual top-level > script) is called Spam -- this may well also be the name of your > top-level package. Then start each top-level script with the single > line > > import Spam_path > > before importing anything else. This should not be necessary if you use the name "sitecustomize" instead of "Spam_path" right? The file sitecustomize.py is automatically imported. Actually all this sounds like site.py all over again. > Your installer, once it knows the absolute pathname of your > application's root directory, crafts a file Spam_path.py which > contains code that inserts the right absolute pathname into sys.path. I don't think this is necessary either. The sys module is available. So sitecustomize.py can say: import sys mydir = sys.path[0] if not mydir: import os mydir = os.getcwd() sys.path = [mydir] # To be really extreme about it # Note: inserting mydir as sys.path[0] should be redundant but is not > Your installer then installs a copy of this file (or a symbolic link > to it) *in each bin directory where it installs top-level Python > scripts*. > > Because the script's directory is first on the default path, the Spam > scripts will pick up Spam_path without any help from $PYTHONPATH. Hmmm. Is this really true? Nothing else, for example the registry, can change sys.path[0]? Ever? Please say yes. > I know this doesn't resolve the relative import thread (how's that > going by the way? :-) but Barry & Fred & I agree that this is the best > solution to the problem stated in Barry's message to which I am > following up here. This is a good idea, but there are a few problems. It depends on sys.path[0] being the directory of the Python file being executed as the main program. I guess I never really trusted this before. I think if this is the case it should never be ''. A relative path or no path on the command line (the __main__ program) should be replaced by the full path in the sys module setup. Then the "mydir = os.getcwd()" above is not necessary. And inserting mydir as sys.path[0] is truly redundant should the current directory change (as it certainly will). This is currently a problem with sys.path[0] which should be fixed no matter what else happens. The files exceptions.py and site.py must be in all the bin directories as well as sitecustomize.py because they are automatically imported in Py_Initialize(). The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point. It seems to me this totally solves Jim Fulton's and Marc's problem and makes "__" unnecessary. You just install zope and mx in zopedir, perform the above, and presto you have a new private name space where you can control all your names. But there must be some problem here I haven't thought of. I still worry that this is not powerful enough. Greg Stein has volunteered to re-write import.c in Python (using imputil.py) and this is a Great Idea. Lots of Python could probably be written in itself. I would like to try writing the main program in Python and eliminating the special freeze main program. Once you start on this road (and I think it is a good road) you have Python code which is more truly part of the binary interpreter than a library. Proposal: Use a special PYTHONPATH on startup to find "special" Python files which are really part of the interpreter. There are three directories Python knows about. Namely sys.path[0] (once it is fixed), sys.executable and sys.dllfullpath, the directory of python15.dll or other shared library (once it is added to sys). How about prepending the single directory sys.executable to sys.path during Py_Initialize()? And demanding that modules like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] be placed there. Actually I would prefer sys.dllfullpath if it exists, since that is where the interpreter is, and I am trying to associate these special internal Python files exactly with their correct Python interpreter. Alternative Proposal: Py_Initialize() first imports its files from sys.executable + '/' + PyInternal.pyl (again I prefer sys.dllfullpath). PyInternal.pyl is a Python library file (like a Java Jar file) which would contain modules like exceptions, etc. The PyInternal.pyl file has the standard Python library file format (whatever that turns out to be). It is not an error if this file is absent. Jim Ahlstrom From guido@CNRI.Reston.VA.US Wed Sep 15 17:15:02 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 12:15:02 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 11:43:13 EDT." <37DFBE91.C93816F1@interet.com> References: <37DFBE91.C93816F1@interet.com> Message-ID: <199909151615.MAA17872@eric.cnri.reston.va.us> [Guido] > > import Spam_path > > > > before importing anything else. [JimA] > This should not be necessary if you use the name "sitecustomize" instead > of "Spam_path" right? The file sitecustomize.py is automatically > imported. > Actually all this sounds like site.py all over again. But the intention here is for the customization to be application specific (hence the Spam in the name). sitecustomize doesn't know whethere I need the Mailman or the Knowbot root added to my path. Or do you mean to imply that we can do this with zero text added to the script, by simply dropping an appropriate sitecustomize.py in the script dir? Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run. > > Your installer, once it knows the absolute pathname of your > > application's root directory, crafts a file Spam_path.py which > > contains code that inserts the right absolute pathname into sys.path. > > I don't think this is necessary either. The sys module is available. > So sitecustomize.py can say: > import sys > mydir = sys.path[0] > if not mydir: > import os > mydir = os.getcwd() > sys.path = [mydir] # To be really extreme about it > # Note: inserting mydir as sys.path[0] should be redundant but is not Hm, guessing based on the script directory might work, but seems less reliable than hardcoding it through the installer. But you can use this if it works for your application. > > Your installer then installs a copy of this file (or a symbolic link > > to it) *in each bin directory where it installs top-level Python > > scripts*. > > > > Because the script's directory is first on the default path, the Spam > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > Hmmm. Is this really true? Nothing else, for example the registry, can > change sys.path[0]? Ever? Please say yes. Yes. (The registry can add module-specific paths, which will be searched before sys.path is even looked at, but this is only for specific modules. It cannot insert a general directory that is searched.) The only way this can fail is if an embedding app fails to call PySys_SetArgv(). > > I know this doesn't resolve the relative import thread (how's that > > going by the way? :-) but Barry & Fred & I agree that this is the best > > solution to the problem stated in Barry's message to which I am > > following up here. > > This is a good idea, but there are a few problems. > > It depends on sys.path[0] being the directory of the Python > file being executed as the main program. I guess I never > really trusted this before. I think if this is the case it > should never be ''. A relative path or no path on the command > line (the __main__ program) should be replaced by the full path > in the sys module setup. Then the "mydir = os.getcwd()" above > is not necessary. And inserting mydir as sys.path[0] is truly > redundant should the current directory change (as it certainly will). > This is currently a problem with sys.path[0] which should be > fixed no matter what else happens. I have always resisted forcing path items to be absolute, although I'm not sure that my reasons are valid any more (it has to do with the fact that getcwd() may fail and the fact that portable path concatenation is a pain). In any case, that's a separate issue -- I agree that if sys.path[0] is '' (as it often is) it's better for site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize it (and everything else on the path) so that it will still work if the app does a chdir(). > The files exceptions.py and site.py must be in all the bin > directories as well as sitecustomize.py because they are > automatically imported in Py_Initialize(). Yes. > The above doesn't work when you start the Python command > interpreter (no main). I know, its a minor point. You could add the "import Spam_path" to your $PYTHONSTARTUP file. > It seems to me this totally solves Jim Fulton's and Marc's > problem and makes "__" unnecessary. You just install zope > and mx in zopedir, perform the above, and presto you have a new > private name space where you can control all your names. But > there must be some problem here I haven't thought of. I think no simple solution that *I* can come up with will satisfy JimF's and Marc's desire for obscurity :-) > I still worry that this is not powerful enough. Greg Stein > has volunteered to re-write import.c in Python (using imputil.py) > and this is a Great Idea. Lots of Python could probably be > written in itself. I would like to try writing the main > program in Python and eliminating the special freeze main > program. Once you start on this road (and I think it is a good road) > you have Python code which is more truly part of the binary > interpreter than a library. Yes, this is the plan for Python 2.0, and some of it may be implemented in Python 1.6. > Proposal: > > Use a special PYTHONPATH on startup to find "special" Python > files which are really part of the interpreter. There are > three directories Python knows about. Namely sys.path[0] > (once it is fixed), sys.executable and sys.dllfullpath, > the directory of python15.dll or other shared library (once it is > added to sys). How about prepending the single directory sys.executable > to sys.path during Py_Initialize()? And demanding that modules > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > be placed there. On Unix, this is a bin directory and it is strongly discouraged to put non-program files there. Python already does something similar -- it looks around in sys.executable's ancestors for a specific landmark, currently lib/python/string.py. Arguably, it should search for execeptions.py instead. > Actually I would prefer sys.dllfullpath if it exists, since that > is where the interpreter is, and I am trying to associate these > special internal Python files exactly with their correct Python > interpreter. Is the full DLL path available at any point? This would certainly be a good starting point -- especially when the DLL is loaded implicitly as the result of some COM operation. > Alternative Proposal: > > Py_Initialize() first imports its files from sys.executable + '/' + > PyInternal.pyl (again I prefer sys.dllfullpath). > PyInternal.pyl is a Python library file (like a Java Jar > file) which would contain modules like exceptions, etc. > The PyInternal.pyl file has the standard Python library file > format (whatever that turns out to be). It is not an error if > this file is absent. I guess this is all up to the redesign of the import mechanism (something like Greg Stein's imputil.py for sure). --Guido van Rossum (home page: http://www.python.org/~guido/) From da@ski.org Wed Sep 15 18:00:05 1999 From: da@ski.org (David Ascher) Date: Wed, 15 Sep 1999 10:00:05 -0700 (Pacific Daylight Time) Subject: [Python-Dev] License cleanup In-Reply-To: <37DF9A87.55AF6279@lemburg.com> Message-ID: Guido, maybe it would make sense to explain the need for a license change. Is my understanding correct that the occasion for the license change is that the copyright is now clearly shifting to CNRI, and as a result CNRI has to forge a license? (BTW, I thought *you* had the copyright transfer from CWI, not CNRI). --david From jim@digicool.com Wed Sep 15 19:10:17 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 14:10:17 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> Message-ID: <37DFE109.96C652B3@digicool.com> Guido van Rossum wrote: > > Jim Fulton wrote: > > > I wouldn't be in favor of making it more complicated if there wasn't > > a good reason. I think that, in working on the Zope framework, > > I've found some pretty good reasons for relative imports. > > And in a later message: > > > I don't think the Package structure of Zope is flawed *except* > > for the fact that it is one level too *shallow*. The ability to > > do relative imports would be very helpful for the work we're doing. > > But I haven't seen explained what it is that Zope is doing where > relative packages would be helpful. I posted an example in an earlier message. I'll recast it here, hopefully more eloquently. :) I'll also offer an alternate proposal that also solves my (and I suspect, Marc-Andre's) problem. Zope is an application platform. It provides a mechanism for developers to plug their own products into Zope. The idea is that someone gets Zope from zope.org and installs it. Then they get third-party products from other places. Zope products are python packages installed as sub-packages of the Zope 'Products' packages. Products are generally self-contained. If they need anything that's not part of standard Python or standard Zope, they need to include it or install what they need in a sub-package of another Zope package, 'Shared'. Because products come from "third parties", it is important that they be self contained. Making assumptions about the Zope or Python environments or, worse, modifying the Zope or Python environments is a bad idea. In this context, consider the following concrete, though fictional example. Aaron has written a collection of modules that implement an RDBMS system for Python, gadfly. He also has a set of modules for parsing, kjParsing, which is needed by gadfly. Currently, these are just a bunch of top-level modules distributed as a combined collection. It would make sense to turn these into two packages, gadfly and kjParsing. Now, if this was done, then the gadfly package would need to use package imports for kjParsing modules, as in: import kjParsing.kjParser So far, so good. Now, suppose that someone wants to create a Zope product, ZGadfly, that used gadfly. The simplest approach would be to include the gadfly and kjParsing packages in their Zope product. Of course, this won't work, because the imports, like the one above, will fail, because kjParsing is no longer a top-level package. It wouldn't do any good to move gadlfy and kjParsing to the shared package, although that might be desirable to share these packages with other products. They could try to stuff the packages into the Zope or Python paths, but that would break the rules and lead to problems in the long term. Hopefully, this illustrates the problem. I think that this will be a common problem in the future, as people build bigger and bigger systems with Python that reuse other people's packages. I'd be curious to hear how folks would solve this problem. Personally, I'd like the problem to go away. :) I'd like the Python import rules to change to make this solvable without import hooks or path hacking. I can think of two ways to approach this: - Relative parent imports: import __.kjParsing.kjParser Note that Python already supports relative imports for going down. For example, we use gadfly and kjParsing together as a single sub-package of our ZGadflyDA product. - Gradually less local searches. Currently, when doing an import in a package, two paths are searched, the package path and then the Python path. If there are intermediate packages, then perhaps their paths should be searched as well. For example, suppose we have the directory structure: Products/ZGadfly/gadfly kjParsing where Products is a top-level package, and we did an import in a module in gadfly: import kjParsing.kjParser Python would search the path of the package Products.ZGadfly.gadfly first, as it does now. This search would fail. Then it would search the path of Products.ZGadfly, where it would find kjParsing, and the import would succeed. This approach has the benefits: o It solves the problem. :) o It has no impact on un-nested packages, o It requires no code changes; it doesn't use the ugly __. Thoughts? Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gmcm@hypernet.com Wed Sep 15 18:35:57 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 13:35:57 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37DFBE91.C93816F1@interet.com> Message-ID: <1274705689-31068526@hypernet.com> Jim Ahlstrom wtoe: > "Guido van Rossum" wrote: > > --> The solution: Did the dev-list miss something? The last I see is Barry's post. - Gordon From guido@CNRI.Reston.VA.US Wed Sep 15 19:17:55 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 14:17:55 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 13:35:57 EDT." <1274705689-31068526@hypernet.com> References: <1274705689-31068526@hypernet.com> Message-ID: <199909151817.OAA18276@eric.cnri.reston.va.us> > From: "Gordon McMillan" > Jim Ahlstrom wtoe: > > > "Guido van Rossum" wrote: > > > --> The solution: > > Did the dev-list miss something? The last I see is Barry's post. Hm. We had an email glitch. Apparently this message got lost: Subject: Re: Path hacking From: Guido van Rossum To: python-dev@python.org Date: Tue, 14 Sep 1999 15:57:51 -0400 I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Wed Sep 15 19:27:58 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 14:27:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <1274705689-31068526@hypernet.com> Message-ID: <37DFE52E.5D6F3981@interet.com> Gordon McMillan wrote: > > Jim Ahlstrom wtoe: > > > "Guido van Rossum" wrote: > > > --> The solution: > > Did the dev-list miss something? The last I see is Barry's post. My mail system is flakey, so I have been reading this list directly on python.org. I didn't get it by list either, so I assumed my mailer ate it. See: http://www.python.org/pipermail/python-dev/1999-September/000851.html Jim Ahlstrom From jim@interet.com Wed Sep 15 19:29:38 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 14:29:38 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> Message-ID: <37DFE592.7FD384A2@interet.com> Guido van Rossum wrote: > > But the intention here is for the customization to be application > specific (hence the Spam in the name). sitecustomize doesn't know > whethere I need the Mailman or the Knowbot root added to my path. Ah, you have multiple scripts in one directory and multiple Foo_path, Bar_path etc. I was thinking with my Windows head. A commercial Windows app generally has its own exclusive install directory, so I was thinking single directory so a single sitecustomize.py. > Or do you mean to imply that we can do this with zero text added to > the script, by simply dropping an appropriate sitecustomize.py in the > script dir? Yes, that is exactly what I was thinking. > Unfortunately this does currently *not* work, because > sys.path[0] is added after Py_Initialize() is run. Yikes! That kills using sitecustomize.py. Your Spam_path still works because it is imported later, but requires an import in each Python main script just as you said. Even worse, it means that exceptions.py and site.py can not be found at all except using the normal PYTHONPATH, and putting their path in Spam_path will *not* work. > > > Because the script's directory is first on the default path, the Spam > > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > > > Hmmm. Is this really true? Nothing else, for example the registry, can > > change sys.path[0]? Ever? Please say yes. > > Yes. (The registry can add module-specific paths, which will be > searched before sys.path is even looked at, but this is only for > specific modules. It cannot insert a general directory that is > searched.) The only way this can fail is if an embedding app fails to > call PySys_SetArgv(). Oh dear, I think I heard no instead of yes. Are you saying that if someone else installs a Python app on my customer's machine after I do, and sets a registry entry which sayes to use c:/other/path/to/site.py for site.py (as he may very well want to do), then if my Python program depends on getting my copy of site.py from my directory, it will then use the other copy instead and may very well fail? > In any case, that's a separate issue -- I > agree that if sys.path[0] is '' (as it often is) it's better for > site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize > it (and everything else on the path) so that it will still work if the > app does a chdir(). Point on the curve: Windows apps generally start from an icon which contains their path and current working directory, and these are generally different. So a Windows app in general will *never* have had a getcwd() equal to the path of either the binary interpreter or the Python main script. > > The files exceptions.py and site.py must be in all the bin > > directories as well as sitecustomize.py because they are > > automatically imported in Py_Initialize(). > > Yes. Well, *no* right? This fails unless the bin directories are in fact on PYTHONPATH. The only way to get exceptions.py is by using sys.path as it exists within Py_Initialize(). So there is no hacked sys.path[0] equal to the script dir. And since the path hacks in site.py haven't happened yet either, we have an incomplete sys.path at that point. > > added to sys). How about prepending the single directory sys.executable > > to sys.path during Py_Initialize()? And demanding that modules > > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > > be placed there. > > On Unix, this is a bin directory and it is strongly discouraged to put > non-program files there. Ok, point taken. > Is the full DLL path available at any point? This would certainly be > a good starting point -- especially when the DLL is loaded implicitly > as the result of some COM operation. I don't know about loading by COM, but if it is a file, its absolute path is reliably known in sys, the code is identical to that currently used for sys.executable (on Windows), and I have a patch if you want. JimA's conjecture: It is currently impossible to ship a Python app which can not be damaged by the installation of a second Python app without using a hacked custom binary. Jim Ahlstrom From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Wed Sep 15 19:41:26 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Wed, 15 Sep 1999 14:41:26 -0400 (EDT) Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <1274705689-31068526@hypernet.com> Message-ID: <14303.59478.806707.880694@anthem.cnri.reston.va.us> >>>>> "Gordo" == Gordon McMillan writes: Gordo> Jim Ahlstrom wtoe: >> "Guido van Rossum" wrote: >> --> The solution: Gordo> Did the dev-list miss something? The last I see is Barry's Gordo> post. I have a suspicion that python.org lost some email yesterday. We had a period of time where mail simply stopped getting delivered (thank you Solaris patch manager) and it took me a little while to realize that things weren't working correctly. Since there's nothing unexpected in the mail queue now, all I can say is that if you didn't get it by now, you ain't gonna. However, everything seemed to make it into the archives, so Guido's message is available at: http://www.python.org/pipermail/python-dev/1999-September/000880.html -Barry From guido@CNRI.Reston.VA.US Wed Sep 15 19:43:56 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 14:43:56 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Wed, 15 Sep 1999 10:00:05 PDT." References: Message-ID: <199909151843.OAA18364@eric.cnri.reston.va.us> [David Ascher] > Guido, maybe it would make sense to explain the need for a license change. > Is my understanding correct that the occasion for the license change is > that the copyright is now clearly shifting to CNRI, and as a result CNRI > has to forge a license? (BTW, I thought *you* had the copyright transfer > from CWI, not CNRI). Correct on both counts. CWI owns the copyright on old Python versions through Python 1.2. I have personally obtained non-exclusive rights to these from CWI. CNRI, by nature of my employment contract, has the copyright on newer versions. CNRI feels the need to protect its intellectual property rights. It feels that the old Python license, even with CNRI added, does not adequately protect CNRI against certain (unlikely) events -- hence the desire to draft a new license. CNRI understands that open source (and now Open Source -- the OSI board has approved the old Python license!) like Python requires different licensing terms than a typical product developed solely by CNRI. I think that the main problem is that CNRI's understanding of what truly constritutes open source is limited, and that my own understanding of legal issues is limited, so that the negotiations with CNRI's legal department (which is headed by CNRI's director) often turn in their favor. I hereby withdraw the posted license. There still is the need for a new license, but we need to go back to the drawing board for it. The CWI ownership of much of the code probably means that the license as it stands doesn't hold anyway. I also think that the Python consortium has a say in the license discussion -- the consortium agreement actually discusses the ownership of intellectual property produced by/for the consortium at some length. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Wed Sep 15 19:46:44 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Wed, 15 Sep 1999 14:46:44 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <14303.59796.4588.287265@anthem.cnri.reston.va.us> JF> Zope products are python packages installed as sub-packages of JF> the Zope 'Products' packages. Products are generally JF> self-contained. If they need anything that's not part of JF> standard Python or standard Zope, they need to include it or JF> install what they need in a sub-package of another Zope JF> package, 'Shared'. Jim, from your description, it sounds like Zope should be putting both the `Shared' and possibly `Products' directories on sys.path. Maybe it does and I'm still missing your point. However if they are not, then wouldn't this solve your problem? A user would drop the kjParsing directory into Shared, and then all the import kjParsing.kjParser statements would Just Work. -Barry From mal@lemburg.com Wed Sep 15 18:02:20 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 19:02:20 +0200 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> Message-ID: <37DFD11C.712653BA@lemburg.com> Guido van Rossum wrote: > > It seems to me this totally solves Jim Fulton's and Marc's > > problem and makes "__" unnecessary. You just install zope > > and mx in zopedir, perform the above, and presto you have a new > > private name space where you can control all your names. But > > there must be some problem here I haven't thought of. > > I think no simple solution that *I* can come up with will satisfy > JimF's and Marc's desire for obscurity :-) Never mind, I'll use an imputil.py based approach to get relative imports to work in my packages. That is when I get imputil.py to work... it doesn't seem to be quite there yet (or I'm using an old version). BTW, I'm 100% behind you guys if you choose to reimplement Python's import mechanism in Python using a similar approach as the one Greg implemented in imputil. Should make everybody happy: those who want obscure syntactic add-ons and others with a taste for zlib'ed packaged byte code, plus those VMS freaks ;-) Perhaps we should start a new thread on that topic... Still needed are: · Python level APIs for the platform specific magic on Win32 and Macs (OS/2, BeOS ?), e.g. access to the Windows registry and the Mac forks · Patches to make the DirectoryImporter 100% backward compatible Greg's imputil.py can be found at: http://www.lyra.org/greg/small/ The trick would then be to install an application specific importer in the setup module Spam_path or MyAppSetup which then takes care of all the rest... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm@hypernet.com Wed Sep 15 20:00:41 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 15:00:41 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <14303.59478.806707.880694@anthem.cnri.reston.va.us> Message-ID: <1274700606-31374305@hypernet.com> Hmm, I'm suspicious of the fact that no message from Barry Warsaw ever gets "lost". Stalin got started by being in charge of the Kremlin's telephone system, you know... > >>>>> "Gordo" == Gordon McMillan writes: > > Gordo> Jim Ahlstrom wtoe: > > >> "Guido van Rossum" wrote: > >> --> The solution: > > Gordo> Did the dev-list miss something? The last I see is > Barry's Gordo> post. > > I have a suspicion that python.org lost some email yesterday. > > We had a period of time where mail simply stopped getting > delivered (thank you Solaris patch manager) and it took me a > little while to realize that things weren't working correctly. > Since there's nothing unexpected in the mail queue now, all I can > say is that if you didn't get it by now, you ain't gonna. > > However, everything seemed to make it into the archives, so > Guido's message is available at: > > http://www.python.org/pipermail/python-dev/1999-September/000880. > html > > -Barry > > _______________________________________________ > Python-Dev maillist - Python-Dev@python.org > http://www.python.org/mailman/listinfo/python-dev - Gordon From jim@interet.com Wed Sep 15 20:25:24 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 15:25:24 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <37DFF2A4.DF26A22B@interet.com> Jim Fulton wrote: > In this context, consider the following concrete, though fictional > example. > ............ The classical solution would be to add "Shared" to sys.path. But how to do this? And what if another installation changes it? > I'd be curious to hear how folks would solve this problem. 1) Ship a custom binary interpreter with critical modules frozen in, and which sets a custom sys.path before anything else happens. So you are not dependent at all on PYTHONPATH. Currently this works. 2) Add a new command line option which sets sys.path and always use it to start your app. It lists only critical zope directories so zope has its own private name space. 3) Add your own import hooks using imputil.py and boot it as in (1). Better if it were part of Python. 4) Use a new Python.pyl file which has seeking from the end and thus allows concattenation of multiple *.pyl. Each package is a foo.pyl, bar.pyl, and to distribute an app, you concatenate packages. The implied PYTHONPATH in each *.pyl file is "." except perhaps for the Python lib which is "Lib". Boot it as in (3). > Personally, I'd like the problem to go away. :) I have the same problem, so me too. > I'd like the Python > import rules to change to make this solvable without import hooks or > path hacking. I can think of two ways to approach this: > > - Relative parent imports: > > import __.kjParsing.kjParser OK by me, but doesn't solve the Grand Problem. > - Gradually less local searches. I am currently bewildered by Python's import mechanism, and this may push me into brain damage. Well, *more* brain damage. Jim Ahlstrom From jim@digicool.com Wed Sep 15 20:34:08 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 15:34:08 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> Message-ID: <37DFF4B0.15390199@digicool.com> "Barry A. Warsaw" wrote: > > JF> Zope products are python packages installed as sub-packages of > JF> the Zope 'Products' packages. Products are generally > JF> self-contained. If they need anything that's not part of > JF> standard Python or standard Zope, they need to include it or > JF> install what they need in a sub-package of another Zope > JF> package, 'Shared'. > > Jim, from your description, it sounds like Zope should be putting both > the `Shared' and possibly `Products' directories on sys.path. It actually does now, although that is bad. What if some future python distribution has a Products package? Or a Products.py? A product is actually a sub-package of Products. Even if Products was not a package, the Product would be and gadfly and kjParsong would be sub-packages. There is a similar issue with Shared, since Shared software is organized by organization, as in 'Shared.DC.ZRDB'. > Maybe > it does and I'm still missing your point. The point is that stuffing arbitrary things into a single global name space is bad. That's why packages were invented. Unless packages can be effectively nested, you haven't gained anything but time. > However if they are not, then wouldn't this solve your problem? No, see above. > A > user would drop the kjParsing directory into Shared, and then all the > import kjParsing.kjParser statements would Just Work. But then they'd be dumping kjParsing into the global package namespace. Eventually, there will be collisions. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@digicool.com Wed Sep 15 20:59:10 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 15:59:10 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <37DFF2A4.DF26A22B@interet.com> Message-ID: <37DFFA8E.FB73A006@digicool.com> "James C. Ahlstrom" wrote: > > Jim Fulton wrote: > > In this context, consider the following concrete, though fictional > > example. > > ............ > > The classical solution would be to add "Shared" to sys.path. But > how to do this? And what if another installation changes it? This doesn't work, because 'Shared' has sub-packages by organization. > > I'd be curious to hear how folks would solve this problem. > > 1) Ship a custom binary interpreter with critical modules frozen in, > and which sets a custom sys.path before anything else happens. So > you are not dependent at all on PYTHONPATH. Currently this works. This is what we do for our binary distribtions, but it doesn't do anything for third-party products added to Zope. > 2) Add a new command line option which sets sys.path and always use > it to start your app. It lists only critical zope directories so > zope has its own private name space. We have a start script that takes care of this. This gets Zope's top-level names (of which there are too many already) into the path. This doesn't help with third-party products added to Zope. > 3) Add your own import hooks using imputil.py and boot it as > in (1). But imputil has a global effect. I suppose I could implement what I want with imputil or by otherwise creating my own importer using the standard import hooks. I suspect that's what I'll have to do eventually. > Better if it were part of Python. > 4) Use a new Python.pyl file which has seeking from the end and > thus allows concattenation of multiple *.pyl. Each package is > a foo.pyl, bar.pyl, and to distribute an app, you concatenate > packages. The implied PYTHONPATH in each *.pyl file is "." > except perhaps for the Python lib which is "Lib". Boot it as > in (3). I'm not looking for a way to replace the current package scheme. Also, I'm not just looking at an app, but at an app that contains apps (aka components). > > Personally, I'd like the problem to go away. :) > > I have the same problem, so me too. > > > I'd like the Python > > import rules to change to make this solvable without import hooks or > > path hacking. I can think of two ways to approach this: > > > > - Relative parent imports: > > > > import __.kjParsing.kjParser > > OK by me, but doesn't solve the Grand Problem. > > > - Gradually less local searches. > > I am currently bewildered by Python's import mechanism, and > this may push me into brain damage. Well, *more* brain damage. I don't see what's so complicated. You just use a gradually widening search, rather than an immediately widening search. I don't think that this is any more complicated than the current rule. I actually prefer the second apprach because it doesn't require any code changes to take advantage of it. I may implement it for Zope if nothing is done in standard Python. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal@lemburg.com Wed Sep 15 21:01:01 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 22:01:01 +0200 Subject: [Python-Dev] Towards a Python based import scheme Message-ID: <37DFFAFD.72F6A806@lemburg.com> Back again... To get a little more constructive I've started hacking away on Greg Stein's imputil.py to make it work with my DateTime package. The DateTime packages does a lot of from...import... and intra-package imports, plus it loads a shared lib as extension. The original version of imputil I fetched from Greg's page did work out of the box (from...import... hassles) and obviously did not support in-package shared libs. I've added both features so that the test script in DateTime can run successfully. Things that remain are: · the win32 registry stuff (needs C code) · the Mac fork stuff (needs C code) · a working __path__ implementation (is anyone using this attribute which only is available in packages ?) · probably a whole bunch of other quirks · some speedups (there currently are too many stat()s) Please give it a try: http://starship.skyport.net/~lemburg/imputil.py in color: http://starship.skyport.net/~lemburg/imputil.py.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm@hypernet.com Wed Sep 15 21:22:34 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 16:22:34 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DFF4B0.15390199@digicool.com> Message-ID: <1274695692-31669872@hypernet.com> Jim Fulton wrote: > It actually does now, although that is bad. What if some future > python distribution has a Products package? Or a Products.py? Name it zopeProducts. Or Zope/Products and Zope/Shared. > A product is actually a sub-package of Products. Even if Products > was not a package, the Product would be and gadfly and kjParsong > would be sub-packages. There is a similar issue with Shared, > since Shared software is organized by organization, as in > 'Shared.DC.ZRDB'. See below. > The point is that stuffing arbitrary things into a single global > name space is bad. That's why packages were invented. Unless > packages can be effectively nested, you haven't gained anything > but time. Packages nest effectively. Subpackages just don't masquerade as top level packages. [Barry] > > A > > user would drop the kjParsing directory into Shared, and then > > all the import kjParsing.kjParser statements would Just Work. > > But then they'd be dumping kjParsing into the global package > namespace. Eventually, there will be collisions. You are trying to do two incompatible things at once. You are trying to be both a standalone application, and a Python installation. You can be both, just not at the same time. To be a standalone application, you can use tricks that Jim Ahlstrom pointed out, or tricks from my Installer. The major point being that sys.path has nothing to do with the PYTHONPATH environment variable. You now have an installation where you can be assured of no collisions, because you control it completely. Of course, that will make liars out of the people at your company who claim you can mix and match pieces of Zope and your own stuff as you please. People who want to mix and match need Zope to play by normal Python rules - and that means packages are installed at the top level. Collisions happen, but your audience is people who *want* to do things differently and *want* control of their installations. By creating "self contained" packages and putting them in a normal Python installation, you're just pushing your headache off on the user. You're co-opting his installation. Yeah, now you have two versions, but they're 99% the same. Build the normal Python one first and distribute it to the hackers. Then install some import and startup hooks, and distribute that to your "users". I, for one, still use Bobo, because Zope doesn't play fair and I have no need or desire for yet another Python installation. - Gordon From gmcm@hypernet.com Wed Sep 15 21:43:27 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 16:43:27 -0400 Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <1274694439-31745225@hypernet.com> Marc-Andre wrote: > To get a little more constructive I've started hacking away on > Greg Stein's imputil.py to make it work with my DateTime package. > http://starship.skyport.net/~lemburg/imputil.py You (and Greg) are missing a rather important patch I submitted to Greg a long time ago (around line 62): def _reload_hook(self, module): # gmcm - Hmmm, reloading of a module may or may not be impossible, # (depending on the importer), but at least we can # look to see if it's ours to reload: if hasattr(module, '__importer__'): if getattr(module, '__importer__') == self: raise SystemError, "reload not yet implemented" return self.__chain_reload(module) - Gordon From jim@digicool.com Wed Sep 15 21:44:31 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 16:44:31 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274695692-31669872@hypernet.com> Message-ID: <37E0052F.A899E985@digicool.com> Gordon McMillan wrote: > > Jim Fulton wrote: > (snip) > Packages nest effectively. Subpackages just don't > masquerade as top level packages. They do currently when imported from the containing package. > [Barry] > > > A > > > user would drop the kjParsing directory into Shared, and then > > > all the import kjParsing.kjParser statements would Just Work. > > > > But then they'd be dumping kjParsing into the global package > > namespace. Eventually, there will be collisions. > > You are trying to do two incompatible things at once. You are > trying to be both a standalone application, and a Python > installation. You can be both, just not at the same time. We don't try to be a Python installation. We try to be a Python application that is a platform for Zope applications. > To be a standalone application, you can use tricks that Jim > Ahlstrom pointed out, or tricks from my Installer. The major > point being that sys.path has nothing to do with the > PYTHONPATH environment variable. You now have an > installation where you can be assured of no collisions, > because you control it completely. > > Of course, that will make liars out of the people at your > company who claim you can mix and match pieces of Zope > and your own stuff as you please. Zope reuses *alot* of code. I think Zope has a pretty good track record of working with existing code. > People who want to mix > and match need Zope to play by normal Python rules - and > that means packages are installed at the top level. Sorry, I just don't buy that. > Collisions > happen, but your audience is people who *want* to do things > differently and *want* control of their installations. Actually, many Zope user's come to Python by way of Zope. They want to be able to install Zope products without worrying about what their zope or python installation looks like. Zope developers want to be able to have their products drop into a Zope installation easily. > By creating "self contained" packages and putting them in a > normal Python installation, you're just pushing your headache > off on the user. You're co-opting his installation. No, putting packages in the python path coops their installation. Putting Zope-specific products that happen to include reusable Python packages in a Zope-specific package leaves their installation alone. > Yeah, now you have two versions, but they're 99% the same. > Build the normal Python one first and distribute it to the > hackers. Then install some import and startup hooks, and > distribute that to your "users". This is exactly what I want to prevent. I'd like package developers to develop their packages once without making them use esoteric import hooks. OTOH I'd like to be able to *use* their packages wherever I damn well please. I don't to be forced to put their packages in the python path. > I, for one, still use Bobo, because Zope doesn't play fair In what way does Zope not play fair? > and I > have no need or desire for yet another Python installation. Cool. You don't need to use another Python installation to use Zope. In fact, for development, we use a totally standard Python distribution. Developers here are not *allowed* to add anything special to the Python installations on the development machines here because we want to *make sure* that we can run with stock python installations. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From bwarsaw@python.org Wed Sep 15 22:55:34 1999 From: bwarsaw@python.org (Barry A. Warsaw) Date: Wed, 15 Sep 1999 17:55:34 -0400 (EDT) Subject: [Python-Dev] Re: Path hacking References: <14303.59478.806707.880694@anthem.cnri.reston.va.us> <1274700606-31374305@hypernet.com> Message-ID: <14304.5590.405325.369903@anthem.cnri.reston.va.us> >>>>> "Gordo" == Gordon McMillan writes: Gordo> Hmm, I'm suspicious of the fact that no message from Barry Gordo> Warsaw ever gets "lost". Gordo> Stalin got started by being in charge of the Kremlin's Gordo> telephone system, you know... Well, I managed to get rid of Ken so my world domination plan is right on schedule! guido-may-be-benevolent-but-you-can-bet-i-won't-be-ly y'rs, -Barry From bwarsaw@python.org Wed Sep 15 23:07:34 1999 From: bwarsaw@python.org (Barry A. Warsaw) Date: Wed, 15 Sep 1999 18:07:34 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> Message-ID: <14304.6310.20571.487708@anthem.cnri.reston.va.us> >>>>> "JF" == Jim Fulton writes: JF> It actually does now, although that is bad. What if some JF> future python distribution has a Products package? Or a JF> Products.py? Wouldn't collide, because in my scheme, Shared isn't a package, it's a directory on sys.path. Everything inside Shared is a top-level package. You'd never "import Shared.DC.ZRDB" but instead you would "import DC.ZRDB". >> A user would drop the kjParsing directory into Shared, and then >> all the import kjParsing.kjParser statements would Just Work. JF> But then they'd be dumping kjParsing into the global package JF> namespace. Eventually, there will be collisions. Do you really think so? Someone else is going to write a kjParsing module? I think we're safe from this for the forseeable future through convention and peer pressure. Or is the concern that there will be two Products which use different versions of kjParsing? I'm also not concerned here because if my product needs a very specific version of kjParsing I can't see any way around shipping my own version, subpackaging it, and using an absolute package name to ensure that I'm getting at exactly the one I need. This is the approach Mailman takes. -Barry From guido@CNRI.Reston.VA.US Wed Sep 15 23:35:20 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 18:35:20 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 14:29:38 EDT." <37DFE592.7FD384A2@interet.com> References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> Message-ID: <199909152235.SAA18693@eric.cnri.reston.va.us> [me] > > But the intention here is for the customization to be application > > specific (hence the Spam in the name). sitecustomize doesn't know > > whethere I need the Mailman or the Knowbot root added to my path. [JimA] > Ah, you have multiple scripts in one directory and multiple > Foo_path, Bar_path etc. I was thinking with my Windows head. > A commercial Windows app generally has its own exclusive > install directory, so I was thinking single directory so a single > sitecustomize.py. > > > Or do you mean to imply that we can do this with zero text added to > > the script, by simply dropping an appropriate sitecustomize.py in the > > script dir? > > Yes, that is exactly what I was thinking. > > > Unfortunately this does currently *not* work, because > > sys.path[0] is added after Py_Initialize() is run. > > Yikes! That kills using sitecustomize.py. Your Spam_path > still works because it is imported later, but requires an > import in each Python main script just as you said. Not too bad (who cares about one more line of boilerplate...). > Even worse, it means that exceptions.py and site.py can not > be found at all except using the normal PYTHONPATH, and > putting their path in Spam_path will *not* work. Why would you want your own exceptions.py and site.py? > > > > Because the script's directory is first on the default path, the Spam > > > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > > > > > Hmmm. Is this really true? Nothing else, for example the registry, can > > > change sys.path[0]? Ever? Please say yes. > > > > Yes. (The registry can add module-specific paths, which will be > > searched before sys.path is even looked at, but this is only for > > specific modules. It cannot insert a general directory that is > > searched.) The only way this can fail is if an embedding app fails to > > call PySys_SetArgv(). > > Oh dear, I think I heard no instead of yes. Are you saying that if > someone else installs a Python app on my customer's machine after I do, > and sets a registry entry which sayes to use c:/other/path/to/site.py > for site.py (as he may very well want to do), then if my Python program > depends on getting my copy of site.py from my directory, it will then > use the other copy instead and may very well fail? Again - why would anyone register their own site.py? > > In any case, that's a separate issue -- I > > agree that if sys.path[0] is '' (as it often is) it's better for > > site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize > > it (and everything else on the path) so that it will still work if the > > app does a chdir(). > > Point on the curve: Windows apps generally start from an icon > which contains their path and current working directory, and > these are generally different. So a Windows app in general will > *never* have had a getcwd() equal to the path of either the > binary interpreter or the Python main script. You're lucky. It turns out that on Windows, under those circumstances at least, sys.path[0] is the absolute pathname of the directory. You only see '' if sys.argv[0] doesn't have any pathname information; that's only possible if the script *does* live in the current directory. > > > The files exceptions.py and site.py must be in all the bin > > > directories as well as sitecustomize.py because they are > > > automatically imported in Py_Initialize(). > > > > Yes. > > Well, *no* right? This fails unless the bin directories are in > fact on PYTHONPATH. The only way to get exceptions.py is by using > sys.path as it exists within Py_Initialize(). So there is no > hacked sys.path[0] equal to the script dir. And since the > path hacks in site.py haven't happened yet either, we have > an incomplete sys.path at that point. Sorry, I've lost track of what we were after here. Indeed the scripts' directory (which I presume you meant by the bin directory) indeed doesn't occur in sys.path until after Py_Initialize() has run. > > > added to sys). How about prepending the single directory sys.executable > > > to sys.path during Py_Initialize()? And demanding that modules > > > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > > > be placed there. > > > > On Unix, this is a bin directory and it is strongly discouraged to put > > non-program files there. > > Ok, point taken. > > > Is the full DLL path available at any point? This would certainly be > > a good starting point -- especially when the DLL is loaded implicitly > > as the result of some COM operation. > > I don't know about loading by COM, but if it is a file, its absolute > path is reliably known in sys, the code is identical to that currently > used for sys.executable (on Windows), and I have a patch if you want. I presume using GetModuleFileName()? Please send me the patch! > JimA's conjecture: It is currently impossible to > ship a Python app which can not be damaged by the installation of a > second Python app without using a hacked custom binary. Sounds right. All tricks to make the app unique require using a different registry key, which requires a change to the DLL. However, you can do this without recompiling! The version string is used is embedded in a resource, so you can patch it using some kind of resource editor. Mark Hammond planned it this way! --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@digicool.com Wed Sep 15 23:49:38 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 18:49:38 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> Message-ID: <37E02282.BDF5FF21@digicool.com> "Barry A. Warsaw" wrote: > > >>>>> "JF" == Jim Fulton writes: > > JF> It actually does now, although that is bad. What if some > JF> future python distribution has a Products package? Or a > JF> Products.py? > > Wouldn't collide, because in my scheme, Shared isn't a package, it's a > directory on sys.path. Everything inside Shared is a top-level > package. You'd never "import Shared.DC.ZRDB" but instead you would > "import DC.ZRDB". I wasn't talking about your scheme. I was talking about how I thought it should be. > >> A user would drop the kjParsing directory into Shared, and then > >> all the import kjParsing.kjParser statements would Just Work. > > JF> But then they'd be dumping kjParsing into the global package > JF> namespace. Eventually, there will be collisions. > > Do you really think so? Yes. > Someone else is going to write a kjParsing > module? No, but sooner or later, two people are going to write two "Parsing" packages, or two "Oracle" packages or two .... Are you suggesting that package names should begin with some hash value to assure that they are unique? (What does kj stand for anyway? :) I guess we could use some form of GUID to be really safe ..... > I think we're safe from this for the forseeable future > through convention and peer pressure. 8^o (I feel like I should respond to this in some way but .... I just can't understand this kind of argument and don't know how to frame a response.) > Or is the concern that there will be two Products which use different > versions of kjParsing? No, at least not primarily. > I'm also not concerned here because if my > product needs a very specific version of kjParsing I can't see any way > around shipping my own version, subpackaging it, and using an absolute > package name to ensure that I'm getting at exactly the one I need. But even in this case, if you used a relative import, you *would* get the specific version you want. I give up. I'll just have to write some import hook to get the behavior I want in Zope. I will probably implement the second alternative I wrote about in my earlier document. I'll get Python to look at intermediate package paths before it looks at the main path. That way, any package that works as a top-level package will also work as a sub-package without modification. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mhammond@skippinet.com.au Thu Sep 16 02:20:10 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 16 Sep 1999 11:20:10 +1000 Subject: [Python-Dev] License cleanup In-Reply-To: <199909151843.OAA18364@eric.cnri.reston.va.us> Message-ID: <004601beffe1$9edb01b0$0801a8c0@bobcat> > CNRI understands that open source (and now Open Source -- the OSI > board has approved the old Python license!) like Python requires > different licensing terms than a typical product developed solely by > CNRI. Actually, this is a good point. I know the issue of payment may get in the way, but it would make sense to have any future proposed licenses reviewed by a lawyer "on our side" - eg, someone whose mandate is to give a legal opinion on the risks and liabilities of the _user_ of the license. Obviously the CNRI lawers are protecting their (ie, CNRI's) interests, and everyone on this group is concerned about their own (ie, personally, their company, or companies they wish to introduct Python into) interests. If the legal jibberish can't be removed (which is likely with lawyers involved) I know I would personally feel much more comfortable with a legal opinion covering my interests.. But as I said, who will pay? If nothing else, we should ensure the OSI approves of the new license... Or maybe we can convince CNRI there is real and serious concern, and they could pay for an external IP lawyer? Mark. From gstein@lyra.org Thu Sep 16 02:37:17 1999 From: gstein@lyra.org (Greg Stein) Date: Wed, 15 Sep 1999 18:37:17 -0700 Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> Message-ID: <37E049CD.6B64DE00@lyra.org> Mark Hammond wrote: >... > Actually, this is a good point. I know the issue of payment may get > in the way, but it would make sense to have any future proposed > licenses reviewed by a lawyer "on our side" - eg, someone whose > mandate is to give a legal opinion on the risks and liabilities of the > _user_ of the license. Interesting thought! > Obviously the CNRI lawers are protecting their > (ie, CNRI's) interests, and everyone on this group is concerned about > their own (ie, personally, their company, or companies they wish to > introduct Python into) interests. Because of these varied interests, I don't think a review by a particular lawyer will be greatly helpful. The question will still remain: "did the lawyer review it from [my/our/company] perspective?" This will lead people back into the same review cycle. >... > But as I said, who will pay? If nothing else, we should ensure the > OSI approves of the new license... Or maybe we can convince CNRI > there is real and serious concern, and they could pay for an external > IP lawyer? Definitely having an OSI certification will be great (cool stuff on the cert for the existing license!). Having Bruce Perens review the license would also be a great boon (see www.perens.com for some of his writings; also see http://perens.com/Termination.html specifically). Licenses are a tough issue. I had to go through this entire morass when deciding what to do with mod_dav. There are a lot of varieties and issues and stuff to cover. I've read a bunch of license (not to mention a bazillion legal documents during the eShop/acquisition days). Not always exciting reading :-), but usually quite interesting. At this point, I think it is a great thing that CNRI is reviewing the license. Unfortunately, the license wasn't as non-controversial as it was thought to be :-(. I'm more than happy to wait and see where they go with the license. Cheers, -g -- Greg Stein, http://www.lyra.org/ From guido@CNRI.Reston.VA.US Thu Sep 16 04:32:18 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 23:32:18 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Wed, 15 Sep 1999 14:10:17 EDT." <37DFE109.96C652B3@digicool.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <199909160332.XAA19278@eric.cnri.reston.va.us> I don't like the pressure to allow relative imports and I don't like the solutions that those who insist on having it propose (they are all writing their own import hooks). The reasoning seems to be something like this: (a) If my package needs non-standard package X, I can't require that X be installed as a top-level package, because there might be another unrelated package also called X; therefore I need to install a copy of package X as a subpackage of my own package. (b) Because any non-standard package X now may be installed at the toplevel or as a subpackage of some other package, X must use relative imports exclusively to access subparts of itself. I find (b) unacceptable for a number of reasons, that have all been explained by others: duplicate installs (and hence possible imports) of the same package as a subpackage of various other packages, the requirement to be religious in the use of relative imports, the ugliness of the __ notation (I do *not* look forward to explaining that to 12-year-olds in the context of CP4E). I think that the fear expressed in (a) is an overreaction to some recent actual top-level name clashes. This was a real pain, but notice that it happened because several popular module collections weren't packagized at all! In the Java world, the right solution exist: a better top-level naming scheme. I forget what the exact rules are, but I seem to recall that Java has something of the same relative import rules af Python currently has: you can import a class from the same leaf package by its simple name, or you can import a class by its full name. As Tim explains, the occasional renames are easily handled with a global substitute in a smart editor. I don't want to do the same thing as Java and require everybody to use their reversed domain name as their package name. Of course you are free to do so: there are currently no toplevel modules or packages that I know of named org, com, edu, gov, mil or net. Two letter domains are all free except for re=Reunion (and ni=Nicaragua, but ni is expired). Oops, mx=Mexico! Well, Python package semantics are slightly different than Java semantics in that shared ownership of a subpackage of (e.g. :-) mx is harder in Python than in Java (Python requires that they are installed in the same directory; Java doesn't). In any case the proper thing to do is to pick a good unique toplevel name. Aaron Watters did a great job with kj (not a country code either!); Marc Andre also has the right spirit with mx. CNRI will probably use cnri as the toplevel package name for CNRI-developed packages that aren't distributed together with Python. If there are still worries about name choices, I volunteer to run a toplevel name registry. This has worked well e.g. for Apple where application developers had to pick a unique 4-letter code; certainly better than the Windows world where conflicting 3-letter extension definitions are common. In the Python world, I would recommend longer toplevel names, so it's more likely that you can use your company name (e.g. pythonware). Note that I still resist the urge to start renaming all the existing standard modules -- too much breakage. This will happen in 2.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one@email.msn.com Thu Sep 16 04:38:22 1999 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 23:38:22 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DF65DB.46F0191A@lemburg.com> Message-ID: <000101befff4$ecc9bde0$712d153f@tim> Sure is a lot of traffic on this -- can't we go back to hating the new license ? [MAL] > ... > Sticking this technology into the import statement is really only > a more elegant approach. Nothing more. The whole thing already > works using the current Python implementation. Your point there has been made many times -- I'm not missing it. My point has also been made many times, so one more can't hurt : it still looks to me like bad practice, so I don't *want* (more, actively oppose) it to be made easier to spell. >> + JimF and MAL's shared notion that they're the only ones making significant >> use of packages is myopic. > Oh, c'mon Tim, we never did say that. We only mentioned having used > package for quite a while. This includes having seen all the different > pitfalls they have to offer. Well, JimF was laying it on a bit thick in two msgs, and sucked you in as his implicit conspirator. My apologies if you feel tarred unfairly. Yet you still can't resist implying that only you two are aware of the pitfalls . >> [on Dragon's "relentlessly straightforward" import policies] > This is perfectly ok if you're just using your own code, but it > gets a mess when third-party packages are involved simply because > you can't be sure they don't use import hooks, local imports or > hacks to __path__ etc. If you still want them to be useable, you'll > have to go down and dive into their structure. This takes time, is > error prone and not necessarily fun. While true, it's much safer to install a "cheating" package at top level, since the cheats presumably *assume* that's where the package is to be run from. If someone distributes a package that does not run correctly from top level, that's one hopeless package. Else it does run correctly from top level, and then the only person with a problem here is one who insists on running it from some embedded location. You then buy the need to dive into their code very carefully indeed -- relative imports don't save you from the huge variety of "clever tricks" they *may* be playing with hooks and path hacks. But note that, unlike Jim (& perhaps you too), I have almost no fear of top-level name collisions. If we do nothing about that, it will be years before it becomes a real problem; and several months before that , we can flock to Java's hokey but effective reverse-domain scheme (which I suspect you're already in favor of -- fine by me!). > Also, I don't know how "explicitly, long-windedly and straightforwardly" > writing module imports is any different from using relative imports. Hugely and profoundly different, as your example shows: > An example: > > mx > .DateTime > .ODBC.Windows > > In ODBC.Windows I would write: > > import __.__.DateTime > > which is just as explicit as writing > > import mx.DateTime > > except that the information about the top-level hierarchy is *not* > included in the import information. Exactly: now instead of a bulletproof absolute import in a self-contained importer, whose precise meaning is clear from casual static inspection, you've got a relative one whose meaning depends vitally on where the importer happens to live. As code moves around, the import may fail to work when you're lucky, or pick up an unintended DateTime package when you're not. The semantics of the import are indeed relative now: you see the opportunity to type three additional characters as a strength, and I see nothing but headaches masquerading as convenience. As further consequence, simple "search-and-replace" strategies are rendered at best useless. That is, a simple grep can no longer find all & only references to mx's flavor of DateTime: it picks up all references to all DateTime guys, with no way short of a structure-aware tree crawler to sort them out again. Complexity spreads. Of course, in the end, I lose no matter what. If *everyone* used dirt-dumb fully-qualified imports, all hassles with shifting package structure could be solved with a text editor in seconds. But that's not whiz-bang enough, so people *will* play endlessly dubious tricks with hooks, paths, relative imports, and metapackages too once Gordon gets around to them . Having an explicit way to spell relative imports would make my life easier because at least those are statically recognizable for what they are. I don't want them and won't use them, but it could ease the pain others will cause by using them. That's not enough to make me in favor of them, though; just enough so I won't pout if they do go in. > Anyway, I'm not too inclined continuing this discussion anymore. > I would never have thought that such a nifty little patch would > have stirred up so much negative response. I'll just hack up my > very own import mechanism using imputil and be done. So there! Don't feel picked on, M-A! Opposition is normal and healthy; and the last idea I got anywhere with was list.pop(). and-even-then-jimf-tried-to-stick-a-default-on-it-ly y'rs - tim From da@ski.org Thu Sep 16 05:19:32 1999 From: da@ski.org (David Ascher) Date: Wed, 15 Sep 1999 21:19:32 -0700 (Pacific Daylight Time) Subject: [Python-Dev] License cleanup In-Reply-To: <004601beffe1$9edb01b0$0801a8c0@bobcat> Message-ID: On Thu, 16 Sep 1999, Mark Hammond wrote: > But as I said, who will pay? If nothing else, we should ensure the > OSI approves of the new license... Or maybe we can convince CNRI > there is real and serious concern, and they could pay for an external > IP lawyer? Eric Raymond's wife is a lawyer, I believe =). From mhammond@skippinet.com.au Thu Sep 16 05:28:19 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 16 Sep 1999 14:28:19 +1000 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <000101befff4$ecc9bde0$712d153f@tim> Message-ID: <004e01befffb$e83a6d90$0801a8c0@bobcat> > Don't feel picked on, M-A! Opposition is normal and healthy; As a (hopefully welcome) tangent to this discussion, I am currently reading "Dynamics of Software Development" by Jim McCarty. He has an interesting section on exactly this. I will quote it verbatim (typing it in, so forgive typos). I present it without comment other than: It appears we are a healthy team! Mark. The Problem of Creativity: Creativity in a group is always limited by the groups defensiveness, and some amount of defensiveness is healthy. In the healthy team, then, no matter how highly valued creativity is ostensibly, change beyond a certain normative quantity or degree will be resisted. And even change within bounds will be accepted only insofar as it can seem to fall under the rubric of "improvement". Change must seem to build on the previously known and accepted reality. Even if there are only two steps involved in a change, with the second building on the first, in a single reality cycle - single product development cycle for us - the change is likely to be rejected by the healthy team. What you need for truly creative change, then, is an environment that transcends good health, an environment that not only accepts a continuum of change, which is normal, but one that positively engenders, nurtures and propels forward wholly new dynamics. The transcendent organization values radical or revolutionary change and esteems utterly new modes of thought. It's possible for a team to be healthy and not particularly creative, but this state of affairs is not especially desirable. What is desirable is team fecundity [MH - had to look this up - "intellectually productive or inventive to a marked degree" - synonym of fertile], the radiating of the new and the original from the normal and the healthy. This kind of creativity requires a flexibility and a courage beyond the reach of most of us most of the time It's ironic and worth remembering that the healthier the team the more effective the resistance to revolutionary thought will be. This is a natural consequence of the teams overall effectiveness on every relevant domain. -- end of quote-- From jcw@equi4.com Thu Sep 16 10:48:33 1999 From: jcw@equi4.com (Jean-Claude Wippler) Date: Thu, 16 Sep 1999 11:48:33 +0200 Subject: [Python-Dev] License cleanup References: <199909151843.OAA18364@eric.cnri.reston.va.us> Message-ID: <37E0BCF1.24FD7431@equi4.com> Guido van Rossum wrote: > I hereby withdraw the posted license. There still is the need for a > new license, but we need to go back to the drawing board for it. One more consideration: some people may compare the "scariness" of the Python license against, say, the Tcl license - and choose accordingly. It's not even about content: seeing that new license, or the Perl licenses for that matter, it sends out a strong message IMO: you are entering the world of lawyers. Proceed with caution. Guard dogs. -- Jean-Claude From mal@lemburg.com Thu Sep 16 11:44:34 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 16 Sep 1999 12:44:34 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> Message-ID: <37E0CA12.1E24EBF2@lemburg.com> Guido van Rossum wrote: > > I don't like the pressure to allow relative imports and I don't like > the solutions that those who insist on having it propose (they are all > writing their own import hooks). I'm wasn't planning to write my own hook until now. I need this feature to be able to organize my package internal stuff swiftly and easily, plus to be able to possibly put it under new top-level packages. Obviously I seem to be more or less alone with this need, so I'll stop argueing for inclusion of "something like relative imports" in the distribution. > The reasoning seems to be something like this: > > (a) If my package needs non-standard package X, I can't require that X > be installed as a top-level package, because there might be another > unrelated package also called X; therefore I need to install a copy of > package X as a subpackage of my own package. > > (b) Because any non-standard package X now may be installed at the > toplevel or as a subpackage of some other package, X must use relative > imports exclusively to access subparts of itself. > > I find (b) unacceptable for a number of reasons, that have all been > explained by others: duplicate installs (and hence possible imports) > of the same package as a subpackage of various other packages, the > requirement to be religious in the use of relative imports, the > ugliness of the __ notation (I do *not* look forward to explaining > that to 12-year-olds in the context of CP4E). (b) is (if at all) a problem only to be taken into account by the author of package X. He may or may not use rel. imports. A 12-year old probably won't (but then: you never know ;). BTW, what is this CP4E thing you're talking about. If it's an syntax aware editor, I have a friend who is really interested in these things... could you send me an URL that I can send him ? > I think that the fear expressed in (a) is an overreaction to some > recent actual top-level name clashes. This was a real pain, but > notice that it happened because several popular module collections > weren't packagized at all! > > In the Java world, the right solution exist: a better top-level naming > scheme. I forget what the exact rules are, but I seem to recall that > Java has something of the same relative import rules af Python > currently has: you can import a class from the same leaf package by > its simple name, or you can import a class by its full name. As Tim > explains, the occasional renames are easily handled with a global > substitute in a smart editor. > > I don't want to do the same thing as Java and require everybody to use > their reversed domain name as their package name. Of course you are > free to do so: there are currently no toplevel modules or packages > that I know of named org, com, edu, gov, mil or net. Two letter > domains are all free except for re=Reunion (and ni=Nicaragua, but ni > is expired). Oops, mx=Mexico! Oh well... I guess they'll have to use 'mex' if they decide to go the reverse domain way ;-) > Well, Python package semantics are > slightly different than Java semantics in that shared ownership of a > subpackage of (e.g. :-) mx is harder in Python than in Java (Python > requires that they are installed in the same directory; Java doesn't). True, perhaps we should lighten this requirement a little when we recode the import mechanism in Python ? E.g. if a local import fails continue the search with the fully qualified name and only if that fails, restart using the local name. This would need some kind of fastpath cache to make the search reasonably fast though. > In any case the proper thing to do is to pick a good unique toplevel > name. Aaron Watters did a great job with kj (not a country code > either!); Marc Andre also has the right spirit with mx. CNRI will > probably use cnri as the toplevel package name for CNRI-developed > packages that aren't distributed together with Python. > > If there are still worries about name choices, I volunteer to run a > toplevel name registry. This has worked well e.g. for Apple where > application developers had to pick a unique 4-letter code; certainly > better than the Windows world where conflicting 3-letter extension > definitions are common. In the Python world, I would recommend longer > toplevel names, so it's more likely that you can use your company name > (e.g. pythonware). > > Note that I still resist the urge to start renaming all the existing > standard modules -- too much breakage. This will happen in 2.0. Wouldn't it suffice to just put them into one package, e.g. 'python.' ? BTW, as Tim argued: the breakage an easily be leveraged by using a smart editor... ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 106 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Thu Sep 16 11:56:01 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 16 Sep 1999 12:56:01 +0200 Subject: [Python-Dev] Relative Package Imports References: <000101befff4$ecc9bde0$712d153f@tim> Message-ID: <37E0CCC1.6EE4E599@lemburg.com> Tim Peters wrote: > > Sure is a lot of traffic on this -- can't we go back to hating the new > license ? The license was withdrawn... I would prefer starting to talk about imputil.py. > [MAL] > > Also, I don't know how "explicitly, long-windedly and straightforwardly" > > writing module imports is any different from using relative imports. > > Hugely and profoundly different, as your example shows: > > > An example: > > > > mx > > .DateTime > > .ODBC.Windows > > > > In ODBC.Windows I would write: > > > > import __.__.DateTime > > > > which is just as explicit as writing > > > > import mx.DateTime > > > > except that the information about the top-level hierarchy is *not* > > included in the import information. > > Exactly: now instead of a bulletproof absolute import in a self-contained > importer, whose precise meaning is clear from casual static inspection, Ah, that's a new idea ! Package local absolute names... wow, that sounds weird. > you've got a relative one whose meaning depends vitally on where the > importer happens to live. As code moves around, the import may fail to work > when you're lucky, or pick up an unintended DateTime package when you're > not. The semantics of the import are indeed relative now: you see the > opportunity to type three additional characters as a strength, and I > see nothing but headaches masquerading as convenience. Tim, the intention was being able to move the mx level around, e.g. renaming it, eliminating it by sticking DateTime and ODBC under some other higher level package (yes, you'll have to watch out here because the higher level package might define DateTime and ODBC too -- but it's not all that likely), etc. The buzz word here is 'relocatable'. > As further consequence, simple "search-and-replace" strategies are rendered > at best useless. That is, a simple grep can no longer find all & only > references to mx's flavor of DateTime: it picks up all references to all > DateTime guys, with no way short of a structure-aware tree crawler to sort > them out again. Complexity spreads. > > Of course, in the end, I lose no matter what. If *everyone* used dirt-dumb > fully-qualified imports, all hassles with shifting package structure could > be solved with a text editor in seconds. But that's not whiz-bang enough, > so people *will* play endlessly dubious tricks with hooks, paths, relative > imports, and metapackages too once Gordon gets around to them . > Having an explicit way to spell relative imports would make my life easier > because at least those are statically recognizable for what they are. I > don't want them and won't use them, but it could ease the pain others will > cause by using them. That's not enough to make me in favor of them, though; > just enough so I won't pout if they do go in. > > > Anyway, I'm not too inclined continuing this discussion anymore. > > I would never have thought that such a nifty little patch would > > have stirred up so much negative response. I'll just hack up my > > very own import mechanism using imputil and be done. So there! > > Don't feel picked on, M-A! Opposition is normal and healthy; and the last > idea I got anywhere with was list.pop(). Yah well. Last thing I succeeded with was making Python methods wrap any callable type. Seems that we're a bit too healthy (see MarkH's post) sometimes, i.e. there isn't all that much room for experiments. Just think of cool developments like Chris' stackless python. Experience shows that these kind of things will never make it into the distribution. Unfortunately, maintaing patches to the dist across releases a real pain and much work, so these ideas will just sit there unused and untested. Much the same happened to gcc ... in the end corporate strength made egcs possible. Perhaps we need such a branch too ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 106 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein@lyra.org Thu Sep 16 12:00:50 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 04:00:50 -0700 Subject: [Python-Dev] Relative Package Imports References: <000101befff4$ecc9bde0$712d153f@tim> <37E0CCC1.6EE4E599@lemburg.com> Message-ID: <37E0CDE2.56A6C6E4@lyra.org> M.-A. Lemburg wrote: > ... > Seems that we're a bit too healthy (see MarkH's post) sometimes, > i.e. there isn't all that much room for experiments. Just think > of cool developments like Chris' stackless python. Experience shows > that these kind of things will never make it into the distribution. A portion of my threading patches went into 1.5. "Big" things can happen and they do... it's just a matter of passing muster with The Head Honcho. That, and the timing (e.g. the thread state changes wouldn't go into a 1.4.n release, but did go into 1.5). > Unfortunately, maintaing patches to the dist across releases a real > pain and much work, so these ideas will just sit there unused and > untested. Much the same happened to gcc ... in the end corporate > strength made egcs possible. Perhaps we need such a branch too ? NO -g -- Greg Stein, http://www.lyra.org/ From fredrik@pythonware.com Thu Sep 16 12:32:35 1999 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 16 Sep 1999 13:32:35 +0200 Subject: [Python-Dev] CP4E References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> Message-ID: <004101bf0037$2dca30d0$f29b12c2@secret.pythonware.com> > BTW, what is this CP4E thing you're talking about. http://www.python.org/doc/essays/cp4e.html also see: http://www.deja.com/=dnc/thread/522068184 (on second thought, don't). > If it's an syntax aware editor, I have a friend who is really > interested in these things... aggressively python-aware editors are fun. want braces? sure, just change the style sheet ;-) From guido@CNRI.Reston.VA.US Thu Sep 16 12:49:44 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 07:49:44 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Thu, 16 Sep 1999 12:44:34 +0200." <37E0CA12.1E24EBF2@lemburg.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> Message-ID: <199909161149.HAA19650@eric.cnri.reston.va.us> > I'm wasn't planning to write my own hook until now. I need this > feature to be able to organize my package internal stuff swiftly > and easily, plus to be able to possibly put it under new top-level > packages. Obviously I seem to be more or less alone with this need, so > I'll stop argueing for inclusion of "something like relative imports" > in the distribution. I still hope against hoping to convince you not to do this. I think this adds lots of complexity for the reader of your package. I think of *all* software that I write as something that eventually someone is going to read and try to understand. I also think of it as something that eventually someone is going to port to a platform that wasn't even designed when I wrote the software. Tricks like depending on a custom import hook make reading a pain. For example, the reader who is just starting to browse a large package often needs to follow the thread of execution from one module to the next. If the import statements contain magic like __magic__.foo.bar (actually proposed magic renamed for generality :-), this reader will first need to understand the custom importer -- which is likely one of the hariest components of the package. The same thing applies with even more force to tools like package browsers. IDLE for example has a class browser which displays the true module and class name for base classes. E.g. if you have code that says from Tkinter import Widget class MyWidget(Widget): ... it correctly deduces that MyWidget's base class is Tkinter.Widget. (And in a future version it will probably allow you to click through to that class, show you its methods, etc.) Custom importers break this feature, and thus make the code harder to analyze for the reader. (Not all custom importers are bad; there are plenty of reasons to augment the standard importer. But, while custom importers make different interpretations of the namespace possible, I frown upon that particular use.) > (b) is (if at all) a problem only to be taken into account by > the author of package X. He may or may not use rel. imports. > A 12-year old probably won't (but then: you never know ;). Which severely limits your ability to do what you want with packages you didn't write. I know that this sounds politically incorrect in a radical free software world, but often in the end I find it more convenient to conform to the rest of the world and "fit in" than to be different in every little way. Note that my hair isn't blue :-); I've also replaced my "signature" glasses with a more conventional model. > BTW, what is this CP4E thing you're talking about. If it's an > syntax aware editor, I have a friend who is really interested > in these things... could you send me an URL that I can send him ? Where have you been? It's on the python.org homepage, has been discussed in c.l.py, c.l.tcl, even c.l.ada, in lwn, and on /.! The syntax aware editor (a proposed super-version of IDLE) is only a small part of it. See python.org/doc/essays/cp4e.html > Oh well... I guess they'll have to use 'mex' if they decide to go > the reverse domain way ;-) Or, in a worse-case scenario, the first Mexican developers using the reverse domain will probably not be aware of the mx toplevel package, and by the time their software hits the street it will be too late. > True, perhaps we should lighten this requirement a little when we > recode the import mechanism in Python ? E.g. if a local import > fails continue the search with the fully qualified name and only > if that fails, restart using the local name. This would need some > kind of fastpath cache to make the search reasonably fast though. Now this is something that I would consider. > Wouldn't it suffice to just put them into one package, e.g. > 'python.' ? And somehow make all user code live implicitly inside that package? I don't see how that solves anything. > BTW, as Tim argued: the breakage an easily be leveraged by using > a smart editor... ;-) What breakage? I think __ is a bit too hard to fix easily with a smart editor. And believe me, at some point *someone* is going to need to rip out your custom importer because it interferes with *his* custom importer, and he'll choose to replace all your __ imports with absolute package names, rather than trying to add the __ feature to his importer. --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein@lyra.org Thu Sep 16 12:57:20 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 04:57:20 -0700 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <37E0DB20.20C72337@lyra.org> Guido van Rossum wrote: >... > What breakage? I think __ is a bit too hard to fix easily with a > smart editor. And believe me, at some point *someone* is going to > need to rip out your custom importer because it interferes with *his* > custom importer, and he'll choose to replace all your __ imports with > absolute package names, rather than trying to add the __ feature to > his importer. This is an aspect of Python's singular import hook. imputil is designed to allow for multiple Importers to be present and to work in harmony. As long as each Importer can identify that a particular import is part of its namespace (e.g. HTTPImporter recognizes _http_... as one of its own), then no conflicts will arise and everything will be hunky-dory. btw, it might be nice to allow a string to the import statement. I'd much rather say: import "http://www.lyra.org/greg/python/qp_xml.py" Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Sep 16 13:00:06 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 05:00:06 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> Message-ID: <37E0DBC6.12D3EB06@lyra.org> Gordon McMillan wrote: >... > def _reload_hook(self, module): > # gmcm - Hmmm, reloading of a module may or may not > be impossible, > # (depending on the importer), but at least we can > # look to see if it's ours to reload: > if hasattr(module, '__importer__'): > if getattr(module, '__importer__') == self: > raise SystemError, "reload not yet implemented" > return self.__chain_reload(module) I've folded this in (finally). New imputil.py to be published in a bit... thx! -g -- Greg Stein, http://www.lyra.org/ From jim@interet.com Thu Sep 16 13:45:39 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 08:45:39 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274695692-31669872@hypernet.com> <37E0052F.A899E985@digicool.com> Message-ID: <37E0E673.29A844BF@interet.com> Jim Fulton wrote: > > Gordon McMillan wrote: > > > > Jim Fulton wrote: > > > [Barry] > > > > A > > > > user would drop the kjParsing directory into Shared, and then > > > > all the import kjParsing.kjParser statements would Just Work. > > > > To be a standalone application, you can use tricks that Jim > > Ahlstrom pointed out, or tricks from my Installer. The major > > point being that sys.path has nothing to do with the > > PYTHONPATH environment variable. You now have an > > installation where you can be assured of no collisions, > > because you control it completely. JimF, I am still trying to understand this. Gordon, Barry and I think a good solution is to put "Shared" on a private sys.path, but you object to this and I don't understand. Specifically zope/ zope/Shared zope/Shared/mx/... zope/Shared/AnotherPackage/... zope/Shared/some_file.py zope/ZopePackage1/... zope/ZopePackage2/... zope/bin/python.exe Suppose zope users must use the special binary zope/bin/python.exe. Suppose further that this binary reliably sets its sys.path to the directories it needs, namely zope, zope/Package*, zope/Shared. Suppose further that all user's packages must go into zope/Shared either as files or package directories and so are at the top level as package imports have always assumed. Then it seems that your zope app would absolutely work, and no other Python installation could damage it by changing PYTHONPATH, and that your zope installation would not damage any other Python installation either. Is the reason this doesn't work the posibility of name collisions in zope/Shared? I thought before that you were worried about name collisions in another co-existing Python installation which may have unknown packages installed, conflicting packages, or packages with incompatible versions. If so, you are trying to solve a problem more severe than I thought. I have always assumed packages would have unique names and so a flat name space suffices. In particular, trying to design for conflicting names can not be supported by the current python *.pyl library file design, and in general would require a PYTHONPATH mechanism. Such PYTHONPATH mechanism would have to be "local" to packages. That is, PYTHONPATH would vary depending on the package doing the importing. Are you sure this problem needs a solution right now? Jim Ahlstrom From jim@interet.com Thu Sep 16 13:57:46 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 08:57:46 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> Message-ID: <37E0E94A.A2F0D37F@interet.com> Guido van Rossum wrote: > > Even worse, it means that exceptions.py and site.py can not > > be found at all except using the normal PYTHONPATH, and > > putting their path in Spam_path will *not* work. > > Why would you want your own exceptions.py and site.py? I don't. I never change Python library files. I am worried that they won't be found because I don't trust PYTHONPATH. > > Oh dear, I think I heard no instead of yes. Are you saying that if > > someone else installs a Python app on my customer's machine after I do, > > and sets a registry entry which sayes to use c:/other/path/to/site.py > > for site.py (as he may very well want to do), then if my Python program > > depends on getting my copy of site.py from my directory, it will then > > use the other copy instead and may very well fail? > > Again - why would anyone register their own site.py? I wouldn't, I am worried that someone else will break my installation. Remember that site.py was invented as a site-specific module, although that function moved to sitecustomize.py. > I presume using GetModuleFileName()? Please send me the patch! Yes, and OK. > > JimA's conjecture: It is currently impossible to > > ship a Python app which can not be damaged by the installation of a > > second Python app without using a hacked custom binary. > > Sounds right. All tricks to make the app unique require using a > different registry key, which requires a change to the DLL. However, > you can do this without recompiling! The version string is used is > embedded in a resource, so you can patch it using some kind of > resource editor. Mark Hammond planned it this way! I don't understand this. Is there documentation? Jim Ahlstrom From guido@CNRI.Reston.VA.US Thu Sep 16 14:49:16 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 09:49:16 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Thu, 16 Sep 1999 08:57:46 EDT." <37E0E94A.A2F0D37F@interet.com> References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> Message-ID: <199909161349.JAA19789@eric.cnri.reston.va.us> [me] > > Why would you want your own exceptions.py and site.py? [JimA] > I don't. I never change Python library files. I am worried > that they won't be found because I don't trust PYTHONPATH. Hmm... PYTHONPATH gets inserted in front of the default sys.path. (Many moons ago that was different. But it has been like this for a loooooong time.) So are you worried that someone put a *different* exceptions.py or site.py on their path? > > Again - why would anyone register their own site.py? > > I wouldn't, I am worried that someone else will break my installation. > Remember that site.py was invented as a site-specific module, although > that function moved to sitecustomize.py. Hm, I dug out the oldest site.py I have (used in Python 1.4), and it doesn't encourage editing it at all -- it tells you to use sitecustomize.py. I guess they could break your installation anyway, but only by messing with the general Python installation. > > Sounds right. All tricks to make the app unique require using a > > different registry key, which requires a change to the DLL. However, > > you can do this without recompiling! The version string is used is > > embedded in a resource, so you can patch it using some kind of > > resource editor. Mark Hammond planned it this way! > > I don't understand this. Is there documentation? The usual :-) Python/import.c shows that import calls PyWin_FindRegisteredModule() to find a registered module before looking in sys.path (but after checking for builtin and frozen modules). PC/import_nt.c shows that PyWin_FindRegisteredModule() uses a registry key of the form "Software\Python\PythonCore\\Modules\" where is the module name, is empty or "\Debug" depending on whether we are compiled with _DEBUG define. The resource value points to a file (either .py, .pyc/.pyo, .pyd or .dll; in fact any of the prefixes returned by imp.get_suffixes()). PC/dl_nt.c shows that PyWin_DLLVersionString is set to string 1000 loaded from the string resource table. PC/python_nt.rc shows that there's a stringtable with item 1000 being the MS_DLL_ID string, set to "1.5" in that file. Note that this value (PyWin_DLLVersionString) is also to Python code as sys.winver. I hope that Mark Hammond can point you to a tool that you can use to edit a string resource in an executable or DLL. --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein@lyra.org Thu Sep 16 14:46:19 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 06:46:19 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <37E0F4AB.38EAB65F@lyra.org> M.-A. Lemburg wrote: >... > The original version of imputil I fetched from Greg's page > did work out of the box (from...import... hassles) and "did not work", I presume. From my original testing, I thought from...import worked. With more testing, I found that something of the form "from xml.dom import builder" did not work. I discovered why it failed (xml.dom was imported by Importer instance I1 but I2 thought it could handle the from...import, and this barfed a check). I've fixed this by delegating to the proper importer (I1 in my example) to complete the import. Your solution to check the __importer__ variable in the globals is probably incorrect. If I read/eval it correctly, that would mean that a module imported by IMP1 could not use modules imported by IMP2. In other words, a package module could not import a top-level module defined by a different importer. (note also that your globals.get() could fail if globals is None) > obviously did not support in-package shared libs. I've added I did not fold this in. Your change isn't "in the spirit" of the Importer mechanism. The "Right Way" to do this is to create a BuiltinImporter and add that to the chain of importers. The DirectoryImporter should only import from directories -- no reason for it to know about builtin stuff. As a result, I did not accept the new methods on Importer for handling builtins/special modules -- those would go in the BuiltinImporter. [BuiltinImporter should be written and included in imputil.py; I don't really have the time at the moment to write the thing... 7am and time for sleep...] However, your change here did raise a very important design issue: get_code() needs to be able to return a loaded module, rather than just a code object. I've folded in your patches for that. I also folded in many of your extended doc/comments (at least in concept; not necessarily verbatim). You and Gordon are recognized in the header now, and I've added a "proper" author notice and licensing (public domain). I did not include the "misses" feature that you added to the DirectoryImporter. I would hate to see a miss-cache get loaded, a module dropped into the filesystem, and the user never being able to import the thing. I didn't fold in your indentation changes or name changes. I liked mine :-). The __main__ thing at the bottom didn't make much sense to me, though, since the call to _test_dir() followed by an exit doesn't really do anything. And yes, I recognize that you can use "python -i imputil.py" but I'd rather just see "python" followed by "import imputil ; imputil._test_dir()". Of course, please feel free to generate a new patch if I've missed something (thinking about it, I missed the OSError thing). > both features so that the test script in DateTime can run > successfully. > > Things that remain are: > · the win32 registry stuff (needs C code) And a new Importer to use it. > · the Mac fork stuff (needs C code) Ditto. > · a working __path__ implementation (is anyone using this attribute > which only is available in packages ?) Per the private mail that I sent to you: I explicitly punted on the __path__ attribute. It can lead to *way* too much confusion. It is also unavailable for frozen packages (boy oh boy did the win32com get some ugliness in there to compensate for being frozen w.r.t. its use of __path__). The DirectoryImporter can insert the attribute, but it definitely wouldn't go into the Importer itself. The __path__ attribute is specific to loading from a filesystem, yet Importer is generic. > · probably a whole bunch of other quirks > · some speedups (there currently are too many stat()s) Yes. I recognize that the "misses" feature was intended to remedy this. I don't have an immediate answer to the stat() issue. Does the Importer mechanism actually perform more stats on an import than Python itself? (it looks like it does one for the isdir() plus two for fetching file timestamps) And a big thanx: I appreciate the patches to imputil! The new module is now available in its "official" location at http://www.lyra.org/greg/python/ Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim@interet.com Thu Sep 16 14:56:00 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 09:56:00 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> Message-ID: <37E0F6F0.8FC49D4@interet.com> Greg Stein wrote: > > Gordon McMillan wrote: > >... > > def _reload_hook(self, module): > > # gmcm - Hmmm, reloading of a module may or may not > > be impossible, > > # (depending on the importer), but at least we can > > # look to see if it's ours to reload: > > if hasattr(module, '__importer__'): > > if getattr(module, '__importer__') == self: > > raise SystemError, "reload not yet implemented" > > return self.__chain_reload(module) > > I've folded this in (finally). > > New imputil.py to be published in a bit... As a result of all this import discussion I am a bit worried that the python library *.pyl file format may not be powerful enough. I have always thought in terms of unique top-level names and a format which supports import of modules and packages. But this does not support the full functionality of PYTHONPATH. For example, PYTHONPATH can (and is) used to select the correct plat-* directory files. And the format may not support Jim Fulton's fancy local import scheme. And what if someone invents a third thing to import besides a module or a package? PYTHONPATH is not going away nor should it. How about if the *.pyl file format is exactly a directory structure? I mean that the table of contents is limited to paths starting with a directory name only, and that the seperator is '/' instead of '.'. So a listing would be identical to the output of 'ls -R'. So: Lib/string.pyc Lib/exceptions.pyc Lib/plat-sunos4/... mx/__init__.pyc mx/... package2/... dir3/... ... The implied PYTHONPATH for this file is ["Lib", "."]. Since the format is exactly a directory tree, it is guaranteed that whatever PYTHONPATH or imports can do now or in the future with a directory tree, it can still do it with a *.pyl file. Jim Ahlstrom From gstein@lyra.org Thu Sep 16 15:05:42 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 07:05:42 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> <37E0F6F0.8FC49D4@interet.com> Message-ID: <37E0F936.6D32E983@lyra.org> James C. Ahlstrom wrote: >... > As a result of all this import discussion I am a bit worried that > the python library *.pyl file format may not be powerful enough. Background for the readers: .pyl is an extension that I used in my "small" distribution. I think Gordon uses it, too. In any case, it is effectively a concatenation of .pyc files along with a TOC mapping fully-qualified dotted module names to seek-positions within the file. [speaking of stat() calls: using a .pyl eliminates them quite nicely -- this may be part of Gordon's observed speed increase when using an archive] The .pyl format was discussed a bit on the distutils-sig list and "sort of" accepted as an okay format for jamming a bunch of modules into a single file. [by "sort of", I mean that the small group who participated in the discussion were okay with it :-); it is a great, minimalist format, so it probably won't please people who like a ton of features in a file format :-) ] >... > How about if the *.pyl file format is exactly a directory structure? > I mean that the table of contents is limited to paths starting with > a directory name only, and that the seperator is '/' instead of '.'. > So a listing would be identical to the output of 'ls -R'. So: > Lib/string.pyc > Lib/exceptions.pyc > Lib/plat-sunos4/... > mx/__init__.pyc > mx/... > package2/... > dir3/... > ... > > The implied PYTHONPATH for this file is ["Lib", "."]. Since the > format is exactly a directory tree, it is guaranteed that whatever > PYTHONPATH or imports can do now or in the future with a directory > tree, it can still do it with a *.pyl file. People import things using a dotted name. Therefore, I think it makes the most sense to map that straight to the resulting .pyc file. No reason to put directories into the file... they make no sense to the end user. During construction of the .pyl, you would walk the tree finding all the available modules (and their corresponding dotted name) and insert them. Note that you can distribute multiple .pyl files. There could be the Python standard lib in one file, the mx package in another, etc. As a module is searched for, the system just peeks into each .pyl in turn, looking for the module. Search order is currently defined by order of install() on the Importer instances. I believe the Right Way to do things is to create sys.importers (as a list of Importers) and deprecate the sys.path variable. Python could start up with an Importer than simply scanned sys.path as a backwards compat measure; it could also leave sys.path empty and create DirectoryImporters for each path component (this could cause problems, though, for some apps that believe sys.path shouldn't be empty, or that use it for magic-munging). I've search the standard lib in the past -- there are only a couple real uses of sys.path if I remember rightly (test package and the traceback module). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Sep 16 15:19:41 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 07:19:41 -0700 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> <199909161349.JAA19789@eric.cnri.reston.va.us> Message-ID: <37E0FC7D.7494B8F7@lyra.org> Guido van Rossum wrote: > > [me] > > > Why would you want your own exceptions.py and site.py? > [JimA] > > I don't. I never change Python library files. I am worried > > that they won't be found because I don't trust PYTHONPATH. > > Hmm... PYTHONPATH gets inserted in front of the default sys.path. > (Many moons ago that was different. But it has been like this for a > loooooong time.) So are you worried that someone put a *different* > exceptions.py or site.py on their path? When you jam every file into a module archive, you still have to leave these two "outside" so that Python can find them when starting up. The problem then breaks down into two parts: 1) locating them 2) ensuring they're the right versions In my "small" distro, I relied on the current-dir always being in the path, so I could always find these. The two files were distributed as part of distro. Anyhow: JimA is saying that he doesn't trust PYTHONPATH. Not so much bad files/versions, but that he won't be able to find them because PYTHONPATH has been monkeyed in some way. > > > Again - why would anyone register their own site.py? > > > > I wouldn't, I am worried that someone else will break my installation. > > Remember that site.py was invented as a site-specific module, although > > that function moved to sitecustomize.py. > > Hm, I dug out the oldest site.py I have (used in Python 1.4), and it > doesn't encourage editing it at all -- it tells you to use > sitecustomize.py. I guess they could break your installation anyway, > but only by messing with the general Python installation. If the file exists, then somebody can mess with it. JimA is trying to create a bulletproof app here. To do this, you can distribute a python.exe, exceptions.py, site.py, and an archive of your other modules. site.py is scrapped in favor of installing an Importer to access the archive (this implies you also distribute imputil.py). These five files are the exact five in my "small" distro. It's pretty cool... no need for registry changes and a very small "file count" footprint. Gordon took this basis and added a bunch of features for bundling an application in there. JimA has also been mixing in some frozen modules (I forget exactly why/what). [oh, my small distro doesn't ship a python15.dll, although it easily could] >... > I hope that Mark Hammond can point you to a tool that you can use to > edit a string resource in an executable or DLL. The win32api module has resource manipulation functions such as BeginUpdateResource, UpdateResource, and EndUpdateResource. Write a Python script to modify your version string :-) A demo of resource munging can be seen in /win32/scripts/VersionStamp/verstamp.py. Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim@digicool.com Thu Sep 16 15:56:53 1999 From: jim@digicool.com (Jim Fulton) Date: Thu, 16 Sep 1999 10:56:53 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <37E10535.CA96A77E@digicool.com> "M.-A. Lemburg" wrote: > > · a working __path__ implementation (is anyone using this attribute > which only is available in packages ?) Yes. I use it for two things: - I modify it to allow a (logical) package to be spread over multiple physical locations. (In Zope, products can be installed in the Zope installation area or in Zope "instance" homes. - I use it to determine the location(s) of a package. Our packages usually contain many files, such as DTML source files, images, data files, etc., that are not Python modules. We have standard utilities for getting at these files in packages. This is extremely useful. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim@interet.com Thu Sep 16 15:58:48 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 10:58:48 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> <199909161349.JAA19789@eric.cnri.reston.va.us> Message-ID: <37E105A8.A2B06A80@interet.com> Guido van Rossum wrote: > Hmm... PYTHONPATH gets inserted in front of the default sys.path. > (Many moons ago that was different. But it has been like this for a > loooooong time.) So are you worried that someone put a *different* > exceptions.py or site.py on their path? Yes, and/or (2) added a sitecustomize.py with their special import hook as has been proposed here over and over, or (3) PYTHONPATH is screwed up and doesn't find anything. Perhaps this is a prejudice of mine. I just look at "print sys.path" and marvel at what I don't understand. At least I can see it is not simple. I hate relying on thing that are not simple. And I hate custom import hooks. Unless they are mine of course ;-) > Hm, I dug out the oldest site.py I have (used in Python 1.4), and it > doesn't encourage editing it at all -- it tells you to use > sitecustomize.py. I guess they could break your installation anyway, > but only by messing with the general Python installation. Just adding sitecustomize.py would do it. And this is encouraged. I think Gordon put his finger on the issue. Either try to co-exist with other installed Python software and take the risk that everyone is playing by the rules, or build your own black-box self-contained Python world and duplicate storage. > The usual :-) > [Documentation goes here...] Thanks. This is very useful. But it doesn't help, perhaps as a result of more of my prejudices. This registry entry is meant to be used by a by-the-rules shared Python installation, so I must not change it. And I hate changing any registry entries at all. My commercial software keeps all its settings in a regular .ini file in its install directory, and makes only minimal and required registry entries. IMHO the Windows registry is a software catastrophe which ranks right up there with JCL (anyone else here old enough to remember that?). Anyone who doesn't agree should go with me to our money center banking clients, and sit there while they grill you on every registry entry and why it is required. Money center banking clients do not like their registry messed with. I do however see your point that I could change the version string to something non-standard and use the registry to control imports. I will think about this further. Maybe it would work. My current "solution" is to use freeze to create a black-box install, and worry about second Python installations and wasted storage when it happens. I was hoping that this thread whould result in a consensis of what to do, but it has not. So now I am hoping that Python library (jar) files will turn out to be a practical solution, so I am pestering Greg and Gordon. We'll see. Jim Ahlstrom From bwarsaw@python.org Thu Sep 16 16:16:42 1999 From: bwarsaw@python.org (Barry A. Warsaw) Date: Thu, 16 Sep 1999 11:16:42 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> Message-ID: <14305.2522.922445.932104@anthem.cnri.reston.va.us> >>>>> "JF" == Jim Fulton writes: >> I think we're safe from this for the forseeable future through >> convention and peer pressure. JF> 8^o JF> (I feel like I should respond to this in some way but .... JF> I just can't understand this kind of argument and don't JF> know how to frame a response.) Maybe I should have included half a smiley. Python may eventually need package naming rules as strict as Java's. I think it's a good idea now for those who are writing packages to consider trying to pick unique top-level package names. We might decide to release our stuff under the `cnri' top level package. That's probably unique enough. Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! Personally, I think `Mailman' as a top-level package name is pretty unique, so I don't plan to change what I'm doing there. -Barry From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Thu Sep 16 16:37:01 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Thu, 16 Sep 1999 11:37:01 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> <37E049CD.6B64DE00@lyra.org> Message-ID: <14305.3741.741370.948251@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Definitely having an OSI certification will be great (cool GS> stuff on the cert for the existing license!). Having Bruce GS> Perens review the license would also be a great boon (see GS> www.perens.com for some of his writings; also see GS> http://perens.com/Termination.html specifically). Interesting article, but IBM's termination clause was different than the JPython one. I fought hard on para7 because IIRC, RMS complained that an earlier version /could/ have been used to arbitrarily terminate. I think the current JPython para7 is better because /you/ have to materially breach, which seems like a much higher threshold. But it still may not be perfect. Aside: don't necessarily think I'm a grinning fan and defender of the JPython license. It's a huge win over what we had before, and I think it's good enough that nearly anybody who wants to do Real Things with JPython, now can. I've had only one question about the license since it was published and that was about the "displayed prominently" clause (i.e. was it okay to include the alternative handle text in an "about" menu pulldown? That seemed prominent enough to my nonlawyerly brain.) I'm glad to see the Python community push hard for the "other side's viewpoint" with reasoned and rational arguments. I think that such responses from Influential Python Users will provide us with useful ammunition when we re-evaluate the licenses. It means that ultimately we'll have the right license for Python (and JPython). Thanks, -Barry From skip@mojam.com (Skip Montanaro) Thu Sep 16 16:54:39 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 16 Sep 1999 10:54:39 -0500 (CDT) Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37E0F6F0.8FC49D4@interet.com> References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> <37E0F6F0.8FC49D4@interet.com> Message-ID: <14305.4799.328120.623217@dolphin.mojam.com> Jim> As a result of all this import discussion I am a bit worried that Jim> the python library *.pyl file format may not be powerful enough. Not to rain on anyone's parade, but I want to remind the folks having this discussion that there are people reading this thread that while fairly well versed in Python have little idea what anyone is talking about anymore. (I don't know. Maybe I'm the only one.) python-dev is clearly the best place to discuss this in the short-term (anyone for an import SIG?), but whatever is implemented will have to be understood by lots of people on c.l.py to be of broad applicability. Perhaps I'm way off base and there are more than a handful of people who will ever run into the problems being solved here, but if we can partition the Python programming community into the package wizards and the mere import mortals, I worry that the potions concocted by the wizards will send a few of us import mortals to the hospital... The Java package scheme, while odious to some perhaps, is extremely easy to understand for anyone who's ever used Windows Explorer or execute "ls -R". just-a-cautionary-peanut-thrown-in-from-the-bleachers-ly y'rs Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From skip@mojam.com (Skip Montanaro) Thu Sep 16 18:01:58 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 16 Sep 1999 12:01:58 -0500 (CDT) Subject: [Python-Dev] Relative Package Imports In-Reply-To: <14305.2522.922445.932104@anthem.cnri.reston.va.us> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> <14305.2522.922445.932104@anthem.cnri.reston.va.us> Message-ID: <14305.8838.329883.696911@dolphin.mojam.com> BAW> Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! Umm, that would be "Knowbots Over Everything" or "Knowledge Of Everyone"? ;-) Skip From guido@CNRI.Reston.VA.US Thu Sep 16 18:04:37 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 13:04:37 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Thu, 16 Sep 1999 12:01:58 CDT." <14305.8838.329883.696911@dolphin.mojam.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> <14305.2522.922445.932104@anthem.cnri.reston.va.us> <14305.8838.329883.696911@dolphin.mojam.com> Message-ID: <199909161704.NAA01253@eric.cnri.reston.va.us> > BAW> Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! > Skip> Umm, that would be "Knowbots Over Everything" or "Knowledge Of Everyone"? Actually, "koe" is dutch for cow. I like the idea of cows over everything -- reminds me of the flying pig on an old Pink Floyd cover :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein@lyra.org Thu Sep 16 18:27:49 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 10:27:49 -0700 Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> <37E049CD.6B64DE00@lyra.org> <14305.3741.741370.948251@anthem.cnri.reston.va.us> Message-ID: <37E12895.443CC6F3@lyra.org> Barry A. Warsaw wrote: >... > Interesting article, but IBM's termination clause was different than > the JPython one. I fought hard on para7 because IIRC, RMS complained > that an earlier version /could/ have been used to arbitrarily > terminate. I think the current JPython para7 is better because /you/ > have to materially breach, which seems like a much higher threshold. > But it still may not be perfect. Yes, I was aware that it was a reactive termination, rather than arbitrary. That makes it quite acceptable, but it still isn't a desirable thing. Especially given some of the grey area in the license ("are we sure we aren't in breach of the license?"). Personally, I'd rather see a license without a termination clause. If it must be there, then I'd like to see it as tight as possible (see the IBM and Apple licenses: IIRC, they only kick in when the user initiates patent litigation against IBM/Apple; the termination cuts them off as an initial response to the suit). The other elements I raised actually caused me more anxiety than the termination. If CNRI finds it acceptable, I'd recommend they use an existing OSD license. They get immediate certfication and, more importantly, a builtin awareness in the open source community of what the license really means. Each time a new license arrives in the community, bunches of people have to go an figure it out; if the new license is the IBM Public License with a search/replace on the company and product name, then people go "oh. all righty. no problem." and move on to doing real stuff. Dang. I keep replying to this stuff. :-) I'm hoping that we wrap this up pending a new release. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gmcm@hypernet.com Thu Sep 16 18:38:03 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37E0F6F0.8FC49D4@interet.com> Message-ID: <1274619156-36273405@hypernet.com> Jim Ahlstrom wrote: > As a result of all this import discussion I am a bit worried that > the python library *.pyl file format may not be powerful enough. > I have always thought in terms of unique top-level names and a > format which supports import of modules and packages. But this > does not support the full functionality of PYTHONPATH. For > example, PYTHONPATH can (and is) used to select the correct > plat-* directory files. And the format may not support Jim > Fulton's fancy local import scheme. And what if someone invents > a third thing to import besides a module or a package? > PYTHONPATH is not going away nor should it. The central idea of imputil is that an importer is responsible for one little chunk of turf. If the desired module / package isn't "his", he just passes the request on to the next element in the chain. So I don't think there's a need for one canonical do-everything importer (or archive format). PYTHONPATH is outside any particular importer. Effectively, you can use a chain of importers to replace PYTHONPATH. So the platform specific modules might be found by one particular importer. In other words, I think it's more effective to specialize individual importers and chain them up than it is to try to create an overly-generalized importer. - Gordon From gmcm@hypernet.com Thu Sep 16 18:38:03 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E0FC7D.7494B8F7@lyra.org> Message-ID: <1274619162-36273021@hypernet.com> [Greg, replying to Guido's confusion on Jim's interest in site.py and exceptions.py] > When you jam every file into a module archive, you still have to > leave these two "outside" so that Python can find them when > starting up. In the soon-to-be-published beta version of my installer, I've got that down to exceptions.py. Background: my (Win32) installer has, as a stated goal, the ability to create quasi-frozen Python apps which won't interfere with (or be influenced by) existing Python installations (if any). And it doesn't require the programmer/user to have a compiler. Thanks to patches given me by Thomas Heller, my python.exe replacement is now a (minimal) embedding app, and I do the same things that Greg does in site.py directly from C code, (and turn off the SiteFlag, too). I believe that if I freeze in exceptions.py and tweak the resource in python15.dll (which is just the stock python15.dll), I can have a completely safe executable. I think the same techniques can be applied on *nix, (although I'm pretty sure programmer/users won't be able to get away without a compiler). This gives me a strong interest in import hooks for two distinct reasons: - I rely completely on Greg's imputil to make this work. - I rely on freeze's modulefinder to help build these things. Bizarre import hooks in a normal Python installation will fool modulefinder. Take a look at what Pmw does (fortunately, Pmw comes with it's own packager). So I'd like to see import hooks follow some sort of pattern that can be followed by a tool like modulefinder. I also want the hooks. Right now I use imputil with archives, but wouldn't it be cool if you could add another imputil importer that checks for more recent versions at some home site on the web and automatically updates the installation? Summary: I'm very interested in seeing import and import hooks get rationalized, and I think Greg's stuff goes a long, long way towards that goal. - Gordon From gmcm@hypernet.com Thu Sep 16 18:38:03 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E105A8.A2B06A80@interet.com> Message-ID: <1274619161-36273101@hypernet.com> James C. Ahlstrom wrote: [Guido explains Windows registry usage] > Thanks. This is very useful. > > But it doesn't help, perhaps as a result of more of my > prejudices. This registry entry is meant to be used by a > by-the-rules shared Python installation, so I must not change it. Without trying it, I doubt you have to. It looks like you could set the resource to something that won't be found in the registry, and then just use normal Python mechanisms. > ... IMHO the Windows registry is a software catastrophe > which ranks right up there with JCL (anyone else here old enough > to remember that?). Ah. In a rush to get it down to ops, tripping at the head of the stairs and sending the card deck flying... Those were the days. - Gordon From skip@mojam.com (Skip Montanaro) Thu Sep 16 18:49:12 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 16 Sep 1999 12:49:12 -0500 (CDT) Subject: [Python-Dev] Re: Path hacking In-Reply-To: <1274619162-36273021@hypernet.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> Message-ID: <14305.11672.687570.290221@dolphin.mojam.com> Gordon> In the soon-to-be-published beta version of my installer, I've Gordon> got that down to exceptions.py. Why not just run exceptions.py through Python2C, visually and experimentally verify that it works, then ship an exceptions.c as an optional module? People wanting to ship self-contained packages could then toss exceptions.py and build the C version of the exceptions module. Greg, is there anything in exceptions.py Python2C couldn't handle? Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From gstein@lyra.org Thu Sep 16 18:47:56 1999 From: gstein@lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 10:47:56 -0700 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <37E12D4C.2394D311@lyra.org> Skip Montanaro wrote: > > Gordon> In the soon-to-be-published beta version of my installer, I've > Gordon> got that down to exceptions.py. > > Why not just run exceptions.py through Python2C, visually and experimentally > verify that it works, then ship an exceptions.c as an optional module? > People wanting to ship self-contained packages could then toss exceptions.py > and build the C version of the exceptions module. > > Greg, is there anything in exceptions.py Python2C couldn't handle? Nah, shouldn't have any problem at all. P2C will even create true class objects and expose them in the interface. I think a person might want to consider hand-tuning the output, though :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim@interet.com Thu Sep 16 19:03:13 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 14:03:13 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <37E130E1.F21CDE8B@interet.com> Skip Montanaro wrote: > Why not just run exceptions.py through Python2C, What is Python2C. Is it the same as freeze? Jim Ahlstrom From guido@CNRI.Reston.VA.US Thu Sep 16 19:08:01 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 14:08:01 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Thu, 16 Sep 1999 12:49:12 CDT." <14305.11672.687570.290221@dolphin.mojam.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <199909161808.OAA01452@eric.cnri.reston.va.us> > Why not just run exceptions.py through Python2C, visually and experimentally > verify that it works, then ship an exceptions.c as an optional module? > People wanting to ship self-contained packages could then toss exceptions.py > and build the C version of the exceptions module. Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@mojam.com (Skip Montanaro) Thu Sep 16 19:34:47 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 16 Sep 1999 13:34:47 -0500 (CDT) Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E130E1.F21CDE8B@interet.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <37E130E1.F21CDE8B@interet.com> Message-ID: <14305.14407.189346.498215@dolphin.mojam.com> Jim> What is Python2C. Is it the same as freeze? Nope. Python in, compilable C out: http://www.mudlib.org/~rassilon/p2c/ Courtesy of Greg Stein and Bill Tutt. Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From jim@interet.com Thu Sep 16 19:37:58 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 14:37:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <199909161808.OAA01452@eric.cnri.reston.va.us> Message-ID: <37E13906.E2E8FB2E@interet.com> Guido van Rossum wrote: > Alternatively (and probably easier) it (and site.py) could be stored > as frozen modules. All it takes is some edits to Python/frozen.c. An excellent design, very simple. A large list of critical Python-language modules including a new importer, imputils.py and a new main could be reliably linked to their interpreter. It would open the door writing a larger part of Python in Python. But editing Python/frozen.c will kill the current freeze feature since a user currently replaces this pointer with their own. We can have a second _PyImport_InternalFrozenModules[] array which is searched FIRST, so that the existing frozen modules feature is retained. Current logic only allows one frozen module array. This is easy. A little harder is turning it off, which may be important to developers. They will probably want to use site.py etc. in PYTHONPATH directories. I would hate to use another command line option. Maybe another method in the imp module, say imp.EnablePyInternalLib(x) to turn on/off the internal frozen modules. A method to print the names in the list wouldn't hurt either. At one time I had code for multiple frozen modules which I may be able to find. Jim Ahlstrom From jim@interet.com Thu Sep 16 20:09:57 1999 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 15:09:57 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <1274619156-36273405@hypernet.com> Message-ID: <37E14085.FE46EB12@interet.com> Gordon McMillan wrote: > So I don't think there's a need for one canonical do-everything > importer (or archive format). PYTHONPATH is outside any > particular importer. Effectively, you can use a chain of > importers to replace PYTHONPATH. So the platform specific > modules might be found by one particular importer. In other > words, I think it's more effective to specialize individual > importers and chain them up than it is to try to create an > overly-generalized importer. Greg agrees with you so I defer to the experts on importers. The feature is meant to support a chain. Greg wrote: > The .pyl format was discussed a bit on the distutils-sig list and "sort > of" accepted as an okay format for jamming a bunch of modules into a > single file. > [by "sort of", I mean that the small group who participated in the > discussion were okay with it :-); it is a great, minimalist format, so > it probably won't please people who like a ton of features in a file >format :-) ] But I still disagree on the .pyl file format. If there is no Standard Format and everyone is linking in his own importer, then we will have exactly the same situation we have now with PYTHONPATH and novel import hooks. There should be a Standard Format to fix this problem. In particular, package authors should be able to publish packages as PYL files and expect them to be usable as is with no further effort. Sysadmins should be able to manage everything PYTHONPATH does with a small (one?) number of PYL files and in a standard way. Jim Ahlstrom From fredrik@pythonware.com Fri Sep 17 08:40:18 1999 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 17 Sep 1999 09:40:18 +0200 Subject: [Python-Dev] Re: Path hacking References: <1274619162-36273021@hypernet.com> Message-ID: <00f801bf00df$e45361c0$f29b12c2@secret.pythonware.com> > Summary: I'm very interested in seeing import and import > hooks get rationalized, and I think Greg's stuff goes a long, > long way towards that goal. don't have time to say much more than: -- ME TOO! (we're also doing this in the pythonworks packager, and Greg's module is essential for that work. might as well turn it into a standard Python feature). From mhammond@skippinet.com.au Fri Sep 17 08:39:51 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 17 Sep 1999 17:39:51 +1000 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E0E94A.A2F0D37F@interet.com> Message-ID: <000501bf00df$d41cafa0$0801a8c0@bobcat> > > you can do this without recompiling! The version string is used is > > embedded in a resource, so you can patch it using some kind of > > resource editor. Mark Hammond planned it this way! > > I don't understand this. Is there documentation? Vague documentation at http://www.python.org/windows/python/registry.html Doesnt explicitely say much about this particular issue, and if I remember I will update it. VC will allow you to edit these resources without rebuilding - simply open the DLL, but select "Resources" as the file type. I havent investigated other resource editors as I have MSVC :-) Greg pointed you to the win32api functions - they do work, as Greg provided code that uses them (the code he pointed at was done by him) - although I never actually worked out how to make them work ( I spent about an hour trying to get code to re-stamp the Python15.dll with a new sys.winver, and gave up. Would be cool to get going tho... Mark. From mhammond@skippinet.com.au Fri Sep 17 08:48:26 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 17 Sep 1999 17:48:26 +1000 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E105A8.A2B06A80@interet.com> Message-ID: <000601bf00e1$072a7d90$0801a8c0@bobcat> > But it doesn't help, perhaps as a result of more of my prejudices. > This registry entry is meant to be used by a by-the-rules shared > Python installation, so I must not change it. No - the point is that the "1.5" key is "reserved" by a standard install. Changing the string value actually allows you to have your own subtree, and you can assume you own that. For commercial apps I help with, re stick a "AppName" in this string (doesnt have to be a number) then simply use "HKLM\Software\Python\AppName" > And I hate changing any registry entries at all. Well, you should learn to get over it! That is what it is designed for. sure, we dont want the users to muck with the registry, but if you dont like your install scripts working with the registry, I definately feel you should get over this prejudice of yours. > only minimal and required registry entries. IMHO the Windows registry > is a software catastrophe which ranks right up there with JCL (anyone > else here old enough to remember that?). Anyone who doesn't agree > should > go with me to our money center banking clients, and sit there > while they > grill you on every registry entry and why it is required. > Money center > banking clients do not like their registry messed with. The registry is a huge bucket. Changing your own, custom kep under either HKLM\Software\ or HKCU\Software is perfectly reasonable. Many apps _require_ you to hit the registry - services, event log, performance monitor, etc. The fact the registry _also_ contains critical information is a problem, and sure, we dont want to mess with that. > My current "solution" is to use freeze to create a black-box install, > and worry about second Python installations and wasted storage when it > happens. well, IMO this is also the correct thing to do. any install that has >100 files is fragile. So I do both - freeze the app, _and_ a custom "sys.winver". > I was hoping that this thread whould result in a consensis of what > to do, but it has not. There is a consesus for people with the same problem. Different problems have different optimal solutions. Mark. From mal@lemburg.com Fri Sep 17 10:33:43 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 11:33:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <37E20AF7.2D05F371@lemburg.com> Guido van Rossum wrote: > > > I'm wasn't planning to write my own hook until now. I need this > > feature to be able to organize my package internal stuff swiftly > > and easily, plus to be able to possibly put it under new top-level > > packages. Obviously I seem to be more or less alone with this need, so > > I'll stop argueing for inclusion of "something like relative imports" > > in the distribution. > > I still hope against hoping to convince you not to do this. I think > this adds lots of complexity for the reader of your package. I think > of *all* software that I write as something that eventually someone is > going to read and try to understand. I also think of it as something > that eventually someone is going to port to a platform that wasn't > even designed when I wrote the software. Tricks like depending on a > custom import hook make reading a pain. > > For example, the reader who is just starting to browse a large package > often needs to follow the thread of execution from one module to the > next. If the import statements contain magic like __magic__.foo.bar > (actually proposed magic renamed for generality :-), this reader will > first need to understand the custom importer -- which is likely one of > the hariest components of the package. While it will perhaps look funny to him at first sight, I'm sure that a proper name will make the idea clear to him: from __superpackage__.foo import bar or just from __super__.foo import bar [Super Foo -- sounds like a comic star ... there's even integrated fun in these names ;-)] > The same thing applies with even more force to tools like package > browsers. IDLE for example has a class browser which displays the > true module and class name for base classes. E.g. if you have code > that says > > from Tkinter import Widget > > class MyWidget(Widget): ... > > it correctly deduces that MyWidget's base class is Tkinter.Widget. > (And in a future version it will probably allow you to click through > to that class, show you its methods, etc.) > > Custom importers break this feature, and thus make the code harder to > analyze for the reader. That's why I tried argue for making it a standard: all tools will then know about the syntax used and take appropriate action. > (Not all custom importers are bad; there are plenty of reasons to > augment the standard importer. But, while custom importers make > different interpretations of the namespace possible, I frown upon that > particular use.) > > > (b) is (if at all) a problem only to be taken into account by > > the author of package X. He may or may not use rel. imports. > > A 12-year old probably won't (but then: you never know ;). > > Which severely limits your ability to do what you want with packages > you didn't write. > > I know that this sounds politically incorrect in a radical free > software world, but often in the end I find it more convenient to > conform to the rest of the world and "fit in" than to be different in > every little way. Note that my hair isn't blue :-); I've also > replaced my "signature" glasses with a more conventional model. Naa, I don't want to be different just to be plain silly different. The case here is one of maintenance. I have around 12 Python packages which I maintain, some of them public, others proprietary. Due to the large number I try to make them all as easy to handle as possible. Now I can't just go and rename the imports in case I want to put them under some other common package name for application A and then do the same with another package name for application B. I would have to do the same thing for every new release of the packages since there are also others who use the standard mx-based name. I just want the packages to work whereever I put them, e.g. I could link them into the new location using symlinks and be done with the work. Of course there are other ways to make this work (and I want to thank everybody who contributed to the sys.path ideas), but they just add omplexity to a whole different area. > > BTW, what is this CP4E thing you're talking about. If it's an > > syntax aware editor, I have a friend who is really interested > > in these things... could you send me an URL that I can send him ? > > Where have you been? It's on the python.org homepage, has been > discussed in c.l.py, c.l.tcl, even c.l.ada, in lwn, and on /.! The > syntax aware editor (a proposed super-version of IDLE) is only a small > part of it. See python.org/doc/essays/cp4e.html Nope, haven't heard anything about it -- perhaps I was on vacation. Sound interesting though. An editor where you can just type away (and the editor fills in the gaps) would be great fun. > > Oh well... I guess they'll have to use 'mex' if they decide to go > > the reverse domain way ;-) > > Or, in a worse-case scenario, the first Mexican developers using the > reverse domain will probably not be aware of the mx toplevel package, > and by the time their software hits the street it will be too late. Well, I guess then they'll have a good reason to upgrade their software. Just like all those companies who made extra money by sticking a "Y2000 + Euro-capable" button on their software. No, seriously, with the proposed extension (see below) this shouldn't be much of a problem because my packages all use capital first letters. The reverse domain name would use lower case ones, e.g. mx.pythonware.pilios or mx.zopistas... > > True, perhaps we should lighten this requirement a little when we > > recode the import mechanism in Python ? E.g. if a local import > > fails continue the search with the fully qualified name and only > > if that fails, restart using the local name. This would need some > > kind of fastpath cache to make the search reasonably fast though. > > Now this is something that I would consider. Should be easy to add to the imputil based core emulation. > > Wouldn't it suffice to just put them into one package, e.g. > > 'python.' ? > > And somehow make all user code live implicitly inside that package? I > don't see how that solves anything. No, to free the top-level namespace and make the origin of the imports clearer, e.g. import python.image import PIL.image import zope.image etc. > And believe me, at some point *someone* is going to > need to rip out your custom importer because it interferes with *his* > custom importer, and he'll choose to replace all your __ imports with > absolute package names, rather than trying to add the __ feature to > his importer. Note that with imputil doing its thing, he won't have to worry about adding code to his importer. It'll just pass on the import to my importer. That's the fun part about imputil. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Vladimir.Marangozov@inrialpes.fr Fri Sep 17 16:07:19 1999 From: Vladimir.Marangozov@inrialpes.fr (Vladimir Marangozov) Date: Fri, 17 Sep 1999 16:07:19 +0100 (NFT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909151843.OAA18364@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 15, 99 02:43:56 pm" Message-ID: <199909171507.QAA33468@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > I hereby withdraw the posted license. There still is the need for a > new license, but we need to go back to the drawing board for it. So, in the end, am I still invited to sign & send the "wet" form or I'd better wait to let it dry? BTW, I'm surprised by the fact that in an Open Source world I'm asked to sign a licence agreement with CNRI or to send e-mails for contributed code. If Python or Linux had had such constraints from the start, they wouldn't have been what they are today. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From guido@CNRI.Reston.VA.US Fri Sep 17 15:07:50 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 17 Sep 1999 10:07:50 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Fri, 17 Sep 1999 16:07:19 BST." <199909171507.QAA33468@pukapuka.inrialpes.fr> References: <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <199909171407.KAA00575@eric.cnri.reston.va.us> > So, in the end, am I still invited to sign & send the "wet" form or > I'd better wait to let it dry? Please send in the form -- the license was a totally separate issue that I shouldn't have brought up in the same mail (or at all, in this stage anyway -- we'll work this out with the Python consortium members first). > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. Unfortunately, that's the price we have to pay. What we get is legal protection from CNRI. In general CNRI has contributed a lot to Python; probably more than you realize. In any case, signing the form and including the email paragraphs is completely voluntary -- if you don't want to do it, just let me know. --Guido van Rossum (home page: http://www.python.org/~guido/) From Vladimir.Marangozov@inrialpes.fr Fri Sep 17 16:35:01 1999 From: Vladimir.Marangozov@inrialpes.fr (Vladimir Marangozov) Date: Fri, 17 Sep 1999 16:35:01 +0100 (NFT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909171407.KAA00575@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 17, 99 10:07:50 am" Message-ID: <199909171535.QAA30684@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > Please send in the form -- the license was a totally separate issue > that I shouldn't have brought up in the same mail (or at all, in this > stage anyway -- we'll work this out with the Python consortium members > first). ok. > > > BTW, I'm surprised by the fact that in an Open Source world I'm asked > > to sign a licence agreement with CNRI or to send e-mails for contributed > > code. If Python or Linux had had such constraints from the start, they > > wouldn't have been what they are today. > > Unfortunately, that's the price we have to pay. What we get is legal > protection from CNRI. In general CNRI has contributed a lot to > Python; probably more than you realize. I realize that. But I also realize that in case of a problem, the wet form protects CNRI, not the contributor. Hm. And what happens if you get hit by a bus? Or in 100 years when we'll dance with angels in Paradise? Will Python stay bound to CNRI with little legal possibilities to detach it, in case our successors (or you) start working in another organization? IANAL and curious. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From Fred L. Drake, Jr." References: <199909151843.OAA18364@eric.cnri.reston.va.us> <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <14306.21291.786632.89652@weyr.cnri.reston.va.us> Vladimir Marangozov writes: > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. You shouldn't be; the FSF certainly requires a signed copyright assignment from contributors. I had to sign one for a bunch of patches I made to oleo many years ago. It was a minor nuissance, but that's all. (The *sad* part is that there hasn't been a new release of oleo that could have included the patches for five years! ;) Just noticed that there is a release at ftp.gnu.org now; I'll have to take a look! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal@lemburg.com Fri Sep 17 15:45:52 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 16:45:52 +0200 Subject: [Python-Dev] Methods and friends Message-ID: <37E25420.6C3F5E3D@lemburg.com> I was just looking into classobject.c again after probably more than a year and wondered... has anyone tried optimizing the method lookup and binding process recently ? It seems that for every method call the following happens: 1. the method is looked up in the instance dict; this fails 2. the method is looked up in the class dict; this succeeds and returns a function 3. the class then turns the function into a new unbound method 4. the instance sees the unbound method and creates a new bound method (deleting the unbound method) and returns it Two possible improvements: · let the instance use a special class_getattr_ex function that does not do the extra unbound method step (3.) or · make methods mutable and have step 4. insert the instance object into the method object to turn it into a bound method Has anyone tried this ? Does it produce anything noticeable w/r to method call speed ? BTW, why does the new module check for instance objects for the INSTANCE argument ? Methods can handle any object in this slot, just as they handle any callable object as "function". Could be put to some use when wrapping types into classes or vice versa... which is what motivated the above ;-) Just curious, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido@CNRI.Reston.VA.US Fri Sep 17 16:09:40 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 17 Sep 1999 11:09:40 -0400 Subject: [Python-Dev] Methods and friends In-Reply-To: Your message of "Fri, 17 Sep 1999 16:45:52 +0200." <37E25420.6C3F5E3D@lemburg.com> References: <37E25420.6C3F5E3D@lemburg.com> Message-ID: <199909171509.LAA00729@eric.cnri.reston.va.us> > It seems that for every method call the following happens: > 1. the method is looked up in the instance dict; this fails > 2. the method is looked up in the class dict; this succeeds > and returns a function > 3. the class then turns the function into a new unbound method > 4. the instance sees the unbound method and creates a new > bound method (deleting the unbound method) and returns it Are you sure? As far as I know, steps 3 and 4 are combined when you do getattr on an instance: instance_getattr() calls instance_getattr1() which calls class_lookup(). The latter doesn't create an unbound method. instance_getattr1() then turns it into a bound method. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Fri Sep 17 17:30:21 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Fri, 17 Sep 1999 12:30:21 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <199909151843.OAA18364@eric.cnri.reston.va.us> <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <14306.27805.461540.758955@anthem.cnri.reston.va.us> >>>>> "VM" == Vladimir Marangozov writes: VM> BTW, I'm surprised by the fact that in an Open Source world VM> I'm asked to sign a licence agreement with CNRI or to send VM> e-mails for contributed code. If Python or Linux had had such VM> constraints from the start, they wouldn't have been what they VM> are today. Note that the FSF has been requiring signatures for a long time. Actually, their requirements are IMHO more onerous because they require you to assign your copyrights to the FSF. Their lawyers have told them that they cannot defend the copyright of, e.g. Emacs, unless they own the copyrights to the entire codebase (or at least, anything that they couldn't rip out, throw away, and not completely cripple the application). CNRI's viewpoint is less drastic, but still important. It means that you retain the copyright on the code (good for you), but you give us permission to use it as we see fit (good for us /and/ for you :). Otherwise, it would be possible for a malicious person to contribute something really vital and useful, wait for it to become indispensible, and then say, "oops, we really didn't mean to let you use that code, sorry!" -Barry From jim@interet.com Fri Sep 17 20:20:24 1999 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 17 Sep 1999 15:20:24 -0400 Subject: [Python-Dev] Re: Path hacking References: <000601bf00e1$072a7d90$0801a8c0@bobcat> Message-ID: <37E29478.316C27C9@interet.com> Perhaps you have heard about the East coast US hurricane. It really hammered us here in New Jersey. I had trouble getting home last night due to high water. When I went to our usual Japanese lunch restaurant it had broken windows and was full of tree branches. Wow... Anyway, Mark Hammond wrote: > > No - the point is that the "1.5" key is "reserved" by a standard > install. Changing the string value actually allows you to have your > own subtree, and you can assume you own that. OK, now that I know the rules I will think about doing that. > > And I hate changing any registry entries at all. > > Well, you should learn to get over it! I will try. Of course, even if the registry helps ship Python apps on Windows, it is no help on Unix, and I care about that almost as much. > > My current "solution" is to use freeze to create a black-box > install, > > and worry about second Python installations and wasted storage when > it > > happens. > > well, IMO this is also the correct thing to do. any install that has > >100 files is fragile. So I do both - freeze the app, _and_ a custom > "sys.winver". Yes, but this requires a compiler. > > I was hoping that this thread whould result in a consensis of what > > to do, but it has not. > > There is a consesus for people with the same problem. Different > problems have different optimal solutions. The problem is that there is no reliable way to ship bullet-proof Python apps without recompiling and rebuilding Python. Each of us has his own pet hack to solve our own problem. Now there is talk of custom import hooks, and this is likely to result in each package requiring its own import hook! Aren't packages supposed to be software IC's? I hate to be a nag, but I will keep pushing for a single solution. Python is totally cross platform, and with all that machinery, there must be portable way to do that. I think this is good for Python. Please don't think I am trying to solve my own selfish problems. I have a compiler, I am happy using freeze, and I don't have any problems. Its just that Python would "sell" better and be more popular if a developer could read the documentation "How to ship and install your Python app in five minutes and make millions". This documentation currently reads "You can't". Jim Ahlstrom From gstein@lyra.org Fri Sep 17 23:14:04 1999 From: gstein@lyra.org (Greg Stein) Date: Fri, 17 Sep 1999 15:14:04 -0700 Subject: [Python-Dev] License cleanup References: <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <37E2BD2C.72792E0F@lyra.org> Vladimir Marangozov wrote: >... > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. Actually, this isn't surprising at all. The Free Software Foundation *requires* this kind of thing to be filed with them before you contribute code to the FSF. Essentially, it is a way for the FSF (and CNRI) to legally state that they own the copyright on the particular code. Without that, the contributor could come along later and claim a copyright on the code. The IBM folks who are working on Apache have provided legal releases to the Apache Software Foundation that basically states that IBM won't try to assume any rights under copyright law on the code they contribute to Apache. In fact, every time that I receive a patch for my mod_dav Apache module, the IBM guy attaches a release to the email that has the patch. In a pure, cooperative, world none of this would be necessary. However, the world simply doesn't work that way and all this stuff (licenses, copyrights, releases) is there to prevent Bad Things from happening. It isn't evil in itself, but simply a reflection of the business environment and the society that we're working within. Cheers, -g -- Greg Stein, http://www.lyra.org/ From tim_one@email.msn.com Sat Sep 18 07:38:27 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 02:38:27 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <14306.21291.786632.89652@weyr.cnri.reston.va.us> Message-ID: <000001bf01a0$6ae8cca0$022d153f@tim> [Vladimir Marangozov] > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for > contributed code. [Fred L. Drake, Jr.] > You shouldn't be; the FSF certainly requires a signed copyright > assignment from contributors. I had to sign one for a bunch of > patches I made to oleo many years ago. It was a minor nuissance, but > that's all. Except they add up: year after year, a new batch of stupid little requirements piles up on top of the last year's, and it's a ratchet effect -- always more, never less. The aggregate gets to be a real weariness on the soul. I had to laugh when François Pinard happened to post this on c.l.py today: > ... > Would it be any volunteer, at least for taking care of filling the FSF > papers, if any are needed? I filled more than enough of those in my > life, I prefer to avoid the burden. Ask Barry how many years we've been trying to sign pymode over to the FSF <0.5 wink>. [Vlad] > If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. I sympathize, but that's really hard to say. You don't get pig-biting weary of this crap until you're my age . AFAIK, Berkeley has never beed sued over the BSD license, MIT over the X license, or the U of Arizona over the Icon license (none, really -- Icon is in the public domain). All the legal mumbo jumbo in the "modern" licenses is like wearing garlic around your neck to ward off vampires: the threat isn't real, and if it were it wouldn't do you any good anyway. a-wooden-stake-thru-the-heart-is-your-only-true-defense-ly y'rs - tim From Vladimir.Marangozov@inrialpes.fr Sat Sep 18 15:17:28 1999 From: Vladimir.Marangozov@inrialpes.fr (Vladimir Marangozov) Date: Sat, 18 Sep 1999 15:17:28 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) Message-ID: <199909181417.PAA20866@pukapuka.inrialpes.fr> The relative import thread has hit me with full force. I'm not sure I understand all of what has been said, but since I know this is mainly a naming issue, I want to point you to one of the best CS papers I have had in my hands. It's not really my style to recommend references, but this one should be of major interest to this audience (and Python respectively): Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. It's not available online (it was written on a typewriter), so I'd be happy to send a hard copy of it to anyone who raises a hand in private mail (or cannot find LNCS 60). It's simply a jewel. This guy has understood everything (and I suspect Guido has read this one before implementing Python ;-) -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From tim_one@email.msn.com Sun Sep 19 00:44:49 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 19:44:49 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909160332.XAA19278@eric.cnri.reston.va.us> Message-ID: <000001bf022f$cc11cba0$072d153f@tim> [Guido] > ... > In the Java world, the right solution exist: a better top-level naming > scheme. I forget what the exact rules are, but I seem to recall that > Java has something of the same relative import rules af Python > currently has: you can import a class from the same leaf package by > its simple name, or you can import a class by its full name. You can import only by the full name. There are no shortcuts for anyone ever in explicit Java imports. The dodge is that to reference classes in exactly the same package, you don't need an import statement at all (you *can* have one, but then you need to use the fully qualified name). The other twist is that all Java code is in *some* package, and code that (wisely) doesn't want to live in the ill-defined unnamed package must contain a package statement declaring which package it's a part of, like package COM.dragonsys.justanexample; The periods there actually have no semantic significance, and package COM.dragonsys.notanexample has no more in common with the above than does package ORG.python.guido.hitmeagain. It's purely a conceptual thing that justanexample and notanexample are *called* "subpackages" of package COM.dragonsys in Java; although implementations are free (but not required) to make up mappings between dotted package names and directory trees. Anyway, the Java rules work fine wrt the issues that have been argued in this thread. Things that are worse than Python: + Magical implied import of same-package names is a PITA for readability and maintenance. + Since subpackages are an illusion, and even packages aren't objects in their own right, there's no possibility to create a shorthand local name for the first M chunks of an N-chunk dotted access path (with M < N). For example, you may use java.awt.Font and java.awt.TextArea a lot. You either spell those out in full all the time, or do import java.awt.Font; import java.awt.TextArea; and use just "Font" and "TextArea" later. Making up a local name for just the "java.awt" part isn't an option. Which leads to gross overuse of the following: + "import package.*" is Java's way of spelling "from package import *", and because of the preceding is the only way to get convenient local names for classes imported from other packages (note that can you can never import a package/module in Java; you can only import a type name). So most Java code will do the above as import java.awt.*; and end up importing a gazillion names. This sucks for the same reasons "import *" sucks in Python, although Java catches the name conflicts at compile time. Things that are better than Python: + Name conflicts are caught at compile time. + At least in theory, the names of packages aren't constrained by platform file name limitations or directory quirks. + Relatedly, explicitly naming the package a compilation unit belongs can be a Good Thing. + They have *an* effective scheme for preventing package-name collisions. > ... > Note that I still resist the urge to start renaming all the existing > standard modules -- too much breakage. This will happen in 2.0. At home, I dropped an empty __init__.py into the std Lib directory, so that I can do e.g. from Lib.random import randrange in my own packages with their own random modules. Works fine. For 1.6 you should pick some explicit way to *allow* explicit import of at least the std library modules. I suggest taking the name "Lib" . Java is very cute with its java.util.etc stuff, but longwindedness is not a virtue here -- "util" would have worked just as well and been just as clear. Nobody should write a package with a generic short name like "util" (or "Lib") anyway, so rather than waste this pleasant little corner of the namespace the language may as well reserve it for its own use. heck-it-already-stole-"and"-and-"or"-ly y'rs - tim From tim_one@email.msn.com Sun Sep 19 04:18:43 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:43 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37E20AF7.2D05F371@lemburg.com> Message-ID: <000001bf024d$ae271f00$f72d153f@tim> [MAL] > ... > Of course there are other ways to make this work (and I want > to thank everybody who contributed to the sys.path ideas), but > they just add complexity to a whole different area. The complexity is already in that area, and for all of us. It can be dealt with. Adding new complexity elsewhere doesn't decrease the complexity that's already there, it just adds more and new kinds of stuff to worry about. [Guido] >> ... >> And believe me, at some point *someone* is going to need to rip out >> your custom importer because it interferes with *his* custom importer, >> and he'll choose to replace all your __ imports with absolute package >> names, rather than trying to add the __ feature to his importer. > Note that with imputil doing its thing, he won't have to > worry about adding code to his importer. It'll just pass on the > import to my importer. That's the fun part about imputil. As I recall, the old ni took the initial "__" as referring to the *current* package. So where you intend to write import __.sibling.func it was written under ni as import __.__.sibling.func Since ni is semi-blessed prior art, some other joker is going to slam in an Importer to use those rules. I mix your packages with theirs, and then the meaning of "__" depends on whose Importer sees the damn thing first. Again complexity spreads. If Jim is wary of top-level name collisions now, wait until he thinks about naming-gimmick collisions . Guido is telling the truth: no package is going into Dragon's products unless it's rewritten to purge dicey name tricks. I think we're all in favor of Greg's imputil approach, but where it adds real value (primarily in allowing imports to get satisfied from sources other than direct file images). if-it's-in-a-file-you-just-import-it-by-name-ly y'rs - tim From tim_one@email.msn.com Sun Sep 19 04:18:52 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:52 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37E0CCC1.6EE4E599@lemburg.com> Message-ID: <000201bf024d$b3213ae0$f72d153f@tim> [MAL] > ... > Seems that we're a bit too healthy (see MarkH's post) sometimes, > i.e. there isn't all that much room for experiments. The odds of a change making it into Python radically decreased when 1.0 hit the net, and have continued to decline (although slowly) since then. In recent years, Guido appears to me to have gotten ever more reluctant to entertain even 100% compatible changes to the internals, if they affect a delicate area of the implementation (ceval.c is the most obvious one there). But that's "normal & healthy" too. Languages & implementations get brittle with age, and it's eventually better to start over -- if Guido didn't have Python2 plans in mind, he'd be the first language designer ever to stop where he started! > Just think of cool developments like Chris' stackless python. Experience > shows that these kind of things will never make it into the distribution. Unfortunately, circumstances piled up and Chris got distracted from that, while nobody else made time to push it in his absence. Large changes have gone in, and even more may make it into the Python1 line, but it generally takes a large or "strategic" user base, and much persistence. GregS mentioned his massive work on threads (still not all in), and I'll add the NumPy extensions (which I wouldn't be surprised to see "mainstreamed"), BarryW's string methods, and DavidA's rich comparisons. > Unfortunately, maintaing patches to the dist across releases a real > pain and much work, so these ideas will just sit there unused and > untested. Much the same happened to gcc ... in the end corporate > strength made egcs possible. Perhaps we need such a branch too ? Don't tell, but I've always been surprised at how few people have tried to release a variant Python! The Alice version (case-insensitive names, and 1/2==0.5) is the only one that comes to mind, and the primary effect that had on today's Python is that raw expressions no longer print their value in non-interactive mode (before Alice, 1 + 2 on a line by itself caused "3" to get printed even in batch scripts; this interfered with the Alice team's favored object.method1().method2().method3() coding style, and Guido endured much pain to change "the real" Python to avoid a code split at that early stage of Python's life; ultimately futile, but then Alice Python didn't catch on anyway). So there's very little Python-related history to go on here. I don't mind seeing variants, but have to predict they won't get very far. Just picture what Python 1.6V would look like if its feature set were drawn from a consensus among you, me, Christian, Greg Ewing, John Skaller and Tom Christiansen . the-value-of-a-benevolent-dictator-is-easy-to-underestimate-ly y'rs - tim From tim_one@email.msn.com Sun Sep 19 04:18:47 1999 From: tim_one@email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:47 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <000101bf024d$af986240$f72d153f@tim> [Guido] > I know that this sounds politically incorrect in a radical free > software world, but often in the end I find it more convenient to > conform to the rest of the world and "fit in" than to be different in > every little way. Note that my hair isn't blue :-); I've also > replaced my "signature" glasses with a more conventional model. !!! No wonder you tried to slam a conventional license down our throats. My advice is to get back the old glasses, and walk around the halls at CNRI counting backwards from 1000 by seventeens loudly and incessantly. you'd-be-amazed-at-how-few-lawyers-talk-to-me-ly y'rs - tim From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Sun Sep 19 08:01:29 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Sun, 19 Sep 1999 03:01:29 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <199909160332.XAA19278@eric.cnri.reston.va.us> <000001bf022f$cc11cba0$072d153f@tim> Message-ID: <14308.35401.20723.782353@anthem.cnri.reston.va.us> >>>>> "TP" == Tim Peters writes: TP> + "import package.*" is Java's way of spelling "from package TP> import *", and because of the preceding is the only way to get TP> convenient local names for classes imported from other TP> packages (note that can you can never import a package/module TP> in Java; you can only import a type name). So most Java code TP> will do the above as TP> import java.awt.*; TP> and end up importing a gazillion names. This sucks for the TP> same reasons "import *" sucks in Python, although Java catches TP> the name conflicts at compile time. The interesting this is that, while the Java developers did this at the language level, at the VM level, every class is "fully qualified"; you see the absolute path for every class name. import pkg.pkg.* is the reason why you still have to have unique named classes like PyThinging or JWiggie, and yup it sucks. -Barry From mal@lemburg.com Fri Sep 17 16:53:38 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 17:53:38 +0200 Subject: [Python-Dev] Methods and friends References: <37E25420.6C3F5E3D@lemburg.com> <199909171509.LAA00729@eric.cnri.reston.va.us> Message-ID: <37E26402.3C519434@lemburg.com> Guido van Rossum wrote: > > > It seems that for every method call the following happens: > > 1. the method is looked up in the instance dict; this fails > > 2. the method is looked up in the class dict; this succeeds > > and returns a function > > 3. the class then turns the function into a new unbound method > > 4. the instance sees the unbound method and creates a new > > bound method (deleting the unbound method) and returns it > > Are you sure? As far as I know, steps 3 and 4 are combined when you > do getattr on an instance: instance_getattr() calls > instance_getattr1() which calls class_lookup(). The latter doesn't > create an unbound method. instance_getattr1() then turns it into a > bound method. Oops. Correct. I should have looked a little closer. So it already works in the first way I mentioned as improvement... your time machine again ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Mon Sep 20 09:59:58 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 20 Sep 1999 10:59:58 +0200 Subject: [Python-Dev] Relative Package Imports References: <000201bf024d$b3213ae0$f72d153f@tim> Message-ID: <37E5F78E.3859D12F@lemburg.com> Tim Peters wrote: > > [MAL] > > ... > > Seems that we're a bit too healthy (see MarkH's post) sometimes, > > i.e. there isn't all that much room for experiments. > > The odds of a change making it into Python radically decreased when 1.0 hit > the net, and have continued to decline (although slowly) since then. In > recent years, Guido appears to me to have gotten ever more reluctant to > entertain even 100% compatible changes to the internals, if they affect a > delicate area of the implementation (ceval.c is the most obvious one there). > > But that's "normal & healthy" too. Languages & implementations get > brittle with age, and it's eventually better to start over -- if Guido > didn't have Python2 plans in mind, he'd be the first language designer ever > to stop where he started! > > > Just think of cool developments like Chris' stackless python. Experience > > shows that these kind of things will never make it into the distribution. > > Unfortunately, circumstances piled up and Chris got distracted from that, > while nobody else made time to push it in his absence. Large changes have > gone in, and even more may make it into the Python1 line, but it generally > takes a large or "strategic" user base, and much persistence. GregS > mentioned his massive work on threads (still not all in), and I'll add the > NumPy extensions (which I wouldn't be surprised to see "mainstreamed"), > BarryW's string methods, and DavidA's rich comparisons. Plus the coercion stuff that's still sleeping in one of my project subdirs (I'll have to get this done *before* 1.6 hits the shelves). > > Unfortunately, maintaing patches to the dist across releases a real > > pain and much work, so these ideas will just sit there unused and > > untested. Much the same happened to gcc ... in the end corporate > > strength made egcs possible. Perhaps we need such a branch too ? > > Don't tell, but I've always been surprised at how few people have tried to > release a variant Python! The Alice version (case-insensitive names, and > 1/2==0.5) is the only one that comes to mind, and the primary effect that > had on today's Python is that raw expressions no longer print their value in > non-interactive mode (before Alice, > > 1 + 2 > > on a line by itself caused "3" to get printed even in batch scripts; this > interfered with the Alice team's favored > > object.method1().method2().method3() > > coding style, and Guido endured much pain to change "the real" Python to > avoid a code split at that early stage of Python's life; ultimately futile, > but then Alice Python didn't catch on anyway). > > So there's very little Python-related history to go on here. I don't mind > seeing variants, but have to predict they won't get very far. Just picture > what Python 1.6V would look like if its feature set were drawn from a > consensus among you, me, Christian, Greg Ewing, John Skaller and Tom > Christiansen . Actually, what I was thinking about here was a Python 2.0 branch starting now rather than in a year or so and thus leaving much room for experiments etc. The intention was the same as with egcs and gcc: to fold the enhancements back into the main branch in a few years. E.g. if Guido points us in the right direction, we could start hacking on that piece of revolutionary work now. BTW, I'd suggest using C++ with namespaces but without templates as target language. By the time Python 2.0 will hit the shelves this setup should have reached the same portability as C has now. Perhaps we could even use RTTI (run time type information) to implement optional type safety... ok, just dreaming a little ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 102 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From skip@mojam.com (Skip Montanaro) Mon Sep 20 16:29:55 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 20 Sep 1999 10:29:55 -0500 Subject: [Python-Dev] Tackling circular dependencies in 2.0? Message-ID: <199909201529.KAA21605@dolphin.mojam.com> I think in Python 2.0 it would be nice to have some way to reclaim circular dependencies without the programmer explicitly having to do something like implement a destroy() method and requiring other programmers to (remember to) call it. I forget what the current state of affairs is w.r.t. future memory management in Python. Not knowing anything much about memory management, would it be possible to have a sort of mixed ref count/garbage collection system where you only use the gc stuff as a last resort? My thought is that it would be useful to use gc to find and reclaim circular garbage. can-you-tell-I-just-got-bitten-by-a-circular-reference?-ly y'rs Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From mal@lemburg.com Mon Sep 20 18:44:57 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 20 Sep 1999 19:44:57 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <37E67299.60899D38@lemburg.com> Skip Montanaro wrote: > > I think in Python 2.0 it would be nice to have some way to reclaim circular > dependencies without the programmer explicitly having to do something like > implement a destroy() method and requiring other programmers to (remember > to) call it. I forget what the current state of affairs is w.r.t. future > memory management in Python. Not knowing anything much about memory > management, would it be possible to have a sort of mixed ref count/garbage > collection system where you only use the gc stuff as a last resort? My > thought is that it would be useful to use gc to find and reclaim circular > garbage. > > can-you-tell-I-just-got-bitten-by-a-circular-reference?-ly y'rs If you don't want to wait until 2.0 becomes GA, you could try weak references: http://starship.skyport.net/~lemburg/mxProxy.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 102 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward@cnri.reston.va.us Mon Sep 20 18:53:03 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 20 Sep 1999 13:53:03 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909201529.KAA21605@dolphin.mojam.com>; from Skip Montanaro on Mon, Sep 20, 1999 at 10:29:55AM -0500 References: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <19990920135302.B393@cnri.reston.va.us> > Not knowing anything much about memory > management, would it be possible to have a sort of mixed ref count/garbage > collection system where you only use the gc stuff as a last resort? My > thought is that it would be useful to use gc to find and reclaim circular > garbage. That's *sorta* what Perl does, 'though you can still be bitten by circular refs in a long-running process. Err, long-running thread. You see, Perl has a mark-and-sweep garbage collector which is run only on thread shutdown. From the "perlobj" man page: [code sample omitted because I'm sure many python-dev readers automatically filter out any message that matches /[\$\@\%]\w+/ it's just a constructor that creates objects for a recursive data structure] If you create nodes like that, they (currently) won't go away unless you break their self reference yourself. (In other words, this is not to be construed as a feature, and you shouldn't depend on it.) Almost. When an interpreter thread finally shuts down (usually when your program exits), then a rather costly but complete mark-and-sweep style of garbage collection is performed, and everything allocated by that thread gets destroyed. This is essential to support Perl as an embedded or a multithreadable language. For example, this program demonstrates Perl's two-phased garbage collection: [more interesting code omitted] Interesting idea, but I don't think it's what Skip had in mind. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Mon Sep 20 18:57:51 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 20 Sep 1999 13:57:51 -0400 Subject: [Python-Dev] Preview of some distutils news Message-ID: <19990920135750.D393@cnri.reston.va.us> Judging by the recent traffic on python-dev, lots of people here are interested in the problems of distributing and installing Python modules. So I thought y'all might want to know that I've finally got some interesting news to report on the distutils-sig, and will shortly be reporting it, soliciting feedback, testers, etc. If you're not already on that list, this might be a good time to join. (Oh, the news? Distutils can now compile and install NumPy, PIL, and mxDateTime (just because I happened to have those three distributions sitting around on my home PC). Of course there are all sorts of caveats and limitations, so c'mon over to the distutils-sig if you want to hear what they are and discuss how to fix them. Later this week, of course, after I've checked in all the relevant code.) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From guido@CNRI.Reston.VA.US Mon Sep 20 19:07:03 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 20 Sep 1999 14:07:03 -0400 Subject: [Python-Dev] wet signature form clarification Message-ID: <199909201807.OAA07799@eric.cnri.reston.va.us> Several of you have asked me a clarification of the "contribution description" required on the wet signature form; given that you are all very prolific contributors with a weak memory, it would be nice if you could write something like "everything I've contributed until Sept. 18, 1999" in the space there rather than dig out all the specific module names. I got a call from CNRI's lawyer today, and she agreed that this was fine. Boy am I pleased today! I think she called it a "group description" or something like that. Anyway, if you were waiting for a ruling on this issue, please send in your signed form! (I'd appreciate it if the signature was dry by the time I got it. :-) The form is at http://python.org/1.5/wetsign.html --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one@email.msn.com Tue Sep 21 07:11:00 1999 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 21 Sep 1999 02:11:00 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <000101bf03f8$13ade740$ae2d153f@tim> [Skip Montanaro] > I think in Python 2.0 it would be nice to have some way to > reclaim circular dependencies without the programmer explicitly > having to do something ... This was debated (again) at great length on c.l.py just a few months ago. Guido chimed in with a proposal to keep track of only the dicts that have been allocated, and now and again mark everything reachable from the root set and nuke whatever dicts don't end up marked. Cycles involving dicts would get reclaimed this way, but not cycles not involving dicts. The approach to destructors for objects in cycles was "tough -- they don't get called". What to do about destructors for objects that are not themselves involved in cycles but are reachable only from dead cycles (so are in fact dead too) wasn't addressed. Seemed possible that stuff reachable from ordinary dicts (not in a cycle, and neither reachable from a cycle) would behave differently than today, since the "list of all dicts" may keep the stuff artificially alive until the next mark+sweep, even if the refcount on the stuff fell to zero; there's probably an OK way around that, though. Anyway, Guido was aiming for the minimal changes that could possibly do real good. It didn't pretend to reclaim all cycles, and was (IMO) too eager to punt on the hard issues (the combo of cycles, destructors and resurrection is a god-awful mess, even in theory; Scheme uses callbacks to dump the problems back on the users Java has incredibily elaborate rules that are both bulletproof and unusable; the Boehm collector lets objects with destructors that are in cycles simply leak, rather than do a wrong thing; Stroustrup has flip-flopped and most recently argued for Guido's "reclaim the memory but don't call the destructors" approach, but a member of the C++ committee told me he's overwhelmingly opposed on this one (I know I would oppose it)). In any case, nothing has come of it, and no easy principled solution is in sight. OTOH, if Guido balks at explaining what "__" means to 12-year-olds, wait until he tries to explain immortal cyclic trash . Perl scores points for its brute-force end-of-thread M&S, but I believe complex user-level data structures are much rarer in Perl, simply due to the clumsiness of the syntax and explicit reference model. Perl's version of "nested functions" don't actually nest (all "def"s are floated to the top level by the compiler, regardless of how deeply they're nested), so Perl's lexical closures don't create cyclic trash either (well, they do, but in the same sense there's a cycle between a Python module namespace and the functions in that module -- Perl also special-cases the snot out of those and busts those cycles by brute force). So there you go! It needs to be solved and nobody has a clue . if-java-hype-hadn't-suffered-exponential-decay-we-could-have- dumped-it-on-the-jvm-by-now-ly y'rs - tim From mal@lemburg.com Tue Sep 21 10:05:05 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 11:05:05 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> Message-ID: <37E74A41.107A1243@lemburg.com> Tim Peters wrote: > > [Skip Montanaro] > > I think in Python 2.0 it would be nice to have some way to > > reclaim circular dependencies without the programmer explicitly > > having to do something ... > > This was debated (again) at great length on c.l.py just a few months ago. > Guido chimed in with a proposal to keep track of only the dicts that have > been allocated, and now and again mark everything reachable from the root > set and nuke whatever dicts don't end up marked. Cycles involving dicts > would get reclaimed this way, but not cycles not involving dicts. The > approach to destructors for objects in cycles was "tough -- they don't get > called". What to do about destructors for objects that are not themselves > involved in cycles but are reachable only from dead cycles (so are in fact > dead too) wasn't addressed. Seemed possible that stuff reachable from > ordinary dicts (not in a cycle, and neither reachable from a cycle) would > behave differently than today, since the "list of all dicts" may keep the > stuff artificially alive until the next mark+sweep, even if the refcount on > the stuff fell to zero; there's probably an OK way around that, though. You could probably tackle the problem by doing local mark&sweep whenever the ref count on a dictionary falls down to 1 (meaning that it is only referenced from the list of all dicts). This is what I do in mxProxy's weak reference implementation and to my surprise it solved all those strange situations where objects are kept alive longer than they would have normally. > Anyway, Guido was aiming for the minimal changes that could possibly do real > good. It didn't pretend to reclaim all cycles, and was (IMO) too eager to > punt on the hard issues (the combo of cycles, destructors and resurrection > is a god-awful mess, even in theory; Scheme uses callbacks to dump the > problems back on the users Java has incredibily elaborate rules that are > both bulletproof and unusable; the Boehm collector lets objects with > destructors that are in cycles simply leak, rather than do a wrong thing; > Stroustrup has flip-flopped and most recently argued for Guido's "reclaim > the memory but don't call the destructors" approach, but a member of the C++ > committee told me he's overwhelmingly opposed on this one (I know I would > oppose it)). Not calling the destructor will cause leakage in all objects allocating extra storage, such as lists, instances and probably just about any dynamically sized object there is in Python... solving the problem only half way. Plus you will definitely run into trouble as soon as external resources are involved, e.g. open files or connections to databases. Perhaps we should give more power to the user instead of trying to give him fuzzy feelings about what's happening underneath the hood. Builtin weak references or other indirect ways of accessing objects (e.g. by giving unique names to the involved objects) can solve many of those circ. ref. problems. BTW, I usually use an instrumented Python interpreter to track down circular references: it uses a tracing hook in the allocation/deallocation code of Python instances which is used when Python is run in debugging mode (python -d). The hook calls a function sys.traceinstances (if present) which allows me to keep a track record of all allocated instances: def traceinstances(action,inst): """ Tracing hook. This is called whenever an instances is created and destroyed. action is either 'create' or 'delete'; inst points to the instance object. """ ... If anyone is interested I can post the patch (against Python 1.5). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido@CNRI.Reston.VA.US Tue Sep 21 14:13:19 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 21 Sep 1999 09:13:19 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: Your message of "Tue, 21 Sep 1999 11:05:05 +0200." <37E74A41.107A1243@lemburg.com> References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> Message-ID: <199909211313.JAA09783@eric.cnri.reston.va.us> > Not calling the destructor will cause leakage in all objects > allocating extra storage, such as lists, instances and > probably just about any dynamically sized object there is in > Python... solving the problem only half way. Plus you will > definitely run into trouble as soon as external resources > are involved, e.g. open files or connections to databases. If I remember well, the only destructors not called would be __del__ methods, since the dependencies between to-be-deleted instances are unknown to the collector. Regular (C-level) destructors would of course be called. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@mojam.com (Skip Montanaro) Tue Sep 21 16:33:31 1999 From: skip@mojam.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 21 Sep 1999 10:33:31 -0500 (CDT) Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <37E74A41.107A1243@lemburg.com> References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> Message-ID: <14311.42315.295460.174012@dolphin.mojam.com> Marc> BTW, I usually use an instrumented Python interpreter to track Marc> down circular references: ... Marc> If anyone is interested I can post the patch (against Python 1.5). That would be interesting to look at. I found my latest circular reference by building Python with Py_DEBUG defined and trudging through the output at the end. Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From mal@lemburg.com Tue Sep 21 17:23:53 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 18:23:53 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> <14311.42315.295460.174012@dolphin.mojam.com> Message-ID: <37E7B119.5A695349@lemburg.com> Skip Montanaro wrote: > > Marc> BTW, I usually use an instrumented Python interpreter to track > Marc> down circular references: ... > > Marc> If anyone is interested I can post the patch (against Python 1.5). > > That would be interesting to look at. I found my latest circular reference > by building Python with Py_DEBUG defined and trudging through the output at > the end. Here it is: --- Objects/orig/classobject.c Thu Jan 1 20:39:04 1998 +++ Objects/classobject.c Sat Aug 8 21:38:57 1998 @@ -403,10 +588,37 @@ PyInstance_New(class, arg, kw) inst = NULL; } Py_DECREF(res); } } + /* sys.traceinstances hook */ + if (Py_DebugFlag && inst) { + PyObject *fct; + + fct = PySys_GetObject("traceinstances"); + if (fct) { + PyObject *v,*arg; + PyObject *error_type, *error_value, *error_traceback; + + /* Save and clear any exception */ + PyErr_Fetch(&error_type,&error_value, + &error_traceback); + PyErr_Clear(); + arg = Py_BuildValue("(sO)","create",(PyObject *)inst); + v = PyEval_CallObject(fct,arg); + Py_DECREF(arg); + if (!v) { + PyErr_Print(); + PyErr_Clear(); + } + else + Py_DECREF(v); + /* Restore exception state */ + PyErr_Restore(error_type,error_value, + error_traceback); + } + } return (PyObject *)inst; } /* Instance methods */ @@ -460,10 +672,31 @@ instance_dealloc(inst) } else Py_DECREF(res); Py_DECREF(del); } + /* sys.traceinstances hook */ + if (Py_DebugFlag) { + PyObject *fct; + + fct = PySys_GetObject("traceinstances"); + if (fct) { + PyObject *v,*arg; + + /* Clear any previous exception */ + PyErr_Clear(); + arg = Py_BuildValue("(sO)","delete",(PyObject *)inst); + v = PyEval_CallObject(fct,arg); + Py_DECREF(arg); + if (!v) { + PyErr_Print(); + PyErr_Clear(); + } + else + Py_DECREF(v); + } + } /* Restore the saved exception and undo the temporary revival */ PyErr_Restore(error_type, error_value, error_traceback); /* Can't use DECREF here, it would cause a recursive call */ if (--inst->ob_refcnt > 0) { #ifdef COUNT_ALLOCS -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Tue Sep 21 17:43:22 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 18:43:22 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> <199909211313.JAA09783@eric.cnri.reston.va.us> Message-ID: <37E7B5AA.6196A3B@lemburg.com> Guido van Rossum wrote: > > > Not calling the destructor will cause leakage in all objects > > allocating extra storage, such as lists, instances and > > probably just about any dynamically sized object there is in > > Python... solving the problem only half way. Plus you will > > definitely run into trouble as soon as external resources > > are involved, e.g. open files or connections to databases. > > If I remember well, the only destructors not called would be __del__ > methods, since the dependencies between to-be-deleted instances are > unknown to the collector. Regular (C-level) destructors would of > course be called. Ok, so low-stuff will not break. But what about e.g. wrappers around these low-level (C-level) objects written in Python, e.g. database abstraction classes ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From tim_one@email.msn.com Wed Sep 22 06:56:05 1999 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 22 Sep 1999 01:56:05 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909211313.JAA09783@eric.cnri.reston.va.us> Message-ID: <000801bf04bf$287e6560$342d153f@tim> [Guido van Rossum] > If I remember well, the only destructors not called would be __del__ > methods, since the dependencies between to-be-deleted instances are > unknown to the collector. Regular (C-level) destructors would of > course be called. Seems to me that the dependencies among to-be-deleted arbitrary C objects are equally unknown to the collector. Assuming there's a fundamental difference between objects implemented in Python and objects implemented in C seems shaky on the face of it. Or if it's not just a convenient assumption, on what is it based? and-what-about-objects-implemented-in-fortran?-ly y'rs - tim From mal@lemburg.com Wed Sep 22 13:47:03 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 22 Sep 1999 14:47:03 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports Message-ID: <37E8CFC7.673E040A@lemburg.com> Hi everybody, I'm currently busy converting my code to use mx.DateTime instead of just DateTime and must say that this is a real pain. Now instead of reviving the __ discussion I'd like to turn to what could be a workable compromise (I think Jim proposed something along these lines already and there was similar code in ni.py). The proposal is really only an addition to the lookup scheme used by the importer. No additional syntax is involved and best of all, it is backward compatible... The current lookup does the following if you want to import a module E from module A.B.C.D: 1. check A.B.C.E 2. check E 3. fail Now instead of failing we could add a lookup method that walks up the package structure: 3. check A.B.E 4. check A.E [5. check E -- already done] 6. fail so that the complete scheme looks like this: 1. check A.B.C.E 2. check E 3. check A.B.E 4. check A.E [5. check E -- already done] 6. fail That way I could leave intra-mx-package imports untouched and still have the convenience of achieving the goal of making my mx subpackages work in the mx context *plus* in the top-level context thus allowing a backward compatible and flexible setup for mx* users. Note that the scheme finds exactly the same modules it did previously, plus perhaps some more (which is intended), and it does not involve any search path hacks. How is that for a compromise ? [Ducking for cover ;-)] -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm@hypernet.com Wed Sep 22 14:12:40 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 22 Sep 1999 09:12:40 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37E8CFC7.673E040A@lemburg.com> Message-ID: <1274116686-66496233@hypernet.com> [M.-A. Lemburg] > The current lookup does the following if you want to import > a module E from module A.B.C.D: > > 1. check A.B.C.E > 2. check E > 3. fail > > Now instead of failing we could add a lookup method that > walks up the package structure: > > 3. check A.B.E > 4. check A.E > [5. check E -- already done] > 6. fail > > so that the complete scheme looks like this: > > 1. check A.B.C.E > 2. check E > 3. check A.B.E > 4. check A.E > [5. check E -- already done] > 6. fail > > That way I could leave intra-mx-package imports untouched and > still have the convenience of achieving the goal of making my mx > subpackages work in the mx context *plus* in the top-level > context thus allowing a backward compatible and flexible setup > for mx* users. Comment 1: You're just giving yourself headaches by allowing your users to install mx in anything other than the prescribed manner. Comment 2: I generally like this scheme, but think (for consistency and confusion-reduction) that it should go straight up the tree, instead of checking the root second. - Gordon From jim@interet.com Wed Sep 22 14:19:58 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 22 Sep 1999 09:19:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <37E12D4C.2394D311@lyra.org> Message-ID: <37E8D77E.2977C61B@interet.com> Greg Stein wrote: > > Skip Montanaro wrote: > > > > Gordon> In the soon-to-be-published beta version of my installer, I've > > Gordon> got that down to exceptions.py. > > > > Why not just run exceptions.py through Python2C, visually and experimentally > > verify that it works, then ship an exceptions.c as an optional module? Optional module... Good idea. But it is easier to use freeze within the optional module. So I wrote a "boot" built in module, where bootmodule.c is created by the Python program bootmake.py. All this lives in ./Modules. Boot incorporates exceptions.pyc etc. as frozen modules without breaking the current freeze feature. I changed pythonrun.c to load "boot" after sys but before any other imports, but only if it exists as a built in module, otherwise no error. Boot has methods to turn it on and off, and to print its contents. Please take a look at ftp://ftp.interet.com/pub/bootmodule.html I think this is a good solution for how to build in imputil.py. Jim Ahlstrom From guido@CNRI.Reston.VA.US Wed Sep 22 15:53:55 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 22 Sep 1999 10:53:55 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: Your message of "Wed, 22 Sep 1999 01:56:05 EDT." <000801bf04bf$287e6560$342d153f@tim> References: <000801bf04bf$287e6560$342d153f@tim> Message-ID: <199909221453.KAA13321@eric.cnri.reston.va.us> > [Guido van Rossum] > > If I remember well, the only destructors not called would be __del__ > > methods, since the dependencies between to-be-deleted instances are > > unknown to the collector. Regular (C-level) destructors would of > > course be called. [Tim] > Seems to me that the dependencies among to-be-deleted arbitrary C objects > are equally unknown to the collector. Assuming there's a fundamental > difference between objects implemented in Python and objects implemented in > C seems shaky on the face of it. Or if it's not just a convenient > assumption, on what is it based? My assumption was based on the standard Python objects. Cycles necessarily have to include dictionaries or lists (modules, classes and instances link to each other through dictionaries; ditto for function objects; i'm crossing my fingers here for stack frame and traceback objects :-) and I can do things to these to get rid of the links without getting rid of the objects: del L[:] or D.clear(). Third party C objects might have interdependencies similar to those found in Python instances, but my gut feeling is that these aren't as problematic -- e.g. interdependencies between C modules are rare (because the machinery is cumbersome) while they are common between Python modules; and C code isn't susceptible to the problems that Python destructors encounter when the modules they import have already been destroyed. I know, it's an unusual amount of handwaving... --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Sep 22 16:13:06 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 22 Sep 1999 17:13:06 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> Message-ID: <37E8F202.2A27508D@lemburg.com> Gordon McMillan wrote: > > [M.-A. Lemburg] > > The current lookup does the following if you want to import > > a module E from module A.B.C.D: > > > > 1. check A.B.C.E > > 2. check E > > 3. fail > > > > Now instead of failing we could add a lookup method that > > walks up the package structure: > > > > 3. check A.B.E > > 4. check A.E > > [5. check E -- already done] > > 6. fail > > > > so that the complete scheme looks like this: > > > > 1. check A.B.C.E > > 2. check E > > 3. check A.B.E > > 4. check A.E > > [5. check E -- already done] > > 6. fail > > > > That way I could leave intra-mx-package imports untouched and > > still have the convenience of achieving the goal of making my mx > > subpackages work in the mx context *plus* in the top-level > > context thus allowing a backward compatible and flexible setup > > for mx* users. > > Comment 1: You're just giving yourself headaches by allowing > your users to install mx in anything other than the prescribed > manner. Actually, I'm trying to provide them a way to smoothly switch from the old setup to the new one. This includes myself, of course ;-). > Comment 2: I generally like this scheme, but think (for > consistency and confusion-reduction) that it should go straight > up the tree, instead of checking the root second. That would probably break code because the search could find some other module having the same name as a top-level one. OTOH, perhaps that situation is not all the common to fear too much about it. Walking up all the way would certainly be easier to explain to a 12-year old ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido@CNRI.Reston.VA.US Wed Sep 22 16:51:49 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 22 Sep 1999 11:51:49 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: Your message of "Wed, 22 Sep 1999 17:13:06 +0200." <37E8F202.2A27508D@lemburg.com> References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> Message-ID: <199909221551.LAA13721@eric.cnri.reston.va.us> > > Comment 2: I generally like this scheme, but think (for > > consistency and confusion-reduction) that it should go straight > > up the tree, instead of checking the root second. > > That would probably break code because the search could > find some other module having the same name as a top-level > one. OTOH, perhaps that situation is not all the common to > fear too much about it. > > Walking up all the way would certainly be easier to explain to > a 12-year old ;-) Yes, please. Do the long-term understandable thing here. I expect not too many packages have defined subpackages (or submodules) whose name conflicts with a standard library module, so you ought to be pretty safe here! --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@digicool.com Wed Sep 22 17:25:25 1999 From: jim@digicool.com (Jim Fulton) Date: Wed, 22 Sep 1999 12:25:25 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> Message-ID: <37E902F5.27630A90@digicool.com> Guido van Rossum wrote: > > > > Comment 2: I generally like this scheme, but think (for > > > consistency and confusion-reduction) that it should go straight > > > up the tree, instead of checking the root second. > > > > That would probably break code because the search could > > find some other module having the same name as a top-level > > one. OTOH, perhaps that situation is not all the common to > > fear too much about it. > > > > Walking up all the way would certainly be easier to explain to > > a 12-year old ;-) > > Yes, please. Do the long-term understandable thing here. I expect > not too many packages have defined subpackages (or submodules) whose > name conflicts with a standard library module, so you ought to be > pretty safe here! Walking straight up the tree is my preference. I think it is very natural. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward@cnri.reston.va.us Wed Sep 22 18:26:12 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 22 Sep 1999 13:26:12 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <199909221551.LAA13721@eric.cnri.reston.va.us>; from Guido van Rossum on Wed, Sep 22, 1999 at 11:51:49AM -0400 References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> Message-ID: <19990922132611.C6623@cnri.reston.va.us> On 22 September 1999, Guido van Rossum said: > Yes, please. Do the long-term understandable thing here. I expect > not too many packages have defined subpackages (or submodules) whose > name conflicts with a standard library module, so you ought to be > pretty safe here! Especially since doing so doesn't work: for example, the distutils.errors module started life as distutils.exceptions. That changed pretty quickly, once I realized why putting import exceptions into other distutils modules didn't work -- obviously it didn't find my distutils.exceptions. Arguably I should have used an absolute import, but what the heck. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal@lemburg.com Fri Sep 24 10:10:59 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 24 Sep 1999 11:10:59 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> <19990922132611.C6623@cnri.reston.va.us> Message-ID: <37EB4023.28DB43A3@lemburg.com> Greg Ward wrote: > > On 22 September 1999, Guido van Rossum said: > > Yes, please. Do the long-term understandable thing here. I expect > > not too many packages have defined subpackages (or submodules) whose > > name conflicts with a standard library module, so you ought to be > > pretty safe here! > > Especially since doing so doesn't work: for example, the > distutils.errors module started life as distutils.exceptions. That > changed pretty quickly, once I realized why putting > > import exceptions > > into other distutils modules didn't work -- obviously it didn't find my > distutils.exceptions. Arguably I should have used an absolute import, > but what the heck. Wow, so many positive answers -- not bad after that last round of relative imports ;-) Ok, then I'll use the walk-me-up approach. That'll be coded into a PathImporter class I'm writing for imputil which will try to mimic the standard behaviour as much as possible (to be released in a few weeks after my vacation). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mhammond@skippinet.com.au Fri Sep 24 13:36:10 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 24 Sep 1999 22:36:10 +1000 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37EB4023.28DB43A3@lemburg.com> Message-ID: <000501bf0689$63614af0$0801a8c0@bobcat> > try to mimic the standard behaviour as much as possible (to be > released in a few weeks after my vacation). no no Marc - you should know the rules by now - you release it mere hours _before_ your vacation :-) Enjoy! Mark. From mal@lemburg.com Sun Sep 26 20:34:15 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 26 Sep 1999 21:34:15 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <000501bf0689$63614af0$0801a8c0@bobcat> Message-ID: <37EE7537.1075369C@lemburg.com> Mark Hammond wrote: > > > try to mimic the standard behaviour as much as possible (to be > > released in a few weeks after my vacation). > > no no Marc - you should know the rules by now - you release it mere > hours _before_ your vacation :-) Enjoy! Ya, well ;-) I'll try my best... I already have something working but it doesn't do the win32 + mac magic yet because that'll require some additions to the builtin imp module. Also, I found that it is rather slow when compared to the builtin one. Caches can speed this up a little, but I still haven't achieved the same performance. BTW, while hacking along I found a few things that might be worth discussing w/r to a general import hook scheme: Currently, the imputil apporach uses a simple chaining technique. Unfortunately, it doesn't allow inspecting the chain for already loaded hooks, so the same type of hook could be loaded more than once. Also, there are at least two types of hooks: 1. hooks that redirect the import to some other data source 2. hooks that modify the way modules are searched Since the first variant may well also be suited to used by the second, the simple chaining method probably won't be powerful enough to handle it. I think what we really need is a set of register/deregister APIs + some framework to differentiate between the two hook types (and possibly other variants). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Mon Sep 27 11:19:25 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 27 Sep 1999 12:19:25 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <000501bf0689$63614af0$0801a8c0@bobcat> <37EE7537.1075369C@lemburg.com> Message-ID: <37EF44AD.7C99D9C5@lemburg.com> M.-A. Lemburg wrote: > > BTW, while hacking along I found a few things that might be worth > discussing w/r to a general import hook scheme: > > Currently, the imputil apporach uses a simple chaining technique. > Unfortunately, it doesn't allow inspecting the chain for already > loaded hooks, so the same type of hook could be loaded more than > once. > > Also, there are at least two types of hooks: > > 1. hooks that redirect the import to some other data source > > 2. hooks that modify the way modules are searched > > Since the first variant may well also be suited to used by > the second, the simple chaining method probably won't be > powerful enough to handle it. > > I think what we really need is a set of register/deregister > APIs + some framework to differentiate between the two > hook types (and possibly other variants). Another quirk that I think needs fixing: When I issues an import: import mx.DateTime the whole import is handled by the importer installed at the start of the import. It is not possible to install a different importer e.g. in mx/__init__.py to handle the rest of the import (in this case the import of subpackage DateTime). I think that the importer should honor the __importer__ function (this is set by imputil) if present to let it continue the import of subsequent elements in the dotted name. Aside: Perhaps this is getting too technical for this list... should I start an egroups mailing list for defining a new and more flexible import mechanism ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm@hypernet.com Mon Sep 27 17:53:30 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 27 Sep 1999 12:53:30 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37EF44AD.7C99D9C5@lemburg.com> Message-ID: <1273671699-5802298@hypernet.com> M.-A. Lemburg wrote: [msg 1] > Currently, the imputil apporach uses a simple chaining > technique. Unfortunately, it doesn't allow inspecting the chain > for already loaded hooks, so the same type of hook could be > loaded more than once. I was hoping Greg would jump in, but since he hasn't - You're associating the hook with the strategy. That's the old style. The imputil style is to associate the hook with the actual stuff being managed. The strategy is a property of the hook. > Also, there are at least two types of hooks: > > 1. hooks that redirect the import to some other data source > > 2. hooks that modify the way modules are searched > > Since the first variant may well also be suited to used by the > second, the simple chaining method probably won't be powerful > enough to handle it. The top level question is "is it mine to import?". Greg provides a framework that makes it easy to use alternate data sources, and alternate ways of finding things but that's not really the key thing. You're a "good" importer if you can (when appropriate) way "no it's not mine" efficiently. [msg 2] > Another quirk that I think needs fixing: > > When I issues an import: > > import mx.DateTime > > the whole import is handled by the importer installed at > the start of the import. It is not possible to install a > different importer e.g. in mx/__init__.py to handle the rest of > the import (in this case the import of subpackage DateTime). I > think that the importer should honor the __importer__ function > (this is set by imputil) if present to let it continue the import > of subsequent elements in the dotted name. Sure you can. Your first importer is the "mx" importer. It has a dict of sub-importers. When mx/DateTime/__init__.py runs, it puts itself into that dict. The importer chain is now a tree. This means, I think, that a "general" relative-path importer (ie, one that uses the default PYTHONPATH strategy), should be careful to install itself as the penultimate importer in the chain, (ie, the last before __builtin__.imp). But putting a relative-path search strategy into the "mx" importer is fine if it can quickly determine that the target is / is not a valid name in the "mx" namespace. - Gordon From mal@lemburg.com Mon Sep 27 23:59:57 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 28 Sep 1999 00:59:57 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1273671699-5802298@hypernet.com> Message-ID: <37EFF6ED.1693ACA3@lemburg.com> Gordon McMillan wrote: > > M.-A. Lemburg wrote: > [msg 1] > > Currently, the imputil apporach uses a simple chaining > > technique. Unfortunately, it doesn't allow inspecting the chain > > for already loaded hooks, so the same type of hook could be > > loaded more than once. > > I was hoping Greg would jump in, but since he hasn't - > > You're associating the hook with the strategy. That's the old > style. The imputil style is to associate the hook with the > actual stuff being managed. The strategy is a property of the > hook. I know, but there still is no way to query what kind of hooks are already loaded and what is worse, you cannot unload or reorder them. I'd suggest using a list of hooks which are then traversed in the order they appear in the list, e.g. __importers__ = [DirectoryImporter('/usr/local/lib/python1.5'), ArchiveImporter('/usr/local/lib/app.pyz'), WebImporter('http://www.python.org/pylib/'), PathImporter(('~/bin','~/lib'))] This also has the advantage of being able to easily query the importers during debugging and eliminates the need to have a predefined attribute naming scheme (such as the one imputil uses). > > Also, there are at least two types of hooks: > > > > 1. hooks that redirect the import to some other data source > > > > 2. hooks that modify the way modules are searched > > > > Since the first variant may well also be suited to used by the > > second, the simple chaining method probably won't be powerful > > enough to handle it. > > The top level question is "is it mine to import?". Greg provides > a framework that makes it easy to use alternate data sources, > and alternate ways of finding things but that's not really the > key thing. You're a "good" importer if you can (when > appropriate) way "no it's not mine" efficiently. It does a good job at this, but doesn't really separate lookup and loading of code too well. Everything is packaged into one single method (.get_code()) which is not always flexible enough, e.g. it wasn't possible to implement the walk-up-the-dotted-name scheme using modifications to .get_code() alone. Also, I can see many uses where you combine a lookup hook (e.g. for loading modules across the web) with a filtering hook (e.g. one which checks a module signature). This should go into the framework as well, IMHO... e.g. by having two methods .find_code() and .make_module() (like the builtin importer). > [msg 2] > > Another quirk that I think needs fixing: > > > > When I issues an import: > > > > import mx.DateTime > > > > the whole import is handled by the importer installed at > > the start of the import. It is not possible to install a > > different importer e.g. in mx/__init__.py to handle the rest of > > the import (in this case the import of subpackage DateTime). I > > think that the importer should honor the __importer__ function > > (this is set by imputil) if present to let it continue the import > > of subsequent elements in the dotted name. > > Sure you can. Your first importer is the "mx" importer. It has a > dict of sub-importers. When mx/DateTime/__init__.py runs, it > puts itself into that dict. The importer chain is now a tree. The problem is that the special importer has to be installed *prior* to doing the mx.DateTime import, because otherwise the importer will not take control over the DateTime subpackage (unless I tell it to do so explicitly, which is not really what I want to have to do). > This means, I think, that a "general" relative-path importer (ie, > one that uses the default PYTHONPATH strategy), should be > careful to install itself as the penultimate importer in the chain, > (ie, the last before __builtin__.imp). But putting a relative-path > search strategy into the "mx" importer is fine if it can quickly > determine that the target is / is not a valid name in the "mx" > namespace. Exactly... and this brings us back to the importer list I mentioned above. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein@lyra.org Tue Sep 28 22:57:34 1999 From: gstein@lyra.org (Greg Stein) Date: Tue, 28 Sep 1999 14:57:34 -0700 (PDT) Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <1273671699-5802298@hypernet.com> Message-ID: On Mon, 27 Sep 1999, Gordon McMillan wrote: > M.-A. Lemburg wrote: > [msg 1] > > Currently, the imputil apporach uses a simple chaining > > technique. Unfortunately, it doesn't allow inspecting the chain > > for already loaded hooks, so the same type of hook could be > > loaded more than once. > > I was hoping Greg would jump in, but since he hasn't - I'm in a middle of a move back to CA. Unpacking now... > You're associating the hook with the strategy. That's the old > style. The imputil style is to associate the hook with the > actual stuff being managed. The strategy is a property of the > hook. Quite true. The chaining is simply an artifact of what has been installed as the import hook. I've always envisioned the potential for a "Importer Manager" that installs just like any other hook, but provides higher-level functions for importers to install themselves. The manager simply delegates the get_code() function to the sub-importers. Of course, the manager could use whatever technique to improve the speed of Importer selection. With respect to speed, I think the main point is to realize that the imputil technique is not inherently slow. It just depends on how you design your Importer subclasses -- do you install one or a hundred Importers? The imputil scheme is more about simplifying how people hook into the process (implement get_code() rather than a load/import combo). It also provides a simple capability (chaining) to allow *multiple* hooks to be installed. > > Also, there are at least two types of hooks: > > > > 1. hooks that redirect the import to some other data source > > > > 2. hooks that modify the way modules are searched Just one way -- your second is a variant of the first. "other data source" is a functional superset which includes searching. Importers don't simply alter searching -- they must perform the actual import (from wherever). This is the big change in mindset from the "ihooks" method -- find it and import it on the spot. The net effect is an Importer either imports a module or it doesn't (and the system can fallback to try another Importer). [ one the examples that people always like to specify was importing via URL which was actually quite difficult to use in the old scheme -- how do you separate an HTTP GET into a find/load step? Effectively, you had to double-fetch, or you had to place the whole module (which you retrieved during the find step) into your context for passing to the load. The other issue was the distinct semantics also implied that you could separate the functions -- I believe that to be quite unnecessary functionality. ] > > Since the first variant may well also be suited to used by the > > second, the simple chaining method probably won't be powerful > > enough to handle it. > > The top level question is "is it mine to import?". Greg provides > a framework that makes it easy to use alternate data sources, > and alternate ways of finding things but that's not really the > key thing. You're a "good" importer if you can (when > appropriate) way "no it's not mine" efficiently. Very true! > [msg 2] > > Another quirk that I think needs fixing: > > > > When I issues an import: > > > > import mx.DateTime > > > > the whole import is handled by the importer installed at > > the start of the import. It is not possible to install a > > different importer e.g. in mx/__init__.py to handle the rest of > > the import (in this case the import of subpackage DateTime). I > > think that the importer should honor the __importer__ function > > (this is set by imputil) if present to let it continue the import > > of subsequent elements in the dotted name. > > Sure you can. Your first importer is the "mx" importer. It has a > dict of sub-importers. When mx/DateTime/__init__.py runs, it > puts itself into that dict. The importer chain is now a tree. Gordon's on top of it here... :-) Yes, it is simply a matter of perspective on the import process. An importer does not have to be a static entity. It also can be much more than a way to search a path... it can be highly dynamic and flexible. Whatever you like. Just implement get_code() to map a module "mx.DateTime" to a code/module object. There are a bazillion ways to do that :-) > This means, I think, that a "general" relative-path importer (ie, > one that uses the default PYTHONPATH strategy), should be > careful to install itself as the penultimate importer in the chain, > (ie, the last before __builtin__.imp). But putting a relative-path > search strategy into the "mx" importer is fine if it can quickly > determine that the target is / is not a valid name in the "mx" > namespace. Part of the Importer work was done to satisfy importing modules from the COM+ namespace. I wanted to be able to say "import COM.foo.bar". The importer would handle all "COM." imports and delegate the "foo.bar" to the underlying Python/COM framework. In other words... yes, the Importer scheme should work *very* well for the "whatever...." type of module namespace. Cheers, -g -- Greg Stein, http://www.lyra.org/ From da@ski.org Wed Sep 29 01:02:20 1999 From: da@ski.org (David Ascher) Date: Tue, 28 Sep 1999 17:02:20 -0700 (Pacific Daylight Time) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) Message-ID: ---------- Forwarded message ---------- On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > It's not available online (it was written on a typewriter), so I'd be > happy to send a hard copy of it to anyone who raises a hand in private > mail (or cannot find LNCS 60). I then asked him for a copy stating: > If you send it to me, I can OCR it and make it available online. and he generously sent it to me. The problem is that I hadn't noticed the length of the manuscript. It's over a hundred pages, and the copy is nth generation, making OCR pretty much useless. So, if anyone wants a copy, I can make and send copies (which would make most sense for folks in the US -- sending things from France isn't cheap). So far, it's good reading. Funny to see "file name" in quotes in the text. --david From mal@lemburg.com Wed Sep 29 11:47:08 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 29 Sep 1999 12:47:08 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: Message-ID: <37F1EE2C.52F9A7E1@lemburg.com> Greg Stein wrote: > > On Mon, 27 Sep 1999, Gordon McMillan wrote: > > M.-A. Lemburg wrote: > > [msg 1] > > > Currently, the imputil apporach uses a simple chaining > > > technique. Unfortunately, it doesn't allow inspecting the chain > > > for already loaded hooks, so the same type of hook could be > > > loaded more than once. > > You're associating the hook with the strategy. That's the old > > style. The imputil style is to associate the hook with the > > actual stuff being managed. The strategy is a property of the > > hook. > > Quite true. The chaining is simply an artifact of what has been installed > as the import hook. I've always envisioned the potential for a "Importer > Manager" that installs just like any other hook, but provides higher-level > functions for importers to install themselves. The manager simply > delegates the get_code() function to the sub-importers. Of course, the > manager could use whatever technique to improve the speed of Importer > selection. > > With respect to speed, I think the main point is to realize that the > imputil technique is not inherently slow. It just depends on how you > design your Importer subclasses -- do you install one or a hundred > Importers? > > The imputil scheme is more about simplifying how people hook into the > process (implement get_code() rather than a load/import combo). It also > provides a simple capability (chaining) to allow *multiple* hooks to be > installed. As I wrote in my reply to Gordon, this setup has some drawbacks which an "Import Manager" could easily solve, e.g. by using a list of importers. > > > Also, there are at least two types of hooks: > > > > > > 1. hooks that redirect the import to some other data source > > > > > > 2. hooks that modify the way modules are searched > > Just one way -- your second is a variant of the first. "other data source" > is a functional superset which includes searching. Importers don't simply > alter searching -- they must perform the actual import (from wherever). Yes, I was just argueing for two types of functionality, not the old scheme. E.g. the Import Manager could provide a set of filters which implement signature checks or know how to un-gzip code plus a set of lookup functions for scanning directories or zip archives. I would like the importers to take advantage of such functionality. Of course, all of this could be implemented in form of classes which the importers then use as mixin classes. > This is the big change in mindset from the "ihooks" method -- find it and > import it on the spot. The net effect is an Importer either imports a > module or it doesn't (and the system can fallback to try another > Importer). > > [ one the examples that people always like to specify was importing via > URL which was actually quite difficult to use in the old scheme -- how do > you separate an HTTP GET into a find/load step? Effectively, you had to > double-fetch, or you had to place the whole module (which you retrieved > during the find step) into your context for passing to the load. The other > issue was the distinct semantics also implied that you could separate the > functions -- I believe that to be quite unnecessary functionality. ] .get_code() is fine for these kind of tasks, but there are some other areas (such as lazy imports) which work better using the split setup. This is pretty easy to implement btw, just have the Import Manager check whether the importer provides .get_code() and then have it revert to using .find_module(), .load_module() if it doesn't. The more I think about it, the more I like the idea of an Import Manager instead of the chaining approach. > > > Since the first variant may well also be suited to used by the > > > second, the simple chaining method probably won't be powerful > > > enough to handle it. > > > > The top level question is "is it mine to import?". Greg provides > > a framework that makes it easy to use alternate data sources, > > and alternate ways of finding things but that's not really the > > key thing. You're a "good" importer if you can (when > > appropriate) way "no it's not mine" efficiently. > > Very true! > > > [msg 2] > > > Another quirk that I think needs fixing: > > > > > > When I issues an import: > > > > > > import mx.DateTime > > > > > > the whole import is handled by the importer installed at > > > the start of the import. It is not possible to install a > > > different importer e.g. in mx/__init__.py to handle the rest of > > > the import (in this case the import of subpackage DateTime). I > > > think that the importer should honor the __importer__ function > > > (this is set by imputil) if present to let it continue the import > > > of subsequent elements in the dotted name. > > > > Sure you can. Your first importer is the "mx" importer. It has a > > dict of sub-importers. When mx/DateTime/__init__.py runs, it > > puts itself into that dict. The importer chain is now a tree. > > Gordon's on top of it here... :-) Yes, it is simply a matter of > perspective on the import process. An importer does not have to be a > static entity. It also can be much more than a way to search a path... it > can be highly dynamic and flexible. Whatever you like. Just implement > get_code() to map a module "mx.DateTime" to a code/module object. There > are a bazillion ways to do that :-) Except that they don't work due to the fact that the builtin importer is not recursively using __import__ for the imports. An Import Manager would help with this too :-) > > This means, I think, that a "general" relative-path importer (ie, > > one that uses the default PYTHONPATH strategy), should be > > careful to install itself as the penultimate importer in the chain, > > (ie, the last before __builtin__.imp). But putting a relative-path > > search strategy into the "mx" importer is fine if it can quickly > > determine that the target is / is not a valid name in the "mx" > > namespace. > > Part of the Importer work was done to satisfy importing modules from the > COM+ namespace. I wanted to be able to say "import COM.foo.bar". The > importer would handle all "COM." imports and delegate the "foo.bar" to the > underlying Python/COM framework. > > In other words... yes, the Importer scheme should work *very* well for > the "whatever...." type of module namespace. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim@interet.com Wed Sep 29 14:59:11 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 29 Sep 1999 09:59:11 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1273671699-5802298@hypernet.com> <37EFF6ED.1693ACA3@lemburg.com> Message-ID: <37F21B2F.DD65738F@interet.com> "M.-A. Lemburg" wrote: > > I know, but there still is no way to query what kind of hooks > are already loaded and what is worse, you cannot unload or reorder > them. I think this is a valid point. I am interested in hooks to read modules from a file archive. If you are a developer, it is necessary to turn this hook OFF, so that you can revert to the usual directory tree where your current source is. I am solving this by leaving a global variable "Importer" in sitecustomize, and calling sitecustomize.Importer.enable(0). This works, but it might be useful if imputil could de-install a hook as well as install it. Jim Ahlstrom From mhammond@skippinet.com.au Thu Sep 30 01:16:44 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 30 Sep 1999 10:16:44 +1000 Subject: [Python-Dev] getopt helper? Message-ID: <007d01bf0ad5$14b6cf60$0801a8c0@bobcat> A little thought I have had a few times now, so it is time to pass it on for comment. Basically _every_ time I use getopt, I write code like this: import getopt try: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") except getopt.error, why: print why print usage % (os.path.basename(sys.argv[0],)) sys.exit(1) Every single time. I have never used getopt without this code. How about we put this functionality into getopt itself? It could be triggered either by adding a new "usage" param defaulting to None, or by adding a new entry point. ie: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:", usage=usage) or opts, args = getopt.getoptex(sys.argv[1:], "slc:d:e:", usage=usage) I know it is fairly trivial, but IMO is such a useful module and the pattern is used so regularly that is seems to make sense to add it. Any thoughts? If it is seen favourably, how should we spell it? Mark. From mal@lemburg.com Thu Sep 30 08:29:24 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 09:29:24 +0200 Subject: [Python-Dev] getopt helper? References: <007d01bf0ad5$14b6cf60$0801a8c0@bobcat> Message-ID: <37F31154.4FDA472C@lemburg.com> Mark Hammond wrote: > > A little thought I have had a few times now, so it is time to pass it > on for comment. > > Basically _every_ time I use getopt, I write code like this: > > import getopt > try: > opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") > except getopt.error, why: > print why > print usage % (os.path.basename(sys.argv[0],)) > sys.exit(1) > > Every single time. I have never used getopt without this code. > > How about we put this functionality into getopt itself? > > It could be triggered either by adding a new "usage" param defaulting > to None, or by adding a new entry point. ie: > > opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:", usage=usage) > or > opts, args = getopt.getoptex(sys.argv[1:], "slc:d:e:", usage=usage) > > I know it is fairly trivial, but IMO is such a useful module and the > pattern is used so regularly that is seems to make sense to add it. > > Any thoughts? If it is seen favourably, how should we spell it? Why not just add a higher level interface ? Something like CommandLine.py which is included in mxDateTime ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mhammond@skippinet.com.au Thu Sep 30 11:09:53 1999 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 30 Sep 1999 20:09:53 +1000 Subject: [Python-Dev] getopt helper? In-Reply-To: <37F31154.4FDA472C@lemburg.com> Message-ID: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> > > Basically _every_ time I use getopt, I write code like this: > Why not just add a higher level interface ? Something > like CommandLine.py which is included in mxDateTime ? Because _every_ time I use getopt, I write code like that :-) A higher level interface would maybe be handy, but it simply occurred to me that every time I used the module I used that pattern. I just got sick of typing it all the time and wondered if it struck a chord with anyone else (and I dont have or use a general purpose "mhutil" module :-) Im really just trying to save myself 10 lines of boilerplate coding, not introduce a new standard module :-) Mark. From Vladimir.Marangozov@inrialpes.fr Thu Sep 30 12:46:48 1999 From: Vladimir.Marangozov@inrialpes.fr (Vladimir Marangozov) Date: Thu, 30 Sep 1999 12:46:48 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: from "David Ascher" at "Sep 28, 99 05:02:20 pm" Message-ID: <199909301146.MAA23138@pukapuka.inrialpes.fr> David Ascher wrote: > > ---------- Forwarded message ---------- > On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > > > It's not available online (it was written on a typewriter), so I'd be > > happy to send a hard copy of it to anyone who raises a hand in private > > mail (or cannot find LNCS 60). > > I then asked him for a copy stating: > > > If you send it to me, I can OCR it and make it available online. > > and he generously sent it to me. The problem is that I hadn't noticed the > length of the manuscript. It's over a hundred pages, and the copy is nth > generation, making OCR pretty much useless. If I find some spare time, I'll do it. This paper is a classic in Comp Sci that we kindly invite our students to read. In research language, we say that Saltzer has made "the turn around the clock" regarding this issue, putting a period on it (i.e. there's hardly something more to say). It's interesting, however, to see how the theory was applied on Python and to establish the fairly easy analogy of the binding model. This analogy proves the good design choices Guido has made, but also reveals some weaknesses or the incompleteness of the current implementation. I hope to discuss this for Python 2 in due time and perhaps settle on a compromise which trades genericity for performance. The naming/binding problem drives the whole implementation logic in Python (objects, classes, scopes, etc.). > > So, if anyone wants a copy, I can make and send copies (which would make > most sense for folks in the US -- sending things from France isn't cheap). I have 2 more copies lying around, ready to be sent. Give me an adress. The first 2 adresses win a hard copy, no matter the location (don't worry about mail costs.) -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From mal@lemburg.com Thu Sep 30 12:22:27 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 13:22:27 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> Message-ID: <37F347F3.6D18C285@lemburg.com> Mark Hammond wrote: > > > > Basically _every_ time I use getopt, I write code like this: > > > Why not just add a higher level interface ? Something > > like CommandLine.py which is included in mxDateTime ? > > Because _every_ time I use getopt, I write code like that :-) > > A higher level interface would maybe be handy, but it simply occurred > to me that every time I used the module I used that pattern. I just > got sick of typing it all the time and wondered if it struck a chord > with anyone else (and I dont have or use a general purpose "mhutil" > module :-) Im really just trying to save myself 10 lines of > boilerplate coding, not introduce a new standard module :-) Just a thought :-) I wrote the CommandLine.py for pretty much the same reason: I have quite a few command line apps lying in my bin/ dir and they all did some kind of getopt/sys.argv tricks to handle the input... way to confusing and not easy to maintain. So I decided to take an OO-approach to have them use the same interface with nice help output and to reduce the coding effort. As an example taken from mxDateTime: #!/usr/local/bin/python -u """ Simple Forking Alarm Sample Application for DateTime types and CommandLine. Only works on OSes which support os.fork(). Author: Marc-Andre Lemburg, mailto:mal@lemburg.com """ import time,sys,os from mx.DateTime import * from CommandLine import Application,ArgumentOption class Alarm(Application): header = "Simple Forking Alarm" options = [ArgumentOption('-s', 'set the alarm to now + arg seconds'), ArgumentOption('-m', 'set the alarm to now + arg minutes'), ArgumentOption('-a', 'set the alarm to ring at arg (hh:mm)'), ] version = '0.1' def main(self): atime = now() + (self.values['-s'] or self.values['-m'] * 60 or self.values['-h'] * 3600) * oneSecond abs = self.values['-a'] if abs: atime = strptime(abs,'%H:%M',today(second=0)) if atime < now(): print 'Alarm time has expired...' return print 'Alarm will ring at',atime if not os.fork(): time.sleep((atime - now()).seconds) alarm() os._exit(0) def alarm(): """ Ring alarm """ for i in range(10): sys.stdout.write('\007') sys.stdout.flush() time.sleep(0.2) if __name__ == '__main__': Alarm() Here's the help output this produces: /home/lemburg> alarm -h ------------------------------------------------------------------------ Simple Forking Alarm ------------------------------------------------------------------------ Synopsis: alarm [option] files... Options and default settings: -s arg set the alarm to now + arg seconds -m arg set the alarm to now + arg minutes -a arg set the alarm to ring at arg (hh:mm) -h show this help text --help show this help text --copyright show copyright --examples show examples of usage Version: 0.1 -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido@CNRI.Reston.VA.US Thu Sep 30 13:31:04 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 30 Sep 1999 08:31:04 -0400 Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: Your message of "Thu, 30 Sep 1999 12:46:48 BST." <199909301146.MAA23138@pukapuka.inrialpes.fr> References: <199909301146.MAA23138@pukapuka.inrialpes.fr> Message-ID: <199909301231.IAA11203@eric.cnri.reston.va.us> I've started reading the paper and agree that it's very good! > It's interesting, however, to see how the theory was applied on Python and > to establish the fairly easy analogy of the binding model. This analogy > proves the good design choices Guido has made, but also reveals some > weaknesses or the incompleteness of the current implementation. I hope > to discuss this for Python 2 in due time and perhaps settle on a compromise > which trades genericity for performance. The naming/binding problem drives > the whole implementation logic in Python (objects, classes, scopes, etc.). I'd like to hear what those weaknesses are in your eyes. I can think of two areas myself: (1) sys.path, $PYTHONPATH etc.; (2) class/instance attributes in the context of subclassing and evolution of the base class. (I don't expect the paper to take a stance on nested scopes.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@CNRI.Reston.VA.US Thu Sep 30 13:35:23 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 30 Sep 1999 08:35:23 -0400 Subject: [Python-Dev] getopt helper? In-Reply-To: Your message of "Thu, 30 Sep 1999 13:22:27 +0200." <37F347F3.6D18C285@lemburg.com> References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> Message-ID: <199909301235.IAA11217@eric.cnri.reston.va.us> [Mark] > > > > Basically _every_ time I use getopt, I write code like this: [Marc-Andre] > > > Why not just add a higher level interface ? Something > > > like CommandLine.py which is included in mxDateTime ? [Mark] > > Because _every_ time I use getopt, I write code like that :-) [Marc-Andre] > I wrote the CommandLine.py for pretty much the same reason: Marc-Andre, you're not hearing what Mark is saying. He wants a change to the standard library, and he knows that small additions to existing modules there stand a better chance of adoption than new modules. I personally liked the idea of getoptex() best, except I would call it getopt_or_die(). If the usage message is omitted it can synthesize one from the (short and long) options arguments and sys.argv[0] (the latter being a bit controversial, but it's just a default). Hmm... Perhaps getopt_or_die() shouldn't take the args argument but extract sys.argv[1:] itself? --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Thu Sep 30 14:22:58 1999 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 30 Sep 1999 15:22:58 +0200 Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> Message-ID: <007501bf0b46$eb2f38e0$f29b12c2@secret.pythonware.com> as reported on slashdot: http://www.perl.com/pub/1999/09/topaz.html (topaz is also the code name for our "rewrite Tkinter in Python", but that's another story ;-) From gward@cnri.reston.va.us Thu Sep 30 15:15:50 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 30 Sep 1999 10:15:50 -0400 Subject: [Python-Dev] getopt helper? In-Reply-To: <199909301235.IAA11217@eric.cnri.reston.va.us>; from Guido van Rossum on Thu, Sep 30, 1999 at 08:35:23AM -0400 References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <19990930101549.A10529@cnri.reston.va.us> On 30 September 1999, Guido van Rossum said: > Marc-Andre, you're not hearing what Mark is saying. He wants a change > to the standard library, and he knows that small additions to existing > modules there stand a better chance of adoption than new modules. > > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). Gasp! Guido hath drunk from the poisoned well of the Great Camel and it doth infest his thinking! "or die" indeed -- not an idiom I've seen since the last time I wrote some Perl code (umm, last week?). > If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? Whatever we call it, it should be able to take an explicit argument list, and only default to sys.argv[1:]. What if I want to parse options from an environment variable or a config file? I also like the "don't clobber sys.argv, but return the modified version instead" model -- it's nice to keep a pristine copy of the original argument list! Another problem with getopt is that it doesn't correlate long and short options. I wrote distutils.fancy_getopt (download your copy today! hurry, don't delay -- at this price, it WON'T LAST LONG!) to address this, and to maybe someday do something with help text. On the other hand, don't listen to me -- I tend to write mammoth, bloated, all-singing, all-dancing command-line parsing modules for every new language I encounter. They get more insane with each iteration. I have yet to top my Getopt::Tabular for Perl, though; see http://search.cpan.org/doc/GWARD/Getopt-Tabular-0.3/Tabular.pod if you've ever wondered how far this sort of thing can be taken in a high-level, dynamically typed language. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fredrik@pythonware.com Thu Sep 30 15:18:15 1999 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 30 Sep 1999 16:18:15 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <00b301bf0b4e$a3ac5cc0$f29b12c2@secret.pythonware.com> > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? sounds reasonable. what are you waiting for ;-) From mal@lemburg.com Thu Sep 30 16:28:31 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 17:28:31 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <37F3819F.2C713970@lemburg.com> Guido van Rossum wrote: > > [Mark] > > > > > Basically _every_ time I use getopt, I write code like this: > > [Marc-Andre] > > > > Why not just add a higher level interface ? Something > > > > like CommandLine.py which is included in mxDateTime ? > > [Mark] > > > Because _every_ time I use getopt, I write code like that :-) > > [Marc-Andre] > > I wrote the CommandLine.py for pretty much the same reason: > > Marc-Andre, you're not hearing what Mark is saying. He wants a change > to the standard library, and he knows that small additions to existing > modules there stand a better chance of adoption than new modules. Oh, I did get the idea... just wanted to plug my module here in a take-it-or-leave-it way ;-) I usually put such things into my lib/ dir for Python to find -- no need to make them a standard. > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? Better not: it's sometimes very useful to call the main(args) function of a script in interactive mode which then passes the args list to getopt(). How about adding something like: def getoptex(...,args=None,helptext='Read the source, Luke ;-)'): if args is None: args = sys.argv[1:] ... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Vladimir.Marangozov@inrialpes.fr Thu Sep 30 20:24:53 1999 From: Vladimir.Marangozov@inrialpes.fr (Vladimir Marangozov) Date: Thu, 30 Sep 1999 20:24:53 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: <199909301231.IAA11203@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 30, 99 08:31:04 am" Message-ID: <199909301924.UAA32482@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > > I've started reading the paper and agree that it's very good! > > > It's interesting, however, to see how the theory was applied on Python and > > to establish the fairly easy analogy of the binding model. This analogy > > proves the good design choices Guido has made, but also reveals some > > weaknesses or the incompleteness of the current implementation. I hope > > to discuss this for Python 2 in due time and perhaps settle on a compromise > > which trades genericity for performance. The naming/binding problem drives > > the whole implementation logic in Python (objects, classes, scopes, etc.). > > I'd like to hear what those weaknesses are in your eyes. I can think > of two areas myself: (1) sys.path, $PYTHONPATH etc.; (2) > class/instance attributes in the context of subclassing and evolution > of the base class. (2) subsumes (1). > > (I don't expect the paper to take a stance on nested scopes.) > But we can. I can't make the time for this right now and I apologize. This subject deserves more attention and I can't describe it quickly, so it has to wait. As I said, in due time ;-). I'm currently overloaded. (In my defense, and Barry will undesrand me very well, I'll say that among othger things I'm reworking the www.inrialpes.fr Web site, which is actually a shame -- now that I'm in charge, come and visit it in a month). Anyway, some quick general notes on what actually exists: Where Python (Guido) really strikes is that almost everything we have at the language level is interpreted as a name. Thus, "sharing can occur". These names are always resolved in some context associated with the object containing the named object. Since all we have is names, the risk of name conflicts is real, especially when half of the contexts against which name resolution occurs are implicit. Therefore, name resolution has to be done in a controlled way (what Guido has successfully tried to provide when designing the language). Simple example: >>> print o.attr This says: print the value of an object, whose name "attr" has to be resolved in the context associated with the object "o", this context being explicitely pointed out by a dot "." Another (not so obvious) one: >>> print 1 This says: print the value of an object, whose name "1" has to be resolved in the current (implicit) context. And this is exactly what happens inside the VM in terms of LOAD_CONST , then PRINT_ITEM. If you don't grasp this, try the same example: >>> a = 1 >>> print a This says: Resolve the name "1" in the current context (thus we reach the object in question) then assign a new binding ("a" -> the object) in the current context. Then for "print a", see "This says" of the previous example. A valuable thing happens in my last example: >>> print o.__dict__ This resolves the name "__dict__" in the context associated with the object named "o" (pointed by a dot "."), returning this same context! (or a fraction of it). Whether we have to get a portion of the context or the full context is debatable. It has been felt that with the dynamicity of Python, it's useful to get access to the context, then play with it in a controlled manner. So from here, I (and you) can deduce what happens in terms of naming and binding on function/class/... definitions, on module imports, on attr lookups, on "global" declarations, and so on, and when and where (and hopefully why) we get name conflicts or strange (implicit) name resolutions for nested functions. A last word towards classes: There's no difference between >>> o = C() # an instance of the class C >>> o.__class__ and >>> o = 1 >>> o.__class__ I'm prevented to type "1.__class__" only for syntactic reasons, but the context for the resolution of o.__class__ exists. What's missing is for a future mail. Later, -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Thu Sep 30 21:29:55 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Thu, 30 Sep 1999 16:29:55 -0400 (EDT) Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> <19990930101549.A10529@cnri.reston.va.us> Message-ID: <14323.51267.54862.538939@anthem.cnri.reston.va.us> I can sympathize with Mark, I have nearly the same code in every script I write. I once wrote a nice (IMO) class for doing all the common things I do with c.l. args, but I can't seem to dig it up at the moment. The idea was basically to have a base class that had all the machinery, while derived classes included specially named methods that did the app-specific option handling. It knew whether the method took 1 or zero arguments (not including self), glommed up the shortarg string and longarg list (with appropriate `:' and `=' thrown in), then parsed the args, dispatching to the handlers, e.g.: class MyOptions(getopt.Options): def handle_a(self): self.alpha = 1 handle_alpha = handle_a def handle_b(self, arg): self.beta = arg handle_beta = handle_b def handle_i(self, arg): try: self.integer = int(arg) except ValueError: self.usage() handle_integer = handle_i and could be used like so: #! /usr/bin/env python # # ... opts = MyOptions(sys.argv[1:], usage=__doc__ % globals()) if opts.alpha: do_my_alpha_thang() if opts.integer = 1: do_something_else() for file in opts.leftoverargs: process_file(file) While I liked this a lot, I seem to remember showing it to Guido, receiving the expected scoffing. So now, I just cut and paste the opt parsing stuff into every script I write :) I think it might be nice to add such a base class to the standard getopt module. -Barry From bwarsaw@cnri.reston.va.us (Barry A. Warsaw) Thu Sep 30 21:42:02 1999 From: bwarsaw@cnri.reston.va.us (Barry A. Warsaw) (Barry A. Warsaw) Date: Thu, 30 Sep 1999 16:42:02 -0400 (EDT) Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <007501bf0b46$eb2f38e0$f29b12c2@secret.pythonware.com> Message-ID: <14323.51994.307017.278161@anthem.cnri.reston.va.us> >>>>> "FL" == Fredrik Lundh writes: FL> as reported on slashdot: FL> http://www.perl.com/pub/1999/09/topaz.html FL> (topaz is also the code name for our "rewrite FL> Tkinter in Python", but that's another story ;-) Here's an interesting quote: When I was trying to figure out how to be persuasive on this subject, I finally realized that Perl may be competing with Java in the problem space, but when you're writing Perl, implementing the Perl runtime, really what you're doing is something equivalent to writing a JVM. You're writing the equivalent of a Java Virtual Machine. Now, would you write a JVM in Eiffel? I don't think so. No, so neither would you write the Perl runtime in Java or in Eiffel. In the context of Python, I disagree that it competes against Java; Python makes a nice complement to Java. And I obviously also think it makes perfect sense to write (a) Python runtime in Java, as JimH has so effectively proven. Another interesting tidbit we've addressed here: Was the syntax appropriate for declaring variables to give appropriate hints to a hypothetical compiler? That is to say MY INT $X or MY STR $Y -- and I thought that the INT and the STR and the NUM should be suffixes, something like MY $X:NUM-and, in fact, that suffix syntax is something that Larry officially has blessed, but just not for this purpose. -Barry From da@ski.org Thu Sep 30 22:30:20 1999 From: da@ski.org (David Ascher) Date: Thu, 30 Sep 1999 14:30:20 -0700 (Pacific Daylight Time) Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys In-Reply-To: <14323.51994.307017.278161@anthem.cnri.reston.va.us> Message-ID: On Thu, 30 Sep 1999, Barry A. Warsaw wrote: > Another interesting tidbit we've addressed here: > > Was the syntax appropriate for declaring variables to give > appropriate hints to a hypothetical compiler? That is to say MY > INT $X or MY STR $Y -- and I thought that the INT and the STR and > the NUM should be suffixes, something like MY $X:NUM-and, in fact, > that suffix syntax is something that Larry officially has blessed, > but just not for this purpose. What struck me about that paragraph is that if I understand the comment correctly, Larry is choosing a mapping from a specific syntax to some purpose (unspecified in the excerpt) while an otherwise well-informed Perl user (Chip) 'naturally' wanted to map said syntax to a different semantics. When this happens in Python-land, Guido says "nope, neither". --david From skip at mojam.com Sat Sep 4 23:26:37 1999 From: skip at mojam.com (Skip Montanaro) Date: Sat, 4 Sep 1999 16:26:37 -0500 Subject: [Python-Dev] Catching "return" and "return expr" at compile time Message-ID: <199909042126.QAA03363@dolphin.mojam.com> Attached is a context diff against the latest version of Python/compile.c that checks at compile time for functions that both return expressions or execute return statements with no expression (or equivalently, fall off the end of the function). I figured I'd post it here to get a little friendly feedback and bug discovery before shooting it off to c.l.py. I modified compile.c instead of some preexisting PyLint script because I don't know what's popular out there. On the other hand, I'm sure most people who would be interested in this sort of thing have access to the C source... The basic idea is that each straight line chunk of code is terminated one of four ways: 1. return with no expression 2. return an expression 3. raise an exception 4. fall off the end of the chunk Falling off the end of the function is obviously treated like return with no expression. (This is, after all, what motivated me to do this. ;-) This information is recorded in a new bit vector added to the struct compiling object that's carried around during the compilation. Compound statements simply aggregate the bit vectors for their various clauses in ways appropriate to their semantics. At the end of a function's compilation, the set of return bits computed up to that point tells you whether or not to spit out a warning. Note that it does nothing to recognize constant expressions. The following function will generate a warning: def f(): i = 0 while 1: i = i + 1 if i > 10: return i even though the only way to return from the function is the return statement. To get the above to shut up the compiler you'd have to do something like class CantGetHere: pass def f(): i = 0 while 1: i = i + 1 if i > 10: return i raise CantGetHere Raise statements are treated as a valid way to "return" from a function. Registering them as separate styles of returns serves effectively to turn off the "no return" bit for a block of code. Raise is compatible with either form of return, though they aren't compatible with each other. The code is run whenever a module is compiled. I didn't bother to add a new flag to the python interpreter to enable/disable warnings during compilation, though a -w switch for Python has been mentioned before. I ran the modified byte code compiler over a silly test module as well as executing ./python Lib/test/regrtest.py ./python Lib/compileall.py It uncovered some interesting programming practices and one item I think is an actual bug. In Lib/gzip.py, GzipFile._read returns EOFError at one point instead of raising it. At other points in the method it raises EOFError. There are no other return statements in the function. (I haven't taken the time/had the nerve to run it against my own Python code yet. ;-) I'm firmly of the opinion that more subtle bugs exist in the way people write functions that return and raise values than in the code that calls those functions, contrary to a few vocal folks on c.l.py who may believe otherwise. Enjoy, Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... -------------- next part -------------- A non-text attachment was scrubbed... Name: compile.diffs Type: application/octet-stream Size: 9771 bytes Desc: not available URL: From guido at CNRI.Reston.VA.US Tue Sep 7 17:20:47 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 07 Sep 1999 11:20:47 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Sat, 04 Sep 1999 16:26:37 CDT." <199909042126.QAA03363@dolphin.mojam.com> References: <199909042126.QAA03363@dolphin.mojam.com> Message-ID: <199909071521.LAA18017@eric.cnri.reston.va.us> This is a valuable service! Even though I'm sure that it will cause some pain for people who were used to this programming style... I'm not sure I like the fact that you can't turn it off -- traditionally, Python has had a "no warnings" policy. That has been diluted a bit (python -t prints warnings) but so far it has been the default. I'm wondering if we should introduce a general '-w' flag to turn on warnings like this (which would subsume -t)? Or perhaps there should be a -W flag ("no warnings") and warnings should be the default? There are also platform problems, e.g. on the Mac, stderr doesn't always exist, and on Windows, it doesn't exist if pythonw.exe is used... --Guido van Rossum (home page: http://www.python.org/~guido/) From gward at cnri.reston.va.us Tue Sep 7 18:13:27 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 12:13:27 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909071521.LAA18017@eric.cnri.reston.va.us>; from Guido van Rossum on Tue, Sep 07, 1999 at 11:20:47AM -0400 References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> Message-ID: <19990907121326.B729@cnri.reston.va.us> On 07 September 1999, Guido van Rossum said: > This is a valuable service! Even though I'm sure that it will cause > some pain for people who were used to this programming style... > > I'm not sure I like the fact that you can't turn it off -- > traditionally, Python has had a "no warnings" policy. That has been > diluted a bit (python -t prints warnings) but so far it has been the > default. > > I'm wondering if we should introduce a general '-w' flag to turn on > warnings like this (which would subsume -t)? Or perhaps there should > be a -W flag ("no warnings") and warnings should be the default? Yes yes yes! While adding "-w" is a long way from having a comprehensive set of compile-time warnings in place, it at least means that someone is *thinking* about it. Also, I would suggest that there should be some standard internal mechanism for reporting errors rather than just calling 'PySys_WriteStderr()'. Something as simple as this would probably do the trick: void Py_Warning (char *filename, int line, char *msg) { if (on_a_platform_where_stderr_means_something) PySys_WriteStderr ("warning: file %s, line %d: %s", filename, line, msg); else do_whatever_it_takes_for_this_platform(); } Well, you get the idea. I make no claim that this is an appropriate name for this function, nor do I have anything to say about where it should live. It should also be smart about unknown filename or line number (eg. skip filename if filename == NULL, skip line number if line == -1). Oh, and of course we'll need to add a global variable $^W so that programmers can turn run-time warnings on and off as needed. *duck* Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time warning system in addition to the compile-time warnings of -t and Skip's patch.) Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gstein at lyra.org Tue Sep 7 19:04:14 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 07 Sep 1999 10:04:14 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> Message-ID: <37D5458E.384B77ED@lyra.org> Greg Ward wrote: > > On 07 September 1999, Guido van Rossum said: > > This is a valuable service! Even though I'm sure that it will cause > > some pain for people who were used to this programming style... > > > > I'm not sure I like the fact that you can't turn it off -- > > traditionally, Python has had a "no warnings" policy. That has been > > diluted a bit (python -t prints warnings) but so far it has been the > > default. > > > > I'm wondering if we should introduce a general '-w' flag to turn on > > warnings like this (which would subsume -t)? Or perhaps there should > > be a -W flag ("no warnings") and warnings should be the default? > > Yes yes yes! While adding "-w" is a long way from having a > comprehensive set of compile-time warnings in place, it at least means > that someone is *thinking* about it. I would recommend no warnings by default, and -Wfeature to add specific types of warnings. This pattern follows that used by gcc (well, gcc has *some* warnings by default). Rather than invent a new set of switches, I'd rather steal an existing semantic :-) > Also, I would suggest that there should be some standard internal > mechanism for reporting errors rather than just calling > 'PySys_WriteStderr()'. Something as simple as this would probably do Why? Why not just use PySys_WriteStdErr() as your requested function? It can easily determine "oops. no stderr. let's do something else." >... > Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time > warning system in addition to the compile-time warnings of -t and Skip's > patch.) There is no such thing as run-time vs compile-time warnings. You always have a compiler at run-time, and it can be used at any time. Therefore, you just have "(compilation) warnings" (I could imagine that people will come up with other kinds of warnings once the feature is provided). I would suggest sys.warnings be a dictionary. python -Wbad-return -Wlines-per-func=50 >>> print sys.warnings {'bad-return': None, 'lines-per-func': '50'} Cheers, -g -- Greg Stein, http://www.lyra.org/ From gward at cnri.reston.va.us Tue Sep 7 19:33:53 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 13:33:53 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <37D5458E.384B77ED@lyra.org>; from Greg Stein on Tue, Sep 07, 1999 at 10:04:14AM -0700 References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> Message-ID: <19990907133353.A1235@cnri.reston.va.us> On 07 September 1999, Greg Stein said: > > Also, I would suggest that there should be some standard internal > > mechanism for reporting errors rather than just calling > > 'PySys_WriteStderr()'. Something as simple as this would probably do > > Why? Why not just use PySys_WriteStdErr() as your requested function? It > can easily determine "oops. no stderr. let's do something else." Hmm, that makes sense for the "what's the local equivalent of stderr?" determination. Probably that actually belongs in mywrite() (the static function in Python/sysmodule.c that PySys_WriteStdout() and PySys_WriteStderr() both invoke), so that the same thing can be done for stdout and stderr. However, I still think a separate function for printing source-code-based warnings is a good idea. This is mainly so that the association from (filename, line_number, message) to "warning: file %s, line %s: %s" % (filename, line_number, message) is done in *one* place, rather than everywhere a warning message is generated. For instance, platforms that don't have stderr, but instead pop up a window with all your compile-time warnings nicely formatted, could take advantage of knowing the filename and line number separately to nicely format those warnings. (Of course, this argues *against* putting the "what's the local equivalent of stderr?" determination in the low-level mywrite() function... arg...) > > Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time > > warning system in addition to the compile-time warnings of -t and Skip's > > patch.) > > There is no such thing as run-time vs compile-time warnings. You always > have a compiler at run-time, and it can be used at any time. Therefore, > you just have "(compilation) warnings" (I could imagine that people will > come up with other kinds of warnings once the feature is provided). Well, currently that's true, since currently Python's only warning is the tab warning from -t -- clearly a compile-time warning. (Is this true? I'm no expert on the internals, but I've certainly not seen any other warnings from Python, and I've included plenty of bugs in my code -- umm, just seeing if it would catch them, yeah that's it...) However, one could certainly envision a world where Python issues runtime warnings. If my time machine were working, I'd zip back and suggest to Guido that mistakes with the % operator should issue warnings rather than raising exceptions. (Ignore the language philosophy issue and presume this would be worthwhile.) There are probably other situations where, ignoring past history and language philosophy, it would make sense to issue a warning and march on ahead rather than blowing up immediately. Sometimes Python has a bit of an itchy trigger finger for that ol' TypeError... Anyways, the focus should probably be on compile-time warnings: I can't think of any major runtime errors offhand that Python currently does *nothing* about, so there's not a great need to go scattering the code with runtime warnings. But they *are* a theoretical possibility, and there *is* a difference with compile-time warnings. > I would suggest sys.warnings be a dictionary. > > python -Wbad-return -Wlines-per-func=50 > > >>> print sys.warnings > {'bad-return': None, 'lines-per-func': '50'} Makes sense -- true to warn (possibly giving some extra meaning to "truth", as in this example), and false to not warn. Or maybe None to not warn, not-None to warn. Of course, if there are only compile-time warnings, then modifying sys.warnings will only affect future imports, execs, evals, etc. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skip at mojam.com Tue Sep 7 19:41:27 1999 From: skip at mojam.com (Skip Montanaro) Date: Tue, 7 Sep 1999 12:41:27 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <19990907133353.A1235@cnri.reston.va.us> References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <14293.19862.328236.265600@dolphin.mojam.com> [ ...Lot's of stuff about details of warning implementation snipped... ] That's why I only provided the code to check for inconsistent use of returns, not the flag to turn it on and off! In a message I accidentally sent only to Guido and myself on the subject, I outlined my take on things, which really does about exhaust my knowledge/interest on how/when to enable warnings: Guido> I'm not sure I like the fact that you can't turn it off -- Guido> traditionally, Python has had a "no warnings" policy. That has Guido> been diluted a bit (python -t prints warnings) but so far it has Guido> been the default. The only reason for not being able to turn it off was that would require introducing some sort of -w flag, which wasn't the point of the exercise. We can have the -w/-t/-W discussion now. I haven't any particular opinion on the best way to do it, although I would much prefer it be a run-time as opposed to compile-time option. One other issue might be whether or not to ignore an existing .pyc file and always recompile .py's if warnings are enabled. Of course, we're still all adults here (I think), so perhaps it's sufficient to remind people in the docs to delete the desired .pyc files before running with warnings enabled. Guido> I'm wondering if we should introduce a general '-w' flag to turn Guido> on warnings like this (which would subsume -t)? Or perhaps there Guido> should be a -W flag ("no warnings") and warnings should be the Guido> default? -w sounds fine to me. Guido> There are also platform problems, e.g. on the Mac, stderr doesn't Guido> always exist, and on Windows, it doesn't exist if pythonw.exe is Guido> used... Perhaps on those platforms a file could be opened in a standard location to catch stderr (I presume you can detect the lack of stderr at run-time?). While that would force some (more) Unix conventions on programmers on those platforms, it would also provide more cross-platform uniformity. Skip From bwarsaw at cnri.reston.va.us Tue Sep 7 20:07:01 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Tue, 7 Sep 1999 14:07:01 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <14293.21573.647149.941018@anthem.cnri.reston.va.us> >>>>> "GW" == Greg Ward writes: GW> However, one could certainly envision a world where Python GW> issues runtime warnings. If my time machine were working, I'd GW> zip back and suggest to Guido that mistakes with the % GW> operator should issue warnings rather than raising exceptions. GW> (Ignore the language philosophy issue and presume this would GW> be worthwhile.) Moderately off-topic, but since you brought it up, here's what I use in Mailman (since site-admins can make mistakes editing their templates, which contains %(keys)s... we'd like to make Mailman more robust so it doesn't totally crap out when that happens). We (hopefully) always interpolate with a SafeDict instead of a raw Python dictionary. -Barry class SafeDict(UserDict): """Dictionary which returns a default value for unknown keys. This is used in maketext so that editing templates is a bit more robust. """ def __init__(self, d): # optional initial dictionary is a Python 1.5.2-ism. Do it this way # for portability UserDict.__init__(self) self.update(d) def __getitem__(self, key): try: return self.data[key] except KeyError: if type(key) == StringType: return '%('+key+')s' else: return '' % `key` From tim_one at email.msn.com Tue Sep 7 20:26:30 1999 From: tim_one at email.msn.com (Tim Peters) Date: Tue, 7 Sep 1999 14:26:30 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909071521.LAA18017@eric.cnri.reston.va.us> Message-ID: <002401bef95e$81963720$3f2d153f@tim> [Guido] > ... > I'm wondering if we should introduce a general '-w' flag to turn on > warnings like this (which would subsume -t)? Or perhaps there should > be a -W flag ("no warnings") and warnings should be the default? The latter, if for no other reason than that new users should get bludgeoned into good practice from their first day. If something's serious enough to trigger a warning, and you insist on doing it anyway, then you should at least know enough about Python to be able to find the -W switch . Note that in response to 1,379 distinct complaints about insane Perl semantics, TomC's stock answer is that every serious Perl programmer runs with -w and "use strict". He's right! Every serious Perl programmer does. Perl picked the wrong default, letting naive programmers hang themselves 1,379 distinct ways by default. Besides, warning by default will enhance your enviable reputation as a ruthless dictator opposed to freedom and creativity . > There are also platform problems, e.g. on the Mac, stderr doesn't > always exist, and on Windows, it doesn't exist if pythonw.exe is > used... But this is already a problem for, e.g., reporting fatal syntax errors, yes? That is, -w/-W isn't creating a new problem here, it's making the lack of a solution to an old problem more evident. all's-for-the-best-in-this-best-of-all-possible-worlds-ly y'rs - tim From gward at cnri.reston.va.us Tue Sep 7 20:47:40 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 14:47:40 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <002401bef95e$81963720$3f2d153f@tim>; from Tim Peters on Tue, Sep 07, 1999 at 02:26:30PM -0400 References: <199909071521.LAA18017@eric.cnri.reston.va.us> <002401bef95e$81963720$3f2d153f@tim> Message-ID: <19990907144739.B1235@cnri.reston.va.us> On 07 September 1999, Tim Peters said: > The latter, if for no other reason than that new users should get bludgeoned > into good practice from their first day. If something's serious enough to > trigger a warning, and you insist on doing it anyway, then you should at > least know enough about Python to be able to find the -W switch . > > Note that in response to 1,379 distinct complaints about insane Perl > semantics, TomC's stock answer is that every serious Perl programmer runs > with -w and "use strict". He's right! Every serious Perl programmer does. > Perl picked the wrong default, letting naive programmers hang themselves > 1,379 distinct ways by default. I agree, but I'm only willing to do so publicly because Tim has. So does the Perl documentation (ie. Tom C., I assume); from "man perl": DIAGNOSTICS The -w switch produces some lovely diagnostics. [...] Did we mention that you should definitely consider using the -w switch? BUGS The -w switch is not mandatory. D'you think that's a hint? Obviously, there *must* be a way to turn off warnings, so we can continue to run our crufty, bug-ridden old code without too many problems. Greg S.'s suggestion for being able to customize *which* warnings are printed is also important. Much hair was pulled when Perl 5.004 was released with a whole bunch of new warning messages -- lots of people had to go back and "fix" working code, or remove the -w switch from production scripts to clean up the mess on their stderr, etc. I suspect most of those people (myself included) were enlightened by the new warnings, but annoyed by having to go and fix what wasn't necessarily broken. A lot of people now recommend using -w only when developing, and removing it for production use, simply because of the risk of new warning messages when you upgrade Perl. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mhammond at skippinet.com.au Wed Sep 8 01:01:25 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 8 Sep 1999 09:01:25 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14293.19862.328236.265600@dolphin.mojam.com> Message-ID: <000e01bef984$e975e6d0$0801a8c0@bobcat> > Perhaps on those platforms a file could be opened in a > standard location to > catch stderr (I presume you can detect the lack of stderr at > run-time?). Not really - there are 2 scenarios here. pythonw.exe, for example, always has a valid stdout handle - it just goes nowhere. When Python is embedded in certain COM servers (such as ASP), the stdout handle is invalid - operations on it will fail (perversely, this also means a single "print" statement in your Python code can raise an exception and make your code fail - and seeing as print statements are the debugging state of the art at the moment, this is less than ideal - but I digress) So Im not sure we can check this reasonably at runtime - invalid handles are easy, but valid handles that go nowhere useful (as in pythonw.exe, and therefore the majority of cases we care about) is obviously difficult. OTOH, pythonw.exe doesnt print tracebacks either. Although not ideal, people arent loudly complaining about this - they know to develop and debug using python.exe. As the warnings we are discussing are compile time warnings, we could simply document that they should run "compileall" over their scripts to generate the warnings before attempting to embed it in some sort of wierd system. On my third hand, I would _really_ like to see this in a lint tool rather than in the core. I realize there is no such tool at the moment, but IMO is where we should be heading. Skip's return statement warnings are fine and a nice addition, but in my experience account for a trivial number of my errors. Stuff like warning about a variable name used only once, for example, will probably never get into core Python but in my opinion is far more valuable. So adding this "-w" switch is fine, but still doesnt give us the framework we need to actually create a truly useful package of warnings for the Python developer. [And I am slowly and painfully starting work in this - a lint tool based on the Python parser module. Dont hold your breath though :-] Mark. From guido at CNRI.Reston.VA.US Wed Sep 8 01:05:19 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 07 Sep 1999 19:05:19 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Wed, 08 Sep 1999 09:01:25 +1000." <000e01bef984$e975e6d0$0801a8c0@bobcat> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <199909072305.TAA20414@eric.cnri.reston.va.us> > > Perhaps on those platforms a file could be opened in a > > standard location to > > catch stderr (I presume you can detect the lack of stderr at > > run-time?). > > Not really - there are 2 scenarios here. pythonw.exe, for example, always > has a valid stdout handle - it just goes nowhere. When Python is embedded > in certain COM servers (such as ASP), the stdout handle is invalid - > operations on it will fail (perversely, this also means a single "print" > statement in your Python code can raise an exception and make your code > fail - and seeing as print statements are the debugging state of the art at > the moment, this is less than ideal - but I digress) > > So Im not sure we can check this reasonably at runtime - invalid handles > are easy, but valid handles that go nowhere useful (as in pythonw.exe, and > therefore the majority of cases we care about) is obviously difficult. > > OTOH, pythonw.exe doesnt print tracebacks either. Although not ideal, > people arent loudly complaining about this - they know to develop and debug > using python.exe. As the warnings we are discussing are compile time > warnings, we could simply document that they should run "compileall" over > their scripts to generate the warnings before attempting to embed it in > some sort of wierd system. Hmm... Perhaps pythonw.exe could use freopen() to point stdout and stderr to a log file in a temp directory? The wizards will know where to look... > On my third hand, I would _really_ like to see this in a lint tool rather > than in the core. I realize there is no such tool at the moment, but IMO > is where we should be heading. Skip's return statement warnings are fine > and a nice addition, but in my experience account for a trivial number of > my errors. Stuff like warning about a variable name used only once, for > example, will probably never get into core Python but in my opinion is far > more valuable. So adding this "-w" switch is fine, but still doesnt give > us the framework we need to actually create a truly useful package of > warnings for the Python developer. > > [And I am slowly and painfully starting work in this - a lint tool based on > the Python parser module. Dont hold your breath though :-] Eventually, I also plan to have some kind of lint in IDLE. If the CP4E money comes, I'll start working on that for earnest... --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Wed Sep 8 13:53:46 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 04:53:46 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <37D64E4A.4A8A206C@lyra.org> Greg Ward wrote: > ... > > I would suggest sys.warnings be a dictionary. > > > > python -Wbad-return -Wlines-per-func=50 > > > > >>> print sys.warnings > > {'bad-return': None, 'lines-per-func': '50'} > > Makes sense -- true to warn (possibly giving some extra meaning to > "truth", as in this example), and false to not warn. Or maybe None to > not warn, not-None to warn. Of course, if there are only compile-time > warnings, then modifying sys.warnings will only affect future imports, > execs, evals, etc. Actually, I had intended *presence* in the dictionary to mean "enabled." I don't think we'd want to pre-populate the dict with all possible flags ahead of time, then check for each of them on the command line. (startup time!) However, if we "simply" parsed the command line, extracted all -W options and dropped them into the dict, then we're set. Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at digicool.com Wed Sep 8 14:41:18 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 08 Sep 1999 08:41:18 -0400 Subject: [Python-Dev] More Python command-line features Message-ID: <37D6596E.A6B80783@digicool.com> While we're talking about Python command-line features, I have a couple of requests that would make life alot easier for us. 1. I'd like the options given before the script name to be accessable to Python scripts. For example, in: python -O foo.py bar spam I'd like to have a sys variable that told me that the arguments ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] (aka sys.argv). This is needed if I want to fork/exec (or span or whatever) Python with the same option. 2. I'd like to be able to supply environment variables to Python on the command line, as in: python -O -e PYTHONHOME=/Zope foo.py bar spam This would be very helpful in environment-variable-challenged environments like windows 9x, and might be handy elswhere as well. If people agree that these would be good ideas, I'd ve happy to supply a patch. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward at cnri.reston.va.us Wed Sep 8 14:48:49 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Wed, 8 Sep 1999 08:48:49 -0400 Subject: [Python-Dev] More Python command-line features In-Reply-To: <37D6596E.A6B80783@digicool.com>; from Jim Fulton on Wed, Sep 08, 1999 at 08:41:18AM -0400 References: <37D6596E.A6B80783@digicool.com> Message-ID: <19990908084848.A18815@cnri.reston.va.us> On 08 September 1999, Jim Fulton said: > > While we're talking about Python command-line features, I have a couple > of requests that would make life alot easier for us. > > 1. I'd like the options given before the script name to > be accessable to Python scripts. For example, in: > > python -O foo.py bar spam > > I'd like to have a sys variable that told me that the arguments > ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] > (aka sys.argv). This is needed if I want to fork/exec (or span or > whatever) Python with the same option. Regarding -O, it would also be nice to have a higher-level way to find out what optimization level the current interpreter is running under. Currently, as I understand it, there's no way to predict whether py_compile will generate .pyc or .pyo files, which is a minor annoyance in the Distutils installation code. However, if Jim's mythical sys variable is guaranteed to canonicalize Python's command-line options so that something like "-O" in sys.python_argv would always answer this question, I'd be satisfied. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fdrake at acm.org Wed Sep 8 15:35:28 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 8 Sep 1999 09:35:28 -0400 (EDT) Subject: [Python-Dev] More Python command-line features In-Reply-To: <19990908084848.A18815@cnri.reston.va.us> References: <37D6596E.A6B80783@digicool.com> <19990908084848.A18815@cnri.reston.va.us> Message-ID: <14294.26144.574453.888184@weyr.cnri.reston.va.us> Greg Ward writes: > Regarding -O, it would also be nice to have a higher-level way to find > out what optimization level the current interpreter is running under. > Currently, as I understand it, there's no way to predict whether I agree. In fact, I'd even be willing to add support to compile the parse-trees produced by the parser module with or without optimization (pick your favorite level); that would be easy with the current implementation. I didn't add this earlier because Guido objected, saying that the internal optimization flag could change. That would require that the parser module implementation change accordingly. I don't think that would be a huge problem, other than for my having to send Guido a pre-forma gripe that he'd given me more work to do two days before a release. ;-) So, should .compile() accept an optional optimization level, with the default being to use the "current" setting? -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From skip at mojam.com Wed Sep 8 16:54:20 1999 From: skip at mojam.com (Skip Montanaro) Date: Wed, 8 Sep 1999 09:54:20 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000e01bef984$e975e6d0$0801a8c0@bobcat> References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <14294.30407.419817.523625@dolphin.mojam.com> Mark> On my third hand, I would _really_ like to see this in a lint tool Mark> rather than in the core. I realize there is no such tool at the Mark> moment, but IMO is where we should be heading. Skip's return Mark> statement warnings are fine and a nice addition, but in my Mark> experience account for a trivial number of my errors. Stuff like Mark> warning about a variable name used only once, for example, will Mark> probably never get into core Python but in my opinion is far more Mark> valuable. So adding this "-w" switch is fine, but still doesnt Mark> give us the framework we need to actually create a truly useful Mark> package of warnings for the Python developer. I'm not sure the stuff I wrote belongs in the core either, certainly not in C code. As I mentioned when I posted it though, I wasn't sure where a PyLint type program already existed that I could simply graft onto. I've fiddled around enough with the compile.c code in the past couple of years that I understand it fairly well already. I do have some Python code that does peephole optimization on Python bytecode. I could have put it in there (it already divides functions into basic blocks), but again, not many people have it laying about to play with. Can we start/settle on a Python-based source code framework for this sort of thing? Ideally, I'd like to see a framework that brings the parser module's output up to a level where mere mortals like me can reason about Python code. Skip From skip at mojam.com Wed Sep 8 16:58:13 1999 From: skip at mojam.com (Skip Montanaro) Date: Wed, 8 Sep 1999 09:58:13 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909072305.TAA20414@eric.cnri.reston.va.us> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> <199909072305.TAA20414@eric.cnri.reston.va.us> Message-ID: <14294.30895.37706.568973@dolphin.mojam.com> Guido> Eventually, I also plan to have some kind of lint in IDLE. If Guido> the CP4E money comes, I'll start working on that for earnest... Speaking of which, just where *is* IDLE? I get the Python source via CVS, but I'll be damned if I have anything called idle.py or IDLE.py or even anything that matches "*idle*" or "*IDLE*" glob patterns. I just executed "cvs update -A ." from the top of my tree and checked again. Still nothing. Is it a separate module from the main Python source? Thx, Skip From fdrake at acm.org Wed Sep 8 17:04:41 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 8 Sep 1999 11:04:41 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30895.37706.568973@dolphin.mojam.com> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> <199909072305.TAA20414@eric.cnri.reston.va.us> <14294.30895.37706.568973@dolphin.mojam.com> Message-ID: <14294.31497.168088.467694@weyr.cnri.reston.va.us> Skip Montanaro writes: > Speaking of which, just where *is* IDLE? I get the Python source via CVS, > but I'll be damned if I have anything called idle.py or IDLE.py or even Look in Tools/idle/ -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From fdrake at acm.org Wed Sep 8 17:09:11 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 8 Sep 1999 11:09:11 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30407.419817.523625@dolphin.mojam.com> References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> <14294.30407.419817.523625@dolphin.mojam.com> Message-ID: <14294.31767.411418.442754@weyr.cnri.reston.va.us> Skip Montanaro writes: > thing? Ideally, I'd like to see a framework that brings the parser module's > output up to a level where mere mortals like me can reason about Python This was exactly what I wanted to prevent when I created the parser module! ;-) I think a wrapper that simplifies the parse tree wouldn't be too hard to do; you simply have to be sure that the simplified version can be re-elaborated to pass back to the byte-code compiler via parser.sequence2ast().compile(). Otherwise you can't modify the tree without loosing line number information, which would be nice to keep around! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal at lemburg.com Wed Sep 8 17:08:02 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 08 Sep 1999 17:08:02 +0200 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909071521.LAA18017@eric.cnri.reston.va.us> <002401bef95e$81963720$3f2d153f@tim> <19990907144739.B1235@cnri.reston.va.us> Message-ID: <37D67BD2.72A43DF5@lemburg.com> Greg Ward wrote: > > On 07 September 1999, Tim Peters said: > > The latter, if for no other reason than that new users should get bludgeoned > > into good practice from their first day. If something's serious enough to > > trigger a warning, and you insist on doing it anyway, then you should at > > least know enough about Python to be able to find the -W switch . > > > > Note that in response to 1,379 distinct complaints about insane Perl > > semantics, TomC's stock answer is that every serious Perl programmer runs > > with -w and "use strict". He's right! Every serious Perl programmer does. > > Perl picked the wrong default, letting naive programmers hang themselves > > 1,379 distinct ways by default. > > I agree, but I'm only willing to do so publicly because Tim has. So > does the Perl documentation (ie. Tom C., I assume); from "man perl": > > DIAGNOSTICS > The -w switch produces some lovely diagnostics. > [...] > Did we mention that you should definitely consider using the > -w switch? > > BUGS > The -w switch is not mandatory. > > D'you think that's a hint? > > Obviously, there *must* be a way to turn off warnings, so we can > continue to run our crufty, bug-ridden old code without too many > problems. > > Greg S.'s suggestion for being able to customize *which* warnings are > printed is also important. Much hair was pulled when Perl 5.004 was > released with a whole bunch of new warning messages -- lots of people > had to go back and "fix" working code, or remove the -w switch from > production scripts to clean up the mess on their stderr, etc. I suspect > most of those people (myself included) were enlightened by the new > warnings, but annoyed by having to go and fix what wasn't necessarily > broken. A lot of people now recommend using -w only when developing, > and removing it for production use, simply because of the risk of new > warning messages when you upgrade Perl. I'd suggest to use the -W [=] kind of command line option interface for warnings and to also add an environment variable to customize the standard settings, e.g. PYTHONWARNINGS. About enabling warning per default: you should consider the fact that much code out there will probably produce such warnings, even if it is perfectly valid (e.g. consider Skip's example with while 1:...). Enabling it is definitely not a good idea for production code -- it is during the development step. Since production code is likely to run using -O, I suggest disabling warnings when -O is used and enabling them otherwise. Also, I'd like to second GregS' idea with the sys.warnings dict. Together with a generic -W = interface this would be great for adding customized warnings to Python scripts (i.e. not only the ones that the interpreter itself produces). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 123 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Wed Sep 8 16:57:59 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 08 Sep 1999 16:57:59 +0200 Subject: [Python-Dev] Accessing internal flag values References: <37D6596E.A6B80783@digicool.com> <19990908084848.A18815@cnri.reston.va.us> Message-ID: <37D67977.1882CFDC@lemburg.com> Greg Ward wrote: > > On 08 September 1999, Jim Fulton said: > > > > While we're talking about Python command-line features, I have a couple > > of requests that would make life alot easier for us. > > > > 1. I'd like the options given before the script name to > > be accessable to Python scripts. For example, in: > > > > python -O foo.py bar spam > > > > I'd like to have a sys variable that told me that the arguments > > ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] > > (aka sys.argv). This is needed if I want to fork/exec (or span or > > whatever) Python with the same option. > > Regarding -O, it would also be nice to have a higher-level way to find > out what optimization level the current interpreter is running under. > Currently, as I understand it, there's no way to predict whether > py_compile will generate .pyc or .pyo files, which is a minor annoyance > in the Distutils installation code. Check out mxTools (from my Python Pages). It has a function which lets you control the value of the optimization flag: """ The following functions are installed as add-ons to the builtin sys module. sys.verbosity([level]) If level is given, the value of the interpreter's verbosity flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to e.g. enable verborse lookup output to stderr for import statements even when the interpreter was not invoked with '-v' or '-vv' switch or to force verbosity to be switched off. sys.debugging([level]) If level is given, the value of the interpreter's debugging flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to check whether the interpreter was called with '-d' flag or not. Some extensions use this flag to enable/disable debugging log output (e.g. all the mx Extensions). sys.optimization([level]) If level is given, the value of the interpreter's optimization flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to e.g. compile Python scripts in optimized mode even though the interpreter was not started with -O. """ -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 123 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein at lyra.org Wed Sep 8 21:20:32 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 12:20:32 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> <14294.30407.419817.523625@dolphin.mojam.com> <14294.31767.411418.442754@weyr.cnri.reston.va.us> Message-ID: <37D6B700.42F07735@lyra.org> Fred L. Drake, Jr. wrote: > > Skip Montanaro writes: > > thing? Ideally, I'd like to see a framework that brings the parser module's > > output up to a level where mere mortals like me can reason about Python > > This was exactly what I wanted to prevent when I created the parser > module! ;-) > I think a wrapper that simplifies the parse tree wouldn't be too > hard to do; you simply have to be sure that the simplified version can > be re-elaborated to pass back to the byte-code compiler via > parser.sequence2ast().compile(). Otherwise you can't modify the > tree without loosing line number information, which would be nice to > keep around! This has already been done. Grab the Python2C distribution from http://www.mudlib.org/~rassilon/p2c/. There is a module named "transformer.py" which does just what you're thinking -- it converts Python's deeply-nested trees into something human-readable. Each of the resulting node types are doc'd at the top of the module. It is also over a couple years old, so it has had some decent debugging/stabilization. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond at skippinet.com.au Thu Sep 9 00:03:07 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 9 Sep 1999 08:03:07 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30407.419817.523625@dolphin.mojam.com> Message-ID: <002801befa45$efed8240$0801a8c0@bobcat> > Can we start/settle on a Python-based source code framework > for this sort of > thing? Ideally, I'd like to see a framework that brings the > parser module's > output up to a level where mere mortals like me can reason > about Python > code. Actually, I struggled with this a _lot_, then found that P2C has a module called "transform" which flattens the parse tree down to something I can understand (which is good :-) I could simply attach it, but it is grafted to P2C IIRC. If there is interest I will rip it out... Mark. From gstein at lyra.org Thu Sep 9 00:11:35 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 15:11:35 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <002801befa45$efed8240$0801a8c0@bobcat> Message-ID: <37D6DF17.6CA3AB94@lyra.org> Mark Hammond wrote: >... > Actually, I struggled with this a _lot_, then found that P2C has a module > called "transform" which flattens the parse tree down to something I can > understand (which is good :-) > > I could simply attach it, but it is grafted to P2C IIRC. If there is > interest I will rip it out... transformer.py operates quite independently. It is a relatively large module, though (33k), so I would recommend people just grab the P2C distribution from http://www.mudlib.org/~rassilon/p2c/. I have started to put together a page at http://www.lyra.org/greg/python/ that includes the various modules that I've "published". I'll get transformer over there in the next day or two. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond at skippinet.com.au Thu Sep 9 00:29:47 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 9 Sep 1999 08:29:47 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <37D6DF17.6CA3AB94@lyra.org> Message-ID: <002c01befa49$a9b74730$0801a8c0@bobcat> Oops - for some reason I thought we were on Python-help where Greg doesnt hang out... > transformer.py operates quite independently. It does - however, the only simple way to see what it does is to use P2C. All I had in mind was a simple "if __name__=='__main__': block to demonstrate its output. Mark. From tim_one at email.msn.com Fri Sep 10 08:16:20 1999 From: tim_one at email.msn.com (Tim Peters) Date: Fri, 10 Sep 1999 02:16:20 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <000001befb54$0095e240$a52d153f@tim> [Mark Hammond] > ... > On my third hand, I would _really_ like to see this in a lint tool > rather than in the core. I realize there is no such tool at the > moment, but IMO is where we should be heading. Following the lead taken by other modern languages, like javalint, c++lint, perllint, dylanlint and even vblint ? C lint was a hack needed due to the combination of bad language design choices and poor compilers, but C compilers are smarter now than lint ever was. Who still uses lint? It's dead, and it's not missed. > Skip's return statement warnings are fine and a nice addition, but in > my experience account for a trivial number of my errors. Stuff like > warning about a variable name used only once, for example, will probably > never get into core Python but in my opinion is far more valuable. The notion that a valuable idea will never get into the core is disturbing. I don't really care how it's implemented, but a *visibly* separate "checking" tool is bad UI, one that even the C community left behind with relief. > So adding this "-w" switch is fine, but still doesnt give us the framework > we need to actually create a truly useful package of warnings for the > Python developer. No, but adding the "-W" switch does give us the means by which (perhaps the illusion of) "a" smarter compiler can be invoked & controlled. > [And I am slowly and painfully starting work in this - a lint tool based > on the Python parser module. Dont hold your breath though :-] Aaron W has had a capable pylint tool for a couple years, & it remains virtually unknown; and, far as I can tell, Aaron reciprocated the lack of interest by dropping active development. So why was C lint successful in its day while every crack at pylint flops (btw, yours will too <0.5 wink>)? I think it's two sides of the same coin: C lint found dozens of deadly problems that infested almost all C code (remember the pre-prototype days?). Versions of pylint offer very little beyond pointing out unique vrbl names, perhaps indentation checking, and ...? I'm drawing a blank here. I suppose they should strive to give better msgs for runaway triple-quoted strings. What else? Skip's "return" checker, and far as I can tell then we're already at the point of diminishing returns. My claim is that pylints don't get used both because they're a separate step, and because the odds of them catching something interesting are comparatively tiny. Python simply doesn't have many errors that *can* be caught at compile-time. It's like me firing up the spell-checker at this point to verify "compile-time" -- the expected payoff is negative. There's little enough useful a pylint could do that a mod to add those few smarts to the core would be a fraction of the size & effort of yet another separate tool. Better, in the core, it would actually do people some good because it would actually get used. Which specific problems do you expect your lint tool to uncover? Perhaps there's a world of mechanically-checkable Python errors I haven't yet bumped into. you-windows-guys-write-strange-code-ly y'rs - tim From da at ski.org Fri Sep 10 08:28:07 1999 From: da at ski.org (David Ascher) Date: Thu, 9 Sep 1999 23:28:07 -0700 (Pacific Daylight Time) Subject: [Python-Dev] FYI: Techniques for scientific C++ Message-ID: In case you haven't seen enough C++ papers, you might find this paper interesting. It's a good way to learn more about templates without trying to actually use them, and it's a very good way to decide to stay away from high-power C++ techniques like expression templates. Yeeagh! (I especially like the program shown in section 1.11.1). --david Date: Thu, 26 Aug 1999 21:59:06 -0500 (EST) From: Todd Veldhuizen Subject: OON: Techniques for scientific C++ I've updated and expanded my slides on "Techniques for Scientific C++" into a document. It's available in HTML and Postscript from: HTML: http://extreme.indiana.edu/~tveldhui/papers/techniques/ PS: http://extreme.indiana.edu/~tveldhui/papers/techniques/techniques.ps Here's the table of contents. 1.1: About this document 1.2: C++ Compilers 1.2.1: Placating sub-standard compilers 1.2.2: The compiler landscape 1.2.3: C++-specific optimization 1.3: Compile times 1.3.1: Headers 1.3.2: Prelinking 1.3.3: The program database approach -- Visual Age C++ 1.3.4: Quadratic/Cubic template algorithms 1.4: Static Polymorphism 1.4.1: Are virtual functions evil? 1.4.2: Solution A: simple engines 1.4.3: Solution B: the Barton and Nackman Trick 1.5: Callback inlining techniques 1.5.1: Callbacks: the typical C++ approach 1.5.1.1: Expression templates 1.5.1.2: STL-style function objects 1.5.1.3: Pointer-to-function as a template parameter 1.6: Managing code bloat 1.6.1: Avoid kitchen-sink template parameters 1.6.2: Put function bodies outside template classes 1.6.3: Inlining levels 1.7: Containers 1.7.1: STL-style containers 1.7.2: Data/View containers 1.8: Aliasing and restrict 1.9: Traits 1.9.1: An example: average() 1.9.2: Type promotion example 1.10: Expression templates 1.10.1: Performance implications of pairwise evaluation 1.10.2: Recursive templates 1.10.3: Expression templates: building parse trees 1.10.4: A minimal implementation 1.10.5: Refinements 1.10.6: Pointers to more information 1.10.7: References 1.11: Template metaprograms 1.11.1: Template metaprograms: some history 1.11.2: The need for specialized algorithms 1.11.3: Using template metaprograms to specialize algorithms 1.12: Comma overloading 1.12.1: An example 1.13: Interfacing with Fortran codes 1.14: Some thoughts on performance tuning 1.14.1: General suggestions 1.14.2: Know what your compiler can do 1.14.3: Data structures and algorithms 1.14.4: Efficient use of the memory hierarchy Cheers, Todd - -- Todd Veldhuizen tveldhui at acm.org Indiana Univ. Comp. Sci. http://extreme.indiana.edu/~tveldhui/ - --------------------- Object Oriented Numerics List -------------------------- From mhammond at skippinet.com.au Fri Sep 10 09:36:39 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 10 Sep 1999 17:36:39 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000001befb54$0095e240$a52d153f@tim> Message-ID: <003a01befb5f$39f25400$0801a8c0@bobcat> [Tim laments the death of lint for C :-] > The notion that a valuable idea will never get into the core > is disturbing. Agreed. I based my assesment simply on my perception of what is likely to happen, not my opinion of what _should_ happen. I do agree that it is far far preferable for Python itself to be capable of issuing these warnings, and if Guido feels that is the best direction then it would be very cool. Only Guido can state if he would support such efforts, and probably should right about now (the funk soul brother - sorry - just got the Fat Boy Slim CD, and its going around in my head :-) > Aaron W has had a capable pylint tool for a couple years, & it remains > virtually unknown; and, far as I can tell, Aaron reciprocated > the lack of > interest by dropping active development. Which tends to be the biggest problem with it. A number of people have tried to use it, but often get stymied by the lack of 1.5.?isms - ie, "raise" (ie re-raise) and "assert". It bombs at these statements, and there is some real magic I didnt want to understand. Aaron agrees that a parser module based one would be better. But your original point still remains - I agree having Python do this is a better solution all round. > (remember the pre-prototype days?). Versions of pylint offer > very little > beyond pointing out unique vrbl names, perhaps indentation > checking, and > ...? I'm drawing a blank here. I suppose they should strive > to give better > msgs for runaway triple-quoted strings. What else? Skip's "return" > checker, and far as I can tell then we're already at the point of > diminishing returns. Agreed. However, all of these would be very valuable and account for the vast majority of my errors. > you-windows-guys-write-strange-code-ly y'rs - tim Only cos we use a strange OS Mark. From guido at CNRI.Reston.VA.US Fri Sep 10 16:05:23 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 10 Sep 1999 10:05:23 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Fri, 10 Sep 1999 17:36:39 +1000." <003a01befb5f$39f25400$0801a8c0@bobcat> References: <003a01befb5f$39f25400$0801a8c0@bobcat> Message-ID: <199909101405.KAA26303@eric.cnri.reston.va.us> > Agreed. I based my assesment simply on my perception of what is likely to > happen, not my opinion of what _should_ happen. I do agree that it is far > far preferable for Python itself to be capable of issuing these warnings, > and if Guido feels that is the best direction then it would be very cool. > Only Guido can state if he would support such efforts, and probably should > right about now (the funk soul brother - sorry - just got the Fat Boy Slim > CD, and its going around in my head :-) I agree it should happen, and Tim's argument about keeping lint and compiler together is a convincing one. What stands in the way? (a) There's so much else to do... (b) *Someone* needs to design a good framework for spitting out warnings, and for giving the programmer a way to control which warnings to ignore. I've seen plenty of good suggestions here; now somebody should simply go off and come up with a proposal too good to refuse. (c) I have a different agenda with CP4E -- I think it would be great if the editor could do syntax checking and beyond *while you type*, like the spell checker in MS Word. (I *like* Word's spell checker, even though I hate the style checker [too stupid], and I gladly put up with the spell checker's spurious complaints -- it's easy to turn off, easy to ignore, and it finds lots of good stuff.) Because the editor has to deal with incomplete and sometimes ungrammatical things, and because it has to work in real time (staying out of the way when you're frantically typing, catching up when your fingers take a rest), it needs a different kind of parser. But that's another project, and once the Python core has a warning framework in place, I'm sure we'll find more things that are worth warning about. I'm not always in agreement with Tim Peters when he says that Python is so dynamic that it's impossible to check for certain errors. It may be impossible to say *for sure* that something is an error, but there sure are lots of situations where you're doing something that's *likely* to be an error. E.g. if the compiler sees len(1), and there's no local or global variable named len, it *could* be the case that the user has set up a parallel universe where the len() built-in accepts an integer argument, but more likely it's an honest mistake, and I would like to get a warning about it. The hard part here is to issue this warning for len(x) where x is some variable or expression that is likely to have a non-sequence value (barring alternate universes); this might require global analysis that's hard or expensive enough that we can't put it in the core (yet). This may be seen as an argument for a separate lint... --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at mojam.com Fri Sep 10 16:16:52 1999 From: skip at mojam.com (Skip Montanaro) Date: Fri, 10 Sep 1999 09:16:52 -0500 (CDT) Subject: [Python-Dev] Where does lint-like functionality belong? Message-ID: <14297.4820.321654.848624@dolphin.mojam.com> Okay, if Tim's assertions are correct (and I have no reason to suspect them), a separate pylint will be doomed, so the only reasonable thing to do is to place it in the core where it will be rarin' to go all the time. Perl's experience with -w seems to suggest that it's best to always enable whatever warnings you can as well. (More and more I see people using gcc's -Wall flag as well.) Now, my return consistency stuff was easy enough to write in C for two reasons. One, I'm fairly comfortable with the compile.c code. Two, adding my checks required no extra memory management overhead. Consider a few other checks you might conceivably add to the byte code compiler: * tab nanny stuff (already enabled with -t, right?) * variables set but not used * variables used before set If all of this sort of stuff is added to the compiler proper, I predict a couple major problems will surface: * The complexity of the code will increase significantly, making it harder to maintain and enhance * Fewer and fewer people will be able to approach the code, making it less likely that new checks are added * Future extensions like pluggable virtual machines will be harder to add because their byte code compilers will be harder to integrate into the core In addition, more global checks probably won't be possible (reasoning about code across module boundaries for instance) because the compiler's view of the world is fairly narrow. I think lint-like tools should be implemented in Python (possibly with the support of an extension module for performance-critical sections) which is then called from the compiler proper under appropriate circumstances (warnings enabled, all necessary modules importable, etc). I believe the code would be much more easily maintained and extended. You'd be able to swap in a new byte code compiler without risking the loss of your checking code. Skip From mal at lemburg.com Fri Sep 10 21:06:28 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 10 Sep 1999 21:06:28 +0200 Subject: [Python-Dev] Relative Package Imports Message-ID: <37D956B4.3E79C2C1@lemburg.com> Hi everybody, I've spent the last two hours trying to get relative package imports to work because I need them for my extension packages which will soon all move under a new top-level package name to overcome the conflicts with PIL and Zope. Here are the results... Demo Package Structure: (see the attached demopkg.zip) [a] [b] bc.py ab.py With the attached patch you can do the following: # Pretty useless... import a.b.__.ab # From inside bc.py: from __ import ab # At top-level (also useless, but shows how this situation is handled): import __.sys # __ is bound to None since we are at top-level; sys is still # being loaded though. Of course, common usage will be of the form: form __.__ import submodule_at_higher_level Please tell me what you think and give it a try. It's a first try and may have some design errors. Especially the way head and tail are treated in Python/import.c:import_module_ex may cause trouble -- I need help here. Note: The patch is against the CVS version. If you run Python in verbose mode, the patch will produce some verbose output of what it's doing. Enjoy, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 113 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ -------------- next part -------------- --- /home/lemburg/orig/Python/Python/import.c Fri Apr 9 19:00:51 1999 +++ Python/import.c Fri Sep 10 20:51:02 1999 @@ -1572,10 +1572,14 @@ load_next(mod, altmod, p_name, buf, p_bu char *dot = strchr(name, '.'); int len; char *p; PyObject *result; + if (Py_VerboseFlag) + printf("# load_next: (1) name='%s', buf='%.*s'\n", + name,*p_buflen,buf); + if (dot == NULL) { *p_name = NULL; len = strlen(name); } else { @@ -1586,10 +1590,39 @@ load_next(mod, altmod, p_name, buf, p_bu PyErr_SetString(PyExc_ValueError, "Empty module name"); return NULL; } + /* Handle "__" indicator telling the import mechanism to + continue the search one level higher in the package + hierarchy */ + if (strncmp(name,"__",len) == 0) { + PyObject *modules = PyImport_GetModuleDict(); + + /* Strip the final dotted name from buf */ + dot = strrchr(buf, '.'); + if (dot == NULL) + *p_buflen = 0; + else + *p_buflen = dot - buf; + buf[*p_buflen] = '\0'; + + /* Fetch the parent module or revert to a top-level search */ + if (*p_buflen > 0) { + mod = PyDict_GetItemString(modules,buf); + if (mod == NULL) { + PyErr_SetString(PyExc_SystemError, + "Parent module missing"); + return NULL; + } + } + else + mod = Py_None; + Py_INCREF(mod); + return mod; + } + p = buf + *p_buflen; if (p != buf) *p++ = '.'; if (p+len-buf >= MAXPATHLEN) { PyErr_SetString(PyExc_ValueError, @@ -1597,10 +1630,14 @@ load_next(mod, altmod, p_name, buf, p_bu return NULL; } strncpy(p, name, len); p[len] = '\0'; *p_buflen = p+len-buf; + + if (Py_VerboseFlag) + printf("# load_next: (2) modname='%s', fullname=buf='%s'\n", + p,buf); result = import_submodule(mod, p, buf); if (result == Py_None && altmod != mod) { Py_DECREF(result); /* Here, altmod must be None and mod must not be None */ -------------- next part -------------- A non-text attachment was scrubbed... Name: demopkg.zip Type: application/x-zip-compressed Size: 851 bytes Desc: not available URL: From guido at CNRI.Reston.VA.US Fri Sep 10 21:16:28 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 10 Sep 1999 15:16:28 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Fri, 10 Sep 1999 21:06:28 +0200." <37D956B4.3E79C2C1@lemburg.com> References: <37D956B4.3E79C2C1@lemburg.com> Message-ID: <199909101916.PAA27287@eric.cnri.reston.va.us> > I've spent the last two hours trying to get relative package > imports to work because I need them for my extension packages > which will soon all move under a new top-level package name > to overcome the conflicts with PIL and Zope. I'd much rather use absolute package names for anything that's not in the same directory as the current module. --Guido van Rossum (home page: http://www.python.org/~guido/) From da at ski.org Sat Sep 11 06:06:54 1999 From: da at ski.org (David Ascher) Date: Fri, 10 Sep 1999 21:06:54 -0700 (Pacific Daylight Time) Subject: [Python-Dev] jpythonc: unreachable Python becomes unreachable Java (fwd) Message-ID: Greg Ward notes on the JPython list: > Looks like I've found a definite bug in jpythonc -- unreachable Python > code compiles to unreachable Java code. Here's a simple example: > [..] > Yikes... does this mean that jpythonc will have to analyze Python code > for reachability? Is this even possible? Guess the answer for now is, > "Don't do that!". This is another sort of warning that a checker could do in the core which could catch some errors. Not sure that it makes sense to try and find all the cases that an arbitrary Java compiler could find, but presumably Greg's original code which led him to the jpythonc bug find had a plain-old python bug in it? Did Aaron's pylint find any nonreachable code? --david From mal at lemburg.com Sat Sep 11 12:29:48 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat, 11 Sep 1999 12:29:48 +0200 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> Message-ID: <37DA2F1C.7ED64960@lemburg.com> Guido van Rossum wrote: > > > I've spent the last two hours trying to get relative package > > imports to work because I need them for my extension packages > > which will soon all move under a new top-level package name > > to overcome the conflicts with PIL and Zope. > > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. Of course, you could do everything with absolute names, but then the package author would dictate the complete absolute path which is not always desirable since it can cause name collisions such as DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) and Mark's win32 stuff. As more and more packages appear, we run into this problem more and more often. Relative imports would greatly reduce the risk these collisions, because packages could be written self-contained meaning that they can reach their internal modules via relative paths only and don't have to know about the absolute paths by which they are reachable from other parts of the Python universe. I could then make all my packages self-contained and distribute them in two forms without having to change a single line for the added support: 1. under the new 'mx' package, e.g. mx.DateTime 2. for backward compatibility under 'DateTime' Another major advantage is that I could take any other self-contained package and install it under absolute paths of my choice, e.g. put Zope under org.zope.core, Python under org.python.core etc., without harming their functionality or having to dive deep into their import structures to fix them manually. To further enhance this mechanism I would like to have an alias mechanism in import, pickle et al. so that changes in the package structures become manageable without user intervention: pickles are a major problem whenever import structures change because they store absolute module names. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 111 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From tim_one at email.msn.com Sat Sep 11 22:29:58 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 11 Sep 1999 16:29:58 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DA2F1C.7ED64960@lemburg.com> Message-ID: <000901befc94$6a463d20$6aa2143f@tim> [Guido] > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. [M.-A. Lemburg] > Of course, you could do everything with absolute names, but then > the package author would dictate the complete absolute path which > is not always desirable since it can cause name collisions such > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > released) and Mark's win32 stuff. > > As more and more packages appear, we run into this problem more > and more often. I never understand package complaints. Maybe it's the imprecision of the language people use, or maybe because it's because people don't give fully fleshed-out examples. Whatever, in the end, I never have any idea what the complaint is, or in what way the solution is "solving" anything. In the above, "absolute" doesn't appear to mean "absolute" in any OS-sensible sense. So what does it really mean? Does it mean the same things to Guido and MAL? In MAL's hint of examples, I don't see any problem. If mxDateTime unpacks itself into a directory named DateTime, then *of course* it's going to collide with other packages that want to do likewise. Install it into mxDateTime instead, and take "absolute" to mean "any module that wants an mxDateTime service and does not itself live directly in mxDateTime/ must import the desired module via a path beginning 'mxDateTime.'", and everything looks straightforward to me (and that outcome makes me infer that this is thus probably what Guido has in mind too). Similarly for win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be good to have an explicit way to say "std library" -- "Lib." comes to mind). > I could then make all my packages self-contained and > distribute them in two forms without having to change > a single line for the added support: > > 1. under the new 'mx' package, e.g. mx.DateTime > 2. for backward compatibility under 'DateTime' Ah, so that's what this is about. I vote screw #2. Releasing it that way was a mistake. Better to live with the brief & finite pain of repairing it than complicating Python to cover up for it. > Another major advantage is that I could take any other > self-contained package and install it under absolute paths > of my choice, e.g. put Zope under org.zope.core, Python under > org.python.core etc., without harming their functionality > or having to dive deep into their import structures to fix > them manually. I view that not as an advantage but as harmful complication. Zope etc add great value to a Python installation, and when I write a killer app full of "import zope.this" and "import zope.that", I don't want the possibility that it's not going to work on my client's machine just because their sysadmin installed Zope into some silly site-specific path named after his soon-to-be ex-girlfriend . I don't want a way to work around him doing that, either: I don't want him to be able to screw me to begin with. > To further enhance this mechanism I would like to have an > alias mechanism in import, pickle et al. so that changes > in the package structures become manageable without user > intervention: pickles are a major problem whenever import > structures change because they store absolute module names. This is a different issue, and may have merits of its own. WRT the relative import scheme, its advantage seems to lie in providing a way to partially recover from the damage the new scheme causes <0.5 wink>. As is, the package name used by a release is part of its published interface. You can't change it without causing pain, any more than you can make incompatible changes to public class methods or input-output behavior. In return, package clients are uniform, simple and portable, making life easiest for the people who know least. The burden is on package authors to choose names wisely, and that's where the burden should be. if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs - tim From tim_one at email.msn.com Sun Sep 12 02:41:16 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 11 Sep 1999 20:41:16 -0400 Subject: [Python-Dev] FYI: Techniques for scientific C++ In-Reply-To: Message-ID: <000101befcb7$85be2b80$4c2d153f@tim> [David Ascher, concerning http://extreme.indiana.edu/~tveldhui/papers/techniques/] ] > In case you haven't seen enough C++ papers, you might find this paper > interesting. It's a good way to learn more about templates without trying > to actually use them, and it's a very good way to decide to stay away from > high-power C++ techniques like expression templates. Yeeagh! Didn't these people get to play with m4 in their youth? Ah, this *is* their youth . As for so much else, this is stuff they could have been doing with Lisp/Scheme compile-time macros 15 years ago, except the latter are much clearer, simpler and easier to use. It's not "a trick" in Scheme, it's a fundamental approach, and the full power of the language is available at every step. If this kind of thing still holds a perverse attraction, check out http://www.fftw.org/ These guys produce what are usually the fastest FFT algorithms in the world across platforms, via a dynamic programming approach that generates algorithm fragments and times them on the target platform, eventually "growing" superior platform-specific FFT code for each vector size. Until C++ templates can do file I/O at compile-time (to take advantage of timings from previous runs), they won't even get close . computation-at-compile-time-is-a-powerful-thing-but- then-so-is-a-skunk's-odor-ly y'rs - tim From gmcm at hypernet.com Sun Sep 12 04:41:44 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Sat, 11 Sep 1999 22:41:44 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DA2F1C.7ED64960@lemburg.com> Message-ID: <1275018545-12250772@hypernet.com> M.-A. Lemburg wrote: > Guido van Rossum wrote: > > > > I'd much rather use absolute package names for anything that's not > > in the same directory as the current module. > > Of course, you could do everything with absolute names, but then > the package author would dictate the complete absolute path which is > not always desirable since it can cause name collisions such as > DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) > and Mark's win32 stuff. I can see your point (although I also believe that authors - Guido excepted - should come up with collision-free names, probably in a Java-ish scheme). But I strongly believe that import.c should be left alone, maybe even to die. There are too many people doing import hooks to make fiddling with its behavior safe. I'm also a strong proponent of Greg's imputil.py scheme, which makes it a breeze to do import hooks. And my experience disproves the notion that the import mechanism needs to be in C. If you don't believe me, try the ZlibArchive stuff (which is cross platform) from my Win32 installer stuff. You can pack the standard library into one 475K file, and get a perceptible performance boost. OTOH, I could see doing a framework of packages, in which case relative imports might be handy. This seems to work fine: def relimp(nm): rpth = string.split(nm, '/') tpth = string.split(__name__, '.')[:-1] for node in rpth: if node == '..': del tpth[-1] else: tpth.append(node) #print `tpth` return __import__(string.join(tpth, '.')) b = relimp('../packageA2.b') > As more and more packages appear, we run into this problem more > and more often. Relative imports would greatly reduce the > risk these collisions, because packages could be written > self-contained meaning that they can reach their internal > modules via relative paths only and don't have to know about > the absolute paths by which they are reachable from other parts > of the Python universe. > > I could then make all my packages self-contained and > distribute them in two forms without having to change > a single line for the added support: > > 1. under the new 'mx' package, e.g. mx.DateTime > 2. for backward compatibility under 'DateTime' > > Another major advantage is that I could take any other > self-contained package and install it under absolute paths > of my choice, e.g. put Zope under org.zope.core, Python under > org.python.core etc., without harming their functionality > or having to dive deep into their import structures to fix > them manually. > > To further enhance this mechanism I would like to have an > alias mechanism in import, pickle et al. so that changes > in the package structures become manageable without user > intervention: pickles are a major problem whenever import > structures change because they store absolute module names. > > -- > Marc-Andre Lemburg > ______________________________________________________________________ > Y2000: 111 days left Business: > http://www.lemburg.com/ Python Pages: > http://www.lemburg.com/python/ > > > > _______________________________________________ > Python-Dev maillist - Python-Dev at python.org > http://www.python.org/mailman/listinfo/python-dev - Gordon From tim_one at email.msn.com Sun Sep 12 11:18:16 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sun, 12 Sep 1999 05:18:16 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909101405.KAA26303@eric.cnri.reston.va.us> Message-ID: <000001befcff$bf654e20$492d153f@tim> [MarkH] > ... > I based my assesment simply on my perception of what is likely to > happen, not my opinion of what _should_ happen. I based mine on what Guido was waiting for someone to say . We worry too much about disagreeing here; different opinions are great! Guido will squash the ones he can't stand anyway. [about Aaron's pylint's lack of 1.5.2 smarts] > ... > Aaron agrees that a parser module based one would be better. You can't beat a real parse, no argument there. Luckily, the compiler parses too. [Guido] > What stands in the way? > > (a) There's so much else to do... How did Perl manage to attract 150 people with nothing to do except hack on Perl internals? "Wow, that code's such a mess I bet even *I* could get something into it" <0.6 wink>. > (b) *Someone* needs to design a good framework for spitting out > warnings, and for giving the programmer a way to control which > warnings to ignore. I've seen plenty of good suggestions here; now > somebody should simply go off and come up with a proposal too good to > refuse. The response has been ... absent. Anyone doing this? I liked JimF's push to make cmd-line options available to Python programs too. Somehow they seem related to me. > (c) I have a different agenda with CP4E -- I think it would be great > if the editor could do syntax checking and beyond *while you type*, > like the spell checker in MS Word. (I *like* Word's spell checker, > even though I hate the style checker [too stupid], and I gladly put up > with the spell checker's spurious complaints -- it's easy to turn off, > easy to ignore, and it finds lots of good stuff.) > > Because the editor has to deal with incomplete and sometimes > ungrammatical things, and because it has to work in real time (staying > out of the way when you're frantically typing, catching up when your > fingers take a rest), it needs a different kind of parser. Different from what? Python's own parser for sure. IDLE has at least two distinct parsers of its own that have nothing in common with Python's parser and little in common with each other. Using the horrid tricks in PyParse.py, it may even be possible to write the kind of parser you need in Python and still have it run fast enough. For parsing-on-the-fly from random positions, take my word for it and Barry's as insurance : the single most frequent question you need to have a fast and reliable answer for is "is this character in a string?". Unfortunately, turns out that's the hardest question to answer too. The next one is "am I on a continuation line, and if so where's the start?". Given rapid & bulletproof ways to answer those, the rest is pretty easy. > But that's another project, and once the Python core has a warning > framework in place, I'm sure we'll find more things that are worth > warning about. That was frequently predicted for various pylint projects too . > I'm not always in agreement with Tim Peters when he says that Python > is so dynamic that it's impossible to check for certain errors. It > may be impossible to say *for sure* that something is an error, but > there sure are lots of situations where you're doing something that's > *likely* to be an error. We have no disagreement there. What a compiler does must match the advertised semantics of the language-- or its own advertised deviations from those --without compromise. A warning system has no such constraint; to the contrary, in the case of a semantic mess like Perl, most of its value is in pointing out *legal* constructs that are unlikely to work the way you intended. > E.g. if the compiler sees len(1), and there's no local or global > variable named len, it *could* be the case that the user has set up a > parallel universe where the len() built-in accepts an integer > argument, but more likely it's an honest mistake, and I would like to > get a warning about it. Me too. More: I'd also like to get a warning for *having* a local or global variable named len! Masking the builtin names is simply bad practice, and is also usually an honest mistake. BTW, I was surprised that the most frequent gotcha among new Python users at Dragon turned out to be exactly that: dropping a "len" or a "str" or whatever (I believe len, str and list were most common) into their previously working code-- because they just learned about that builtin --and getting runtime errors as a result. That is, they already had a local var of that name, and forgot. Then they were irked that Python didn't nag them from the start (with a msg they understood, of course). > The hard part here is to issue this warning for len(x) where x is some > variable or expression that is likely to have a non-sequence value > (barring alternate universes); this might require global analysis > that's hard or expensive enough that we can't put it in the core > (yet). This may be seen as an argument for a separate lint... Curiously, Haskell is statically type-safe but doesn't require declarations of any kind -- it does global analysis, and has a 100% reliable type inference engine (the language was, of course, designed to make this true). Yet I don't think I've ever seen a Haskell program on the web that didn't explicitly declare the type of every global anyway. I think this is natural, too: while it's a PITA to declare the type of every stinking local that lives for two lines and then vanishes, the types of non-local names aren't self-evident: type decls really help for them. So if anyone is thinking of doing the kind of global analysis Guido mentions here, and is capable of doing it , I'd much rather they put their effort into optional static type decls for Python2. Many of the same questions need to be answered either way (like "what's a type?", and "how do we spell a type?" -- the global analysis warnings won't do any good if you can't communicate the substance of an error ), and optional decls are likely to have bigger bang for the buck. [Skip Montanaro] > ... > Perl's experience with -w seems to suggest that it's best to always > enable whatever warnings you can as well. While that's my position, I don't want to oversell the Perl experience. That language allows so many goofy constructs, and does so many wild guesses at runtime, that Perl is flatly unusable without -w for non-trivial programs. Not true of Python, although the kinds of warnings people have suggested so far certainly do seem worthy of loud complaint by default. > (More and more I see people using gcc's -Wall flag as well.) If you have to write portable C++ code, and don't enable every warning you can get on every compiler you have, and don't also turn on "treat warnings as errors", non-portable code will sneak into the project rapidly. That's my experience, over & over. gcc catches stuff MS doesn't, and vice versa, and MetroWerks yet another blob, and platform-specific cruft *still* gets in. It's irksome. > Now, my return consistency stuff was easy enough to write in C for two > reasons. One, I'm fairly comfortable with the compile.c code. I don't anticipate dozens of people submitting new warning code. It would be unprecendented if even two of us decided this was our thing. If would be almost unprecendented if even one of us followed up on it <0.6 wink>. > Two, adding my checks required no extra memory management overhead. Really good global analysis likely requires again as much C code as already exists. Luckily, I don't think putting in some warnings requires that all conceivable warnings be implemented at once . For stuff that complex, I'd rather make it optional and write it in Python; I don't believe any law prevents the compiler from running Python code. > Consider a few other checks you might conceivably add to the byte code > compiler: > > * tab nanny stuff (already enabled with -t, right?) Very timidly, yes . Doesn't complain by default, and you need -tt to make it an error. Only catches 4 vs 8 tab size ambiguities, but that's good enough for almost everyone almost all the time. > * variables set but not used > * variables used before set These would be wonderful. The Perl/pylint "gripe about names unique in a module" is a cheap approximation that gets a surprising percentage of the benefit for the cost of a dict and an exception list. > If all of this sort of stuff is added to the compiler proper, I predict a > couple major problems will surface: > > * The complexity of the code will increase significantly, making it > harder to maintain and enhance The complexity of the existing code should be almost entirely unaffected, because non-trivial semantic analysis belongs in a new subsystem with its own code. > * Fewer and fewer people will be able to approach the code, making it > less likely that new checks are added As opposed to what? The status quo, with no checks at all? Somehow, facing the prospect of *some* checks doesn't frighten me away . Besides, I don't buy the premise: if someone takes this on as their project, worrying that they'll decline to add new valuable checks is like MarkH worrying that I wouldn't finish adding full support for stinking tabs to the common IDLE/PythonWin editing components. People take pride in their hackery. > * Future extensions like pluggable virtual machines will be harder > to add because their byte code compilers will be harder to integrate > into the core If you're picturing adding this stuff sprayed throughout the guts of the existing com_xxx routines, we've got different pictures in mind. Semantic analysis is usually a pass between parsing and code generation, transforming the parse tree and complaining about source it thinks is fishy. If done in any conventional way, it has no crosstalk at all with either the parsing work that precedes it or the code generation that follows it. It's a pipe stage between them, whose output is of the same type as its input. That is, it's a "pluggable component" in its own right, and doesn't even need to be run. So potential for interference just isn't there. At present, Python is very unusual both in: 1) Having no identifiable semantic pass at all, parsing directly to byte code, and enforcing its few semantic constraints (like "'continue' not properly in loop") lumped in with both of those. and 2) Having one trivial optimization pass-- 76 lines of code instead of the usual 76,000 --running after the byte code has been generated. However, the sole transformation made here (distinguishing local from non-local names) is much more properly viewed as being a part of semantic analysis than as being "an optimization". It's deducing trivial info about what names *mean* (i.e., basic semantics), and is called optimization here only because Python didn't do it at first. So relating this to a traditional compiler, I'd say that "optimize()" is truly Python's semantic analysis pass, and all that comes before it is the parsing pass -- a parsing pass with output in a form that's unfortunately clumsy for further semantic analysis, but so it goes. The byte code is such a direct reflection of the parse tree that there's really little fundamental difference between them. So for minimal disruption, I'd move "optimize" into a new module and call it the semantic analysis pass, and it would work with the byte code. Just as now, you wouldn't *need* to call it at all. Unlike now, the parsing pass probably needs to save away some more info (e.g., I don't *think* it keeps track of what all is in a module now in any usable way). For Python2, I hope Guido adopts a more traditional structure (i.e., parsing produces a parse tree, codegen produces bytecode from a parse tree, and other tree->tree transformers can be plugged in between them). Almost all compilers follow this structure, and not because compiler writers are unimaginative droids . Compile-time for Python isn't anywhere near being a problem now, even on my creaky old 166MHz machine; I suspect the current structure reflects worry about that on much older & slower machines. Some of the most useful Perl msgs need to be in com_xxx, though, or even earlier. The most glaring example is runaway triple-quoted strings. Python's "invalid token" pointing at the end of the file is maddeningly unhelpful; Perl says it looks like you have a runaway string, and gives the line number it thinks it may have started on. That guess is usually correct, or points you to what you *thought* was the end of a different string. Either way your recovery work is slashed. (Of course IDLE is even better: the whole tail of the file changes to "string color", and you just need to look up until the color changes!) > In addition, more global checks probably won't be possible (reasoning about > code across module boundaries for instance) because the compiler's view of > the world is fairly narrow. As above, I don't think there's enough now even to analyze one module in isolation. > I think lint-like tools should be implemented in Python (possibly with the > support of an extension module for performance-critical sections) which is > then called from the compiler proper under appropriate circumstances > (warnings enabled, all necessary modules importable, etc). I have no objection to that. I do object to the barely conceivable getting in the way of the plainly achievable, though -- the pylint tools out there now, just like your return consistency checker, do a real service already without any global analysis. Moving that much into the core (implemented in Python if possible, recoded in C if not) gets a large chunk of the potential benefit for maybe 5% of the eventual work. It's nice that Guido is suddenly keen on global analysis too, but I don't see him volunteering to do any work either . > I believe the code would be much more easily maintained and extended. If it's written in Python, of course. > You'd be able to swap in a new byte code compiler without risking the > loss of your checking code. I never understood this one; even if there *were* a competing byte code compiler out there <0.1 wink>, keeping as much as possible out of com_xxx should render it a non-issue. If I'm missing your point and there is some fundamental conflict here, fine, then it's another basis on which bytecode compilers will compete. more-concerned-about-things-that-exist-than-things-that-don't-ly y'rs - tim From iporres at abo.fi Sun Sep 12 11:54:36 1999 From: iporres at abo.fi (Ivan Porres) Date: Sun, 12 Sep 1999 12:54:36 +0300 Subject: [Python-Dev] Memory usage in Python Message-ID: <19990912125436.B735@marchica.abo.fi> Dear all, I've been reading this list for a while but this is the first time I post a message. I am interested on using Python as a scripting language for small embedded systems. One of the limiting factors on a embedded system is the memory. Memory cost money, uses space and drains power. We have released a big patch to selectively disable some features of a Python interpreter at compile time. With the patch, it is possible to disable, i.e., complex numbers or file objects, in order to reduce the memory foot print of the Python interpreter. The patch is available at http://www.abo.fi/~iporres/python Now, I am interested on studying and reducing the memory consumption of Python programs. That is all the memory allocated by the interpreter in order to run a python program. I would like to know if somebody else has been working on the same direction. Is there any document explaining how Python uses the memory? I would be nice to know which structures are most used, their size, how often are accessed, etc... I have the impression that most people is concerned about optimising Python for speed, but I still think that some Python users will also benefit from size optimisations. Ivan -- Ivan Porres Paltor Turku Centre for Computer Science ?bo Akademi, Department of Computer Science Phone: +358-2-2154033 Lemmink?inengatan 14A FIN-20520 Turku - Finland http://www.abo.fi/~iporres From skip at mojam.com Sun Sep 12 16:20:14 1999 From: skip at mojam.com (Skip Montanaro) Date: Sun, 12 Sep 1999 09:20:14 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000001befcff$bf654e20$492d153f@tim> References: <199909101405.KAA26303@eric.cnri.reston.va.us> <000001befcff$bf654e20$492d153f@tim> Message-ID: <14299.46750.214399.735453@dolphin.mojam.com> Skip> * Future extensions like pluggable virtual machines will be harder Skip> to add because their byte code compilers will be harder to Skip> integrate into the core Tim> If you're picturing adding this stuff sprayed throughout the guts Tim> of the existing com_xxx routines, we've got different pictures in Tim> mind. This was precisely my example, because that's the way I implemented the return warning stuff, by modifying the com_xxx routines. I believe that's the wrong way to go in the long run, and I see by the rest of your message you feel the same way as well. To the greatest extent possible, I think this stuff should be implemented in Python. (We may disagree on that point.) Being able to plug in new parse tree analysis/transformation modules between parse tree creation and code generation could at least be controlled from Python. Skip P.S. Something I just noticed: Since the node typedef (node.h) and the macros that manipulate nodes are shared across multiple files shouldn't they be named something slightly less likely to clash with other packages? From mal at lemburg.com Mon Sep 13 12:05:46 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:05:46 +0200 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> Message-ID: <37DCCC7A.545B44E6@lemburg.com> Tim Peters wrote: > > [Guido] > > I'd much rather use absolute package names for anything that's not in > > the same directory as the current module. > > [M.-A. Lemburg] > > Of course, you could do everything with absolute names, but then > > the package author would dictate the complete absolute path which > > is not always desirable since it can cause name collisions such > > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > > released) and Mark's win32 stuff. > > > > As more and more packages appear, we run into this problem more > > and more often. > > I never understand package complaints. Maybe it's the imprecision of the > language people use, or maybe because it's because people don't give fully > fleshed-out examples. Whatever, in the end, I never have any idea what the > complaint is, or in what way the solution is "solving" anything. My original post contained an example package using relative imports. The example uses intra-package imports across subpackage levels which currently is only possible using absolute module names (see below for a definition). Note that I did not want to start a discussion about absolute vs. relative names (I believe everybody agrees that realtive file names are a Good Thing). The 'import __.module' thing is not new: ni.py had support for this and my patch simply adds it back to the implementation. Think of the '__' as an additional feature that authors can use at their own will. They don't *have* to, just as you don't have to use relative file paths in your config files. It's just a convenience that some authors may want to use in their code. > In the above, "absolute" doesn't appear to mean "absolute" in any > OS-sensible sense. So what does it really mean? Does it mean the same > things to Guido and MAL? "Absolute" means fully qualified name, i.e. the complete path to the modules from the top-level root via all subpackage down to the module name itself, e.g. TextTools.Constants.TagTable. > In MAL's hint of examples, I don't see any problem. If mxDateTime unpacks > itself into a directory named DateTime, then *of course* it's going to > collide with other packages that want to do likewise. Install it into > mxDateTime instead, and take "absolute" to mean "any module that wants an > mxDateTime service and does not itself live directly in mxDateTime/ must > import the desired module via a path beginning 'mxDateTime.'", and > everything looks straightforward to me (and that outcome makes me infer that > this is thus probably what Guido has in mind too). This is what I intend to do: move all my stuff under a 'mx' package and then continue reinventing (faster, different, doesn't-slip-when-wet) wheels ;-). BTW, the queue thingie is just a proof of concept fun project. > Similarly for > win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be > good to have an explicit way to say "std library" -- "Lib." comes to mind). I'd prefer 'org.python.core.' or 'stdlib.' or just simply 'python.' but that's a different issue. > > I could then make all my packages self-contained and > > distribute them in two forms without having to change > > a single line for the added support: > > > > 1. under the new 'mx' package, e.g. mx.DateTime > > 2. for backward compatibility under 'DateTime' > > Ah, so that's what this is about. I vote screw #2. Releasing it that way > was a mistake. Not until Zope went Open Source ;-) ... > Better to live with the brief & finite pain of repairing it > than complicating Python to cover up for it. Screwing #2 is not possible unless I want all those already stored DateTime pickles to fail loading... ok, I could probably provide some kind of compatibility package which then redirects the import to mx.DateTime. > > Another major advantage is that I could take any other > > self-contained package and install it under absolute paths > > of my choice, e.g. put Zope under org.zope.core, Python under > > org.python.core etc., without harming their functionality > > or having to dive deep into their import structures to fix > > them manually. > > I view that not as an advantage but as harmful complication. Zope etc add > great value to a Python installation, and when I write a killer app full of > "import zope.this" and "import zope.that", I don't want the possibility that > it's not going to work on my client's machine just because their sysadmin > installed Zope into some silly site-specific path named after his soon-to-be > ex-girlfriend . I don't want a way to work around him doing that, > either: I don't want him to be able to screw me to begin with. Zope is not a package AFAIK (and probably never will be due to the pickle complications), so 'import zope.this' won't work anyways unless you add a Zope wrapping package of your own -- and this will only work for Zope modules not relying on other Zope modules unless they use relative imports. > > To further enhance this mechanism I would like to have an > > alias mechanism in import, pickle et al. so that changes > > in the package structures become manageable without user > > intervention: pickles are a major problem whenever import > > structures change because they store absolute module names. > > This is a different issue, and may have merits of its own. WRT the relative > import scheme, its advantage seems to lie in providing a way to partially > recover from the damage the new scheme causes <0.5 wink>. I'm not proposing a new scheme... only a convenience for package authors. > As is, the package name used by a release is part of its published > interface. You can't change it without causing pain, any more than you can > make incompatible changes to public class methods or input-output behavior. > In return, package clients are uniform, simple and portable, making life > easiest for the people who know least. The burden is on package authors to > choose names wisely, and that's where the burden should be. Sure, but we are heading into these kind of problems just now and have no proper solution at hand. Simply coping out by blaiming the package authors is not the right way to handle the situation; referring them to some global name registry (the one at NIST) isn't either. As the Zope example shows, it's not easy to move from half-packaged to full packaged (relative imports will ease this move, though) due to external references using absolute object type/class names. Changes in the package structure produce the same problems. It's not so much the burden of the programmer having to adjust his code to the new layout I'm talking about here: it's the user with all his data stored in object databases that worries me. > if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs - tim Not if just one of them decides to make it top-level... grabbing the name until eternity ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 13 12:17:31 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:17:31 +0200 Subject: [Python-Dev] Relative Package Imports References: <1275018545-12250772@hypernet.com> Message-ID: <37DCCF3B.2F1E962D@lemburg.com> Gordon McMillan wrote: > > M.-A. Lemburg wrote: > > Guido van Rossum wrote: > > > > > > I'd much rather use absolute package names for anything that's not > > > in the same directory as the current module. > > > > Of course, you could do everything with absolute names, but then > > the package author would dictate the complete absolute path which is > > not always desirable since it can cause name collisions such as > > DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) > > and Mark's win32 stuff. > > I can see your point (although I also believe that authors - Guido > excepted - should come up with collision-free names, probably in a > Java-ish scheme). Agreed. > But I strongly believe that import.c should be left alone, maybe > even to die. There are too many people doing import hooks to > make fiddling with its behavior safe. > > I'm also a strong proponent of Greg's imputil.py scheme, which > makes it a breeze to do import hooks. And my experience > disproves the notion that the import mechanism needs to be in C. If > you don't believe me, try the ZlibArchive stuff (which is cross > platform) from my Win32 installer stuff. You can pack the standard > library into one 475K file, and get a perceptible performance boost. You're probably right in saying that we don't need the code in C. I just wanted to avoid yet another import hook being incompatible with all the other existing hooks. Perhaps we should restart the import discussion all over and come up with a more flexbile 100% compatible framework based on Greg's imputil scheme. Then I could add my hook for the relative imports and be happy ;-) > OTOH, I could see doing a framework of packages, in which case > relative imports might be handy. This seems to work fine: > > def relimp(nm): > rpth = string.split(nm, '/') > tpth = string.split(__name__, '.')[:-1] > for node in rpth: > if node == '..': > del tpth[-1] > else: > tpth.append(node) > #print `tpth` > return __import__(string.join(tpth, '.')) > > b = relimp('../packageA2.b') This is pretty much how my patch works... except that I use the ni.py style '__' pseudo package instead of '../'. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 13 12:27:04 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:27:04 +0200 Subject: [Python-Dev] Memory usage in Python References: <19990912125436.B735@marchica.abo.fi> Message-ID: <37DCD178.15676B9F@lemburg.com> Ivan Porres wrote: > > Dear all, > > I've been reading this list for a while but this is the first time I > post a message. I am interested on using Python as a scripting > language for small embedded systems. One of the limiting factors on a > embedded system is the memory. Memory cost money, uses space and > drains power. > > We have released a big patch to selectively disable some features of a > Python interpreter at compile time. With the patch, it is possible to > disable, i.e., complex numbers or file objects, in order to reduce > the memory foot print of the Python interpreter. The patch is > available at http://www.abo.fi/~iporres/python > > Now, I am interested on studying and reducing the memory consumption of > Python programs. That is all the memory allocated by the interpreter > in order to run a python program. I would like to know if somebody > else has been working on the same direction. Is there any document > explaining how Python uses the memory? I would be nice to know which > structures are most used, their size, how often are accessed, > etc... You should have a look at Vladimir Marangozov's pymalloc (hope this URL still works): http://sirac.inrialpes.fr/~marangoz/python/ It has some logging facilities which give detailed reports about the storage sizes etc. > I have the impression that most people is concerned about optimising > Python for speed, but I still think that some Python users will > also benefit from size optimisations. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at digicool.com Mon Sep 13 14:44:56 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 08:44:56 -0400 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> Message-ID: <37DCF1C8.27CF657F@digicool.com> Guido van Rossum wrote: > > > I've spent the last two hours trying to get relative package > > imports to work because I need them for my extension packages > > which will soon all move under a new top-level package name > > to overcome the conflicts with PIL and Zope. > > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. I'll second Marc-Andre here. A significant headache occurs when you have a package that has sub-packages. Sub-packages need to be able to reference other sub-packages within the same package without knowing where the containing package is installed. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From guido at CNRI.Reston.VA.US Mon Sep 13 15:09:59 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 13 Sep 1999 09:09:59 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 08:44:56 EDT." <37DCF1C8.27CF657F@digicool.com> References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> Message-ID: <199909131309.JAA29239@eric.cnri.reston.va.us> [MAL] > > > I've spent the last two hours trying to get relative package > > > imports to work because I need them for my extension packages > > > which will soon all move under a new top-level package name > > > to overcome the conflicts with PIL and Zope. [GvR] > > I'd much rather use absolute package names for anything that's not in > > the same directory as the current module. [JimF] > I'll second Marc-Andre here. > > A significant headache occurs when you have a package > that has sub-packages. Sub-packages need to be able to > reference other sub-packages within the same package without > knowing where the containing package is installed. You never need to know where it is installed. When I said absolute package name I meant package name (e.g. zope.foo.bar.subpack) not filename. As Tim has argued, the ability to change the name of the toplevel here is a liability, not a feature. (On reduced bandwidth here, hence not so subtle.) --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond at skippinet.com.au Mon Sep 13 15:14:56 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Mon, 13 Sep 1999 23:14:56 +1000 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DCF1C8.27CF657F@digicool.com> Message-ID: <000d01befde9$f9c70690$0801a8c0@bobcat> [Jim writes] > I'll second Marc-Andre here. > > A significant headache occurs when you have a package > that has sub-packages. Sub-packages need to be able to > reference other sub-packages within the same package without > knowing where the containing package is installed. I certainly dont want to know _where_ the package is installed, but do want to know the package name. A package name, IMO, is like a module name - once named, you are largely stuck with it. I for one use packages quite extensively, but have never lamented the loss of __ from the ni days. If someone, for example, wanted to change the package name of "win32com" or "pywin", then I would expect lots of code to break, in the same way as changing the name any standard module would break code. I see Marc's issue with pickle, but I dont see the problem with absolute package names any more than I see a problem with explicitely naming modules in an import statement... Mark. From jim at digicool.com Mon Sep 13 15:32:25 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:32:25 -0400 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> Message-ID: <37DCFCE9.A00DB3E0@digicool.com> Tim Peters wrote: > (snip) > As is, the package name used by a release is part of its published > interface. You can't change it without causing pain, any more than you can > make incompatible changes to public class methods or input-output behavior. > In return, package clients are uniform, simple and portable, making life > easiest for the people who know least. The burden is on package authors to > choose names wisely, and that's where the burden should be. Not all packages are part of the external interface. In fact, all Zope names are essentially internal, since Zope is an application. The issue is not so much access to access from outside as it is access between packages within Zope. Further, the current support for relative imports allows a package to be moved into another package without breaking the pulic interface wrt the containing package. Here's an example that I hope will be motivating: Suppose Marc-Andre has a package mx with subpackages DateTime and stringtools. If mx was installed in the Python path then a module in the mx.DateTime package could get at stringtools like: import mx.stringtools So far, so good. Zope has a notion of products which are *self contained* packages that are sub-packages of the Products package. So, suppose someone wants to write a NiftyDB product, which is a Zope product that provides access to an external database. Now the author of the NiftyDB product wants to use the mx package. The mx package is not a standard part of Zope, or of Python, so they simpley include it in the NiftyDB product directory. Becase relative imports are allowed in the current import scheme, they can use mx as usual. A NiftyDB module can import DateTime as follows: import mx.DateTime So even though mx is istalled as a sub-package, the public interface is unchanged, at least wrt the containing package. Unfortunately, the internal import of stringtools in the DateTime package: import mx.stringtools will fail, because mx is no longer a top-level module. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Mon Sep 13 15:32:39 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:32:39 -0400 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> Message-ID: <37DCFCF7.DACEF21D@digicool.com> "M.-A. Lemburg" wrote: > > Tim Peters wrote: > > > > [Guido] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > > > > [M.-A. Lemburg] > > > Of course, you could do everything with absolute names, but then > > > the package author would dictate the complete absolute path which > > > is not always desirable since it can cause name collisions such > > > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > > > released) and Mark's win32 stuff. > > > > > > As more and more packages appear, we run into this problem more > > > and more often. > > > > I never understand package complaints. Maybe it's the imprecision of the > > language people use, or maybe because it's because people don't give fully > > fleshed-out examples. Whatever, in the end, I never have any idea what the > > complaint is, or in what way the solution is "solving" anything. > > My original post contained an example package using relative > imports. The example uses intra-package imports across subpackage > levels which currently is only possible using absolute module > names (see below for a definition). I agree that this is a valid complaint about the current scheme. > Note that I did not want to start a discussion about absolute vs. > relative names (I believe everybody agrees that realtive file names > are a Good Thing). The 'import __.module' thing is not new: ni.py > had support for this and my patch simply adds it back to the > implementation. And, in fact, the current scheme does let you use relative paths to go down. (snip) > > > I could then make all my packages self-contained and > > > distribute them in two forms without having to change > > > a single line for the added support: > > > > > > 1. under the new 'mx' package, e.g. mx.DateTime > > > 2. for backward compatibility under 'DateTime' > > > > Ah, so that's what this is about. I vote screw #2. Releasing it that way > > was a mistake. > > Not until Zope went Open Source ;-) ... > > > Better to live with the brief & finite pain of repairing it > > than complicating Python to cover up for it. > > Screwing #2 is not possible unless I want all those already > stored DateTime pickles to fail loading... ok, I could probably > provide some kind of compatibility package which then redirects > the import to mx.DateTime. Regardless of how you spell the import, the pickles *must* reflect the absolute path. Otherwise, pickleability depends on where you unpickle. If you rearrange packages, or rename modules, there is a pickling issue. This is clearly a problem in need of a better solution. (snip) > Zope is not a package AFAIK Nope. If I had it to do over it would be. It will be eventually. (While Zope is not a package, it makes extensive use of packges. Zope has a total of about 350 modules and packages, of which about 44 are in the top-level namespace.) > (and probably never will be due to the pickle complications), The pickling issues are solvable in a number of ways, although it is a bit painfull. I considered fixing the package layout in Zope 2, but I ran out of time. :) > so 'import zope.this' won't work anyways > unless you add a Zope wrapping package of your own -- and this will > only work for Zope modules not relying on other Zope modules unless > they use relative imports. This is a good example. It should be possible to make Zope package- portable but it isn't, at least not without writing import hooks, which make my head hurt alot more than meta-classes. ;) > > > To further enhance this mechanism I would like to have an > > > alias mechanism in import, pickle et al. so that changes > > > in the package structures become manageable without user > > > intervention: pickles are a major problem whenever import > > > structures change because they store absolute module names. > > > > This is a different issue, and may have merits of its own. WRT the relative > > import scheme, its advantage seems to lie in providing a way to partially > > recover from the damage the new scheme causes <0.5 wink>. > > I'm not proposing a new scheme... only a convenience for package > authors. I think that this is an important convenience that is probably not appreciated until you write a complex package structure. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Mon Sep 13 15:35:21 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:35:21 -0400 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> <199909131309.JAA29239@eric.cnri.reston.va.us> Message-ID: <37DCFD99.F48FC65E@digicool.com> Guido van Rossum wrote: > > [MAL] > > > > I've spent the last two hours trying to get relative package > > > > imports to work because I need them for my extension packages > > > > which will soon all move under a new top-level package name > > > > to overcome the conflicts with PIL and Zope. > [GvR] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > [JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. > > You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. I wasn't refering to file path location either, but to package name-space location. See the example I just posted in reply to Tim. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From guido at CNRI.Reston.VA.US Mon Sep 13 15:40:10 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 13 Sep 1999 09:40:10 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 12:05:46 +0200." <37DCCC7A.545B44E6@lemburg.com> References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> Message-ID: <199909131340.JAA29340@eric.cnri.reston.va.us> > Note that I did not want to start a discussion about absolute vs. > relative names (I believe everybody agrees that realtive file names > are a Good Thing). The 'import __.module' thing is not new: ni.py > had support for this and my patch simply adds it back to the > implementation. It was left out of the 1.5 package implementation on purpose, and I will fight proposals to get it back. It is an ugly hack that deserves to die. So there! > "Absolute" means fully qualified name, i.e. the complete path to > the modules from the top-level root via all subpackage down to the > module name itself, e.g. TextTools.Constants.TagTable. No disagreement there. BTW, The compatibility issues with class references in object databases need to be resolved in a different way -- package renamings are but one of the problems here. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at lemburg.com Mon Sep 13 15:57:32 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 15:57:32 +0200 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> <199909131309.JAA29239@eric.cnri.reston.va.us> Message-ID: <37DD02CC.26B20BDD@lemburg.com> Guido van Rossum wrote: > > [MAL] > > > > I've spent the last two hours trying to get relative package > > > > imports to work because I need them for my extension packages > > > > which will soon all move under a new top-level package name > > > > to overcome the conflicts with PIL and Zope. > [GvR] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > [JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. > > You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. I think Jim meant the absolute package name just like you and I did. I don't really understand why you and Tim oppose to relative imports... Tim's argument about sysadmins installing packages under fantasy names isn't really all that realistic, IMHO, even though it would work. The ability to host a subpackage inside another package is a very attractive feature for large package based applications: you don't have to edit a single line in the subpackage's source code and can install/update the subpackage without harming its functionality within the system. BTW: The patch I posted does not add a circular reference to the modules as the old ni.py implementation (sometimes) did. Since the __ handling is done dynamically and on import only there is no need to add a module attribute __ pointing to the parent module. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 13 16:08:36 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 16:08:36 +0200 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> <199909131340.JAA29340@eric.cnri.reston.va.us> Message-ID: <37DD0564.7361B789@lemburg.com> Guido van Rossum wrote: > > > Note that I did not want to start a discussion about absolute vs. > > relative names (I believe everybody agrees that realtive file names > > are a Good Thing). The 'import __.module' thing is not new: ni.py > > had support for this and my patch simply adds it back to the > > implementation. > > It was left out of the 1.5 package implementation on purpose, and I > will fight proposals to get it back. It is an ugly hack that deserves > to die. So there! I agree that it's ugly to include the __ attribute in the module namespace due to the possible circular reference (parent->module, module->parent), but the patch I sent doesn't do this... or was "ugly" referring to the two underscores looking strange ? Could you elaborate a bit on the reasons for dropping __ support ? Not-really-looking-for-a-fight-here--ly, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Mon Sep 13 16:35:16 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Mon, 13 Sep 1999 10:35:16 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909131309.JAA29239@eric.cnri.reston.va.us> References: Your message of "Mon, 13 Sep 1999 08:44:56 EDT." <37DCF1C8.27CF657F@digicool.com> Message-ID: <1274889332-20022765@hypernet.com> A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 1587 bytes Desc: not available URL: From gmcm at hypernet.com Mon Sep 13 16:56:12 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Mon, 13 Sep 1999 10:56:12 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DCFCE9.A00DB3E0@digicool.com> Message-ID: <1274888077-20098343@hypernet.com> [Jim Fulton] > Here's an example that I hope will be motivating: > > Suppose Marc-Andre has a package mx with subpackages DateTime > and stringtools. ... > Zope has a notion of products which are *self contained* packages that > are sub-packages of the Products package. So, suppose someone wants > to write a NiftyDB product, which is a Zope product that provides > access to an external database. Now the author of the NiftyDB product > wants to use the mx package. The mx package is not a standard part of > Zope, or of Python, so they simpley include it in the NiftyDB product > directory. This would motivate me to strangle the author of NiftyDB. He should say "This requires Marc-Andre's mx package", and put error handling on the import. Otherwise I might end up with a gazillion separate copies of the mx package installed on my system and no way to clean it up. Your notion of "self contained packages" is flawed. Reducto ad adsurdum, it leads to downloading the OS with every app you install. (Crap, I hope I haven't given Sun any new ideas...). - Gordon From jim at digicool.com Mon Sep 13 17:16:03 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 11:16:03 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> Message-ID: <37DD1533.24A05F69@digicool.com> Gordon McMillan wrote: > > [Jim Fulton] > > Here's an example that I hope will be motivating: > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > and stringtools. > ... > > Zope has a notion of products which are *self contained* packages that > > are sub-packages of the Products package. So, suppose someone wants > > to write a NiftyDB product, which is a Zope product that provides > > access to an external database. Now the author of the NiftyDB product > > wants to use the mx package. The mx package is not a standard part of > > Zope, or of Python, so they simpley include it in the NiftyDB product > > directory. > > This would motivate me to strangle the author of NiftyDB. He > should say "This requires Marc-Andre's mx package", and put error > handling on the import. This is unacceptable. People should not have to hack a global namespace to get a local feature. > Otherwise I might end up with a gazillion > separate copies of the mx package installed on my system and no > way to clean it up. Or you might not. In any case, the ability to have localized references should be there. Actually, Zope does provide a place to share packages that multiple products use. This involves putting shared packages in a sub-package of the 'Shared' package. For example, Digital Creations shared packages go in Shared.DC. Putting mx in Shared.ACME runs into the same problem. Although the product author can: import Shared.ACME.mx.DateTime the import of mx.stringtools in DateTime will still fail. Now, maybe ACME will hack mx to change the internal imports: import Shared.ACME.mx.stringutils but their hacks will get broken when we make Zope a package and Shared becomes a sub-package of Zope. > Your notion of "self contained packages" is flawed. Reducto ad > adsurdum, it leads to downloading the OS with every app you > install. (Crap, I hope I haven't given Sun any new ideas...). I'm sorry, but relying on existing Python installations for commercial applications is just not practical. In fact, Zope binary distributions contain their own Python distributions because relying on preinstalled Python distributions is just way too much of a support burden and a hassle for out customers. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Mon Sep 13 17:21:15 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 11:21:15 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> Message-ID: <37DD166B.746B19AD@digicool.com> Gordon McMillan wrote: > > [relative package imports] > [JimF] > > > I'll second Marc-Andre here. > > > > > > A significant headache occurs when you have a package > > > that has sub-packages. Sub-packages need to be able to > > > reference other sub-packages within the same package without > > > knowing where the containing package is installed. > [GvR] > > You never need to know where it is installed. When I said absolute > > package name I meant package name (e.g. zope.foo.bar.subpack) not > > filename. As Tim has argued, the ability to change the name of the > > toplevel here is a liability, not a feature. > > In between. I can see relative packages as *one* way of handling > certain problems. Consider: > > zope.Win32 > zope.Unix > > with both of these having alternate implementations of subpackages foo > and bar. Then for (the current) foo.a to get to (the current) bar.b, > using a relative import seems a natural. I wouldn't use relative imports for this. > This can, of course, be done in pure Python. So can doing things in > zope.__init__.py that make the appropriate implementations of foo and > bar appear to be zope.foo and zope.bar. On any criteria I can think of, > this would be a superior solution. (*) I agree. This is not a good motivating example for relative imports. > What I am against is further complicating the already over complicated > built in import mechanism. The current scheme doesn't seem all that complicated to me. I wouldn't be in favor of making it more complicated if there wasn't a good reason. I think that, in working on the Zope framework, I've found some pretty good reasons for relative imports. > > (*) such as a zope.__init__.py that looks like this: > > import sys > if sys.platform[:3] == 'win': > nm = __name__ + '.Win32' > else: > nm = __name__ + '.Unix' > new = __import__(nm) > sys.modules[__name__] = sys.modules[nm] Right, this would be silly. This is not a good example. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal at lemburg.com Mon Sep 13 17:27:31 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 17:27:31 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> Message-ID: <37DD17E3.74FEB43F@lemburg.com> Gordon McMillan wrote: > > [Jim Fulton] > > Here's an example that I hope will be motivating: > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > and stringtools. > ... > > Zope has a notion of products which are *self contained* packages that > > are sub-packages of the Products package. So, suppose someone wants > > to write a NiftyDB product, which is a Zope product that provides > > access to an external database. Now the author of the NiftyDB product > > wants to use the mx package. The mx package is not a standard part of > > Zope, or of Python, so they simpley include it in the NiftyDB product > > directory. > > This would motivate me to strangle the author of NiftyDB. He > should say "This requires Marc-Andre's mx package", and put error > handling on the import. Otherwise I might end up with a gazillion > separate copies of the mx package installed on my system and no > way to clean it up. The mx package installed in product dir won't be visible to outside scripts: it is self-contained and only works in the context of Zope. There's nothing bad about it if you take the POV of a typical Zope user who doesn't know about the internals too much. OTOH, if the user does have the skill to install packages himself, he could install the mx package top-level and then leave it out of the NiftyDB product dir. Both setups will work without any source code change... that's the real advantage. Currently, only the top-level installation of the mx package will work and this of course introduces other problems like version control, e.g. the NiftyDB product (relying on mx 1.9) may not support version 2.0 of mx which the sysadmin installed at top-level. There would be no way (apart from patching NiftyDB) to get back to a working setup. > Your notion of "self contained packages" is flawed. Reducto ad > adsurdum, it leads to downloading the OS with every app you > install. (Crap, I hope I haven't given Sun any new ideas...). While you do of course have a point in that packages should not be loaded twice (a smarter import mechanism written in Python could handle these situations along with the relative import), you should consider that Zope is an application. It could go the WinXX way and simply hack around in the OS or wrap everything it needs up in the package itself. Both have their pros and cons, both are possible using relative imports (I'm starting to feel like one of those marketing guys ;). BTW, I really don't care much about this feature being coded in Python or C. All I would like is to have it enabled and available per default, so that writing self-contained packages becomes a standard in the Python world. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jcw at equi4.com Mon Sep 13 18:23:35 1999 From: jcw at equi4.com (Jean-Claude Wippler) Date: Mon, 13 Sep 1999 18:23:35 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> Message-ID: <37DD2507.8A7AD0D1@equi4.com> "M.-A. Lemburg" wrote: > Gordon McMillan wrote: > > Jim Fulton wrote: [deeply nested snippety snip] > > > > This would motivate me to strangle the author of NiftyDB. He > > should say "This requires Marc-Andre's mx package", [...] Yes. > The mx package installed in product dir won't be visible to outside > scripts: it is self-contained and only works in the context of Zope. But why shouldn't it be visible? The only approach which will not self-destruct IMO, is to segment on source-of-origin. Your package in your namespace, Gordon's in his, and Jim's in yet another. Given that there is no ordering relationship, that means three areas next to each other. If Jim takes M-A's package and modifies it, then that would be a reason to put M-A's-modified-by-Jim-package within Jim's area. > Currently, only the top-level installation of the mx package > will work and this of course introduces other problems like > version control, e.g. the NiftyDB product (relying on mx 1.9) > may not support version 2.0 of mx which the sysadmin installed > at top-level. There would be no way (apart from patching NiftyDB) > to get back to a working setup. Which points to a solution in a completely different direction: version numbers in package directories, like GNU and Tcl do, to name two. > [...] have it enabled and available per default, so that writing > self-contained packages becomes a standard in the Python world. A noble goal. Now this may be a silly question, but what's the issue here? I'm used to writing things which sort of look like this: import myGoobledygook utils = myGoobledygook print utils.fun(123) Couldn't this be applied here as well: mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") In other words: figure out a way to get at the proper modules, then use an alias in your own code to stay away from naming/access dependencies? And if a parent wants to tell a submodule how to find it, can't it just set a variable at module level in that submodule, to reach it? Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, DNS or LDAP servers, FS mount points - it's all the same issue. -- Jean-Claude From jim at digicool.com Mon Sep 13 18:56:42 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 12:56:42 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> Message-ID: <37DD2CCA.B5C29881@digicool.com> Jean-Claude Wippler wrote: > (snip) > > The mx package installed in product dir won't be visible to outside > > scripts: it is self-contained and only works in the context of Zope. > > But why shouldn't it be visible? Because visibility has a cost. Making mx a top-level name means that someone else can't make it a top-level name. This is why packages are a good idea. If the need is local, the definition shouldn't be made global *just* to get around a limitation in spelling. > The only approach which will not > self-destruct IMO, is to segment on source-of-origin. Your package in > your namespace, Gordon's in his, and Jim's in yet another. Given that > there is no ordering relationship, that means three areas next to each > other. It must be possible to package things together regardless of point of origin. Otherwise, you have a big obstical to reuse. BTW, I think that there is ample evidence that relative imports will not cause anything to self destruct. > If Jim takes M-A's package and modifies it, then that would be a > reason to put M-A's-modified-by-Jim-package within Jim's area. But Jim wants to use M-A's package as a black box. I don't want to hack all of his imports due to a packaging detail. (snip) > Now this may be a silly question, but what's the issue here? I'm used > to writing things which sort of look like this: > import myGoobledygook > utils = myGoobledygook > print utils.fun(123) > Couldn't this be applied here as well: > mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") > In other words: figure out a way to get at the proper modules, then use > an alias in your own code to stay away from naming/access dependencies? > > And if a parent wants to tell a submodule how to find it, can't it just > set a variable at module level in that submodule, to reach it? > > Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, > DNS or LDAP servers, FS mount points - it's all the same issue. Yes, these are options, as are import hooks. While these are possible alternatives, they are far less attractive than simply making it possible to spell relative imports. Why? Well, for one, they require a lot of cooperation among packages that might be written by totally different authors. I might come up with a handy relative import function, but how do I get M-A to use it? Well, I'm sure I could get him to use it, but how about all of the other people whose packages I want to reuse. No problem, we'll come up with a standard mechansism, maybe even a standard library module. Of course, someone is bound to realize sooner or later that it's silly to have a standard relative import library module *and* a standard import statement and add the feature to standard import. :) Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal at lemburg.com Mon Sep 13 20:21:08 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 20:21:08 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> Message-ID: <37DD4094.1BDCE45F@lemburg.com> Jean-Claude Wippler wrote: > > "M.-A. Lemburg" wrote: > > Gordon McMillan wrote: > > > Jim Fulton wrote: > [deeply nested snippety snip] > > > > > > This would motivate me to strangle the author of NiftyDB. He > > > should say "This requires Marc-Andre's mx package", [...] > > Yes. Of course ;-) ... but that's not the point. Commercial products have to come with all batteries included, not just some of them. But you can't include everything known in the Python universe just to avoid conflicts when someone writes a cool plugin for your package. > > The mx package installed in product dir won't be visible to outside > > scripts: it is self-contained and only works in the context of Zope. > > But why shouldn't it be visible? Because it would probably affect other apps using the same package. Just think of what happens to Win9x if you constantly update the DLLs... > The only approach which will not > self-destruct IMO, is to segment on source-of-origin. Your package in > your namespace, Gordon's in his, and Jim's in yet another. Given that > there is no ordering relationship, that means three areas next to each > other. If Jim takes M-A's package and modifies it, then that would be a > reason to put M-A's-modified-by-Jim-package within Jim's area. > > > Currently, only the top-level installation of the mx package > > will work and this of course introduces other problems like > > version control, e.g. the NiftyDB product (relying on mx 1.9) > > may not support version 2.0 of mx which the sysadmin installed > > at top-level. There would be no way (apart from patching NiftyDB) > > to get back to a working setup. > > Which points to a solution in a completely different direction: version > numbers in package directories, like GNU and Tcl do, to name two. A different topic -- an interesting one too :-) How does Java handle this problem ? > > [...] have it enabled and available per default, so that writing > > self-contained packages becomes a standard in the Python world. > > A noble goal. And one which is only reachable by getting whatever mechanism is needed to make it work into the standard distribution -- not only a hook for relative imports but the real thing ! > Now this may be a silly question, but what's the issue here? I'm used > to writing things which sort of look like this: > import myGoobledygook > utils = myGoobledygook > print utils.fun(123) > Couldn't this be applied here as well: > mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") > In other words: figure out a way to get at the proper modules, then use > an alias in your own code to stay away from naming/access dependencies? > > And if a parent wants to tell a submodule how to find it, can't it just > set a variable at module level in that submodule, to reach it? > > Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, > DNS or LDAP servers, FS mount points - it's all the same issue. True, but please remember that we are not talking about adding some esoteric module storage to the standard mechanism (the one reachable via the "import" statement). Relative import only adds a logical notion to imports that opens up new possibilities which are not easily available using other techniques. BTW, would you want to miss relative file name paths ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at interet.com Mon Sep 13 22:52:57 1999 From: jim at interet.com (James C. Ahlstrom) Date: Mon, 13 Sep 1999 16:52:57 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> Message-ID: <37DD6429.E4F6BF92@interet.com> Jim Fulton wrote: > > Gordon McMillan wrote: > > > > [Jim Fulton] > > > Here's an example that I hope will be motivating: > > > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > > and stringtools. > > ... > > > Zope has a notion of products which are *self contained* packages that > > > are sub-packages of the Products package. So, suppose someone wants > > > to write a NiftyDB product, which is a Zope product that provides > > > access to an external database. Now the author of the NiftyDB product > > > wants to use the mx package. The mx package is not a standard part of > > > Zope, or of Python, so they simpley include it in the NiftyDB product > > > directory. First, I am not all that opposed to having a notion of ".." available in the import statement. If we can write zope.dir1.mod1 which is a relative import going down, maybe we can write ../dir2/mod2 or something spelled differently. But I think there would still be problems. We would be relying on all package authors to use ".." or "__" within their package. But it is more natural to write zope.this.that or mx.this.that everywhere, and that is what people have in fact done. Probably there would be bugs and at least annoyance. And it seems unfortunate to need this feature. A proper package scheme should really isolate package internals from the outside world, and if this isn't happening we need a new scheme. I also don't think there is any escape from needing standard package names, at least the head name "zope" for example. Of course all this currently works if (in Jim's example) both "zope" and "mx" are top-level names. Then the ".." is not necessary. But, as Jim points out: >> But why shouldn't it be visible? >Because visibility has a cost. Making mx a top-level name >means that someone else can't make it a top-level name. >This is why packages are a good idea. The global shared nature of PYTHONPATH and its name space makes it difficult to guarantee that all required packages are going to be present in a complicated installation like zope. And if anyone else installs another Python package, it can easily break the first installation. One solution is for an application to establish its own PYTHONPATH which can not be altered. If this points to "zopedir" then the installer can freely install mx to the directory zopedir/mx and be confident that another mx installation is not damaged, nor used. The logical extension is to place each package into its own file using a scheme like Gordon McMillan is using. If the package contents is obtained by seeking from the END of the file, then multiple package files can be concattenated with cat package1 package2 zope mx >> bigpackage and a large installation like zope can be shipped with its own "bigpackage" library which is essentially a normal PYTHONPATH archive with everything above the head directory names thrown away. The Python library is included under "Lib" just as it is now. The bigpackage library implies its own PYTHONPATH of "Lib;." If the Python library files are placed in the root, the implied PYTHONPATH is ".". Jim Ahlstrom From jcw at equi4.com Mon Sep 13 23:55:43 1999 From: jcw at equi4.com (Jean-Claude Wippler) Date: Mon, 13 Sep 1999 23:55:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> <37DD4094.1BDCE45F@lemburg.com> Message-ID: <37DD72DF.D5904CC@equi4.com> Marc-Andre, > BTW, would you want to miss relative file name paths ? Well, that thought did cross my mind while posting that reply... I'm not sure. Yes, they are convenient, but "current working directory" is not always a pleasant concept (yet more global state - think of tree walks and the inconvenience of having to get, alter, work, restore it - when using chdir to implement this). I don't use .. as often as might be expected. I use "cd ~/something" a lot more. While that may look relative at another level, it really is not (as "echo ~" shows). It does illustrate how nearly equivalent the two approaches are. DOS/Windows has always had a broken relative path: the current drive. And the "hash/bang" headers of Unix seem to always use absolute paths. Some of this may seem to point to the need for relative paths. But I think it hides a more fundamental issue: you need to find out context. Once you do, relativeness no longer matters (obviously). A system which has a current directory and ".." is equivalent to one which has no such thing and passes a "starting directory" in the environment, say. I think a system with less global state is more modular -> preferable. Another angle: I have built many types of tree structures, still do. Less and less of those contain a "parent" link. Instead of storing a parent you can just as easily keep state while descending into children. - Unix file info does not maintain a parent directory, it's redundant. - Directories do, and file system mount points are messy because of it. - Afaik, it's considered bad style to use "../header.h" in C includes. - Upward pointers can introduce cycles (no not here, as has been said). - In C++, member objects rarely need pointers to the enclosing object. It is not for a module to know where it "is" in a hierarchy, it is for a parent to (sometimes) provide that reference when it uses the module. If you want want access to a module called mx, and there may be many of them - then you ask another module to locate it for you. One which is able to choose / decide for you. Proximity may be a good discriminator, but the decision of proximity was not taken by you - and you won't know what rule is best. So you ask an "importer" (which could well always be the standard Python mechanism). Given the task to write such as beast, I'd probably want to implement the following module search as default: - look for the module first in the parent (i.e. as sibling) - move one level up, look again, etc - all the way to the top That's just one way to do it - proximity is not *always* what you want. So if Zope chooses a different packaging style, let it override import. But please don't build ".." into your modules, it doesn't belong there. Sorry for all the handwaving generalities. The issues are the same IMO, whether relativeness is provided or context, and therefore the outcome of this discussion will never be conclusive - they both work. -- Jean-Claude From tim_one at email.msn.com Tue Sep 14 09:02:24 1999 From: tim_one at email.msn.com (Tim Peters) Date: Tue, 14 Sep 1999 03:02:24 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DD0564.7361B789@lemburg.com> Message-ID: <000001befe7f$191104c0$fe2d153f@tim> [MAL] > I agree that it's ugly to include the __ attribute in the module > namespace due to the possible circular reference (parent->module, > module->parent), but the patch I sent doesn't do this... or was > "ugly" referring to the two underscores looking strange ? > > Could you elaborate a bit on the reasons for dropping __ support ? There are two sections on why __ was dropped in http://www.python.org/doc/essays/packages.html They don't refer to circularity, but to "limited use", "poor readability" and "awkwardness". A deeper reason may be hiding in the essay's "most packages will have a relative shallow substructure": this is Guido , the man who invented two-level scoping, and class inheritance without a "super" hook back to the (anonymous) parent. For all Python's dynamicism, it very much favors shallow, static name hierarchies. I don't think it's coincidence that Python's own source code is in a two-level directory structure either! The only #include with a ".." is in grammar.h, and there it's in a comment : #include "bitset.h" /* Sigh... */ So if we cut to the core here, I'd bet Guido doesn't object so much to relative imports as to the idea that anyone would go off and create a package structure so fractally convoluted that relative imports are strongly more attractive than naming the target package in full. Or maybe Guido doesn't care about that at all. I do regardless. I know Python's restrictions can grate, but in all, and in my repeated experience, they force you to rethink complicated designs and refactor them into simpler schemes that fit what Python is best at spelling. Nesting packages 8 deep is clumsy now? Damn straight, and I'm thankful for that: the clumsier it is, the less gratuitous inherited complexity I'll have to deal with in my future lives <0.5 wink>. Things that came up in this thread that are worth fixing include: + Problems with persistent class references (incl. pickles). + Dealing with incompatible versions of packages. If someone wants to embed a copy of (say) mxDateTime in their own package, the only excuses are that they're afraid of overwriting the user's existing mxDateTime installation (if any), and/or of having some future installation of something else overwrite mxDateTime with an incompatible version. Those are bad problems, but package embedding is no solution. You have a much better approach to that already via the DateTime.__version__ string! "Something like that" needs to be formalized and imposed on all public packages. at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim PS: > Just think of what happens to Win9x if you constantly update the DLLs... As a matter of personal experience, it gets much stabler! The older DLLs get replaced by less-buggy newer ones, thanks to version numbers, rules, and installers that finally play by the rules. The mean time between crashes when I installed Win95 a few years ago was about an hour; now it's at least days and possibly weeks (don't know -- never leave the puter on that long). When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. From mal at lemburg.com Tue Sep 14 10:32:38 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 10:32:38 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> Message-ID: <37DE0825.10D2BC6F@lemburg.com> James C. Ahlstrom wrote: > > First, I am not all that opposed to having a notion of ".." > available in the import statement. If we can write zope.dir1.mod1 > which is a relative import going down, maybe we can write > ../dir2/mod2 or something spelled differently. But I think > there would still be problems. > > We would be relying on all package authors to use ".." > or "__" within their package. But it is more natural to write > zope.this.that or mx.this.that everywhere, and that is what > people have in fact done. Probably there would be bugs and > at least annoyance. Think of it as an opportunity: it makes intra-package imports very simple without hard-wiring paths to submodules into the package. In the end, having written a self-contained package is a "feature" of that package. Other authors can then say: ok, I can plugin your package if it's self-contained. -- I have a feeling that this discussion is loosing its grip on reality a bit. Jim and I are not argueing to change Python's face, internal structure or look&feel. The '__' thing doesn't harm anyone, doesn't break any functionality or code. It is even now already possible using import hooks and has been implemented before in ni.py using just that technology. All that we are talking about is reviving relative imports so that its usage is possible *without* adding yet another hook. This is not about the pros and cons of deeply nested packages and neither about advantages of shallow structures. Its only about adding semantics to the string '__' used in import statements. Nothing more. People are not forced to use it. Besides, if used it will be a package internal technique and not be visible to users of that package. Now why would anyone want to make life harder for package authors ? Give the people something to play with so that they have more fun at what their doing for the benefit of the community ! -- > And it seems unfortunate to need this feature. A proper > package scheme should really isolate package internals from > the outside world, and if this isn't happening we need a new scheme. > > I also don't think there is any escape from needing standard > package names, at least the head name "zope" for example. > > Of course all this currently works if (in Jim's example) both > "zope" and "mx" are top-level names. Then the ".." is not > necessary. But, as Jim points out: > > >> But why shouldn't it be visible? > > >Because visibility has a cost. Making mx a top-level name > >means that someone else can't make it a top-level name. > >This is why packages are a good idea. > > The global shared nature of PYTHONPATH and its name space makes > it difficult to guarantee that all required packages are going > to be present in a complicated installation like zope. And if > anyone else installs another Python package, it can easily break > the first installation. > > One solution is for an application to establish its own > PYTHONPATH which can not be altered. If this points to "zopedir" > then the installer can freely install mx to the directory > zopedir/mx and be confident that another mx installation > is not damaged, nor used. > > The logical extension is to place each package into its own > file using a scheme like Gordon McMillan is using. If the > package contents is obtained by seeking from the END of the > file, then multiple package files can be concattenated with > cat package1 package2 zope mx >> bigpackage > and a large installation like zope can be shipped with its > own "bigpackage" library which is essentially a normal > PYTHONPATH archive with everything above the head directory > names thrown away. The Python library is included under > "Lib" just as it is now. The bigpackage library implies > its own PYTHONPATH of "Lib;." If the Python library files are > placed in the root, the implied PYTHONPATH is ".". This is a possibility and in fact I'm using such a setup in my current application. Still it doesn't avoid possible conflicts due to being top-level, e.g. the user could install an extension which relies on a specific top-level name already taken by the hosting package, e.g. the host defines a DateTime package and the extension comes with its own DateTime package. This is the current situation with Zope and mxODBC (which needs mxDateTime). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Tue Sep 14 10:49:52 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 10:49:52 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> <37DD4094.1BDCE45F@lemburg.com> <37DD72DF.D5904CC@equi4.com> Message-ID: <37DE0C30.4B0521B@lemburg.com> Jean-Claude Wippler wrote: > > Marc-Andre, > > > BTW, would you want to miss relative file name paths ? > > Well, that thought did cross my mind while posting that reply... > > I'm not sure. Yes, they are convenient, but "current working directory" > is not always a pleasant concept (yet more global state - think of tree > walks and the inconvenience of having to get, alter, work, restore it - > when using chdir to implement this). I don't use .. as often as might > be expected. I use "cd ~/something" a lot more. While that may look > relative at another level, it really is not (as "echo ~" shows). It > does illustrate how nearly equivalent the two approaches are. > > DOS/Windows has always had a broken relative path: the current drive. > > And the "hash/bang" headers of Unix seem to always use absolute paths. ...and people have turned to /usr/bin/env to get around this necessity :-) > Some of this may seem to point to the need for relative paths. But I > think it hides a more fundamental issue: you need to find out context. > Once you do, relativeness no longer matters (obviously). A system which > has a current directory and ".." is equivalent to one which has no such > thing and passes a "starting directory" in the environment, say. > > I think a system with less global state is more modular -> preferable. Agreed. I would much prefer imputil.py to become a standard -- then I could add my hook into its chain and wouldn't have to argue for using relative imports ;-) But then, how would I tell if someone else already has integrated such a hook in the chain ? I sure wouldn't want to add another one just to make sure... If the standard mechanism already knows about '__' then I wouldn't have to worry. > Another angle: I have built many types of tree structures, still do. > Less and less of those contain a "parent" link. Instead of storing a > parent you can just as easily keep state while descending into children. > > - Unix file info does not maintain a parent directory, it's redundant. > - Directories do, and file system mount points are messy because of it. > - Afaik, it's considered bad style to use "../header.h" in C includes. > - Upward pointers can introduce cycles (no not here, as has been said). > - In C++, member objects rarely need pointers to the enclosing object. Note that my patch does exactly this: it resolves the __ while descending into the package structure -- without any explicit back reference stored in the modules. > It is not for a module to know where it "is" in a hierarchy, it is for a > parent to (sometimes) provide that reference when it uses the module. Right. > If you want want access to a module called mx, and there may be many of > them - then you ask another module to locate it for you. One which is > able to choose / decide for you. Proximity may be a good discriminator, > but the decision of proximity was not taken by you - and you won't know > what rule is best. So you ask an "importer" (which could well always be > the standard Python mechanism). Given the task to write such as beast, > I'd probably want to implement the following module search as default: > - look for the module first in the parent (i.e. as sibling) > - move one level up, look again, etc > - all the way to the top > That's just one way to do it - proximity is not *always* what you want. Now this is an interesting idea... looks like acquisition at the module import level. Right now the scheme is as follows: - look for the module in the import module's dir - look for the module on the PYTHONPATH You would just add a third lookup step in between the two... it would cause many more module lookups though. > So if Zope chooses a different packaging style, let it override import. > But please don't build ".." into your modules, it doesn't belong there. > > Sorry for all the handwaving generalities. The issues are the same IMO, > whether relativeness is provided or context, and therefore the outcome > of this discussion will never be conclusive - they both work. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Tue Sep 14 11:21:45 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 11:21:45 +0200 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <37DE13A9.4150AF0E@lemburg.com> Tim Peters wrote: > > [MAL] > > I agree that it's ugly to include the __ attribute in the module > > namespace due to the possible circular reference (parent->module, > > module->parent), but the patch I sent doesn't do this... or was > > "ugly" referring to the two underscores looking strange ? > > > > Could you elaborate a bit on the reasons for dropping __ support ? > > There are two sections on why __ was dropped in > > http://www.python.org/doc/essays/packages.html > > They don't refer to circularity, but to "limited use", "poor readability" > and "awkwardness". Ok, so the two underscores look strange... that's arguable. "Limited use" is not: there clearly is a use for relative imports. Please remember that we are talking about package internals here -- not user interfaces of that package. > A deeper reason may be hiding in the essay's "most > packages will have a relative shallow substructure": this is Guido , > the man who invented two-level scoping, and class inheritance without a > "super" hook back to the (anonymous) parent. For all Python's dynamicism, > it very much favors shallow, static name hierarchies. I don't think it's > coincidence that Python's own source code is in a two-level directory > structure either! The only #include with a ".." is in grammar.h, and there > it's in a comment : > > #include "bitset.h" /* Sigh... */ > > So if we cut to the core here, I'd bet Guido doesn't object so much to > relative imports as to the idea that anyone would go off and create a > package structure so fractally convoluted that relative imports are strongly > more attractive than naming the target package in full. This is not the intention of relative imports. Its all about making packages relocatable. And that's a Good Thing IMHO. > Or maybe Guido doesn't care about that at all. I do regardless. I know > Python's restrictions can grate, but in all, and in my repeated experience, > they force you to rethink complicated designs and refactor them into simpler > schemes that fit what Python is best at spelling. Nesting packages 8 deep > is clumsy now? Damn straight, and I'm thankful for that: the clumsier it > is, the less gratuitous inherited complexity I'll have to deal with in my > future lives <0.5 wink>. Tim, I promise you not use 8 levels deep ;-) I usually create my packages using at most two levels (moving to at most three levels in future mx package releases) and even better: the common package interfaces are all available at the top-most level while internal stuff hides in higher levels. You don't have to think much about the internal structure of the package while you use it. That's something I have to worry about and the advantages of relative imports show up in such a context which is why I posted the patch. > Things that came up in this thread that are worth fixing include: > > + Problems with persistent class references (incl. pickles). > > + Dealing with incompatible versions of packages. If someone wants to embed > a copy of (say) mxDateTime in their own package, the only excuses are that > they're afraid of overwriting the user's existing mxDateTime installation > (if any), and/or of having some future installation of something else > overwrite mxDateTime with an incompatible version. Those are bad problems, > but package embedding is no solution. You have a much better approach to > that already via the DateTime.__version__ string! "Something like that" > needs to be formalized and imposed on all public packages. So the distutils will have to check for already installed package versions and warn about conflicts. Right, that's something which definitely needs to become a standard (too ;). > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim > > PS: > > > Just think of what happens to Win9x if you constantly update the DLLs... > > As a matter of personal experience, it gets much stabler! The older DLLs > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > installers that finally play by the rules. The mean time between crashes > when I installed Win95 a few years ago was about an hour; now it's at least > days and possibly weeks (don't know -- never leave the puter on that long). > When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. My experience is that intra-DLL references simply don't match anymore and cause the system to become instable. Also, some weird installers don't care about the version numbers and install older versions at their will. The outcome is a complete version mess. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Tue Sep 14 14:21:26 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Tue, 14 Sep 1999 08:21:26 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE0C30.4B0521B@lemburg.com> Message-ID: <1274810963-24736486@hypernet.com> A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 892 bytes Desc: not available URL: From jim at digicool.com Tue Sep 14 14:20:53 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 12:20:53 +0000 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <37DE3DA4.3C7C14A8@digicool.com> Tim Peters wrote: > > [MAL] > > I agree that it's ugly to include the __ attribute in the module > > namespace due to the possible circular reference (parent->module, > > module->parent), but the patch I sent doesn't do this... or was > > "ugly" referring to the two underscores looking strange ? > > > > Could you elaborate a bit on the reasons for dropping __ support ? > > There are two sections on why __ was dropped in > > http://www.python.org/doc/essays/packages.html > > They don't refer to circularity, but to "limited use", "poor readability" > and "awkwardness". Which are all quite subjective. > A deeper reason may be hiding in the essay's "most > packages will have a relative shallow substructure": this is Guido , > the man who invented two-level scoping, and class inheritance without a > "super" hook back to the (anonymous) parent. For all Python's dynamicism, > it very much favors shallow, static name hierarchies. I don't think it's > coincidence that Python's own source code is in a two-level directory > structure either! The only #include with a ".." is in grammar.h, and there > it's in a comment : > > #include "bitset.h" /* Sigh... */ > > So if we cut to the core here, I'd bet Guido doesn't object so much to > relative imports as to the idea that anyone would go off and create a > package structure so fractally convoluted that relative imports are strongly > more attractive than naming the target package in full. > > Or maybe Guido doesn't care about that at all. I do regardless. I know > Python's restrictions can grate, but in all, and in my repeated experience, > they force you to rethink complicated designs and refactor them into simpler > schemes that fit what Python is best at spelling. Nesting packages 8 deep > is clumsy now? Damn straight, and I'm thankful for that: the clumsier it > is, the less gratuitous inherited complexity I'll have to deal with in my > future lives <0.5 wink>. Nobody has suggested building 8-level package hierarchies. In fact, the example I gave occured when a two-level package hierarchy was used in a two-level hierarchy. I don't think the Package structure of Zope is flawed *except* for the fact that it is one level too *shallow*. The ability to do relative imports would be very helpful for the work we're doing. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at interet.com Tue Sep 14 15:30:38 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 09:30:38 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> Message-ID: <37DE4DFE.FFD199ED@interet.com> "M.-A. Lemburg" wrote: > > Still it doesn't avoid possible conflicts > due to being top-level, e.g. the user could install an extension > which relies on a specific top-level name already taken by the hosting > package, e.g. the host defines a DateTime package and the > extension comes with its own DateTime package. This is the current > situation with Zope and mxODBC (which needs mxDateTime). Yes, the name conflict at the global level is exactly the problem. And to repeat, I don't really mind if "__" is added to imports, although I don't like the spelling. But... Perhaps a better solution is a multiple global name space. Suppose there is a command line option (or other mechanism such as a special file in sys.executable) which specifies sys.path for the application. To be specific, suppose zope is installed in the "zopehome" directory and the zope package lives in zopehome/zope. And suppose zope needs mx, so it installs it in zopehome/mx. This really is an obvious structure, and anyone could see that replacing mx/* would upgrade the mx package. The sys.path would be zopehome plus perhaps some subdirectories of zopehome. Doesn't this solve the problem? Why not? And don't complain about wasted disk space. The smallest disk you can buy today is 4 gigs and costs $150. I find the PYTHONPATH mechanism totally unreliable for commercial programs anyway. It is a global object, and an installation of a second Python program can break the first one. I don't think there is any solution to this other than specify sys.path on a per-application basis. If this is false, what is the other solution? Jim Ahlstrom From jim at digicool.com Tue Sep 14 14:42:10 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 12:42:10 +0000 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> Message-ID: <37DE42A2.E27F7E55@digicool.com> "M.-A. Lemburg" wrote: > > James C. Ahlstrom wrote: > > > > First, I am not all that opposed to having a notion of ".." > > available in the import statement. If we can write zope.dir1.mod1 > > which is a relative import going down, maybe we can write > > ../dir2/mod2 or something spelled differently. But I think > > there would still be problems. > > > > We would be relying on all package authors to use ".." > > or "__" within their package. But it is more natural to write > > zope.this.that or mx.this.that everywhere, and that is what > > people have in fact done. Probably there would be bugs and > > at least annoyance. > > Think of it as an opportunity: it makes intra-package imports > very simple without hard-wiring paths to submodules > into the package. > > In the end, having written a self-contained package is a "feature" > of that package. Other authors can then say: ok, I can > plugin your package if it's self-contained. Well said. It really should be possible to provide *self-contained* package hierarchies. > I have a feeling that this discussion is loosing its grip > on reality a bit. Jim and I are not argueing to change > Python's face, internal structure or look&feel. The '__' thing > doesn't harm anyone, doesn't break any functionality or > code. It is even now already possible using import hooks and > has been implemented before in ni.py using just that technology. > > All that we are talking about is reviving relative imports > so that its usage is possible *without* adding yet another > hook. This is not about the pros and cons of deeply nested > packages and neither about advantages of shallow structures. > Its only about adding semantics to the string '__' used > in import statements. Nothing more. People are not forced > to use it. Besides, if used it will be a package internal > technique and not be visible to users of that package. > > Now why would anyone want to make life harder for package > authors ? I don't think nearly enough Python code has been packagized. Packages are critcal for large projects and for reuse of code within large projects. There are at least two people in this discussion that are trying hard to make use of packages and have observed a significant problem. If we really want to encourage developers of reusable Python software to use packages, then I think it would be wise to pay attention to people who are actively using packages and have encountered problems. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at interet.com Tue Sep 14 15:46:23 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 09:46:23 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE42A2.E27F7E55@digicool.com> Message-ID: <37DE51AF.67BB9CC0@interet.com> Jim Fulton wrote: > > "M.-A. Lemburg" wrote: > > In the end, having written a self-contained package is a "feature" > > of that package. Other authors can then say: ok, I can > > plugin your package if it's self-contained. > > Well said. It really should be possible to provide > *self-contained* package hierarchies. > There are at > least two people in this discussion that are trying hard to > make use of packages and have observed a significant problem. OK, just to make sure I understand this. You are saying that a ".." operation is needed in imports so that a package author with a package which looks like this: mypackage mypackage/sub1/mod1.py mypackage/sub2/mod2.py can write "import __.sub2.mod2" instead of "import mypackage.sub2.mod2" within the file mod1.py. Right? Jim Ahlstrom From jim at digicool.com Tue Sep 14 15:24:10 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 13:24:10 +0000 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE42A2.E27F7E55@digicool.com> <37DE51AF.67BB9CC0@interet.com> Message-ID: <37DE4C7A.A553A8F2@digicool.com> "James C. Ahlstrom" wrote: > > Jim Fulton wrote: > > > > "M.-A. Lemburg" wrote: > > > In the end, having written a self-contained package is a "feature" > > > of that package. Other authors can then say: ok, I can > > > plugin your package if it's self-contained. > > > > Well said. It really should be possible to provide > > *self-contained* package hierarchies. > > > There are at > > least two people in this discussion that are trying hard to > > make use of packages and have observed a significant problem. > > OK, just to make sure I understand this. You are saying that > a ".." operation is needed in imports so that a package author > with a package which looks like this: > mypackage > mypackage/sub1/mod1.py > mypackage/sub2/mod2.py > > can write "import __.sub2.mod2" instead of "import mypackage.sub2.mod2" > within the file mod1.py. Right? Yup. Note that sys.modules would still have the absolute paths for keys. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward at cnri.reston.va.us Tue Sep 14 16:17:01 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 14 Sep 1999 10:17:01 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <000001befe7f$191104c0$fe2d153f@tim>; from Tim Peters on Tue, Sep 14, 1999 at 03:02:24AM -0400 References: <37DD0564.7361B789@lemburg.com> <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <19990914101700.A4401@cnri.reston.va.us> On 14 September 1999, Tim Peters said: > + Dealing with incompatible versions of packages. If someone wants to embed > a copy of (say) mxDateTime in their own package, the only excuses are that > they're afraid of overwriting the user's existing mxDateTime installation > (if any), and/or of having some future installation of something else > overwrite mxDateTime with an incompatible version. Those are bad problems, > but package embedding is no solution. You have a much better approach to > that already via the DateTime.__version__ string! "Something like that" > needs to be formalized and imposed on all public packages. > > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim Been there, tried that, bought the flame war. I made the mistake of kicking off the Distutils SIG back in Decemver with a proposal for a standard version numbering scheme for Python module distributions. See http://www.python.org/pipermail/distutils-sig/1998-December/000016.html for the kick-off of that "heated discussion". ;-) FWIW, if I was posting that message today, I would s/must/should/ and that's about it. And I would stress that these are "Proposed Recommended Guidelines" rather than "Things You Must Do or Distutils Will Slam the Door in Your Face", which is what it must have sounded like based on some of the responses. Folks: please don't reheat this flamewar on python-dev -- if you're interested in it, it will undoubtedly come up again on distutils-sig in due course! Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal at lemburg.com Tue Sep 14 16:24:06 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 16:24:06 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> Message-ID: <37DE5A86.4CD04049@lemburg.com> James C. Ahlstrom wrote: > > "M.-A. Lemburg" wrote: > > > > Still it doesn't avoid possible conflicts > > due to being top-level, e.g. the user could install an extension > > which relies on a specific top-level name already taken by the hosting > > package, e.g. the host defines a DateTime package and the > > extension comes with its own DateTime package. This is the current > > situation with Zope and mxODBC (which needs mxDateTime). > > Yes, the name conflict at the global level is exactly the problem. > And to repeat, I don't really mind if "__" is added to imports, > although I don't like the spelling. But... Doesn't look nice, but what other syntax would look better ? It will have to use identifiers and thus is restricted to [a-zA-Z0-9_]+. > Perhaps a better solution is a multiple global name space. Suppose > there is a command line option (or other mechanism such as a special > file in sys.executable) which specifies sys.path for the application. > To be specific, suppose zope is installed in the "zopehome" > directory and the zope package lives in zopehome/zope. And suppose > zope needs mx, so it installs it in zopehome/mx. This really is an > obvious structure, and anyone could see that replacing mx/* would > upgrade the mx package. The sys.path would be zopehome plus perhaps > some subdirectories of zopehome. Doesn't this solve the problem? > Why not? And don't complain about wasted disk space. The smallest > disk you can buy today is 4 gigs and costs $150. This would solve the problem for Zope (but only after I have restructured my packages to all go under the mx package -- which is what started this thread; it wouldn't be needed using relative imports). But what about other installations or tools like mxCrypto which plug into existing packages (Andrew's crypto package) ? Today such plugins only work side-by-side, ie. you have to install one part which sits in the plugin slot and one part which gets installed somewhere in a top-level PYTHONPATH dir. The plugin slot part then imports the Real Thing from outside the host package (this is how mxCrypto integrates itself with AMK's lib). > I find the PYTHONPATH mechanism totally unreliable for commercial > programs anyway. It is a global object, and an installation of > a second Python program can break the first one. I don't think > there is any solution to this other than specify sys.path on a > per-application basis. If this is false, what is the other > solution? The solution I'm using (and Zope is too) is to wrap a complete Python installation into the product, all setup to do the right thing. It makes the archive a little heavier... which is actually not so bad from a marketing POV. Bizarre, but people seem to want all those megs. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at digicool.com Tue Sep 14 16:13:13 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 14:13:13 +0000 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> <37DE13A9.4150AF0E@lemburg.com> Message-ID: <37DE57F9.9B3BFDC4@digicool.com> "M.-A. Lemburg" wrote: > > Tim Peters wrote: (snip) > > As a matter of personal experience, it gets much stabler! The older DLLs > > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > > installers that finally play by the rules. The mean time between crashes > > when I installed Win95 a few years ago was about an hour; now it's at least > > days and possibly weeks (don't know -- never leave the puter on that long). > > When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. > > My experience is that intra-DLL references simply don't match anymore > and cause the system to become instable. Also, some weird installers > don't care about the version numbers and install older versions at > their will. The outcome is a complete version mess. This has been my experince too. I cringe anytime I see some installer stuff DLLs in my system areas. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From skip at mojam.com Tue Sep 14 17:34:48 1999 From: skip at mojam.com (Skip Montanaro) Date: Tue, 14 Sep 1999 10:34:48 -0500 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda Message-ID: <199909141534.KAA03648@dolphin.mojam.com> I haven't been paying real close attention to the relative package import thread on python-dev, but some way or other I strongly believe something has to be done to improve the situation I now find myself in or more and more people are going to start getting bitten by the same sort of problem. Allow me to 'splain. I use a web server based on Zope's ZServer (which is itself based on medusa) sitting behind an Apache web server as a long-running process to handle what many people traditionally used CGI for. It's a big performance boost over CGI. Unfortunately, I have recently been experiencing frequent server hangs. So far I've been unable to figure out what the cause is. I do notice occasional tracebacks like: Unhandled exception in thread: Traceback (innermost last): File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send _push_mode = 0 File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer self.ac_in_buffer = '' File "/usr/lib/python1.5/asynchat.py", line 253, in pop # this could maybe be made faster with a computed regex? IndexError: list index out of range and also notice that the server can pile up a huge number of connections in the ESTABLISHED state, at which point the whole mess grinds to a halt with not much computation or network traffic happening. A separate shell script uses netstat to detect when a large number of sockets have piled up and restarts the server. Brutal, but crudely effective. (When in doubt, treat the symptoms...) Today it just dawned on me looking at the above traceback that ZServer is getting the wrong version of asynchat.py (and presumably of asyncore.py as well). It's using the version delivered with Python 1.5.2 distribution instead of the version that was delivered with the version of Zope I'm using (1.something). So I started looking around at the versions and dates of various copies of asynchat.py. Here's what I found: source version number:date owner Python 1.5.2 1.2:1999/06/18 guido Python CVS 1.2:1999/06/18 guido Zope 1.? 1.7:1999/04/09 amos Zope 2.0 1.9:1999/07/19 amos Medusa 990902 2.24:1999/07/07 rushing What's apparently been happening is that people have picked up asyncore and asynchat at various time and stuck them in their own CVS repositories without somehow freezing the Id string of the version they originally got from Sam Rushing. It's not clear what the differences are until you compare the actual files. It turns out that the Zope 2.0 and Medusa versions have no content differences, only wildly different version numbers. The Medusa and Python CVS versions only have one difference: if index > 0: # don't bother reporting the empty string (source of subtle bugs) self.collect_incoming_data (self.ac_in_buffer[:index]) which *may* be what's causing my problems (note the IndexError in my traceback). Somewhere along the way I think we need to apply more restraint to our packaging and numbering of Python modules. I don't know what form that restraint will eventually take, but at this point I'm willing to replace Greg Ward's s/must/should/ with s/should/must/. a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From bwarsaw at cnri.reston.va.us Tue Sep 14 18:19:34 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Tue, 14 Sep 1999 12:19:34 -0400 (EDT) Subject: Path hacking [Long] (was Re: [Python-Dev] Relative Package Imports) References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> Message-ID: <14302.30102.360791.488002@anthem.cnri.reston.va.us> Finally, something I can relate to. Although I have a goal of packagizing everything I write these days, I haven't experienced any of the problems that lead others to suggest relative imports. The most complicated app that I hack on (continuously) is Mailman, which has a main package and several subpackages off the main one. I always use absolute paths in my import statements, so I don't see what the fuss is about. But I'm perfectly willing to admit that I don't have enough experience. However... >>>>> "JCA" == James C Ahlstrom writes: JCA> I find the PYTHONPATH mechanism totally unreliable for JCA> commercial programs anyway. It is a global object, and an JCA> installation of a second Python program can break the first JCA> one. I don't think there is any solution to this other than JCA> specify sys.path on a per-application basis. If this is JCA> false, what is the other solution? I completely agree with JimA here. It's been a pain with the Knowbot stuff, a pain with Mailman, and a pain with other packages that I've installed for shared use within CNRI. The .pth files solve part of the problem nicely. They let me install, say PIL or PCT in a shared location, for access by all the Python users at my site, without the users having to individually hack their dot-files, etc. But this doesn't work so well for apps like Mailman or the Knowbot stuff because we can't expect that the person installing those applications will be able to install a .pth file in the right place. Also, .pth files don't let you tightly control sys.path, e.g. you can only add paths, not delete or reorder them. Plus you have a global naming problem. Mailman's top level package is called "Mailman", so I can be fairly confident that I'm not going to collide, but it means that I have an extra directory level within my install that contains all the core importable modules. I don't think that's a big deal, but it's a convention that other packaged app writers should follow. The problem is getting Mailman's (or the Knowbots') top level directory on sys.path, and in exactly controlling the contents of sys.path. Our first approach with Knowbots was to do direct sys.path.insert()s, which is quite ugly and error prone. Plus if you're adding many paths, or adding and deleting, that's a lot of gibberish at the top of your entry level executables. And now let's say that you have a dozen or two dozen entry level executables that all have to perform the same sys.path magic. That's a lot of cutting-and-pasting (and /highly/ error prone patching when directory structures change). It's a lose. So for Knowbots we wrote a small module called pathhack that all entry level executables imported. pathhack was good because it put all that sys.path munging nonsense in one place so it was manageable from a s/w engineering standpoint. But it sucked because those executables had to /find/ pathhack.py! Bootstrap lossage (we've actually gone back to sys.path.insert). With Mailman, I could solve that problem because I added a configure/make phase. This let me write a module template called paths.py.in which configure flippered into paths.py containing path hackage based on --prefix. The next trick was that "make install" copied that paths.py file into all the subdirectories that had top level entry points into the Mailman system (e.g. the bin directory, the cron directory, the cgi directory). So now, an executable need only do import paths import Mailman.Utils import Mailman.Logging.Utils and absolute paths work like a charm. I can even provide a `pythonlib' directory that contains newer versions of standard modules that have fixes for folks running older Pythons. Thus I do from Mailman.pythonlib import rfc822 and the rest of my code uses my special rfc822 module with no changes. I'm very happy with how this works for Mailman, however we can't use the same approach (or let's say Guido doesn't want to use this approach) for the Knowbots stuff because there /is/ no "make install" step. You just unpack it and go. But it still has to play lots of games searching the file system for various things. What I've been thinking is that Python needs a registry . JPython's already got such a beast, and it integrates with Java's system properties, so that things like the PYTHONPATH equivalent are set in the registry and immediately available. But it's not very flexible, and you still need an install step in order to bootstrap the locating of the registry. I think we can do a little bit better. Python already knows how to find it's sys module. We can add an object into sys, call it sys.registry, which would contain things like sys.path definitions, and all sorts of other application specific keys. This object would be tied to a file (or files) which might be human readable, a marshal/pickle (or both). Bootstrap location of this file(s) is an issue, but see below. This would let you do things like the following at the beginning of every top level executable: import sys sys.application = 'zope' sys.registry.setpath(sys.application+'.pythonpath') I'm sure all kinds of lengthy discussion will now ensue about the exact interface of the registry object, but I'll make just a few observations: - There should be a system wide registry and a user specific registry. This let's an admin install shared applications easily, but also lets individual users have their own overrides. - The system-wide registry can be located in say sys.prefix/lib/python/site-packages. The user registry would reside somewhere in $HOME. This could all be platform specific so that on Windows, maybe the Python registry is integrated with the Windows registry, while in JPython it would be integrated with the standard JPython registry mechanism. - You should be able to specify registry entries on the command line. - There needs to be defined rules for resolving registry keys b/w system, user, and command line specifications. JPython has some experience here (although there have been requests to change JPython's lookup order), and at the very least, JPython and CPython should be as consistent as possible (CPython won't have to merge in Java's system properties). - The sys.registry object should be read/writable. This would let an install script do something like: import sys sys.registry.lock() sys.registry.put('zope.pythonpath', '@prefix@:@prefix@/matools:@prefix@/pythonlib') sys.registry.write() sys.registry.unlock() which would write either the global system registry or the local user registry, depending on permissions (or maybe that's spelled explicitly in the API). - In a sense you're pushing the namespace issue up a level into the registry, but at least this is a domain we can completely control from Python; it abstracts away the file system, and I don't think there's any way to avoid requiring conventions and cooperation for registry key naming. I also don't think it'll be a big problem in practice. When I packagize and re-release my Zarathustra's Ocular Python Experience virtual reality system, I'll try to think of a non-colliding top level package name. - (oh darn, I know I had more points, but Guido just popped in and I lost my train of thought). Well, this has gone on long enough so I might as well let you guys shoot this idea all to hell. Let me close by saying that while I think the Windows registry is a mess, I also think that it might be useful for Python. Does it solve the same problem that the relative imports is trying to solve? I dunno, but that's why I changed the Subject: line above. :) -Barry From gmcm at hypernet.com Tue Sep 14 18:47:25 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Tue, 14 Sep 1999 12:47:25 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE5A86.4CD04049@lemburg.com> Message-ID: <1274795004-25696413@hypernet.com> [MA Lemburg] > The solution I'm using (and Zope is too) is to wrap a > complete Python installation into the product, all setup to > do the right thing. It makes the archive a little heavier... > which is actually not so bad from a marketing POV. Bizarre, but > people seem to want all those megs. This is fine if you're distributing an app - it's a black box, it's standalone, no dependencies outside the app. Cool. I do that too. I use imputil to serve modules out of archives - no need to tweak the modules at all. In the upcoming version (of my installer) sys.path has exactly one entry, and the only python that needs to be outside an archive is exceptions.pyc. (I repeat here: while the pieces of my installer that handle binary dependencies are Win32 only, the parts that handle pure Python are cross-platform.) But both you and Jim are aiming for this non-existant in- between space - in some respects you want a "normal" Python installation, but you want to black-box your turf. I'm sorry, but if you're going to fit into a Python installation, you should make public your dependencies. Yes, installation becomes more complicated, but you're dealing with *Python users*. They're at least as likely to get screwed by having multiple different copies of the same thing around as you are in having dependencies outside of your personal control. If you also want to distribute as a standalone, freeze/squeeze/ whatever it. Just make sure that what you distribute this way won't interfere (or be confounded by) any existing Python installations. - Gordon From gstein at lyra.org Tue Sep 14 19:23:36 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 10:23:36 -0700 (PDT) Subject: [Python-Dev] revised import code (was: Relative Package Imports) In-Reply-To: <37DCCF3B.2F1E962D@lemburg.com> Message-ID: On Mon, 13 Sep 1999, M.-A. Lemburg wrote: > Gordon McMillan wrote: > > But I strongly believe that import.c should be left alone, maybe > > even to die. There are too many people doing import hooks to > > make fiddling with its behavior safe. > > > > I'm also a strong proponent of Greg's imputil.py scheme, which > > makes it a breeze to do import hooks. And my experience > > disproves the notion that the import mechanism needs to be in C. If > > you don't believe me, try the ZlibArchive stuff (which is cross > > platform) from my Win32 installer stuff. You can pack the standard > > library into one 475K file, and get a perceptible performance boost. > > You're probably right in saying that we don't need the code in C. > I just wanted to avoid yet another import hook being incompatible > with all the other existing hooks. > > Perhaps we should restart the import discussion all over and > come up with a more flexbile 100% compatible framework based > on Greg's imputil scheme. Then I could add my hook for the relative > imports and be happy ;-) I'll code up some patches to strip down import.c and replace with imputil.py and some bootstrap/support C code -- as soon as 1.6 development opens up. Guido? Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Tue Sep 14 19:36:03 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:36:03 -0400 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> <37DE13A9.4150AF0E@lemburg.com> <37DE57F9.9B3BFDC4@digicool.com> Message-ID: <37DE8783.2B8D6455@interet.com> Jim Fulton wrote: > > "M.-A. Lemburg" wrote: > > > > Tim Peters wrote: > (snip) > > > As a matter of personal experience, it gets much stabler! The older DLLs > > > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > > > > My experience is that intra-DLL references simply don't match anymore > > and cause the system to become instable. Also, some weird installers > > don't care about the version numbers and install older versions at > > their will. The outcome is a complete version mess. > > This has been my experince too. I cringe anytime I see some > installer stuff DLLs in my system areas. My experience has been the same as Tim's. You should generally upgrade DLL's as a matter of course, because your system will be more stable. Installers are required to check DLL versions before replacing them. Not doing this is a major sin, and reason enough to stop dealing with a vendor. Jim Ahlstrom From jim at interet.com Tue Sep 14 19:49:41 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:49:41 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <37DE5A86.4CD04049@lemburg.com> Message-ID: <37DE8AB5.DDB8C432@interet.com> "M.-A. Lemburg" wrote: > Doesn't look nice, but what other syntax would look better ? It will > have to use identifiers and thus is restricted to [a-zA-Z0-9_]+. Maybe "__up__" or "__up" following the convention that double-under names are special? Just a thought... > But what about other installations or tools like mxCrypto > which plug into existing packages (Andrew's crypto package) ? Today > such plugins only work side-by-side, ie. you have to install one > part which sits in the plugin slot and one part which gets installed > somewhere in a top-level PYTHONPATH dir. The plugin slot part then > imports the Real Thing from outside the host package (this is how > mxCrypto integrates itself with AMK's lib). I am not familiar with this. Dosen't this work?: zopehome/part_in_top_level_slot.py zopemome/plugin_slot/part_in_plugin_slot.py > The solution I'm using (and Zope is too) is to wrap a > complete Python installation into the product, all setup to > do the right thing. It makes the archive a little heavier... > which is actually not so bad from a marketing POV. Bizarre, > but people seem to want all those megs. For the record, I distribute a hacked main program which sets its own sys.path. I think your and zope's solution may break if another Python program is installed, and such program changes PYTHONPATH and/or the registry. That is why I never change PYTHONPATH nor the registry. Jim Ahlstrom From jim at interet.com Tue Sep 14 19:59:03 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:59:03 -0400 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <37DE8CE7.F211F6FA@interet.com> Skip Montanaro wrote: > So I started looking around at the versions and dates of various copies of > asynchat.py. >... > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have Yes, I have had this happen too. I am unwilling and unable to risk this sort of problem at a customer's site. So I ship a complete app with no external dependencies. Also crude but effective. Jim Ahlstrom From gstein at lyra.org Tue Sep 14 20:55:47 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 11:55:47 -0700 (PDT) Subject: [Python-Dev] relative package imports, version , numbering, yadda yadda In-Reply-To: <37DE8CE7.F211F6FA@interet.com> Message-ID: On Tue, 14 Sep 1999, James C. Ahlstrom wrote: > Skip Montanaro wrote: > > So I started looking around at the versions and dates of various copies of > > asynchat.py. > >... > > What's apparently been happening is that people have picked up asyncore and > > asynchat at various time and stuck them in their own CVS repositories > > without somehow freezing the Id string of the version they originally got > > from Sam Rushing. It's not clear what the differences are until you compare > > the actual files. It turns out that the Zope 2.0 and Medusa versions have > > Yes, I have had this happen too. I am unwilling and unable to risk > this sort of problem at a customer's site. So I ship a complete > app with no external dependencies. Also crude but effective. We did the same with the Python-based apps/servers at Microsoft. Merchant Server was a big frozen app (based on the non-sensical requirement to hide the fact that Python was used). In Site Server 2.0 and 3.0, we used a mini-install -- just the Lib files we needed plus our extensions. In the Site Server (non-frozen) case, we did use the registry, but built the interpreter with a custom "version". SS20 and SS30. In the registry, this meant we used Python/PythonCore/SS20 (I think that's the layout). Worked great, no complaints. Oh, and the DLLs we put into the system directory (pretty necessary for COM) were named SS20.DLL to prevent conflicts. I'm not sure who said it, but I agree with the following statement: * ship your app as a complete black box, or ship your app with dependencies on modules/packages [at the top level] This monkeying around with "mx.foo" working where mx is at the top level or is embedded is just scary. That said, personally, I would just do something like the following at the startup of my app: ZopeImporter("zopedir").install() [where ZopeImporter is a imputil.Importer subclass] The importer would just Do The Right Thing for all imports, and only defer to the Python library for things that weren't shipped with Zope (the empty set?) Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Tue Sep 14 21:11:54 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 15:11:54 -0400 Subject: Path hacking [Long] (was Re: [Python-Dev] Relative Package Imports) References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <14302.30102.360791.488002@anthem.cnri.reston.va.us> Message-ID: <37DE9DFA.269967B5@interet.com> "Barry A. Warsaw" wrote: > > So for Knowbots we wrote a small module called pathhack that all entry > level executables imported. pathhack was good because it put all that > sys.path munging nonsense in one place so it was manageable from a s/w > engineering standpoint. But it sucked because those executables had > to /find/ pathhack.py! Bootstrap lossage (we've actually gone back to > sys.path.insert). Yes, exactly the problem I had, bootstraping the import of pathhack. Actually it gets worse because Python imports exceptions.py, site.py and sitecustomize.py during Py_Initialize(), so if you are having a really bad day, you might pick up the wrong version of those. AFAIK, the only way to solve that currently is to use freeze to build pathhack into the binary executable. That is what I do anyway. But it is not an ideal solution. > What I've been thinking is that Python needs a registry . Yikes! As you say, Window's registry is a mess. > [Lots of good ideas omitted...] > - The system-wide registry can be located in say > sys.prefix/lib/python/site-packages. The user registry > would reside somewhere in $HOME. This could all be platform > specific so that on Windows, maybe the Python registry is integrated > with the Windows registry, while in JPython it would be integrated > with the standard JPython registry mechanism. Python already has three directories it knows about: sys.executable is the directory of the interpreter binary, sys.dllfullpath could be the directory of the interpreter as a shared library (I have a patch for this), and there is the directory of the main Python program as given on the command line. Perhaps we can put the registry in one of these directories. That would be consistent on all platforms. > - You should be able to specify registry entries on the command line. This is vital because I am worried about a bad registry. > - There needs to be defined rules for resolving registry keys b/w > system, user, and command line specifications. JPython has some > experience here (although there have been requests to change I am not sure a full registry is required. Once you can control sys.path and can get an accurate import of sitecustomize.py, you can do everything else there. Maybe just a command line option is enough. But I will think about it... Jim Ahlstrom From jim at interet.com Tue Sep 14 21:37:11 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 15:37:11 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274795004-25696413@hypernet.com> Message-ID: <37DEA3E7.389CE@interet.com> Gordon McMillan wrote: > > But both you and Jim are aiming for this non-existant in- > between space - in some respects you want a "normal" > Python installation, but you want to black-box your turf. I don't really have much choice here. For some purposes, PYTHONPATH is better than a black-box. It is better for development because it avoids creating the black-box. And I can look at and modify Python app and library files directly. But for a customer install I need a bullet-proof dumb-simple set of required state which I can explain to customer support staff. Thus the need for both. > I'm sorry, but if you're going to fit into a Python installation, > you should make public your dependencies. Yes, installation > becomes more complicated, but you're dealing with *Python > users*. They're at least as likely to get screwed by having > multiple different copies of the same thing around as you are > in having dependencies outside of your personal control. I can't fit into a Python installation because Python installations do not support commercial software concerns. I am not dealing with Python users, my customers are office workers. They don't know what Python is, have no other Python apps, and if they did, my install wouldn't hurt them anyway. > If you also want to distribute as a standalone, freeze/squeeze/ > whatever it. Just make sure that what you distribute this > way won't interfere (or be confounded by) any existing Python > installations. Yes, exactly my goal. Jim Ahlstrom From guido at CNRI.Reston.VA.US Tue Sep 14 21:57:51 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 15:57:51 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Tue, 14 Sep 1999 12:19:34 EDT." <14302.30102.360791.488002@anthem.cnri.reston.va.us> References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <14302.30102.360791.488002@anthem.cnri.reston.va.us> Message-ID: <199909141957.PAA15470@eric.cnri.reston.va.us> I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at CNRI.Reston.VA.US Tue Sep 14 22:09:07 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 16:09:07 -0400 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda In-Reply-To: Your message of "Tue, 14 Sep 1999 10:34:48 CDT." <199909141534.KAA03648@dolphin.mojam.com> References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <199909142009.QAA15534@eric.cnri.reston.va.us> > So I started looking around at the versions and dates of various copies of > asynchat.py. Here's what I found: > > source version number:date owner > Python 1.5.2 1.2:1999/06/18 guido > Python CVS 1.2:1999/06/18 guido > Zope 1.? 1.7:1999/04/09 amos > Zope 2.0 1.9:1999/07/19 amos > Medusa 990902 2.24:1999/07/07 rushing > > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have > no content differences, only wildly different version numbers. The Medusa > and Python CVS versions only have one difference: > > if index > 0: > # don't bother reporting the empty string (source of subtle bugs) > self.collect_incoming_data (self.ac_in_buffer[:index]) > > which *may* be what's causing my problems (note the IndexError in my > traceback). My bad. I despise putting RCS identifiers in code I release, but I don't always freeze them before incorporating other people's code in my CVS tree. I will fix this if I can and I promise to try not to repeat this mistake in the future. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at lemburg.com Tue Sep 14 22:57:19 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 22:57:19 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274795004-25696413@hypernet.com> Message-ID: <37DEB6AF.53974AAB@lemburg.com> Gordon McMillan wrote: > > [MA Lemburg] > > The solution I'm using (and Zope is too) is to wrap a > > complete Python installation into the product, all setup to > > do the right thing. It makes the archive a little heavier... > > which is actually not so bad from a marketing POV. Bizarre, but > > people seem to want all those megs. > > This is fine if you're distributing an app - it's a black box, it's > standalone, no dependencies outside the app. Cool. I do that > too. I use imputil to serve modules out of archives - no need to > tweak the modules at all. In the upcoming version (of my > installer) sys.path has exactly one entry, and the only python > that needs to be outside an archive is exceptions.pyc. > > (I repeat here: while the pieces of my installer that handle > binary dependencies are Win32 only, the parts that handle > pure Python are cross-platform.) > > But both you and Jim are aiming for this non-existant in- > between space - in some respects you want a "normal" > Python installation, but you want to black-box your turf. Actually, I want my stuff to be as flexible as possible. Of course I'm using it in my product too, but that's a different story. I have the unsatisfying impression that this discussion doesn't lead anywhere. I can't really understand all the evil sides Tim and Guido seem to see in their views of relative imports. For people like Jim and me, who have quite some experience in doing Python packages, this small (! hey, it's only about 20 lines of code !) additional feature could be of great use. I'm still waiting for some ultimate argument that blows relative imports away. The arguments put forward so far have all been in the category "evil", "ugly", "I don't like it". Hey, we can do better than that... Note that if this doesn't work out, Jim and I could always agree on some standard import hook that we'd both use, but that's exactly what we would very much like to avoid in favour of some established standard hard-wired into the Python distribution. > I'm sorry, but if you're going to fit into a Python installation, > you should make public your dependencies. Yes, installation > becomes more complicated, but you're dealing with *Python > users*. They're at least as likely to get screwed by having > multiple different copies of the same thing around as you are > in having dependencies outside of your personal control. I am probably going to ship all the mx* stuff in one big package -- not as separate packages anymore. That way I can write code which depends on other parts of the mx universe more easily. > If you also want to distribute as a standalone, freeze/squeeze/ > whatever it. Just make sure that what you distribute this > way won't interfere (or be confounded by) any existing Python > installations. It won't interfere with anything since the Python interpreter I use is configured *not* too look in any standard places for extensions and the like. Plus it's configured to run faster, but that's a different story ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Tue Sep 14 23:09:53 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 17:09:53 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 11:21:15 EDT." <37DD166B.746B19AD@digicool.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> Message-ID: <199909142109.RAA15752@eric.cnri.reston.va.us> Jim Fulton wrote: > I wouldn't be in favor of making it more complicated if there wasn't > a good reason. I think that, in working on the Zope framework, > I've found some pretty good reasons for relative imports. And in a later message: > I don't think the Package structure of Zope is flawed *except* > for the fact that it is one level too *shallow*. The ability to > do relative imports would be very helpful for the work we're doing. But I haven't seen explained what it is that Zope is doing where relative packages would be helpful. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at CNRI.Reston.VA.US Wed Sep 15 00:47:18 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 18:47:18 -0400 Subject: [Python-Dev] License cleanup Message-ID: <199909142247.SAA16609@eric.cnri.reston.va.us> After more than four years of living with an out-of-dated license for Python, CNRI has finally agreed to clean up Python's copyright status. I expect that this won't have any real effect before Python 1.6 is released, but I am required to start preparing for the transition now. We will use a new license (a clone of the JPython license) and we will require that all contributors explicitly allow us the use of their contribution: either a few email paragraphs in an email message, or a longer form with a wet signature, depending on the size of the contribution. I believe the text of the license and forms we use is quite uncontroversial; these very same words have been used for JPython for quite a while. The words are all on the web: http://www.python.org/1.5/pylicense.html [proposed license] http://www.python.org/1.5/bugrelease.html [email release] http://www.python.org/1.5/wetsign.html [wet signature release] If you are reading python-dev but you never contributed any code to Python, you can stop reading now. If you *did* contribute code to Python, however, I'd love it if you saved me some work and filled out the wet signature form and mailed it to me at the given address. If you need help jogging your memory what your contributions were, send me email; I can try grepping the CVS files for your name. If you believe that special circumstances exist that make it impossible or difficult for you to sign the form, please send me email, and we'll discuss the matter. If you contributed something and I don't hear from you, you will eventually hear from me again -- but I hope I can save myself the hassle of writing each of you through this mass mailing. Thanks in advance! --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Wed Sep 15 01:08:32 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 16:08:32 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: Actually, I don't like them all that much :-( [I don't recall any specific discussion about it, but I may have missed it and/or simply because I've never used JPython.] The BSD-ish license that Python has always used is much more preferable. I dislike the regulation of the "Python" name, the requirement to prominently discuss modifications made, and the revocation clause. I might find other items, but that is from a quick read using Lynx on a tiny monitor... Heck, how could people like PPSI, PythonWare, or D.C. truely like that license? Each of those companies uses "Python" significantly in their marketing and their business. I can certainly state that PPSI will never do anything in an official capacity to recognize that license. [there is a separate issue of whether "Python" can be trademarked, but the license does use the term "trade name" which could easily be argued to include the term "Python" and thus subject the name to the license.] > If you *did* contribute code to Python, however, I'd love it if you > saved me some work and filled out the wet signature form and mailed it > to me at the given address. No problem. Future contributions and agreemend to abide by that license are a different issue. It doesn't have the "feels good" feeling that the old license does. I'm not sure that bodes well, and it doesn't sit well with me at the moment. Regards, -g -- Greg Stein, http://www.lyra.org/ From guido at CNRI.Reston.VA.US Wed Sep 15 01:31:47 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 19:31:47 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Tue, 14 Sep 1999 16:08:32 PDT." References: Message-ID: <199909142331.TAA16699@eric.cnri.reston.va.us> > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. I might > find other items, but that is from a quick read using Lynx on a tiny > monitor... Hm... We may have to review the regulation of the Python name. This made sense in the context of the previous uses of this license (JPython and Grail) but Python is a different thing -- the name Python stands for more than just the implementation. I'll discuss this with CNRI's legal team. I don't see how the other things you mention can be much of a problem (most Open Source licenses have a revocation clause these days, I think, and I don't see how discussing the modifications made can be a problem with open source users). > Heck, how could people like PPSI, PythonWare, or D.C. truely like that > license? Each of those companies uses "Python" significantly in their > marketing and their business. I can certainly state that PPSI will never > do anything in an official capacity to recognize that license. How can you say that without consulting with the board? And I am *on* that board! I despise your attitude. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw at cnri.reston.va.us Wed Sep 15 01:44:54 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Tue, 14 Sep 1999 19:44:54 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <14302.56822.478320.862970@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Heck, how could people like PPSI, PythonWare, or D.C. truely GS> like that license? Each of those companies uses "Python" GS> significantly in their marketing and their business. Data point: I know that there are a number of companies that have embedded JPython in their commercial products. So far I've had zero complaints from them on the JPython license. -Barry From gstein at lyra.org Wed Sep 15 02:11:56 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:11:56 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142331.TAA16699@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > Hm... We may have to review the regulation of the Python name. This > made sense in the context of the previous uses of this license > (JPython and Grail) but Python is a different thing -- the name Python > stands for more than just the implementation. I'll discuss this with > CNRI's legal team. Cool. > I don't see how the other things you mention can be much of a problem > (most Open Source licenses have a revocation clause these days, I > think, and I don't see how discussing the modifications made can be a > problem with open source users). I'll do some more reading. As I said: that was my first cut. The revocation clause doesn't sit well with me. Maybe other OSS packages have it, but I believe that is usually because the license was developed by a company and its legal team. I don't think the GPL, BSD, MPL, and Apache licenses have revocation clauses, and I consider those to be the "most open" types of licenses (MPL less so). The Python 1.5 license is just as open, more so than most. > > Heck, how could people like PPSI, PythonWare, or D.C. truely like that > > license? Each of those companies uses "Python" significantly in their > > marketing and their business. I can certainly state that PPSI will never > > do anything in an official capacity to recognize that license. > > How can you say that without consulting with the board? And I am *on* > that board! I despise your attitude. Because the President (me) runs the day-to-day operation and direction of the company. The Board advises. The Board typically has other duties such as replacing me :-), handling stock issues, etc, but the Board is typically not involved with most issues. This is standard practice for corporate organization. Therefore, I *can* make that choice, and even do it unilaterally if I wanted to be an ass about it. Will I refuse to listen to the board or the shareholders or the employees? Of course I'll listen. [further PPSI issues should be taken offline] Regardless: it boils down to the "Python" requirement in that license. PPSI simply cannot operate under that license. If it gets dropped, then cool. -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Wed Sep 15 02:16:10 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:16:10 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Barry A. Warsaw wrote: > >>>>> "GS" == Greg Stein writes: > GS> Heck, how could people like PPSI, PythonWare, or D.C. truely > GS> like that license? Each of those companies uses "Python" > GS> significantly in their marketing and their business. > > Data point: I know that there are a number of companies that have > embedded JPython in their commercial products. So far I've had zero > complaints from them on the JPython license. Are they using it in their marketing, or simply as an underlying driving force for their products? If they *are* using it in their marketing, then they have exposed themselves to a liability. According to the license that they are using, they are not allowed to use JPython in their marketing. If they do, then they are in breach of the license and it could be terminated on them. Their products could no longer include JPython and they'd be SOL. I would be interested to hear from somebody using JPython and marketing it and how they interpreted that license. Possibly I'm missing something, but that language seems pretty darn clear to me. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Wed Sep 15 02:24:25 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:24:25 -0700 (PDT) Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DEB6AF.53974AAB@lemburg.com> Message-ID: On Tue, 14 Sep 1999, M.-A. Lemburg wrote: >... > I have the unsatisfying impression that this discussion doesn't > lead anywhere. I can't really understand all the evil sides Tim and > Guido seem to see in their views of relative imports. For people > like Jim and me, who have quite some experience in doing > Python packages, this small (! hey, it's only about 20 lines of > code !) additional feature could be of great use. $| was probably a small addition to Perl, but I don't think you could then argue that it was a good thing to do. Small doesn't mean good. Any change "could be of great use" to *somebody*, but does it make it good for Python as a whole? Changes imply doc, maintenance, future compatibility, etc. > I'm still waiting for some ultimate argument that blows > relative imports away. The arguments put forward so far have > all been in the category "evil", "ugly", "I don't like it". > Hey, we can do better than that... People seem to be disagreeing with your stated requirement. In other words, they're saying that you shouldn't be attempting to make your package "portable across the module-import-space." I tend to agree. Fix it at one location. If an app doesn't like that, then they can individually compensate. I don't believe that Python's standard machinery is required to handle this particular notion of packaging. > Note that if this doesn't work out, Jim and I could always agree on > some standard import hook that we'd both use, but that's > exactly what we would very much like to avoid in favour of some > established standard hard-wired into the Python distribution. No reason the standard hook could not go into imputil.py. I believe people mostly care that it doesn't become *standard* or *default* behavior -- that it only is available when explicitly requested by an app. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond at skippinet.com.au Wed Sep 15 02:49:35 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 15 Sep 1999 10:49:35 +1000 Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <002e01beff14$306e5b00$0801a8c0@bobcat> > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: Obviously IANAL. However, this language does make me feel less comfortable than the existing one. The ability to terminate would appear an issue - it would seem to take a braver CEO to base their technology on Python with this hanging over them. Sure, it may rarely be invoked, but I certainly wouldnt want to fight it in court if it was. If I was writing in C, I could worst-case grudgingly accept needing to change compilers - but I dont have that luxury for Python. If my license was terminated, I have nowhere else to turn. It is a real shame when lawyers get so involved. Obviously Guido has no say in this, but IMO the ideal scenario would be to use the exsting language, but simply change the names and dates. Im guessing this would be unacceptable to CNRI. Being NAL, I suppose I have no choice other than to trust this licence. However, Im not looking forward to showing this licence to people as they are deciding if Python is the appropriate technology choice - to date, there has never been an issue - all they need to is not remove any copyright notice from the code (which is not actually seen in most apps) and add the copyright notice to the documentation. This new one seems much scarier to me.. Just my $200.00 worth (remember, we are talking lawyers fees here :-) I dont have a real concern as I dont understand the legal implications; just a slight uneasiness about it all...Not being controversial for the sake of it, just airing my possibly il-informed opinion - no opinions were solicitied, but that has never stopped me before :-) Of course, I will be sending my "wet" signature on the form. Im not sure what to put in the "contribution description" - maybe just "various small changes to the Windows port"?? I can't say Ive added entire modules, but my name appears against a number of small patches to a fairly large set of files... Mark. From tim_one at email.msn.com Wed Sep 15 07:19:48 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 01:19:48 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <19990914101700.A4401@cnri.reston.va.us> Message-ID: <000301beff39$ee3015a0$612d153f@tim> [Tim, speaks of the devil ...] > "Something like that" [MAL's __version__ string] needs to be formalized > and imposed on all public packages. > > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim [... and Greg Ward of his legions appears!] > Been there, tried that, bought the flame war. I made the mistake of > kicking off the Distutils SIG back in Decemver with a proposal for a > standard version numbering scheme for Python module distributions. See > > http://www.python.org/pipermail/distutils-sig/1998-December/000016.html > > for the kick-off of that "heated discussion". ;-) Greg, if you call that a flame war, your credentials as an ex-Perl'er are in serious doubt . Except for the cowboy contingent, most participants were moving swiftly to consensus! > FWIW, if I was posting that message today, I would s/must/should/ and > that's about it. No, it's "must" or it's useless. What wasn't brought up in that thread is that the Distutil "version number" is an artficial construct created for the primary benefit of Distutil tools -- it needn't have anything whatsoever to do with whatever silly string the developer wants to *display* as being their "version number". It's instead a coordinate in an abstract but rigidly defined Distutil space, specifically designed to make programmatic navigation of that space reliable in a shared and uniform way. If a developer chooses, users need never be exposed to it. I'd use the x.y.z Distutil version number directly to keep my own life simpler, but if someone else wants to display a GUID followed by a 3-letter country code and the number of nanoseconds since the birth of Mohammed, fine -- they still have to map that to Distutil VN space internally or write their own stinkin' disttools. You may have went overboard on the *semantics* of the Distutil VN, though: its only real meaning is in what Distutil tools *do* with it. Fight this battle again. Without a uniform way for an installer to *know* when it's thought safe to replace a package with another version of that package, Python installations will never move beyond the similar hell of Windows 3.1. even-herds-of-cats-wear-collars-ly y'rs - tim From tim_one at email.msn.com Wed Sep 15 07:40:35 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 01:40:35 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <000501beff3c$d58a8be0$612d153f@tim> I dislike the new license. Selling Python at work wasn't easy, but the short & straightforward CWI license went a *long* way toward convincing the suits there was little to worry about. The new license has several blobs of lawyer-speak that ensure the next battle will be much harder -- the prospect of license revocation, some fuzzy concept of derivative works, and vague "prominent display" requirements? Boston lawyers charge Really Big Bux to guess what that gibberish might mean in Virginia. The only bright side is that we now get explicit rights to "perform" and "display" Python . > If you believe that special circumstances exist that make it > impossible or difficult for you to sign the form, please send me > email, and we'll discuss the matter. It would less hassle for me if you took all my contributions out <0.9 wink>. i'll-sign-but-it's-really-really-really-depressing-ly y'rs - tim From tim_one at email.msn.com Wed Sep 15 08:42:02 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 02:42:02 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE0825.10D2BC6F@lemburg.com> Message-ID: <000701beff45$6b51c5a0$612d153f@tim> Don't tell, but this is getting repetitive. One more batch of repeats from me, and I'm done: + The notion that Python packages aren't self-contained now is wrong: a package module can import other modules in the package without qualification. This is good design. + Where self-containment breaks down is in going across *sub*-packages. I was surprised Guido pretended to cater to them, since Python (wisely, IMO) stayed away from sub-modules from the start. Hierarchy is a great tool for managing complexity, but it also introduces complexity of its own. And when a flat package space suffices, hierarchy introduces artificial complications. + I agree with Gordon that, if anything, the default "import" behavior is already too complicated. By the time you wade thru PYTHONPATH, and PYTHONSTARTUP, and .pth files, and people mucking with sys.path, and site config files, and symlinks under Unix, and packages mucking with their own __path__ attributes, running Python with -v is the only hope of figuring out why nothing ever works <0.6 wink>. + All this dynamicism is in support of a concept that's actually static: the structure of an installation. As Barry laments, you can't always know the structure in advance of installation, but the solution is (I believe) along the lines he suggests: invent a way to register/query the structure once & for all after it is known. + JimF and MAL's shared notion that they're the only ones making significant use of packages is myopic. Dragon also uses packages, and heavily, although products using them have not been announced and I can't talk about them yet. Anything I'm within 10 miles of doesn't use *any* import hooks, or play any other tricks on the system. Instead I urge people to write their imports as explicitly, long-windedly and straightforwardly as possible (with binding to a convenient local alias name on the following line). Do we embed sub-packages? Sure. Do modules move around in the hierarchy over time? Of course. So how do we deal with this supposed nightmare? Because all imports are relentlessly straightforward (even intra-package imports are fully qualified), and none are hiding in dynamic functions, we get into Emacs and do a global search-and-replace. Takes 30 seconds, and the new structure is plain as day to everyone reading the code. low-tech-it's-not-just-for-breakfast-anymore-ly y'rs - tim From ping at lfw.org Wed Sep 15 09:31:29 1999 From: ping at lfw.org (ping at lfw.org) Date: Wed, 15 Sep 1999 00:31:29 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: > > http://www.python.org/1.5/pylicense.html [proposed license] > http://www.python.org/1.5/bugrelease.html [email release] > http://www.python.org/1.5/wetsign.html [wet signature release] Hi all. I'm sorry i haven't contributed anything to the relative-import and python-path discussions of late, but that's because so far i haven't had any ideas that have crossed my threshold of being sufficiently insightful to propose. I will follow the discussion with much interest. I'm afraid i have to say that the revocation clause makes me pretty uncomfortable. I know that it says CNRI will revoke only on a "material breach", but i still have a nasty suspicion that it sounds frightening enough to scare many people away. I don't think we want that. I suppose Greg's other points of contention are valid too but it's really the revocation that bugs me the most. -- ?!ng From fredrik at pythonware.com Wed Sep 15 09:42:28 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Sep 1999 09:42:28 +0200 Subject: [Python-Dev] License cleanup References: Message-ID: <008701beff4d$dcebfb10$f29b12c2@secret.pythonware.com> > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. same here. reading the new one made me feel very uneasy, but I cannot really say much about it before I've discussed it with people who know more about this... just a few small notes: the BSD-ish license used up to now has been a major selling argument for Python, while this one seems to really push the bounds of what qualifies as an open source license... (it also seems to imply that Python is a trademark, which is, as far as I can tell, is not true at this time. and archive corporation/seagate already owns the trademark wrt. software). the worst thing is that we will have to run this by our lawyers before we can decide whether to continue contributing to 1.6 development :-( From mal at lemburg.com Wed Sep 15 11:24:43 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 11:24:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <000701beff45$6b51c5a0$612d153f@tim> Message-ID: <37DF65DB.46F0191A@lemburg.com> Tim Peters wrote: > > + The notion that Python packages aren't self-contained now is wrong: a > package module can import other modules in the package without > qualification. This is good design. Agreed. And we would like to take that notion one step further -- without breaking the two-scope resolution Python uses now. The '__' tags are just place holders which could just as well be handled by a simple Python fuction (like the one Gordon posted) using perfectly valid technology. Sticking this technology into the import statement is really only a more elegant approach. Nothing more. The whole thing already works using the current Python implementation. > + JimF and MAL's shared notion that they're the only ones making significant > use of packages is myopic. Oh, c'mon Tim, we never did say that. We only mentioned having used package for quite a while. This includes having seen all the different pitfalls they have to offer. > Dragon also uses packages, and heavily, although > products using them have not been announced and I can't talk about them yet. > Anything I'm within 10 miles of doesn't use *any* import hooks, or play any > other tricks on the system. Instead I urge people to write their imports as > explicitly, long-windedly and straightforwardly as possible (with binding to > a convenient local alias name on the following line). Do we embed > sub-packages? Sure. Do modules move around in the hierarchy over time? Of > course. So how do we deal with this supposed nightmare? Because all > imports are relentlessly straightforward (even intra-package imports are > fully qualified), and none are hiding in dynamic functions, we get into > Emacs and do a global search-and-replace. Takes 30 seconds, and the new > structure is plain as day to everyone reading the code. This is perfectly ok if you're just using your own code, but it gets a mess when third-party packages are involved simply because you can't be sure they don't use import hooks, local imports or hacks to __path__ etc. If you still want them to be useable, you'll have to go down and dive into their structure. This takes time, is error prone and not necessarily fun. Also, I don't know how "explicitly, long-windedly and straightforwardly" writing module imports is any different from using relative imports. An example: mx .DateTime .ODBC.Windows In ODBC.Windows I would write: import __.__.DateTime which is just as explicit as writing import mx.DateTime except that the information about the top-level hierarchy is *not* included in the import information. Anyway, I'm not too inclined continuing this discussion anymore. I would never have thought that such a nifty little patch would have stirred up so much negative response. I'll just hack up my very own import mechanism using imputil and be done. So there! -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Wed Sep 15 12:05:27 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 12:05:27 +0200 Subject: [Python-Dev] License cleanup References: <008701beff4d$dcebfb10$f29b12c2@secret.pythonware.com> Message-ID: <37DF6F67.59FCEA6D@lemburg.com> Fredrik Lundh wrote: > > > The BSD-ish license that Python has always used is much more preferable. I > > dislike the regulation of the "Python" name, the requirement to > > prominently discuss modifications made, and the revocation clause. > > same here. reading the new one made me feel very > uneasy, but I cannot really say much about it before > I've discussed it with people who know more about > this... Dito. Some comments: """ 4.Licensee may not use CNRI trademarks or trade name, including Python or CNRI, in a trademark sense to endorse or promote products or services of Licensee, or any third party. Licensee may use the mark Python in connection with Licensee's derivative versions that are based on or incorporate the Software, but only in the form "Python-based ___________________," or equivalent. """ Say I want to sell Python 1.6 training, how would I promote this ? Since I'm not producing a derivative work, I guess I couldn't use the name 'Python' at all... hmm, I could probably try Pyth*n ;-) """ 3.In the event Licensee prepares a derivative work that is based on or incorporates the Software or any part thereof, and wants to make the derivative work available to the public as provided herein, then Licensee hereby agrees to indicate in any such work, in a prominently visible way, the nature of the modifications made to CNRI's Software. """ How explicit would that indication have to be ? E.g. do I have to provide a patch or would a simple run-down of new features suffice ? Needless to say, I would not be able to sell products based on Python 1.6 with the revocation clause in the license. In the end, I'd probably have to negotiate a separate license with CNRI not having this clause. Anything else would be unacceptable in a commercial setting. Is this intended ? And finally in the "Python Contribution Agreement": """ Licensee confirms to CNRI that, to the best of Licensee's knowledge and belief, the Contribution is free of any claims of parties other than Licensee under copyright, patent or other rights or interests ("claims"). """ Best knowledge and belief do not guard against law suit. Why doesn't this text protect the contributor in some way against charges forwarded by CNRI to the contributor ? (Note that the disclaimer in the Python License is not valid everywhere.) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From bwarsaw at cnri.reston.va.us Wed Sep 15 13:56:48 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 07:56:48 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: <14303.35200.979238.868146@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Are they using it in their marketing, or simply as an GS> underlying driving force for their products? I'm not sure that JPython is much of a marketing advantage right now, so AFAIK none of them are actively promoting their use of JPython in their product. However, my reading of the second half of item 4 would allow them to say something like "You can even extend your flapjabs using our keen JPython-based scripting capabilities". GS> If they *are* using it in their marketing, then they have GS> exposed themselves to a liability. According to the license GS> that they are using, they are not allowed to use JPython in GS> their marketing. If they do, then they are in breach of the GS> license and it could be terminated on them. Their products GS> could no longer include JPython and they'd be SOL. I hope that wouldn't really be the case, but it's an interesting point, so I'm sure we'll bring it up. -Barry From jim at interet.com Wed Sep 15 14:20:54 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 08:20:54 -0400 Subject: [Python-Dev] License cleanup References: Message-ID: <37DF8F26.37BC155E@interet.com> Greg Stein wrote: > > On Tue, 14 Sep 1999, Guido van Rossum wrote: > > I believe the text of the license and forms we use is quite > > uncontroversial; these very same words have been used for JPython for > > quite a while. The words are all on the web: Oh boy, this is really going to cause trouble. Where's my flame suit... > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. I might > find other items, but that is from a quick read using Lynx on a tiny > monitor... I guess I am used to reading license agreements, and I am not very worried about the new one. Before we all get upset, lets remember that Guido works in a large company with lots of lawyers, and he trapped between a group of Internet geeks (hey, I like Internet geeks) and his buracracy. And remember that lawyers respond better to specific proposals for language changes than philosophical discussion. First off, the license is not revokable. It is only revokable on breach. If a license can not be revoked on breach it doesn't really mean anything. This is totally standard. Suppose someone else claims to own Python and starts selling "The True Standard Python" for $100. Suppose they change the standard library names so software only runs on their version. CNRI should be able to revoke their license to use Python. This is something we would all want CNRI to do. The protection of the Python name is a necessity. That is really all CNRI has, since the license gives away use of the software itself. If CNRI doesn't own "Python" then it can't object when someone else claims they own it. Don't we want them to object? The license doesn't say you can't use "Python", it sayes you can't use it in a trademark sense. I think that means you can say "I am teaching a course on Python, which is CNRI's software" but not "I am teaching a course on my Python, all rights reserved". Actually this is a little unclear, perhaps (4) could be made a little clearer. Paragraph (3) is a little troublesome. I seems to mean that if you ship a modified Python, you must say it is modified. I presume it doesn't mean that you must describe your own code in the event it incorporates Python. Really, we need to know what CNRI wants us to do here. On the contributions side (wetsign.html) it says you are contributing software free of third party claims "to the best of your knowledge and belief" not "represents and warrants" which is different. CNRI really has to be told that as far as you know, you didn't steal the software you are contributing. This is reasonable. Actually I might like to see a warranty disclaimer "NO WARRANTIES etc." like the license paragraph (5) and (6). I am not sure I need it since the contribution is free, but I usually ship free software with a disclaimer for "fitness for any particular purpose etc.". This is a pretty weak license agreement. Remember that if it is too weak, it prevents CNRI from defending Python against others who would claim they own it or who claim they are the true source of the language design (paranoia department: Microsoft's Python++). We want CNRI to defend Python, right? Jim Ahlstrom From gward at cnri.reston.va.us Wed Sep 15 14:30:32 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Wed, 15 Sep 1999 08:30:32 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <14302.56822.478320.862970@anthem.cnri.reston.va.us>; from Barry A. Warsaw on Tue, Sep 14, 1999 at 07:44:54PM -0400 References: <199909142247.SAA16609@eric.cnri.reston.va.us> <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: <19990915083032.A17433@cnri.reston.va.us> On 14 September 1999, Barry A. Warsaw said: > Data point: I know that there are a number of companies that have > embedded JPython in their commercial products. So far I've had zero > complaints from them on the JPython license. Just thought I should join the tide of opposition: heck, I *work* for CNRI and I still don't like the license. I didn't say much about the new JPython license because a) I trust Barry's judgement, b) it was certainly an improvement over the old JPython license, and c) I wasn't especially worried about one part of CNRI (Guido's group) taking JPython away from another part (the group that Andrew and I are on). However, that doesn't change the fact that the "new" license is a nasty piece of legalistic gibberish. Making it the license for Python 1.6 would be a major setback -- while it was better than the old JPython license, it's a damn sight worse than the old Python license. I have zero sympathy for the legal beagles here with their narrow corporatist viewpoint; trying to treat Python as just another potential piece of intellectual property is wrong-headed in the extreme. The free software world simply does not work that way. BTW, I suspect that the companies embedding JPython haven't minded the license because they come from the Java world, a world that seems to me to be dominated by corporate pin-headed thinking. The idea of community, openness, and sharing is utterly alien to these suit-wearing, smarmy Java frat-boy types, so JPython's licensing terms were probably a breath of fresh air to them. ("What? No $100,000 source license fee? Wow!") ("But wait Chip -- it's not BUZZWORD COMPLIANT! I can't find enough TLAs!!!") Hmmm, enough flaming Java weenies. Please, don't anybody take the last paragraphy too seriously or personally... Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fredrik at pythonware.com Wed Sep 15 14:39:30 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Sep 1999 14:39:30 +0200 Subject: [Python-Dev] License cleanup References: <37DF8F26.37BC155E@interet.com> Message-ID: <003e01beff77$5c87fe40$f29b12c2@secret.pythonware.com> > The license doesn't say you can't use "Python", it sayes you > can't use it in a trademark sense. quick check: which of these uses "Python" in a trademark sense, and thus violates the license: pythonware? professional python services? pythonworks? programming python? python training? python powered? the viper python implementation? python imaging library? wxpython? pythonwin? etc. all of them? none of them? From mal at lemburg.com Wed Sep 15 15:09:27 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 15:09:27 +0200 Subject: [Python-Dev] License cleanup References: <37DF8F26.37BC155E@interet.com> <003e01beff77$5c87fe40$f29b12c2@secret.pythonware.com> Message-ID: <37DF9A87.55AF6279@lemburg.com> Fredrik Lundh wrote: > > > The license doesn't say you can't use "Python", it sayes you > > can't use it in a trademark sense. > > quick check: which of these uses "Python" in a trademark > sense, and thus violates the license: > > pythonware? professional python services? pythonworks? > programming python? python training? python powered? > the viper python implementation? python imaging library? > wxpython? pythonwin? etc. > > all of them? none of them? Using a word in a trademark sense usually simply means using it in corporate relationships (at least that's how it works in Germany). If you are a company and talk about, write about or otherwise use the word in a commercial context then you are using the word in a trademark sense. There are several ways to declare a trademark, e.g. there are word marks, logo marks, sound marks, color marks etc. (don't know if these are the right translations). A word mark, for example, refers to a specific spelling of the word regardeless of the font, style or color. Logo marks refer to a specific design including font, style and color. Note that a trademark owner can still give you permission to use the mark in any decent way without paying fees or royalties. So even if CNRI does own the mark, they could still make it usable by others. In fact, if done right, this is a Good Thing. The answer to your question depends on what kind mark CNRI owns. [There currently is a very strong movement in Germany against people who are applying what they learned from domain grabbing to trademarks. Prominent examples include "WWW" and "Webspace". Even the color violet is trademarked (by a company producing chocolate)] -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at interet.com Wed Sep 15 17:43:13 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 11:43:13 -0400 Subject: [Python-Dev] Re: Path hacking Message-ID: <37DFBE91.C93816F1@interet.com> "Guido van Rossum" wrote: > --> The solution: Ah, finally a specific proposal... > Suppose your application (as a whole, not the individual top-level > script) is called Spam -- this may well also be the name of your > top-level package. Then start each top-level script with the single > line > > import Spam_path > > before importing anything else. This should not be necessary if you use the name "sitecustomize" instead of "Spam_path" right? The file sitecustomize.py is automatically imported. Actually all this sounds like site.py all over again. > Your installer, once it knows the absolute pathname of your > application's root directory, crafts a file Spam_path.py which > contains code that inserts the right absolute pathname into sys.path. I don't think this is necessary either. The sys module is available. So sitecustomize.py can say: import sys mydir = sys.path[0] if not mydir: import os mydir = os.getcwd() sys.path = [mydir] # To be really extreme about it # Note: inserting mydir as sys.path[0] should be redundant but is not > Your installer then installs a copy of this file (or a symbolic link > to it) *in each bin directory where it installs top-level Python > scripts*. > > Because the script's directory is first on the default path, the Spam > scripts will pick up Spam_path without any help from $PYTHONPATH. Hmmm. Is this really true? Nothing else, for example the registry, can change sys.path[0]? Ever? Please say yes. > I know this doesn't resolve the relative import thread (how's that > going by the way? :-) but Barry & Fred & I agree that this is the best > solution to the problem stated in Barry's message to which I am > following up here. This is a good idea, but there are a few problems. It depends on sys.path[0] being the directory of the Python file being executed as the main program. I guess I never really trusted this before. I think if this is the case it should never be ''. A relative path or no path on the command line (the __main__ program) should be replaced by the full path in the sys module setup. Then the "mydir = os.getcwd()" above is not necessary. And inserting mydir as sys.path[0] is truly redundant should the current directory change (as it certainly will). This is currently a problem with sys.path[0] which should be fixed no matter what else happens. The files exceptions.py and site.py must be in all the bin directories as well as sitecustomize.py because they are automatically imported in Py_Initialize(). The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point. It seems to me this totally solves Jim Fulton's and Marc's problem and makes "__" unnecessary. You just install zope and mx in zopedir, perform the above, and presto you have a new private name space where you can control all your names. But there must be some problem here I haven't thought of. I still worry that this is not powerful enough. Greg Stein has volunteered to re-write import.c in Python (using imputil.py) and this is a Great Idea. Lots of Python could probably be written in itself. I would like to try writing the main program in Python and eliminating the special freeze main program. Once you start on this road (and I think it is a good road) you have Python code which is more truly part of the binary interpreter than a library. Proposal: Use a special PYTHONPATH on startup to find "special" Python files which are really part of the interpreter. There are three directories Python knows about. Namely sys.path[0] (once it is fixed), sys.executable and sys.dllfullpath, the directory of python15.dll or other shared library (once it is added to sys). How about prepending the single directory sys.executable to sys.path during Py_Initialize()? And demanding that modules like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] be placed there. Actually I would prefer sys.dllfullpath if it exists, since that is where the interpreter is, and I am trying to associate these special internal Python files exactly with their correct Python interpreter. Alternative Proposal: Py_Initialize() first imports its files from sys.executable + '/' + PyInternal.pyl (again I prefer sys.dllfullpath). PyInternal.pyl is a Python library file (like a Java Jar file) which would contain modules like exceptions, etc. The PyInternal.pyl file has the standard Python library file format (whatever that turns out to be). It is not an error if this file is absent. Jim Ahlstrom From guido at CNRI.Reston.VA.US Wed Sep 15 18:15:02 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 12:15:02 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 11:43:13 EDT." <37DFBE91.C93816F1@interet.com> References: <37DFBE91.C93816F1@interet.com> Message-ID: <199909151615.MAA17872@eric.cnri.reston.va.us> [Guido] > > import Spam_path > > > > before importing anything else. [JimA] > This should not be necessary if you use the name "sitecustomize" instead > of "Spam_path" right? The file sitecustomize.py is automatically > imported. > Actually all this sounds like site.py all over again. But the intention here is for the customization to be application specific (hence the Spam in the name). sitecustomize doesn't know whethere I need the Mailman or the Knowbot root added to my path. Or do you mean to imply that we can do this with zero text added to the script, by simply dropping an appropriate sitecustomize.py in the script dir? Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run. > > Your installer, once it knows the absolute pathname of your > > application's root directory, crafts a file Spam_path.py which > > contains code that inserts the right absolute pathname into sys.path. > > I don't think this is necessary either. The sys module is available. > So sitecustomize.py can say: > import sys > mydir = sys.path[0] > if not mydir: > import os > mydir = os.getcwd() > sys.path = [mydir] # To be really extreme about it > # Note: inserting mydir as sys.path[0] should be redundant but is not Hm, guessing based on the script directory might work, but seems less reliable than hardcoding it through the installer. But you can use this if it works for your application. > > Your installer then installs a copy of this file (or a symbolic link > > to it) *in each bin directory where it installs top-level Python > > scripts*. > > > > Because the script's directory is first on the default path, the Spam > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > Hmmm. Is this really true? Nothing else, for example the registry, can > change sys.path[0]? Ever? Please say yes. Yes. (The registry can add module-specific paths, which will be searched before sys.path is even looked at, but this is only for specific modules. It cannot insert a general directory that is searched.) The only way this can fail is if an embedding app fails to call PySys_SetArgv(). > > I know this doesn't resolve the relative import thread (how's that > > going by the way? :-) but Barry & Fred & I agree that this is the best > > solution to the problem stated in Barry's message to which I am > > following up here. > > This is a good idea, but there are a few problems. > > It depends on sys.path[0] being the directory of the Python > file being executed as the main program. I guess I never > really trusted this before. I think if this is the case it > should never be ''. A relative path or no path on the command > line (the __main__ program) should be replaced by the full path > in the sys module setup. Then the "mydir = os.getcwd()" above > is not necessary. And inserting mydir as sys.path[0] is truly > redundant should the current directory change (as it certainly will). > This is currently a problem with sys.path[0] which should be > fixed no matter what else happens. I have always resisted forcing path items to be absolute, although I'm not sure that my reasons are valid any more (it has to do with the fact that getcwd() may fail and the fact that portable path concatenation is a pain). In any case, that's a separate issue -- I agree that if sys.path[0] is '' (as it often is) it's better for site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize it (and everything else on the path) so that it will still work if the app does a chdir(). > The files exceptions.py and site.py must be in all the bin > directories as well as sitecustomize.py because they are > automatically imported in Py_Initialize(). Yes. > The above doesn't work when you start the Python command > interpreter (no main). I know, its a minor point. You could add the "import Spam_path" to your $PYTHONSTARTUP file. > It seems to me this totally solves Jim Fulton's and Marc's > problem and makes "__" unnecessary. You just install zope > and mx in zopedir, perform the above, and presto you have a new > private name space where you can control all your names. But > there must be some problem here I haven't thought of. I think no simple solution that *I* can come up with will satisfy JimF's and Marc's desire for obscurity :-) > I still worry that this is not powerful enough. Greg Stein > has volunteered to re-write import.c in Python (using imputil.py) > and this is a Great Idea. Lots of Python could probably be > written in itself. I would like to try writing the main > program in Python and eliminating the special freeze main > program. Once you start on this road (and I think it is a good road) > you have Python code which is more truly part of the binary > interpreter than a library. Yes, this is the plan for Python 2.0, and some of it may be implemented in Python 1.6. > Proposal: > > Use a special PYTHONPATH on startup to find "special" Python > files which are really part of the interpreter. There are > three directories Python knows about. Namely sys.path[0] > (once it is fixed), sys.executable and sys.dllfullpath, > the directory of python15.dll or other shared library (once it is > added to sys). How about prepending the single directory sys.executable > to sys.path during Py_Initialize()? And demanding that modules > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > be placed there. On Unix, this is a bin directory and it is strongly discouraged to put non-program files there. Python already does something similar -- it looks around in sys.executable's ancestors for a specific landmark, currently lib/python/string.py. Arguably, it should search for execeptions.py instead. > Actually I would prefer sys.dllfullpath if it exists, since that > is where the interpreter is, and I am trying to associate these > special internal Python files exactly with their correct Python > interpreter. Is the full DLL path available at any point? This would certainly be a good starting point -- especially when the DLL is loaded implicitly as the result of some COM operation. > Alternative Proposal: > > Py_Initialize() first imports its files from sys.executable + '/' + > PyInternal.pyl (again I prefer sys.dllfullpath). > PyInternal.pyl is a Python library file (like a Java Jar > file) which would contain modules like exceptions, etc. > The PyInternal.pyl file has the standard Python library file > format (whatever that turns out to be). It is not an error if > this file is absent. I guess this is all up to the redesign of the import mechanism (something like Greg Stein's imputil.py for sure). --Guido van Rossum (home page: http://www.python.org/~guido/) From da at ski.org Wed Sep 15 19:00:05 1999 From: da at ski.org (David Ascher) Date: Wed, 15 Sep 1999 10:00:05 -0700 (Pacific Daylight Time) Subject: [Python-Dev] License cleanup In-Reply-To: <37DF9A87.55AF6279@lemburg.com> Message-ID: Guido, maybe it would make sense to explain the need for a license change. Is my understanding correct that the occasion for the license change is that the copyright is now clearly shifting to CNRI, and as a result CNRI has to forge a license? (BTW, I thought *you* had the copyright transfer from CWI, not CNRI). --david From jim at digicool.com Wed Sep 15 20:10:17 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 14:10:17 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> Message-ID: <37DFE109.96C652B3@digicool.com> Guido van Rossum wrote: > > Jim Fulton wrote: > > > I wouldn't be in favor of making it more complicated if there wasn't > > a good reason. I think that, in working on the Zope framework, > > I've found some pretty good reasons for relative imports. > > And in a later message: > > > I don't think the Package structure of Zope is flawed *except* > > for the fact that it is one level too *shallow*. The ability to > > do relative imports would be very helpful for the work we're doing. > > But I haven't seen explained what it is that Zope is doing where > relative packages would be helpful. I posted an example in an earlier message. I'll recast it here, hopefully more eloquently. :) I'll also offer an alternate proposal that also solves my (and I suspect, Marc-Andre's) problem. Zope is an application platform. It provides a mechanism for developers to plug their own products into Zope. The idea is that someone gets Zope from zope.org and installs it. Then they get third-party products from other places. Zope products are python packages installed as sub-packages of the Zope 'Products' packages. Products are generally self-contained. If they need anything that's not part of standard Python or standard Zope, they need to include it or install what they need in a sub-package of another Zope package, 'Shared'. Because products come from "third parties", it is important that they be self contained. Making assumptions about the Zope or Python environments or, worse, modifying the Zope or Python environments is a bad idea. In this context, consider the following concrete, though fictional example. Aaron has written a collection of modules that implement an RDBMS system for Python, gadfly. He also has a set of modules for parsing, kjParsing, which is needed by gadfly. Currently, these are just a bunch of top-level modules distributed as a combined collection. It would make sense to turn these into two packages, gadfly and kjParsing. Now, if this was done, then the gadfly package would need to use package imports for kjParsing modules, as in: import kjParsing.kjParser So far, so good. Now, suppose that someone wants to create a Zope product, ZGadfly, that used gadfly. The simplest approach would be to include the gadfly and kjParsing packages in their Zope product. Of course, this won't work, because the imports, like the one above, will fail, because kjParsing is no longer a top-level package. It wouldn't do any good to move gadlfy and kjParsing to the shared package, although that might be desirable to share these packages with other products. They could try to stuff the packages into the Zope or Python paths, but that would break the rules and lead to problems in the long term. Hopefully, this illustrates the problem. I think that this will be a common problem in the future, as people build bigger and bigger systems with Python that reuse other people's packages. I'd be curious to hear how folks would solve this problem. Personally, I'd like the problem to go away. :) I'd like the Python import rules to change to make this solvable without import hooks or path hacking. I can think of two ways to approach this: - Relative parent imports: import __.kjParsing.kjParser Note that Python already supports relative imports for going down. For example, we use gadfly and kjParsing together as a single sub-package of our ZGadflyDA product. - Gradually less local searches. Currently, when doing an import in a package, two paths are searched, the package path and then the Python path. If there are intermediate packages, then perhaps their paths should be searched as well. For example, suppose we have the directory structure: Products/ZGadfly/gadfly kjParsing where Products is a top-level package, and we did an import in a module in gadfly: import kjParsing.kjParser Python would search the path of the package Products.ZGadfly.gadfly first, as it does now. This search would fail. Then it would search the path of Products.ZGadfly, where it would find kjParsing, and the import would succeed. This approach has the benefits: o It solves the problem. :) o It has no impact on un-nested packages, o It requires no code changes; it doesn't use the ugly __. Thoughts? Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gmcm at hypernet.com Wed Sep 15 19:35:57 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 13:35:57 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37DFBE91.C93816F1@interet.com> Message-ID: <1274705689-31068526@hypernet.com> Jim Ahlstrom wtoe: > "Guido van Rossum" wrote: > > --> The solution: Did the dev-list miss something? The last I see is Barry's post. - Gordon From guido at CNRI.Reston.VA.US Wed Sep 15 20:17:55 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 14:17:55 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 13:35:57 EDT." <1274705689-31068526@hypernet.com> References: <1274705689-31068526@hypernet.com> Message-ID: <199909151817.OAA18276@eric.cnri.reston.va.us> > From: "Gordon McMillan" > Jim Ahlstrom wtoe: > > > "Guido van Rossum" wrote: > > > --> The solution: > > Did the dev-list miss something? The last I see is Barry's post. Hm. We had an email glitch. Apparently this message got lost: Subject: Re: Path hacking From: Guido van Rossum To: python-dev at python.org Date: Tue, 14 Sep 1999 15:57:51 -0400 I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at interet.com Wed Sep 15 20:27:58 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 14:27:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <1274705689-31068526@hypernet.com> Message-ID: <37DFE52E.5D6F3981@interet.com> Gordon McMillan wrote: > > Jim Ahlstrom wtoe: > > > "Guido van Rossum" wrote: > > > --> The solution: > > Did the dev-list miss something? The last I see is Barry's post. My mail system is flakey, so I have been reading this list directly on python.org. I didn't get it by list either, so I assumed my mailer ate it. See: http://www.python.org/pipermail/python-dev/1999-September/000851.html Jim Ahlstrom From jim at interet.com Wed Sep 15 20:29:38 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 14:29:38 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> Message-ID: <37DFE592.7FD384A2@interet.com> Guido van Rossum wrote: > > But the intention here is for the customization to be application > specific (hence the Spam in the name). sitecustomize doesn't know > whethere I need the Mailman or the Knowbot root added to my path. Ah, you have multiple scripts in one directory and multiple Foo_path, Bar_path etc. I was thinking with my Windows head. A commercial Windows app generally has its own exclusive install directory, so I was thinking single directory so a single sitecustomize.py. > Or do you mean to imply that we can do this with zero text added to > the script, by simply dropping an appropriate sitecustomize.py in the > script dir? Yes, that is exactly what I was thinking. > Unfortunately this does currently *not* work, because > sys.path[0] is added after Py_Initialize() is run. Yikes! That kills using sitecustomize.py. Your Spam_path still works because it is imported later, but requires an import in each Python main script just as you said. Even worse, it means that exceptions.py and site.py can not be found at all except using the normal PYTHONPATH, and putting their path in Spam_path will *not* work. > > > Because the script's directory is first on the default path, the Spam > > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > > > Hmmm. Is this really true? Nothing else, for example the registry, can > > change sys.path[0]? Ever? Please say yes. > > Yes. (The registry can add module-specific paths, which will be > searched before sys.path is even looked at, but this is only for > specific modules. It cannot insert a general directory that is > searched.) The only way this can fail is if an embedding app fails to > call PySys_SetArgv(). Oh dear, I think I heard no instead of yes. Are you saying that if someone else installs a Python app on my customer's machine after I do, and sets a registry entry which sayes to use c:/other/path/to/site.py for site.py (as he may very well want to do), then if my Python program depends on getting my copy of site.py from my directory, it will then use the other copy instead and may very well fail? > In any case, that's a separate issue -- I > agree that if sys.path[0] is '' (as it often is) it's better for > site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize > it (and everything else on the path) so that it will still work if the > app does a chdir(). Point on the curve: Windows apps generally start from an icon which contains their path and current working directory, and these are generally different. So a Windows app in general will *never* have had a getcwd() equal to the path of either the binary interpreter or the Python main script. > > The files exceptions.py and site.py must be in all the bin > > directories as well as sitecustomize.py because they are > > automatically imported in Py_Initialize(). > > Yes. Well, *no* right? This fails unless the bin directories are in fact on PYTHONPATH. The only way to get exceptions.py is by using sys.path as it exists within Py_Initialize(). So there is no hacked sys.path[0] equal to the script dir. And since the path hacks in site.py haven't happened yet either, we have an incomplete sys.path at that point. > > added to sys). How about prepending the single directory sys.executable > > to sys.path during Py_Initialize()? And demanding that modules > > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > > be placed there. > > On Unix, this is a bin directory and it is strongly discouraged to put > non-program files there. Ok, point taken. > Is the full DLL path available at any point? This would certainly be > a good starting point -- especially when the DLL is loaded implicitly > as the result of some COM operation. I don't know about loading by COM, but if it is a file, its absolute path is reliably known in sys, the code is identical to that currently used for sys.executable (on Windows), and I have a patch if you want. JimA's conjecture: It is currently impossible to ship a Python app which can not be damaged by the installation of a second Python app without using a hacked custom binary. Jim Ahlstrom From bwarsaw at cnri.reston.va.us Wed Sep 15 20:41:26 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 14:41:26 -0400 (EDT) Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <1274705689-31068526@hypernet.com> Message-ID: <14303.59478.806707.880694@anthem.cnri.reston.va.us> >>>>> "Gordo" == Gordon McMillan writes: Gordo> Jim Ahlstrom wtoe: >> "Guido van Rossum" wrote: >> --> The solution: Gordo> Did the dev-list miss something? The last I see is Barry's Gordo> post. I have a suspicion that python.org lost some email yesterday. We had a period of time where mail simply stopped getting delivered (thank you Solaris patch manager) and it took me a little while to realize that things weren't working correctly. Since there's nothing unexpected in the mail queue now, all I can say is that if you didn't get it by now, you ain't gonna. However, everything seemed to make it into the archives, so Guido's message is available at: http://www.python.org/pipermail/python-dev/1999-September/000880.html -Barry From guido at CNRI.Reston.VA.US Wed Sep 15 20:43:56 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 14:43:56 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Wed, 15 Sep 1999 10:00:05 PDT." References: Message-ID: <199909151843.OAA18364@eric.cnri.reston.va.us> [David Ascher] > Guido, maybe it would make sense to explain the need for a license change. > Is my understanding correct that the occasion for the license change is > that the copyright is now clearly shifting to CNRI, and as a result CNRI > has to forge a license? (BTW, I thought *you* had the copyright transfer > from CWI, not CNRI). Correct on both counts. CWI owns the copyright on old Python versions through Python 1.2. I have personally obtained non-exclusive rights to these from CWI. CNRI, by nature of my employment contract, has the copyright on newer versions. CNRI feels the need to protect its intellectual property rights. It feels that the old Python license, even with CNRI added, does not adequately protect CNRI against certain (unlikely) events -- hence the desire to draft a new license. CNRI understands that open source (and now Open Source -- the OSI board has approved the old Python license!) like Python requires different licensing terms than a typical product developed solely by CNRI. I think that the main problem is that CNRI's understanding of what truly constritutes open source is limited, and that my own understanding of legal issues is limited, so that the negotiations with CNRI's legal department (which is headed by CNRI's director) often turn in their favor. I hereby withdraw the posted license. There still is the need for a new license, but we need to go back to the drawing board for it. The CWI ownership of much of the code probably means that the license as it stands doesn't hold anyway. I also think that the Python consortium has a say in the license discussion -- the consortium agreement actually discusses the ownership of intellectual property produced by/for the consortium at some length. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw at cnri.reston.va.us Wed Sep 15 20:46:44 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 14:46:44 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <14303.59796.4588.287265@anthem.cnri.reston.va.us> JF> Zope products are python packages installed as sub-packages of JF> the Zope 'Products' packages. Products are generally JF> self-contained. If they need anything that's not part of JF> standard Python or standard Zope, they need to include it or JF> install what they need in a sub-package of another Zope JF> package, 'Shared'. Jim, from your description, it sounds like Zope should be putting both the `Shared' and possibly `Products' directories on sys.path. Maybe it does and I'm still missing your point. However if they are not, then wouldn't this solve your problem? A user would drop the kjParsing directory into Shared, and then all the import kjParsing.kjParser statements would Just Work. -Barry From mal at lemburg.com Wed Sep 15 19:02:20 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 19:02:20 +0200 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> Message-ID: <37DFD11C.712653BA@lemburg.com> Guido van Rossum wrote: > > It seems to me this totally solves Jim Fulton's and Marc's > > problem and makes "__" unnecessary. You just install zope > > and mx in zopedir, perform the above, and presto you have a new > > private name space where you can control all your names. But > > there must be some problem here I haven't thought of. > > I think no simple solution that *I* can come up with will satisfy > JimF's and Marc's desire for obscurity :-) Never mind, I'll use an imputil.py based approach to get relative imports to work in my packages. That is when I get imputil.py to work... it doesn't seem to be quite there yet (or I'm using an old version). BTW, I'm 100% behind you guys if you choose to reimplement Python's import mechanism in Python using a similar approach as the one Greg implemented in imputil. Should make everybody happy: those who want obscure syntactic add-ons and others with a taste for zlib'ed packaged byte code, plus those VMS freaks ;-) Perhaps we should start a new thread on that topic... Still needed are: ? Python level APIs for the platform specific magic on Win32 and Macs (OS/2, BeOS ?), e.g. access to the Windows registry and the Mac forks ? Patches to make the DirectoryImporter 100% backward compatible Greg's imputil.py can be found at: http://www.lyra.org/greg/small/ The trick would then be to install an application specific importer in the setup module Spam_path or MyAppSetup which then takes care of all the rest... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Wed Sep 15 21:00:41 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 15:00:41 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <14303.59478.806707.880694@anthem.cnri.reston.va.us> Message-ID: <1274700606-31374305@hypernet.com> Hmm, I'm suspicious of the fact that no message from Barry Warsaw ever gets "lost". Stalin got started by being in charge of the Kremlin's telephone system, you know... > >>>>> "Gordo" == Gordon McMillan writes: > > Gordo> Jim Ahlstrom wtoe: > > >> "Guido van Rossum" wrote: > >> --> The solution: > > Gordo> Did the dev-list miss something? The last I see is > Barry's Gordo> post. > > I have a suspicion that python.org lost some email yesterday. > > We had a period of time where mail simply stopped getting > delivered (thank you Solaris patch manager) and it took me a > little while to realize that things weren't working correctly. > Since there's nothing unexpected in the mail queue now, all I can > say is that if you didn't get it by now, you ain't gonna. > > However, everything seemed to make it into the archives, so > Guido's message is available at: > > http://www.python.org/pipermail/python-dev/1999-September/000880. > html > > -Barry > > _______________________________________________ > Python-Dev maillist - Python-Dev at python.org > http://www.python.org/mailman/listinfo/python-dev - Gordon From jim at interet.com Wed Sep 15 21:25:24 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 15:25:24 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <37DFF2A4.DF26A22B@interet.com> Jim Fulton wrote: > In this context, consider the following concrete, though fictional > example. > ............ The classical solution would be to add "Shared" to sys.path. But how to do this? And what if another installation changes it? > I'd be curious to hear how folks would solve this problem. 1) Ship a custom binary interpreter with critical modules frozen in, and which sets a custom sys.path before anything else happens. So you are not dependent at all on PYTHONPATH. Currently this works. 2) Add a new command line option which sets sys.path and always use it to start your app. It lists only critical zope directories so zope has its own private name space. 3) Add your own import hooks using imputil.py and boot it as in (1). Better if it were part of Python. 4) Use a new Python.pyl file which has seeking from the end and thus allows concattenation of multiple *.pyl. Each package is a foo.pyl, bar.pyl, and to distribute an app, you concatenate packages. The implied PYTHONPATH in each *.pyl file is "." except perhaps for the Python lib which is "Lib". Boot it as in (3). > Personally, I'd like the problem to go away. :) I have the same problem, so me too. > I'd like the Python > import rules to change to make this solvable without import hooks or > path hacking. I can think of two ways to approach this: > > - Relative parent imports: > > import __.kjParsing.kjParser OK by me, but doesn't solve the Grand Problem. > - Gradually less local searches. I am currently bewildered by Python's import mechanism, and this may push me into brain damage. Well, *more* brain damage. Jim Ahlstrom From jim at digicool.com Wed Sep 15 21:34:08 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 15:34:08 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> Message-ID: <37DFF4B0.15390199@digicool.com> "Barry A. Warsaw" wrote: > > JF> Zope products are python packages installed as sub-packages of > JF> the Zope 'Products' packages. Products are generally > JF> self-contained. If they need anything that's not part of > JF> standard Python or standard Zope, they need to include it or > JF> install what they need in a sub-package of another Zope > JF> package, 'Shared'. > > Jim, from your description, it sounds like Zope should be putting both > the `Shared' and possibly `Products' directories on sys.path. It actually does now, although that is bad. What if some future python distribution has a Products package? Or a Products.py? A product is actually a sub-package of Products. Even if Products was not a package, the Product would be and gadfly and kjParsong would be sub-packages. There is a similar issue with Shared, since Shared software is organized by organization, as in 'Shared.DC.ZRDB'. > Maybe > it does and I'm still missing your point. The point is that stuffing arbitrary things into a single global name space is bad. That's why packages were invented. Unless packages can be effectively nested, you haven't gained anything but time. > However if they are not, then wouldn't this solve your problem? No, see above. > A > user would drop the kjParsing directory into Shared, and then all the > import kjParsing.kjParser statements would Just Work. But then they'd be dumping kjParsing into the global package namespace. Eventually, there will be collisions. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Wed Sep 15 21:59:10 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 15:59:10 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <37DFF2A4.DF26A22B@interet.com> Message-ID: <37DFFA8E.FB73A006@digicool.com> "James C. Ahlstrom" wrote: > > Jim Fulton wrote: > > In this context, consider the following concrete, though fictional > > example. > > ............ > > The classical solution would be to add "Shared" to sys.path. But > how to do this? And what if another installation changes it? This doesn't work, because 'Shared' has sub-packages by organization. > > I'd be curious to hear how folks would solve this problem. > > 1) Ship a custom binary interpreter with critical modules frozen in, > and which sets a custom sys.path before anything else happens. So > you are not dependent at all on PYTHONPATH. Currently this works. This is what we do for our binary distribtions, but it doesn't do anything for third-party products added to Zope. > 2) Add a new command line option which sets sys.path and always use > it to start your app. It lists only critical zope directories so > zope has its own private name space. We have a start script that takes care of this. This gets Zope's top-level names (of which there are too many already) into the path. This doesn't help with third-party products added to Zope. > 3) Add your own import hooks using imputil.py and boot it as > in (1). But imputil has a global effect. I suppose I could implement what I want with imputil or by otherwise creating my own importer using the standard import hooks. I suspect that's what I'll have to do eventually. > Better if it were part of Python. > 4) Use a new Python.pyl file which has seeking from the end and > thus allows concattenation of multiple *.pyl. Each package is > a foo.pyl, bar.pyl, and to distribute an app, you concatenate > packages. The implied PYTHONPATH in each *.pyl file is "." > except perhaps for the Python lib which is "Lib". Boot it as > in (3). I'm not looking for a way to replace the current package scheme. Also, I'm not just looking at an app, but at an app that contains apps (aka components). > > Personally, I'd like the problem to go away. :) > > I have the same problem, so me too. > > > I'd like the Python > > import rules to change to make this solvable without import hooks or > > path hacking. I can think of two ways to approach this: > > > > - Relative parent imports: > > > > import __.kjParsing.kjParser > > OK by me, but doesn't solve the Grand Problem. > > > - Gradually less local searches. > > I am currently bewildered by Python's import mechanism, and > this may push me into brain damage. Well, *more* brain damage. I don't see what's so complicated. You just use a gradually widening search, rather than an immediately widening search. I don't think that this is any more complicated than the current rule. I actually prefer the second apprach because it doesn't require any code changes to take advantage of it. I may implement it for Zope if nothing is done in standard Python. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal at lemburg.com Wed Sep 15 22:01:01 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 22:01:01 +0200 Subject: [Python-Dev] Towards a Python based import scheme Message-ID: <37DFFAFD.72F6A806@lemburg.com> Back again... To get a little more constructive I've started hacking away on Greg Stein's imputil.py to make it work with my DateTime package. The DateTime packages does a lot of from...import... and intra-package imports, plus it loads a shared lib as extension. The original version of imputil I fetched from Greg's page did work out of the box (from...import... hassles) and obviously did not support in-package shared libs. I've added both features so that the test script in DateTime can run successfully. Things that remain are: ? the win32 registry stuff (needs C code) ? the Mac fork stuff (needs C code) ? a working __path__ implementation (is anyone using this attribute which only is available in packages ?) ? probably a whole bunch of other quirks ? some speedups (there currently are too many stat()s) Please give it a try: http://starship.skyport.net/~lemburg/imputil.py in color: http://starship.skyport.net/~lemburg/imputil.py.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Wed Sep 15 22:22:34 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 16:22:34 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DFF4B0.15390199@digicool.com> Message-ID: <1274695692-31669872@hypernet.com> Jim Fulton wrote: > It actually does now, although that is bad. What if some future > python distribution has a Products package? Or a Products.py? Name it zopeProducts. Or Zope/Products and Zope/Shared. > A product is actually a sub-package of Products. Even if Products > was not a package, the Product would be and gadfly and kjParsong > would be sub-packages. There is a similar issue with Shared, > since Shared software is organized by organization, as in > 'Shared.DC.ZRDB'. See below. > The point is that stuffing arbitrary things into a single global > name space is bad. That's why packages were invented. Unless > packages can be effectively nested, you haven't gained anything > but time. Packages nest effectively. Subpackages just don't masquerade as top level packages. [Barry] > > A > > user would drop the kjParsing directory into Shared, and then > > all the import kjParsing.kjParser statements would Just Work. > > But then they'd be dumping kjParsing into the global package > namespace. Eventually, there will be collisions. You are trying to do two incompatible things at once. You are trying to be both a standalone application, and a Python installation. You can be both, just not at the same time. To be a standalone application, you can use tricks that Jim Ahlstrom pointed out, or tricks from my Installer. The major point being that sys.path has nothing to do with the PYTHONPATH environment variable. You now have an installation where you can be assured of no collisions, because you control it completely. Of course, that will make liars out of the people at your company who claim you can mix and match pieces of Zope and your own stuff as you please. People who want to mix and match need Zope to play by normal Python rules - and that means packages are installed at the top level. Collisions happen, but your audience is people who *want* to do things differently and *want* control of their installations. By creating "self contained" packages and putting them in a normal Python installation, you're just pushing your headache off on the user. You're co-opting his installation. Yeah, now you have two versions, but they're 99% the same. Build the normal Python one first and distribute it to the hackers. Then install some import and startup hooks, and distribute that to your "users". I, for one, still use Bobo, because Zope doesn't play fair and I have no need or desire for yet another Python installation. - Gordon From gmcm at hypernet.com Wed Sep 15 22:43:27 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 16:43:27 -0400 Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <1274694439-31745225@hypernet.com> Marc-Andre wrote: > To get a little more constructive I've started hacking away on > Greg Stein's imputil.py to make it work with my DateTime package. > http://starship.skyport.net/~lemburg/imputil.py You (and Greg) are missing a rather important patch I submitted to Greg a long time ago (around line 62): def _reload_hook(self, module): # gmcm - Hmmm, reloading of a module may or may not be impossible, # (depending on the importer), but at least we can # look to see if it's ours to reload: if hasattr(module, '__importer__'): if getattr(module, '__importer__') == self: raise SystemError, "reload not yet implemented" return self.__chain_reload(module) - Gordon From jim at digicool.com Wed Sep 15 22:44:31 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 16:44:31 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274695692-31669872@hypernet.com> Message-ID: <37E0052F.A899E985@digicool.com> Gordon McMillan wrote: > > Jim Fulton wrote: > (snip) > Packages nest effectively. Subpackages just don't > masquerade as top level packages. They do currently when imported from the containing package. > [Barry] > > > A > > > user would drop the kjParsing directory into Shared, and then > > > all the import kjParsing.kjParser statements would Just Work. > > > > But then they'd be dumping kjParsing into the global package > > namespace. Eventually, there will be collisions. > > You are trying to do two incompatible things at once. You are > trying to be both a standalone application, and a Python > installation. You can be both, just not at the same time. We don't try to be a Python installation. We try to be a Python application that is a platform for Zope applications. > To be a standalone application, you can use tricks that Jim > Ahlstrom pointed out, or tricks from my Installer. The major > point being that sys.path has nothing to do with the > PYTHONPATH environment variable. You now have an > installation where you can be assured of no collisions, > because you control it completely. > > Of course, that will make liars out of the people at your > company who claim you can mix and match pieces of Zope > and your own stuff as you please. Zope reuses *alot* of code. I think Zope has a pretty good track record of working with existing code. > People who want to mix > and match need Zope to play by normal Python rules - and > that means packages are installed at the top level. Sorry, I just don't buy that. > Collisions > happen, but your audience is people who *want* to do things > differently and *want* control of their installations. Actually, many Zope user's come to Python by way of Zope. They want to be able to install Zope products without worrying about what their zope or python installation looks like. Zope developers want to be able to have their products drop into a Zope installation easily. > By creating "self contained" packages and putting them in a > normal Python installation, you're just pushing your headache > off on the user. You're co-opting his installation. No, putting packages in the python path coops their installation. Putting Zope-specific products that happen to include reusable Python packages in a Zope-specific package leaves their installation alone. > Yeah, now you have two versions, but they're 99% the same. > Build the normal Python one first and distribute it to the > hackers. Then install some import and startup hooks, and > distribute that to your "users". This is exactly what I want to prevent. I'd like package developers to develop their packages once without making them use esoteric import hooks. OTOH I'd like to be able to *use* their packages wherever I damn well please. I don't to be forced to put their packages in the python path. > I, for one, still use Bobo, because Zope doesn't play fair In what way does Zope not play fair? > and I > have no need or desire for yet another Python installation. Cool. You don't need to use another Python installation to use Zope. In fact, for development, we use a totally standard Python distribution. Developers here are not *allowed* to add anything special to the Python installations on the development machines here because we want to *make sure* that we can run with stock python installations. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From bwarsaw at cnri.reston.va.us Wed Sep 15 23:55:34 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 17:55:34 -0400 (EDT) Subject: [Python-Dev] Re: Path hacking References: <14303.59478.806707.880694@anthem.cnri.reston.va.us> <1274700606-31374305@hypernet.com> Message-ID: <14304.5590.405325.369903@anthem.cnri.reston.va.us> >>>>> "Gordo" == Gordon McMillan writes: Gordo> Hmm, I'm suspicious of the fact that no message from Barry Gordo> Warsaw ever gets "lost". Gordo> Stalin got started by being in charge of the Kremlin's Gordo> telephone system, you know... Well, I managed to get rid of Ken so my world domination plan is right on schedule! guido-may-be-benevolent-but-you-can-bet-i-won't-be-ly y'rs, -Barry From bwarsaw at cnri.reston.va.us Thu Sep 16 00:07:34 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 18:07:34 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> Message-ID: <14304.6310.20571.487708@anthem.cnri.reston.va.us> >>>>> "JF" == Jim Fulton writes: JF> It actually does now, although that is bad. What if some JF> future python distribution has a Products package? Or a JF> Products.py? Wouldn't collide, because in my scheme, Shared isn't a package, it's a directory on sys.path. Everything inside Shared is a top-level package. You'd never "import Shared.DC.ZRDB" but instead you would "import DC.ZRDB". >> A user would drop the kjParsing directory into Shared, and then >> all the import kjParsing.kjParser statements would Just Work. JF> But then they'd be dumping kjParsing into the global package JF> namespace. Eventually, there will be collisions. Do you really think so? Someone else is going to write a kjParsing module? I think we're safe from this for the forseeable future through convention and peer pressure. Or is the concern that there will be two Products which use different versions of kjParsing? I'm also not concerned here because if my product needs a very specific version of kjParsing I can't see any way around shipping my own version, subpackaging it, and using an absolute package name to ensure that I'm getting at exactly the one I need. This is the approach Mailman takes. -Barry From guido at CNRI.Reston.VA.US Thu Sep 16 00:35:20 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 18:35:20 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 14:29:38 EDT." <37DFE592.7FD384A2@interet.com> References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> Message-ID: <199909152235.SAA18693@eric.cnri.reston.va.us> [me] > > But the intention here is for the customization to be application > > specific (hence the Spam in the name). sitecustomize doesn't know > > whethere I need the Mailman or the Knowbot root added to my path. [JimA] > Ah, you have multiple scripts in one directory and multiple > Foo_path, Bar_path etc. I was thinking with my Windows head. > A commercial Windows app generally has its own exclusive > install directory, so I was thinking single directory so a single > sitecustomize.py. > > > Or do you mean to imply that we can do this with zero text added to > > the script, by simply dropping an appropriate sitecustomize.py in the > > script dir? > > Yes, that is exactly what I was thinking. > > > Unfortunately this does currently *not* work, because > > sys.path[0] is added after Py_Initialize() is run. > > Yikes! That kills using sitecustomize.py. Your Spam_path > still works because it is imported later, but requires an > import in each Python main script just as you said. Not too bad (who cares about one more line of boilerplate...). > Even worse, it means that exceptions.py and site.py can not > be found at all except using the normal PYTHONPATH, and > putting their path in Spam_path will *not* work. Why would you want your own exceptions.py and site.py? > > > > Because the script's directory is first on the default path, the Spam > > > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > > > > > Hmmm. Is this really true? Nothing else, for example the registry, can > > > change sys.path[0]? Ever? Please say yes. > > > > Yes. (The registry can add module-specific paths, which will be > > searched before sys.path is even looked at, but this is only for > > specific modules. It cannot insert a general directory that is > > searched.) The only way this can fail is if an embedding app fails to > > call PySys_SetArgv(). > > Oh dear, I think I heard no instead of yes. Are you saying that if > someone else installs a Python app on my customer's machine after I do, > and sets a registry entry which sayes to use c:/other/path/to/site.py > for site.py (as he may very well want to do), then if my Python program > depends on getting my copy of site.py from my directory, it will then > use the other copy instead and may very well fail? Again - why would anyone register their own site.py? > > In any case, that's a separate issue -- I > > agree that if sys.path[0] is '' (as it often is) it's better for > > site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize > > it (and everything else on the path) so that it will still work if the > > app does a chdir(). > > Point on the curve: Windows apps generally start from an icon > which contains their path and current working directory, and > these are generally different. So a Windows app in general will > *never* have had a getcwd() equal to the path of either the > binary interpreter or the Python main script. You're lucky. It turns out that on Windows, under those circumstances at least, sys.path[0] is the absolute pathname of the directory. You only see '' if sys.argv[0] doesn't have any pathname information; that's only possible if the script *does* live in the current directory. > > > The files exceptions.py and site.py must be in all the bin > > > directories as well as sitecustomize.py because they are > > > automatically imported in Py_Initialize(). > > > > Yes. > > Well, *no* right? This fails unless the bin directories are in > fact on PYTHONPATH. The only way to get exceptions.py is by using > sys.path as it exists within Py_Initialize(). So there is no > hacked sys.path[0] equal to the script dir. And since the > path hacks in site.py haven't happened yet either, we have > an incomplete sys.path at that point. Sorry, I've lost track of what we were after here. Indeed the scripts' directory (which I presume you meant by the bin directory) indeed doesn't occur in sys.path until after Py_Initialize() has run. > > > added to sys). How about prepending the single directory sys.executable > > > to sys.path during Py_Initialize()? And demanding that modules > > > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > > > be placed there. > > > > On Unix, this is a bin directory and it is strongly discouraged to put > > non-program files there. > > Ok, point taken. > > > Is the full DLL path available at any point? This would certainly be > > a good starting point -- especially when the DLL is loaded implicitly > > as the result of some COM operation. > > I don't know about loading by COM, but if it is a file, its absolute > path is reliably known in sys, the code is identical to that currently > used for sys.executable (on Windows), and I have a patch if you want. I presume using GetModuleFileName()? Please send me the patch! > JimA's conjecture: It is currently impossible to > ship a Python app which can not be damaged by the installation of a > second Python app without using a hacked custom binary. Sounds right. All tricks to make the app unique require using a different registry key, which requires a change to the DLL. However, you can do this without recompiling! The version string is used is embedded in a resource, so you can patch it using some kind of resource editor. Mark Hammond planned it this way! --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at digicool.com Thu Sep 16 00:49:38 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 18:49:38 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> Message-ID: <37E02282.BDF5FF21@digicool.com> "Barry A. Warsaw" wrote: > > >>>>> "JF" == Jim Fulton writes: > > JF> It actually does now, although that is bad. What if some > JF> future python distribution has a Products package? Or a > JF> Products.py? > > Wouldn't collide, because in my scheme, Shared isn't a package, it's a > directory on sys.path. Everything inside Shared is a top-level > package. You'd never "import Shared.DC.ZRDB" but instead you would > "import DC.ZRDB". I wasn't talking about your scheme. I was talking about how I thought it should be. > >> A user would drop the kjParsing directory into Shared, and then > >> all the import kjParsing.kjParser statements would Just Work. > > JF> But then they'd be dumping kjParsing into the global package > JF> namespace. Eventually, there will be collisions. > > Do you really think so? Yes. > Someone else is going to write a kjParsing > module? No, but sooner or later, two people are going to write two "Parsing" packages, or two "Oracle" packages or two .... Are you suggesting that package names should begin with some hash value to assure that they are unique? (What does kj stand for anyway? :) I guess we could use some form of GUID to be really safe ..... > I think we're safe from this for the forseeable future > through convention and peer pressure. 8^o (I feel like I should respond to this in some way but .... I just can't understand this kind of argument and don't know how to frame a response.) > Or is the concern that there will be two Products which use different > versions of kjParsing? No, at least not primarily. > I'm also not concerned here because if my > product needs a very specific version of kjParsing I can't see any way > around shipping my own version, subpackaging it, and using an absolute > package name to ensure that I'm getting at exactly the one I need. But even in this case, if you used a relative import, you *would* get the specific version you want. I give up. I'll just have to write some import hook to get the behavior I want in Zope. I will probably implement the second alternative I wrote about in my earlier document. I'll get Python to look at intermediate package paths before it looks at the main path. That way, any package that works as a top-level package will also work as a sub-package without modification. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mhammond at skippinet.com.au Thu Sep 16 03:20:10 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 16 Sep 1999 11:20:10 +1000 Subject: [Python-Dev] License cleanup In-Reply-To: <199909151843.OAA18364@eric.cnri.reston.va.us> Message-ID: <004601beffe1$9edb01b0$0801a8c0@bobcat> > CNRI understands that open source (and now Open Source -- the OSI > board has approved the old Python license!) like Python requires > different licensing terms than a typical product developed solely by > CNRI. Actually, this is a good point. I know the issue of payment may get in the way, but it would make sense to have any future proposed licenses reviewed by a lawyer "on our side" - eg, someone whose mandate is to give a legal opinion on the risks and liabilities of the _user_ of the license. Obviously the CNRI lawers are protecting their (ie, CNRI's) interests, and everyone on this group is concerned about their own (ie, personally, their company, or companies they wish to introduct Python into) interests. If the legal jibberish can't be removed (which is likely with lawyers involved) I know I would personally feel much more comfortable with a legal opinion covering my interests.. But as I said, who will pay? If nothing else, we should ensure the OSI approves of the new license... Or maybe we can convince CNRI there is real and serious concern, and they could pay for an external IP lawyer? Mark. From gstein at lyra.org Thu Sep 16 03:37:17 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 15 Sep 1999 18:37:17 -0700 Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> Message-ID: <37E049CD.6B64DE00@lyra.org> Mark Hammond wrote: >... > Actually, this is a good point. I know the issue of payment may get > in the way, but it would make sense to have any future proposed > licenses reviewed by a lawyer "on our side" - eg, someone whose > mandate is to give a legal opinion on the risks and liabilities of the > _user_ of the license. Interesting thought! > Obviously the CNRI lawers are protecting their > (ie, CNRI's) interests, and everyone on this group is concerned about > their own (ie, personally, their company, or companies they wish to > introduct Python into) interests. Because of these varied interests, I don't think a review by a particular lawyer will be greatly helpful. The question will still remain: "did the lawyer review it from [my/our/company] perspective?" This will lead people back into the same review cycle. >... > But as I said, who will pay? If nothing else, we should ensure the > OSI approves of the new license... Or maybe we can convince CNRI > there is real and serious concern, and they could pay for an external > IP lawyer? Definitely having an OSI certification will be great (cool stuff on the cert for the existing license!). Having Bruce Perens review the license would also be a great boon (see www.perens.com for some of his writings; also see http://perens.com/Termination.html specifically). Licenses are a tough issue. I had to go through this entire morass when deciding what to do with mod_dav. There are a lot of varieties and issues and stuff to cover. I've read a bunch of license (not to mention a bazillion legal documents during the eShop/acquisition days). Not always exciting reading :-), but usually quite interesting. At this point, I think it is a great thing that CNRI is reviewing the license. Unfortunately, the license wasn't as non-controversial as it was thought to be :-(. I'm more than happy to wait and see where they go with the license. Cheers, -g -- Greg Stein, http://www.lyra.org/ From guido at CNRI.Reston.VA.US Thu Sep 16 05:32:18 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 23:32:18 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Wed, 15 Sep 1999 14:10:17 EDT." <37DFE109.96C652B3@digicool.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <199909160332.XAA19278@eric.cnri.reston.va.us> I don't like the pressure to allow relative imports and I don't like the solutions that those who insist on having it propose (they are all writing their own import hooks). The reasoning seems to be something like this: (a) If my package needs non-standard package X, I can't require that X be installed as a top-level package, because there might be another unrelated package also called X; therefore I need to install a copy of package X as a subpackage of my own package. (b) Because any non-standard package X now may be installed at the toplevel or as a subpackage of some other package, X must use relative imports exclusively to access subparts of itself. I find (b) unacceptable for a number of reasons, that have all been explained by others: duplicate installs (and hence possible imports) of the same package as a subpackage of various other packages, the requirement to be religious in the use of relative imports, the ugliness of the __ notation (I do *not* look forward to explaining that to 12-year-olds in the context of CP4E). I think that the fear expressed in (a) is an overreaction to some recent actual top-level name clashes. This was a real pain, but notice that it happened because several popular module collections weren't packagized at all! In the Java world, the right solution exist: a better top-level naming scheme. I forget what the exact rules are, but I seem to recall that Java has something of the same relative import rules af Python currently has: you can import a class from the same leaf package by its simple name, or you can import a class by its full name. As Tim explains, the occasional renames are easily handled with a global substitute in a smart editor. I don't want to do the same thing as Java and require everybody to use their reversed domain name as their package name. Of course you are free to do so: there are currently no toplevel modules or packages that I know of named org, com, edu, gov, mil or net. Two letter domains are all free except for re=Reunion (and ni=Nicaragua, but ni is expired). Oops, mx=Mexico! Well, Python package semantics are slightly different than Java semantics in that shared ownership of a subpackage of (e.g. :-) mx is harder in Python than in Java (Python requires that they are installed in the same directory; Java doesn't). In any case the proper thing to do is to pick a good unique toplevel name. Aaron Watters did a great job with kj (not a country code either!); Marc Andre also has the right spirit with mx. CNRI will probably use cnri as the toplevel package name for CNRI-developed packages that aren't distributed together with Python. If there are still worries about name choices, I volunteer to run a toplevel name registry. This has worked well e.g. for Apple where application developers had to pick a unique 4-letter code; certainly better than the Windows world where conflicting 3-letter extension definitions are common. In the Python world, I would recommend longer toplevel names, so it's more likely that you can use your company name (e.g. pythonware). Note that I still resist the urge to start renaming all the existing standard modules -- too much breakage. This will happen in 2.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one at email.msn.com Thu Sep 16 05:38:22 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 23:38:22 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DF65DB.46F0191A@lemburg.com> Message-ID: <000101befff4$ecc9bde0$712d153f@tim> Sure is a lot of traffic on this -- can't we go back to hating the new license ? [MAL] > ... > Sticking this technology into the import statement is really only > a more elegant approach. Nothing more. The whole thing already > works using the current Python implementation. Your point there has been made many times -- I'm not missing it. My point has also been made many times, so one more can't hurt : it still looks to me like bad practice, so I don't *want* (more, actively oppose) it to be made easier to spell. >> + JimF and MAL's shared notion that they're the only ones making significant >> use of packages is myopic. > Oh, c'mon Tim, we never did say that. We only mentioned having used > package for quite a while. This includes having seen all the different > pitfalls they have to offer. Well, JimF was laying it on a bit thick in two msgs, and sucked you in as his implicit conspirator. My apologies if you feel tarred unfairly. Yet you still can't resist implying that only you two are aware of the pitfalls . >> [on Dragon's "relentlessly straightforward" import policies] > This is perfectly ok if you're just using your own code, but it > gets a mess when third-party packages are involved simply because > you can't be sure they don't use import hooks, local imports or > hacks to __path__ etc. If you still want them to be useable, you'll > have to go down and dive into their structure. This takes time, is > error prone and not necessarily fun. While true, it's much safer to install a "cheating" package at top level, since the cheats presumably *assume* that's where the package is to be run from. If someone distributes a package that does not run correctly from top level, that's one hopeless package. Else it does run correctly from top level, and then the only person with a problem here is one who insists on running it from some embedded location. You then buy the need to dive into their code very carefully indeed -- relative imports don't save you from the huge variety of "clever tricks" they *may* be playing with hooks and path hacks. But note that, unlike Jim (& perhaps you too), I have almost no fear of top-level name collisions. If we do nothing about that, it will be years before it becomes a real problem; and several months before that , we can flock to Java's hokey but effective reverse-domain scheme (which I suspect you're already in favor of -- fine by me!). > Also, I don't know how "explicitly, long-windedly and straightforwardly" > writing module imports is any different from using relative imports. Hugely and profoundly different, as your example shows: > An example: > > mx > .DateTime > .ODBC.Windows > > In ODBC.Windows I would write: > > import __.__.DateTime > > which is just as explicit as writing > > import mx.DateTime > > except that the information about the top-level hierarchy is *not* > included in the import information. Exactly: now instead of a bulletproof absolute import in a self-contained importer, whose precise meaning is clear from casual static inspection, you've got a relative one whose meaning depends vitally on where the importer happens to live. As code moves around, the import may fail to work when you're lucky, or pick up an unintended DateTime package when you're not. The semantics of the import are indeed relative now: you see the opportunity to type three additional characters as a strength, and I see nothing but headaches masquerading as convenience. As further consequence, simple "search-and-replace" strategies are rendered at best useless. That is, a simple grep can no longer find all & only references to mx's flavor of DateTime: it picks up all references to all DateTime guys, with no way short of a structure-aware tree crawler to sort them out again. Complexity spreads. Of course, in the end, I lose no matter what. If *everyone* used dirt-dumb fully-qualified imports, all hassles with shifting package structure could be solved with a text editor in seconds. But that's not whiz-bang enough, so people *will* play endlessly dubious tricks with hooks, paths, relative imports, and metapackages too once Gordon gets around to them . Having an explicit way to spell relative imports would make my life easier because at least those are statically recognizable for what they are. I don't want them and won't use them, but it could ease the pain others will cause by using them. That's not enough to make me in favor of them, though; just enough so I won't pout if they do go in. > Anyway, I'm not too inclined continuing this discussion anymore. > I would never have thought that such a nifty little patch would > have stirred up so much negative response. I'll just hack up my > very own import mechanism using imputil and be done. So there! Don't feel picked on, M-A! Opposition is normal and healthy; and the last idea I got anywhere with was list.pop(). and-even-then-jimf-tried-to-stick-a-default-on-it-ly y'rs - tim From da at ski.org Thu Sep 16 06:19:32 1999 From: da at ski.org (David Ascher) Date: Wed, 15 Sep 1999 21:19:32 -0700 (Pacific Daylight Time) Subject: [Python-Dev] License cleanup In-Reply-To: <004601beffe1$9edb01b0$0801a8c0@bobcat> Message-ID: On Thu, 16 Sep 1999, Mark Hammond wrote: > But as I said, who will pay? If nothing else, we should ensure the > OSI approves of the new license... Or maybe we can convince CNRI > there is real and serious concern, and they could pay for an external > IP lawyer? Eric Raymond's wife is a lawyer, I believe =). From mhammond at skippinet.com.au Thu Sep 16 06:28:19 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 16 Sep 1999 14:28:19 +1000 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <000101befff4$ecc9bde0$712d153f@tim> Message-ID: <004e01befffb$e83a6d90$0801a8c0@bobcat> > Don't feel picked on, M-A! Opposition is normal and healthy; As a (hopefully welcome) tangent to this discussion, I am currently reading "Dynamics of Software Development" by Jim McCarty. He has an interesting section on exactly this. I will quote it verbatim (typing it in, so forgive typos). I present it without comment other than: It appears we are a healthy team! Mark. The Problem of Creativity: Creativity in a group is always limited by the groups defensiveness, and some amount of defensiveness is healthy. In the healthy team, then, no matter how highly valued creativity is ostensibly, change beyond a certain normative quantity or degree will be resisted. And even change within bounds will be accepted only insofar as it can seem to fall under the rubric of "improvement". Change must seem to build on the previously known and accepted reality. Even if there are only two steps involved in a change, with the second building on the first, in a single reality cycle - single product development cycle for us - the change is likely to be rejected by the healthy team. What you need for truly creative change, then, is an environment that transcends good health, an environment that not only accepts a continuum of change, which is normal, but one that positively engenders, nurtures and propels forward wholly new dynamics. The transcendent organization values radical or revolutionary change and esteems utterly new modes of thought. It's possible for a team to be healthy and not particularly creative, but this state of affairs is not especially desirable. What is desirable is team fecundity [MH - had to look this up - "intellectually productive or inventive to a marked degree" - synonym of fertile], the radiating of the new and the original from the normal and the healthy. This kind of creativity requires a flexibility and a courage beyond the reach of most of us most of the time It's ironic and worth remembering that the healthier the team the more effective the resistance to revolutionary thought will be. This is a natural consequence of the teams overall effectiveness on every relevant domain. -- end of quote-- From jcw at equi4.com Thu Sep 16 11:48:33 1999 From: jcw at equi4.com (Jean-Claude Wippler) Date: Thu, 16 Sep 1999 11:48:33 +0200 Subject: [Python-Dev] License cleanup References: <199909151843.OAA18364@eric.cnri.reston.va.us> Message-ID: <37E0BCF1.24FD7431@equi4.com> Guido van Rossum wrote: > I hereby withdraw the posted license. There still is the need for a > new license, but we need to go back to the drawing board for it. One more consideration: some people may compare the "scariness" of the Python license against, say, the Tcl license - and choose accordingly. It's not even about content: seeing that new license, or the Perl licenses for that matter, it sends out a strong message IMO: you are entering the world of lawyers. Proceed with caution. Guard dogs. -- Jean-Claude From mal at lemburg.com Thu Sep 16 12:44:34 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 16 Sep 1999 12:44:34 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> Message-ID: <37E0CA12.1E24EBF2@lemburg.com> Guido van Rossum wrote: > > I don't like the pressure to allow relative imports and I don't like > the solutions that those who insist on having it propose (they are all > writing their own import hooks). I'm wasn't planning to write my own hook until now. I need this feature to be able to organize my package internal stuff swiftly and easily, plus to be able to possibly put it under new top-level packages. Obviously I seem to be more or less alone with this need, so I'll stop argueing for inclusion of "something like relative imports" in the distribution. > The reasoning seems to be something like this: > > (a) If my package needs non-standard package X, I can't require that X > be installed as a top-level package, because there might be another > unrelated package also called X; therefore I need to install a copy of > package X as a subpackage of my own package. > > (b) Because any non-standard package X now may be installed at the > toplevel or as a subpackage of some other package, X must use relative > imports exclusively to access subparts of itself. > > I find (b) unacceptable for a number of reasons, that have all been > explained by others: duplicate installs (and hence possible imports) > of the same package as a subpackage of various other packages, the > requirement to be religious in the use of relative imports, the > ugliness of the __ notation (I do *not* look forward to explaining > that to 12-year-olds in the context of CP4E). (b) is (if at all) a problem only to be taken into account by the author of package X. He may or may not use rel. imports. A 12-year old probably won't (but then: you never know ;). BTW, what is this CP4E thing you're talking about. If it's an syntax aware editor, I have a friend who is really interested in these things... could you send me an URL that I can send him ? > I think that the fear expressed in (a) is an overreaction to some > recent actual top-level name clashes. This was a real pain, but > notice that it happened because several popular module collections > weren't packagized at all! > > In the Java world, the right solution exist: a better top-level naming > scheme. I forget what the exact rules are, but I seem to recall that > Java has something of the same relative import rules af Python > currently has: you can import a class from the same leaf package by > its simple name, or you can import a class by its full name. As Tim > explains, the occasional renames are easily handled with a global > substitute in a smart editor. > > I don't want to do the same thing as Java and require everybody to use > their reversed domain name as their package name. Of course you are > free to do so: there are currently no toplevel modules or packages > that I know of named org, com, edu, gov, mil or net. Two letter > domains are all free except for re=Reunion (and ni=Nicaragua, but ni > is expired). Oops, mx=Mexico! Oh well... I guess they'll have to use 'mex' if they decide to go the reverse domain way ;-) > Well, Python package semantics are > slightly different than Java semantics in that shared ownership of a > subpackage of (e.g. :-) mx is harder in Python than in Java (Python > requires that they are installed in the same directory; Java doesn't). True, perhaps we should lighten this requirement a little when we recode the import mechanism in Python ? E.g. if a local import fails continue the search with the fully qualified name and only if that fails, restart using the local name. This would need some kind of fastpath cache to make the search reasonably fast though. > In any case the proper thing to do is to pick a good unique toplevel > name. Aaron Watters did a great job with kj (not a country code > either!); Marc Andre also has the right spirit with mx. CNRI will > probably use cnri as the toplevel package name for CNRI-developed > packages that aren't distributed together with Python. > > If there are still worries about name choices, I volunteer to run a > toplevel name registry. This has worked well e.g. for Apple where > application developers had to pick a unique 4-letter code; certainly > better than the Windows world where conflicting 3-letter extension > definitions are common. In the Python world, I would recommend longer > toplevel names, so it's more likely that you can use your company name > (e.g. pythonware). > > Note that I still resist the urge to start renaming all the existing > standard modules -- too much breakage. This will happen in 2.0. Wouldn't it suffice to just put them into one package, e.g. 'python.' ? BTW, as Tim argued: the breakage an easily be leveraged by using a smart editor... ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 106 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Thu Sep 16 12:56:01 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 16 Sep 1999 12:56:01 +0200 Subject: [Python-Dev] Relative Package Imports References: <000101befff4$ecc9bde0$712d153f@tim> Message-ID: <37E0CCC1.6EE4E599@lemburg.com> Tim Peters wrote: > > Sure is a lot of traffic on this -- can't we go back to hating the new > license ? The license was withdrawn... I would prefer starting to talk about imputil.py. > [MAL] > > Also, I don't know how "explicitly, long-windedly and straightforwardly" > > writing module imports is any different from using relative imports. > > Hugely and profoundly different, as your example shows: > > > An example: > > > > mx > > .DateTime > > .ODBC.Windows > > > > In ODBC.Windows I would write: > > > > import __.__.DateTime > > > > which is just as explicit as writing > > > > import mx.DateTime > > > > except that the information about the top-level hierarchy is *not* > > included in the import information. > > Exactly: now instead of a bulletproof absolute import in a self-contained > importer, whose precise meaning is clear from casual static inspection, Ah, that's a new idea ! Package local absolute names... wow, that sounds weird. > you've got a relative one whose meaning depends vitally on where the > importer happens to live. As code moves around, the import may fail to work > when you're lucky, or pick up an unintended DateTime package when you're > not. The semantics of the import are indeed relative now: you see the > opportunity to type three additional characters as a strength, and I > see nothing but headaches masquerading as convenience. Tim, the intention was being able to move the mx level around, e.g. renaming it, eliminating it by sticking DateTime and ODBC under some other higher level package (yes, you'll have to watch out here because the higher level package might define DateTime and ODBC too -- but it's not all that likely), etc. The buzz word here is 'relocatable'. > As further consequence, simple "search-and-replace" strategies are rendered > at best useless. That is, a simple grep can no longer find all & only > references to mx's flavor of DateTime: it picks up all references to all > DateTime guys, with no way short of a structure-aware tree crawler to sort > them out again. Complexity spreads. > > Of course, in the end, I lose no matter what. If *everyone* used dirt-dumb > fully-qualified imports, all hassles with shifting package structure could > be solved with a text editor in seconds. But that's not whiz-bang enough, > so people *will* play endlessly dubious tricks with hooks, paths, relative > imports, and metapackages too once Gordon gets around to them . > Having an explicit way to spell relative imports would make my life easier > because at least those are statically recognizable for what they are. I > don't want them and won't use them, but it could ease the pain others will > cause by using them. That's not enough to make me in favor of them, though; > just enough so I won't pout if they do go in. > > > Anyway, I'm not too inclined continuing this discussion anymore. > > I would never have thought that such a nifty little patch would > > have stirred up so much negative response. I'll just hack up my > > very own import mechanism using imputil and be done. So there! > > Don't feel picked on, M-A! Opposition is normal and healthy; and the last > idea I got anywhere with was list.pop(). Yah well. Last thing I succeeded with was making Python methods wrap any callable type. Seems that we're a bit too healthy (see MarkH's post) sometimes, i.e. there isn't all that much room for experiments. Just think of cool developments like Chris' stackless python. Experience shows that these kind of things will never make it into the distribution. Unfortunately, maintaing patches to the dist across releases a real pain and much work, so these ideas will just sit there unused and untested. Much the same happened to gcc ... in the end corporate strength made egcs possible. Perhaps we need such a branch too ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 106 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein at lyra.org Thu Sep 16 13:00:50 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 04:00:50 -0700 Subject: [Python-Dev] Relative Package Imports References: <000101befff4$ecc9bde0$712d153f@tim> <37E0CCC1.6EE4E599@lemburg.com> Message-ID: <37E0CDE2.56A6C6E4@lyra.org> M.-A. Lemburg wrote: > ... > Seems that we're a bit too healthy (see MarkH's post) sometimes, > i.e. there isn't all that much room for experiments. Just think > of cool developments like Chris' stackless python. Experience shows > that these kind of things will never make it into the distribution. A portion of my threading patches went into 1.5. "Big" things can happen and they do... it's just a matter of passing muster with The Head Honcho. That, and the timing (e.g. the thread state changes wouldn't go into a 1.4.n release, but did go into 1.5). > Unfortunately, maintaing patches to the dist across releases a real > pain and much work, so these ideas will just sit there unused and > untested. Much the same happened to gcc ... in the end corporate > strength made egcs possible. Perhaps we need such a branch too ? NO -g -- Greg Stein, http://www.lyra.org/ From fredrik at pythonware.com Thu Sep 16 13:32:35 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Sep 1999 13:32:35 +0200 Subject: [Python-Dev] CP4E References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> Message-ID: <004101bf0037$2dca30d0$f29b12c2@secret.pythonware.com> > BTW, what is this CP4E thing you're talking about. http://www.python.org/doc/essays/cp4e.html also see: http://www.deja.com/=dnc/thread/522068184 (on second thought, don't). > If it's an syntax aware editor, I have a friend who is really > interested in these things... aggressively python-aware editors are fun. want braces? sure, just change the style sheet ;-) From guido at CNRI.Reston.VA.US Thu Sep 16 13:49:44 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 07:49:44 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Thu, 16 Sep 1999 12:44:34 +0200." <37E0CA12.1E24EBF2@lemburg.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> Message-ID: <199909161149.HAA19650@eric.cnri.reston.va.us> > I'm wasn't planning to write my own hook until now. I need this > feature to be able to organize my package internal stuff swiftly > and easily, plus to be able to possibly put it under new top-level > packages. Obviously I seem to be more or less alone with this need, so > I'll stop argueing for inclusion of "something like relative imports" > in the distribution. I still hope against hoping to convince you not to do this. I think this adds lots of complexity for the reader of your package. I think of *all* software that I write as something that eventually someone is going to read and try to understand. I also think of it as something that eventually someone is going to port to a platform that wasn't even designed when I wrote the software. Tricks like depending on a custom import hook make reading a pain. For example, the reader who is just starting to browse a large package often needs to follow the thread of execution from one module to the next. If the import statements contain magic like __magic__.foo.bar (actually proposed magic renamed for generality :-), this reader will first need to understand the custom importer -- which is likely one of the hariest components of the package. The same thing applies with even more force to tools like package browsers. IDLE for example has a class browser which displays the true module and class name for base classes. E.g. if you have code that says from Tkinter import Widget class MyWidget(Widget): ... it correctly deduces that MyWidget's base class is Tkinter.Widget. (And in a future version it will probably allow you to click through to that class, show you its methods, etc.) Custom importers break this feature, and thus make the code harder to analyze for the reader. (Not all custom importers are bad; there are plenty of reasons to augment the standard importer. But, while custom importers make different interpretations of the namespace possible, I frown upon that particular use.) > (b) is (if at all) a problem only to be taken into account by > the author of package X. He may or may not use rel. imports. > A 12-year old probably won't (but then: you never know ;). Which severely limits your ability to do what you want with packages you didn't write. I know that this sounds politically incorrect in a radical free software world, but often in the end I find it more convenient to conform to the rest of the world and "fit in" than to be different in every little way. Note that my hair isn't blue :-); I've also replaced my "signature" glasses with a more conventional model. > BTW, what is this CP4E thing you're talking about. If it's an > syntax aware editor, I have a friend who is really interested > in these things... could you send me an URL that I can send him ? Where have you been? It's on the python.org homepage, has been discussed in c.l.py, c.l.tcl, even c.l.ada, in lwn, and on /.! The syntax aware editor (a proposed super-version of IDLE) is only a small part of it. See python.org/doc/essays/cp4e.html > Oh well... I guess they'll have to use 'mex' if they decide to go > the reverse domain way ;-) Or, in a worse-case scenario, the first Mexican developers using the reverse domain will probably not be aware of the mx toplevel package, and by the time their software hits the street it will be too late. > True, perhaps we should lighten this requirement a little when we > recode the import mechanism in Python ? E.g. if a local import > fails continue the search with the fully qualified name and only > if that fails, restart using the local name. This would need some > kind of fastpath cache to make the search reasonably fast though. Now this is something that I would consider. > Wouldn't it suffice to just put them into one package, e.g. > 'python.' ? And somehow make all user code live implicitly inside that package? I don't see how that solves anything. > BTW, as Tim argued: the breakage an easily be leveraged by using > a smart editor... ;-) What breakage? I think __ is a bit too hard to fix easily with a smart editor. And believe me, at some point *someone* is going to need to rip out your custom importer because it interferes with *his* custom importer, and he'll choose to replace all your __ imports with absolute package names, rather than trying to add the __ feature to his importer. --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Thu Sep 16 13:57:20 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 04:57:20 -0700 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <37E0DB20.20C72337@lyra.org> Guido van Rossum wrote: >... > What breakage? I think __ is a bit too hard to fix easily with a > smart editor. And believe me, at some point *someone* is going to > need to rip out your custom importer because it interferes with *his* > custom importer, and he'll choose to replace all your __ imports with > absolute package names, rather than trying to add the __ feature to > his importer. This is an aspect of Python's singular import hook. imputil is designed to allow for multiple Importers to be present and to work in harmony. As long as each Importer can identify that a particular import is part of its namespace (e.g. HTTPImporter recognizes _http_... as one of its own), then no conflicts will arise and everything will be hunky-dory. btw, it might be nice to allow a string to the import statement. I'd much rather say: import "http://www.lyra.org/greg/python/qp_xml.py" Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Thu Sep 16 14:00:06 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 05:00:06 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> Message-ID: <37E0DBC6.12D3EB06@lyra.org> Gordon McMillan wrote: >... > def _reload_hook(self, module): > # gmcm - Hmmm, reloading of a module may or may not > be impossible, > # (depending on the importer), but at least we can > # look to see if it's ours to reload: > if hasattr(module, '__importer__'): > if getattr(module, '__importer__') == self: > raise SystemError, "reload not yet implemented" > return self.__chain_reload(module) I've folded this in (finally). New imputil.py to be published in a bit... thx! -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Thu Sep 16 14:45:39 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 08:45:39 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274695692-31669872@hypernet.com> <37E0052F.A899E985@digicool.com> Message-ID: <37E0E673.29A844BF@interet.com> Jim Fulton wrote: > > Gordon McMillan wrote: > > > > Jim Fulton wrote: > > > [Barry] > > > > A > > > > user would drop the kjParsing directory into Shared, and then > > > > all the import kjParsing.kjParser statements would Just Work. > > > > To be a standalone application, you can use tricks that Jim > > Ahlstrom pointed out, or tricks from my Installer. The major > > point being that sys.path has nothing to do with the > > PYTHONPATH environment variable. You now have an > > installation where you can be assured of no collisions, > > because you control it completely. JimF, I am still trying to understand this. Gordon, Barry and I think a good solution is to put "Shared" on a private sys.path, but you object to this and I don't understand. Specifically zope/ zope/Shared zope/Shared/mx/... zope/Shared/AnotherPackage/... zope/Shared/some_file.py zope/ZopePackage1/... zope/ZopePackage2/... zope/bin/python.exe Suppose zope users must use the special binary zope/bin/python.exe. Suppose further that this binary reliably sets its sys.path to the directories it needs, namely zope, zope/Package*, zope/Shared. Suppose further that all user's packages must go into zope/Shared either as files or package directories and so are at the top level as package imports have always assumed. Then it seems that your zope app would absolutely work, and no other Python installation could damage it by changing PYTHONPATH, and that your zope installation would not damage any other Python installation either. Is the reason this doesn't work the posibility of name collisions in zope/Shared? I thought before that you were worried about name collisions in another co-existing Python installation which may have unknown packages installed, conflicting packages, or packages with incompatible versions. If so, you are trying to solve a problem more severe than I thought. I have always assumed packages would have unique names and so a flat name space suffices. In particular, trying to design for conflicting names can not be supported by the current python *.pyl library file design, and in general would require a PYTHONPATH mechanism. Such PYTHONPATH mechanism would have to be "local" to packages. That is, PYTHONPATH would vary depending on the package doing the importing. Are you sure this problem needs a solution right now? Jim Ahlstrom From jim at interet.com Thu Sep 16 14:57:46 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 08:57:46 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> Message-ID: <37E0E94A.A2F0D37F@interet.com> Guido van Rossum wrote: > > Even worse, it means that exceptions.py and site.py can not > > be found at all except using the normal PYTHONPATH, and > > putting their path in Spam_path will *not* work. > > Why would you want your own exceptions.py and site.py? I don't. I never change Python library files. I am worried that they won't be found because I don't trust PYTHONPATH. > > Oh dear, I think I heard no instead of yes. Are you saying that if > > someone else installs a Python app on my customer's machine after I do, > > and sets a registry entry which sayes to use c:/other/path/to/site.py > > for site.py (as he may very well want to do), then if my Python program > > depends on getting my copy of site.py from my directory, it will then > > use the other copy instead and may very well fail? > > Again - why would anyone register their own site.py? I wouldn't, I am worried that someone else will break my installation. Remember that site.py was invented as a site-specific module, although that function moved to sitecustomize.py. > I presume using GetModuleFileName()? Please send me the patch! Yes, and OK. > > JimA's conjecture: It is currently impossible to > > ship a Python app which can not be damaged by the installation of a > > second Python app without using a hacked custom binary. > > Sounds right. All tricks to make the app unique require using a > different registry key, which requires a change to the DLL. However, > you can do this without recompiling! The version string is used is > embedded in a resource, so you can patch it using some kind of > resource editor. Mark Hammond planned it this way! I don't understand this. Is there documentation? Jim Ahlstrom From guido at CNRI.Reston.VA.US Thu Sep 16 15:49:16 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 09:49:16 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Thu, 16 Sep 1999 08:57:46 EDT." <37E0E94A.A2F0D37F@interet.com> References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> Message-ID: <199909161349.JAA19789@eric.cnri.reston.va.us> [me] > > Why would you want your own exceptions.py and site.py? [JimA] > I don't. I never change Python library files. I am worried > that they won't be found because I don't trust PYTHONPATH. Hmm... PYTHONPATH gets inserted in front of the default sys.path. (Many moons ago that was different. But it has been like this for a loooooong time.) So are you worried that someone put a *different* exceptions.py or site.py on their path? > > Again - why would anyone register their own site.py? > > I wouldn't, I am worried that someone else will break my installation. > Remember that site.py was invented as a site-specific module, although > that function moved to sitecustomize.py. Hm, I dug out the oldest site.py I have (used in Python 1.4), and it doesn't encourage editing it at all -- it tells you to use sitecustomize.py. I guess they could break your installation anyway, but only by messing with the general Python installation. > > Sounds right. All tricks to make the app unique require using a > > different registry key, which requires a change to the DLL. However, > > you can do this without recompiling! The version string is used is > > embedded in a resource, so you can patch it using some kind of > > resource editor. Mark Hammond planned it this way! > > I don't understand this. Is there documentation? The usual :-) Python/import.c shows that import calls PyWin_FindRegisteredModule() to find a registered module before looking in sys.path (but after checking for builtin and frozen modules). PC/import_nt.c shows that PyWin_FindRegisteredModule() uses a registry key of the form "Software\Python\PythonCore\\Modules\" where is the module name, is empty or "\Debug" depending on whether we are compiled with _DEBUG define. The resource value points to a file (either .py, .pyc/.pyo, .pyd or .dll; in fact any of the prefixes returned by imp.get_suffixes()). PC/dl_nt.c shows that PyWin_DLLVersionString is set to string 1000 loaded from the string resource table. PC/python_nt.rc shows that there's a stringtable with item 1000 being the MS_DLL_ID string, set to "1.5" in that file. Note that this value (PyWin_DLLVersionString) is also to Python code as sys.winver. I hope that Mark Hammond can point you to a tool that you can use to edit a string resource in an executable or DLL. --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Thu Sep 16 15:46:19 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 06:46:19 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <37E0F4AB.38EAB65F@lyra.org> M.-A. Lemburg wrote: >... > The original version of imputil I fetched from Greg's page > did work out of the box (from...import... hassles) and "did not work", I presume. From my original testing, I thought from...import worked. With more testing, I found that something of the form "from xml.dom import builder" did not work. I discovered why it failed (xml.dom was imported by Importer instance I1 but I2 thought it could handle the from...import, and this barfed a check). I've fixed this by delegating to the proper importer (I1 in my example) to complete the import. Your solution to check the __importer__ variable in the globals is probably incorrect. If I read/eval it correctly, that would mean that a module imported by IMP1 could not use modules imported by IMP2. In other words, a package module could not import a top-level module defined by a different importer. (note also that your globals.get() could fail if globals is None) > obviously did not support in-package shared libs. I've added I did not fold this in. Your change isn't "in the spirit" of the Importer mechanism. The "Right Way" to do this is to create a BuiltinImporter and add that to the chain of importers. The DirectoryImporter should only import from directories -- no reason for it to know about builtin stuff. As a result, I did not accept the new methods on Importer for handling builtins/special modules -- those would go in the BuiltinImporter. [BuiltinImporter should be written and included in imputil.py; I don't really have the time at the moment to write the thing... 7am and time for sleep...] However, your change here did raise a very important design issue: get_code() needs to be able to return a loaded module, rather than just a code object. I've folded in your patches for that. I also folded in many of your extended doc/comments (at least in concept; not necessarily verbatim). You and Gordon are recognized in the header now, and I've added a "proper" author notice and licensing (public domain). I did not include the "misses" feature that you added to the DirectoryImporter. I would hate to see a miss-cache get loaded, a module dropped into the filesystem, and the user never being able to import the thing. I didn't fold in your indentation changes or name changes. I liked mine :-). The __main__ thing at the bottom didn't make much sense to me, though, since the call to _test_dir() followed by an exit doesn't really do anything. And yes, I recognize that you can use "python -i imputil.py" but I'd rather just see "python" followed by "import imputil ; imputil._test_dir()". Of course, please feel free to generate a new patch if I've missed something (thinking about it, I missed the OSError thing). > both features so that the test script in DateTime can run > successfully. > > Things that remain are: > ? the win32 registry stuff (needs C code) And a new Importer to use it. > ? the Mac fork stuff (needs C code) Ditto. > ? a working __path__ implementation (is anyone using this attribute > which only is available in packages ?) Per the private mail that I sent to you: I explicitly punted on the __path__ attribute. It can lead to *way* too much confusion. It is also unavailable for frozen packages (boy oh boy did the win32com get some ugliness in there to compensate for being frozen w.r.t. its use of __path__). The DirectoryImporter can insert the attribute, but it definitely wouldn't go into the Importer itself. The __path__ attribute is specific to loading from a filesystem, yet Importer is generic. > ? probably a whole bunch of other quirks > ? some speedups (there currently are too many stat()s) Yes. I recognize that the "misses" feature was intended to remedy this. I don't have an immediate answer to the stat() issue. Does the Importer mechanism actually perform more stats on an import than Python itself? (it looks like it does one for the isdir() plus two for fetching file timestamps) And a big thanx: I appreciate the patches to imputil! The new module is now available in its "official" location at http://www.lyra.org/greg/python/ Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Thu Sep 16 15:56:00 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 09:56:00 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> Message-ID: <37E0F6F0.8FC49D4@interet.com> Greg Stein wrote: > > Gordon McMillan wrote: > >... > > def _reload_hook(self, module): > > # gmcm - Hmmm, reloading of a module may or may not > > be impossible, > > # (depending on the importer), but at least we can > > # look to see if it's ours to reload: > > if hasattr(module, '__importer__'): > > if getattr(module, '__importer__') == self: > > raise SystemError, "reload not yet implemented" > > return self.__chain_reload(module) > > I've folded this in (finally). > > New imputil.py to be published in a bit... As a result of all this import discussion I am a bit worried that the python library *.pyl file format may not be powerful enough. I have always thought in terms of unique top-level names and a format which supports import of modules and packages. But this does not support the full functionality of PYTHONPATH. For example, PYTHONPATH can (and is) used to select the correct plat-* directory files. And the format may not support Jim Fulton's fancy local import scheme. And what if someone invents a third thing to import besides a module or a package? PYTHONPATH is not going away nor should it. How about if the *.pyl file format is exactly a directory structure? I mean that the table of contents is limited to paths starting with a directory name only, and that the seperator is '/' instead of '.'. So a listing would be identical to the output of 'ls -R'. So: Lib/string.pyc Lib/exceptions.pyc Lib/plat-sunos4/... mx/__init__.pyc mx/... package2/... dir3/... ... The implied PYTHONPATH for this file is ["Lib", "."]. Since the format is exactly a directory tree, it is guaranteed that whatever PYTHONPATH or imports can do now or in the future with a directory tree, it can still do it with a *.pyl file. Jim Ahlstrom From gstein at lyra.org Thu Sep 16 16:05:42 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 07:05:42 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> <37E0F6F0.8FC49D4@interet.com> Message-ID: <37E0F936.6D32E983@lyra.org> James C. Ahlstrom wrote: >... > As a result of all this import discussion I am a bit worried that > the python library *.pyl file format may not be powerful enough. Background for the readers: .pyl is an extension that I used in my "small" distribution. I think Gordon uses it, too. In any case, it is effectively a concatenation of .pyc files along with a TOC mapping fully-qualified dotted module names to seek-positions within the file. [speaking of stat() calls: using a .pyl eliminates them quite nicely -- this may be part of Gordon's observed speed increase when using an archive] The .pyl format was discussed a bit on the distutils-sig list and "sort of" accepted as an okay format for jamming a bunch of modules into a single file. [by "sort of", I mean that the small group who participated in the discussion were okay with it :-); it is a great, minimalist format, so it probably won't please people who like a ton of features in a file format :-) ] >... > How about if the *.pyl file format is exactly a directory structure? > I mean that the table of contents is limited to paths starting with > a directory name only, and that the seperator is '/' instead of '.'. > So a listing would be identical to the output of 'ls -R'. So: > Lib/string.pyc > Lib/exceptions.pyc > Lib/plat-sunos4/... > mx/__init__.pyc > mx/... > package2/... > dir3/... > ... > > The implied PYTHONPATH for this file is ["Lib", "."]. Since the > format is exactly a directory tree, it is guaranteed that whatever > PYTHONPATH or imports can do now or in the future with a directory > tree, it can still do it with a *.pyl file. People import things using a dotted name. Therefore, I think it makes the most sense to map that straight to the resulting .pyc file. No reason to put directories into the file... they make no sense to the end user. During construction of the .pyl, you would walk the tree finding all the available modules (and their corresponding dotted name) and insert them. Note that you can distribute multiple .pyl files. There could be the Python standard lib in one file, the mx package in another, etc. As a module is searched for, the system just peeks into each .pyl in turn, looking for the module. Search order is currently defined by order of install() on the Importer instances. I believe the Right Way to do things is to create sys.importers (as a list of Importers) and deprecate the sys.path variable. Python could start up with an Importer than simply scanned sys.path as a backwards compat measure; it could also leave sys.path empty and create DirectoryImporters for each path component (this could cause problems, though, for some apps that believe sys.path shouldn't be empty, or that use it for magic-munging). I've search the standard lib in the past -- there are only a couple real uses of sys.path if I remember rightly (test package and the traceback module). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Thu Sep 16 16:19:41 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 07:19:41 -0700 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> <199909161349.JAA19789@eric.cnri.reston.va.us> Message-ID: <37E0FC7D.7494B8F7@lyra.org> Guido van Rossum wrote: > > [me] > > > Why would you want your own exceptions.py and site.py? > [JimA] > > I don't. I never change Python library files. I am worried > > that they won't be found because I don't trust PYTHONPATH. > > Hmm... PYTHONPATH gets inserted in front of the default sys.path. > (Many moons ago that was different. But it has been like this for a > loooooong time.) So are you worried that someone put a *different* > exceptions.py or site.py on their path? When you jam every file into a module archive, you still have to leave these two "outside" so that Python can find them when starting up. The problem then breaks down into two parts: 1) locating them 2) ensuring they're the right versions In my "small" distro, I relied on the current-dir always being in the path, so I could always find these. The two files were distributed as part of distro. Anyhow: JimA is saying that he doesn't trust PYTHONPATH. Not so much bad files/versions, but that he won't be able to find them because PYTHONPATH has been monkeyed in some way. > > > Again - why would anyone register their own site.py? > > > > I wouldn't, I am worried that someone else will break my installation. > > Remember that site.py was invented as a site-specific module, although > > that function moved to sitecustomize.py. > > Hm, I dug out the oldest site.py I have (used in Python 1.4), and it > doesn't encourage editing it at all -- it tells you to use > sitecustomize.py. I guess they could break your installation anyway, > but only by messing with the general Python installation. If the file exists, then somebody can mess with it. JimA is trying to create a bulletproof app here. To do this, you can distribute a python.exe, exceptions.py, site.py, and an archive of your other modules. site.py is scrapped in favor of installing an Importer to access the archive (this implies you also distribute imputil.py). These five files are the exact five in my "small" distro. It's pretty cool... no need for registry changes and a very small "file count" footprint. Gordon took this basis and added a bunch of features for bundling an application in there. JimA has also been mixing in some frozen modules (I forget exactly why/what). [oh, my small distro doesn't ship a python15.dll, although it easily could] >... > I hope that Mark Hammond can point you to a tool that you can use to > edit a string resource in an executable or DLL. The win32api module has resource manipulation functions such as BeginUpdateResource, UpdateResource, and EndUpdateResource. Write a Python script to modify your version string :-) A demo of resource munging can be seen in /win32/scripts/VersionStamp/verstamp.py. Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at digicool.com Thu Sep 16 16:56:53 1999 From: jim at digicool.com (Jim Fulton) Date: Thu, 16 Sep 1999 10:56:53 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <37E10535.CA96A77E@digicool.com> "M.-A. Lemburg" wrote: > > ? a working __path__ implementation (is anyone using this attribute > which only is available in packages ?) Yes. I use it for two things: - I modify it to allow a (logical) package to be spread over multiple physical locations. (In Zope, products can be installed in the Zope installation area or in Zope "instance" homes. - I use it to determine the location(s) of a package. Our packages usually contain many files, such as DTML source files, images, data files, etc., that are not Python modules. We have standard utilities for getting at these files in packages. This is extremely useful. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at interet.com Thu Sep 16 16:58:48 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 10:58:48 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> <199909161349.JAA19789@eric.cnri.reston.va.us> Message-ID: <37E105A8.A2B06A80@interet.com> Guido van Rossum wrote: > Hmm... PYTHONPATH gets inserted in front of the default sys.path. > (Many moons ago that was different. But it has been like this for a > loooooong time.) So are you worried that someone put a *different* > exceptions.py or site.py on their path? Yes, and/or (2) added a sitecustomize.py with their special import hook as has been proposed here over and over, or (3) PYTHONPATH is screwed up and doesn't find anything. Perhaps this is a prejudice of mine. I just look at "print sys.path" and marvel at what I don't understand. At least I can see it is not simple. I hate relying on thing that are not simple. And I hate custom import hooks. Unless they are mine of course ;-) > Hm, I dug out the oldest site.py I have (used in Python 1.4), and it > doesn't encourage editing it at all -- it tells you to use > sitecustomize.py. I guess they could break your installation anyway, > but only by messing with the general Python installation. Just adding sitecustomize.py would do it. And this is encouraged. I think Gordon put his finger on the issue. Either try to co-exist with other installed Python software and take the risk that everyone is playing by the rules, or build your own black-box self-contained Python world and duplicate storage. > The usual :-) > [Documentation goes here...] Thanks. This is very useful. But it doesn't help, perhaps as a result of more of my prejudices. This registry entry is meant to be used by a by-the-rules shared Python installation, so I must not change it. And I hate changing any registry entries at all. My commercial software keeps all its settings in a regular .ini file in its install directory, and makes only minimal and required registry entries. IMHO the Windows registry is a software catastrophe which ranks right up there with JCL (anyone else here old enough to remember that?). Anyone who doesn't agree should go with me to our money center banking clients, and sit there while they grill you on every registry entry and why it is required. Money center banking clients do not like their registry messed with. I do however see your point that I could change the version string to something non-standard and use the registry to control imports. I will think about this further. Maybe it would work. My current "solution" is to use freeze to create a black-box install, and worry about second Python installations and wasted storage when it happens. I was hoping that this thread whould result in a consensis of what to do, but it has not. So now I am hoping that Python library (jar) files will turn out to be a practical solution, so I am pestering Greg and Gordon. We'll see. Jim Ahlstrom From bwarsaw at cnri.reston.va.us Thu Sep 16 17:16:42 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 16 Sep 1999 11:16:42 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> Message-ID: <14305.2522.922445.932104@anthem.cnri.reston.va.us> >>>>> "JF" == Jim Fulton writes: >> I think we're safe from this for the forseeable future through >> convention and peer pressure. JF> 8^o JF> (I feel like I should respond to this in some way but .... JF> I just can't understand this kind of argument and don't JF> know how to frame a response.) Maybe I should have included half a smiley. Python may eventually need package naming rules as strict as Java's. I think it's a good idea now for those who are writing packages to consider trying to pick unique top-level package names. We might decide to release our stuff under the `cnri' top level package. That's probably unique enough. Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! Personally, I think `Mailman' as a top-level package name is pretty unique, so I don't plan to change what I'm doing there. -Barry From bwarsaw at cnri.reston.va.us Thu Sep 16 17:37:01 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 16 Sep 1999 11:37:01 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> <37E049CD.6B64DE00@lyra.org> Message-ID: <14305.3741.741370.948251@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Definitely having an OSI certification will be great (cool GS> stuff on the cert for the existing license!). Having Bruce GS> Perens review the license would also be a great boon (see GS> www.perens.com for some of his writings; also see GS> http://perens.com/Termination.html specifically). Interesting article, but IBM's termination clause was different than the JPython one. I fought hard on para7 because IIRC, RMS complained that an earlier version /could/ have been used to arbitrarily terminate. I think the current JPython para7 is better because /you/ have to materially breach, which seems like a much higher threshold. But it still may not be perfect. Aside: don't necessarily think I'm a grinning fan and defender of the JPython license. It's a huge win over what we had before, and I think it's good enough that nearly anybody who wants to do Real Things with JPython, now can. I've had only one question about the license since it was published and that was about the "displayed prominently" clause (i.e. was it okay to include the alternative handle text in an "about" menu pulldown? That seemed prominent enough to my nonlawyerly brain.) I'm glad to see the Python community push hard for the "other side's viewpoint" with reasoned and rational arguments. I think that such responses from Influential Python Users will provide us with useful ammunition when we re-evaluate the licenses. It means that ultimately we'll have the right license for Python (and JPython). Thanks, -Barry From skip at mojam.com Thu Sep 16 17:54:39 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 10:54:39 -0500 (CDT) Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37E0F6F0.8FC49D4@interet.com> References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> <37E0F6F0.8FC49D4@interet.com> Message-ID: <14305.4799.328120.623217@dolphin.mojam.com> Jim> As a result of all this import discussion I am a bit worried that Jim> the python library *.pyl file format may not be powerful enough. Not to rain on anyone's parade, but I want to remind the folks having this discussion that there are people reading this thread that while fairly well versed in Python have little idea what anyone is talking about anymore. (I don't know. Maybe I'm the only one.) python-dev is clearly the best place to discuss this in the short-term (anyone for an import SIG?), but whatever is implemented will have to be understood by lots of people on c.l.py to be of broad applicability. Perhaps I'm way off base and there are more than a handful of people who will ever run into the problems being solved here, but if we can partition the Python programming community into the package wizards and the mere import mortals, I worry that the potions concocted by the wizards will send a few of us import mortals to the hospital... The Java package scheme, while odious to some perhaps, is extremely easy to understand for anyone who's ever used Windows Explorer or execute "ls -R". just-a-cautionary-peanut-thrown-in-from-the-bleachers-ly y'rs Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From skip at mojam.com Thu Sep 16 19:01:58 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 12:01:58 -0500 (CDT) Subject: [Python-Dev] Relative Package Imports In-Reply-To: <14305.2522.922445.932104@anthem.cnri.reston.va.us> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> <14305.2522.922445.932104@anthem.cnri.reston.va.us> Message-ID: <14305.8838.329883.696911@dolphin.mojam.com> BAW> Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! Umm, that would be "Knowbots Over Everything" or "Knowledge Of Everyone"? ;-) Skip From guido at CNRI.Reston.VA.US Thu Sep 16 19:04:37 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 13:04:37 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Thu, 16 Sep 1999 12:01:58 CDT." <14305.8838.329883.696911@dolphin.mojam.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> <14305.2522.922445.932104@anthem.cnri.reston.va.us> <14305.8838.329883.696911@dolphin.mojam.com> Message-ID: <199909161704.NAA01253@eric.cnri.reston.va.us> > BAW> Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! > Skip> Umm, that would be "Knowbots Over Everything" or "Knowledge Of Everyone"? Actually, "koe" is dutch for cow. I like the idea of cows over everything -- reminds me of the flying pig on an old Pink Floyd cover :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Thu Sep 16 19:27:49 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 10:27:49 -0700 Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> <37E049CD.6B64DE00@lyra.org> <14305.3741.741370.948251@anthem.cnri.reston.va.us> Message-ID: <37E12895.443CC6F3@lyra.org> Barry A. Warsaw wrote: >... > Interesting article, but IBM's termination clause was different than > the JPython one. I fought hard on para7 because IIRC, RMS complained > that an earlier version /could/ have been used to arbitrarily > terminate. I think the current JPython para7 is better because /you/ > have to materially breach, which seems like a much higher threshold. > But it still may not be perfect. Yes, I was aware that it was a reactive termination, rather than arbitrary. That makes it quite acceptable, but it still isn't a desirable thing. Especially given some of the grey area in the license ("are we sure we aren't in breach of the license?"). Personally, I'd rather see a license without a termination clause. If it must be there, then I'd like to see it as tight as possible (see the IBM and Apple licenses: IIRC, they only kick in when the user initiates patent litigation against IBM/Apple; the termination cuts them off as an initial response to the suit). The other elements I raised actually caused me more anxiety than the termination. If CNRI finds it acceptable, I'd recommend they use an existing OSD license. They get immediate certfication and, more importantly, a builtin awareness in the open source community of what the license really means. Each time a new license arrives in the community, bunches of people have to go an figure it out; if the new license is the IBM Public License with a search/replace on the company and product name, then people go "oh. all righty. no problem." and move on to doing real stuff. Dang. I keep replying to this stuff. :-) I'm hoping that we wrap this up pending a new release. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gmcm at hypernet.com Thu Sep 16 19:38:03 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37E0F6F0.8FC49D4@interet.com> Message-ID: <1274619156-36273405@hypernet.com> Jim Ahlstrom wrote: > As a result of all this import discussion I am a bit worried that > the python library *.pyl file format may not be powerful enough. > I have always thought in terms of unique top-level names and a > format which supports import of modules and packages. But this > does not support the full functionality of PYTHONPATH. For > example, PYTHONPATH can (and is) used to select the correct > plat-* directory files. And the format may not support Jim > Fulton's fancy local import scheme. And what if someone invents > a third thing to import besides a module or a package? > PYTHONPATH is not going away nor should it. The central idea of imputil is that an importer is responsible for one little chunk of turf. If the desired module / package isn't "his", he just passes the request on to the next element in the chain. So I don't think there's a need for one canonical do-everything importer (or archive format). PYTHONPATH is outside any particular importer. Effectively, you can use a chain of importers to replace PYTHONPATH. So the platform specific modules might be found by one particular importer. In other words, I think it's more effective to specialize individual importers and chain them up than it is to try to create an overly-generalized importer. - Gordon From gmcm at hypernet.com Thu Sep 16 19:38:03 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E0FC7D.7494B8F7@lyra.org> Message-ID: <1274619162-36273021@hypernet.com> [Greg, replying to Guido's confusion on Jim's interest in site.py and exceptions.py] > When you jam every file into a module archive, you still have to > leave these two "outside" so that Python can find them when > starting up. In the soon-to-be-published beta version of my installer, I've got that down to exceptions.py. Background: my (Win32) installer has, as a stated goal, the ability to create quasi-frozen Python apps which won't interfere with (or be influenced by) existing Python installations (if any). And it doesn't require the programmer/user to have a compiler. Thanks to patches given me by Thomas Heller, my python.exe replacement is now a (minimal) embedding app, and I do the same things that Greg does in site.py directly from C code, (and turn off the SiteFlag, too). I believe that if I freeze in exceptions.py and tweak the resource in python15.dll (which is just the stock python15.dll), I can have a completely safe executable. I think the same techniques can be applied on *nix, (although I'm pretty sure programmer/users won't be able to get away without a compiler). This gives me a strong interest in import hooks for two distinct reasons: - I rely completely on Greg's imputil to make this work. - I rely on freeze's modulefinder to help build these things. Bizarre import hooks in a normal Python installation will fool modulefinder. Take a look at what Pmw does (fortunately, Pmw comes with it's own packager). So I'd like to see import hooks follow some sort of pattern that can be followed by a tool like modulefinder. I also want the hooks. Right now I use imputil with archives, but wouldn't it be cool if you could add another imputil importer that checks for more recent versions at some home site on the web and automatically updates the installation? Summary: I'm very interested in seeing import and import hooks get rationalized, and I think Greg's stuff goes a long, long way towards that goal. - Gordon From gmcm at hypernet.com Thu Sep 16 19:38:03 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E105A8.A2B06A80@interet.com> Message-ID: <1274619161-36273101@hypernet.com> James C. Ahlstrom wrote: [Guido explains Windows registry usage] > Thanks. This is very useful. > > But it doesn't help, perhaps as a result of more of my > prejudices. This registry entry is meant to be used by a > by-the-rules shared Python installation, so I must not change it. Without trying it, I doubt you have to. It looks like you could set the resource to something that won't be found in the registry, and then just use normal Python mechanisms. > ... IMHO the Windows registry is a software catastrophe > which ranks right up there with JCL (anyone else here old enough > to remember that?). Ah. In a rush to get it down to ops, tripping at the head of the stairs and sending the card deck flying... Those were the days. - Gordon From skip at mojam.com Thu Sep 16 19:49:12 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 12:49:12 -0500 (CDT) Subject: [Python-Dev] Re: Path hacking In-Reply-To: <1274619162-36273021@hypernet.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> Message-ID: <14305.11672.687570.290221@dolphin.mojam.com> Gordon> In the soon-to-be-published beta version of my installer, I've Gordon> got that down to exceptions.py. Why not just run exceptions.py through Python2C, visually and experimentally verify that it works, then ship an exceptions.c as an optional module? People wanting to ship self-contained packages could then toss exceptions.py and build the C version of the exceptions module. Greg, is there anything in exceptions.py Python2C couldn't handle? Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From gstein at lyra.org Thu Sep 16 19:47:56 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 10:47:56 -0700 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <37E12D4C.2394D311@lyra.org> Skip Montanaro wrote: > > Gordon> In the soon-to-be-published beta version of my installer, I've > Gordon> got that down to exceptions.py. > > Why not just run exceptions.py through Python2C, visually and experimentally > verify that it works, then ship an exceptions.c as an optional module? > People wanting to ship self-contained packages could then toss exceptions.py > and build the C version of the exceptions module. > > Greg, is there anything in exceptions.py Python2C couldn't handle? Nah, shouldn't have any problem at all. P2C will even create true class objects and expose them in the interface. I think a person might want to consider hand-tuning the output, though :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Thu Sep 16 20:03:13 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 14:03:13 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <37E130E1.F21CDE8B@interet.com> Skip Montanaro wrote: > Why not just run exceptions.py through Python2C, What is Python2C. Is it the same as freeze? Jim Ahlstrom From guido at CNRI.Reston.VA.US Thu Sep 16 20:08:01 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 14:08:01 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Thu, 16 Sep 1999 12:49:12 CDT." <14305.11672.687570.290221@dolphin.mojam.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <199909161808.OAA01452@eric.cnri.reston.va.us> > Why not just run exceptions.py through Python2C, visually and experimentally > verify that it works, then ship an exceptions.c as an optional module? > People wanting to ship self-contained packages could then toss exceptions.py > and build the C version of the exceptions module. Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at mojam.com Thu Sep 16 20:34:47 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 13:34:47 -0500 (CDT) Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E130E1.F21CDE8B@interet.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <37E130E1.F21CDE8B@interet.com> Message-ID: <14305.14407.189346.498215@dolphin.mojam.com> Jim> What is Python2C. Is it the same as freeze? Nope. Python in, compilable C out: http://www.mudlib.org/~rassilon/p2c/ Courtesy of Greg Stein and Bill Tutt. Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From jim at interet.com Thu Sep 16 20:37:58 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 14:37:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <199909161808.OAA01452@eric.cnri.reston.va.us> Message-ID: <37E13906.E2E8FB2E@interet.com> Guido van Rossum wrote: > Alternatively (and probably easier) it (and site.py) could be stored > as frozen modules. All it takes is some edits to Python/frozen.c. An excellent design, very simple. A large list of critical Python-language modules including a new importer, imputils.py and a new main could be reliably linked to their interpreter. It would open the door writing a larger part of Python in Python. But editing Python/frozen.c will kill the current freeze feature since a user currently replaces this pointer with their own. We can have a second _PyImport_InternalFrozenModules[] array which is searched FIRST, so that the existing frozen modules feature is retained. Current logic only allows one frozen module array. This is easy. A little harder is turning it off, which may be important to developers. They will probably want to use site.py etc. in PYTHONPATH directories. I would hate to use another command line option. Maybe another method in the imp module, say imp.EnablePyInternalLib(x) to turn on/off the internal frozen modules. A method to print the names in the list wouldn't hurt either. At one time I had code for multiple frozen modules which I may be able to find. Jim Ahlstrom From jim at interet.com Thu Sep 16 21:09:57 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 15:09:57 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <1274619156-36273405@hypernet.com> Message-ID: <37E14085.FE46EB12@interet.com> Gordon McMillan wrote: > So I don't think there's a need for one canonical do-everything > importer (or archive format). PYTHONPATH is outside any > particular importer. Effectively, you can use a chain of > importers to replace PYTHONPATH. So the platform specific > modules might be found by one particular importer. In other > words, I think it's more effective to specialize individual > importers and chain them up than it is to try to create an > overly-generalized importer. Greg agrees with you so I defer to the experts on importers. The feature is meant to support a chain. Greg wrote: > The .pyl format was discussed a bit on the distutils-sig list and "sort > of" accepted as an okay format for jamming a bunch of modules into a > single file. > [by "sort of", I mean that the small group who participated in the > discussion were okay with it :-); it is a great, minimalist format, so > it probably won't please people who like a ton of features in a file >format :-) ] But I still disagree on the .pyl file format. If there is no Standard Format and everyone is linking in his own importer, then we will have exactly the same situation we have now with PYTHONPATH and novel import hooks. There should be a Standard Format to fix this problem. In particular, package authors should be able to publish packages as PYL files and expect them to be usable as is with no further effort. Sysadmins should be able to manage everything PYTHONPATH does with a small (one?) number of PYL files and in a standard way. Jim Ahlstrom From fredrik at pythonware.com Fri Sep 17 09:40:18 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 17 Sep 1999 09:40:18 +0200 Subject: [Python-Dev] Re: Path hacking References: <1274619162-36273021@hypernet.com> Message-ID: <00f801bf00df$e45361c0$f29b12c2@secret.pythonware.com> > Summary: I'm very interested in seeing import and import > hooks get rationalized, and I think Greg's stuff goes a long, > long way towards that goal. don't have time to say much more than: -- ME TOO! (we're also doing this in the pythonworks packager, and Greg's module is essential for that work. might as well turn it into a standard Python feature). From mhammond at skippinet.com.au Fri Sep 17 09:39:51 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 17 Sep 1999 17:39:51 +1000 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E0E94A.A2F0D37F@interet.com> Message-ID: <000501bf00df$d41cafa0$0801a8c0@bobcat> > > you can do this without recompiling! The version string is used is > > embedded in a resource, so you can patch it using some kind of > > resource editor. Mark Hammond planned it this way! > > I don't understand this. Is there documentation? Vague documentation at http://www.python.org/windows/python/registry.html Doesnt explicitely say much about this particular issue, and if I remember I will update it. VC will allow you to edit these resources without rebuilding - simply open the DLL, but select "Resources" as the file type. I havent investigated other resource editors as I have MSVC :-) Greg pointed you to the win32api functions - they do work, as Greg provided code that uses them (the code he pointed at was done by him) - although I never actually worked out how to make them work ( I spent about an hour trying to get code to re-stamp the Python15.dll with a new sys.winver, and gave up. Would be cool to get going tho... Mark. From mhammond at skippinet.com.au Fri Sep 17 09:48:26 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 17 Sep 1999 17:48:26 +1000 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E105A8.A2B06A80@interet.com> Message-ID: <000601bf00e1$072a7d90$0801a8c0@bobcat> > But it doesn't help, perhaps as a result of more of my prejudices. > This registry entry is meant to be used by a by-the-rules shared > Python installation, so I must not change it. No - the point is that the "1.5" key is "reserved" by a standard install. Changing the string value actually allows you to have your own subtree, and you can assume you own that. For commercial apps I help with, re stick a "AppName" in this string (doesnt have to be a number) then simply use "HKLM\Software\Python\AppName" > And I hate changing any registry entries at all. Well, you should learn to get over it! That is what it is designed for. sure, we dont want the users to muck with the registry, but if you dont like your install scripts working with the registry, I definately feel you should get over this prejudice of yours. > only minimal and required registry entries. IMHO the Windows registry > is a software catastrophe which ranks right up there with JCL (anyone > else here old enough to remember that?). Anyone who doesn't agree > should > go with me to our money center banking clients, and sit there > while they > grill you on every registry entry and why it is required. > Money center > banking clients do not like their registry messed with. The registry is a huge bucket. Changing your own, custom kep under either HKLM\Software\ or HKCU\Software is perfectly reasonable. Many apps _require_ you to hit the registry - services, event log, performance monitor, etc. The fact the registry _also_ contains critical information is a problem, and sure, we dont want to mess with that. > My current "solution" is to use freeze to create a black-box install, > and worry about second Python installations and wasted storage when it > happens. well, IMO this is also the correct thing to do. any install that has >100 files is fragile. So I do both - freeze the app, _and_ a custom "sys.winver". > I was hoping that this thread whould result in a consensis of what > to do, but it has not. There is a consesus for people with the same problem. Different problems have different optimal solutions. Mark. From mal at lemburg.com Fri Sep 17 11:33:43 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 11:33:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <37E20AF7.2D05F371@lemburg.com> Guido van Rossum wrote: > > > I'm wasn't planning to write my own hook until now. I need this > > feature to be able to organize my package internal stuff swiftly > > and easily, plus to be able to possibly put it under new top-level > > packages. Obviously I seem to be more or less alone with this need, so > > I'll stop argueing for inclusion of "something like relative imports" > > in the distribution. > > I still hope against hoping to convince you not to do this. I think > this adds lots of complexity for the reader of your package. I think > of *all* software that I write as something that eventually someone is > going to read and try to understand. I also think of it as something > that eventually someone is going to port to a platform that wasn't > even designed when I wrote the software. Tricks like depending on a > custom import hook make reading a pain. > > For example, the reader who is just starting to browse a large package > often needs to follow the thread of execution from one module to the > next. If the import statements contain magic like __magic__.foo.bar > (actually proposed magic renamed for generality :-), this reader will > first need to understand the custom importer -- which is likely one of > the hariest components of the package. While it will perhaps look funny to him at first sight, I'm sure that a proper name will make the idea clear to him: from __superpackage__.foo import bar or just from __super__.foo import bar [Super Foo -- sounds like a comic star ... there's even integrated fun in these names ;-)] > The same thing applies with even more force to tools like package > browsers. IDLE for example has a class browser which displays the > true module and class name for base classes. E.g. if you have code > that says > > from Tkinter import Widget > > class MyWidget(Widget): ... > > it correctly deduces that MyWidget's base class is Tkinter.Widget. > (And in a future version it will probably allow you to click through > to that class, show you its methods, etc.) > > Custom importers break this feature, and thus make the code harder to > analyze for the reader. That's why I tried argue for making it a standard: all tools will then know about the syntax used and take appropriate action. > (Not all custom importers are bad; there are plenty of reasons to > augment the standard importer. But, while custom importers make > different interpretations of the namespace possible, I frown upon that > particular use.) > > > (b) is (if at all) a problem only to be taken into account by > > the author of package X. He may or may not use rel. imports. > > A 12-year old probably won't (but then: you never know ;). > > Which severely limits your ability to do what you want with packages > you didn't write. > > I know that this sounds politically incorrect in a radical free > software world, but often in the end I find it more convenient to > conform to the rest of the world and "fit in" than to be different in > every little way. Note that my hair isn't blue :-); I've also > replaced my "signature" glasses with a more conventional model. Naa, I don't want to be different just to be plain silly different. The case here is one of maintenance. I have around 12 Python packages which I maintain, some of them public, others proprietary. Due to the large number I try to make them all as easy to handle as possible. Now I can't just go and rename the imports in case I want to put them under some other common package name for application A and then do the same with another package name for application B. I would have to do the same thing for every new release of the packages since there are also others who use the standard mx-based name. I just want the packages to work whereever I put them, e.g. I could link them into the new location using symlinks and be done with the work. Of course there are other ways to make this work (and I want to thank everybody who contributed to the sys.path ideas), but they just add omplexity to a whole different area. > > BTW, what is this CP4E thing you're talking about. If it's an > > syntax aware editor, I have a friend who is really interested > > in these things... could you send me an URL that I can send him ? > > Where have you been? It's on the python.org homepage, has been > discussed in c.l.py, c.l.tcl, even c.l.ada, in lwn, and on /.! The > syntax aware editor (a proposed super-version of IDLE) is only a small > part of it. See python.org/doc/essays/cp4e.html Nope, haven't heard anything about it -- perhaps I was on vacation. Sound interesting though. An editor where you can just type away (and the editor fills in the gaps) would be great fun. > > Oh well... I guess they'll have to use 'mex' if they decide to go > > the reverse domain way ;-) > > Or, in a worse-case scenario, the first Mexican developers using the > reverse domain will probably not be aware of the mx toplevel package, > and by the time their software hits the street it will be too late. Well, I guess then they'll have a good reason to upgrade their software. Just like all those companies who made extra money by sticking a "Y2000 + Euro-capable" button on their software. No, seriously, with the proposed extension (see below) this shouldn't be much of a problem because my packages all use capital first letters. The reverse domain name would use lower case ones, e.g. mx.pythonware.pilios or mx.zopistas... > > True, perhaps we should lighten this requirement a little when we > > recode the import mechanism in Python ? E.g. if a local import > > fails continue the search with the fully qualified name and only > > if that fails, restart using the local name. This would need some > > kind of fastpath cache to make the search reasonably fast though. > > Now this is something that I would consider. Should be easy to add to the imputil based core emulation. > > Wouldn't it suffice to just put them into one package, e.g. > > 'python.' ? > > And somehow make all user code live implicitly inside that package? I > don't see how that solves anything. No, to free the top-level namespace and make the origin of the imports clearer, e.g. import python.image import PIL.image import zope.image etc. > And believe me, at some point *someone* is going to > need to rip out your custom importer because it interferes with *his* > custom importer, and he'll choose to replace all your __ imports with > absolute package names, rather than trying to add the __ feature to > his importer. Note that with imputil doing its thing, he won't have to worry about adding code to his importer. It'll just pass on the import to my importer. That's the fun part about imputil. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Vladimir.Marangozov at inrialpes.fr Fri Sep 17 17:07:19 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Fri, 17 Sep 1999 16:07:19 +0100 (NFT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909151843.OAA18364@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 15, 99 02:43:56 pm" Message-ID: <199909171507.QAA33468@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > I hereby withdraw the posted license. There still is the need for a > new license, but we need to go back to the drawing board for it. So, in the end, am I still invited to sign & send the "wet" form or I'd better wait to let it dry? BTW, I'm surprised by the fact that in an Open Source world I'm asked to sign a licence agreement with CNRI or to send e-mails for contributed code. If Python or Linux had had such constraints from the start, they wouldn't have been what they are today. -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From guido at CNRI.Reston.VA.US Fri Sep 17 16:07:50 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 17 Sep 1999 10:07:50 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Fri, 17 Sep 1999 16:07:19 BST." <199909171507.QAA33468@pukapuka.inrialpes.fr> References: <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <199909171407.KAA00575@eric.cnri.reston.va.us> > So, in the end, am I still invited to sign & send the "wet" form or > I'd better wait to let it dry? Please send in the form -- the license was a totally separate issue that I shouldn't have brought up in the same mail (or at all, in this stage anyway -- we'll work this out with the Python consortium members first). > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. Unfortunately, that's the price we have to pay. What we get is legal protection from CNRI. In general CNRI has contributed a lot to Python; probably more than you realize. In any case, signing the form and including the email paragraphs is completely voluntary -- if you don't want to do it, just let me know. --Guido van Rossum (home page: http://www.python.org/~guido/) From Vladimir.Marangozov at inrialpes.fr Fri Sep 17 17:35:01 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Fri, 17 Sep 1999 16:35:01 +0100 (NFT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909171407.KAA00575@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 17, 99 10:07:50 am" Message-ID: <199909171535.QAA30684@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > Please send in the form -- the license was a totally separate issue > that I shouldn't have brought up in the same mail (or at all, in this > stage anyway -- we'll work this out with the Python consortium members > first). ok. > > > BTW, I'm surprised by the fact that in an Open Source world I'm asked > > to sign a licence agreement with CNRI or to send e-mails for contributed > > code. If Python or Linux had had such constraints from the start, they > > wouldn't have been what they are today. > > Unfortunately, that's the price we have to pay. What we get is legal > protection from CNRI. In general CNRI has contributed a lot to > Python; probably more than you realize. I realize that. But I also realize that in case of a problem, the wet form protects CNRI, not the contributor. Hm. And what happens if you get hit by a bus? Or in 100 years when we'll dance with angels in Paradise? Will Python stay bound to CNRI with little legal possibilities to detach it, in case our successors (or you) start working in another organization? IANAL and curious. -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From fdrake at acm.org Fri Sep 17 16:41:47 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 17 Sep 1999 10:41:47 -0400 (EDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909171507.QAA33468@pukapuka.inrialpes.fr> References: <199909151843.OAA18364@eric.cnri.reston.va.us> <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <14306.21291.786632.89652@weyr.cnri.reston.va.us> Vladimir Marangozov writes: > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. You shouldn't be; the FSF certainly requires a signed copyright assignment from contributors. I had to sign one for a bunch of patches I made to oleo many years ago. It was a minor nuissance, but that's all. (The *sad* part is that there hasn't been a new release of oleo that could have included the patches for five years! ;) Just noticed that there is a release at ftp.gnu.org now; I'll have to take a look! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal at lemburg.com Fri Sep 17 16:45:52 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 16:45:52 +0200 Subject: [Python-Dev] Methods and friends Message-ID: <37E25420.6C3F5E3D@lemburg.com> I was just looking into classobject.c again after probably more than a year and wondered... has anyone tried optimizing the method lookup and binding process recently ? It seems that for every method call the following happens: 1. the method is looked up in the instance dict; this fails 2. the method is looked up in the class dict; this succeeds and returns a function 3. the class then turns the function into a new unbound method 4. the instance sees the unbound method and creates a new bound method (deleting the unbound method) and returns it Two possible improvements: ? let the instance use a special class_getattr_ex function that does not do the extra unbound method step (3.) or ? make methods mutable and have step 4. insert the instance object into the method object to turn it into a bound method Has anyone tried this ? Does it produce anything noticeable w/r to method call speed ? BTW, why does the new module check for instance objects for the INSTANCE argument ? Methods can handle any object in this slot, just as they handle any callable object as "function". Could be put to some use when wrapping types into classes or vice versa... which is what motivated the above ;-) Just curious, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Fri Sep 17 17:09:40 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 17 Sep 1999 11:09:40 -0400 Subject: [Python-Dev] Methods and friends In-Reply-To: Your message of "Fri, 17 Sep 1999 16:45:52 +0200." <37E25420.6C3F5E3D@lemburg.com> References: <37E25420.6C3F5E3D@lemburg.com> Message-ID: <199909171509.LAA00729@eric.cnri.reston.va.us> > It seems that for every method call the following happens: > 1. the method is looked up in the instance dict; this fails > 2. the method is looked up in the class dict; this succeeds > and returns a function > 3. the class then turns the function into a new unbound method > 4. the instance sees the unbound method and creates a new > bound method (deleting the unbound method) and returns it Are you sure? As far as I know, steps 3 and 4 are combined when you do getattr on an instance: instance_getattr() calls instance_getattr1() which calls class_lookup(). The latter doesn't create an unbound method. instance_getattr1() then turns it into a bound method. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw at cnri.reston.va.us Fri Sep 17 18:30:21 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Fri, 17 Sep 1999 12:30:21 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <199909151843.OAA18364@eric.cnri.reston.va.us> <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <14306.27805.461540.758955@anthem.cnri.reston.va.us> >>>>> "VM" == Vladimir Marangozov writes: VM> BTW, I'm surprised by the fact that in an Open Source world VM> I'm asked to sign a licence agreement with CNRI or to send VM> e-mails for contributed code. If Python or Linux had had such VM> constraints from the start, they wouldn't have been what they VM> are today. Note that the FSF has been requiring signatures for a long time. Actually, their requirements are IMHO more onerous because they require you to assign your copyrights to the FSF. Their lawyers have told them that they cannot defend the copyright of, e.g. Emacs, unless they own the copyrights to the entire codebase (or at least, anything that they couldn't rip out, throw away, and not completely cripple the application). CNRI's viewpoint is less drastic, but still important. It means that you retain the copyright on the code (good for you), but you give us permission to use it as we see fit (good for us /and/ for you :). Otherwise, it would be possible for a malicious person to contribute something really vital and useful, wait for it to become indispensible, and then say, "oops, we really didn't mean to let you use that code, sorry!" -Barry From jim at interet.com Fri Sep 17 21:20:24 1999 From: jim at interet.com (James C. Ahlstrom) Date: Fri, 17 Sep 1999 15:20:24 -0400 Subject: [Python-Dev] Re: Path hacking References: <000601bf00e1$072a7d90$0801a8c0@bobcat> Message-ID: <37E29478.316C27C9@interet.com> Perhaps you have heard about the East coast US hurricane. It really hammered us here in New Jersey. I had trouble getting home last night due to high water. When I went to our usual Japanese lunch restaurant it had broken windows and was full of tree branches. Wow... Anyway, Mark Hammond wrote: > > No - the point is that the "1.5" key is "reserved" by a standard > install. Changing the string value actually allows you to have your > own subtree, and you can assume you own that. OK, now that I know the rules I will think about doing that. > > And I hate changing any registry entries at all. > > Well, you should learn to get over it! I will try. Of course, even if the registry helps ship Python apps on Windows, it is no help on Unix, and I care about that almost as much. > > My current "solution" is to use freeze to create a black-box > install, > > and worry about second Python installations and wasted storage when > it > > happens. > > well, IMO this is also the correct thing to do. any install that has > >100 files is fragile. So I do both - freeze the app, _and_ a custom > "sys.winver". Yes, but this requires a compiler. > > I was hoping that this thread whould result in a consensis of what > > to do, but it has not. > > There is a consesus for people with the same problem. Different > problems have different optimal solutions. The problem is that there is no reliable way to ship bullet-proof Python apps without recompiling and rebuilding Python. Each of us has his own pet hack to solve our own problem. Now there is talk of custom import hooks, and this is likely to result in each package requiring its own import hook! Aren't packages supposed to be software IC's? I hate to be a nag, but I will keep pushing for a single solution. Python is totally cross platform, and with all that machinery, there must be portable way to do that. I think this is good for Python. Please don't think I am trying to solve my own selfish problems. I have a compiler, I am happy using freeze, and I don't have any problems. Its just that Python would "sell" better and be more popular if a developer could read the documentation "How to ship and install your Python app in five minutes and make millions". This documentation currently reads "You can't". Jim Ahlstrom From gstein at lyra.org Sat Sep 18 00:14:04 1999 From: gstein at lyra.org (Greg Stein) Date: Fri, 17 Sep 1999 15:14:04 -0700 Subject: [Python-Dev] License cleanup References: <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <37E2BD2C.72792E0F@lyra.org> Vladimir Marangozov wrote: >... > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. Actually, this isn't surprising at all. The Free Software Foundation *requires* this kind of thing to be filed with them before you contribute code to the FSF. Essentially, it is a way for the FSF (and CNRI) to legally state that they own the copyright on the particular code. Without that, the contributor could come along later and claim a copyright on the code. The IBM folks who are working on Apache have provided legal releases to the Apache Software Foundation that basically states that IBM won't try to assume any rights under copyright law on the code they contribute to Apache. In fact, every time that I receive a patch for my mod_dav Apache module, the IBM guy attaches a release to the email that has the patch. In a pure, cooperative, world none of this would be necessary. However, the world simply doesn't work that way and all this stuff (licenses, copyrights, releases) is there to prevent Bad Things from happening. It isn't evil in itself, but simply a reflection of the business environment and the society that we're working within. Cheers, -g -- Greg Stein, http://www.lyra.org/ From tim_one at email.msn.com Sat Sep 18 08:38:27 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 02:38:27 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <14306.21291.786632.89652@weyr.cnri.reston.va.us> Message-ID: <000001bf01a0$6ae8cca0$022d153f@tim> [Vladimir Marangozov] > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for > contributed code. [Fred L. Drake, Jr.] > You shouldn't be; the FSF certainly requires a signed copyright > assignment from contributors. I had to sign one for a bunch of > patches I made to oleo many years ago. It was a minor nuissance, but > that's all. Except they add up: year after year, a new batch of stupid little requirements piles up on top of the last year's, and it's a ratchet effect -- always more, never less. The aggregate gets to be a real weariness on the soul. I had to laugh when Fran?ois Pinard happened to post this on c.l.py today: > ... > Would it be any volunteer, at least for taking care of filling the FSF > papers, if any are needed? I filled more than enough of those in my > life, I prefer to avoid the burden. Ask Barry how many years we've been trying to sign pymode over to the FSF <0.5 wink>. [Vlad] > If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. I sympathize, but that's really hard to say. You don't get pig-biting weary of this crap until you're my age . AFAIK, Berkeley has never beed sued over the BSD license, MIT over the X license, or the U of Arizona over the Icon license (none, really -- Icon is in the public domain). All the legal mumbo jumbo in the "modern" licenses is like wearing garlic around your neck to ward off vampires: the threat isn't real, and if it were it wouldn't do you any good anyway. a-wooden-stake-thru-the-heart-is-your-only-true-defense-ly y'rs - tim From Vladimir.Marangozov at inrialpes.fr Sat Sep 18 16:17:28 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Sat, 18 Sep 1999 15:17:28 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) Message-ID: <199909181417.PAA20866@pukapuka.inrialpes.fr> The relative import thread has hit me with full force. I'm not sure I understand all of what has been said, but since I know this is mainly a naming issue, I want to point you to one of the best CS papers I have had in my hands. It's not really my style to recommend references, but this one should be of major interest to this audience (and Python respectively): Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. It's not available online (it was written on a typewriter), so I'd be happy to send a hard copy of it to anyone who raises a hand in private mail (or cannot find LNCS 60). It's simply a jewel. This guy has understood everything (and I suspect Guido has read this one before implementing Python ;-) -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From tim_one at email.msn.com Sun Sep 19 01:44:49 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 19:44:49 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909160332.XAA19278@eric.cnri.reston.va.us> Message-ID: <000001bf022f$cc11cba0$072d153f@tim> [Guido] > ... > In the Java world, the right solution exist: a better top-level naming > scheme. I forget what the exact rules are, but I seem to recall that > Java has something of the same relative import rules af Python > currently has: you can import a class from the same leaf package by > its simple name, or you can import a class by its full name. You can import only by the full name. There are no shortcuts for anyone ever in explicit Java imports. The dodge is that to reference classes in exactly the same package, you don't need an import statement at all (you *can* have one, but then you need to use the fully qualified name). The other twist is that all Java code is in *some* package, and code that (wisely) doesn't want to live in the ill-defined unnamed package must contain a package statement declaring which package it's a part of, like package COM.dragonsys.justanexample; The periods there actually have no semantic significance, and package COM.dragonsys.notanexample has no more in common with the above than does package ORG.python.guido.hitmeagain. It's purely a conceptual thing that justanexample and notanexample are *called* "subpackages" of package COM.dragonsys in Java; although implementations are free (but not required) to make up mappings between dotted package names and directory trees. Anyway, the Java rules work fine wrt the issues that have been argued in this thread. Things that are worse than Python: + Magical implied import of same-package names is a PITA for readability and maintenance. + Since subpackages are an illusion, and even packages aren't objects in their own right, there's no possibility to create a shorthand local name for the first M chunks of an N-chunk dotted access path (with M < N). For example, you may use java.awt.Font and java.awt.TextArea a lot. You either spell those out in full all the time, or do import java.awt.Font; import java.awt.TextArea; and use just "Font" and "TextArea" later. Making up a local name for just the "java.awt" part isn't an option. Which leads to gross overuse of the following: + "import package.*" is Java's way of spelling "from package import *", and because of the preceding is the only way to get convenient local names for classes imported from other packages (note that can you can never import a package/module in Java; you can only import a type name). So most Java code will do the above as import java.awt.*; and end up importing a gazillion names. This sucks for the same reasons "import *" sucks in Python, although Java catches the name conflicts at compile time. Things that are better than Python: + Name conflicts are caught at compile time. + At least in theory, the names of packages aren't constrained by platform file name limitations or directory quirks. + Relatedly, explicitly naming the package a compilation unit belongs can be a Good Thing. + They have *an* effective scheme for preventing package-name collisions. > ... > Note that I still resist the urge to start renaming all the existing > standard modules -- too much breakage. This will happen in 2.0. At home, I dropped an empty __init__.py into the std Lib directory, so that I can do e.g. from Lib.random import randrange in my own packages with their own random modules. Works fine. For 1.6 you should pick some explicit way to *allow* explicit import of at least the std library modules. I suggest taking the name "Lib" . Java is very cute with its java.util.etc stuff, but longwindedness is not a virtue here -- "util" would have worked just as well and been just as clear. Nobody should write a package with a generic short name like "util" (or "Lib") anyway, so rather than waste this pleasant little corner of the namespace the language may as well reserve it for its own use. heck-it-already-stole-"and"-and-"or"-ly y'rs - tim From tim_one at email.msn.com Sun Sep 19 05:18:43 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:43 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37E20AF7.2D05F371@lemburg.com> Message-ID: <000001bf024d$ae271f00$f72d153f@tim> [MAL] > ... > Of course there are other ways to make this work (and I want > to thank everybody who contributed to the sys.path ideas), but > they just add complexity to a whole different area. The complexity is already in that area, and for all of us. It can be dealt with. Adding new complexity elsewhere doesn't decrease the complexity that's already there, it just adds more and new kinds of stuff to worry about. [Guido] >> ... >> And believe me, at some point *someone* is going to need to rip out >> your custom importer because it interferes with *his* custom importer, >> and he'll choose to replace all your __ imports with absolute package >> names, rather than trying to add the __ feature to his importer. > Note that with imputil doing its thing, he won't have to > worry about adding code to his importer. It'll just pass on the > import to my importer. That's the fun part about imputil. As I recall, the old ni took the initial "__" as referring to the *current* package. So where you intend to write import __.sibling.func it was written under ni as import __.__.sibling.func Since ni is semi-blessed prior art, some other joker is going to slam in an Importer to use those rules. I mix your packages with theirs, and then the meaning of "__" depends on whose Importer sees the damn thing first. Again complexity spreads. If Jim is wary of top-level name collisions now, wait until he thinks about naming-gimmick collisions . Guido is telling the truth: no package is going into Dragon's products unless it's rewritten to purge dicey name tricks. I think we're all in favor of Greg's imputil approach, but where it adds real value (primarily in allowing imports to get satisfied from sources other than direct file images). if-it's-in-a-file-you-just-import-it-by-name-ly y'rs - tim From tim_one at email.msn.com Sun Sep 19 05:18:52 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:52 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37E0CCC1.6EE4E599@lemburg.com> Message-ID: <000201bf024d$b3213ae0$f72d153f@tim> [MAL] > ... > Seems that we're a bit too healthy (see MarkH's post) sometimes, > i.e. there isn't all that much room for experiments. The odds of a change making it into Python radically decreased when 1.0 hit the net, and have continued to decline (although slowly) since then. In recent years, Guido appears to me to have gotten ever more reluctant to entertain even 100% compatible changes to the internals, if they affect a delicate area of the implementation (ceval.c is the most obvious one there). But that's "normal & healthy" too. Languages & implementations get brittle with age, and it's eventually better to start over -- if Guido didn't have Python2 plans in mind, he'd be the first language designer ever to stop where he started! > Just think of cool developments like Chris' stackless python. Experience > shows that these kind of things will never make it into the distribution. Unfortunately, circumstances piled up and Chris got distracted from that, while nobody else made time to push it in his absence. Large changes have gone in, and even more may make it into the Python1 line, but it generally takes a large or "strategic" user base, and much persistence. GregS mentioned his massive work on threads (still not all in), and I'll add the NumPy extensions (which I wouldn't be surprised to see "mainstreamed"), BarryW's string methods, and DavidA's rich comparisons. > Unfortunately, maintaing patches to the dist across releases a real > pain and much work, so these ideas will just sit there unused and > untested. Much the same happened to gcc ... in the end corporate > strength made egcs possible. Perhaps we need such a branch too ? Don't tell, but I've always been surprised at how few people have tried to release a variant Python! The Alice version (case-insensitive names, and 1/2==0.5) is the only one that comes to mind, and the primary effect that had on today's Python is that raw expressions no longer print their value in non-interactive mode (before Alice, 1 + 2 on a line by itself caused "3" to get printed even in batch scripts; this interfered with the Alice team's favored object.method1().method2().method3() coding style, and Guido endured much pain to change "the real" Python to avoid a code split at that early stage of Python's life; ultimately futile, but then Alice Python didn't catch on anyway). So there's very little Python-related history to go on here. I don't mind seeing variants, but have to predict they won't get very far. Just picture what Python 1.6V would look like if its feature set were drawn from a consensus among you, me, Christian, Greg Ewing, John Skaller and Tom Christiansen . the-value-of-a-benevolent-dictator-is-easy-to-underestimate-ly y'rs - tim From tim_one at email.msn.com Sun Sep 19 05:18:47 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:47 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <000101bf024d$af986240$f72d153f@tim> [Guido] > I know that this sounds politically incorrect in a radical free > software world, but often in the end I find it more convenient to > conform to the rest of the world and "fit in" than to be different in > every little way. Note that my hair isn't blue :-); I've also > replaced my "signature" glasses with a more conventional model. !!! No wonder you tried to slam a conventional license down our throats. My advice is to get back the old glasses, and walk around the halls at CNRI counting backwards from 1000 by seventeens loudly and incessantly. you'd-be-amazed-at-how-few-lawyers-talk-to-me-ly y'rs - tim From bwarsaw at cnri.reston.va.us Sun Sep 19 09:01:29 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Sun, 19 Sep 1999 03:01:29 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <199909160332.XAA19278@eric.cnri.reston.va.us> <000001bf022f$cc11cba0$072d153f@tim> Message-ID: <14308.35401.20723.782353@anthem.cnri.reston.va.us> >>>>> "TP" == Tim Peters writes: TP> + "import package.*" is Java's way of spelling "from package TP> import *", and because of the preceding is the only way to get TP> convenient local names for classes imported from other TP> packages (note that can you can never import a package/module TP> in Java; you can only import a type name). So most Java code TP> will do the above as TP> import java.awt.*; TP> and end up importing a gazillion names. This sucks for the TP> same reasons "import *" sucks in Python, although Java catches TP> the name conflicts at compile time. The interesting this is that, while the Java developers did this at the language level, at the VM level, every class is "fully qualified"; you see the absolute path for every class name. import pkg.pkg.* is the reason why you still have to have unique named classes like PyThinging or JWiggie, and yup it sucks. -Barry From mal at lemburg.com Fri Sep 17 17:53:38 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 17:53:38 +0200 Subject: [Python-Dev] Methods and friends References: <37E25420.6C3F5E3D@lemburg.com> <199909171509.LAA00729@eric.cnri.reston.va.us> Message-ID: <37E26402.3C519434@lemburg.com> Guido van Rossum wrote: > > > It seems that for every method call the following happens: > > 1. the method is looked up in the instance dict; this fails > > 2. the method is looked up in the class dict; this succeeds > > and returns a function > > 3. the class then turns the function into a new unbound method > > 4. the instance sees the unbound method and creates a new > > bound method (deleting the unbound method) and returns it > > Are you sure? As far as I know, steps 3 and 4 are combined when you > do getattr on an instance: instance_getattr() calls > instance_getattr1() which calls class_lookup(). The latter doesn't > create an unbound method. instance_getattr1() then turns it into a > bound method. Oops. Correct. I should have looked a little closer. So it already works in the first way I mentioned as improvement... your time machine again ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 20 10:59:58 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 20 Sep 1999 10:59:58 +0200 Subject: [Python-Dev] Relative Package Imports References: <000201bf024d$b3213ae0$f72d153f@tim> Message-ID: <37E5F78E.3859D12F@lemburg.com> Tim Peters wrote: > > [MAL] > > ... > > Seems that we're a bit too healthy (see MarkH's post) sometimes, > > i.e. there isn't all that much room for experiments. > > The odds of a change making it into Python radically decreased when 1.0 hit > the net, and have continued to decline (although slowly) since then. In > recent years, Guido appears to me to have gotten ever more reluctant to > entertain even 100% compatible changes to the internals, if they affect a > delicate area of the implementation (ceval.c is the most obvious one there). > > But that's "normal & healthy" too. Languages & implementations get > brittle with age, and it's eventually better to start over -- if Guido > didn't have Python2 plans in mind, he'd be the first language designer ever > to stop where he started! > > > Just think of cool developments like Chris' stackless python. Experience > > shows that these kind of things will never make it into the distribution. > > Unfortunately, circumstances piled up and Chris got distracted from that, > while nobody else made time to push it in his absence. Large changes have > gone in, and even more may make it into the Python1 line, but it generally > takes a large or "strategic" user base, and much persistence. GregS > mentioned his massive work on threads (still not all in), and I'll add the > NumPy extensions (which I wouldn't be surprised to see "mainstreamed"), > BarryW's string methods, and DavidA's rich comparisons. Plus the coercion stuff that's still sleeping in one of my project subdirs (I'll have to get this done *before* 1.6 hits the shelves). > > Unfortunately, maintaing patches to the dist across releases a real > > pain and much work, so these ideas will just sit there unused and > > untested. Much the same happened to gcc ... in the end corporate > > strength made egcs possible. Perhaps we need such a branch too ? > > Don't tell, but I've always been surprised at how few people have tried to > release a variant Python! The Alice version (case-insensitive names, and > 1/2==0.5) is the only one that comes to mind, and the primary effect that > had on today's Python is that raw expressions no longer print their value in > non-interactive mode (before Alice, > > 1 + 2 > > on a line by itself caused "3" to get printed even in batch scripts; this > interfered with the Alice team's favored > > object.method1().method2().method3() > > coding style, and Guido endured much pain to change "the real" Python to > avoid a code split at that early stage of Python's life; ultimately futile, > but then Alice Python didn't catch on anyway). > > So there's very little Python-related history to go on here. I don't mind > seeing variants, but have to predict they won't get very far. Just picture > what Python 1.6V would look like if its feature set were drawn from a > consensus among you, me, Christian, Greg Ewing, John Skaller and Tom > Christiansen . Actually, what I was thinking about here was a Python 2.0 branch starting now rather than in a year or so and thus leaving much room for experiments etc. The intention was the same as with egcs and gcc: to fold the enhancements back into the main branch in a few years. E.g. if Guido points us in the right direction, we could start hacking on that piece of revolutionary work now. BTW, I'd suggest using C++ with namespaces but without templates as target language. By the time Python 2.0 will hit the shelves this setup should have reached the same portability as C has now. Perhaps we could even use RTTI (run time type information) to implement optional type safety... ok, just dreaming a little ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 102 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From skip at mojam.com Mon Sep 20 17:29:55 1999 From: skip at mojam.com (Skip Montanaro) Date: Mon, 20 Sep 1999 10:29:55 -0500 Subject: [Python-Dev] Tackling circular dependencies in 2.0? Message-ID: <199909201529.KAA21605@dolphin.mojam.com> I think in Python 2.0 it would be nice to have some way to reclaim circular dependencies without the programmer explicitly having to do something like implement a destroy() method and requiring other programmers to (remember to) call it. I forget what the current state of affairs is w.r.t. future memory management in Python. Not knowing anything much about memory management, would it be possible to have a sort of mixed ref count/garbage collection system where you only use the gc stuff as a last resort? My thought is that it would be useful to use gc to find and reclaim circular garbage. can-you-tell-I-just-got-bitten-by-a-circular-reference?-ly y'rs Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From mal at lemburg.com Mon Sep 20 19:44:57 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 20 Sep 1999 19:44:57 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <37E67299.60899D38@lemburg.com> Skip Montanaro wrote: > > I think in Python 2.0 it would be nice to have some way to reclaim circular > dependencies without the programmer explicitly having to do something like > implement a destroy() method and requiring other programmers to (remember > to) call it. I forget what the current state of affairs is w.r.t. future > memory management in Python. Not knowing anything much about memory > management, would it be possible to have a sort of mixed ref count/garbage > collection system where you only use the gc stuff as a last resort? My > thought is that it would be useful to use gc to find and reclaim circular > garbage. > > can-you-tell-I-just-got-bitten-by-a-circular-reference?-ly y'rs If you don't want to wait until 2.0 becomes GA, you could try weak references: http://starship.skyport.net/~lemburg/mxProxy.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 102 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward at cnri.reston.va.us Mon Sep 20 19:53:03 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Mon, 20 Sep 1999 13:53:03 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909201529.KAA21605@dolphin.mojam.com>; from Skip Montanaro on Mon, Sep 20, 1999 at 10:29:55AM -0500 References: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <19990920135302.B393@cnri.reston.va.us> > Not knowing anything much about memory > management, would it be possible to have a sort of mixed ref count/garbage > collection system where you only use the gc stuff as a last resort? My > thought is that it would be useful to use gc to find and reclaim circular > garbage. That's *sorta* what Perl does, 'though you can still be bitten by circular refs in a long-running process. Err, long-running thread. You see, Perl has a mark-and-sweep garbage collector which is run only on thread shutdown. From gward at cnri.reston.va.us Mon Sep 20 19:57:51 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Mon, 20 Sep 1999 13:57:51 -0400 Subject: [Python-Dev] Preview of some distutils news Message-ID: <19990920135750.D393@cnri.reston.va.us> Judging by the recent traffic on python-dev, lots of people here are interested in the problems of distributing and installing Python modules. So I thought y'all might want to know that I've finally got some interesting news to report on the distutils-sig, and will shortly be reporting it, soliciting feedback, testers, etc. If you're not already on that list, this might be a good time to join. (Oh, the news? Distutils can now compile and install NumPy, PIL, and mxDateTime (just because I happened to have those three distributions sitting around on my home PC). Of course there are all sorts of caveats and limitations, so c'mon over to the distutils-sig if you want to hear what they are and discuss how to fix them. Later this week, of course, after I've checked in all the relevant code.) Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From guido at CNRI.Reston.VA.US Mon Sep 20 20:07:03 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 20 Sep 1999 14:07:03 -0400 Subject: [Python-Dev] wet signature form clarification Message-ID: <199909201807.OAA07799@eric.cnri.reston.va.us> Several of you have asked me a clarification of the "contribution description" required on the wet signature form; given that you are all very prolific contributors with a weak memory, it would be nice if you could write something like "everything I've contributed until Sept. 18, 1999" in the space there rather than dig out all the specific module names. I got a call from CNRI's lawyer today, and she agreed that this was fine. Boy am I pleased today! I think she called it a "group description" or something like that. Anyway, if you were waiting for a ruling on this issue, please send in your signed form! (I'd appreciate it if the signature was dry by the time I got it. :-) The form is at http://python.org/1.5/wetsign.html --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one at email.msn.com Tue Sep 21 08:11:00 1999 From: tim_one at email.msn.com (Tim Peters) Date: Tue, 21 Sep 1999 02:11:00 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <000101bf03f8$13ade740$ae2d153f@tim> [Skip Montanaro] > I think in Python 2.0 it would be nice to have some way to > reclaim circular dependencies without the programmer explicitly > having to do something ... This was debated (again) at great length on c.l.py just a few months ago. Guido chimed in with a proposal to keep track of only the dicts that have been allocated, and now and again mark everything reachable from the root set and nuke whatever dicts don't end up marked. Cycles involving dicts would get reclaimed this way, but not cycles not involving dicts. The approach to destructors for objects in cycles was "tough -- they don't get called". What to do about destructors for objects that are not themselves involved in cycles but are reachable only from dead cycles (so are in fact dead too) wasn't addressed. Seemed possible that stuff reachable from ordinary dicts (not in a cycle, and neither reachable from a cycle) would behave differently than today, since the "list of all dicts" may keep the stuff artificially alive until the next mark+sweep, even if the refcount on the stuff fell to zero; there's probably an OK way around that, though. Anyway, Guido was aiming for the minimal changes that could possibly do real good. It didn't pretend to reclaim all cycles, and was (IMO) too eager to punt on the hard issues (the combo of cycles, destructors and resurrection is a god-awful mess, even in theory; Scheme uses callbacks to dump the problems back on the users Java has incredibily elaborate rules that are both bulletproof and unusable; the Boehm collector lets objects with destructors that are in cycles simply leak, rather than do a wrong thing; Stroustrup has flip-flopped and most recently argued for Guido's "reclaim the memory but don't call the destructors" approach, but a member of the C++ committee told me he's overwhelmingly opposed on this one (I know I would oppose it)). In any case, nothing has come of it, and no easy principled solution is in sight. OTOH, if Guido balks at explaining what "__" means to 12-year-olds, wait until he tries to explain immortal cyclic trash . Perl scores points for its brute-force end-of-thread M&S, but I believe complex user-level data structures are much rarer in Perl, simply due to the clumsiness of the syntax and explicit reference model. Perl's version of "nested functions" don't actually nest (all "def"s are floated to the top level by the compiler, regardless of how deeply they're nested), so Perl's lexical closures don't create cyclic trash either (well, they do, but in the same sense there's a cycle between a Python module namespace and the functions in that module -- Perl also special-cases the snot out of those and busts those cycles by brute force). So there you go! It needs to be solved and nobody has a clue . if-java-hype-hadn't-suffered-exponential-decay-we-could-have- dumped-it-on-the-jvm-by-now-ly y'rs - tim From mal at lemburg.com Tue Sep 21 11:05:05 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 11:05:05 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> Message-ID: <37E74A41.107A1243@lemburg.com> Tim Peters wrote: > > [Skip Montanaro] > > I think in Python 2.0 it would be nice to have some way to > > reclaim circular dependencies without the programmer explicitly > > having to do something ... > > This was debated (again) at great length on c.l.py just a few months ago. > Guido chimed in with a proposal to keep track of only the dicts that have > been allocated, and now and again mark everything reachable from the root > set and nuke whatever dicts don't end up marked. Cycles involving dicts > would get reclaimed this way, but not cycles not involving dicts. The > approach to destructors for objects in cycles was "tough -- they don't get > called". What to do about destructors for objects that are not themselves > involved in cycles but are reachable only from dead cycles (so are in fact > dead too) wasn't addressed. Seemed possible that stuff reachable from > ordinary dicts (not in a cycle, and neither reachable from a cycle) would > behave differently than today, since the "list of all dicts" may keep the > stuff artificially alive until the next mark+sweep, even if the refcount on > the stuff fell to zero; there's probably an OK way around that, though. You could probably tackle the problem by doing local mark&sweep whenever the ref count on a dictionary falls down to 1 (meaning that it is only referenced from the list of all dicts). This is what I do in mxProxy's weak reference implementation and to my surprise it solved all those strange situations where objects are kept alive longer than they would have normally. > Anyway, Guido was aiming for the minimal changes that could possibly do real > good. It didn't pretend to reclaim all cycles, and was (IMO) too eager to > punt on the hard issues (the combo of cycles, destructors and resurrection > is a god-awful mess, even in theory; Scheme uses callbacks to dump the > problems back on the users Java has incredibily elaborate rules that are > both bulletproof and unusable; the Boehm collector lets objects with > destructors that are in cycles simply leak, rather than do a wrong thing; > Stroustrup has flip-flopped and most recently argued for Guido's "reclaim > the memory but don't call the destructors" approach, but a member of the C++ > committee told me he's overwhelmingly opposed on this one (I know I would > oppose it)). Not calling the destructor will cause leakage in all objects allocating extra storage, such as lists, instances and probably just about any dynamically sized object there is in Python... solving the problem only half way. Plus you will definitely run into trouble as soon as external resources are involved, e.g. open files or connections to databases. Perhaps we should give more power to the user instead of trying to give him fuzzy feelings about what's happening underneath the hood. Builtin weak references or other indirect ways of accessing objects (e.g. by giving unique names to the involved objects) can solve many of those circ. ref. problems. BTW, I usually use an instrumented Python interpreter to track down circular references: it uses a tracing hook in the allocation/deallocation code of Python instances which is used when Python is run in debugging mode (python -d). The hook calls a function sys.traceinstances (if present) which allows me to keep a track record of all allocated instances: def traceinstances(action,inst): """ Tracing hook. This is called whenever an instances is created and destroyed. action is either 'create' or 'delete'; inst points to the instance object. """ ... If anyone is interested I can post the patch (against Python 1.5). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Tue Sep 21 15:13:19 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 21 Sep 1999 09:13:19 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: Your message of "Tue, 21 Sep 1999 11:05:05 +0200." <37E74A41.107A1243@lemburg.com> References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> Message-ID: <199909211313.JAA09783@eric.cnri.reston.va.us> > Not calling the destructor will cause leakage in all objects > allocating extra storage, such as lists, instances and > probably just about any dynamically sized object there is in > Python... solving the problem only half way. Plus you will > definitely run into trouble as soon as external resources > are involved, e.g. open files or connections to databases. If I remember well, the only destructors not called would be __del__ methods, since the dependencies between to-be-deleted instances are unknown to the collector. Regular (C-level) destructors would of course be called. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at mojam.com Tue Sep 21 17:33:31 1999 From: skip at mojam.com (Skip Montanaro) Date: Tue, 21 Sep 1999 10:33:31 -0500 (CDT) Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <37E74A41.107A1243@lemburg.com> References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> Message-ID: <14311.42315.295460.174012@dolphin.mojam.com> Marc> BTW, I usually use an instrumented Python interpreter to track Marc> down circular references: ... Marc> If anyone is interested I can post the patch (against Python 1.5). That would be interesting to look at. I found my latest circular reference by building Python with Py_DEBUG defined and trudging through the output at the end. Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From mal at lemburg.com Tue Sep 21 18:23:53 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 18:23:53 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> <14311.42315.295460.174012@dolphin.mojam.com> Message-ID: <37E7B119.5A695349@lemburg.com> Skip Montanaro wrote: > > Marc> BTW, I usually use an instrumented Python interpreter to track > Marc> down circular references: ... > > Marc> If anyone is interested I can post the patch (against Python 1.5). > > That would be interesting to look at. I found my latest circular reference > by building Python with Py_DEBUG defined and trudging through the output at > the end. Here it is: --- Objects/orig/classobject.c Thu Jan 1 20:39:04 1998 +++ Objects/classobject.c Sat Aug 8 21:38:57 1998 @@ -403,10 +588,37 @@ PyInstance_New(class, arg, kw) inst = NULL; } Py_DECREF(res); } } + /* sys.traceinstances hook */ + if (Py_DebugFlag && inst) { + PyObject *fct; + + fct = PySys_GetObject("traceinstances"); + if (fct) { + PyObject *v,*arg; + PyObject *error_type, *error_value, *error_traceback; + + /* Save and clear any exception */ + PyErr_Fetch(&error_type,&error_value, + &error_traceback); + PyErr_Clear(); + arg = Py_BuildValue("(sO)","create",(PyObject *)inst); + v = PyEval_CallObject(fct,arg); + Py_DECREF(arg); + if (!v) { + PyErr_Print(); + PyErr_Clear(); + } + else + Py_DECREF(v); + /* Restore exception state */ + PyErr_Restore(error_type,error_value, + error_traceback); + } + } return (PyObject *)inst; } /* Instance methods */ @@ -460,10 +672,31 @@ instance_dealloc(inst) } else Py_DECREF(res); Py_DECREF(del); } + /* sys.traceinstances hook */ + if (Py_DebugFlag) { + PyObject *fct; + + fct = PySys_GetObject("traceinstances"); + if (fct) { + PyObject *v,*arg; + + /* Clear any previous exception */ + PyErr_Clear(); + arg = Py_BuildValue("(sO)","delete",(PyObject *)inst); + v = PyEval_CallObject(fct,arg); + Py_DECREF(arg); + if (!v) { + PyErr_Print(); + PyErr_Clear(); + } + else + Py_DECREF(v); + } + } /* Restore the saved exception and undo the temporary revival */ PyErr_Restore(error_type, error_value, error_traceback); /* Can't use DECREF here, it would cause a recursive call */ if (--inst->ob_refcnt > 0) { #ifdef COUNT_ALLOCS -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Tue Sep 21 18:43:22 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 18:43:22 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> <199909211313.JAA09783@eric.cnri.reston.va.us> Message-ID: <37E7B5AA.6196A3B@lemburg.com> Guido van Rossum wrote: > > > Not calling the destructor will cause leakage in all objects > > allocating extra storage, such as lists, instances and > > probably just about any dynamically sized object there is in > > Python... solving the problem only half way. Plus you will > > definitely run into trouble as soon as external resources > > are involved, e.g. open files or connections to databases. > > If I remember well, the only destructors not called would be __del__ > methods, since the dependencies between to-be-deleted instances are > unknown to the collector. Regular (C-level) destructors would of > course be called. Ok, so low-stuff will not break. But what about e.g. wrappers around these low-level (C-level) objects written in Python, e.g. database abstraction classes ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From tim_one at email.msn.com Wed Sep 22 07:56:05 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 22 Sep 1999 01:56:05 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909211313.JAA09783@eric.cnri.reston.va.us> Message-ID: <000801bf04bf$287e6560$342d153f@tim> [Guido van Rossum] > If I remember well, the only destructors not called would be __del__ > methods, since the dependencies between to-be-deleted instances are > unknown to the collector. Regular (C-level) destructors would of > course be called. Seems to me that the dependencies among to-be-deleted arbitrary C objects are equally unknown to the collector. Assuming there's a fundamental difference between objects implemented in Python and objects implemented in C seems shaky on the face of it. Or if it's not just a convenient assumption, on what is it based? and-what-about-objects-implemented-in-fortran?-ly y'rs - tim From mal at lemburg.com Wed Sep 22 14:47:03 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 22 Sep 1999 14:47:03 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports Message-ID: <37E8CFC7.673E040A@lemburg.com> Hi everybody, I'm currently busy converting my code to use mx.DateTime instead of just DateTime and must say that this is a real pain. Now instead of reviving the __ discussion I'd like to turn to what could be a workable compromise (I think Jim proposed something along these lines already and there was similar code in ni.py). The proposal is really only an addition to the lookup scheme used by the importer. No additional syntax is involved and best of all, it is backward compatible... The current lookup does the following if you want to import a module E from module A.B.C.D: 1. check A.B.C.E 2. check E 3. fail Now instead of failing we could add a lookup method that walks up the package structure: 3. check A.B.E 4. check A.E [5. check E -- already done] 6. fail so that the complete scheme looks like this: 1. check A.B.C.E 2. check E 3. check A.B.E 4. check A.E [5. check E -- already done] 6. fail That way I could leave intra-mx-package imports untouched and still have the convenience of achieving the goal of making my mx subpackages work in the mx context *plus* in the top-level context thus allowing a backward compatible and flexible setup for mx* users. Note that the scheme finds exactly the same modules it did previously, plus perhaps some more (which is intended), and it does not involve any search path hacks. How is that for a compromise ? [Ducking for cover ;-)] -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Wed Sep 22 15:12:40 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 22 Sep 1999 09:12:40 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37E8CFC7.673E040A@lemburg.com> Message-ID: <1274116686-66496233@hypernet.com> [M.-A. Lemburg] > The current lookup does the following if you want to import > a module E from module A.B.C.D: > > 1. check A.B.C.E > 2. check E > 3. fail > > Now instead of failing we could add a lookup method that > walks up the package structure: > > 3. check A.B.E > 4. check A.E > [5. check E -- already done] > 6. fail > > so that the complete scheme looks like this: > > 1. check A.B.C.E > 2. check E > 3. check A.B.E > 4. check A.E > [5. check E -- already done] > 6. fail > > That way I could leave intra-mx-package imports untouched and > still have the convenience of achieving the goal of making my mx > subpackages work in the mx context *plus* in the top-level > context thus allowing a backward compatible and flexible setup > for mx* users. Comment 1: You're just giving yourself headaches by allowing your users to install mx in anything other than the prescribed manner. Comment 2: I generally like this scheme, but think (for consistency and confusion-reduction) that it should go straight up the tree, instead of checking the root second. - Gordon From jim at interet.com Wed Sep 22 15:19:58 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 22 Sep 1999 09:19:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <37E12D4C.2394D311@lyra.org> Message-ID: <37E8D77E.2977C61B@interet.com> Greg Stein wrote: > > Skip Montanaro wrote: > > > > Gordon> In the soon-to-be-published beta version of my installer, I've > > Gordon> got that down to exceptions.py. > > > > Why not just run exceptions.py through Python2C, visually and experimentally > > verify that it works, then ship an exceptions.c as an optional module? Optional module... Good idea. But it is easier to use freeze within the optional module. So I wrote a "boot" built in module, where bootmodule.c is created by the Python program bootmake.py. All this lives in ./Modules. Boot incorporates exceptions.pyc etc. as frozen modules without breaking the current freeze feature. I changed pythonrun.c to load "boot" after sys but before any other imports, but only if it exists as a built in module, otherwise no error. Boot has methods to turn it on and off, and to print its contents. Please take a look at ftp://ftp.interet.com/pub/bootmodule.html I think this is a good solution for how to build in imputil.py. Jim Ahlstrom From guido at CNRI.Reston.VA.US Wed Sep 22 16:53:55 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 22 Sep 1999 10:53:55 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: Your message of "Wed, 22 Sep 1999 01:56:05 EDT." <000801bf04bf$287e6560$342d153f@tim> References: <000801bf04bf$287e6560$342d153f@tim> Message-ID: <199909221453.KAA13321@eric.cnri.reston.va.us> > [Guido van Rossum] > > If I remember well, the only destructors not called would be __del__ > > methods, since the dependencies between to-be-deleted instances are > > unknown to the collector. Regular (C-level) destructors would of > > course be called. [Tim] > Seems to me that the dependencies among to-be-deleted arbitrary C objects > are equally unknown to the collector. Assuming there's a fundamental > difference between objects implemented in Python and objects implemented in > C seems shaky on the face of it. Or if it's not just a convenient > assumption, on what is it based? My assumption was based on the standard Python objects. Cycles necessarily have to include dictionaries or lists (modules, classes and instances link to each other through dictionaries; ditto for function objects; i'm crossing my fingers here for stack frame and traceback objects :-) and I can do things to these to get rid of the links without getting rid of the objects: del L[:] or D.clear(). Third party C objects might have interdependencies similar to those found in Python instances, but my gut feeling is that these aren't as problematic -- e.g. interdependencies between C modules are rare (because the machinery is cumbersome) while they are common between Python modules; and C code isn't susceptible to the problems that Python destructors encounter when the modules they import have already been destroyed. I know, it's an unusual amount of handwaving... --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at lemburg.com Wed Sep 22 17:13:06 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 22 Sep 1999 17:13:06 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> Message-ID: <37E8F202.2A27508D@lemburg.com> Gordon McMillan wrote: > > [M.-A. Lemburg] > > The current lookup does the following if you want to import > > a module E from module A.B.C.D: > > > > 1. check A.B.C.E > > 2. check E > > 3. fail > > > > Now instead of failing we could add a lookup method that > > walks up the package structure: > > > > 3. check A.B.E > > 4. check A.E > > [5. check E -- already done] > > 6. fail > > > > so that the complete scheme looks like this: > > > > 1. check A.B.C.E > > 2. check E > > 3. check A.B.E > > 4. check A.E > > [5. check E -- already done] > > 6. fail > > > > That way I could leave intra-mx-package imports untouched and > > still have the convenience of achieving the goal of making my mx > > subpackages work in the mx context *plus* in the top-level > > context thus allowing a backward compatible and flexible setup > > for mx* users. > > Comment 1: You're just giving yourself headaches by allowing > your users to install mx in anything other than the prescribed > manner. Actually, I'm trying to provide them a way to smoothly switch from the old setup to the new one. This includes myself, of course ;-). > Comment 2: I generally like this scheme, but think (for > consistency and confusion-reduction) that it should go straight > up the tree, instead of checking the root second. That would probably break code because the search could find some other module having the same name as a top-level one. OTOH, perhaps that situation is not all the common to fear too much about it. Walking up all the way would certainly be easier to explain to a 12-year old ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Wed Sep 22 17:51:49 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 22 Sep 1999 11:51:49 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: Your message of "Wed, 22 Sep 1999 17:13:06 +0200." <37E8F202.2A27508D@lemburg.com> References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> Message-ID: <199909221551.LAA13721@eric.cnri.reston.va.us> > > Comment 2: I generally like this scheme, but think (for > > consistency and confusion-reduction) that it should go straight > > up the tree, instead of checking the root second. > > That would probably break code because the search could > find some other module having the same name as a top-level > one. OTOH, perhaps that situation is not all the common to > fear too much about it. > > Walking up all the way would certainly be easier to explain to > a 12-year old ;-) Yes, please. Do the long-term understandable thing here. I expect not too many packages have defined subpackages (or submodules) whose name conflicts with a standard library module, so you ought to be pretty safe here! --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at digicool.com Wed Sep 22 18:25:25 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 22 Sep 1999 12:25:25 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> Message-ID: <37E902F5.27630A90@digicool.com> Guido van Rossum wrote: > > > > Comment 2: I generally like this scheme, but think (for > > > consistency and confusion-reduction) that it should go straight > > > up the tree, instead of checking the root second. > > > > That would probably break code because the search could > > find some other module having the same name as a top-level > > one. OTOH, perhaps that situation is not all the common to > > fear too much about it. > > > > Walking up all the way would certainly be easier to explain to > > a 12-year old ;-) > > Yes, please. Do the long-term understandable thing here. I expect > not too many packages have defined subpackages (or submodules) whose > name conflicts with a standard library module, so you ought to be > pretty safe here! Walking straight up the tree is my preference. I think it is very natural. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward at cnri.reston.va.us Wed Sep 22 19:26:12 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Wed, 22 Sep 1999 13:26:12 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <199909221551.LAA13721@eric.cnri.reston.va.us>; from Guido van Rossum on Wed, Sep 22, 1999 at 11:51:49AM -0400 References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> Message-ID: <19990922132611.C6623@cnri.reston.va.us> On 22 September 1999, Guido van Rossum said: > Yes, please. Do the long-term understandable thing here. I expect > not too many packages have defined subpackages (or submodules) whose > name conflicts with a standard library module, so you ought to be > pretty safe here! Especially since doing so doesn't work: for example, the distutils.errors module started life as distutils.exceptions. That changed pretty quickly, once I realized why putting import exceptions into other distutils modules didn't work -- obviously it didn't find my distutils.exceptions. Arguably I should have used an absolute import, but what the heck. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal at lemburg.com Fri Sep 24 11:10:59 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 24 Sep 1999 11:10:59 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> <19990922132611.C6623@cnri.reston.va.us> Message-ID: <37EB4023.28DB43A3@lemburg.com> Greg Ward wrote: > > On 22 September 1999, Guido van Rossum said: > > Yes, please. Do the long-term understandable thing here. I expect > > not too many packages have defined subpackages (or submodules) whose > > name conflicts with a standard library module, so you ought to be > > pretty safe here! > > Especially since doing so doesn't work: for example, the > distutils.errors module started life as distutils.exceptions. That > changed pretty quickly, once I realized why putting > > import exceptions > > into other distutils modules didn't work -- obviously it didn't find my > distutils.exceptions. Arguably I should have used an absolute import, > but what the heck. Wow, so many positive answers -- not bad after that last round of relative imports ;-) Ok, then I'll use the walk-me-up approach. That'll be coded into a PathImporter class I'm writing for imputil which will try to mimic the standard behaviour as much as possible (to be released in a few weeks after my vacation). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mhammond at skippinet.com.au Fri Sep 24 14:36:10 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 24 Sep 1999 22:36:10 +1000 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37EB4023.28DB43A3@lemburg.com> Message-ID: <000501bf0689$63614af0$0801a8c0@bobcat> > try to mimic the standard behaviour as much as possible (to be > released in a few weeks after my vacation). no no Marc - you should know the rules by now - you release it mere hours _before_ your vacation :-) Enjoy! Mark. From mal at lemburg.com Sun Sep 26 21:34:15 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun, 26 Sep 1999 21:34:15 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <000501bf0689$63614af0$0801a8c0@bobcat> Message-ID: <37EE7537.1075369C@lemburg.com> Mark Hammond wrote: > > > try to mimic the standard behaviour as much as possible (to be > > released in a few weeks after my vacation). > > no no Marc - you should know the rules by now - you release it mere > hours _before_ your vacation :-) Enjoy! Ya, well ;-) I'll try my best... I already have something working but it doesn't do the win32 + mac magic yet because that'll require some additions to the builtin imp module. Also, I found that it is rather slow when compared to the builtin one. Caches can speed this up a little, but I still haven't achieved the same performance. BTW, while hacking along I found a few things that might be worth discussing w/r to a general import hook scheme: Currently, the imputil apporach uses a simple chaining technique. Unfortunately, it doesn't allow inspecting the chain for already loaded hooks, so the same type of hook could be loaded more than once. Also, there are at least two types of hooks: 1. hooks that redirect the import to some other data source 2. hooks that modify the way modules are searched Since the first variant may well also be suited to used by the second, the simple chaining method probably won't be powerful enough to handle it. I think what we really need is a set of register/deregister APIs + some framework to differentiate between the two hook types (and possibly other variants). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 27 12:19:25 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 27 Sep 1999 12:19:25 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <000501bf0689$63614af0$0801a8c0@bobcat> <37EE7537.1075369C@lemburg.com> Message-ID: <37EF44AD.7C99D9C5@lemburg.com> M.-A. Lemburg wrote: > > BTW, while hacking along I found a few things that might be worth > discussing w/r to a general import hook scheme: > > Currently, the imputil apporach uses a simple chaining technique. > Unfortunately, it doesn't allow inspecting the chain for already > loaded hooks, so the same type of hook could be loaded more than > once. > > Also, there are at least two types of hooks: > > 1. hooks that redirect the import to some other data source > > 2. hooks that modify the way modules are searched > > Since the first variant may well also be suited to used by > the second, the simple chaining method probably won't be > powerful enough to handle it. > > I think what we really need is a set of register/deregister > APIs + some framework to differentiate between the two > hook types (and possibly other variants). Another quirk that I think needs fixing: When I issues an import: import mx.DateTime the whole import is handled by the importer installed at the start of the import. It is not possible to install a different importer e.g. in mx/__init__.py to handle the rest of the import (in this case the import of subpackage DateTime). I think that the importer should honor the __importer__ function (this is set by imputil) if present to let it continue the import of subsequent elements in the dotted name. Aside: Perhaps this is getting too technical for this list... should I start an egroups mailing list for defining a new and more flexible import mechanism ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Mon Sep 27 18:53:30 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Mon, 27 Sep 1999 12:53:30 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37EF44AD.7C99D9C5@lemburg.com> Message-ID: <1273671699-5802298@hypernet.com> M.-A. Lemburg wrote: [msg 1] > Currently, the imputil apporach uses a simple chaining > technique. Unfortunately, it doesn't allow inspecting the chain > for already loaded hooks, so the same type of hook could be > loaded more than once. I was hoping Greg would jump in, but since he hasn't - You're associating the hook with the strategy. That's the old style. The imputil style is to associate the hook with the actual stuff being managed. The strategy is a property of the hook. > Also, there are at least two types of hooks: > > 1. hooks that redirect the import to some other data source > > 2. hooks that modify the way modules are searched > > Since the first variant may well also be suited to used by the > second, the simple chaining method probably won't be powerful > enough to handle it. The top level question is "is it mine to import?". Greg provides a framework that makes it easy to use alternate data sources, and alternate ways of finding things but that's not really the key thing. You're a "good" importer if you can (when appropriate) way "no it's not mine" efficiently. [msg 2] > Another quirk that I think needs fixing: > > When I issues an import: > > import mx.DateTime > > the whole import is handled by the importer installed at > the start of the import. It is not possible to install a > different importer e.g. in mx/__init__.py to handle the rest of > the import (in this case the import of subpackage DateTime). I > think that the importer should honor the __importer__ function > (this is set by imputil) if present to let it continue the import > of subsequent elements in the dotted name. Sure you can. Your first importer is the "mx" importer. It has a dict of sub-importers. When mx/DateTime/__init__.py runs, it puts itself into that dict. The importer chain is now a tree. This means, I think, that a "general" relative-path importer (ie, one that uses the default PYTHONPATH strategy), should be careful to install itself as the penultimate importer in the chain, (ie, the last before __builtin__.imp). But putting a relative-path search strategy into the "mx" importer is fine if it can quickly determine that the target is / is not a valid name in the "mx" namespace. - Gordon From mal at lemburg.com Tue Sep 28 00:59:57 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 28 Sep 1999 00:59:57 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1273671699-5802298@hypernet.com> Message-ID: <37EFF6ED.1693ACA3@lemburg.com> Gordon McMillan wrote: > > M.-A. Lemburg wrote: > [msg 1] > > Currently, the imputil apporach uses a simple chaining > > technique. Unfortunately, it doesn't allow inspecting the chain > > for already loaded hooks, so the same type of hook could be > > loaded more than once. > > I was hoping Greg would jump in, but since he hasn't - > > You're associating the hook with the strategy. That's the old > style. The imputil style is to associate the hook with the > actual stuff being managed. The strategy is a property of the > hook. I know, but there still is no way to query what kind of hooks are already loaded and what is worse, you cannot unload or reorder them. I'd suggest using a list of hooks which are then traversed in the order they appear in the list, e.g. __importers__ = [DirectoryImporter('/usr/local/lib/python1.5'), ArchiveImporter('/usr/local/lib/app.pyz'), WebImporter('http://www.python.org/pylib/'), PathImporter(('~/bin','~/lib'))] This also has the advantage of being able to easily query the importers during debugging and eliminates the need to have a predefined attribute naming scheme (such as the one imputil uses). > > Also, there are at least two types of hooks: > > > > 1. hooks that redirect the import to some other data source > > > > 2. hooks that modify the way modules are searched > > > > Since the first variant may well also be suited to used by the > > second, the simple chaining method probably won't be powerful > > enough to handle it. > > The top level question is "is it mine to import?". Greg provides > a framework that makes it easy to use alternate data sources, > and alternate ways of finding things but that's not really the > key thing. You're a "good" importer if you can (when > appropriate) way "no it's not mine" efficiently. It does a good job at this, but doesn't really separate lookup and loading of code too well. Everything is packaged into one single method (.get_code()) which is not always flexible enough, e.g. it wasn't possible to implement the walk-up-the-dotted-name scheme using modifications to .get_code() alone. Also, I can see many uses where you combine a lookup hook (e.g. for loading modules across the web) with a filtering hook (e.g. one which checks a module signature). This should go into the framework as well, IMHO... e.g. by having two methods .find_code() and .make_module() (like the builtin importer). > [msg 2] > > Another quirk that I think needs fixing: > > > > When I issues an import: > > > > import mx.DateTime > > > > the whole import is handled by the importer installed at > > the start of the import. It is not possible to install a > > different importer e.g. in mx/__init__.py to handle the rest of > > the import (in this case the import of subpackage DateTime). I > > think that the importer should honor the __importer__ function > > (this is set by imputil) if present to let it continue the import > > of subsequent elements in the dotted name. > > Sure you can. Your first importer is the "mx" importer. It has a > dict of sub-importers. When mx/DateTime/__init__.py runs, it > puts itself into that dict. The importer chain is now a tree. The problem is that the special importer has to be installed *prior* to doing the mx.DateTime import, because otherwise the importer will not take control over the DateTime subpackage (unless I tell it to do so explicitly, which is not really what I want to have to do). > This means, I think, that a "general" relative-path importer (ie, > one that uses the default PYTHONPATH strategy), should be > careful to install itself as the penultimate importer in the chain, > (ie, the last before __builtin__.imp). But putting a relative-path > search strategy into the "mx" importer is fine if it can quickly > determine that the target is / is not a valid name in the "mx" > namespace. Exactly... and this brings us back to the importer list I mentioned above. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein at lyra.org Tue Sep 28 23:57:34 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 28 Sep 1999 14:57:34 -0700 (PDT) Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <1273671699-5802298@hypernet.com> Message-ID: On Mon, 27 Sep 1999, Gordon McMillan wrote: > M.-A. Lemburg wrote: > [msg 1] > > Currently, the imputil apporach uses a simple chaining > > technique. Unfortunately, it doesn't allow inspecting the chain > > for already loaded hooks, so the same type of hook could be > > loaded more than once. > > I was hoping Greg would jump in, but since he hasn't - I'm in a middle of a move back to CA. Unpacking now... > You're associating the hook with the strategy. That's the old > style. The imputil style is to associate the hook with the > actual stuff being managed. The strategy is a property of the > hook. Quite true. The chaining is simply an artifact of what has been installed as the import hook. I've always envisioned the potential for a "Importer Manager" that installs just like any other hook, but provides higher-level functions for importers to install themselves. The manager simply delegates the get_code() function to the sub-importers. Of course, the manager could use whatever technique to improve the speed of Importer selection. With respect to speed, I think the main point is to realize that the imputil technique is not inherently slow. It just depends on how you design your Importer subclasses -- do you install one or a hundred Importers? The imputil scheme is more about simplifying how people hook into the process (implement get_code() rather than a load/import combo). It also provides a simple capability (chaining) to allow *multiple* hooks to be installed. > > Also, there are at least two types of hooks: > > > > 1. hooks that redirect the import to some other data source > > > > 2. hooks that modify the way modules are searched Just one way -- your second is a variant of the first. "other data source" is a functional superset which includes searching. Importers don't simply alter searching -- they must perform the actual import (from wherever). This is the big change in mindset from the "ihooks" method -- find it and import it on the spot. The net effect is an Importer either imports a module or it doesn't (and the system can fallback to try another Importer). [ one the examples that people always like to specify was importing via URL which was actually quite difficult to use in the old scheme -- how do you separate an HTTP GET into a find/load step? Effectively, you had to double-fetch, or you had to place the whole module (which you retrieved during the find step) into your context for passing to the load. The other issue was the distinct semantics also implied that you could separate the functions -- I believe that to be quite unnecessary functionality. ] > > Since the first variant may well also be suited to used by the > > second, the simple chaining method probably won't be powerful > > enough to handle it. > > The top level question is "is it mine to import?". Greg provides > a framework that makes it easy to use alternate data sources, > and alternate ways of finding things but that's not really the > key thing. You're a "good" importer if you can (when > appropriate) way "no it's not mine" efficiently. Very true! > [msg 2] > > Another quirk that I think needs fixing: > > > > When I issues an import: > > > > import mx.DateTime > > > > the whole import is handled by the importer installed at > > the start of the import. It is not possible to install a > > different importer e.g. in mx/__init__.py to handle the rest of > > the import (in this case the import of subpackage DateTime). I > > think that the importer should honor the __importer__ function > > (this is set by imputil) if present to let it continue the import > > of subsequent elements in the dotted name. > > Sure you can. Your first importer is the "mx" importer. It has a > dict of sub-importers. When mx/DateTime/__init__.py runs, it > puts itself into that dict. The importer chain is now a tree. Gordon's on top of it here... :-) Yes, it is simply a matter of perspective on the import process. An importer does not have to be a static entity. It also can be much more than a way to search a path... it can be highly dynamic and flexible. Whatever you like. Just implement get_code() to map a module "mx.DateTime" to a code/module object. There are a bazillion ways to do that :-) > This means, I think, that a "general" relative-path importer (ie, > one that uses the default PYTHONPATH strategy), should be > careful to install itself as the penultimate importer in the chain, > (ie, the last before __builtin__.imp). But putting a relative-path > search strategy into the "mx" importer is fine if it can quickly > determine that the target is / is not a valid name in the "mx" > namespace. Part of the Importer work was done to satisfy importing modules from the COM+ namespace. I wanted to be able to say "import COM.foo.bar". The importer would handle all "COM." imports and delegate the "foo.bar" to the underlying Python/COM framework. In other words... yes, the Importer scheme should work *very* well for the "whatever...." type of module namespace. Cheers, -g -- Greg Stein, http://www.lyra.org/ From da at ski.org Wed Sep 29 02:02:20 1999 From: da at ski.org (David Ascher) Date: Tue, 28 Sep 1999 17:02:20 -0700 (Pacific Daylight Time) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) Message-ID: ---------- Forwarded message ---------- On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > It's not available online (it was written on a typewriter), so I'd be > happy to send a hard copy of it to anyone who raises a hand in private > mail (or cannot find LNCS 60). I then asked him for a copy stating: > If you send it to me, I can OCR it and make it available online. and he generously sent it to me. The problem is that I hadn't noticed the length of the manuscript. It's over a hundred pages, and the copy is nth generation, making OCR pretty much useless. So, if anyone wants a copy, I can make and send copies (which would make most sense for folks in the US -- sending things from France isn't cheap). So far, it's good reading. Funny to see "file name" in quotes in the text. --david From mal at lemburg.com Wed Sep 29 12:47:08 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 29 Sep 1999 12:47:08 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: Message-ID: <37F1EE2C.52F9A7E1@lemburg.com> Greg Stein wrote: > > On Mon, 27 Sep 1999, Gordon McMillan wrote: > > M.-A. Lemburg wrote: > > [msg 1] > > > Currently, the imputil apporach uses a simple chaining > > > technique. Unfortunately, it doesn't allow inspecting the chain > > > for already loaded hooks, so the same type of hook could be > > > loaded more than once. > > You're associating the hook with the strategy. That's the old > > style. The imputil style is to associate the hook with the > > actual stuff being managed. The strategy is a property of the > > hook. > > Quite true. The chaining is simply an artifact of what has been installed > as the import hook. I've always envisioned the potential for a "Importer > Manager" that installs just like any other hook, but provides higher-level > functions for importers to install themselves. The manager simply > delegates the get_code() function to the sub-importers. Of course, the > manager could use whatever technique to improve the speed of Importer > selection. > > With respect to speed, I think the main point is to realize that the > imputil technique is not inherently slow. It just depends on how you > design your Importer subclasses -- do you install one or a hundred > Importers? > > The imputil scheme is more about simplifying how people hook into the > process (implement get_code() rather than a load/import combo). It also > provides a simple capability (chaining) to allow *multiple* hooks to be > installed. As I wrote in my reply to Gordon, this setup has some drawbacks which an "Import Manager" could easily solve, e.g. by using a list of importers. > > > Also, there are at least two types of hooks: > > > > > > 1. hooks that redirect the import to some other data source > > > > > > 2. hooks that modify the way modules are searched > > Just one way -- your second is a variant of the first. "other data source" > is a functional superset which includes searching. Importers don't simply > alter searching -- they must perform the actual import (from wherever). Yes, I was just argueing for two types of functionality, not the old scheme. E.g. the Import Manager could provide a set of filters which implement signature checks or know how to un-gzip code plus a set of lookup functions for scanning directories or zip archives. I would like the importers to take advantage of such functionality. Of course, all of this could be implemented in form of classes which the importers then use as mixin classes. > This is the big change in mindset from the "ihooks" method -- find it and > import it on the spot. The net effect is an Importer either imports a > module or it doesn't (and the system can fallback to try another > Importer). > > [ one the examples that people always like to specify was importing via > URL which was actually quite difficult to use in the old scheme -- how do > you separate an HTTP GET into a find/load step? Effectively, you had to > double-fetch, or you had to place the whole module (which you retrieved > during the find step) into your context for passing to the load. The other > issue was the distinct semantics also implied that you could separate the > functions -- I believe that to be quite unnecessary functionality. ] .get_code() is fine for these kind of tasks, but there are some other areas (such as lazy imports) which work better using the split setup. This is pretty easy to implement btw, just have the Import Manager check whether the importer provides .get_code() and then have it revert to using .find_module(), .load_module() if it doesn't. The more I think about it, the more I like the idea of an Import Manager instead of the chaining approach. > > > Since the first variant may well also be suited to used by the > > > second, the simple chaining method probably won't be powerful > > > enough to handle it. > > > > The top level question is "is it mine to import?". Greg provides > > a framework that makes it easy to use alternate data sources, > > and alternate ways of finding things but that's not really the > > key thing. You're a "good" importer if you can (when > > appropriate) way "no it's not mine" efficiently. > > Very true! > > > [msg 2] > > > Another quirk that I think needs fixing: > > > > > > When I issues an import: > > > > > > import mx.DateTime > > > > > > the whole import is handled by the importer installed at > > > the start of the import. It is not possible to install a > > > different importer e.g. in mx/__init__.py to handle the rest of > > > the import (in this case the import of subpackage DateTime). I > > > think that the importer should honor the __importer__ function > > > (this is set by imputil) if present to let it continue the import > > > of subsequent elements in the dotted name. > > > > Sure you can. Your first importer is the "mx" importer. It has a > > dict of sub-importers. When mx/DateTime/__init__.py runs, it > > puts itself into that dict. The importer chain is now a tree. > > Gordon's on top of it here... :-) Yes, it is simply a matter of > perspective on the import process. An importer does not have to be a > static entity. It also can be much more than a way to search a path... it > can be highly dynamic and flexible. Whatever you like. Just implement > get_code() to map a module "mx.DateTime" to a code/module object. There > are a bazillion ways to do that :-) Except that they don't work due to the fact that the builtin importer is not recursively using __import__ for the imports. An Import Manager would help with this too :-) > > This means, I think, that a "general" relative-path importer (ie, > > one that uses the default PYTHONPATH strategy), should be > > careful to install itself as the penultimate importer in the chain, > > (ie, the last before __builtin__.imp). But putting a relative-path > > search strategy into the "mx" importer is fine if it can quickly > > determine that the target is / is not a valid name in the "mx" > > namespace. > > Part of the Importer work was done to satisfy importing modules from the > COM+ namespace. I wanted to be able to say "import COM.foo.bar". The > importer would handle all "COM." imports and delegate the "foo.bar" to the > underlying Python/COM framework. > > In other words... yes, the Importer scheme should work *very* well for > the "whatever...." type of module namespace. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at interet.com Wed Sep 29 15:59:11 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 29 Sep 1999 09:59:11 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1273671699-5802298@hypernet.com> <37EFF6ED.1693ACA3@lemburg.com> Message-ID: <37F21B2F.DD65738F@interet.com> "M.-A. Lemburg" wrote: > > I know, but there still is no way to query what kind of hooks > are already loaded and what is worse, you cannot unload or reorder > them. I think this is a valid point. I am interested in hooks to read modules from a file archive. If you are a developer, it is necessary to turn this hook OFF, so that you can revert to the usual directory tree where your current source is. I am solving this by leaving a global variable "Importer" in sitecustomize, and calling sitecustomize.Importer.enable(0). This works, but it might be useful if imputil could de-install a hook as well as install it. Jim Ahlstrom From mhammond at skippinet.com.au Thu Sep 30 02:16:44 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 30 Sep 1999 10:16:44 +1000 Subject: [Python-Dev] getopt helper? Message-ID: <007d01bf0ad5$14b6cf60$0801a8c0@bobcat> A little thought I have had a few times now, so it is time to pass it on for comment. Basically _every_ time I use getopt, I write code like this: import getopt try: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") except getopt.error, why: print why print usage % (os.path.basename(sys.argv[0],)) sys.exit(1) Every single time. I have never used getopt without this code. How about we put this functionality into getopt itself? It could be triggered either by adding a new "usage" param defaulting to None, or by adding a new entry point. ie: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:", usage=usage) or opts, args = getopt.getoptex(sys.argv[1:], "slc:d:e:", usage=usage) I know it is fairly trivial, but IMO is such a useful module and the pattern is used so regularly that is seems to make sense to add it. Any thoughts? If it is seen favourably, how should we spell it? Mark. From mal at lemburg.com Thu Sep 30 09:29:24 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 09:29:24 +0200 Subject: [Python-Dev] getopt helper? References: <007d01bf0ad5$14b6cf60$0801a8c0@bobcat> Message-ID: <37F31154.4FDA472C@lemburg.com> Mark Hammond wrote: > > A little thought I have had a few times now, so it is time to pass it > on for comment. > > Basically _every_ time I use getopt, I write code like this: > > import getopt > try: > opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") > except getopt.error, why: > print why > print usage % (os.path.basename(sys.argv[0],)) > sys.exit(1) > > Every single time. I have never used getopt without this code. > > How about we put this functionality into getopt itself? > > It could be triggered either by adding a new "usage" param defaulting > to None, or by adding a new entry point. ie: > > opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:", usage=usage) > or > opts, args = getopt.getoptex(sys.argv[1:], "slc:d:e:", usage=usage) > > I know it is fairly trivial, but IMO is such a useful module and the > pattern is used so regularly that is seems to make sense to add it. > > Any thoughts? If it is seen favourably, how should we spell it? Why not just add a higher level interface ? Something like CommandLine.py which is included in mxDateTime ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mhammond at skippinet.com.au Thu Sep 30 12:09:53 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 30 Sep 1999 20:09:53 +1000 Subject: [Python-Dev] getopt helper? In-Reply-To: <37F31154.4FDA472C@lemburg.com> Message-ID: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> > > Basically _every_ time I use getopt, I write code like this: > Why not just add a higher level interface ? Something > like CommandLine.py which is included in mxDateTime ? Because _every_ time I use getopt, I write code like that :-) A higher level interface would maybe be handy, but it simply occurred to me that every time I used the module I used that pattern. I just got sick of typing it all the time and wondered if it struck a chord with anyone else (and I dont have or use a general purpose "mhutil" module :-) Im really just trying to save myself 10 lines of boilerplate coding, not introduce a new standard module :-) Mark. From Vladimir.Marangozov at inrialpes.fr Thu Sep 30 13:46:48 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Thu, 30 Sep 1999 12:46:48 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: from "David Ascher" at "Sep 28, 99 05:02:20 pm" Message-ID: <199909301146.MAA23138@pukapuka.inrialpes.fr> David Ascher wrote: > > ---------- Forwarded message ---------- > On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > > > It's not available online (it was written on a typewriter), so I'd be > > happy to send a hard copy of it to anyone who raises a hand in private > > mail (or cannot find LNCS 60). > > I then asked him for a copy stating: > > > If you send it to me, I can OCR it and make it available online. > > and he generously sent it to me. The problem is that I hadn't noticed the > length of the manuscript. It's over a hundred pages, and the copy is nth > generation, making OCR pretty much useless. If I find some spare time, I'll do it. This paper is a classic in Comp Sci that we kindly invite our students to read. In research language, we say that Saltzer has made "the turn around the clock" regarding this issue, putting a period on it (i.e. there's hardly something more to say). It's interesting, however, to see how the theory was applied on Python and to establish the fairly easy analogy of the binding model. This analogy proves the good design choices Guido has made, but also reveals some weaknesses or the incompleteness of the current implementation. I hope to discuss this for Python 2 in due time and perhaps settle on a compromise which trades genericity for performance. The naming/binding problem drives the whole implementation logic in Python (objects, classes, scopes, etc.). > > So, if anyone wants a copy, I can make and send copies (which would make > most sense for folks in the US -- sending things from France isn't cheap). I have 2 more copies lying around, ready to be sent. Give me an adress. The first 2 adresses win a hard copy, no matter the location (don't worry about mail costs.) -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From mal at lemburg.com Thu Sep 30 13:22:27 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 13:22:27 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> Message-ID: <37F347F3.6D18C285@lemburg.com> Mark Hammond wrote: > > > > Basically _every_ time I use getopt, I write code like this: > > > Why not just add a higher level interface ? Something > > like CommandLine.py which is included in mxDateTime ? > > Because _every_ time I use getopt, I write code like that :-) > > A higher level interface would maybe be handy, but it simply occurred > to me that every time I used the module I used that pattern. I just > got sick of typing it all the time and wondered if it struck a chord > with anyone else (and I dont have or use a general purpose "mhutil" > module :-) Im really just trying to save myself 10 lines of > boilerplate coding, not introduce a new standard module :-) Just a thought :-) I wrote the CommandLine.py for pretty much the same reason: I have quite a few command line apps lying in my bin/ dir and they all did some kind of getopt/sys.argv tricks to handle the input... way to confusing and not easy to maintain. So I decided to take an OO-approach to have them use the same interface with nice help output and to reduce the coding effort. As an example taken from mxDateTime: #!/usr/local/bin/python -u """ Simple Forking Alarm Sample Application for DateTime types and CommandLine. Only works on OSes which support os.fork(). Author: Marc-Andre Lemburg, mailto:mal at lemburg.com """ import time,sys,os from mx.DateTime import * from CommandLine import Application,ArgumentOption class Alarm(Application): header = "Simple Forking Alarm" options = [ArgumentOption('-s', 'set the alarm to now + arg seconds'), ArgumentOption('-m', 'set the alarm to now + arg minutes'), ArgumentOption('-a', 'set the alarm to ring at arg (hh:mm)'), ] version = '0.1' def main(self): atime = now() + (self.values['-s'] or self.values['-m'] * 60 or self.values['-h'] * 3600) * oneSecond abs = self.values['-a'] if abs: atime = strptime(abs,'%H:%M',today(second=0)) if atime < now(): print 'Alarm time has expired...' return print 'Alarm will ring at',atime if not os.fork(): time.sleep((atime - now()).seconds) alarm() os._exit(0) def alarm(): """ Ring alarm """ for i in range(10): sys.stdout.write('\007') sys.stdout.flush() time.sleep(0.2) if __name__ == '__main__': Alarm() Here's the help output this produces: /home/lemburg> alarm -h ------------------------------------------------------------------------ Simple Forking Alarm ------------------------------------------------------------------------ Synopsis: alarm [option] files... Options and default settings: -s arg set the alarm to now + arg seconds -m arg set the alarm to now + arg minutes -a arg set the alarm to ring at arg (hh:mm) -h show this help text --help show this help text --copyright show copyright --examples show examples of usage Version: 0.1 -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Thu Sep 30 14:31:04 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 30 Sep 1999 08:31:04 -0400 Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: Your message of "Thu, 30 Sep 1999 12:46:48 BST." <199909301146.MAA23138@pukapuka.inrialpes.fr> References: <199909301146.MAA23138@pukapuka.inrialpes.fr> Message-ID: <199909301231.IAA11203@eric.cnri.reston.va.us> I've started reading the paper and agree that it's very good! > It's interesting, however, to see how the theory was applied on Python and > to establish the fairly easy analogy of the binding model. This analogy > proves the good design choices Guido has made, but also reveals some > weaknesses or the incompleteness of the current implementation. I hope > to discuss this for Python 2 in due time and perhaps settle on a compromise > which trades genericity for performance. The naming/binding problem drives > the whole implementation logic in Python (objects, classes, scopes, etc.). I'd like to hear what those weaknesses are in your eyes. I can think of two areas myself: (1) sys.path, $PYTHONPATH etc.; (2) class/instance attributes in the context of subclassing and evolution of the base class. (I don't expect the paper to take a stance on nested scopes.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at CNRI.Reston.VA.US Thu Sep 30 14:35:23 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 30 Sep 1999 08:35:23 -0400 Subject: [Python-Dev] getopt helper? In-Reply-To: Your message of "Thu, 30 Sep 1999 13:22:27 +0200." <37F347F3.6D18C285@lemburg.com> References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> Message-ID: <199909301235.IAA11217@eric.cnri.reston.va.us> [Mark] > > > > Basically _every_ time I use getopt, I write code like this: [Marc-Andre] > > > Why not just add a higher level interface ? Something > > > like CommandLine.py which is included in mxDateTime ? [Mark] > > Because _every_ time I use getopt, I write code like that :-) [Marc-Andre] > I wrote the CommandLine.py for pretty much the same reason: Marc-Andre, you're not hearing what Mark is saying. He wants a change to the standard library, and he knows that small additions to existing modules there stand a better chance of adoption than new modules. I personally liked the idea of getoptex() best, except I would call it getopt_or_die(). If the usage message is omitted it can synthesize one from the (short and long) options arguments and sys.argv[0] (the latter being a bit controversial, but it's just a default). Hmm... Perhaps getopt_or_die() shouldn't take the args argument but extract sys.argv[1:] itself? --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Thu Sep 30 15:22:58 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 30 Sep 1999 15:22:58 +0200 Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> Message-ID: <007501bf0b46$eb2f38e0$f29b12c2@secret.pythonware.com> as reported on slashdot: http://www.perl.com/pub/1999/09/topaz.html (topaz is also the code name for our "rewrite Tkinter in Python", but that's another story ;-) From gward at cnri.reston.va.us Thu Sep 30 16:15:50 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Thu, 30 Sep 1999 10:15:50 -0400 Subject: [Python-Dev] getopt helper? In-Reply-To: <199909301235.IAA11217@eric.cnri.reston.va.us>; from Guido van Rossum on Thu, Sep 30, 1999 at 08:35:23AM -0400 References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <19990930101549.A10529@cnri.reston.va.us> On 30 September 1999, Guido van Rossum said: > Marc-Andre, you're not hearing what Mark is saying. He wants a change > to the standard library, and he knows that small additions to existing > modules there stand a better chance of adoption than new modules. > > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). Gasp! Guido hath drunk from the poisoned well of the Great Camel and it doth infest his thinking! "or die" indeed -- not an idiom I've seen since the last time I wrote some Perl code (umm, last week?). > If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? Whatever we call it, it should be able to take an explicit argument list, and only default to sys.argv[1:]. What if I want to parse options from an environment variable or a config file? I also like the "don't clobber sys.argv, but return the modified version instead" model -- it's nice to keep a pristine copy of the original argument list! Another problem with getopt is that it doesn't correlate long and short options. I wrote distutils.fancy_getopt (download your copy today! hurry, don't delay -- at this price, it WON'T LAST LONG!) to address this, and to maybe someday do something with help text. On the other hand, don't listen to me -- I tend to write mammoth, bloated, all-singing, all-dancing command-line parsing modules for every new language I encounter. They get more insane with each iteration. I have yet to top my Getopt::Tabular for Perl, though; see http://search.cpan.org/doc/GWARD/Getopt-Tabular-0.3/Tabular.pod if you've ever wondered how far this sort of thing can be taken in a high-level, dynamically typed language. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fredrik at pythonware.com Thu Sep 30 16:18:15 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 30 Sep 1999 16:18:15 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <00b301bf0b4e$a3ac5cc0$f29b12c2@secret.pythonware.com> > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? sounds reasonable. what are you waiting for ;-) From mal at lemburg.com Thu Sep 30 17:28:31 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 17:28:31 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <37F3819F.2C713970@lemburg.com> Guido van Rossum wrote: > > [Mark] > > > > > Basically _every_ time I use getopt, I write code like this: > > [Marc-Andre] > > > > Why not just add a higher level interface ? Something > > > > like CommandLine.py which is included in mxDateTime ? > > [Mark] > > > Because _every_ time I use getopt, I write code like that :-) > > [Marc-Andre] > > I wrote the CommandLine.py for pretty much the same reason: > > Marc-Andre, you're not hearing what Mark is saying. He wants a change > to the standard library, and he knows that small additions to existing > modules there stand a better chance of adoption than new modules. Oh, I did get the idea... just wanted to plug my module here in a take-it-or-leave-it way ;-) I usually put such things into my lib/ dir for Python to find -- no need to make them a standard. > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? Better not: it's sometimes very useful to call the main(args) function of a script in interactive mode which then passes the args list to getopt(). How about adding something like: def getoptex(...,args=None,helptext='Read the source, Luke ;-)'): if args is None: args = sys.argv[1:] ... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Vladimir.Marangozov at inrialpes.fr Thu Sep 30 21:24:53 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Thu, 30 Sep 1999 20:24:53 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: <199909301231.IAA11203@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 30, 99 08:31:04 am" Message-ID: <199909301924.UAA32482@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > > I've started reading the paper and agree that it's very good! > > > It's interesting, however, to see how the theory was applied on Python and > > to establish the fairly easy analogy of the binding model. This analogy > > proves the good design choices Guido has made, but also reveals some > > weaknesses or the incompleteness of the current implementation. I hope > > to discuss this for Python 2 in due time and perhaps settle on a compromise > > which trades genericity for performance. The naming/binding problem drives > > the whole implementation logic in Python (objects, classes, scopes, etc.). > > I'd like to hear what those weaknesses are in your eyes. I can think > of two areas myself: (1) sys.path, $PYTHONPATH etc.; (2) > class/instance attributes in the context of subclassing and evolution > of the base class. (2) subsumes (1). > > (I don't expect the paper to take a stance on nested scopes.) > But we can. I can't make the time for this right now and I apologize. This subject deserves more attention and I can't describe it quickly, so it has to wait. As I said, in due time ;-). I'm currently overloaded. (In my defense, and Barry will undesrand me very well, I'll say that among othger things I'm reworking the www.inrialpes.fr Web site, which is actually a shame -- now that I'm in charge, come and visit it in a month). Anyway, some quick general notes on what actually exists: Where Python (Guido) really strikes is that almost everything we have at the language level is interpreted as a name. Thus, "sharing can occur". These names are always resolved in some context associated with the object containing the named object. Since all we have is names, the risk of name conflicts is real, especially when half of the contexts against which name resolution occurs are implicit. Therefore, name resolution has to be done in a controlled way (what Guido has successfully tried to provide when designing the language). Simple example: >>> print o.attr This says: print the value of an object, whose name "attr" has to be resolved in the context associated with the object "o", this context being explicitely pointed out by a dot "." Another (not so obvious) one: >>> print 1 This says: print the value of an object, whose name "1" has to be resolved in the current (implicit) context. And this is exactly what happens inside the VM in terms of LOAD_CONST , then PRINT_ITEM. If you don't grasp this, try the same example: >>> a = 1 >>> print a This says: Resolve the name "1" in the current context (thus we reach the object in question) then assign a new binding ("a" -> the object) in the current context. Then for "print a", see "This says" of the previous example. A valuable thing happens in my last example: >>> print o.__dict__ This resolves the name "__dict__" in the context associated with the object named "o" (pointed by a dot "."), returning this same context! (or a fraction of it). Whether we have to get a portion of the context or the full context is debatable. It has been felt that with the dynamicity of Python, it's useful to get access to the context, then play with it in a controlled manner. So from here, I (and you) can deduce what happens in terms of naming and binding on function/class/... definitions, on module imports, on attr lookups, on "global" declarations, and so on, and when and where (and hopefully why) we get name conflicts or strange (implicit) name resolutions for nested functions. A last word towards classes: There's no difference between >>> o = C() # an instance of the class C >>> o.__class__ and >>> o = 1 >>> o.__class__ I'm prevented to type "1.__class__" only for syntactic reasons, but the context for the resolution of o.__class__ exists. What's missing is for a future mail. Later, -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From bwarsaw at cnri.reston.va.us Thu Sep 30 22:29:55 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 30 Sep 1999 16:29:55 -0400 (EDT) Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> <19990930101549.A10529@cnri.reston.va.us> Message-ID: <14323.51267.54862.538939@anthem.cnri.reston.va.us> I can sympathize with Mark, I have nearly the same code in every script I write. I once wrote a nice (IMO) class for doing all the common things I do with c.l. args, but I can't seem to dig it up at the moment. The idea was basically to have a base class that had all the machinery, while derived classes included specially named methods that did the app-specific option handling. It knew whether the method took 1 or zero arguments (not including self), glommed up the shortarg string and longarg list (with appropriate `:' and `=' thrown in), then parsed the args, dispatching to the handlers, e.g.: class MyOptions(getopt.Options): def handle_a(self): self.alpha = 1 handle_alpha = handle_a def handle_b(self, arg): self.beta = arg handle_beta = handle_b def handle_i(self, arg): try: self.integer = int(arg) except ValueError: self.usage() handle_integer = handle_i and could be used like so: #! /usr/bin/env python # # ... opts = MyOptions(sys.argv[1:], usage=__doc__ % globals()) if opts.alpha: do_my_alpha_thang() if opts.integer = 1: do_something_else() for file in opts.leftoverargs: process_file(file) While I liked this a lot, I seem to remember showing it to Guido, receiving the expected scoffing. So now, I just cut and paste the opt parsing stuff into every script I write :) I think it might be nice to add such a base class to the standard getopt module. -Barry From bwarsaw at cnri.reston.va.us Thu Sep 30 22:42:02 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 30 Sep 1999 16:42:02 -0400 (EDT) Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <007501bf0b46$eb2f38e0$f29b12c2@secret.pythonware.com> Message-ID: <14323.51994.307017.278161@anthem.cnri.reston.va.us> >>>>> "FL" == Fredrik Lundh writes: FL> as reported on slashdot: FL> http://www.perl.com/pub/1999/09/topaz.html FL> (topaz is also the code name for our "rewrite FL> Tkinter in Python", but that's another story ;-) Here's an interesting quote: When I was trying to figure out how to be persuasive on this subject, I finally realized that Perl may be competing with Java in the problem space, but when you're writing Perl, implementing the Perl runtime, really what you're doing is something equivalent to writing a JVM. You're writing the equivalent of a Java Virtual Machine. Now, would you write a JVM in Eiffel? I don't think so. No, so neither would you write the Perl runtime in Java or in Eiffel. In the context of Python, I disagree that it competes against Java; Python makes a nice complement to Java. And I obviously also think it makes perfect sense to write (a) Python runtime in Java, as JimH has so effectively proven. Another interesting tidbit we've addressed here: Was the syntax appropriate for declaring variables to give appropriate hints to a hypothetical compiler? That is to say MY INT $X or MY STR $Y -- and I thought that the INT and the STR and the NUM should be suffixes, something like MY $X:NUM-and, in fact, that suffix syntax is something that Larry officially has blessed, but just not for this purpose. -Barry From da at ski.org Thu Sep 30 23:30:20 1999 From: da at ski.org (David Ascher) Date: Thu, 30 Sep 1999 14:30:20 -0700 (Pacific Daylight Time) Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys In-Reply-To: <14323.51994.307017.278161@anthem.cnri.reston.va.us> Message-ID: On Thu, 30 Sep 1999, Barry A. Warsaw wrote: > Another interesting tidbit we've addressed here: > > Was the syntax appropriate for declaring variables to give > appropriate hints to a hypothetical compiler? That is to say MY > INT $X or MY STR $Y -- and I thought that the INT and the STR and > the NUM should be suffixes, something like MY $X:NUM-and, in fact, > that suffix syntax is something that Larry officially has blessed, > but just not for this purpose. What struck me about that paragraph is that if I understand the comment correctly, Larry is choosing a mapping from a specific syntax to some purpose (unspecified in the excerpt) while an otherwise well-informed Perl user (Chip) 'naturally' wanted to map said syntax to a different semantics. When this happens in Python-land, Guido says "nope, neither". --david From skip at mojam.com Sat Sep 4 23:26:37 1999 From: skip at mojam.com (Skip Montanaro) Date: Sat, 4 Sep 1999 16:26:37 -0500 Subject: [Python-Dev] Catching "return" and "return expr" at compile time Message-ID: <199909042126.QAA03363@dolphin.mojam.com> Attached is a context diff against the latest version of Python/compile.c that checks at compile time for functions that both return expressions or execute return statements with no expression (or equivalently, fall off the end of the function). I figured I'd post it here to get a little friendly feedback and bug discovery before shooting it off to c.l.py. I modified compile.c instead of some preexisting PyLint script because I don't know what's popular out there. On the other hand, I'm sure most people who would be interested in this sort of thing have access to the C source... The basic idea is that each straight line chunk of code is terminated one of four ways: 1. return with no expression 2. return an expression 3. raise an exception 4. fall off the end of the chunk Falling off the end of the function is obviously treated like return with no expression. (This is, after all, what motivated me to do this. ;-) This information is recorded in a new bit vector added to the struct compiling object that's carried around during the compilation. Compound statements simply aggregate the bit vectors for their various clauses in ways appropriate to their semantics. At the end of a function's compilation, the set of return bits computed up to that point tells you whether or not to spit out a warning. Note that it does nothing to recognize constant expressions. The following function will generate a warning: def f(): i = 0 while 1: i = i + 1 if i > 10: return i even though the only way to return from the function is the return statement. To get the above to shut up the compiler you'd have to do something like class CantGetHere: pass def f(): i = 0 while 1: i = i + 1 if i > 10: return i raise CantGetHere Raise statements are treated as a valid way to "return" from a function. Registering them as separate styles of returns serves effectively to turn off the "no return" bit for a block of code. Raise is compatible with either form of return, though they aren't compatible with each other. The code is run whenever a module is compiled. I didn't bother to add a new flag to the python interpreter to enable/disable warnings during compilation, though a -w switch for Python has been mentioned before. I ran the modified byte code compiler over a silly test module as well as executing ./python Lib/test/regrtest.py ./python Lib/compileall.py It uncovered some interesting programming practices and one item I think is an actual bug. In Lib/gzip.py, GzipFile._read returns EOFError at one point instead of raising it. At other points in the method it raises EOFError. There are no other return statements in the function. (I haven't taken the time/had the nerve to run it against my own Python code yet. ;-) I'm firmly of the opinion that more subtle bugs exist in the way people write functions that return and raise values than in the code that calls those functions, contrary to a few vocal folks on c.l.py who may believe otherwise. Enjoy, Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... -------------- next part -------------- A non-text attachment was scrubbed... Name: compile.diffs Type: application/octet-stream Size: 9771 bytes Desc: not available URL: From guido at CNRI.Reston.VA.US Tue Sep 7 17:20:47 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 07 Sep 1999 11:20:47 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Sat, 04 Sep 1999 16:26:37 CDT." <199909042126.QAA03363@dolphin.mojam.com> References: <199909042126.QAA03363@dolphin.mojam.com> Message-ID: <199909071521.LAA18017@eric.cnri.reston.va.us> This is a valuable service! Even though I'm sure that it will cause some pain for people who were used to this programming style... I'm not sure I like the fact that you can't turn it off -- traditionally, Python has had a "no warnings" policy. That has been diluted a bit (python -t prints warnings) but so far it has been the default. I'm wondering if we should introduce a general '-w' flag to turn on warnings like this (which would subsume -t)? Or perhaps there should be a -W flag ("no warnings") and warnings should be the default? There are also platform problems, e.g. on the Mac, stderr doesn't always exist, and on Windows, it doesn't exist if pythonw.exe is used... --Guido van Rossum (home page: http://www.python.org/~guido/) From gward at cnri.reston.va.us Tue Sep 7 18:13:27 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 12:13:27 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909071521.LAA18017@eric.cnri.reston.va.us>; from Guido van Rossum on Tue, Sep 07, 1999 at 11:20:47AM -0400 References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> Message-ID: <19990907121326.B729@cnri.reston.va.us> On 07 September 1999, Guido van Rossum said: > This is a valuable service! Even though I'm sure that it will cause > some pain for people who were used to this programming style... > > I'm not sure I like the fact that you can't turn it off -- > traditionally, Python has had a "no warnings" policy. That has been > diluted a bit (python -t prints warnings) but so far it has been the > default. > > I'm wondering if we should introduce a general '-w' flag to turn on > warnings like this (which would subsume -t)? Or perhaps there should > be a -W flag ("no warnings") and warnings should be the default? Yes yes yes! While adding "-w" is a long way from having a comprehensive set of compile-time warnings in place, it at least means that someone is *thinking* about it. Also, I would suggest that there should be some standard internal mechanism for reporting errors rather than just calling 'PySys_WriteStderr()'. Something as simple as this would probably do the trick: void Py_Warning (char *filename, int line, char *msg) { if (on_a_platform_where_stderr_means_something) PySys_WriteStderr ("warning: file %s, line %d: %s", filename, line, msg); else do_whatever_it_takes_for_this_platform(); } Well, you get the idea. I make no claim that this is an appropriate name for this function, nor do I have anything to say about where it should live. It should also be smart about unknown filename or line number (eg. skip filename if filename == NULL, skip line number if line == -1). Oh, and of course we'll need to add a global variable $^W so that programmers can turn run-time warnings on and off as needed. *duck* Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time warning system in addition to the compile-time warnings of -t and Skip's patch.) Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gstein at lyra.org Tue Sep 7 19:04:14 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 07 Sep 1999 10:04:14 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> Message-ID: <37D5458E.384B77ED@lyra.org> Greg Ward wrote: > > On 07 September 1999, Guido van Rossum said: > > This is a valuable service! Even though I'm sure that it will cause > > some pain for people who were used to this programming style... > > > > I'm not sure I like the fact that you can't turn it off -- > > traditionally, Python has had a "no warnings" policy. That has been > > diluted a bit (python -t prints warnings) but so far it has been the > > default. > > > > I'm wondering if we should introduce a general '-w' flag to turn on > > warnings like this (which would subsume -t)? Or perhaps there should > > be a -W flag ("no warnings") and warnings should be the default? > > Yes yes yes! While adding "-w" is a long way from having a > comprehensive set of compile-time warnings in place, it at least means > that someone is *thinking* about it. I would recommend no warnings by default, and -Wfeature to add specific types of warnings. This pattern follows that used by gcc (well, gcc has *some* warnings by default). Rather than invent a new set of switches, I'd rather steal an existing semantic :-) > Also, I would suggest that there should be some standard internal > mechanism for reporting errors rather than just calling > 'PySys_WriteStderr()'. Something as simple as this would probably do Why? Why not just use PySys_WriteStdErr() as your requested function? It can easily determine "oops. no stderr. let's do something else." >... > Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time > warning system in addition to the compile-time warnings of -t and Skip's > patch.) There is no such thing as run-time vs compile-time warnings. You always have a compiler at run-time, and it can be used at any time. Therefore, you just have "(compilation) warnings" (I could imagine that people will come up with other kinds of warnings once the feature is provided). I would suggest sys.warnings be a dictionary. python -Wbad-return -Wlines-per-func=50 >>> print sys.warnings {'bad-return': None, 'lines-per-func': '50'} Cheers, -g -- Greg Stein, http://www.lyra.org/ From gward at cnri.reston.va.us Tue Sep 7 19:33:53 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 13:33:53 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <37D5458E.384B77ED@lyra.org>; from Greg Stein on Tue, Sep 07, 1999 at 10:04:14AM -0700 References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> Message-ID: <19990907133353.A1235@cnri.reston.va.us> On 07 September 1999, Greg Stein said: > > Also, I would suggest that there should be some standard internal > > mechanism for reporting errors rather than just calling > > 'PySys_WriteStderr()'. Something as simple as this would probably do > > Why? Why not just use PySys_WriteStdErr() as your requested function? It > can easily determine "oops. no stderr. let's do something else." Hmm, that makes sense for the "what's the local equivalent of stderr?" determination. Probably that actually belongs in mywrite() (the static function in Python/sysmodule.c that PySys_WriteStdout() and PySys_WriteStderr() both invoke), so that the same thing can be done for stdout and stderr. However, I still think a separate function for printing source-code-based warnings is a good idea. This is mainly so that the association from (filename, line_number, message) to "warning: file %s, line %s: %s" % (filename, line_number, message) is done in *one* place, rather than everywhere a warning message is generated. For instance, platforms that don't have stderr, but instead pop up a window with all your compile-time warnings nicely formatted, could take advantage of knowing the filename and line number separately to nicely format those warnings. (Of course, this argues *against* putting the "what's the local equivalent of stderr?" determination in the low-level mywrite() function... arg...) > > Maybe sys.show_warnings? ;-) (Of course, that's assuming a run-time > > warning system in addition to the compile-time warnings of -t and Skip's > > patch.) > > There is no such thing as run-time vs compile-time warnings. You always > have a compiler at run-time, and it can be used at any time. Therefore, > you just have "(compilation) warnings" (I could imagine that people will > come up with other kinds of warnings once the feature is provided). Well, currently that's true, since currently Python's only warning is the tab warning from -t -- clearly a compile-time warning. (Is this true? I'm no expert on the internals, but I've certainly not seen any other warnings from Python, and I've included plenty of bugs in my code -- umm, just seeing if it would catch them, yeah that's it...) However, one could certainly envision a world where Python issues runtime warnings. If my time machine were working, I'd zip back and suggest to Guido that mistakes with the % operator should issue warnings rather than raising exceptions. (Ignore the language philosophy issue and presume this would be worthwhile.) There are probably other situations where, ignoring past history and language philosophy, it would make sense to issue a warning and march on ahead rather than blowing up immediately. Sometimes Python has a bit of an itchy trigger finger for that ol' TypeError... Anyways, the focus should probably be on compile-time warnings: I can't think of any major runtime errors offhand that Python currently does *nothing* about, so there's not a great need to go scattering the code with runtime warnings. But they *are* a theoretical possibility, and there *is* a difference with compile-time warnings. > I would suggest sys.warnings be a dictionary. > > python -Wbad-return -Wlines-per-func=50 > > >>> print sys.warnings > {'bad-return': None, 'lines-per-func': '50'} Makes sense -- true to warn (possibly giving some extra meaning to "truth", as in this example), and false to not warn. Or maybe None to not warn, not-None to warn. Of course, if there are only compile-time warnings, then modifying sys.warnings will only affect future imports, execs, evals, etc. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skip at mojam.com Tue Sep 7 19:41:27 1999 From: skip at mojam.com (Skip Montanaro) Date: Tue, 7 Sep 1999 12:41:27 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <19990907133353.A1235@cnri.reston.va.us> References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <14293.19862.328236.265600@dolphin.mojam.com> [ ...Lot's of stuff about details of warning implementation snipped... ] That's why I only provided the code to check for inconsistent use of returns, not the flag to turn it on and off! In a message I accidentally sent only to Guido and myself on the subject, I outlined my take on things, which really does about exhaust my knowledge/interest on how/when to enable warnings: Guido> I'm not sure I like the fact that you can't turn it off -- Guido> traditionally, Python has had a "no warnings" policy. That has Guido> been diluted a bit (python -t prints warnings) but so far it has Guido> been the default. The only reason for not being able to turn it off was that would require introducing some sort of -w flag, which wasn't the point of the exercise. We can have the -w/-t/-W discussion now. I haven't any particular opinion on the best way to do it, although I would much prefer it be a run-time as opposed to compile-time option. One other issue might be whether or not to ignore an existing .pyc file and always recompile .py's if warnings are enabled. Of course, we're still all adults here (I think), so perhaps it's sufficient to remind people in the docs to delete the desired .pyc files before running with warnings enabled. Guido> I'm wondering if we should introduce a general '-w' flag to turn Guido> on warnings like this (which would subsume -t)? Or perhaps there Guido> should be a -W flag ("no warnings") and warnings should be the Guido> default? -w sounds fine to me. Guido> There are also platform problems, e.g. on the Mac, stderr doesn't Guido> always exist, and on Windows, it doesn't exist if pythonw.exe is Guido> used... Perhaps on those platforms a file could be opened in a standard location to catch stderr (I presume you can detect the lack of stderr at run-time?). While that would force some (more) Unix conventions on programmers on those platforms, it would also provide more cross-platform uniformity. Skip From bwarsaw at cnri.reston.va.us Tue Sep 7 20:07:01 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Tue, 7 Sep 1999 14:07:01 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <14293.21573.647149.941018@anthem.cnri.reston.va.us> >>>>> "GW" == Greg Ward writes: GW> However, one could certainly envision a world where Python GW> issues runtime warnings. If my time machine were working, I'd GW> zip back and suggest to Guido that mistakes with the % GW> operator should issue warnings rather than raising exceptions. GW> (Ignore the language philosophy issue and presume this would GW> be worthwhile.) Moderately off-topic, but since you brought it up, here's what I use in Mailman (since site-admins can make mistakes editing their templates, which contains %(keys)s... we'd like to make Mailman more robust so it doesn't totally crap out when that happens). We (hopefully) always interpolate with a SafeDict instead of a raw Python dictionary. -Barry class SafeDict(UserDict): """Dictionary which returns a default value for unknown keys. This is used in maketext so that editing templates is a bit more robust. """ def __init__(self, d): # optional initial dictionary is a Python 1.5.2-ism. Do it this way # for portability UserDict.__init__(self) self.update(d) def __getitem__(self, key): try: return self.data[key] except KeyError: if type(key) == StringType: return '%('+key+')s' else: return '' % `key` From tim_one at email.msn.com Tue Sep 7 20:26:30 1999 From: tim_one at email.msn.com (Tim Peters) Date: Tue, 7 Sep 1999 14:26:30 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909071521.LAA18017@eric.cnri.reston.va.us> Message-ID: <002401bef95e$81963720$3f2d153f@tim> [Guido] > ... > I'm wondering if we should introduce a general '-w' flag to turn on > warnings like this (which would subsume -t)? Or perhaps there should > be a -W flag ("no warnings") and warnings should be the default? The latter, if for no other reason than that new users should get bludgeoned into good practice from their first day. If something's serious enough to trigger a warning, and you insist on doing it anyway, then you should at least know enough about Python to be able to find the -W switch . Note that in response to 1,379 distinct complaints about insane Perl semantics, TomC's stock answer is that every serious Perl programmer runs with -w and "use strict". He's right! Every serious Perl programmer does. Perl picked the wrong default, letting naive programmers hang themselves 1,379 distinct ways by default. Besides, warning by default will enhance your enviable reputation as a ruthless dictator opposed to freedom and creativity . > There are also platform problems, e.g. on the Mac, stderr doesn't > always exist, and on Windows, it doesn't exist if pythonw.exe is > used... But this is already a problem for, e.g., reporting fatal syntax errors, yes? That is, -w/-W isn't creating a new problem here, it's making the lack of a solution to an old problem more evident. all's-for-the-best-in-this-best-of-all-possible-worlds-ly y'rs - tim From gward at cnri.reston.va.us Tue Sep 7 20:47:40 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 7 Sep 1999 14:47:40 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <002401bef95e$81963720$3f2d153f@tim>; from Tim Peters on Tue, Sep 07, 1999 at 02:26:30PM -0400 References: <199909071521.LAA18017@eric.cnri.reston.va.us> <002401bef95e$81963720$3f2d153f@tim> Message-ID: <19990907144739.B1235@cnri.reston.va.us> On 07 September 1999, Tim Peters said: > The latter, if for no other reason than that new users should get bludgeoned > into good practice from their first day. If something's serious enough to > trigger a warning, and you insist on doing it anyway, then you should at > least know enough about Python to be able to find the -W switch . > > Note that in response to 1,379 distinct complaints about insane Perl > semantics, TomC's stock answer is that every serious Perl programmer runs > with -w and "use strict". He's right! Every serious Perl programmer does. > Perl picked the wrong default, letting naive programmers hang themselves > 1,379 distinct ways by default. I agree, but I'm only willing to do so publicly because Tim has. So does the Perl documentation (ie. Tom C., I assume); from "man perl": DIAGNOSTICS The -w switch produces some lovely diagnostics. [...] Did we mention that you should definitely consider using the -w switch? BUGS The -w switch is not mandatory. D'you think that's a hint? Obviously, there *must* be a way to turn off warnings, so we can continue to run our crufty, bug-ridden old code without too many problems. Greg S.'s suggestion for being able to customize *which* warnings are printed is also important. Much hair was pulled when Perl 5.004 was released with a whole bunch of new warning messages -- lots of people had to go back and "fix" working code, or remove the -w switch from production scripts to clean up the mess on their stderr, etc. I suspect most of those people (myself included) were enlightened by the new warnings, but annoyed by having to go and fix what wasn't necessarily broken. A lot of people now recommend using -w only when developing, and removing it for production use, simply because of the risk of new warning messages when you upgrade Perl. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mhammond at skippinet.com.au Wed Sep 8 01:01:25 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 8 Sep 1999 09:01:25 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14293.19862.328236.265600@dolphin.mojam.com> Message-ID: <000e01bef984$e975e6d0$0801a8c0@bobcat> > Perhaps on those platforms a file could be opened in a > standard location to > catch stderr (I presume you can detect the lack of stderr at > run-time?). Not really - there are 2 scenarios here. pythonw.exe, for example, always has a valid stdout handle - it just goes nowhere. When Python is embedded in certain COM servers (such as ASP), the stdout handle is invalid - operations on it will fail (perversely, this also means a single "print" statement in your Python code can raise an exception and make your code fail - and seeing as print statements are the debugging state of the art at the moment, this is less than ideal - but I digress) So Im not sure we can check this reasonably at runtime - invalid handles are easy, but valid handles that go nowhere useful (as in pythonw.exe, and therefore the majority of cases we care about) is obviously difficult. OTOH, pythonw.exe doesnt print tracebacks either. Although not ideal, people arent loudly complaining about this - they know to develop and debug using python.exe. As the warnings we are discussing are compile time warnings, we could simply document that they should run "compileall" over their scripts to generate the warnings before attempting to embed it in some sort of wierd system. On my third hand, I would _really_ like to see this in a lint tool rather than in the core. I realize there is no such tool at the moment, but IMO is where we should be heading. Skip's return statement warnings are fine and a nice addition, but in my experience account for a trivial number of my errors. Stuff like warning about a variable name used only once, for example, will probably never get into core Python but in my opinion is far more valuable. So adding this "-w" switch is fine, but still doesnt give us the framework we need to actually create a truly useful package of warnings for the Python developer. [And I am slowly and painfully starting work in this - a lint tool based on the Python parser module. Dont hold your breath though :-] Mark. From guido at CNRI.Reston.VA.US Wed Sep 8 01:05:19 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 07 Sep 1999 19:05:19 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Wed, 08 Sep 1999 09:01:25 +1000." <000e01bef984$e975e6d0$0801a8c0@bobcat> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <199909072305.TAA20414@eric.cnri.reston.va.us> > > Perhaps on those platforms a file could be opened in a > > standard location to > > catch stderr (I presume you can detect the lack of stderr at > > run-time?). > > Not really - there are 2 scenarios here. pythonw.exe, for example, always > has a valid stdout handle - it just goes nowhere. When Python is embedded > in certain COM servers (such as ASP), the stdout handle is invalid - > operations on it will fail (perversely, this also means a single "print" > statement in your Python code can raise an exception and make your code > fail - and seeing as print statements are the debugging state of the art at > the moment, this is less than ideal - but I digress) > > So Im not sure we can check this reasonably at runtime - invalid handles > are easy, but valid handles that go nowhere useful (as in pythonw.exe, and > therefore the majority of cases we care about) is obviously difficult. > > OTOH, pythonw.exe doesnt print tracebacks either. Although not ideal, > people arent loudly complaining about this - they know to develop and debug > using python.exe. As the warnings we are discussing are compile time > warnings, we could simply document that they should run "compileall" over > their scripts to generate the warnings before attempting to embed it in > some sort of wierd system. Hmm... Perhaps pythonw.exe could use freopen() to point stdout and stderr to a log file in a temp directory? The wizards will know where to look... > On my third hand, I would _really_ like to see this in a lint tool rather > than in the core. I realize there is no such tool at the moment, but IMO > is where we should be heading. Skip's return statement warnings are fine > and a nice addition, but in my experience account for a trivial number of > my errors. Stuff like warning about a variable name used only once, for > example, will probably never get into core Python but in my opinion is far > more valuable. So adding this "-w" switch is fine, but still doesnt give > us the framework we need to actually create a truly useful package of > warnings for the Python developer. > > [And I am slowly and painfully starting work in this - a lint tool based on > the Python parser module. Dont hold your breath though :-] Eventually, I also plan to have some kind of lint in IDLE. If the CP4E money comes, I'll start working on that for earnest... --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Wed Sep 8 13:53:46 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 04:53:46 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909042126.QAA03363@dolphin.mojam.com> <199909071521.LAA18017@eric.cnri.reston.va.us> <19990907121326.B729@cnri.reston.va.us> <37D5458E.384B77ED@lyra.org> <19990907133353.A1235@cnri.reston.va.us> Message-ID: <37D64E4A.4A8A206C@lyra.org> Greg Ward wrote: > ... > > I would suggest sys.warnings be a dictionary. > > > > python -Wbad-return -Wlines-per-func=50 > > > > >>> print sys.warnings > > {'bad-return': None, 'lines-per-func': '50'} > > Makes sense -- true to warn (possibly giving some extra meaning to > "truth", as in this example), and false to not warn. Or maybe None to > not warn, not-None to warn. Of course, if there are only compile-time > warnings, then modifying sys.warnings will only affect future imports, > execs, evals, etc. Actually, I had intended *presence* in the dictionary to mean "enabled." I don't think we'd want to pre-populate the dict with all possible flags ahead of time, then check for each of them on the command line. (startup time!) However, if we "simply" parsed the command line, extracted all -W options and dropped them into the dict, then we're set. Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at digicool.com Wed Sep 8 14:41:18 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 08 Sep 1999 08:41:18 -0400 Subject: [Python-Dev] More Python command-line features Message-ID: <37D6596E.A6B80783@digicool.com> While we're talking about Python command-line features, I have a couple of requests that would make life alot easier for us. 1. I'd like the options given before the script name to be accessable to Python scripts. For example, in: python -O foo.py bar spam I'd like to have a sys variable that told me that the arguments ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] (aka sys.argv). This is needed if I want to fork/exec (or span or whatever) Python with the same option. 2. I'd like to be able to supply environment variables to Python on the command line, as in: python -O -e PYTHONHOME=/Zope foo.py bar spam This would be very helpful in environment-variable-challenged environments like windows 9x, and might be handy elswhere as well. If people agree that these would be good ideas, I'd ve happy to supply a patch. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward at cnri.reston.va.us Wed Sep 8 14:48:49 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Wed, 8 Sep 1999 08:48:49 -0400 Subject: [Python-Dev] More Python command-line features In-Reply-To: <37D6596E.A6B80783@digicool.com>; from Jim Fulton on Wed, Sep 08, 1999 at 08:41:18AM -0400 References: <37D6596E.A6B80783@digicool.com> Message-ID: <19990908084848.A18815@cnri.reston.va.us> On 08 September 1999, Jim Fulton said: > > While we're talking about Python command-line features, I have a couple > of requests that would make life alot easier for us. > > 1. I'd like the options given before the script name to > be accessable to Python scripts. For example, in: > > python -O foo.py bar spam > > I'd like to have a sys variable that told me that the arguments > ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] > (aka sys.argv). This is needed if I want to fork/exec (or span or > whatever) Python with the same option. Regarding -O, it would also be nice to have a higher-level way to find out what optimization level the current interpreter is running under. Currently, as I understand it, there's no way to predict whether py_compile will generate .pyc or .pyo files, which is a minor annoyance in the Distutils installation code. However, if Jim's mythical sys variable is guaranteed to canonicalize Python's command-line options so that something like "-O" in sys.python_argv would always answer this question, I'd be satisfied. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fdrake at acm.org Wed Sep 8 15:35:28 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 8 Sep 1999 09:35:28 -0400 (EDT) Subject: [Python-Dev] More Python command-line features In-Reply-To: <19990908084848.A18815@cnri.reston.va.us> References: <37D6596E.A6B80783@digicool.com> <19990908084848.A18815@cnri.reston.va.us> Message-ID: <14294.26144.574453.888184@weyr.cnri.reston.va.us> Greg Ward writes: > Regarding -O, it would also be nice to have a higher-level way to find > out what optimization level the current interpreter is running under. > Currently, as I understand it, there's no way to predict whether I agree. In fact, I'd even be willing to add support to compile the parse-trees produced by the parser module with or without optimization (pick your favorite level); that would be easy with the current implementation. I didn't add this earlier because Guido objected, saying that the internal optimization flag could change. That would require that the parser module implementation change accordingly. I don't think that would be a huge problem, other than for my having to send Guido a pre-forma gripe that he'd given me more work to do two days before a release. ;-) So, should .compile() accept an optional optimization level, with the default being to use the "current" setting? -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From skip at mojam.com Wed Sep 8 16:54:20 1999 From: skip at mojam.com (Skip Montanaro) Date: Wed, 8 Sep 1999 09:54:20 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000e01bef984$e975e6d0$0801a8c0@bobcat> References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <14294.30407.419817.523625@dolphin.mojam.com> Mark> On my third hand, I would _really_ like to see this in a lint tool Mark> rather than in the core. I realize there is no such tool at the Mark> moment, but IMO is where we should be heading. Skip's return Mark> statement warnings are fine and a nice addition, but in my Mark> experience account for a trivial number of my errors. Stuff like Mark> warning about a variable name used only once, for example, will Mark> probably never get into core Python but in my opinion is far more Mark> valuable. So adding this "-w" switch is fine, but still doesnt Mark> give us the framework we need to actually create a truly useful Mark> package of warnings for the Python developer. I'm not sure the stuff I wrote belongs in the core either, certainly not in C code. As I mentioned when I posted it though, I wasn't sure where a PyLint type program already existed that I could simply graft onto. I've fiddled around enough with the compile.c code in the past couple of years that I understand it fairly well already. I do have some Python code that does peephole optimization on Python bytecode. I could have put it in there (it already divides functions into basic blocks), but again, not many people have it laying about to play with. Can we start/settle on a Python-based source code framework for this sort of thing? Ideally, I'd like to see a framework that brings the parser module's output up to a level where mere mortals like me can reason about Python code. Skip From skip at mojam.com Wed Sep 8 16:58:13 1999 From: skip at mojam.com (Skip Montanaro) Date: Wed, 8 Sep 1999 09:58:13 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909072305.TAA20414@eric.cnri.reston.va.us> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> <199909072305.TAA20414@eric.cnri.reston.va.us> Message-ID: <14294.30895.37706.568973@dolphin.mojam.com> Guido> Eventually, I also plan to have some kind of lint in IDLE. If Guido> the CP4E money comes, I'll start working on that for earnest... Speaking of which, just where *is* IDLE? I get the Python source via CVS, but I'll be damned if I have anything called idle.py or IDLE.py or even anything that matches "*idle*" or "*IDLE*" glob patterns. I just executed "cvs update -A ." from the top of my tree and checked again. Still nothing. Is it a separate module from the main Python source? Thx, Skip From fdrake at acm.org Wed Sep 8 17:04:41 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 8 Sep 1999 11:04:41 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30895.37706.568973@dolphin.mojam.com> References: <000e01bef984$e975e6d0$0801a8c0@bobcat> <199909072305.TAA20414@eric.cnri.reston.va.us> <14294.30895.37706.568973@dolphin.mojam.com> Message-ID: <14294.31497.168088.467694@weyr.cnri.reston.va.us> Skip Montanaro writes: > Speaking of which, just where *is* IDLE? I get the Python source via CVS, > but I'll be damned if I have anything called idle.py or IDLE.py or even Look in Tools/idle/ -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From fdrake at acm.org Wed Sep 8 17:09:11 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 8 Sep 1999 11:09:11 -0400 (EDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30407.419817.523625@dolphin.mojam.com> References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> <14294.30407.419817.523625@dolphin.mojam.com> Message-ID: <14294.31767.411418.442754@weyr.cnri.reston.va.us> Skip Montanaro writes: > thing? Ideally, I'd like to see a framework that brings the parser module's > output up to a level where mere mortals like me can reason about Python This was exactly what I wanted to prevent when I created the parser module! ;-) I think a wrapper that simplifies the parse tree wouldn't be too hard to do; you simply have to be sure that the simplified version can be re-elaborated to pass back to the byte-code compiler via parser.sequence2ast().compile(). Otherwise you can't modify the tree without loosing line number information, which would be nice to keep around! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal at lemburg.com Wed Sep 8 17:08:02 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 08 Sep 1999 17:08:02 +0200 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <199909071521.LAA18017@eric.cnri.reston.va.us> <002401bef95e$81963720$3f2d153f@tim> <19990907144739.B1235@cnri.reston.va.us> Message-ID: <37D67BD2.72A43DF5@lemburg.com> Greg Ward wrote: > > On 07 September 1999, Tim Peters said: > > The latter, if for no other reason than that new users should get bludgeoned > > into good practice from their first day. If something's serious enough to > > trigger a warning, and you insist on doing it anyway, then you should at > > least know enough about Python to be able to find the -W switch . > > > > Note that in response to 1,379 distinct complaints about insane Perl > > semantics, TomC's stock answer is that every serious Perl programmer runs > > with -w and "use strict". He's right! Every serious Perl programmer does. > > Perl picked the wrong default, letting naive programmers hang themselves > > 1,379 distinct ways by default. > > I agree, but I'm only willing to do so publicly because Tim has. So > does the Perl documentation (ie. Tom C., I assume); from "man perl": > > DIAGNOSTICS > The -w switch produces some lovely diagnostics. > [...] > Did we mention that you should definitely consider using the > -w switch? > > BUGS > The -w switch is not mandatory. > > D'you think that's a hint? > > Obviously, there *must* be a way to turn off warnings, so we can > continue to run our crufty, bug-ridden old code without too many > problems. > > Greg S.'s suggestion for being able to customize *which* warnings are > printed is also important. Much hair was pulled when Perl 5.004 was > released with a whole bunch of new warning messages -- lots of people > had to go back and "fix" working code, or remove the -w switch from > production scripts to clean up the mess on their stderr, etc. I suspect > most of those people (myself included) were enlightened by the new > warnings, but annoyed by having to go and fix what wasn't necessarily > broken. A lot of people now recommend using -w only when developing, > and removing it for production use, simply because of the risk of new > warning messages when you upgrade Perl. I'd suggest to use the -W [=] kind of command line option interface for warnings and to also add an environment variable to customize the standard settings, e.g. PYTHONWARNINGS. About enabling warning per default: you should consider the fact that much code out there will probably produce such warnings, even if it is perfectly valid (e.g. consider Skip's example with while 1:...). Enabling it is definitely not a good idea for production code -- it is during the development step. Since production code is likely to run using -O, I suggest disabling warnings when -O is used and enabling them otherwise. Also, I'd like to second GregS' idea with the sys.warnings dict. Together with a generic -W = interface this would be great for adding customized warnings to Python scripts (i.e. not only the ones that the interpreter itself produces). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 123 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Wed Sep 8 16:57:59 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 08 Sep 1999 16:57:59 +0200 Subject: [Python-Dev] Accessing internal flag values References: <37D6596E.A6B80783@digicool.com> <19990908084848.A18815@cnri.reston.va.us> Message-ID: <37D67977.1882CFDC@lemburg.com> Greg Ward wrote: > > On 08 September 1999, Jim Fulton said: > > > > While we're talking about Python command-line features, I have a couple > > of requests that would make life alot easier for us. > > > > 1. I'd like the options given before the script name to > > be accessable to Python scripts. For example, in: > > > > python -O foo.py bar spam > > > > I'd like to have a sys variable that told me that the arguments > > ['-O'] were included before the arguments ['foo.py', 'bar', 'spam'] > > (aka sys.argv). This is needed if I want to fork/exec (or span or > > whatever) Python with the same option. > > Regarding -O, it would also be nice to have a higher-level way to find > out what optimization level the current interpreter is running under. > Currently, as I understand it, there's no way to predict whether > py_compile will generate .pyc or .pyo files, which is a minor annoyance > in the Distutils installation code. Check out mxTools (from my Python Pages). It has a function which lets you control the value of the optimization flag: """ The following functions are installed as add-ons to the builtin sys module. sys.verbosity([level]) If level is given, the value of the interpreter's verbosity flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to e.g. enable verborse lookup output to stderr for import statements even when the interpreter was not invoked with '-v' or '-vv' switch or to force verbosity to be switched off. sys.debugging([level]) If level is given, the value of the interpreter's debugging flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to check whether the interpreter was called with '-d' flag or not. Some extensions use this flag to enable/disable debugging log output (e.g. all the mx Extensions). sys.optimization([level]) If level is given, the value of the interpreter's optimization flag is set to level and the previous value of that flag is returned. Otherwise, the current value is returned. You can use this function to e.g. compile Python scripts in optimized mode even though the interpreter was not started with -O. """ -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 123 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein at lyra.org Wed Sep 8 21:20:32 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 12:20:32 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <14293.19862.328236.265600@dolphin.mojam.com> <000e01bef984$e975e6d0$0801a8c0@bobcat> <14294.30407.419817.523625@dolphin.mojam.com> <14294.31767.411418.442754@weyr.cnri.reston.va.us> Message-ID: <37D6B700.42F07735@lyra.org> Fred L. Drake, Jr. wrote: > > Skip Montanaro writes: > > thing? Ideally, I'd like to see a framework that brings the parser module's > > output up to a level where mere mortals like me can reason about Python > > This was exactly what I wanted to prevent when I created the parser > module! ;-) > I think a wrapper that simplifies the parse tree wouldn't be too > hard to do; you simply have to be sure that the simplified version can > be re-elaborated to pass back to the byte-code compiler via > parser.sequence2ast().compile(). Otherwise you can't modify the > tree without loosing line number information, which would be nice to > keep around! This has already been done. Grab the Python2C distribution from http://www.mudlib.org/~rassilon/p2c/. There is a module named "transformer.py" which does just what you're thinking -- it converts Python's deeply-nested trees into something human-readable. Each of the resulting node types are doc'd at the top of the module. It is also over a couple years old, so it has had some decent debugging/stabilization. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond at skippinet.com.au Thu Sep 9 00:03:07 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 9 Sep 1999 08:03:07 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <14294.30407.419817.523625@dolphin.mojam.com> Message-ID: <002801befa45$efed8240$0801a8c0@bobcat> > Can we start/settle on a Python-based source code framework > for this sort of > thing? Ideally, I'd like to see a framework that brings the > parser module's > output up to a level where mere mortals like me can reason > about Python > code. Actually, I struggled with this a _lot_, then found that P2C has a module called "transform" which flattens the parse tree down to something I can understand (which is good :-) I could simply attach it, but it is grafted to P2C IIRC. If there is interest I will rip it out... Mark. From gstein at lyra.org Thu Sep 9 00:11:35 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 08 Sep 1999 15:11:35 -0700 Subject: [Python-Dev] Catching "return" and "return expr" at compile time References: <002801befa45$efed8240$0801a8c0@bobcat> Message-ID: <37D6DF17.6CA3AB94@lyra.org> Mark Hammond wrote: >... > Actually, I struggled with this a _lot_, then found that P2C has a module > called "transform" which flattens the parse tree down to something I can > understand (which is good :-) > > I could simply attach it, but it is grafted to P2C IIRC. If there is > interest I will rip it out... transformer.py operates quite independently. It is a relatively large module, though (33k), so I would recommend people just grab the P2C distribution from http://www.mudlib.org/~rassilon/p2c/. I have started to put together a page at http://www.lyra.org/greg/python/ that includes the various modules that I've "published". I'll get transformer over there in the next day or two. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond at skippinet.com.au Thu Sep 9 00:29:47 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 9 Sep 1999 08:29:47 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <37D6DF17.6CA3AB94@lyra.org> Message-ID: <002c01befa49$a9b74730$0801a8c0@bobcat> Oops - for some reason I thought we were on Python-help where Greg doesnt hang out... > transformer.py operates quite independently. It does - however, the only simple way to see what it does is to use P2C. All I had in mind was a simple "if __name__=='__main__': block to demonstrate its output. Mark. From tim_one at email.msn.com Fri Sep 10 08:16:20 1999 From: tim_one at email.msn.com (Tim Peters) Date: Fri, 10 Sep 1999 02:16:20 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000e01bef984$e975e6d0$0801a8c0@bobcat> Message-ID: <000001befb54$0095e240$a52d153f@tim> [Mark Hammond] > ... > On my third hand, I would _really_ like to see this in a lint tool > rather than in the core. I realize there is no such tool at the > moment, but IMO is where we should be heading. Following the lead taken by other modern languages, like javalint, c++lint, perllint, dylanlint and even vblint ? C lint was a hack needed due to the combination of bad language design choices and poor compilers, but C compilers are smarter now than lint ever was. Who still uses lint? It's dead, and it's not missed. > Skip's return statement warnings are fine and a nice addition, but in > my experience account for a trivial number of my errors. Stuff like > warning about a variable name used only once, for example, will probably > never get into core Python but in my opinion is far more valuable. The notion that a valuable idea will never get into the core is disturbing. I don't really care how it's implemented, but a *visibly* separate "checking" tool is bad UI, one that even the C community left behind with relief. > So adding this "-w" switch is fine, but still doesnt give us the framework > we need to actually create a truly useful package of warnings for the > Python developer. No, but adding the "-W" switch does give us the means by which (perhaps the illusion of) "a" smarter compiler can be invoked & controlled. > [And I am slowly and painfully starting work in this - a lint tool based > on the Python parser module. Dont hold your breath though :-] Aaron W has had a capable pylint tool for a couple years, & it remains virtually unknown; and, far as I can tell, Aaron reciprocated the lack of interest by dropping active development. So why was C lint successful in its day while every crack at pylint flops (btw, yours will too <0.5 wink>)? I think it's two sides of the same coin: C lint found dozens of deadly problems that infested almost all C code (remember the pre-prototype days?). Versions of pylint offer very little beyond pointing out unique vrbl names, perhaps indentation checking, and ...? I'm drawing a blank here. I suppose they should strive to give better msgs for runaway triple-quoted strings. What else? Skip's "return" checker, and far as I can tell then we're already at the point of diminishing returns. My claim is that pylints don't get used both because they're a separate step, and because the odds of them catching something interesting are comparatively tiny. Python simply doesn't have many errors that *can* be caught at compile-time. It's like me firing up the spell-checker at this point to verify "compile-time" -- the expected payoff is negative. There's little enough useful a pylint could do that a mod to add those few smarts to the core would be a fraction of the size & effort of yet another separate tool. Better, in the core, it would actually do people some good because it would actually get used. Which specific problems do you expect your lint tool to uncover? Perhaps there's a world of mechanically-checkable Python errors I haven't yet bumped into. you-windows-guys-write-strange-code-ly y'rs - tim From da at ski.org Fri Sep 10 08:28:07 1999 From: da at ski.org (David Ascher) Date: Thu, 9 Sep 1999 23:28:07 -0700 (Pacific Daylight Time) Subject: [Python-Dev] FYI: Techniques for scientific C++ Message-ID: In case you haven't seen enough C++ papers, you might find this paper interesting. It's a good way to learn more about templates without trying to actually use them, and it's a very good way to decide to stay away from high-power C++ techniques like expression templates. Yeeagh! (I especially like the program shown in section 1.11.1). --david Date: Thu, 26 Aug 1999 21:59:06 -0500 (EST) From: Todd Veldhuizen Subject: OON: Techniques for scientific C++ I've updated and expanded my slides on "Techniques for Scientific C++" into a document. It's available in HTML and Postscript from: HTML: http://extreme.indiana.edu/~tveldhui/papers/techniques/ PS: http://extreme.indiana.edu/~tveldhui/papers/techniques/techniques.ps Here's the table of contents. 1.1: About this document 1.2: C++ Compilers 1.2.1: Placating sub-standard compilers 1.2.2: The compiler landscape 1.2.3: C++-specific optimization 1.3: Compile times 1.3.1: Headers 1.3.2: Prelinking 1.3.3: The program database approach -- Visual Age C++ 1.3.4: Quadratic/Cubic template algorithms 1.4: Static Polymorphism 1.4.1: Are virtual functions evil? 1.4.2: Solution A: simple engines 1.4.3: Solution B: the Barton and Nackman Trick 1.5: Callback inlining techniques 1.5.1: Callbacks: the typical C++ approach 1.5.1.1: Expression templates 1.5.1.2: STL-style function objects 1.5.1.3: Pointer-to-function as a template parameter 1.6: Managing code bloat 1.6.1: Avoid kitchen-sink template parameters 1.6.2: Put function bodies outside template classes 1.6.3: Inlining levels 1.7: Containers 1.7.1: STL-style containers 1.7.2: Data/View containers 1.8: Aliasing and restrict 1.9: Traits 1.9.1: An example: average() 1.9.2: Type promotion example 1.10: Expression templates 1.10.1: Performance implications of pairwise evaluation 1.10.2: Recursive templates 1.10.3: Expression templates: building parse trees 1.10.4: A minimal implementation 1.10.5: Refinements 1.10.6: Pointers to more information 1.10.7: References 1.11: Template metaprograms 1.11.1: Template metaprograms: some history 1.11.2: The need for specialized algorithms 1.11.3: Using template metaprograms to specialize algorithms 1.12: Comma overloading 1.12.1: An example 1.13: Interfacing with Fortran codes 1.14: Some thoughts on performance tuning 1.14.1: General suggestions 1.14.2: Know what your compiler can do 1.14.3: Data structures and algorithms 1.14.4: Efficient use of the memory hierarchy Cheers, Todd - -- Todd Veldhuizen tveldhui at acm.org Indiana Univ. Comp. Sci. http://extreme.indiana.edu/~tveldhui/ - --------------------- Object Oriented Numerics List -------------------------- From mhammond at skippinet.com.au Fri Sep 10 09:36:39 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 10 Sep 1999 17:36:39 +1000 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000001befb54$0095e240$a52d153f@tim> Message-ID: <003a01befb5f$39f25400$0801a8c0@bobcat> [Tim laments the death of lint for C :-] > The notion that a valuable idea will never get into the core > is disturbing. Agreed. I based my assesment simply on my perception of what is likely to happen, not my opinion of what _should_ happen. I do agree that it is far far preferable for Python itself to be capable of issuing these warnings, and if Guido feels that is the best direction then it would be very cool. Only Guido can state if he would support such efforts, and probably should right about now (the funk soul brother - sorry - just got the Fat Boy Slim CD, and its going around in my head :-) > Aaron W has had a capable pylint tool for a couple years, & it remains > virtually unknown; and, far as I can tell, Aaron reciprocated > the lack of > interest by dropping active development. Which tends to be the biggest problem with it. A number of people have tried to use it, but often get stymied by the lack of 1.5.?isms - ie, "raise" (ie re-raise) and "assert". It bombs at these statements, and there is some real magic I didnt want to understand. Aaron agrees that a parser module based one would be better. But your original point still remains - I agree having Python do this is a better solution all round. > (remember the pre-prototype days?). Versions of pylint offer > very little > beyond pointing out unique vrbl names, perhaps indentation > checking, and > ...? I'm drawing a blank here. I suppose they should strive > to give better > msgs for runaway triple-quoted strings. What else? Skip's "return" > checker, and far as I can tell then we're already at the point of > diminishing returns. Agreed. However, all of these would be very valuable and account for the vast majority of my errors. > you-windows-guys-write-strange-code-ly y'rs - tim Only cos we use a strange OS Mark. From guido at CNRI.Reston.VA.US Fri Sep 10 16:05:23 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 10 Sep 1999 10:05:23 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: Your message of "Fri, 10 Sep 1999 17:36:39 +1000." <003a01befb5f$39f25400$0801a8c0@bobcat> References: <003a01befb5f$39f25400$0801a8c0@bobcat> Message-ID: <199909101405.KAA26303@eric.cnri.reston.va.us> > Agreed. I based my assesment simply on my perception of what is likely to > happen, not my opinion of what _should_ happen. I do agree that it is far > far preferable for Python itself to be capable of issuing these warnings, > and if Guido feels that is the best direction then it would be very cool. > Only Guido can state if he would support such efforts, and probably should > right about now (the funk soul brother - sorry - just got the Fat Boy Slim > CD, and its going around in my head :-) I agree it should happen, and Tim's argument about keeping lint and compiler together is a convincing one. What stands in the way? (a) There's so much else to do... (b) *Someone* needs to design a good framework for spitting out warnings, and for giving the programmer a way to control which warnings to ignore. I've seen plenty of good suggestions here; now somebody should simply go off and come up with a proposal too good to refuse. (c) I have a different agenda with CP4E -- I think it would be great if the editor could do syntax checking and beyond *while you type*, like the spell checker in MS Word. (I *like* Word's spell checker, even though I hate the style checker [too stupid], and I gladly put up with the spell checker's spurious complaints -- it's easy to turn off, easy to ignore, and it finds lots of good stuff.) Because the editor has to deal with incomplete and sometimes ungrammatical things, and because it has to work in real time (staying out of the way when you're frantically typing, catching up when your fingers take a rest), it needs a different kind of parser. But that's another project, and once the Python core has a warning framework in place, I'm sure we'll find more things that are worth warning about. I'm not always in agreement with Tim Peters when he says that Python is so dynamic that it's impossible to check for certain errors. It may be impossible to say *for sure* that something is an error, but there sure are lots of situations where you're doing something that's *likely* to be an error. E.g. if the compiler sees len(1), and there's no local or global variable named len, it *could* be the case that the user has set up a parallel universe where the len() built-in accepts an integer argument, but more likely it's an honest mistake, and I would like to get a warning about it. The hard part here is to issue this warning for len(x) where x is some variable or expression that is likely to have a non-sequence value (barring alternate universes); this might require global analysis that's hard or expensive enough that we can't put it in the core (yet). This may be seen as an argument for a separate lint... --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at mojam.com Fri Sep 10 16:16:52 1999 From: skip at mojam.com (Skip Montanaro) Date: Fri, 10 Sep 1999 09:16:52 -0500 (CDT) Subject: [Python-Dev] Where does lint-like functionality belong? Message-ID: <14297.4820.321654.848624@dolphin.mojam.com> Okay, if Tim's assertions are correct (and I have no reason to suspect them), a separate pylint will be doomed, so the only reasonable thing to do is to place it in the core where it will be rarin' to go all the time. Perl's experience with -w seems to suggest that it's best to always enable whatever warnings you can as well. (More and more I see people using gcc's -Wall flag as well.) Now, my return consistency stuff was easy enough to write in C for two reasons. One, I'm fairly comfortable with the compile.c code. Two, adding my checks required no extra memory management overhead. Consider a few other checks you might conceivably add to the byte code compiler: * tab nanny stuff (already enabled with -t, right?) * variables set but not used * variables used before set If all of this sort of stuff is added to the compiler proper, I predict a couple major problems will surface: * The complexity of the code will increase significantly, making it harder to maintain and enhance * Fewer and fewer people will be able to approach the code, making it less likely that new checks are added * Future extensions like pluggable virtual machines will be harder to add because their byte code compilers will be harder to integrate into the core In addition, more global checks probably won't be possible (reasoning about code across module boundaries for instance) because the compiler's view of the world is fairly narrow. I think lint-like tools should be implemented in Python (possibly with the support of an extension module for performance-critical sections) which is then called from the compiler proper under appropriate circumstances (warnings enabled, all necessary modules importable, etc). I believe the code would be much more easily maintained and extended. You'd be able to swap in a new byte code compiler without risking the loss of your checking code. Skip From mal at lemburg.com Fri Sep 10 21:06:28 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 10 Sep 1999 21:06:28 +0200 Subject: [Python-Dev] Relative Package Imports Message-ID: <37D956B4.3E79C2C1@lemburg.com> Hi everybody, I've spent the last two hours trying to get relative package imports to work because I need them for my extension packages which will soon all move under a new top-level package name to overcome the conflicts with PIL and Zope. Here are the results... Demo Package Structure: (see the attached demopkg.zip) [a] [b] bc.py ab.py With the attached patch you can do the following: # Pretty useless... import a.b.__.ab # From inside bc.py: from __ import ab # At top-level (also useless, but shows how this situation is handled): import __.sys # __ is bound to None since we are at top-level; sys is still # being loaded though. Of course, common usage will be of the form: form __.__ import submodule_at_higher_level Please tell me what you think and give it a try. It's a first try and may have some design errors. Especially the way head and tail are treated in Python/import.c:import_module_ex may cause trouble -- I need help here. Note: The patch is against the CVS version. If you run Python in verbose mode, the patch will produce some verbose output of what it's doing. Enjoy, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 113 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ -------------- next part -------------- --- /home/lemburg/orig/Python/Python/import.c Fri Apr 9 19:00:51 1999 +++ Python/import.c Fri Sep 10 20:51:02 1999 @@ -1572,10 +1572,14 @@ load_next(mod, altmod, p_name, buf, p_bu char *dot = strchr(name, '.'); int len; char *p; PyObject *result; + if (Py_VerboseFlag) + printf("# load_next: (1) name='%s', buf='%.*s'\n", + name,*p_buflen,buf); + if (dot == NULL) { *p_name = NULL; len = strlen(name); } else { @@ -1586,10 +1590,39 @@ load_next(mod, altmod, p_name, buf, p_bu PyErr_SetString(PyExc_ValueError, "Empty module name"); return NULL; } + /* Handle "__" indicator telling the import mechanism to + continue the search one level higher in the package + hierarchy */ + if (strncmp(name,"__",len) == 0) { + PyObject *modules = PyImport_GetModuleDict(); + + /* Strip the final dotted name from buf */ + dot = strrchr(buf, '.'); + if (dot == NULL) + *p_buflen = 0; + else + *p_buflen = dot - buf; + buf[*p_buflen] = '\0'; + + /* Fetch the parent module or revert to a top-level search */ + if (*p_buflen > 0) { + mod = PyDict_GetItemString(modules,buf); + if (mod == NULL) { + PyErr_SetString(PyExc_SystemError, + "Parent module missing"); + return NULL; + } + } + else + mod = Py_None; + Py_INCREF(mod); + return mod; + } + p = buf + *p_buflen; if (p != buf) *p++ = '.'; if (p+len-buf >= MAXPATHLEN) { PyErr_SetString(PyExc_ValueError, @@ -1597,10 +1630,14 @@ load_next(mod, altmod, p_name, buf, p_bu return NULL; } strncpy(p, name, len); p[len] = '\0'; *p_buflen = p+len-buf; + + if (Py_VerboseFlag) + printf("# load_next: (2) modname='%s', fullname=buf='%s'\n", + p,buf); result = import_submodule(mod, p, buf); if (result == Py_None && altmod != mod) { Py_DECREF(result); /* Here, altmod must be None and mod must not be None */ -------------- next part -------------- A non-text attachment was scrubbed... Name: demopkg.zip Type: application/x-zip-compressed Size: 851 bytes Desc: not available URL: From guido at CNRI.Reston.VA.US Fri Sep 10 21:16:28 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 10 Sep 1999 15:16:28 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Fri, 10 Sep 1999 21:06:28 +0200." <37D956B4.3E79C2C1@lemburg.com> References: <37D956B4.3E79C2C1@lemburg.com> Message-ID: <199909101916.PAA27287@eric.cnri.reston.va.us> > I've spent the last two hours trying to get relative package > imports to work because I need them for my extension packages > which will soon all move under a new top-level package name > to overcome the conflicts with PIL and Zope. I'd much rather use absolute package names for anything that's not in the same directory as the current module. --Guido van Rossum (home page: http://www.python.org/~guido/) From da at ski.org Sat Sep 11 06:06:54 1999 From: da at ski.org (David Ascher) Date: Fri, 10 Sep 1999 21:06:54 -0700 (Pacific Daylight Time) Subject: [Python-Dev] jpythonc: unreachable Python becomes unreachable Java (fwd) Message-ID: Greg Ward notes on the JPython list: > Looks like I've found a definite bug in jpythonc -- unreachable Python > code compiles to unreachable Java code. Here's a simple example: > [..] > Yikes... does this mean that jpythonc will have to analyze Python code > for reachability? Is this even possible? Guess the answer for now is, > "Don't do that!". This is another sort of warning that a checker could do in the core which could catch some errors. Not sure that it makes sense to try and find all the cases that an arbitrary Java compiler could find, but presumably Greg's original code which led him to the jpythonc bug find had a plain-old python bug in it? Did Aaron's pylint find any nonreachable code? --david From mal at lemburg.com Sat Sep 11 12:29:48 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat, 11 Sep 1999 12:29:48 +0200 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> Message-ID: <37DA2F1C.7ED64960@lemburg.com> Guido van Rossum wrote: > > > I've spent the last two hours trying to get relative package > > imports to work because I need them for my extension packages > > which will soon all move under a new top-level package name > > to overcome the conflicts with PIL and Zope. > > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. Of course, you could do everything with absolute names, but then the package author would dictate the complete absolute path which is not always desirable since it can cause name collisions such as DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) and Mark's win32 stuff. As more and more packages appear, we run into this problem more and more often. Relative imports would greatly reduce the risk these collisions, because packages could be written self-contained meaning that they can reach their internal modules via relative paths only and don't have to know about the absolute paths by which they are reachable from other parts of the Python universe. I could then make all my packages self-contained and distribute them in two forms without having to change a single line for the added support: 1. under the new 'mx' package, e.g. mx.DateTime 2. for backward compatibility under 'DateTime' Another major advantage is that I could take any other self-contained package and install it under absolute paths of my choice, e.g. put Zope under org.zope.core, Python under org.python.core etc., without harming their functionality or having to dive deep into their import structures to fix them manually. To further enhance this mechanism I would like to have an alias mechanism in import, pickle et al. so that changes in the package structures become manageable without user intervention: pickles are a major problem whenever import structures change because they store absolute module names. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 111 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From tim_one at email.msn.com Sat Sep 11 22:29:58 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 11 Sep 1999 16:29:58 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DA2F1C.7ED64960@lemburg.com> Message-ID: <000901befc94$6a463d20$6aa2143f@tim> [Guido] > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. [M.-A. Lemburg] > Of course, you could do everything with absolute names, but then > the package author would dictate the complete absolute path which > is not always desirable since it can cause name collisions such > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > released) and Mark's win32 stuff. > > As more and more packages appear, we run into this problem more > and more often. I never understand package complaints. Maybe it's the imprecision of the language people use, or maybe because it's because people don't give fully fleshed-out examples. Whatever, in the end, I never have any idea what the complaint is, or in what way the solution is "solving" anything. In the above, "absolute" doesn't appear to mean "absolute" in any OS-sensible sense. So what does it really mean? Does it mean the same things to Guido and MAL? In MAL's hint of examples, I don't see any problem. If mxDateTime unpacks itself into a directory named DateTime, then *of course* it's going to collide with other packages that want to do likewise. Install it into mxDateTime instead, and take "absolute" to mean "any module that wants an mxDateTime service and does not itself live directly in mxDateTime/ must import the desired module via a path beginning 'mxDateTime.'", and everything looks straightforward to me (and that outcome makes me infer that this is thus probably what Guido has in mind too). Similarly for win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be good to have an explicit way to say "std library" -- "Lib." comes to mind). > I could then make all my packages self-contained and > distribute them in two forms without having to change > a single line for the added support: > > 1. under the new 'mx' package, e.g. mx.DateTime > 2. for backward compatibility under 'DateTime' Ah, so that's what this is about. I vote screw #2. Releasing it that way was a mistake. Better to live with the brief & finite pain of repairing it than complicating Python to cover up for it. > Another major advantage is that I could take any other > self-contained package and install it under absolute paths > of my choice, e.g. put Zope under org.zope.core, Python under > org.python.core etc., without harming their functionality > or having to dive deep into their import structures to fix > them manually. I view that not as an advantage but as harmful complication. Zope etc add great value to a Python installation, and when I write a killer app full of "import zope.this" and "import zope.that", I don't want the possibility that it's not going to work on my client's machine just because their sysadmin installed Zope into some silly site-specific path named after his soon-to-be ex-girlfriend . I don't want a way to work around him doing that, either: I don't want him to be able to screw me to begin with. > To further enhance this mechanism I would like to have an > alias mechanism in import, pickle et al. so that changes > in the package structures become manageable without user > intervention: pickles are a major problem whenever import > structures change because they store absolute module names. This is a different issue, and may have merits of its own. WRT the relative import scheme, its advantage seems to lie in providing a way to partially recover from the damage the new scheme causes <0.5 wink>. As is, the package name used by a release is part of its published interface. You can't change it without causing pain, any more than you can make incompatible changes to public class methods or input-output behavior. In return, package clients are uniform, simple and portable, making life easiest for the people who know least. The burden is on package authors to choose names wisely, and that's where the burden should be. if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs - tim From tim_one at email.msn.com Sun Sep 12 02:41:16 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 11 Sep 1999 20:41:16 -0400 Subject: [Python-Dev] FYI: Techniques for scientific C++ In-Reply-To: Message-ID: <000101befcb7$85be2b80$4c2d153f@tim> [David Ascher, concerning http://extreme.indiana.edu/~tveldhui/papers/techniques/] ] > In case you haven't seen enough C++ papers, you might find this paper > interesting. It's a good way to learn more about templates without trying > to actually use them, and it's a very good way to decide to stay away from > high-power C++ techniques like expression templates. Yeeagh! Didn't these people get to play with m4 in their youth? Ah, this *is* their youth . As for so much else, this is stuff they could have been doing with Lisp/Scheme compile-time macros 15 years ago, except the latter are much clearer, simpler and easier to use. It's not "a trick" in Scheme, it's a fundamental approach, and the full power of the language is available at every step. If this kind of thing still holds a perverse attraction, check out http://www.fftw.org/ These guys produce what are usually the fastest FFT algorithms in the world across platforms, via a dynamic programming approach that generates algorithm fragments and times them on the target platform, eventually "growing" superior platform-specific FFT code for each vector size. Until C++ templates can do file I/O at compile-time (to take advantage of timings from previous runs), they won't even get close . computation-at-compile-time-is-a-powerful-thing-but- then-so-is-a-skunk's-odor-ly y'rs - tim From gmcm at hypernet.com Sun Sep 12 04:41:44 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Sat, 11 Sep 1999 22:41:44 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DA2F1C.7ED64960@lemburg.com> Message-ID: <1275018545-12250772@hypernet.com> M.-A. Lemburg wrote: > Guido van Rossum wrote: > > > > I'd much rather use absolute package names for anything that's not > > in the same directory as the current module. > > Of course, you could do everything with absolute names, but then > the package author would dictate the complete absolute path which is > not always desirable since it can cause name collisions such as > DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) > and Mark's win32 stuff. I can see your point (although I also believe that authors - Guido excepted - should come up with collision-free names, probably in a Java-ish scheme). But I strongly believe that import.c should be left alone, maybe even to die. There are too many people doing import hooks to make fiddling with its behavior safe. I'm also a strong proponent of Greg's imputil.py scheme, which makes it a breeze to do import hooks. And my experience disproves the notion that the import mechanism needs to be in C. If you don't believe me, try the ZlibArchive stuff (which is cross platform) from my Win32 installer stuff. You can pack the standard library into one 475K file, and get a perceptible performance boost. OTOH, I could see doing a framework of packages, in which case relative imports might be handy. This seems to work fine: def relimp(nm): rpth = string.split(nm, '/') tpth = string.split(__name__, '.')[:-1] for node in rpth: if node == '..': del tpth[-1] else: tpth.append(node) #print `tpth` return __import__(string.join(tpth, '.')) b = relimp('../packageA2.b') > As more and more packages appear, we run into this problem more > and more often. Relative imports would greatly reduce the > risk these collisions, because packages could be written > self-contained meaning that they can reach their internal > modules via relative paths only and don't have to know about > the absolute paths by which they are reachable from other parts > of the Python universe. > > I could then make all my packages self-contained and > distribute them in two forms without having to change > a single line for the added support: > > 1. under the new 'mx' package, e.g. mx.DateTime > 2. for backward compatibility under 'DateTime' > > Another major advantage is that I could take any other > self-contained package and install it under absolute paths > of my choice, e.g. put Zope under org.zope.core, Python under > org.python.core etc., without harming their functionality > or having to dive deep into their import structures to fix > them manually. > > To further enhance this mechanism I would like to have an > alias mechanism in import, pickle et al. so that changes > in the package structures become manageable without user > intervention: pickles are a major problem whenever import > structures change because they store absolute module names. > > -- > Marc-Andre Lemburg > ______________________________________________________________________ > Y2000: 111 days left Business: > http://www.lemburg.com/ Python Pages: > http://www.lemburg.com/python/ > > > > _______________________________________________ > Python-Dev maillist - Python-Dev at python.org > http://www.python.org/mailman/listinfo/python-dev - Gordon From tim_one at email.msn.com Sun Sep 12 11:18:16 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sun, 12 Sep 1999 05:18:16 -0400 Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <199909101405.KAA26303@eric.cnri.reston.va.us> Message-ID: <000001befcff$bf654e20$492d153f@tim> [MarkH] > ... > I based my assesment simply on my perception of what is likely to > happen, not my opinion of what _should_ happen. I based mine on what Guido was waiting for someone to say . We worry too much about disagreeing here; different opinions are great! Guido will squash the ones he can't stand anyway. [about Aaron's pylint's lack of 1.5.2 smarts] > ... > Aaron agrees that a parser module based one would be better. You can't beat a real parse, no argument there. Luckily, the compiler parses too. [Guido] > What stands in the way? > > (a) There's so much else to do... How did Perl manage to attract 150 people with nothing to do except hack on Perl internals? "Wow, that code's such a mess I bet even *I* could get something into it" <0.6 wink>. > (b) *Someone* needs to design a good framework for spitting out > warnings, and for giving the programmer a way to control which > warnings to ignore. I've seen plenty of good suggestions here; now > somebody should simply go off and come up with a proposal too good to > refuse. The response has been ... absent. Anyone doing this? I liked JimF's push to make cmd-line options available to Python programs too. Somehow they seem related to me. > (c) I have a different agenda with CP4E -- I think it would be great > if the editor could do syntax checking and beyond *while you type*, > like the spell checker in MS Word. (I *like* Word's spell checker, > even though I hate the style checker [too stupid], and I gladly put up > with the spell checker's spurious complaints -- it's easy to turn off, > easy to ignore, and it finds lots of good stuff.) > > Because the editor has to deal with incomplete and sometimes > ungrammatical things, and because it has to work in real time (staying > out of the way when you're frantically typing, catching up when your > fingers take a rest), it needs a different kind of parser. Different from what? Python's own parser for sure. IDLE has at least two distinct parsers of its own that have nothing in common with Python's parser and little in common with each other. Using the horrid tricks in PyParse.py, it may even be possible to write the kind of parser you need in Python and still have it run fast enough. For parsing-on-the-fly from random positions, take my word for it and Barry's as insurance : the single most frequent question you need to have a fast and reliable answer for is "is this character in a string?". Unfortunately, turns out that's the hardest question to answer too. The next one is "am I on a continuation line, and if so where's the start?". Given rapid & bulletproof ways to answer those, the rest is pretty easy. > But that's another project, and once the Python core has a warning > framework in place, I'm sure we'll find more things that are worth > warning about. That was frequently predicted for various pylint projects too . > I'm not always in agreement with Tim Peters when he says that Python > is so dynamic that it's impossible to check for certain errors. It > may be impossible to say *for sure* that something is an error, but > there sure are lots of situations where you're doing something that's > *likely* to be an error. We have no disagreement there. What a compiler does must match the advertised semantics of the language-- or its own advertised deviations from those --without compromise. A warning system has no such constraint; to the contrary, in the case of a semantic mess like Perl, most of its value is in pointing out *legal* constructs that are unlikely to work the way you intended. > E.g. if the compiler sees len(1), and there's no local or global > variable named len, it *could* be the case that the user has set up a > parallel universe where the len() built-in accepts an integer > argument, but more likely it's an honest mistake, and I would like to > get a warning about it. Me too. More: I'd also like to get a warning for *having* a local or global variable named len! Masking the builtin names is simply bad practice, and is also usually an honest mistake. BTW, I was surprised that the most frequent gotcha among new Python users at Dragon turned out to be exactly that: dropping a "len" or a "str" or whatever (I believe len, str and list were most common) into their previously working code-- because they just learned about that builtin --and getting runtime errors as a result. That is, they already had a local var of that name, and forgot. Then they were irked that Python didn't nag them from the start (with a msg they understood, of course). > The hard part here is to issue this warning for len(x) where x is some > variable or expression that is likely to have a non-sequence value > (barring alternate universes); this might require global analysis > that's hard or expensive enough that we can't put it in the core > (yet). This may be seen as an argument for a separate lint... Curiously, Haskell is statically type-safe but doesn't require declarations of any kind -- it does global analysis, and has a 100% reliable type inference engine (the language was, of course, designed to make this true). Yet I don't think I've ever seen a Haskell program on the web that didn't explicitly declare the type of every global anyway. I think this is natural, too: while it's a PITA to declare the type of every stinking local that lives for two lines and then vanishes, the types of non-local names aren't self-evident: type decls really help for them. So if anyone is thinking of doing the kind of global analysis Guido mentions here, and is capable of doing it , I'd much rather they put their effort into optional static type decls for Python2. Many of the same questions need to be answered either way (like "what's a type?", and "how do we spell a type?" -- the global analysis warnings won't do any good if you can't communicate the substance of an error ), and optional decls are likely to have bigger bang for the buck. [Skip Montanaro] > ... > Perl's experience with -w seems to suggest that it's best to always > enable whatever warnings you can as well. While that's my position, I don't want to oversell the Perl experience. That language allows so many goofy constructs, and does so many wild guesses at runtime, that Perl is flatly unusable without -w for non-trivial programs. Not true of Python, although the kinds of warnings people have suggested so far certainly do seem worthy of loud complaint by default. > (More and more I see people using gcc's -Wall flag as well.) If you have to write portable C++ code, and don't enable every warning you can get on every compiler you have, and don't also turn on "treat warnings as errors", non-portable code will sneak into the project rapidly. That's my experience, over & over. gcc catches stuff MS doesn't, and vice versa, and MetroWerks yet another blob, and platform-specific cruft *still* gets in. It's irksome. > Now, my return consistency stuff was easy enough to write in C for two > reasons. One, I'm fairly comfortable with the compile.c code. I don't anticipate dozens of people submitting new warning code. It would be unprecendented if even two of us decided this was our thing. If would be almost unprecendented if even one of us followed up on it <0.6 wink>. > Two, adding my checks required no extra memory management overhead. Really good global analysis likely requires again as much C code as already exists. Luckily, I don't think putting in some warnings requires that all conceivable warnings be implemented at once . For stuff that complex, I'd rather make it optional and write it in Python; I don't believe any law prevents the compiler from running Python code. > Consider a few other checks you might conceivably add to the byte code > compiler: > > * tab nanny stuff (already enabled with -t, right?) Very timidly, yes . Doesn't complain by default, and you need -tt to make it an error. Only catches 4 vs 8 tab size ambiguities, but that's good enough for almost everyone almost all the time. > * variables set but not used > * variables used before set These would be wonderful. The Perl/pylint "gripe about names unique in a module" is a cheap approximation that gets a surprising percentage of the benefit for the cost of a dict and an exception list. > If all of this sort of stuff is added to the compiler proper, I predict a > couple major problems will surface: > > * The complexity of the code will increase significantly, making it > harder to maintain and enhance The complexity of the existing code should be almost entirely unaffected, because non-trivial semantic analysis belongs in a new subsystem with its own code. > * Fewer and fewer people will be able to approach the code, making it > less likely that new checks are added As opposed to what? The status quo, with no checks at all? Somehow, facing the prospect of *some* checks doesn't frighten me away . Besides, I don't buy the premise: if someone takes this on as their project, worrying that they'll decline to add new valuable checks is like MarkH worrying that I wouldn't finish adding full support for stinking tabs to the common IDLE/PythonWin editing components. People take pride in their hackery. > * Future extensions like pluggable virtual machines will be harder > to add because their byte code compilers will be harder to integrate > into the core If you're picturing adding this stuff sprayed throughout the guts of the existing com_xxx routines, we've got different pictures in mind. Semantic analysis is usually a pass between parsing and code generation, transforming the parse tree and complaining about source it thinks is fishy. If done in any conventional way, it has no crosstalk at all with either the parsing work that precedes it or the code generation that follows it. It's a pipe stage between them, whose output is of the same type as its input. That is, it's a "pluggable component" in its own right, and doesn't even need to be run. So potential for interference just isn't there. At present, Python is very unusual both in: 1) Having no identifiable semantic pass at all, parsing directly to byte code, and enforcing its few semantic constraints (like "'continue' not properly in loop") lumped in with both of those. and 2) Having one trivial optimization pass-- 76 lines of code instead of the usual 76,000 --running after the byte code has been generated. However, the sole transformation made here (distinguishing local from non-local names) is much more properly viewed as being a part of semantic analysis than as being "an optimization". It's deducing trivial info about what names *mean* (i.e., basic semantics), and is called optimization here only because Python didn't do it at first. So relating this to a traditional compiler, I'd say that "optimize()" is truly Python's semantic analysis pass, and all that comes before it is the parsing pass -- a parsing pass with output in a form that's unfortunately clumsy for further semantic analysis, but so it goes. The byte code is such a direct reflection of the parse tree that there's really little fundamental difference between them. So for minimal disruption, I'd move "optimize" into a new module and call it the semantic analysis pass, and it would work with the byte code. Just as now, you wouldn't *need* to call it at all. Unlike now, the parsing pass probably needs to save away some more info (e.g., I don't *think* it keeps track of what all is in a module now in any usable way). For Python2, I hope Guido adopts a more traditional structure (i.e., parsing produces a parse tree, codegen produces bytecode from a parse tree, and other tree->tree transformers can be plugged in between them). Almost all compilers follow this structure, and not because compiler writers are unimaginative droids . Compile-time for Python isn't anywhere near being a problem now, even on my creaky old 166MHz machine; I suspect the current structure reflects worry about that on much older & slower machines. Some of the most useful Perl msgs need to be in com_xxx, though, or even earlier. The most glaring example is runaway triple-quoted strings. Python's "invalid token" pointing at the end of the file is maddeningly unhelpful; Perl says it looks like you have a runaway string, and gives the line number it thinks it may have started on. That guess is usually correct, or points you to what you *thought* was the end of a different string. Either way your recovery work is slashed. (Of course IDLE is even better: the whole tail of the file changes to "string color", and you just need to look up until the color changes!) > In addition, more global checks probably won't be possible (reasoning about > code across module boundaries for instance) because the compiler's view of > the world is fairly narrow. As above, I don't think there's enough now even to analyze one module in isolation. > I think lint-like tools should be implemented in Python (possibly with the > support of an extension module for performance-critical sections) which is > then called from the compiler proper under appropriate circumstances > (warnings enabled, all necessary modules importable, etc). I have no objection to that. I do object to the barely conceivable getting in the way of the plainly achievable, though -- the pylint tools out there now, just like your return consistency checker, do a real service already without any global analysis. Moving that much into the core (implemented in Python if possible, recoded in C if not) gets a large chunk of the potential benefit for maybe 5% of the eventual work. It's nice that Guido is suddenly keen on global analysis too, but I don't see him volunteering to do any work either . > I believe the code would be much more easily maintained and extended. If it's written in Python, of course. > You'd be able to swap in a new byte code compiler without risking the > loss of your checking code. I never understood this one; even if there *were* a competing byte code compiler out there <0.1 wink>, keeping as much as possible out of com_xxx should render it a non-issue. If I'm missing your point and there is some fundamental conflict here, fine, then it's another basis on which bytecode compilers will compete. more-concerned-about-things-that-exist-than-things-that-don't-ly y'rs - tim From iporres at abo.fi Sun Sep 12 11:54:36 1999 From: iporres at abo.fi (Ivan Porres) Date: Sun, 12 Sep 1999 12:54:36 +0300 Subject: [Python-Dev] Memory usage in Python Message-ID: <19990912125436.B735@marchica.abo.fi> Dear all, I've been reading this list for a while but this is the first time I post a message. I am interested on using Python as a scripting language for small embedded systems. One of the limiting factors on a embedded system is the memory. Memory cost money, uses space and drains power. We have released a big patch to selectively disable some features of a Python interpreter at compile time. With the patch, it is possible to disable, i.e., complex numbers or file objects, in order to reduce the memory foot print of the Python interpreter. The patch is available at http://www.abo.fi/~iporres/python Now, I am interested on studying and reducing the memory consumption of Python programs. That is all the memory allocated by the interpreter in order to run a python program. I would like to know if somebody else has been working on the same direction. Is there any document explaining how Python uses the memory? I would be nice to know which structures are most used, their size, how often are accessed, etc... I have the impression that most people is concerned about optimising Python for speed, but I still think that some Python users will also benefit from size optimisations. Ivan -- Ivan Porres Paltor Turku Centre for Computer Science ?bo Akademi, Department of Computer Science Phone: +358-2-2154033 Lemmink?inengatan 14A FIN-20520 Turku - Finland http://www.abo.fi/~iporres From skip at mojam.com Sun Sep 12 16:20:14 1999 From: skip at mojam.com (Skip Montanaro) Date: Sun, 12 Sep 1999 09:20:14 -0500 (CDT) Subject: [Python-Dev] Catching "return" and "return expr" at compile time In-Reply-To: <000001befcff$bf654e20$492d153f@tim> References: <199909101405.KAA26303@eric.cnri.reston.va.us> <000001befcff$bf654e20$492d153f@tim> Message-ID: <14299.46750.214399.735453@dolphin.mojam.com> Skip> * Future extensions like pluggable virtual machines will be harder Skip> to add because their byte code compilers will be harder to Skip> integrate into the core Tim> If you're picturing adding this stuff sprayed throughout the guts Tim> of the existing com_xxx routines, we've got different pictures in Tim> mind. This was precisely my example, because that's the way I implemented the return warning stuff, by modifying the com_xxx routines. I believe that's the wrong way to go in the long run, and I see by the rest of your message you feel the same way as well. To the greatest extent possible, I think this stuff should be implemented in Python. (We may disagree on that point.) Being able to plug in new parse tree analysis/transformation modules between parse tree creation and code generation could at least be controlled from Python. Skip P.S. Something I just noticed: Since the node typedef (node.h) and the macros that manipulate nodes are shared across multiple files shouldn't they be named something slightly less likely to clash with other packages? From mal at lemburg.com Mon Sep 13 12:05:46 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:05:46 +0200 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> Message-ID: <37DCCC7A.545B44E6@lemburg.com> Tim Peters wrote: > > [Guido] > > I'd much rather use absolute package names for anything that's not in > > the same directory as the current module. > > [M.-A. Lemburg] > > Of course, you could do everything with absolute names, but then > > the package author would dictate the complete absolute path which > > is not always desirable since it can cause name collisions such > > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > > released) and Mark's win32 stuff. > > > > As more and more packages appear, we run into this problem more > > and more often. > > I never understand package complaints. Maybe it's the imprecision of the > language people use, or maybe because it's because people don't give fully > fleshed-out examples. Whatever, in the end, I never have any idea what the > complaint is, or in what way the solution is "solving" anything. My original post contained an example package using relative imports. The example uses intra-package imports across subpackage levels which currently is only possible using absolute module names (see below for a definition). Note that I did not want to start a discussion about absolute vs. relative names (I believe everybody agrees that realtive file names are a Good Thing). The 'import __.module' thing is not new: ni.py had support for this and my patch simply adds it back to the implementation. Think of the '__' as an additional feature that authors can use at their own will. They don't *have* to, just as you don't have to use relative file paths in your config files. It's just a convenience that some authors may want to use in their code. > In the above, "absolute" doesn't appear to mean "absolute" in any > OS-sensible sense. So what does it really mean? Does it mean the same > things to Guido and MAL? "Absolute" means fully qualified name, i.e. the complete path to the modules from the top-level root via all subpackage down to the module name itself, e.g. TextTools.Constants.TagTable. > In MAL's hint of examples, I don't see any problem. If mxDateTime unpacks > itself into a directory named DateTime, then *of course* it's going to > collide with other packages that want to do likewise. Install it into > mxDateTime instead, and take "absolute" to mean "any module that wants an > mxDateTime service and does not itself live directly in mxDateTime/ must > import the desired module via a path beginning 'mxDateTime.'", and > everything looks straightforward to me (and that outcome makes me infer that > this is thus probably what Guido has in mind too). This is what I intend to do: move all my stuff under a 'mx' package and then continue reinventing (faster, different, doesn't-slip-when-wet) wheels ;-). BTW, the queue thingie is just a proof of concept fun project. > Similarly for > win32.Queue vs mxQueue.Queue vs the Queue in the std library (it *would* be > good to have an explicit way to say "std library" -- "Lib." comes to mind). I'd prefer 'org.python.core.' or 'stdlib.' or just simply 'python.' but that's a different issue. > > I could then make all my packages self-contained and > > distribute them in two forms without having to change > > a single line for the added support: > > > > 1. under the new 'mx' package, e.g. mx.DateTime > > 2. for backward compatibility under 'DateTime' > > Ah, so that's what this is about. I vote screw #2. Releasing it that way > was a mistake. Not until Zope went Open Source ;-) ... > Better to live with the brief & finite pain of repairing it > than complicating Python to cover up for it. Screwing #2 is not possible unless I want all those already stored DateTime pickles to fail loading... ok, I could probably provide some kind of compatibility package which then redirects the import to mx.DateTime. > > Another major advantage is that I could take any other > > self-contained package and install it under absolute paths > > of my choice, e.g. put Zope under org.zope.core, Python under > > org.python.core etc., without harming their functionality > > or having to dive deep into their import structures to fix > > them manually. > > I view that not as an advantage but as harmful complication. Zope etc add > great value to a Python installation, and when I write a killer app full of > "import zope.this" and "import zope.that", I don't want the possibility that > it's not going to work on my client's machine just because their sysadmin > installed Zope into some silly site-specific path named after his soon-to-be > ex-girlfriend . I don't want a way to work around him doing that, > either: I don't want him to be able to screw me to begin with. Zope is not a package AFAIK (and probably never will be due to the pickle complications), so 'import zope.this' won't work anyways unless you add a Zope wrapping package of your own -- and this will only work for Zope modules not relying on other Zope modules unless they use relative imports. > > To further enhance this mechanism I would like to have an > > alias mechanism in import, pickle et al. so that changes > > in the package structures become manageable without user > > intervention: pickles are a major problem whenever import > > structures change because they store absolute module names. > > This is a different issue, and may have merits of its own. WRT the relative > import scheme, its advantage seems to lie in providing a way to partially > recover from the damage the new scheme causes <0.5 wink>. I'm not proposing a new scheme... only a convenience for package authors. > As is, the package name used by a release is part of its published > interface. You can't change it without causing pain, any more than you can > make incompatible changes to public class methods or input-output behavior. > In return, package clients are uniform, simple and portable, making life > easiest for the people who know least. The burden is on package authors to > choose names wisely, and that's where the burden should be. Sure, but we are heading into these kind of problems just now and have no proper solution at hand. Simply coping out by blaiming the package authors is not the right way to handle the situation; referring them to some global name registry (the one at NIST) isn't either. As the Zope example shows, it's not easy to move from half-packaged to full packaged (relative imports will ease this move, though) due to external references using absolute object type/class names. Changes in the package structure produce the same problems. It's not so much the burden of the programmer having to adjust his code to the new layout I'm talking about here: it's the user with all his data stored in object databases that worries me. > if-100-pkgs-all-want-their-own-queue-they-already-can-ly y'rs - tim Not if just one of them decides to make it top-level... grabbing the name until eternity ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 13 12:17:31 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:17:31 +0200 Subject: [Python-Dev] Relative Package Imports References: <1275018545-12250772@hypernet.com> Message-ID: <37DCCF3B.2F1E962D@lemburg.com> Gordon McMillan wrote: > > M.-A. Lemburg wrote: > > Guido van Rossum wrote: > > > > > > I'd much rather use absolute package names for anything that's not > > > in the same directory as the current module. > > > > Of course, you could do everything with absolute names, but then > > the package author would dictate the complete absolute path which is > > not always desirable since it can cause name collisions such as > > DateTime in mxDateTime and Zope or Queue in mxQueue (to be released) > > and Mark's win32 stuff. > > I can see your point (although I also believe that authors - Guido > excepted - should come up with collision-free names, probably in a > Java-ish scheme). Agreed. > But I strongly believe that import.c should be left alone, maybe > even to die. There are too many people doing import hooks to > make fiddling with its behavior safe. > > I'm also a strong proponent of Greg's imputil.py scheme, which > makes it a breeze to do import hooks. And my experience > disproves the notion that the import mechanism needs to be in C. If > you don't believe me, try the ZlibArchive stuff (which is cross > platform) from my Win32 installer stuff. You can pack the standard > library into one 475K file, and get a perceptible performance boost. You're probably right in saying that we don't need the code in C. I just wanted to avoid yet another import hook being incompatible with all the other existing hooks. Perhaps we should restart the import discussion all over and come up with a more flexbile 100% compatible framework based on Greg's imputil scheme. Then I could add my hook for the relative imports and be happy ;-) > OTOH, I could see doing a framework of packages, in which case > relative imports might be handy. This seems to work fine: > > def relimp(nm): > rpth = string.split(nm, '/') > tpth = string.split(__name__, '.')[:-1] > for node in rpth: > if node == '..': > del tpth[-1] > else: > tpth.append(node) > #print `tpth` > return __import__(string.join(tpth, '.')) > > b = relimp('../packageA2.b') This is pretty much how my patch works... except that I use the ni.py style '__' pseudo package instead of '../'. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 13 12:27:04 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 12:27:04 +0200 Subject: [Python-Dev] Memory usage in Python References: <19990912125436.B735@marchica.abo.fi> Message-ID: <37DCD178.15676B9F@lemburg.com> Ivan Porres wrote: > > Dear all, > > I've been reading this list for a while but this is the first time I > post a message. I am interested on using Python as a scripting > language for small embedded systems. One of the limiting factors on a > embedded system is the memory. Memory cost money, uses space and > drains power. > > We have released a big patch to selectively disable some features of a > Python interpreter at compile time. With the patch, it is possible to > disable, i.e., complex numbers or file objects, in order to reduce > the memory foot print of the Python interpreter. The patch is > available at http://www.abo.fi/~iporres/python > > Now, I am interested on studying and reducing the memory consumption of > Python programs. That is all the memory allocated by the interpreter > in order to run a python program. I would like to know if somebody > else has been working on the same direction. Is there any document > explaining how Python uses the memory? I would be nice to know which > structures are most used, their size, how often are accessed, > etc... You should have a look at Vladimir Marangozov's pymalloc (hope this URL still works): http://sirac.inrialpes.fr/~marangoz/python/ It has some logging facilities which give detailed reports about the storage sizes etc. > I have the impression that most people is concerned about optimising > Python for speed, but I still think that some Python users will > also benefit from size optimisations. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at digicool.com Mon Sep 13 14:44:56 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 08:44:56 -0400 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> Message-ID: <37DCF1C8.27CF657F@digicool.com> Guido van Rossum wrote: > > > I've spent the last two hours trying to get relative package > > imports to work because I need them for my extension packages > > which will soon all move under a new top-level package name > > to overcome the conflicts with PIL and Zope. > > I'd much rather use absolute package names for anything that's not in > the same directory as the current module. I'll second Marc-Andre here. A significant headache occurs when you have a package that has sub-packages. Sub-packages need to be able to reference other sub-packages within the same package without knowing where the containing package is installed. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From guido at CNRI.Reston.VA.US Mon Sep 13 15:09:59 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 13 Sep 1999 09:09:59 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 08:44:56 EDT." <37DCF1C8.27CF657F@digicool.com> References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> Message-ID: <199909131309.JAA29239@eric.cnri.reston.va.us> [MAL] > > > I've spent the last two hours trying to get relative package > > > imports to work because I need them for my extension packages > > > which will soon all move under a new top-level package name > > > to overcome the conflicts with PIL and Zope. [GvR] > > I'd much rather use absolute package names for anything that's not in > > the same directory as the current module. [JimF] > I'll second Marc-Andre here. > > A significant headache occurs when you have a package > that has sub-packages. Sub-packages need to be able to > reference other sub-packages within the same package without > knowing where the containing package is installed. You never need to know where it is installed. When I said absolute package name I meant package name (e.g. zope.foo.bar.subpack) not filename. As Tim has argued, the ability to change the name of the toplevel here is a liability, not a feature. (On reduced bandwidth here, hence not so subtle.) --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond at skippinet.com.au Mon Sep 13 15:14:56 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Mon, 13 Sep 1999 23:14:56 +1000 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DCF1C8.27CF657F@digicool.com> Message-ID: <000d01befde9$f9c70690$0801a8c0@bobcat> [Jim writes] > I'll second Marc-Andre here. > > A significant headache occurs when you have a package > that has sub-packages. Sub-packages need to be able to > reference other sub-packages within the same package without > knowing where the containing package is installed. I certainly dont want to know _where_ the package is installed, but do want to know the package name. A package name, IMO, is like a module name - once named, you are largely stuck with it. I for one use packages quite extensively, but have never lamented the loss of __ from the ni days. If someone, for example, wanted to change the package name of "win32com" or "pywin", then I would expect lots of code to break, in the same way as changing the name any standard module would break code. I see Marc's issue with pickle, but I dont see the problem with absolute package names any more than I see a problem with explicitely naming modules in an import statement... Mark. From jim at digicool.com Mon Sep 13 15:32:25 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:32:25 -0400 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> Message-ID: <37DCFCE9.A00DB3E0@digicool.com> Tim Peters wrote: > (snip) > As is, the package name used by a release is part of its published > interface. You can't change it without causing pain, any more than you can > make incompatible changes to public class methods or input-output behavior. > In return, package clients are uniform, simple and portable, making life > easiest for the people who know least. The burden is on package authors to > choose names wisely, and that's where the burden should be. Not all packages are part of the external interface. In fact, all Zope names are essentially internal, since Zope is an application. The issue is not so much access to access from outside as it is access between packages within Zope. Further, the current support for relative imports allows a package to be moved into another package without breaking the pulic interface wrt the containing package. Here's an example that I hope will be motivating: Suppose Marc-Andre has a package mx with subpackages DateTime and stringtools. If mx was installed in the Python path then a module in the mx.DateTime package could get at stringtools like: import mx.stringtools So far, so good. Zope has a notion of products which are *self contained* packages that are sub-packages of the Products package. So, suppose someone wants to write a NiftyDB product, which is a Zope product that provides access to an external database. Now the author of the NiftyDB product wants to use the mx package. The mx package is not a standard part of Zope, or of Python, so they simpley include it in the NiftyDB product directory. Becase relative imports are allowed in the current import scheme, they can use mx as usual. A NiftyDB module can import DateTime as follows: import mx.DateTime So even though mx is istalled as a sub-package, the public interface is unchanged, at least wrt the containing package. Unfortunately, the internal import of stringtools in the DateTime package: import mx.stringtools will fail, because mx is no longer a top-level module. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Mon Sep 13 15:32:39 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:32:39 -0400 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> Message-ID: <37DCFCF7.DACEF21D@digicool.com> "M.-A. Lemburg" wrote: > > Tim Peters wrote: > > > > [Guido] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > > > > [M.-A. Lemburg] > > > Of course, you could do everything with absolute names, but then > > > the package author would dictate the complete absolute path which > > > is not always desirable since it can cause name collisions such > > > as DateTime in mxDateTime and Zope or Queue in mxQueue (to be > > > released) and Mark's win32 stuff. > > > > > > As more and more packages appear, we run into this problem more > > > and more often. > > > > I never understand package complaints. Maybe it's the imprecision of the > > language people use, or maybe because it's because people don't give fully > > fleshed-out examples. Whatever, in the end, I never have any idea what the > > complaint is, or in what way the solution is "solving" anything. > > My original post contained an example package using relative > imports. The example uses intra-package imports across subpackage > levels which currently is only possible using absolute module > names (see below for a definition). I agree that this is a valid complaint about the current scheme. > Note that I did not want to start a discussion about absolute vs. > relative names (I believe everybody agrees that realtive file names > are a Good Thing). The 'import __.module' thing is not new: ni.py > had support for this and my patch simply adds it back to the > implementation. And, in fact, the current scheme does let you use relative paths to go down. (snip) > > > I could then make all my packages self-contained and > > > distribute them in two forms without having to change > > > a single line for the added support: > > > > > > 1. under the new 'mx' package, e.g. mx.DateTime > > > 2. for backward compatibility under 'DateTime' > > > > Ah, so that's what this is about. I vote screw #2. Releasing it that way > > was a mistake. > > Not until Zope went Open Source ;-) ... > > > Better to live with the brief & finite pain of repairing it > > than complicating Python to cover up for it. > > Screwing #2 is not possible unless I want all those already > stored DateTime pickles to fail loading... ok, I could probably > provide some kind of compatibility package which then redirects > the import to mx.DateTime. Regardless of how you spell the import, the pickles *must* reflect the absolute path. Otherwise, pickleability depends on where you unpickle. If you rearrange packages, or rename modules, there is a pickling issue. This is clearly a problem in need of a better solution. (snip) > Zope is not a package AFAIK Nope. If I had it to do over it would be. It will be eventually. (While Zope is not a package, it makes extensive use of packges. Zope has a total of about 350 modules and packages, of which about 44 are in the top-level namespace.) > (and probably never will be due to the pickle complications), The pickling issues are solvable in a number of ways, although it is a bit painfull. I considered fixing the package layout in Zope 2, but I ran out of time. :) > so 'import zope.this' won't work anyways > unless you add a Zope wrapping package of your own -- and this will > only work for Zope modules not relying on other Zope modules unless > they use relative imports. This is a good example. It should be possible to make Zope package- portable but it isn't, at least not without writing import hooks, which make my head hurt alot more than meta-classes. ;) > > > To further enhance this mechanism I would like to have an > > > alias mechanism in import, pickle et al. so that changes > > > in the package structures become manageable without user > > > intervention: pickles are a major problem whenever import > > > structures change because they store absolute module names. > > > > This is a different issue, and may have merits of its own. WRT the relative > > import scheme, its advantage seems to lie in providing a way to partially > > recover from the damage the new scheme causes <0.5 wink>. > > I'm not proposing a new scheme... only a convenience for package > authors. I think that this is an important convenience that is probably not appreciated until you write a complex package structure. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Mon Sep 13 15:35:21 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 09:35:21 -0400 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> <199909131309.JAA29239@eric.cnri.reston.va.us> Message-ID: <37DCFD99.F48FC65E@digicool.com> Guido van Rossum wrote: > > [MAL] > > > > I've spent the last two hours trying to get relative package > > > > imports to work because I need them for my extension packages > > > > which will soon all move under a new top-level package name > > > > to overcome the conflicts with PIL and Zope. > [GvR] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > [JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. > > You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. I wasn't refering to file path location either, but to package name-space location. See the example I just posted in reply to Tim. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From guido at CNRI.Reston.VA.US Mon Sep 13 15:40:10 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 13 Sep 1999 09:40:10 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 12:05:46 +0200." <37DCCC7A.545B44E6@lemburg.com> References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> Message-ID: <199909131340.JAA29340@eric.cnri.reston.va.us> > Note that I did not want to start a discussion about absolute vs. > relative names (I believe everybody agrees that realtive file names > are a Good Thing). The 'import __.module' thing is not new: ni.py > had support for this and my patch simply adds it back to the > implementation. It was left out of the 1.5 package implementation on purpose, and I will fight proposals to get it back. It is an ugly hack that deserves to die. So there! > "Absolute" means fully qualified name, i.e. the complete path to > the modules from the top-level root via all subpackage down to the > module name itself, e.g. TextTools.Constants.TagTable. No disagreement there. BTW, The compatibility issues with class references in object databases need to be resolved in a different way -- package renamings are but one of the problems here. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at lemburg.com Mon Sep 13 15:57:32 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 15:57:32 +0200 Subject: [Python-Dev] Relative Package Imports References: <37D956B4.3E79C2C1@lemburg.com> <199909101916.PAA27287@eric.cnri.reston.va.us> <37DCF1C8.27CF657F@digicool.com> <199909131309.JAA29239@eric.cnri.reston.va.us> Message-ID: <37DD02CC.26B20BDD@lemburg.com> Guido van Rossum wrote: > > [MAL] > > > > I've spent the last two hours trying to get relative package > > > > imports to work because I need them for my extension packages > > > > which will soon all move under a new top-level package name > > > > to overcome the conflicts with PIL and Zope. > [GvR] > > > I'd much rather use absolute package names for anything that's not in > > > the same directory as the current module. > [JimF] > > I'll second Marc-Andre here. > > > > A significant headache occurs when you have a package > > that has sub-packages. Sub-packages need to be able to > > reference other sub-packages within the same package without > > knowing where the containing package is installed. > > You never need to know where it is installed. When I said absolute > package name I meant package name (e.g. zope.foo.bar.subpack) not > filename. As Tim has argued, the ability to change the name of the > toplevel here is a liability, not a feature. I think Jim meant the absolute package name just like you and I did. I don't really understand why you and Tim oppose to relative imports... Tim's argument about sysadmins installing packages under fantasy names isn't really all that realistic, IMHO, even though it would work. The ability to host a subpackage inside another package is a very attractive feature for large package based applications: you don't have to edit a single line in the subpackage's source code and can install/update the subpackage without harming its functionality within the system. BTW: The patch I posted does not add a circular reference to the modules as the old ni.py implementation (sometimes) did. Since the __ handling is done dynamically and on import only there is no need to add a module attribute __ pointing to the parent module. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 13 16:08:36 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 16:08:36 +0200 Subject: [Python-Dev] Relative Package Imports References: <000901befc94$6a463d20$6aa2143f@tim> <37DCCC7A.545B44E6@lemburg.com> <199909131340.JAA29340@eric.cnri.reston.va.us> Message-ID: <37DD0564.7361B789@lemburg.com> Guido van Rossum wrote: > > > Note that I did not want to start a discussion about absolute vs. > > relative names (I believe everybody agrees that realtive file names > > are a Good Thing). The 'import __.module' thing is not new: ni.py > > had support for this and my patch simply adds it back to the > > implementation. > > It was left out of the 1.5 package implementation on purpose, and I > will fight proposals to get it back. It is an ugly hack that deserves > to die. So there! I agree that it's ugly to include the __ attribute in the module namespace due to the possible circular reference (parent->module, module->parent), but the patch I sent doesn't do this... or was "ugly" referring to the two underscores looking strange ? Could you elaborate a bit on the reasons for dropping __ support ? Not-really-looking-for-a-fight-here--ly, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Mon Sep 13 16:35:16 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Mon, 13 Sep 1999 10:35:16 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909131309.JAA29239@eric.cnri.reston.va.us> References: Your message of "Mon, 13 Sep 1999 08:44:56 EDT." <37DCF1C8.27CF657F@digicool.com> Message-ID: <1274889332-20022765@hypernet.com> A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 1587 bytes Desc: not available URL: From gmcm at hypernet.com Mon Sep 13 16:56:12 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Mon, 13 Sep 1999 10:56:12 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DCFCE9.A00DB3E0@digicool.com> Message-ID: <1274888077-20098343@hypernet.com> [Jim Fulton] > Here's an example that I hope will be motivating: > > Suppose Marc-Andre has a package mx with subpackages DateTime > and stringtools. ... > Zope has a notion of products which are *self contained* packages that > are sub-packages of the Products package. So, suppose someone wants > to write a NiftyDB product, which is a Zope product that provides > access to an external database. Now the author of the NiftyDB product > wants to use the mx package. The mx package is not a standard part of > Zope, or of Python, so they simpley include it in the NiftyDB product > directory. This would motivate me to strangle the author of NiftyDB. He should say "This requires Marc-Andre's mx package", and put error handling on the import. Otherwise I might end up with a gazillion separate copies of the mx package installed on my system and no way to clean it up. Your notion of "self contained packages" is flawed. Reducto ad adsurdum, it leads to downloading the OS with every app you install. (Crap, I hope I haven't given Sun any new ideas...). - Gordon From jim at digicool.com Mon Sep 13 17:16:03 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 11:16:03 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> Message-ID: <37DD1533.24A05F69@digicool.com> Gordon McMillan wrote: > > [Jim Fulton] > > Here's an example that I hope will be motivating: > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > and stringtools. > ... > > Zope has a notion of products which are *self contained* packages that > > are sub-packages of the Products package. So, suppose someone wants > > to write a NiftyDB product, which is a Zope product that provides > > access to an external database. Now the author of the NiftyDB product > > wants to use the mx package. The mx package is not a standard part of > > Zope, or of Python, so they simpley include it in the NiftyDB product > > directory. > > This would motivate me to strangle the author of NiftyDB. He > should say "This requires Marc-Andre's mx package", and put error > handling on the import. This is unacceptable. People should not have to hack a global namespace to get a local feature. > Otherwise I might end up with a gazillion > separate copies of the mx package installed on my system and no > way to clean it up. Or you might not. In any case, the ability to have localized references should be there. Actually, Zope does provide a place to share packages that multiple products use. This involves putting shared packages in a sub-package of the 'Shared' package. For example, Digital Creations shared packages go in Shared.DC. Putting mx in Shared.ACME runs into the same problem. Although the product author can: import Shared.ACME.mx.DateTime the import of mx.stringtools in DateTime will still fail. Now, maybe ACME will hack mx to change the internal imports: import Shared.ACME.mx.stringutils but their hacks will get broken when we make Zope a package and Shared becomes a sub-package of Zope. > Your notion of "self contained packages" is flawed. Reducto ad > adsurdum, it leads to downloading the OS with every app you > install. (Crap, I hope I haven't given Sun any new ideas...). I'm sorry, but relying on existing Python installations for commercial applications is just not practical. In fact, Zope binary distributions contain their own Python distributions because relying on preinstalled Python distributions is just way too much of a support burden and a hassle for out customers. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Mon Sep 13 17:21:15 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 11:21:15 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> Message-ID: <37DD166B.746B19AD@digicool.com> Gordon McMillan wrote: > > [relative package imports] > [JimF] > > > I'll second Marc-Andre here. > > > > > > A significant headache occurs when you have a package > > > that has sub-packages. Sub-packages need to be able to > > > reference other sub-packages within the same package without > > > knowing where the containing package is installed. > [GvR] > > You never need to know where it is installed. When I said absolute > > package name I meant package name (e.g. zope.foo.bar.subpack) not > > filename. As Tim has argued, the ability to change the name of the > > toplevel here is a liability, not a feature. > > In between. I can see relative packages as *one* way of handling > certain problems. Consider: > > zope.Win32 > zope.Unix > > with both of these having alternate implementations of subpackages foo > and bar. Then for (the current) foo.a to get to (the current) bar.b, > using a relative import seems a natural. I wouldn't use relative imports for this. > This can, of course, be done in pure Python. So can doing things in > zope.__init__.py that make the appropriate implementations of foo and > bar appear to be zope.foo and zope.bar. On any criteria I can think of, > this would be a superior solution. (*) I agree. This is not a good motivating example for relative imports. > What I am against is further complicating the already over complicated > built in import mechanism. The current scheme doesn't seem all that complicated to me. I wouldn't be in favor of making it more complicated if there wasn't a good reason. I think that, in working on the Zope framework, I've found some pretty good reasons for relative imports. > > (*) such as a zope.__init__.py that looks like this: > > import sys > if sys.platform[:3] == 'win': > nm = __name__ + '.Win32' > else: > nm = __name__ + '.Unix' > new = __import__(nm) > sys.modules[__name__] = sys.modules[nm] Right, this would be silly. This is not a good example. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal at lemburg.com Mon Sep 13 17:27:31 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 17:27:31 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> Message-ID: <37DD17E3.74FEB43F@lemburg.com> Gordon McMillan wrote: > > [Jim Fulton] > > Here's an example that I hope will be motivating: > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > and stringtools. > ... > > Zope has a notion of products which are *self contained* packages that > > are sub-packages of the Products package. So, suppose someone wants > > to write a NiftyDB product, which is a Zope product that provides > > access to an external database. Now the author of the NiftyDB product > > wants to use the mx package. The mx package is not a standard part of > > Zope, or of Python, so they simpley include it in the NiftyDB product > > directory. > > This would motivate me to strangle the author of NiftyDB. He > should say "This requires Marc-Andre's mx package", and put error > handling on the import. Otherwise I might end up with a gazillion > separate copies of the mx package installed on my system and no > way to clean it up. The mx package installed in product dir won't be visible to outside scripts: it is self-contained and only works in the context of Zope. There's nothing bad about it if you take the POV of a typical Zope user who doesn't know about the internals too much. OTOH, if the user does have the skill to install packages himself, he could install the mx package top-level and then leave it out of the NiftyDB product dir. Both setups will work without any source code change... that's the real advantage. Currently, only the top-level installation of the mx package will work and this of course introduces other problems like version control, e.g. the NiftyDB product (relying on mx 1.9) may not support version 2.0 of mx which the sysadmin installed at top-level. There would be no way (apart from patching NiftyDB) to get back to a working setup. > Your notion of "self contained packages" is flawed. Reducto ad > adsurdum, it leads to downloading the OS with every app you > install. (Crap, I hope I haven't given Sun any new ideas...). While you do of course have a point in that packages should not be loaded twice (a smarter import mechanism written in Python could handle these situations along with the relative import), you should consider that Zope is an application. It could go the WinXX way and simply hack around in the OS or wrap everything it needs up in the package itself. Both have their pros and cons, both are possible using relative imports (I'm starting to feel like one of those marketing guys ;). BTW, I really don't care much about this feature being coded in Python or C. All I would like is to have it enabled and available per default, so that writing self-contained packages becomes a standard in the Python world. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jcw at equi4.com Mon Sep 13 18:23:35 1999 From: jcw at equi4.com (Jean-Claude Wippler) Date: Mon, 13 Sep 1999 18:23:35 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> Message-ID: <37DD2507.8A7AD0D1@equi4.com> "M.-A. Lemburg" wrote: > Gordon McMillan wrote: > > Jim Fulton wrote: [deeply nested snippety snip] > > > > This would motivate me to strangle the author of NiftyDB. He > > should say "This requires Marc-Andre's mx package", [...] Yes. > The mx package installed in product dir won't be visible to outside > scripts: it is self-contained and only works in the context of Zope. But why shouldn't it be visible? The only approach which will not self-destruct IMO, is to segment on source-of-origin. Your package in your namespace, Gordon's in his, and Jim's in yet another. Given that there is no ordering relationship, that means three areas next to each other. If Jim takes M-A's package and modifies it, then that would be a reason to put M-A's-modified-by-Jim-package within Jim's area. > Currently, only the top-level installation of the mx package > will work and this of course introduces other problems like > version control, e.g. the NiftyDB product (relying on mx 1.9) > may not support version 2.0 of mx which the sysadmin installed > at top-level. There would be no way (apart from patching NiftyDB) > to get back to a working setup. Which points to a solution in a completely different direction: version numbers in package directories, like GNU and Tcl do, to name two. > [...] have it enabled and available per default, so that writing > self-contained packages becomes a standard in the Python world. A noble goal. Now this may be a silly question, but what's the issue here? I'm used to writing things which sort of look like this: import myGoobledygook utils = myGoobledygook print utils.fun(123) Couldn't this be applied here as well: mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") In other words: figure out a way to get at the proper modules, then use an alias in your own code to stay away from naming/access dependencies? And if a parent wants to tell a submodule how to find it, can't it just set a variable at module level in that submodule, to reach it? Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, DNS or LDAP servers, FS mount points - it's all the same issue. -- Jean-Claude From jim at digicool.com Mon Sep 13 18:56:42 1999 From: jim at digicool.com (Jim Fulton) Date: Mon, 13 Sep 1999 12:56:42 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> Message-ID: <37DD2CCA.B5C29881@digicool.com> Jean-Claude Wippler wrote: > (snip) > > The mx package installed in product dir won't be visible to outside > > scripts: it is self-contained and only works in the context of Zope. > > But why shouldn't it be visible? Because visibility has a cost. Making mx a top-level name means that someone else can't make it a top-level name. This is why packages are a good idea. If the need is local, the definition shouldn't be made global *just* to get around a limitation in spelling. > The only approach which will not > self-destruct IMO, is to segment on source-of-origin. Your package in > your namespace, Gordon's in his, and Jim's in yet another. Given that > there is no ordering relationship, that means three areas next to each > other. It must be possible to package things together regardless of point of origin. Otherwise, you have a big obstical to reuse. BTW, I think that there is ample evidence that relative imports will not cause anything to self destruct. > If Jim takes M-A's package and modifies it, then that would be a > reason to put M-A's-modified-by-Jim-package within Jim's area. But Jim wants to use M-A's package as a black box. I don't want to hack all of his imports due to a packaging detail. (snip) > Now this may be a silly question, but what's the issue here? I'm used > to writing things which sort of look like this: > import myGoobledygook > utils = myGoobledygook > print utils.fun(123) > Couldn't this be applied here as well: > mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") > In other words: figure out a way to get at the proper modules, then use > an alias in your own code to stay away from naming/access dependencies? > > And if a parent wants to tell a submodule how to find it, can't it just > set a variable at module level in that submodule, to reach it? > > Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, > DNS or LDAP servers, FS mount points - it's all the same issue. Yes, these are options, as are import hooks. While these are possible alternatives, they are far less attractive than simply making it possible to spell relative imports. Why? Well, for one, they require a lot of cooperation among packages that might be written by totally different authors. I might come up with a handy relative import function, but how do I get M-A to use it? Well, I'm sure I could get him to use it, but how about all of the other people whose packages I want to reuse. No problem, we'll come up with a standard mechansism, maybe even a standard library module. Of course, someone is bound to realize sooner or later that it's silly to have a standard relative import library module *and* a standard import statement and add the feature to standard import. :) Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal at lemburg.com Mon Sep 13 20:21:08 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 13 Sep 1999 20:21:08 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> Message-ID: <37DD4094.1BDCE45F@lemburg.com> Jean-Claude Wippler wrote: > > "M.-A. Lemburg" wrote: > > Gordon McMillan wrote: > > > Jim Fulton wrote: > [deeply nested snippety snip] > > > > > > This would motivate me to strangle the author of NiftyDB. He > > > should say "This requires Marc-Andre's mx package", [...] > > Yes. Of course ;-) ... but that's not the point. Commercial products have to come with all batteries included, not just some of them. But you can't include everything known in the Python universe just to avoid conflicts when someone writes a cool plugin for your package. > > The mx package installed in product dir won't be visible to outside > > scripts: it is self-contained and only works in the context of Zope. > > But why shouldn't it be visible? Because it would probably affect other apps using the same package. Just think of what happens to Win9x if you constantly update the DLLs... > The only approach which will not > self-destruct IMO, is to segment on source-of-origin. Your package in > your namespace, Gordon's in his, and Jim's in yet another. Given that > there is no ordering relationship, that means three areas next to each > other. If Jim takes M-A's package and modifies it, then that would be a > reason to put M-A's-modified-by-Jim-package within Jim's area. > > > Currently, only the top-level installation of the mx package > > will work and this of course introduces other problems like > > version control, e.g. the NiftyDB product (relying on mx 1.9) > > may not support version 2.0 of mx which the sysadmin installed > > at top-level. There would be no way (apart from patching NiftyDB) > > to get back to a working setup. > > Which points to a solution in a completely different direction: version > numbers in package directories, like GNU and Tcl do, to name two. A different topic -- an interesting one too :-) How does Java handle this problem ? > > [...] have it enabled and available per default, so that writing > > self-contained packages becomes a standard in the Python world. > > A noble goal. And one which is only reachable by getting whatever mechanism is needed to make it work into the standard distribution -- not only a hook for relative imports but the real thing ! > Now this may be a silly question, but what's the issue here? I'm used > to writing things which sort of look like this: > import myGoobledygook > utils = myGoobledygook > print utils.fun(123) > Couldn't this be applied here as well: > mxDT = aCoolImporter("Marc-Andre's latest date/time utilities") > In other words: figure out a way to get at the proper modules, then use > an alias in your own code to stay away from naming/access dependencies? > > And if a parent wants to tell a submodule how to find it, can't it just > set a variable at module level in that submodule, to reach it? > > Aliases, bookmarks, symlinks, shortcuts, env vars, globals, registries, > DNS or LDAP servers, FS mount points - it's all the same issue. True, but please remember that we are not talking about adding some esoteric module storage to the standard mechanism (the one reachable via the "import" statement). Relative import only adds a logical notion to imports that opens up new possibilities which are not easily available using other techniques. BTW, would you want to miss relative file name paths ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 109 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at interet.com Mon Sep 13 22:52:57 1999 From: jim at interet.com (James C. Ahlstrom) Date: Mon, 13 Sep 1999 16:52:57 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> Message-ID: <37DD6429.E4F6BF92@interet.com> Jim Fulton wrote: > > Gordon McMillan wrote: > > > > [Jim Fulton] > > > Here's an example that I hope will be motivating: > > > > > > Suppose Marc-Andre has a package mx with subpackages DateTime > > > and stringtools. > > ... > > > Zope has a notion of products which are *self contained* packages that > > > are sub-packages of the Products package. So, suppose someone wants > > > to write a NiftyDB product, which is a Zope product that provides > > > access to an external database. Now the author of the NiftyDB product > > > wants to use the mx package. The mx package is not a standard part of > > > Zope, or of Python, so they simpley include it in the NiftyDB product > > > directory. First, I am not all that opposed to having a notion of ".." available in the import statement. If we can write zope.dir1.mod1 which is a relative import going down, maybe we can write ../dir2/mod2 or something spelled differently. But I think there would still be problems. We would be relying on all package authors to use ".." or "__" within their package. But it is more natural to write zope.this.that or mx.this.that everywhere, and that is what people have in fact done. Probably there would be bugs and at least annoyance. And it seems unfortunate to need this feature. A proper package scheme should really isolate package internals from the outside world, and if this isn't happening we need a new scheme. I also don't think there is any escape from needing standard package names, at least the head name "zope" for example. Of course all this currently works if (in Jim's example) both "zope" and "mx" are top-level names. Then the ".." is not necessary. But, as Jim points out: >> But why shouldn't it be visible? >Because visibility has a cost. Making mx a top-level name >means that someone else can't make it a top-level name. >This is why packages are a good idea. The global shared nature of PYTHONPATH and its name space makes it difficult to guarantee that all required packages are going to be present in a complicated installation like zope. And if anyone else installs another Python package, it can easily break the first installation. One solution is for an application to establish its own PYTHONPATH which can not be altered. If this points to "zopedir" then the installer can freely install mx to the directory zopedir/mx and be confident that another mx installation is not damaged, nor used. The logical extension is to place each package into its own file using a scheme like Gordon McMillan is using. If the package contents is obtained by seeking from the END of the file, then multiple package files can be concattenated with cat package1 package2 zope mx >> bigpackage and a large installation like zope can be shipped with its own "bigpackage" library which is essentially a normal PYTHONPATH archive with everything above the head directory names thrown away. The Python library is included under "Lib" just as it is now. The bigpackage library implies its own PYTHONPATH of "Lib;." If the Python library files are placed in the root, the implied PYTHONPATH is ".". Jim Ahlstrom From jcw at equi4.com Mon Sep 13 23:55:43 1999 From: jcw at equi4.com (Jean-Claude Wippler) Date: Mon, 13 Sep 1999 23:55:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> <37DD4094.1BDCE45F@lemburg.com> Message-ID: <37DD72DF.D5904CC@equi4.com> Marc-Andre, > BTW, would you want to miss relative file name paths ? Well, that thought did cross my mind while posting that reply... I'm not sure. Yes, they are convenient, but "current working directory" is not always a pleasant concept (yet more global state - think of tree walks and the inconvenience of having to get, alter, work, restore it - when using chdir to implement this). I don't use .. as often as might be expected. I use "cd ~/something" a lot more. While that may look relative at another level, it really is not (as "echo ~" shows). It does illustrate how nearly equivalent the two approaches are. DOS/Windows has always had a broken relative path: the current drive. And the "hash/bang" headers of Unix seem to always use absolute paths. Some of this may seem to point to the need for relative paths. But I think it hides a more fundamental issue: you need to find out context. Once you do, relativeness no longer matters (obviously). A system which has a current directory and ".." is equivalent to one which has no such thing and passes a "starting directory" in the environment, say. I think a system with less global state is more modular -> preferable. Another angle: I have built many types of tree structures, still do. Less and less of those contain a "parent" link. Instead of storing a parent you can just as easily keep state while descending into children. - Unix file info does not maintain a parent directory, it's redundant. - Directories do, and file system mount points are messy because of it. - Afaik, it's considered bad style to use "../header.h" in C includes. - Upward pointers can introduce cycles (no not here, as has been said). - In C++, member objects rarely need pointers to the enclosing object. It is not for a module to know where it "is" in a hierarchy, it is for a parent to (sometimes) provide that reference when it uses the module. If you want want access to a module called mx, and there may be many of them - then you ask another module to locate it for you. One which is able to choose / decide for you. Proximity may be a good discriminator, but the decision of proximity was not taken by you - and you won't know what rule is best. So you ask an "importer" (which could well always be the standard Python mechanism). Given the task to write such as beast, I'd probably want to implement the following module search as default: - look for the module first in the parent (i.e. as sibling) - move one level up, look again, etc - all the way to the top That's just one way to do it - proximity is not *always* what you want. So if Zope chooses a different packaging style, let it override import. But please don't build ".." into your modules, it doesn't belong there. Sorry for all the handwaving generalities. The issues are the same IMO, whether relativeness is provided or context, and therefore the outcome of this discussion will never be conclusive - they both work. -- Jean-Claude From tim_one at email.msn.com Tue Sep 14 09:02:24 1999 From: tim_one at email.msn.com (Tim Peters) Date: Tue, 14 Sep 1999 03:02:24 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DD0564.7361B789@lemburg.com> Message-ID: <000001befe7f$191104c0$fe2d153f@tim> [MAL] > I agree that it's ugly to include the __ attribute in the module > namespace due to the possible circular reference (parent->module, > module->parent), but the patch I sent doesn't do this... or was > "ugly" referring to the two underscores looking strange ? > > Could you elaborate a bit on the reasons for dropping __ support ? There are two sections on why __ was dropped in http://www.python.org/doc/essays/packages.html They don't refer to circularity, but to "limited use", "poor readability" and "awkwardness". A deeper reason may be hiding in the essay's "most packages will have a relative shallow substructure": this is Guido , the man who invented two-level scoping, and class inheritance without a "super" hook back to the (anonymous) parent. For all Python's dynamicism, it very much favors shallow, static name hierarchies. I don't think it's coincidence that Python's own source code is in a two-level directory structure either! The only #include with a ".." is in grammar.h, and there it's in a comment : #include "bitset.h" /* Sigh... */ So if we cut to the core here, I'd bet Guido doesn't object so much to relative imports as to the idea that anyone would go off and create a package structure so fractally convoluted that relative imports are strongly more attractive than naming the target package in full. Or maybe Guido doesn't care about that at all. I do regardless. I know Python's restrictions can grate, but in all, and in my repeated experience, they force you to rethink complicated designs and refactor them into simpler schemes that fit what Python is best at spelling. Nesting packages 8 deep is clumsy now? Damn straight, and I'm thankful for that: the clumsier it is, the less gratuitous inherited complexity I'll have to deal with in my future lives <0.5 wink>. Things that came up in this thread that are worth fixing include: + Problems with persistent class references (incl. pickles). + Dealing with incompatible versions of packages. If someone wants to embed a copy of (say) mxDateTime in their own package, the only excuses are that they're afraid of overwriting the user's existing mxDateTime installation (if any), and/or of having some future installation of something else overwrite mxDateTime with an incompatible version. Those are bad problems, but package embedding is no solution. You have a much better approach to that already via the DateTime.__version__ string! "Something like that" needs to be formalized and imposed on all public packages. at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim PS: > Just think of what happens to Win9x if you constantly update the DLLs... As a matter of personal experience, it gets much stabler! The older DLLs get replaced by less-buggy newer ones, thanks to version numbers, rules, and installers that finally play by the rules. The mean time between crashes when I installed Win95 a few years ago was about an hour; now it's at least days and possibly weeks (don't know -- never leave the puter on that long). When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. From mal at lemburg.com Tue Sep 14 10:32:38 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 10:32:38 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> Message-ID: <37DE0825.10D2BC6F@lemburg.com> James C. Ahlstrom wrote: > > First, I am not all that opposed to having a notion of ".." > available in the import statement. If we can write zope.dir1.mod1 > which is a relative import going down, maybe we can write > ../dir2/mod2 or something spelled differently. But I think > there would still be problems. > > We would be relying on all package authors to use ".." > or "__" within their package. But it is more natural to write > zope.this.that or mx.this.that everywhere, and that is what > people have in fact done. Probably there would be bugs and > at least annoyance. Think of it as an opportunity: it makes intra-package imports very simple without hard-wiring paths to submodules into the package. In the end, having written a self-contained package is a "feature" of that package. Other authors can then say: ok, I can plugin your package if it's self-contained. -- I have a feeling that this discussion is loosing its grip on reality a bit. Jim and I are not argueing to change Python's face, internal structure or look&feel. The '__' thing doesn't harm anyone, doesn't break any functionality or code. It is even now already possible using import hooks and has been implemented before in ni.py using just that technology. All that we are talking about is reviving relative imports so that its usage is possible *without* adding yet another hook. This is not about the pros and cons of deeply nested packages and neither about advantages of shallow structures. Its only about adding semantics to the string '__' used in import statements. Nothing more. People are not forced to use it. Besides, if used it will be a package internal technique and not be visible to users of that package. Now why would anyone want to make life harder for package authors ? Give the people something to play with so that they have more fun at what their doing for the benefit of the community ! -- > And it seems unfortunate to need this feature. A proper > package scheme should really isolate package internals from > the outside world, and if this isn't happening we need a new scheme. > > I also don't think there is any escape from needing standard > package names, at least the head name "zope" for example. > > Of course all this currently works if (in Jim's example) both > "zope" and "mx" are top-level names. Then the ".." is not > necessary. But, as Jim points out: > > >> But why shouldn't it be visible? > > >Because visibility has a cost. Making mx a top-level name > >means that someone else can't make it a top-level name. > >This is why packages are a good idea. > > The global shared nature of PYTHONPATH and its name space makes > it difficult to guarantee that all required packages are going > to be present in a complicated installation like zope. And if > anyone else installs another Python package, it can easily break > the first installation. > > One solution is for an application to establish its own > PYTHONPATH which can not be altered. If this points to "zopedir" > then the installer can freely install mx to the directory > zopedir/mx and be confident that another mx installation > is not damaged, nor used. > > The logical extension is to place each package into its own > file using a scheme like Gordon McMillan is using. If the > package contents is obtained by seeking from the END of the > file, then multiple package files can be concattenated with > cat package1 package2 zope mx >> bigpackage > and a large installation like zope can be shipped with its > own "bigpackage" library which is essentially a normal > PYTHONPATH archive with everything above the head directory > names thrown away. The Python library is included under > "Lib" just as it is now. The bigpackage library implies > its own PYTHONPATH of "Lib;." If the Python library files are > placed in the root, the implied PYTHONPATH is ".". This is a possibility and in fact I'm using such a setup in my current application. Still it doesn't avoid possible conflicts due to being top-level, e.g. the user could install an extension which relies on a specific top-level name already taken by the hosting package, e.g. the host defines a DateTime package and the extension comes with its own DateTime package. This is the current situation with Zope and mxODBC (which needs mxDateTime). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Tue Sep 14 10:49:52 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 10:49:52 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD17E3.74FEB43F@lemburg.com> <37DD2507.8A7AD0D1@equi4.com> <37DD4094.1BDCE45F@lemburg.com> <37DD72DF.D5904CC@equi4.com> Message-ID: <37DE0C30.4B0521B@lemburg.com> Jean-Claude Wippler wrote: > > Marc-Andre, > > > BTW, would you want to miss relative file name paths ? > > Well, that thought did cross my mind while posting that reply... > > I'm not sure. Yes, they are convenient, but "current working directory" > is not always a pleasant concept (yet more global state - think of tree > walks and the inconvenience of having to get, alter, work, restore it - > when using chdir to implement this). I don't use .. as often as might > be expected. I use "cd ~/something" a lot more. While that may look > relative at another level, it really is not (as "echo ~" shows). It > does illustrate how nearly equivalent the two approaches are. > > DOS/Windows has always had a broken relative path: the current drive. > > And the "hash/bang" headers of Unix seem to always use absolute paths. ...and people have turned to /usr/bin/env to get around this necessity :-) > Some of this may seem to point to the need for relative paths. But I > think it hides a more fundamental issue: you need to find out context. > Once you do, relativeness no longer matters (obviously). A system which > has a current directory and ".." is equivalent to one which has no such > thing and passes a "starting directory" in the environment, say. > > I think a system with less global state is more modular -> preferable. Agreed. I would much prefer imputil.py to become a standard -- then I could add my hook into its chain and wouldn't have to argue for using relative imports ;-) But then, how would I tell if someone else already has integrated such a hook in the chain ? I sure wouldn't want to add another one just to make sure... If the standard mechanism already knows about '__' then I wouldn't have to worry. > Another angle: I have built many types of tree structures, still do. > Less and less of those contain a "parent" link. Instead of storing a > parent you can just as easily keep state while descending into children. > > - Unix file info does not maintain a parent directory, it's redundant. > - Directories do, and file system mount points are messy because of it. > - Afaik, it's considered bad style to use "../header.h" in C includes. > - Upward pointers can introduce cycles (no not here, as has been said). > - In C++, member objects rarely need pointers to the enclosing object. Note that my patch does exactly this: it resolves the __ while descending into the package structure -- without any explicit back reference stored in the modules. > It is not for a module to know where it "is" in a hierarchy, it is for a > parent to (sometimes) provide that reference when it uses the module. Right. > If you want want access to a module called mx, and there may be many of > them - then you ask another module to locate it for you. One which is > able to choose / decide for you. Proximity may be a good discriminator, > but the decision of proximity was not taken by you - and you won't know > what rule is best. So you ask an "importer" (which could well always be > the standard Python mechanism). Given the task to write such as beast, > I'd probably want to implement the following module search as default: > - look for the module first in the parent (i.e. as sibling) > - move one level up, look again, etc > - all the way to the top > That's just one way to do it - proximity is not *always* what you want. Now this is an interesting idea... looks like acquisition at the module import level. Right now the scheme is as follows: - look for the module in the import module's dir - look for the module on the PYTHONPATH You would just add a third lookup step in between the two... it would cause many more module lookups though. > So if Zope chooses a different packaging style, let it override import. > But please don't build ".." into your modules, it doesn't belong there. > > Sorry for all the handwaving generalities. The issues are the same IMO, > whether relativeness is provided or context, and therefore the outcome > of this discussion will never be conclusive - they both work. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Tue Sep 14 11:21:45 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 11:21:45 +0200 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <37DE13A9.4150AF0E@lemburg.com> Tim Peters wrote: > > [MAL] > > I agree that it's ugly to include the __ attribute in the module > > namespace due to the possible circular reference (parent->module, > > module->parent), but the patch I sent doesn't do this... or was > > "ugly" referring to the two underscores looking strange ? > > > > Could you elaborate a bit on the reasons for dropping __ support ? > > There are two sections on why __ was dropped in > > http://www.python.org/doc/essays/packages.html > > They don't refer to circularity, but to "limited use", "poor readability" > and "awkwardness". Ok, so the two underscores look strange... that's arguable. "Limited use" is not: there clearly is a use for relative imports. Please remember that we are talking about package internals here -- not user interfaces of that package. > A deeper reason may be hiding in the essay's "most > packages will have a relative shallow substructure": this is Guido , > the man who invented two-level scoping, and class inheritance without a > "super" hook back to the (anonymous) parent. For all Python's dynamicism, > it very much favors shallow, static name hierarchies. I don't think it's > coincidence that Python's own source code is in a two-level directory > structure either! The only #include with a ".." is in grammar.h, and there > it's in a comment : > > #include "bitset.h" /* Sigh... */ > > So if we cut to the core here, I'd bet Guido doesn't object so much to > relative imports as to the idea that anyone would go off and create a > package structure so fractally convoluted that relative imports are strongly > more attractive than naming the target package in full. This is not the intention of relative imports. Its all about making packages relocatable. And that's a Good Thing IMHO. > Or maybe Guido doesn't care about that at all. I do regardless. I know > Python's restrictions can grate, but in all, and in my repeated experience, > they force you to rethink complicated designs and refactor them into simpler > schemes that fit what Python is best at spelling. Nesting packages 8 deep > is clumsy now? Damn straight, and I'm thankful for that: the clumsier it > is, the less gratuitous inherited complexity I'll have to deal with in my > future lives <0.5 wink>. Tim, I promise you not use 8 levels deep ;-) I usually create my packages using at most two levels (moving to at most three levels in future mx package releases) and even better: the common package interfaces are all available at the top-most level while internal stuff hides in higher levels. You don't have to think much about the internal structure of the package while you use it. That's something I have to worry about and the advantages of relative imports show up in such a context which is why I posted the patch. > Things that came up in this thread that are worth fixing include: > > + Problems with persistent class references (incl. pickles). > > + Dealing with incompatible versions of packages. If someone wants to embed > a copy of (say) mxDateTime in their own package, the only excuses are that > they're afraid of overwriting the user's existing mxDateTime installation > (if any), and/or of having some future installation of something else > overwrite mxDateTime with an incompatible version. Those are bad problems, > but package embedding is no solution. You have a much better approach to > that already via the DateTime.__version__ string! "Something like that" > needs to be formalized and imposed on all public packages. So the distutils will have to check for already installed package versions and warn about conflicts. Right, that's something which definitely needs to become a standard (too ;). > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim > > PS: > > > Just think of what happens to Win9x if you constantly update the DLLs... > > As a matter of personal experience, it gets much stabler! The older DLLs > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > installers that finally play by the rules. The mean time between crashes > when I installed Win95 a few years ago was about an hour; now it's at least > days and possibly weeks (don't know -- never leave the puter on that long). > When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. My experience is that intra-DLL references simply don't match anymore and cause the system to become instable. Also, some weird installers don't care about the version numbers and install older versions at their will. The outcome is a complete version mess. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Tue Sep 14 14:21:26 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Tue, 14 Sep 1999 08:21:26 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE0C30.4B0521B@lemburg.com> Message-ID: <1274810963-24736486@hypernet.com> A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 892 bytes Desc: not available URL: From jim at digicool.com Tue Sep 14 14:20:53 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 12:20:53 +0000 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <37DE3DA4.3C7C14A8@digicool.com> Tim Peters wrote: > > [MAL] > > I agree that it's ugly to include the __ attribute in the module > > namespace due to the possible circular reference (parent->module, > > module->parent), but the patch I sent doesn't do this... or was > > "ugly" referring to the two underscores looking strange ? > > > > Could you elaborate a bit on the reasons for dropping __ support ? > > There are two sections on why __ was dropped in > > http://www.python.org/doc/essays/packages.html > > They don't refer to circularity, but to "limited use", "poor readability" > and "awkwardness". Which are all quite subjective. > A deeper reason may be hiding in the essay's "most > packages will have a relative shallow substructure": this is Guido , > the man who invented two-level scoping, and class inheritance without a > "super" hook back to the (anonymous) parent. For all Python's dynamicism, > it very much favors shallow, static name hierarchies. I don't think it's > coincidence that Python's own source code is in a two-level directory > structure either! The only #include with a ".." is in grammar.h, and there > it's in a comment : > > #include "bitset.h" /* Sigh... */ > > So if we cut to the core here, I'd bet Guido doesn't object so much to > relative imports as to the idea that anyone would go off and create a > package structure so fractally convoluted that relative imports are strongly > more attractive than naming the target package in full. > > Or maybe Guido doesn't care about that at all. I do regardless. I know > Python's restrictions can grate, but in all, and in my repeated experience, > they force you to rethink complicated designs and refactor them into simpler > schemes that fit what Python is best at spelling. Nesting packages 8 deep > is clumsy now? Damn straight, and I'm thankful for that: the clumsier it > is, the less gratuitous inherited complexity I'll have to deal with in my > future lives <0.5 wink>. Nobody has suggested building 8-level package hierarchies. In fact, the example I gave occured when a two-level package hierarchy was used in a two-level hierarchy. I don't think the Package structure of Zope is flawed *except* for the fact that it is one level too *shallow*. The ability to do relative imports would be very helpful for the work we're doing. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at interet.com Tue Sep 14 15:30:38 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 09:30:38 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> Message-ID: <37DE4DFE.FFD199ED@interet.com> "M.-A. Lemburg" wrote: > > Still it doesn't avoid possible conflicts > due to being top-level, e.g. the user could install an extension > which relies on a specific top-level name already taken by the hosting > package, e.g. the host defines a DateTime package and the > extension comes with its own DateTime package. This is the current > situation with Zope and mxODBC (which needs mxDateTime). Yes, the name conflict at the global level is exactly the problem. And to repeat, I don't really mind if "__" is added to imports, although I don't like the spelling. But... Perhaps a better solution is a multiple global name space. Suppose there is a command line option (or other mechanism such as a special file in sys.executable) which specifies sys.path for the application. To be specific, suppose zope is installed in the "zopehome" directory and the zope package lives in zopehome/zope. And suppose zope needs mx, so it installs it in zopehome/mx. This really is an obvious structure, and anyone could see that replacing mx/* would upgrade the mx package. The sys.path would be zopehome plus perhaps some subdirectories of zopehome. Doesn't this solve the problem? Why not? And don't complain about wasted disk space. The smallest disk you can buy today is 4 gigs and costs $150. I find the PYTHONPATH mechanism totally unreliable for commercial programs anyway. It is a global object, and an installation of a second Python program can break the first one. I don't think there is any solution to this other than specify sys.path on a per-application basis. If this is false, what is the other solution? Jim Ahlstrom From jim at digicool.com Tue Sep 14 14:42:10 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 12:42:10 +0000 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> Message-ID: <37DE42A2.E27F7E55@digicool.com> "M.-A. Lemburg" wrote: > > James C. Ahlstrom wrote: > > > > First, I am not all that opposed to having a notion of ".." > > available in the import statement. If we can write zope.dir1.mod1 > > which is a relative import going down, maybe we can write > > ../dir2/mod2 or something spelled differently. But I think > > there would still be problems. > > > > We would be relying on all package authors to use ".." > > or "__" within their package. But it is more natural to write > > zope.this.that or mx.this.that everywhere, and that is what > > people have in fact done. Probably there would be bugs and > > at least annoyance. > > Think of it as an opportunity: it makes intra-package imports > very simple without hard-wiring paths to submodules > into the package. > > In the end, having written a self-contained package is a "feature" > of that package. Other authors can then say: ok, I can > plugin your package if it's self-contained. Well said. It really should be possible to provide *self-contained* package hierarchies. > I have a feeling that this discussion is loosing its grip > on reality a bit. Jim and I are not argueing to change > Python's face, internal structure or look&feel. The '__' thing > doesn't harm anyone, doesn't break any functionality or > code. It is even now already possible using import hooks and > has been implemented before in ni.py using just that technology. > > All that we are talking about is reviving relative imports > so that its usage is possible *without* adding yet another > hook. This is not about the pros and cons of deeply nested > packages and neither about advantages of shallow structures. > Its only about adding semantics to the string '__' used > in import statements. Nothing more. People are not forced > to use it. Besides, if used it will be a package internal > technique and not be visible to users of that package. > > Now why would anyone want to make life harder for package > authors ? I don't think nearly enough Python code has been packagized. Packages are critcal for large projects and for reuse of code within large projects. There are at least two people in this discussion that are trying hard to make use of packages and have observed a significant problem. If we really want to encourage developers of reusable Python software to use packages, then I think it would be wise to pay attention to people who are actively using packages and have encountered problems. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at interet.com Tue Sep 14 15:46:23 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 09:46:23 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE42A2.E27F7E55@digicool.com> Message-ID: <37DE51AF.67BB9CC0@interet.com> Jim Fulton wrote: > > "M.-A. Lemburg" wrote: > > In the end, having written a self-contained package is a "feature" > > of that package. Other authors can then say: ok, I can > > plugin your package if it's self-contained. > > Well said. It really should be possible to provide > *self-contained* package hierarchies. > There are at > least two people in this discussion that are trying hard to > make use of packages and have observed a significant problem. OK, just to make sure I understand this. You are saying that a ".." operation is needed in imports so that a package author with a package which looks like this: mypackage mypackage/sub1/mod1.py mypackage/sub2/mod2.py can write "import __.sub2.mod2" instead of "import mypackage.sub2.mod2" within the file mod1.py. Right? Jim Ahlstrom From jim at digicool.com Tue Sep 14 15:24:10 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 13:24:10 +0000 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE42A2.E27F7E55@digicool.com> <37DE51AF.67BB9CC0@interet.com> Message-ID: <37DE4C7A.A553A8F2@digicool.com> "James C. Ahlstrom" wrote: > > Jim Fulton wrote: > > > > "M.-A. Lemburg" wrote: > > > In the end, having written a self-contained package is a "feature" > > > of that package. Other authors can then say: ok, I can > > > plugin your package if it's self-contained. > > > > Well said. It really should be possible to provide > > *self-contained* package hierarchies. > > > There are at > > least two people in this discussion that are trying hard to > > make use of packages and have observed a significant problem. > > OK, just to make sure I understand this. You are saying that > a ".." operation is needed in imports so that a package author > with a package which looks like this: > mypackage > mypackage/sub1/mod1.py > mypackage/sub2/mod2.py > > can write "import __.sub2.mod2" instead of "import mypackage.sub2.mod2" > within the file mod1.py. Right? Yup. Note that sys.modules would still have the absolute paths for keys. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward at cnri.reston.va.us Tue Sep 14 16:17:01 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Tue, 14 Sep 1999 10:17:01 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <000001befe7f$191104c0$fe2d153f@tim>; from Tim Peters on Tue, Sep 14, 1999 at 03:02:24AM -0400 References: <37DD0564.7361B789@lemburg.com> <000001befe7f$191104c0$fe2d153f@tim> Message-ID: <19990914101700.A4401@cnri.reston.va.us> On 14 September 1999, Tim Peters said: > + Dealing with incompatible versions of packages. If someone wants to embed > a copy of (say) mxDateTime in their own package, the only excuses are that > they're afraid of overwriting the user's existing mxDateTime installation > (if any), and/or of having some future installation of something else > overwrite mxDateTime with an incompatible version. Those are bad problems, > but package embedding is no solution. You have a much better approach to > that already via the DateTime.__version__ string! "Something like that" > needs to be formalized and imposed on all public packages. > > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim Been there, tried that, bought the flame war. I made the mistake of kicking off the Distutils SIG back in Decemver with a proposal for a standard version numbering scheme for Python module distributions. See http://www.python.org/pipermail/distutils-sig/1998-December/000016.html for the kick-off of that "heated discussion". ;-) FWIW, if I was posting that message today, I would s/must/should/ and that's about it. And I would stress that these are "Proposed Recommended Guidelines" rather than "Things You Must Do or Distutils Will Slam the Door in Your Face", which is what it must have sounded like based on some of the responses. Folks: please don't reheat this flamewar on python-dev -- if you're interested in it, it will undoubtedly come up again on distutils-sig in due course! Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal at lemburg.com Tue Sep 14 16:24:06 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 16:24:06 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> Message-ID: <37DE5A86.4CD04049@lemburg.com> James C. Ahlstrom wrote: > > "M.-A. Lemburg" wrote: > > > > Still it doesn't avoid possible conflicts > > due to being top-level, e.g. the user could install an extension > > which relies on a specific top-level name already taken by the hosting > > package, e.g. the host defines a DateTime package and the > > extension comes with its own DateTime package. This is the current > > situation with Zope and mxODBC (which needs mxDateTime). > > Yes, the name conflict at the global level is exactly the problem. > And to repeat, I don't really mind if "__" is added to imports, > although I don't like the spelling. But... Doesn't look nice, but what other syntax would look better ? It will have to use identifiers and thus is restricted to [a-zA-Z0-9_]+. > Perhaps a better solution is a multiple global name space. Suppose > there is a command line option (or other mechanism such as a special > file in sys.executable) which specifies sys.path for the application. > To be specific, suppose zope is installed in the "zopehome" > directory and the zope package lives in zopehome/zope. And suppose > zope needs mx, so it installs it in zopehome/mx. This really is an > obvious structure, and anyone could see that replacing mx/* would > upgrade the mx package. The sys.path would be zopehome plus perhaps > some subdirectories of zopehome. Doesn't this solve the problem? > Why not? And don't complain about wasted disk space. The smallest > disk you can buy today is 4 gigs and costs $150. This would solve the problem for Zope (but only after I have restructured my packages to all go under the mx package -- which is what started this thread; it wouldn't be needed using relative imports). But what about other installations or tools like mxCrypto which plug into existing packages (Andrew's crypto package) ? Today such plugins only work side-by-side, ie. you have to install one part which sits in the plugin slot and one part which gets installed somewhere in a top-level PYTHONPATH dir. The plugin slot part then imports the Real Thing from outside the host package (this is how mxCrypto integrates itself with AMK's lib). > I find the PYTHONPATH mechanism totally unreliable for commercial > programs anyway. It is a global object, and an installation of > a second Python program can break the first one. I don't think > there is any solution to this other than specify sys.path on a > per-application basis. If this is false, what is the other > solution? The solution I'm using (and Zope is too) is to wrap a complete Python installation into the product, all setup to do the right thing. It makes the archive a little heavier... which is actually not so bad from a marketing POV. Bizarre, but people seem to want all those megs. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at digicool.com Tue Sep 14 16:13:13 1999 From: jim at digicool.com (Jim Fulton) Date: Tue, 14 Sep 1999 14:13:13 +0000 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> <37DE13A9.4150AF0E@lemburg.com> Message-ID: <37DE57F9.9B3BFDC4@digicool.com> "M.-A. Lemburg" wrote: > > Tim Peters wrote: (snip) > > As a matter of personal experience, it gets much stabler! The older DLLs > > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > > installers that finally play by the rules. The mean time between crashes > > when I installed Win95 a few years ago was about an hour; now it's at least > > days and possibly weeks (don't know -- never leave the puter on that long). > > When a version upgrade fails, it's not a mystery, it's a bug <0.9 wink>. > > My experience is that intra-DLL references simply don't match anymore > and cause the system to become instable. Also, some weird installers > don't care about the version numbers and install older versions at > their will. The outcome is a complete version mess. This has been my experince too. I cringe anytime I see some installer stuff DLLs in my system areas. Jim -- Jim Fulton mailto:jim at digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From skip at mojam.com Tue Sep 14 17:34:48 1999 From: skip at mojam.com (Skip Montanaro) Date: Tue, 14 Sep 1999 10:34:48 -0500 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda Message-ID: <199909141534.KAA03648@dolphin.mojam.com> I haven't been paying real close attention to the relative package import thread on python-dev, but some way or other I strongly believe something has to be done to improve the situation I now find myself in or more and more people are going to start getting bitten by the same sort of problem. Allow me to 'splain. I use a web server based on Zope's ZServer (which is itself based on medusa) sitting behind an Apache web server as a long-running process to handle what many people traditionally used CGI for. It's a big performance boost over CGI. Unfortunately, I have recently been experiencing frequent server hangs. So far I've been unable to figure out what the cause is. I do notice occasional tracebacks like: Unhandled exception in thread: Traceback (innermost last): File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send _push_mode = 0 File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer self.ac_in_buffer = '' File "/usr/lib/python1.5/asynchat.py", line 253, in pop # this could maybe be made faster with a computed regex? IndexError: list index out of range and also notice that the server can pile up a huge number of connections in the ESTABLISHED state, at which point the whole mess grinds to a halt with not much computation or network traffic happening. A separate shell script uses netstat to detect when a large number of sockets have piled up and restarts the server. Brutal, but crudely effective. (When in doubt, treat the symptoms...) Today it just dawned on me looking at the above traceback that ZServer is getting the wrong version of asynchat.py (and presumably of asyncore.py as well). It's using the version delivered with Python 1.5.2 distribution instead of the version that was delivered with the version of Zope I'm using (1.something). So I started looking around at the versions and dates of various copies of asynchat.py. Here's what I found: source version number:date owner Python 1.5.2 1.2:1999/06/18 guido Python CVS 1.2:1999/06/18 guido Zope 1.? 1.7:1999/04/09 amos Zope 2.0 1.9:1999/07/19 amos Medusa 990902 2.24:1999/07/07 rushing What's apparently been happening is that people have picked up asyncore and asynchat at various time and stuck them in their own CVS repositories without somehow freezing the Id string of the version they originally got from Sam Rushing. It's not clear what the differences are until you compare the actual files. It turns out that the Zope 2.0 and Medusa versions have no content differences, only wildly different version numbers. The Medusa and Python CVS versions only have one difference: if index > 0: # don't bother reporting the empty string (source of subtle bugs) self.collect_incoming_data (self.ac_in_buffer[:index]) which *may* be what's causing my problems (note the IndexError in my traceback). Somewhere along the way I think we need to apply more restraint to our packaging and numbering of Python modules. I don't know what form that restraint will eventually take, but at this point I'm willing to replace Greg Ward's s/must/should/ with s/should/must/. a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From bwarsaw at cnri.reston.va.us Tue Sep 14 18:19:34 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Tue, 14 Sep 1999 12:19:34 -0400 (EDT) Subject: Path hacking [Long] (was Re: [Python-Dev] Relative Package Imports) References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> Message-ID: <14302.30102.360791.488002@anthem.cnri.reston.va.us> Finally, something I can relate to. Although I have a goal of packagizing everything I write these days, I haven't experienced any of the problems that lead others to suggest relative imports. The most complicated app that I hack on (continuously) is Mailman, which has a main package and several subpackages off the main one. I always use absolute paths in my import statements, so I don't see what the fuss is about. But I'm perfectly willing to admit that I don't have enough experience. However... >>>>> "JCA" == James C Ahlstrom writes: JCA> I find the PYTHONPATH mechanism totally unreliable for JCA> commercial programs anyway. It is a global object, and an JCA> installation of a second Python program can break the first JCA> one. I don't think there is any solution to this other than JCA> specify sys.path on a per-application basis. If this is JCA> false, what is the other solution? I completely agree with JimA here. It's been a pain with the Knowbot stuff, a pain with Mailman, and a pain with other packages that I've installed for shared use within CNRI. The .pth files solve part of the problem nicely. They let me install, say PIL or PCT in a shared location, for access by all the Python users at my site, without the users having to individually hack their dot-files, etc. But this doesn't work so well for apps like Mailman or the Knowbot stuff because we can't expect that the person installing those applications will be able to install a .pth file in the right place. Also, .pth files don't let you tightly control sys.path, e.g. you can only add paths, not delete or reorder them. Plus you have a global naming problem. Mailman's top level package is called "Mailman", so I can be fairly confident that I'm not going to collide, but it means that I have an extra directory level within my install that contains all the core importable modules. I don't think that's a big deal, but it's a convention that other packaged app writers should follow. The problem is getting Mailman's (or the Knowbots') top level directory on sys.path, and in exactly controlling the contents of sys.path. Our first approach with Knowbots was to do direct sys.path.insert()s, which is quite ugly and error prone. Plus if you're adding many paths, or adding and deleting, that's a lot of gibberish at the top of your entry level executables. And now let's say that you have a dozen or two dozen entry level executables that all have to perform the same sys.path magic. That's a lot of cutting-and-pasting (and /highly/ error prone patching when directory structures change). It's a lose. So for Knowbots we wrote a small module called pathhack that all entry level executables imported. pathhack was good because it put all that sys.path munging nonsense in one place so it was manageable from a s/w engineering standpoint. But it sucked because those executables had to /find/ pathhack.py! Bootstrap lossage (we've actually gone back to sys.path.insert). With Mailman, I could solve that problem because I added a configure/make phase. This let me write a module template called paths.py.in which configure flippered into paths.py containing path hackage based on --prefix. The next trick was that "make install" copied that paths.py file into all the subdirectories that had top level entry points into the Mailman system (e.g. the bin directory, the cron directory, the cgi directory). So now, an executable need only do import paths import Mailman.Utils import Mailman.Logging.Utils and absolute paths work like a charm. I can even provide a `pythonlib' directory that contains newer versions of standard modules that have fixes for folks running older Pythons. Thus I do from Mailman.pythonlib import rfc822 and the rest of my code uses my special rfc822 module with no changes. I'm very happy with how this works for Mailman, however we can't use the same approach (or let's say Guido doesn't want to use this approach) for the Knowbots stuff because there /is/ no "make install" step. You just unpack it and go. But it still has to play lots of games searching the file system for various things. What I've been thinking is that Python needs a registry . JPython's already got such a beast, and it integrates with Java's system properties, so that things like the PYTHONPATH equivalent are set in the registry and immediately available. But it's not very flexible, and you still need an install step in order to bootstrap the locating of the registry. I think we can do a little bit better. Python already knows how to find it's sys module. We can add an object into sys, call it sys.registry, which would contain things like sys.path definitions, and all sorts of other application specific keys. This object would be tied to a file (or files) which might be human readable, a marshal/pickle (or both). Bootstrap location of this file(s) is an issue, but see below. This would let you do things like the following at the beginning of every top level executable: import sys sys.application = 'zope' sys.registry.setpath(sys.application+'.pythonpath') I'm sure all kinds of lengthy discussion will now ensue about the exact interface of the registry object, but I'll make just a few observations: - There should be a system wide registry and a user specific registry. This let's an admin install shared applications easily, but also lets individual users have their own overrides. - The system-wide registry can be located in say sys.prefix/lib/python/site-packages. The user registry would reside somewhere in $HOME. This could all be platform specific so that on Windows, maybe the Python registry is integrated with the Windows registry, while in JPython it would be integrated with the standard JPython registry mechanism. - You should be able to specify registry entries on the command line. - There needs to be defined rules for resolving registry keys b/w system, user, and command line specifications. JPython has some experience here (although there have been requests to change JPython's lookup order), and at the very least, JPython and CPython should be as consistent as possible (CPython won't have to merge in Java's system properties). - The sys.registry object should be read/writable. This would let an install script do something like: import sys sys.registry.lock() sys.registry.put('zope.pythonpath', '@prefix@:@prefix@/matools:@prefix@/pythonlib') sys.registry.write() sys.registry.unlock() which would write either the global system registry or the local user registry, depending on permissions (or maybe that's spelled explicitly in the API). - In a sense you're pushing the namespace issue up a level into the registry, but at least this is a domain we can completely control from Python; it abstracts away the file system, and I don't think there's any way to avoid requiring conventions and cooperation for registry key naming. I also don't think it'll be a big problem in practice. When I packagize and re-release my Zarathustra's Ocular Python Experience virtual reality system, I'll try to think of a non-colliding top level package name. - (oh darn, I know I had more points, but Guido just popped in and I lost my train of thought). Well, this has gone on long enough so I might as well let you guys shoot this idea all to hell. Let me close by saying that while I think the Windows registry is a mess, I also think that it might be useful for Python. Does it solve the same problem that the relative imports is trying to solve? I dunno, but that's why I changed the Subject: line above. :) -Barry From gmcm at hypernet.com Tue Sep 14 18:47:25 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Tue, 14 Sep 1999 12:47:25 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE5A86.4CD04049@lemburg.com> Message-ID: <1274795004-25696413@hypernet.com> [MA Lemburg] > The solution I'm using (and Zope is too) is to wrap a > complete Python installation into the product, all setup to > do the right thing. It makes the archive a little heavier... > which is actually not so bad from a marketing POV. Bizarre, but > people seem to want all those megs. This is fine if you're distributing an app - it's a black box, it's standalone, no dependencies outside the app. Cool. I do that too. I use imputil to serve modules out of archives - no need to tweak the modules at all. In the upcoming version (of my installer) sys.path has exactly one entry, and the only python that needs to be outside an archive is exceptions.pyc. (I repeat here: while the pieces of my installer that handle binary dependencies are Win32 only, the parts that handle pure Python are cross-platform.) But both you and Jim are aiming for this non-existant in- between space - in some respects you want a "normal" Python installation, but you want to black-box your turf. I'm sorry, but if you're going to fit into a Python installation, you should make public your dependencies. Yes, installation becomes more complicated, but you're dealing with *Python users*. They're at least as likely to get screwed by having multiple different copies of the same thing around as you are in having dependencies outside of your personal control. If you also want to distribute as a standalone, freeze/squeeze/ whatever it. Just make sure that what you distribute this way won't interfere (or be confounded by) any existing Python installations. - Gordon From gstein at lyra.org Tue Sep 14 19:23:36 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 10:23:36 -0700 (PDT) Subject: [Python-Dev] revised import code (was: Relative Package Imports) In-Reply-To: <37DCCF3B.2F1E962D@lemburg.com> Message-ID: On Mon, 13 Sep 1999, M.-A. Lemburg wrote: > Gordon McMillan wrote: > > But I strongly believe that import.c should be left alone, maybe > > even to die. There are too many people doing import hooks to > > make fiddling with its behavior safe. > > > > I'm also a strong proponent of Greg's imputil.py scheme, which > > makes it a breeze to do import hooks. And my experience > > disproves the notion that the import mechanism needs to be in C. If > > you don't believe me, try the ZlibArchive stuff (which is cross > > platform) from my Win32 installer stuff. You can pack the standard > > library into one 475K file, and get a perceptible performance boost. > > You're probably right in saying that we don't need the code in C. > I just wanted to avoid yet another import hook being incompatible > with all the other existing hooks. > > Perhaps we should restart the import discussion all over and > come up with a more flexbile 100% compatible framework based > on Greg's imputil scheme. Then I could add my hook for the relative > imports and be happy ;-) I'll code up some patches to strip down import.c and replace with imputil.py and some bootstrap/support C code -- as soon as 1.6 development opens up. Guido? Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Tue Sep 14 19:36:03 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:36:03 -0400 Subject: [Python-Dev] Relative Package Imports References: <000001befe7f$191104c0$fe2d153f@tim> <37DE13A9.4150AF0E@lemburg.com> <37DE57F9.9B3BFDC4@digicool.com> Message-ID: <37DE8783.2B8D6455@interet.com> Jim Fulton wrote: > > "M.-A. Lemburg" wrote: > > > > Tim Peters wrote: > (snip) > > > As a matter of personal experience, it gets much stabler! The older DLLs > > > get replaced by less-buggy newer ones, thanks to version numbers, rules, and > > > > My experience is that intra-DLL references simply don't match anymore > > and cause the system to become instable. Also, some weird installers > > don't care about the version numbers and install older versions at > > their will. The outcome is a complete version mess. > > This has been my experince too. I cringe anytime I see some > installer stuff DLLs in my system areas. My experience has been the same as Tim's. You should generally upgrade DLL's as a matter of course, because your system will be more stable. Installers are required to check DLL versions before replacing them. Not doing this is a major sin, and reason enough to stop dealing with a vendor. Jim Ahlstrom From jim at interet.com Tue Sep 14 19:49:41 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:49:41 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <37DE5A86.4CD04049@lemburg.com> Message-ID: <37DE8AB5.DDB8C432@interet.com> "M.-A. Lemburg" wrote: > Doesn't look nice, but what other syntax would look better ? It will > have to use identifiers and thus is restricted to [a-zA-Z0-9_]+. Maybe "__up__" or "__up" following the convention that double-under names are special? Just a thought... > But what about other installations or tools like mxCrypto > which plug into existing packages (Andrew's crypto package) ? Today > such plugins only work side-by-side, ie. you have to install one > part which sits in the plugin slot and one part which gets installed > somewhere in a top-level PYTHONPATH dir. The plugin slot part then > imports the Real Thing from outside the host package (this is how > mxCrypto integrates itself with AMK's lib). I am not familiar with this. Dosen't this work?: zopehome/part_in_top_level_slot.py zopemome/plugin_slot/part_in_plugin_slot.py > The solution I'm using (and Zope is too) is to wrap a > complete Python installation into the product, all setup to > do the right thing. It makes the archive a little heavier... > which is actually not so bad from a marketing POV. Bizarre, > but people seem to want all those megs. For the record, I distribute a hacked main program which sets its own sys.path. I think your and zope's solution may break if another Python program is installed, and such program changes PYTHONPATH and/or the registry. That is why I never change PYTHONPATH nor the registry. Jim Ahlstrom From jim at interet.com Tue Sep 14 19:59:03 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 13:59:03 -0400 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <37DE8CE7.F211F6FA@interet.com> Skip Montanaro wrote: > So I started looking around at the versions and dates of various copies of > asynchat.py. >... > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have Yes, I have had this happen too. I am unwilling and unable to risk this sort of problem at a customer's site. So I ship a complete app with no external dependencies. Also crude but effective. Jim Ahlstrom From gstein at lyra.org Tue Sep 14 20:55:47 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 11:55:47 -0700 (PDT) Subject: [Python-Dev] relative package imports, version , numbering, yadda yadda In-Reply-To: <37DE8CE7.F211F6FA@interet.com> Message-ID: On Tue, 14 Sep 1999, James C. Ahlstrom wrote: > Skip Montanaro wrote: > > So I started looking around at the versions and dates of various copies of > > asynchat.py. > >... > > What's apparently been happening is that people have picked up asyncore and > > asynchat at various time and stuck them in their own CVS repositories > > without somehow freezing the Id string of the version they originally got > > from Sam Rushing. It's not clear what the differences are until you compare > > the actual files. It turns out that the Zope 2.0 and Medusa versions have > > Yes, I have had this happen too. I am unwilling and unable to risk > this sort of problem at a customer's site. So I ship a complete > app with no external dependencies. Also crude but effective. We did the same with the Python-based apps/servers at Microsoft. Merchant Server was a big frozen app (based on the non-sensical requirement to hide the fact that Python was used). In Site Server 2.0 and 3.0, we used a mini-install -- just the Lib files we needed plus our extensions. In the Site Server (non-frozen) case, we did use the registry, but built the interpreter with a custom "version". SS20 and SS30. In the registry, this meant we used Python/PythonCore/SS20 (I think that's the layout). Worked great, no complaints. Oh, and the DLLs we put into the system directory (pretty necessary for COM) were named SS20.DLL to prevent conflicts. I'm not sure who said it, but I agree with the following statement: * ship your app as a complete black box, or ship your app with dependencies on modules/packages [at the top level] This monkeying around with "mx.foo" working where mx is at the top level or is embedded is just scary. That said, personally, I would just do something like the following at the startup of my app: ZopeImporter("zopedir").install() [where ZopeImporter is a imputil.Importer subclass] The importer would just Do The Right Thing for all imports, and only defer to the Python library for things that weren't shipped with Zope (the empty set?) Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Tue Sep 14 21:11:54 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 15:11:54 -0400 Subject: Path hacking [Long] (was Re: [Python-Dev] Relative Package Imports) References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <14302.30102.360791.488002@anthem.cnri.reston.va.us> Message-ID: <37DE9DFA.269967B5@interet.com> "Barry A. Warsaw" wrote: > > So for Knowbots we wrote a small module called pathhack that all entry > level executables imported. pathhack was good because it put all that > sys.path munging nonsense in one place so it was manageable from a s/w > engineering standpoint. But it sucked because those executables had > to /find/ pathhack.py! Bootstrap lossage (we've actually gone back to > sys.path.insert). Yes, exactly the problem I had, bootstraping the import of pathhack. Actually it gets worse because Python imports exceptions.py, site.py and sitecustomize.py during Py_Initialize(), so if you are having a really bad day, you might pick up the wrong version of those. AFAIK, the only way to solve that currently is to use freeze to build pathhack into the binary executable. That is what I do anyway. But it is not an ideal solution. > What I've been thinking is that Python needs a registry . Yikes! As you say, Window's registry is a mess. > [Lots of good ideas omitted...] > - The system-wide registry can be located in say > sys.prefix/lib/python/site-packages. The user registry > would reside somewhere in $HOME. This could all be platform > specific so that on Windows, maybe the Python registry is integrated > with the Windows registry, while in JPython it would be integrated > with the standard JPython registry mechanism. Python already has three directories it knows about: sys.executable is the directory of the interpreter binary, sys.dllfullpath could be the directory of the interpreter as a shared library (I have a patch for this), and there is the directory of the main Python program as given on the command line. Perhaps we can put the registry in one of these directories. That would be consistent on all platforms. > - You should be able to specify registry entries on the command line. This is vital because I am worried about a bad registry. > - There needs to be defined rules for resolving registry keys b/w > system, user, and command line specifications. JPython has some > experience here (although there have been requests to change I am not sure a full registry is required. Once you can control sys.path and can get an accurate import of sitecustomize.py, you can do everything else there. Maybe just a command line option is enough. But I will think about it... Jim Ahlstrom From jim at interet.com Tue Sep 14 21:37:11 1999 From: jim at interet.com (James C. Ahlstrom) Date: Tue, 14 Sep 1999 15:37:11 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274795004-25696413@hypernet.com> Message-ID: <37DEA3E7.389CE@interet.com> Gordon McMillan wrote: > > But both you and Jim are aiming for this non-existant in- > between space - in some respects you want a "normal" > Python installation, but you want to black-box your turf. I don't really have much choice here. For some purposes, PYTHONPATH is better than a black-box. It is better for development because it avoids creating the black-box. And I can look at and modify Python app and library files directly. But for a customer install I need a bullet-proof dumb-simple set of required state which I can explain to customer support staff. Thus the need for both. > I'm sorry, but if you're going to fit into a Python installation, > you should make public your dependencies. Yes, installation > becomes more complicated, but you're dealing with *Python > users*. They're at least as likely to get screwed by having > multiple different copies of the same thing around as you are > in having dependencies outside of your personal control. I can't fit into a Python installation because Python installations do not support commercial software concerns. I am not dealing with Python users, my customers are office workers. They don't know what Python is, have no other Python apps, and if they did, my install wouldn't hurt them anyway. > If you also want to distribute as a standalone, freeze/squeeze/ > whatever it. Just make sure that what you distribute this > way won't interfere (or be confounded by) any existing Python > installations. Yes, exactly my goal. Jim Ahlstrom From guido at CNRI.Reston.VA.US Tue Sep 14 21:57:51 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 15:57:51 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Tue, 14 Sep 1999 12:19:34 EDT." <14302.30102.360791.488002@anthem.cnri.reston.va.us> References: <1274888077-20098343@hypernet.com> <37DD1533.24A05F69@digicool.com> <37DD6429.E4F6BF92@interet.com> <37DE0825.10D2BC6F@lemburg.com> <37DE4DFE.FFD199ED@interet.com> <14302.30102.360791.488002@anthem.cnri.reston.va.us> Message-ID: <199909141957.PAA15470@eric.cnri.reston.va.us> I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at CNRI.Reston.VA.US Tue Sep 14 22:09:07 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 16:09:07 -0400 Subject: [Python-Dev] relative package imports, version numbering, yadda yadda In-Reply-To: Your message of "Tue, 14 Sep 1999 10:34:48 CDT." <199909141534.KAA03648@dolphin.mojam.com> References: <199909141534.KAA03648@dolphin.mojam.com> Message-ID: <199909142009.QAA15534@eric.cnri.reston.va.us> > So I started looking around at the versions and dates of various copies of > asynchat.py. Here's what I found: > > source version number:date owner > Python 1.5.2 1.2:1999/06/18 guido > Python CVS 1.2:1999/06/18 guido > Zope 1.? 1.7:1999/04/09 amos > Zope 2.0 1.9:1999/07/19 amos > Medusa 990902 2.24:1999/07/07 rushing > > What's apparently been happening is that people have picked up asyncore and > asynchat at various time and stuck them in their own CVS repositories > without somehow freezing the Id string of the version they originally got > from Sam Rushing. It's not clear what the differences are until you compare > the actual files. It turns out that the Zope 2.0 and Medusa versions have > no content differences, only wildly different version numbers. The Medusa > and Python CVS versions only have one difference: > > if index > 0: > # don't bother reporting the empty string (source of subtle bugs) > self.collect_incoming_data (self.ac_in_buffer[:index]) > > which *may* be what's causing my problems (note the IndexError in my > traceback). My bad. I despise putting RCS identifiers in code I release, but I don't always freeze them before incorporating other people's code in my CVS tree. I will fix this if I can and I promise to try not to repeat this mistake in the future. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at lemburg.com Tue Sep 14 22:57:19 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 14 Sep 1999 22:57:19 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274795004-25696413@hypernet.com> Message-ID: <37DEB6AF.53974AAB@lemburg.com> Gordon McMillan wrote: > > [MA Lemburg] > > The solution I'm using (and Zope is too) is to wrap a > > complete Python installation into the product, all setup to > > do the right thing. It makes the archive a little heavier... > > which is actually not so bad from a marketing POV. Bizarre, but > > people seem to want all those megs. > > This is fine if you're distributing an app - it's a black box, it's > standalone, no dependencies outside the app. Cool. I do that > too. I use imputil to serve modules out of archives - no need to > tweak the modules at all. In the upcoming version (of my > installer) sys.path has exactly one entry, and the only python > that needs to be outside an archive is exceptions.pyc. > > (I repeat here: while the pieces of my installer that handle > binary dependencies are Win32 only, the parts that handle > pure Python are cross-platform.) > > But both you and Jim are aiming for this non-existant in- > between space - in some respects you want a "normal" > Python installation, but you want to black-box your turf. Actually, I want my stuff to be as flexible as possible. Of course I'm using it in my product too, but that's a different story. I have the unsatisfying impression that this discussion doesn't lead anywhere. I can't really understand all the evil sides Tim and Guido seem to see in their views of relative imports. For people like Jim and me, who have quite some experience in doing Python packages, this small (! hey, it's only about 20 lines of code !) additional feature could be of great use. I'm still waiting for some ultimate argument that blows relative imports away. The arguments put forward so far have all been in the category "evil", "ugly", "I don't like it". Hey, we can do better than that... Note that if this doesn't work out, Jim and I could always agree on some standard import hook that we'd both use, but that's exactly what we would very much like to avoid in favour of some established standard hard-wired into the Python distribution. > I'm sorry, but if you're going to fit into a Python installation, > you should make public your dependencies. Yes, installation > becomes more complicated, but you're dealing with *Python > users*. They're at least as likely to get screwed by having > multiple different copies of the same thing around as you are > in having dependencies outside of your personal control. I am probably going to ship all the mx* stuff in one big package -- not as separate packages anymore. That way I can write code which depends on other parts of the mx universe more easily. > If you also want to distribute as a standalone, freeze/squeeze/ > whatever it. Just make sure that what you distribute this > way won't interfere (or be confounded by) any existing Python > installations. It won't interfere with anything since the Python interpreter I use is configured *not* too look in any standard places for extensions and the like. Plus it's configured to run faster, but that's a different story ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 108 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Tue Sep 14 23:09:53 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 17:09:53 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Mon, 13 Sep 1999 11:21:15 EDT." <37DD166B.746B19AD@digicool.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> Message-ID: <199909142109.RAA15752@eric.cnri.reston.va.us> Jim Fulton wrote: > I wouldn't be in favor of making it more complicated if there wasn't > a good reason. I think that, in working on the Zope framework, > I've found some pretty good reasons for relative imports. And in a later message: > I don't think the Package structure of Zope is flawed *except* > for the fact that it is one level too *shallow*. The ability to > do relative imports would be very helpful for the work we're doing. But I haven't seen explained what it is that Zope is doing where relative packages would be helpful. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at CNRI.Reston.VA.US Wed Sep 15 00:47:18 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 18:47:18 -0400 Subject: [Python-Dev] License cleanup Message-ID: <199909142247.SAA16609@eric.cnri.reston.va.us> After more than four years of living with an out-of-dated license for Python, CNRI has finally agreed to clean up Python's copyright status. I expect that this won't have any real effect before Python 1.6 is released, but I am required to start preparing for the transition now. We will use a new license (a clone of the JPython license) and we will require that all contributors explicitly allow us the use of their contribution: either a few email paragraphs in an email message, or a longer form with a wet signature, depending on the size of the contribution. I believe the text of the license and forms we use is quite uncontroversial; these very same words have been used for JPython for quite a while. The words are all on the web: http://www.python.org/1.5/pylicense.html [proposed license] http://www.python.org/1.5/bugrelease.html [email release] http://www.python.org/1.5/wetsign.html [wet signature release] If you are reading python-dev but you never contributed any code to Python, you can stop reading now. If you *did* contribute code to Python, however, I'd love it if you saved me some work and filled out the wet signature form and mailed it to me at the given address. If you need help jogging your memory what your contributions were, send me email; I can try grepping the CVS files for your name. If you believe that special circumstances exist that make it impossible or difficult for you to sign the form, please send me email, and we'll discuss the matter. If you contributed something and I don't hear from you, you will eventually hear from me again -- but I hope I can save myself the hassle of writing each of you through this mass mailing. Thanks in advance! --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Wed Sep 15 01:08:32 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 16:08:32 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: Actually, I don't like them all that much :-( [I don't recall any specific discussion about it, but I may have missed it and/or simply because I've never used JPython.] The BSD-ish license that Python has always used is much more preferable. I dislike the regulation of the "Python" name, the requirement to prominently discuss modifications made, and the revocation clause. I might find other items, but that is from a quick read using Lynx on a tiny monitor... Heck, how could people like PPSI, PythonWare, or D.C. truely like that license? Each of those companies uses "Python" significantly in their marketing and their business. I can certainly state that PPSI will never do anything in an official capacity to recognize that license. [there is a separate issue of whether "Python" can be trademarked, but the license does use the term "trade name" which could easily be argued to include the term "Python" and thus subject the name to the license.] > If you *did* contribute code to Python, however, I'd love it if you > saved me some work and filled out the wet signature form and mailed it > to me at the given address. No problem. Future contributions and agreemend to abide by that license are a different issue. It doesn't have the "feels good" feeling that the old license does. I'm not sure that bodes well, and it doesn't sit well with me at the moment. Regards, -g -- Greg Stein, http://www.lyra.org/ From guido at CNRI.Reston.VA.US Wed Sep 15 01:31:47 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 14 Sep 1999 19:31:47 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Tue, 14 Sep 1999 16:08:32 PDT." References: Message-ID: <199909142331.TAA16699@eric.cnri.reston.va.us> > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. I might > find other items, but that is from a quick read using Lynx on a tiny > monitor... Hm... We may have to review the regulation of the Python name. This made sense in the context of the previous uses of this license (JPython and Grail) but Python is a different thing -- the name Python stands for more than just the implementation. I'll discuss this with CNRI's legal team. I don't see how the other things you mention can be much of a problem (most Open Source licenses have a revocation clause these days, I think, and I don't see how discussing the modifications made can be a problem with open source users). > Heck, how could people like PPSI, PythonWare, or D.C. truely like that > license? Each of those companies uses "Python" significantly in their > marketing and their business. I can certainly state that PPSI will never > do anything in an official capacity to recognize that license. How can you say that without consulting with the board? And I am *on* that board! I despise your attitude. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw at cnri.reston.va.us Wed Sep 15 01:44:54 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Tue, 14 Sep 1999 19:44:54 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <14302.56822.478320.862970@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Heck, how could people like PPSI, PythonWare, or D.C. truely GS> like that license? Each of those companies uses "Python" GS> significantly in their marketing and their business. Data point: I know that there are a number of companies that have embedded JPython in their commercial products. So far I've had zero complaints from them on the JPython license. -Barry From gstein at lyra.org Wed Sep 15 02:11:56 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:11:56 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142331.TAA16699@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > Hm... We may have to review the regulation of the Python name. This > made sense in the context of the previous uses of this license > (JPython and Grail) but Python is a different thing -- the name Python > stands for more than just the implementation. I'll discuss this with > CNRI's legal team. Cool. > I don't see how the other things you mention can be much of a problem > (most Open Source licenses have a revocation clause these days, I > think, and I don't see how discussing the modifications made can be a > problem with open source users). I'll do some more reading. As I said: that was my first cut. The revocation clause doesn't sit well with me. Maybe other OSS packages have it, but I believe that is usually because the license was developed by a company and its legal team. I don't think the GPL, BSD, MPL, and Apache licenses have revocation clauses, and I consider those to be the "most open" types of licenses (MPL less so). The Python 1.5 license is just as open, more so than most. > > Heck, how could people like PPSI, PythonWare, or D.C. truely like that > > license? Each of those companies uses "Python" significantly in their > > marketing and their business. I can certainly state that PPSI will never > > do anything in an official capacity to recognize that license. > > How can you say that without consulting with the board? And I am *on* > that board! I despise your attitude. Because the President (me) runs the day-to-day operation and direction of the company. The Board advises. The Board typically has other duties such as replacing me :-), handling stock issues, etc, but the Board is typically not involved with most issues. This is standard practice for corporate organization. Therefore, I *can* make that choice, and even do it unilaterally if I wanted to be an ass about it. Will I refuse to listen to the board or the shareholders or the employees? Of course I'll listen. [further PPSI issues should be taken offline] Regardless: it boils down to the "Python" requirement in that license. PPSI simply cannot operate under that license. If it gets dropped, then cool. -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Wed Sep 15 02:16:10 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:16:10 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Barry A. Warsaw wrote: > >>>>> "GS" == Greg Stein writes: > GS> Heck, how could people like PPSI, PythonWare, or D.C. truely > GS> like that license? Each of those companies uses "Python" > GS> significantly in their marketing and their business. > > Data point: I know that there are a number of companies that have > embedded JPython in their commercial products. So far I've had zero > complaints from them on the JPython license. Are they using it in their marketing, or simply as an underlying driving force for their products? If they *are* using it in their marketing, then they have exposed themselves to a liability. According to the license that they are using, they are not allowed to use JPython in their marketing. If they do, then they are in breach of the license and it could be terminated on them. Their products could no longer include JPython and they'd be SOL. I would be interested to hear from somebody using JPython and marketing it and how they interpreted that license. Possibly I'm missing something, but that language seems pretty darn clear to me. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Wed Sep 15 02:24:25 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 14 Sep 1999 17:24:25 -0700 (PDT) Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DEB6AF.53974AAB@lemburg.com> Message-ID: On Tue, 14 Sep 1999, M.-A. Lemburg wrote: >... > I have the unsatisfying impression that this discussion doesn't > lead anywhere. I can't really understand all the evil sides Tim and > Guido seem to see in their views of relative imports. For people > like Jim and me, who have quite some experience in doing > Python packages, this small (! hey, it's only about 20 lines of > code !) additional feature could be of great use. $| was probably a small addition to Perl, but I don't think you could then argue that it was a good thing to do. Small doesn't mean good. Any change "could be of great use" to *somebody*, but does it make it good for Python as a whole? Changes imply doc, maintenance, future compatibility, etc. > I'm still waiting for some ultimate argument that blows > relative imports away. The arguments put forward so far have > all been in the category "evil", "ugly", "I don't like it". > Hey, we can do better than that... People seem to be disagreeing with your stated requirement. In other words, they're saying that you shouldn't be attempting to make your package "portable across the module-import-space." I tend to agree. Fix it at one location. If an app doesn't like that, then they can individually compensate. I don't believe that Python's standard machinery is required to handle this particular notion of packaging. > Note that if this doesn't work out, Jim and I could always agree on > some standard import hook that we'd both use, but that's > exactly what we would very much like to avoid in favour of some > established standard hard-wired into the Python distribution. No reason the standard hook could not go into imputil.py. I believe people mostly care that it doesn't become *standard* or *default* behavior -- that it only is available when explicitly requested by an app. Cheers, -g -- Greg Stein, http://www.lyra.org/ From mhammond at skippinet.com.au Wed Sep 15 02:49:35 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 15 Sep 1999 10:49:35 +1000 Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <002e01beff14$306e5b00$0801a8c0@bobcat> > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: Obviously IANAL. However, this language does make me feel less comfortable than the existing one. The ability to terminate would appear an issue - it would seem to take a braver CEO to base their technology on Python with this hanging over them. Sure, it may rarely be invoked, but I certainly wouldnt want to fight it in court if it was. If I was writing in C, I could worst-case grudgingly accept needing to change compilers - but I dont have that luxury for Python. If my license was terminated, I have nowhere else to turn. It is a real shame when lawyers get so involved. Obviously Guido has no say in this, but IMO the ideal scenario would be to use the exsting language, but simply change the names and dates. Im guessing this would be unacceptable to CNRI. Being NAL, I suppose I have no choice other than to trust this licence. However, Im not looking forward to showing this licence to people as they are deciding if Python is the appropriate technology choice - to date, there has never been an issue - all they need to is not remove any copyright notice from the code (which is not actually seen in most apps) and add the copyright notice to the documentation. This new one seems much scarier to me.. Just my $200.00 worth (remember, we are talking lawyers fees here :-) I dont have a real concern as I dont understand the legal implications; just a slight uneasiness about it all...Not being controversial for the sake of it, just airing my possibly il-informed opinion - no opinions were solicitied, but that has never stopped me before :-) Of course, I will be sending my "wet" signature on the form. Im not sure what to put in the "contribution description" - maybe just "various small changes to the Windows port"?? I can't say Ive added entire modules, but my name appears against a number of small patches to a fairly large set of files... Mark. From tim_one at email.msn.com Wed Sep 15 07:19:48 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 01:19:48 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <19990914101700.A4401@cnri.reston.va.us> Message-ID: <000301beff39$ee3015a0$612d153f@tim> [Tim, speaks of the devil ...] > "Something like that" [MAL's __version__ string] needs to be formalized > and imposed on all public packages. > > at-which-point-the-distutils-sig-jumps-in-and-saves-the-day-ly y'rs - tim [... and Greg Ward of his legions appears!] > Been there, tried that, bought the flame war. I made the mistake of > kicking off the Distutils SIG back in Decemver with a proposal for a > standard version numbering scheme for Python module distributions. See > > http://www.python.org/pipermail/distutils-sig/1998-December/000016.html > > for the kick-off of that "heated discussion". ;-) Greg, if you call that a flame war, your credentials as an ex-Perl'er are in serious doubt . Except for the cowboy contingent, most participants were moving swiftly to consensus! > FWIW, if I was posting that message today, I would s/must/should/ and > that's about it. No, it's "must" or it's useless. What wasn't brought up in that thread is that the Distutil "version number" is an artficial construct created for the primary benefit of Distutil tools -- it needn't have anything whatsoever to do with whatever silly string the developer wants to *display* as being their "version number". It's instead a coordinate in an abstract but rigidly defined Distutil space, specifically designed to make programmatic navigation of that space reliable in a shared and uniform way. If a developer chooses, users need never be exposed to it. I'd use the x.y.z Distutil version number directly to keep my own life simpler, but if someone else wants to display a GUID followed by a 3-letter country code and the number of nanoseconds since the birth of Mohammed, fine -- they still have to map that to Distutil VN space internally or write their own stinkin' disttools. You may have went overboard on the *semantics* of the Distutil VN, though: its only real meaning is in what Distutil tools *do* with it. Fight this battle again. Without a uniform way for an installer to *know* when it's thought safe to replace a package with another version of that package, Python installations will never move beyond the similar hell of Windows 3.1. even-herds-of-cats-wear-collars-ly y'rs - tim From tim_one at email.msn.com Wed Sep 15 07:40:35 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 01:40:35 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: <000501beff3c$d58a8be0$612d153f@tim> I dislike the new license. Selling Python at work wasn't easy, but the short & straightforward CWI license went a *long* way toward convincing the suits there was little to worry about. The new license has several blobs of lawyer-speak that ensure the next battle will be much harder -- the prospect of license revocation, some fuzzy concept of derivative works, and vague "prominent display" requirements? Boston lawyers charge Really Big Bux to guess what that gibberish might mean in Virginia. The only bright side is that we now get explicit rights to "perform" and "display" Python . > If you believe that special circumstances exist that make it > impossible or difficult for you to sign the form, please send me > email, and we'll discuss the matter. It would less hassle for me if you took all my contributions out <0.9 wink>. i'll-sign-but-it's-really-really-really-depressing-ly y'rs - tim From tim_one at email.msn.com Wed Sep 15 08:42:02 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 02:42:02 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DE0825.10D2BC6F@lemburg.com> Message-ID: <000701beff45$6b51c5a0$612d153f@tim> Don't tell, but this is getting repetitive. One more batch of repeats from me, and I'm done: + The notion that Python packages aren't self-contained now is wrong: a package module can import other modules in the package without qualification. This is good design. + Where self-containment breaks down is in going across *sub*-packages. I was surprised Guido pretended to cater to them, since Python (wisely, IMO) stayed away from sub-modules from the start. Hierarchy is a great tool for managing complexity, but it also introduces complexity of its own. And when a flat package space suffices, hierarchy introduces artificial complications. + I agree with Gordon that, if anything, the default "import" behavior is already too complicated. By the time you wade thru PYTHONPATH, and PYTHONSTARTUP, and .pth files, and people mucking with sys.path, and site config files, and symlinks under Unix, and packages mucking with their own __path__ attributes, running Python with -v is the only hope of figuring out why nothing ever works <0.6 wink>. + All this dynamicism is in support of a concept that's actually static: the structure of an installation. As Barry laments, you can't always know the structure in advance of installation, but the solution is (I believe) along the lines he suggests: invent a way to register/query the structure once & for all after it is known. + JimF and MAL's shared notion that they're the only ones making significant use of packages is myopic. Dragon also uses packages, and heavily, although products using them have not been announced and I can't talk about them yet. Anything I'm within 10 miles of doesn't use *any* import hooks, or play any other tricks on the system. Instead I urge people to write their imports as explicitly, long-windedly and straightforwardly as possible (with binding to a convenient local alias name on the following line). Do we embed sub-packages? Sure. Do modules move around in the hierarchy over time? Of course. So how do we deal with this supposed nightmare? Because all imports are relentlessly straightforward (even intra-package imports are fully qualified), and none are hiding in dynamic functions, we get into Emacs and do a global search-and-replace. Takes 30 seconds, and the new structure is plain as day to everyone reading the code. low-tech-it's-not-just-for-breakfast-anymore-ly y'rs - tim From ping at lfw.org Wed Sep 15 09:31:29 1999 From: ping at lfw.org (ping at lfw.org) Date: Wed, 15 Sep 1999 00:31:29 -0700 (PDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909142247.SAA16609@eric.cnri.reston.va.us> Message-ID: On Tue, 14 Sep 1999, Guido van Rossum wrote: > I believe the text of the license and forms we use is quite > uncontroversial; these very same words have been used for JPython for > quite a while. The words are all on the web: > > http://www.python.org/1.5/pylicense.html [proposed license] > http://www.python.org/1.5/bugrelease.html [email release] > http://www.python.org/1.5/wetsign.html [wet signature release] Hi all. I'm sorry i haven't contributed anything to the relative-import and python-path discussions of late, but that's because so far i haven't had any ideas that have crossed my threshold of being sufficiently insightful to propose. I will follow the discussion with much interest. I'm afraid i have to say that the revocation clause makes me pretty uncomfortable. I know that it says CNRI will revoke only on a "material breach", but i still have a nasty suspicion that it sounds frightening enough to scare many people away. I don't think we want that. I suppose Greg's other points of contention are valid too but it's really the revocation that bugs me the most. -- ?!ng From fredrik at pythonware.com Wed Sep 15 09:42:28 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Sep 1999 09:42:28 +0200 Subject: [Python-Dev] License cleanup References: Message-ID: <008701beff4d$dcebfb10$f29b12c2@secret.pythonware.com> > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. same here. reading the new one made me feel very uneasy, but I cannot really say much about it before I've discussed it with people who know more about this... just a few small notes: the BSD-ish license used up to now has been a major selling argument for Python, while this one seems to really push the bounds of what qualifies as an open source license... (it also seems to imply that Python is a trademark, which is, as far as I can tell, is not true at this time. and archive corporation/seagate already owns the trademark wrt. software). the worst thing is that we will have to run this by our lawyers before we can decide whether to continue contributing to 1.6 development :-( From mal at lemburg.com Wed Sep 15 11:24:43 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 11:24:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <000701beff45$6b51c5a0$612d153f@tim> Message-ID: <37DF65DB.46F0191A@lemburg.com> Tim Peters wrote: > > + The notion that Python packages aren't self-contained now is wrong: a > package module can import other modules in the package without > qualification. This is good design. Agreed. And we would like to take that notion one step further -- without breaking the two-scope resolution Python uses now. The '__' tags are just place holders which could just as well be handled by a simple Python fuction (like the one Gordon posted) using perfectly valid technology. Sticking this technology into the import statement is really only a more elegant approach. Nothing more. The whole thing already works using the current Python implementation. > + JimF and MAL's shared notion that they're the only ones making significant > use of packages is myopic. Oh, c'mon Tim, we never did say that. We only mentioned having used package for quite a while. This includes having seen all the different pitfalls they have to offer. > Dragon also uses packages, and heavily, although > products using them have not been announced and I can't talk about them yet. > Anything I'm within 10 miles of doesn't use *any* import hooks, or play any > other tricks on the system. Instead I urge people to write their imports as > explicitly, long-windedly and straightforwardly as possible (with binding to > a convenient local alias name on the following line). Do we embed > sub-packages? Sure. Do modules move around in the hierarchy over time? Of > course. So how do we deal with this supposed nightmare? Because all > imports are relentlessly straightforward (even intra-package imports are > fully qualified), and none are hiding in dynamic functions, we get into > Emacs and do a global search-and-replace. Takes 30 seconds, and the new > structure is plain as day to everyone reading the code. This is perfectly ok if you're just using your own code, but it gets a mess when third-party packages are involved simply because you can't be sure they don't use import hooks, local imports or hacks to __path__ etc. If you still want them to be useable, you'll have to go down and dive into their structure. This takes time, is error prone and not necessarily fun. Also, I don't know how "explicitly, long-windedly and straightforwardly" writing module imports is any different from using relative imports. An example: mx .DateTime .ODBC.Windows In ODBC.Windows I would write: import __.__.DateTime which is just as explicit as writing import mx.DateTime except that the information about the top-level hierarchy is *not* included in the import information. Anyway, I'm not too inclined continuing this discussion anymore. I would never have thought that such a nifty little patch would have stirred up so much negative response. I'll just hack up my very own import mechanism using imputil and be done. So there! -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Wed Sep 15 12:05:27 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 12:05:27 +0200 Subject: [Python-Dev] License cleanup References: <008701beff4d$dcebfb10$f29b12c2@secret.pythonware.com> Message-ID: <37DF6F67.59FCEA6D@lemburg.com> Fredrik Lundh wrote: > > > The BSD-ish license that Python has always used is much more preferable. I > > dislike the regulation of the "Python" name, the requirement to > > prominently discuss modifications made, and the revocation clause. > > same here. reading the new one made me feel very > uneasy, but I cannot really say much about it before > I've discussed it with people who know more about > this... Dito. Some comments: """ 4.Licensee may not use CNRI trademarks or trade name, including Python or CNRI, in a trademark sense to endorse or promote products or services of Licensee, or any third party. Licensee may use the mark Python in connection with Licensee's derivative versions that are based on or incorporate the Software, but only in the form "Python-based ___________________," or equivalent. """ Say I want to sell Python 1.6 training, how would I promote this ? Since I'm not producing a derivative work, I guess I couldn't use the name 'Python' at all... hmm, I could probably try Pyth*n ;-) """ 3.In the event Licensee prepares a derivative work that is based on or incorporates the Software or any part thereof, and wants to make the derivative work available to the public as provided herein, then Licensee hereby agrees to indicate in any such work, in a prominently visible way, the nature of the modifications made to CNRI's Software. """ How explicit would that indication have to be ? E.g. do I have to provide a patch or would a simple run-down of new features suffice ? Needless to say, I would not be able to sell products based on Python 1.6 with the revocation clause in the license. In the end, I'd probably have to negotiate a separate license with CNRI not having this clause. Anything else would be unacceptable in a commercial setting. Is this intended ? And finally in the "Python Contribution Agreement": """ Licensee confirms to CNRI that, to the best of Licensee's knowledge and belief, the Contribution is free of any claims of parties other than Licensee under copyright, patent or other rights or interests ("claims"). """ Best knowledge and belief do not guard against law suit. Why doesn't this text protect the contributor in some way against charges forwarded by CNRI to the contributor ? (Note that the disclaimer in the Python License is not valid everywhere.) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From bwarsaw at cnri.reston.va.us Wed Sep 15 13:56:48 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 07:56:48 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: <14303.35200.979238.868146@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Are they using it in their marketing, or simply as an GS> underlying driving force for their products? I'm not sure that JPython is much of a marketing advantage right now, so AFAIK none of them are actively promoting their use of JPython in their product. However, my reading of the second half of item 4 would allow them to say something like "You can even extend your flapjabs using our keen JPython-based scripting capabilities". GS> If they *are* using it in their marketing, then they have GS> exposed themselves to a liability. According to the license GS> that they are using, they are not allowed to use JPython in GS> their marketing. If they do, then they are in breach of the GS> license and it could be terminated on them. Their products GS> could no longer include JPython and they'd be SOL. I hope that wouldn't really be the case, but it's an interesting point, so I'm sure we'll bring it up. -Barry From jim at interet.com Wed Sep 15 14:20:54 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 08:20:54 -0400 Subject: [Python-Dev] License cleanup References: Message-ID: <37DF8F26.37BC155E@interet.com> Greg Stein wrote: > > On Tue, 14 Sep 1999, Guido van Rossum wrote: > > I believe the text of the license and forms we use is quite > > uncontroversial; these very same words have been used for JPython for > > quite a while. The words are all on the web: Oh boy, this is really going to cause trouble. Where's my flame suit... > The BSD-ish license that Python has always used is much more preferable. I > dislike the regulation of the "Python" name, the requirement to > prominently discuss modifications made, and the revocation clause. I might > find other items, but that is from a quick read using Lynx on a tiny > monitor... I guess I am used to reading license agreements, and I am not very worried about the new one. Before we all get upset, lets remember that Guido works in a large company with lots of lawyers, and he trapped between a group of Internet geeks (hey, I like Internet geeks) and his buracracy. And remember that lawyers respond better to specific proposals for language changes than philosophical discussion. First off, the license is not revokable. It is only revokable on breach. If a license can not be revoked on breach it doesn't really mean anything. This is totally standard. Suppose someone else claims to own Python and starts selling "The True Standard Python" for $100. Suppose they change the standard library names so software only runs on their version. CNRI should be able to revoke their license to use Python. This is something we would all want CNRI to do. The protection of the Python name is a necessity. That is really all CNRI has, since the license gives away use of the software itself. If CNRI doesn't own "Python" then it can't object when someone else claims they own it. Don't we want them to object? The license doesn't say you can't use "Python", it sayes you can't use it in a trademark sense. I think that means you can say "I am teaching a course on Python, which is CNRI's software" but not "I am teaching a course on my Python, all rights reserved". Actually this is a little unclear, perhaps (4) could be made a little clearer. Paragraph (3) is a little troublesome. I seems to mean that if you ship a modified Python, you must say it is modified. I presume it doesn't mean that you must describe your own code in the event it incorporates Python. Really, we need to know what CNRI wants us to do here. On the contributions side (wetsign.html) it says you are contributing software free of third party claims "to the best of your knowledge and belief" not "represents and warrants" which is different. CNRI really has to be told that as far as you know, you didn't steal the software you are contributing. This is reasonable. Actually I might like to see a warranty disclaimer "NO WARRANTIES etc." like the license paragraph (5) and (6). I am not sure I need it since the contribution is free, but I usually ship free software with a disclaimer for "fitness for any particular purpose etc.". This is a pretty weak license agreement. Remember that if it is too weak, it prevents CNRI from defending Python against others who would claim they own it or who claim they are the true source of the language design (paranoia department: Microsoft's Python++). We want CNRI to defend Python, right? Jim Ahlstrom From gward at cnri.reston.va.us Wed Sep 15 14:30:32 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Wed, 15 Sep 1999 08:30:32 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <14302.56822.478320.862970@anthem.cnri.reston.va.us>; from Barry A. Warsaw on Tue, Sep 14, 1999 at 07:44:54PM -0400 References: <199909142247.SAA16609@eric.cnri.reston.va.us> <14302.56822.478320.862970@anthem.cnri.reston.va.us> Message-ID: <19990915083032.A17433@cnri.reston.va.us> On 14 September 1999, Barry A. Warsaw said: > Data point: I know that there are a number of companies that have > embedded JPython in their commercial products. So far I've had zero > complaints from them on the JPython license. Just thought I should join the tide of opposition: heck, I *work* for CNRI and I still don't like the license. I didn't say much about the new JPython license because a) I trust Barry's judgement, b) it was certainly an improvement over the old JPython license, and c) I wasn't especially worried about one part of CNRI (Guido's group) taking JPython away from another part (the group that Andrew and I are on). However, that doesn't change the fact that the "new" license is a nasty piece of legalistic gibberish. Making it the license for Python 1.6 would be a major setback -- while it was better than the old JPython license, it's a damn sight worse than the old Python license. I have zero sympathy for the legal beagles here with their narrow corporatist viewpoint; trying to treat Python as just another potential piece of intellectual property is wrong-headed in the extreme. The free software world simply does not work that way. BTW, I suspect that the companies embedding JPython haven't minded the license because they come from the Java world, a world that seems to me to be dominated by corporate pin-headed thinking. The idea of community, openness, and sharing is utterly alien to these suit-wearing, smarmy Java frat-boy types, so JPython's licensing terms were probably a breath of fresh air to them. ("What? No $100,000 source license fee? Wow!") ("But wait Chip -- it's not BUZZWORD COMPLIANT! I can't find enough TLAs!!!") Hmmm, enough flaming Java weenies. Please, don't anybody take the last paragraphy too seriously or personally... Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fredrik at pythonware.com Wed Sep 15 14:39:30 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 15 Sep 1999 14:39:30 +0200 Subject: [Python-Dev] License cleanup References: <37DF8F26.37BC155E@interet.com> Message-ID: <003e01beff77$5c87fe40$f29b12c2@secret.pythonware.com> > The license doesn't say you can't use "Python", it sayes you > can't use it in a trademark sense. quick check: which of these uses "Python" in a trademark sense, and thus violates the license: pythonware? professional python services? pythonworks? programming python? python training? python powered? the viper python implementation? python imaging library? wxpython? pythonwin? etc. all of them? none of them? From mal at lemburg.com Wed Sep 15 15:09:27 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 15:09:27 +0200 Subject: [Python-Dev] License cleanup References: <37DF8F26.37BC155E@interet.com> <003e01beff77$5c87fe40$f29b12c2@secret.pythonware.com> Message-ID: <37DF9A87.55AF6279@lemburg.com> Fredrik Lundh wrote: > > > The license doesn't say you can't use "Python", it sayes you > > can't use it in a trademark sense. > > quick check: which of these uses "Python" in a trademark > sense, and thus violates the license: > > pythonware? professional python services? pythonworks? > programming python? python training? python powered? > the viper python implementation? python imaging library? > wxpython? pythonwin? etc. > > all of them? none of them? Using a word in a trademark sense usually simply means using it in corporate relationships (at least that's how it works in Germany). If you are a company and talk about, write about or otherwise use the word in a commercial context then you are using the word in a trademark sense. There are several ways to declare a trademark, e.g. there are word marks, logo marks, sound marks, color marks etc. (don't know if these are the right translations). A word mark, for example, refers to a specific spelling of the word regardeless of the font, style or color. Logo marks refer to a specific design including font, style and color. Note that a trademark owner can still give you permission to use the mark in any decent way without paying fees or royalties. So even if CNRI does own the mark, they could still make it usable by others. In fact, if done right, this is a Good Thing. The answer to your question depends on what kind mark CNRI owns. [There currently is a very strong movement in Germany against people who are applying what they learned from domain grabbing to trademarks. Prominent examples include "WWW" and "Webspace". Even the color violet is trademarked (by a company producing chocolate)] -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at interet.com Wed Sep 15 17:43:13 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 11:43:13 -0400 Subject: [Python-Dev] Re: Path hacking Message-ID: <37DFBE91.C93816F1@interet.com> "Guido van Rossum" wrote: > --> The solution: Ah, finally a specific proposal... > Suppose your application (as a whole, not the individual top-level > script) is called Spam -- this may well also be the name of your > top-level package. Then start each top-level script with the single > line > > import Spam_path > > before importing anything else. This should not be necessary if you use the name "sitecustomize" instead of "Spam_path" right? The file sitecustomize.py is automatically imported. Actually all this sounds like site.py all over again. > Your installer, once it knows the absolute pathname of your > application's root directory, crafts a file Spam_path.py which > contains code that inserts the right absolute pathname into sys.path. I don't think this is necessary either. The sys module is available. So sitecustomize.py can say: import sys mydir = sys.path[0] if not mydir: import os mydir = os.getcwd() sys.path = [mydir] # To be really extreme about it # Note: inserting mydir as sys.path[0] should be redundant but is not > Your installer then installs a copy of this file (or a symbolic link > to it) *in each bin directory where it installs top-level Python > scripts*. > > Because the script's directory is first on the default path, the Spam > scripts will pick up Spam_path without any help from $PYTHONPATH. Hmmm. Is this really true? Nothing else, for example the registry, can change sys.path[0]? Ever? Please say yes. > I know this doesn't resolve the relative import thread (how's that > going by the way? :-) but Barry & Fred & I agree that this is the best > solution to the problem stated in Barry's message to which I am > following up here. This is a good idea, but there are a few problems. It depends on sys.path[0] being the directory of the Python file being executed as the main program. I guess I never really trusted this before. I think if this is the case it should never be ''. A relative path or no path on the command line (the __main__ program) should be replaced by the full path in the sys module setup. Then the "mydir = os.getcwd()" above is not necessary. And inserting mydir as sys.path[0] is truly redundant should the current directory change (as it certainly will). This is currently a problem with sys.path[0] which should be fixed no matter what else happens. The files exceptions.py and site.py must be in all the bin directories as well as sitecustomize.py because they are automatically imported in Py_Initialize(). The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point. It seems to me this totally solves Jim Fulton's and Marc's problem and makes "__" unnecessary. You just install zope and mx in zopedir, perform the above, and presto you have a new private name space where you can control all your names. But there must be some problem here I haven't thought of. I still worry that this is not powerful enough. Greg Stein has volunteered to re-write import.c in Python (using imputil.py) and this is a Great Idea. Lots of Python could probably be written in itself. I would like to try writing the main program in Python and eliminating the special freeze main program. Once you start on this road (and I think it is a good road) you have Python code which is more truly part of the binary interpreter than a library. Proposal: Use a special PYTHONPATH on startup to find "special" Python files which are really part of the interpreter. There are three directories Python knows about. Namely sys.path[0] (once it is fixed), sys.executable and sys.dllfullpath, the directory of python15.dll or other shared library (once it is added to sys). How about prepending the single directory sys.executable to sys.path during Py_Initialize()? And demanding that modules like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] be placed there. Actually I would prefer sys.dllfullpath if it exists, since that is where the interpreter is, and I am trying to associate these special internal Python files exactly with their correct Python interpreter. Alternative Proposal: Py_Initialize() first imports its files from sys.executable + '/' + PyInternal.pyl (again I prefer sys.dllfullpath). PyInternal.pyl is a Python library file (like a Java Jar file) which would contain modules like exceptions, etc. The PyInternal.pyl file has the standard Python library file format (whatever that turns out to be). It is not an error if this file is absent. Jim Ahlstrom From guido at CNRI.Reston.VA.US Wed Sep 15 18:15:02 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 12:15:02 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 11:43:13 EDT." <37DFBE91.C93816F1@interet.com> References: <37DFBE91.C93816F1@interet.com> Message-ID: <199909151615.MAA17872@eric.cnri.reston.va.us> [Guido] > > import Spam_path > > > > before importing anything else. [JimA] > This should not be necessary if you use the name "sitecustomize" instead > of "Spam_path" right? The file sitecustomize.py is automatically > imported. > Actually all this sounds like site.py all over again. But the intention here is for the customization to be application specific (hence the Spam in the name). sitecustomize doesn't know whethere I need the Mailman or the Knowbot root added to my path. Or do you mean to imply that we can do this with zero text added to the script, by simply dropping an appropriate sitecustomize.py in the script dir? Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run. > > Your installer, once it knows the absolute pathname of your > > application's root directory, crafts a file Spam_path.py which > > contains code that inserts the right absolute pathname into sys.path. > > I don't think this is necessary either. The sys module is available. > So sitecustomize.py can say: > import sys > mydir = sys.path[0] > if not mydir: > import os > mydir = os.getcwd() > sys.path = [mydir] # To be really extreme about it > # Note: inserting mydir as sys.path[0] should be redundant but is not Hm, guessing based on the script directory might work, but seems less reliable than hardcoding it through the installer. But you can use this if it works for your application. > > Your installer then installs a copy of this file (or a symbolic link > > to it) *in each bin directory where it installs top-level Python > > scripts*. > > > > Because the script's directory is first on the default path, the Spam > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > Hmmm. Is this really true? Nothing else, for example the registry, can > change sys.path[0]? Ever? Please say yes. Yes. (The registry can add module-specific paths, which will be searched before sys.path is even looked at, but this is only for specific modules. It cannot insert a general directory that is searched.) The only way this can fail is if an embedding app fails to call PySys_SetArgv(). > > I know this doesn't resolve the relative import thread (how's that > > going by the way? :-) but Barry & Fred & I agree that this is the best > > solution to the problem stated in Barry's message to which I am > > following up here. > > This is a good idea, but there are a few problems. > > It depends on sys.path[0] being the directory of the Python > file being executed as the main program. I guess I never > really trusted this before. I think if this is the case it > should never be ''. A relative path or no path on the command > line (the __main__ program) should be replaced by the full path > in the sys module setup. Then the "mydir = os.getcwd()" above > is not necessary. And inserting mydir as sys.path[0] is truly > redundant should the current directory change (as it certainly will). > This is currently a problem with sys.path[0] which should be > fixed no matter what else happens. I have always resisted forcing path items to be absolute, although I'm not sure that my reasons are valid any more (it has to do with the fact that getcwd() may fail and the fact that portable path concatenation is a pain). In any case, that's a separate issue -- I agree that if sys.path[0] is '' (as it often is) it's better for site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize it (and everything else on the path) so that it will still work if the app does a chdir(). > The files exceptions.py and site.py must be in all the bin > directories as well as sitecustomize.py because they are > automatically imported in Py_Initialize(). Yes. > The above doesn't work when you start the Python command > interpreter (no main). I know, its a minor point. You could add the "import Spam_path" to your $PYTHONSTARTUP file. > It seems to me this totally solves Jim Fulton's and Marc's > problem and makes "__" unnecessary. You just install zope > and mx in zopedir, perform the above, and presto you have a new > private name space where you can control all your names. But > there must be some problem here I haven't thought of. I think no simple solution that *I* can come up with will satisfy JimF's and Marc's desire for obscurity :-) > I still worry that this is not powerful enough. Greg Stein > has volunteered to re-write import.c in Python (using imputil.py) > and this is a Great Idea. Lots of Python could probably be > written in itself. I would like to try writing the main > program in Python and eliminating the special freeze main > program. Once you start on this road (and I think it is a good road) > you have Python code which is more truly part of the binary > interpreter than a library. Yes, this is the plan for Python 2.0, and some of it may be implemented in Python 1.6. > Proposal: > > Use a special PYTHONPATH on startup to find "special" Python > files which are really part of the interpreter. There are > three directories Python knows about. Namely sys.path[0] > (once it is fixed), sys.executable and sys.dllfullpath, > the directory of python15.dll or other shared library (once it is > added to sys). How about prepending the single directory sys.executable > to sys.path during Py_Initialize()? And demanding that modules > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > be placed there. On Unix, this is a bin directory and it is strongly discouraged to put non-program files there. Python already does something similar -- it looks around in sys.executable's ancestors for a specific landmark, currently lib/python/string.py. Arguably, it should search for execeptions.py instead. > Actually I would prefer sys.dllfullpath if it exists, since that > is where the interpreter is, and I am trying to associate these > special internal Python files exactly with their correct Python > interpreter. Is the full DLL path available at any point? This would certainly be a good starting point -- especially when the DLL is loaded implicitly as the result of some COM operation. > Alternative Proposal: > > Py_Initialize() first imports its files from sys.executable + '/' + > PyInternal.pyl (again I prefer sys.dllfullpath). > PyInternal.pyl is a Python library file (like a Java Jar > file) which would contain modules like exceptions, etc. > The PyInternal.pyl file has the standard Python library file > format (whatever that turns out to be). It is not an error if > this file is absent. I guess this is all up to the redesign of the import mechanism (something like Greg Stein's imputil.py for sure). --Guido van Rossum (home page: http://www.python.org/~guido/) From da at ski.org Wed Sep 15 19:00:05 1999 From: da at ski.org (David Ascher) Date: Wed, 15 Sep 1999 10:00:05 -0700 (Pacific Daylight Time) Subject: [Python-Dev] License cleanup In-Reply-To: <37DF9A87.55AF6279@lemburg.com> Message-ID: Guido, maybe it would make sense to explain the need for a license change. Is my understanding correct that the occasion for the license change is that the copyright is now clearly shifting to CNRI, and as a result CNRI has to forge a license? (BTW, I thought *you* had the copyright transfer from CWI, not CNRI). --david From jim at digicool.com Wed Sep 15 20:10:17 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 14:10:17 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> Message-ID: <37DFE109.96C652B3@digicool.com> Guido van Rossum wrote: > > Jim Fulton wrote: > > > I wouldn't be in favor of making it more complicated if there wasn't > > a good reason. I think that, in working on the Zope framework, > > I've found some pretty good reasons for relative imports. > > And in a later message: > > > I don't think the Package structure of Zope is flawed *except* > > for the fact that it is one level too *shallow*. The ability to > > do relative imports would be very helpful for the work we're doing. > > But I haven't seen explained what it is that Zope is doing where > relative packages would be helpful. I posted an example in an earlier message. I'll recast it here, hopefully more eloquently. :) I'll also offer an alternate proposal that also solves my (and I suspect, Marc-Andre's) problem. Zope is an application platform. It provides a mechanism for developers to plug their own products into Zope. The idea is that someone gets Zope from zope.org and installs it. Then they get third-party products from other places. Zope products are python packages installed as sub-packages of the Zope 'Products' packages. Products are generally self-contained. If they need anything that's not part of standard Python or standard Zope, they need to include it or install what they need in a sub-package of another Zope package, 'Shared'. Because products come from "third parties", it is important that they be self contained. Making assumptions about the Zope or Python environments or, worse, modifying the Zope or Python environments is a bad idea. In this context, consider the following concrete, though fictional example. Aaron has written a collection of modules that implement an RDBMS system for Python, gadfly. He also has a set of modules for parsing, kjParsing, which is needed by gadfly. Currently, these are just a bunch of top-level modules distributed as a combined collection. It would make sense to turn these into two packages, gadfly and kjParsing. Now, if this was done, then the gadfly package would need to use package imports for kjParsing modules, as in: import kjParsing.kjParser So far, so good. Now, suppose that someone wants to create a Zope product, ZGadfly, that used gadfly. The simplest approach would be to include the gadfly and kjParsing packages in their Zope product. Of course, this won't work, because the imports, like the one above, will fail, because kjParsing is no longer a top-level package. It wouldn't do any good to move gadlfy and kjParsing to the shared package, although that might be desirable to share these packages with other products. They could try to stuff the packages into the Zope or Python paths, but that would break the rules and lead to problems in the long term. Hopefully, this illustrates the problem. I think that this will be a common problem in the future, as people build bigger and bigger systems with Python that reuse other people's packages. I'd be curious to hear how folks would solve this problem. Personally, I'd like the problem to go away. :) I'd like the Python import rules to change to make this solvable without import hooks or path hacking. I can think of two ways to approach this: - Relative parent imports: import __.kjParsing.kjParser Note that Python already supports relative imports for going down. For example, we use gadfly and kjParsing together as a single sub-package of our ZGadflyDA product. - Gradually less local searches. Currently, when doing an import in a package, two paths are searched, the package path and then the Python path. If there are intermediate packages, then perhaps their paths should be searched as well. For example, suppose we have the directory structure: Products/ZGadfly/gadfly kjParsing where Products is a top-level package, and we did an import in a module in gadfly: import kjParsing.kjParser Python would search the path of the package Products.ZGadfly.gadfly first, as it does now. This search would fail. Then it would search the path of Products.ZGadfly, where it would find kjParsing, and the import would succeed. This approach has the benefits: o It solves the problem. :) o It has no impact on un-nested packages, o It requires no code changes; it doesn't use the ugly __. Thoughts? Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gmcm at hypernet.com Wed Sep 15 19:35:57 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 13:35:57 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37DFBE91.C93816F1@interet.com> Message-ID: <1274705689-31068526@hypernet.com> Jim Ahlstrom wtoe: > "Guido van Rossum" wrote: > > --> The solution: Did the dev-list miss something? The last I see is Barry's post. - Gordon From guido at CNRI.Reston.VA.US Wed Sep 15 20:17:55 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 14:17:55 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 13:35:57 EDT." <1274705689-31068526@hypernet.com> References: <1274705689-31068526@hypernet.com> Message-ID: <199909151817.OAA18276@eric.cnri.reston.va.us> > From: "Gordon McMillan" > Jim Ahlstrom wtoe: > > > "Guido van Rossum" wrote: > > > --> The solution: > > Did the dev-list miss something? The last I see is Barry's post. Hm. We had an email glitch. Apparently this message got lost: Subject: Re: Path hacking From: Guido van Rossum To: python-dev at python.org Date: Tue, 14 Sep 1999 15:57:51 -0400 I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at interet.com Wed Sep 15 20:27:58 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 14:27:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <1274705689-31068526@hypernet.com> Message-ID: <37DFE52E.5D6F3981@interet.com> Gordon McMillan wrote: > > Jim Ahlstrom wtoe: > > > "Guido van Rossum" wrote: > > > --> The solution: > > Did the dev-list miss something? The last I see is Barry's post. My mail system is flakey, so I have been reading this list directly on python.org. I didn't get it by list either, so I assumed my mailer ate it. See: http://www.python.org/pipermail/python-dev/1999-September/000851.html Jim Ahlstrom From jim at interet.com Wed Sep 15 20:29:38 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 14:29:38 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> Message-ID: <37DFE592.7FD384A2@interet.com> Guido van Rossum wrote: > > But the intention here is for the customization to be application > specific (hence the Spam in the name). sitecustomize doesn't know > whethere I need the Mailman or the Knowbot root added to my path. Ah, you have multiple scripts in one directory and multiple Foo_path, Bar_path etc. I was thinking with my Windows head. A commercial Windows app generally has its own exclusive install directory, so I was thinking single directory so a single sitecustomize.py. > Or do you mean to imply that we can do this with zero text added to > the script, by simply dropping an appropriate sitecustomize.py in the > script dir? Yes, that is exactly what I was thinking. > Unfortunately this does currently *not* work, because > sys.path[0] is added after Py_Initialize() is run. Yikes! That kills using sitecustomize.py. Your Spam_path still works because it is imported later, but requires an import in each Python main script just as you said. Even worse, it means that exceptions.py and site.py can not be found at all except using the normal PYTHONPATH, and putting their path in Spam_path will *not* work. > > > Because the script's directory is first on the default path, the Spam > > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > > > Hmmm. Is this really true? Nothing else, for example the registry, can > > change sys.path[0]? Ever? Please say yes. > > Yes. (The registry can add module-specific paths, which will be > searched before sys.path is even looked at, but this is only for > specific modules. It cannot insert a general directory that is > searched.) The only way this can fail is if an embedding app fails to > call PySys_SetArgv(). Oh dear, I think I heard no instead of yes. Are you saying that if someone else installs a Python app on my customer's machine after I do, and sets a registry entry which sayes to use c:/other/path/to/site.py for site.py (as he may very well want to do), then if my Python program depends on getting my copy of site.py from my directory, it will then use the other copy instead and may very well fail? > In any case, that's a separate issue -- I > agree that if sys.path[0] is '' (as it often is) it's better for > site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize > it (and everything else on the path) so that it will still work if the > app does a chdir(). Point on the curve: Windows apps generally start from an icon which contains their path and current working directory, and these are generally different. So a Windows app in general will *never* have had a getcwd() equal to the path of either the binary interpreter or the Python main script. > > The files exceptions.py and site.py must be in all the bin > > directories as well as sitecustomize.py because they are > > automatically imported in Py_Initialize(). > > Yes. Well, *no* right? This fails unless the bin directories are in fact on PYTHONPATH. The only way to get exceptions.py is by using sys.path as it exists within Py_Initialize(). So there is no hacked sys.path[0] equal to the script dir. And since the path hacks in site.py haven't happened yet either, we have an incomplete sys.path at that point. > > added to sys). How about prepending the single directory sys.executable > > to sys.path during Py_Initialize()? And demanding that modules > > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > > be placed there. > > On Unix, this is a bin directory and it is strongly discouraged to put > non-program files there. Ok, point taken. > Is the full DLL path available at any point? This would certainly be > a good starting point -- especially when the DLL is loaded implicitly > as the result of some COM operation. I don't know about loading by COM, but if it is a file, its absolute path is reliably known in sys, the code is identical to that currently used for sys.executable (on Windows), and I have a patch if you want. JimA's conjecture: It is currently impossible to ship a Python app which can not be damaged by the installation of a second Python app without using a hacked custom binary. Jim Ahlstrom From bwarsaw at cnri.reston.va.us Wed Sep 15 20:41:26 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 14:41:26 -0400 (EDT) Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <1274705689-31068526@hypernet.com> Message-ID: <14303.59478.806707.880694@anthem.cnri.reston.va.us> >>>>> "Gordo" == Gordon McMillan writes: Gordo> Jim Ahlstrom wtoe: >> "Guido van Rossum" wrote: >> --> The solution: Gordo> Did the dev-list miss something? The last I see is Barry's Gordo> post. I have a suspicion that python.org lost some email yesterday. We had a period of time where mail simply stopped getting delivered (thank you Solaris patch manager) and it took me a little while to realize that things weren't working correctly. Since there's nothing unexpected in the mail queue now, all I can say is that if you didn't get it by now, you ain't gonna. However, everything seemed to make it into the archives, so Guido's message is available at: http://www.python.org/pipermail/python-dev/1999-September/000880.html -Barry From guido at CNRI.Reston.VA.US Wed Sep 15 20:43:56 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 14:43:56 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Wed, 15 Sep 1999 10:00:05 PDT." References: Message-ID: <199909151843.OAA18364@eric.cnri.reston.va.us> [David Ascher] > Guido, maybe it would make sense to explain the need for a license change. > Is my understanding correct that the occasion for the license change is > that the copyright is now clearly shifting to CNRI, and as a result CNRI > has to forge a license? (BTW, I thought *you* had the copyright transfer > from CWI, not CNRI). Correct on both counts. CWI owns the copyright on old Python versions through Python 1.2. I have personally obtained non-exclusive rights to these from CWI. CNRI, by nature of my employment contract, has the copyright on newer versions. CNRI feels the need to protect its intellectual property rights. It feels that the old Python license, even with CNRI added, does not adequately protect CNRI against certain (unlikely) events -- hence the desire to draft a new license. CNRI understands that open source (and now Open Source -- the OSI board has approved the old Python license!) like Python requires different licensing terms than a typical product developed solely by CNRI. I think that the main problem is that CNRI's understanding of what truly constritutes open source is limited, and that my own understanding of legal issues is limited, so that the negotiations with CNRI's legal department (which is headed by CNRI's director) often turn in their favor. I hereby withdraw the posted license. There still is the need for a new license, but we need to go back to the drawing board for it. The CWI ownership of much of the code probably means that the license as it stands doesn't hold anyway. I also think that the Python consortium has a say in the license discussion -- the consortium agreement actually discusses the ownership of intellectual property produced by/for the consortium at some length. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw at cnri.reston.va.us Wed Sep 15 20:46:44 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 14:46:44 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <14303.59796.4588.287265@anthem.cnri.reston.va.us> JF> Zope products are python packages installed as sub-packages of JF> the Zope 'Products' packages. Products are generally JF> self-contained. If they need anything that's not part of JF> standard Python or standard Zope, they need to include it or JF> install what they need in a sub-package of another Zope JF> package, 'Shared'. Jim, from your description, it sounds like Zope should be putting both the `Shared' and possibly `Products' directories on sys.path. Maybe it does and I'm still missing your point. However if they are not, then wouldn't this solve your problem? A user would drop the kjParsing directory into Shared, and then all the import kjParsing.kjParser statements would Just Work. -Barry From mal at lemburg.com Wed Sep 15 19:02:20 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 19:02:20 +0200 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> Message-ID: <37DFD11C.712653BA@lemburg.com> Guido van Rossum wrote: > > It seems to me this totally solves Jim Fulton's and Marc's > > problem and makes "__" unnecessary. You just install zope > > and mx in zopedir, perform the above, and presto you have a new > > private name space where you can control all your names. But > > there must be some problem here I haven't thought of. > > I think no simple solution that *I* can come up with will satisfy > JimF's and Marc's desire for obscurity :-) Never mind, I'll use an imputil.py based approach to get relative imports to work in my packages. That is when I get imputil.py to work... it doesn't seem to be quite there yet (or I'm using an old version). BTW, I'm 100% behind you guys if you choose to reimplement Python's import mechanism in Python using a similar approach as the one Greg implemented in imputil. Should make everybody happy: those who want obscure syntactic add-ons and others with a taste for zlib'ed packaged byte code, plus those VMS freaks ;-) Perhaps we should start a new thread on that topic... Still needed are: ? Python level APIs for the platform specific magic on Win32 and Macs (OS/2, BeOS ?), e.g. access to the Windows registry and the Mac forks ? Patches to make the DirectoryImporter 100% backward compatible Greg's imputil.py can be found at: http://www.lyra.org/greg/small/ The trick would then be to install an application specific importer in the setup module Spam_path or MyAppSetup which then takes care of all the rest... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Wed Sep 15 21:00:41 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 15:00:41 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <14303.59478.806707.880694@anthem.cnri.reston.va.us> Message-ID: <1274700606-31374305@hypernet.com> Hmm, I'm suspicious of the fact that no message from Barry Warsaw ever gets "lost". Stalin got started by being in charge of the Kremlin's telephone system, you know... > >>>>> "Gordo" == Gordon McMillan writes: > > Gordo> Jim Ahlstrom wtoe: > > >> "Guido van Rossum" wrote: > >> --> The solution: > > Gordo> Did the dev-list miss something? The last I see is > Barry's Gordo> post. > > I have a suspicion that python.org lost some email yesterday. > > We had a period of time where mail simply stopped getting > delivered (thank you Solaris patch manager) and it took me a > little while to realize that things weren't working correctly. > Since there's nothing unexpected in the mail queue now, all I can > say is that if you didn't get it by now, you ain't gonna. > > However, everything seemed to make it into the archives, so > Guido's message is available at: > > http://www.python.org/pipermail/python-dev/1999-September/000880. > html > > -Barry > > _______________________________________________ > Python-Dev maillist - Python-Dev at python.org > http://www.python.org/mailman/listinfo/python-dev - Gordon From jim at interet.com Wed Sep 15 21:25:24 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 15 Sep 1999 15:25:24 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <37DFF2A4.DF26A22B@interet.com> Jim Fulton wrote: > In this context, consider the following concrete, though fictional > example. > ............ The classical solution would be to add "Shared" to sys.path. But how to do this? And what if another installation changes it? > I'd be curious to hear how folks would solve this problem. 1) Ship a custom binary interpreter with critical modules frozen in, and which sets a custom sys.path before anything else happens. So you are not dependent at all on PYTHONPATH. Currently this works. 2) Add a new command line option which sets sys.path and always use it to start your app. It lists only critical zope directories so zope has its own private name space. 3) Add your own import hooks using imputil.py and boot it as in (1). Better if it were part of Python. 4) Use a new Python.pyl file which has seeking from the end and thus allows concattenation of multiple *.pyl. Each package is a foo.pyl, bar.pyl, and to distribute an app, you concatenate packages. The implied PYTHONPATH in each *.pyl file is "." except perhaps for the Python lib which is "Lib". Boot it as in (3). > Personally, I'd like the problem to go away. :) I have the same problem, so me too. > I'd like the Python > import rules to change to make this solvable without import hooks or > path hacking. I can think of two ways to approach this: > > - Relative parent imports: > > import __.kjParsing.kjParser OK by me, but doesn't solve the Grand Problem. > - Gradually less local searches. I am currently bewildered by Python's import mechanism, and this may push me into brain damage. Well, *more* brain damage. Jim Ahlstrom From jim at digicool.com Wed Sep 15 21:34:08 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 15:34:08 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> Message-ID: <37DFF4B0.15390199@digicool.com> "Barry A. Warsaw" wrote: > > JF> Zope products are python packages installed as sub-packages of > JF> the Zope 'Products' packages. Products are generally > JF> self-contained. If they need anything that's not part of > JF> standard Python or standard Zope, they need to include it or > JF> install what they need in a sub-package of another Zope > JF> package, 'Shared'. > > Jim, from your description, it sounds like Zope should be putting both > the `Shared' and possibly `Products' directories on sys.path. It actually does now, although that is bad. What if some future python distribution has a Products package? Or a Products.py? A product is actually a sub-package of Products. Even if Products was not a package, the Product would be and gadfly and kjParsong would be sub-packages. There is a similar issue with Shared, since Shared software is organized by organization, as in 'Shared.DC.ZRDB'. > Maybe > it does and I'm still missing your point. The point is that stuffing arbitrary things into a single global name space is bad. That's why packages were invented. Unless packages can be effectively nested, you haven't gained anything but time. > However if they are not, then wouldn't this solve your problem? No, see above. > A > user would drop the kjParsing directory into Shared, and then all the > import kjParsing.kjParser statements would Just Work. But then they'd be dumping kjParsing into the global package namespace. Eventually, there will be collisions. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at digicool.com Wed Sep 15 21:59:10 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 15:59:10 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <37DFF2A4.DF26A22B@interet.com> Message-ID: <37DFFA8E.FB73A006@digicool.com> "James C. Ahlstrom" wrote: > > Jim Fulton wrote: > > In this context, consider the following concrete, though fictional > > example. > > ............ > > The classical solution would be to add "Shared" to sys.path. But > how to do this? And what if another installation changes it? This doesn't work, because 'Shared' has sub-packages by organization. > > I'd be curious to hear how folks would solve this problem. > > 1) Ship a custom binary interpreter with critical modules frozen in, > and which sets a custom sys.path before anything else happens. So > you are not dependent at all on PYTHONPATH. Currently this works. This is what we do for our binary distribtions, but it doesn't do anything for third-party products added to Zope. > 2) Add a new command line option which sets sys.path and always use > it to start your app. It lists only critical zope directories so > zope has its own private name space. We have a start script that takes care of this. This gets Zope's top-level names (of which there are too many already) into the path. This doesn't help with third-party products added to Zope. > 3) Add your own import hooks using imputil.py and boot it as > in (1). But imputil has a global effect. I suppose I could implement what I want with imputil or by otherwise creating my own importer using the standard import hooks. I suspect that's what I'll have to do eventually. > Better if it were part of Python. > 4) Use a new Python.pyl file which has seeking from the end and > thus allows concattenation of multiple *.pyl. Each package is > a foo.pyl, bar.pyl, and to distribute an app, you concatenate > packages. The implied PYTHONPATH in each *.pyl file is "." > except perhaps for the Python lib which is "Lib". Boot it as > in (3). I'm not looking for a way to replace the current package scheme. Also, I'm not just looking at an app, but at an app that contains apps (aka components). > > Personally, I'd like the problem to go away. :) > > I have the same problem, so me too. > > > I'd like the Python > > import rules to change to make this solvable without import hooks or > > path hacking. I can think of two ways to approach this: > > > > - Relative parent imports: > > > > import __.kjParsing.kjParser > > OK by me, but doesn't solve the Grand Problem. > > > - Gradually less local searches. > > I am currently bewildered by Python's import mechanism, and > this may push me into brain damage. Well, *more* brain damage. I don't see what's so complicated. You just use a gradually widening search, rather than an immediately widening search. I don't think that this is any more complicated than the current rule. I actually prefer the second apprach because it doesn't require any code changes to take advantage of it. I may implement it for Zope if nothing is done in standard Python. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mal at lemburg.com Wed Sep 15 22:01:01 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 15 Sep 1999 22:01:01 +0200 Subject: [Python-Dev] Towards a Python based import scheme Message-ID: <37DFFAFD.72F6A806@lemburg.com> Back again... To get a little more constructive I've started hacking away on Greg Stein's imputil.py to make it work with my DateTime package. The DateTime packages does a lot of from...import... and intra-package imports, plus it loads a shared lib as extension. The original version of imputil I fetched from Greg's page did work out of the box (from...import... hassles) and obviously did not support in-package shared libs. I've added both features so that the test script in DateTime can run successfully. Things that remain are: ? the win32 registry stuff (needs C code) ? the Mac fork stuff (needs C code) ? a working __path__ implementation (is anyone using this attribute which only is available in packages ?) ? probably a whole bunch of other quirks ? some speedups (there currently are too many stat()s) Please give it a try: http://starship.skyport.net/~lemburg/imputil.py in color: http://starship.skyport.net/~lemburg/imputil.py.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Wed Sep 15 22:22:34 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 16:22:34 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DFF4B0.15390199@digicool.com> Message-ID: <1274695692-31669872@hypernet.com> Jim Fulton wrote: > It actually does now, although that is bad. What if some future > python distribution has a Products package? Or a Products.py? Name it zopeProducts. Or Zope/Products and Zope/Shared. > A product is actually a sub-package of Products. Even if Products > was not a package, the Product would be and gadfly and kjParsong > would be sub-packages. There is a similar issue with Shared, > since Shared software is organized by organization, as in > 'Shared.DC.ZRDB'. See below. > The point is that stuffing arbitrary things into a single global > name space is bad. That's why packages were invented. Unless > packages can be effectively nested, you haven't gained anything > but time. Packages nest effectively. Subpackages just don't masquerade as top level packages. [Barry] > > A > > user would drop the kjParsing directory into Shared, and then > > all the import kjParsing.kjParser statements would Just Work. > > But then they'd be dumping kjParsing into the global package > namespace. Eventually, there will be collisions. You are trying to do two incompatible things at once. You are trying to be both a standalone application, and a Python installation. You can be both, just not at the same time. To be a standalone application, you can use tricks that Jim Ahlstrom pointed out, or tricks from my Installer. The major point being that sys.path has nothing to do with the PYTHONPATH environment variable. You now have an installation where you can be assured of no collisions, because you control it completely. Of course, that will make liars out of the people at your company who claim you can mix and match pieces of Zope and your own stuff as you please. People who want to mix and match need Zope to play by normal Python rules - and that means packages are installed at the top level. Collisions happen, but your audience is people who *want* to do things differently and *want* control of their installations. By creating "self contained" packages and putting them in a normal Python installation, you're just pushing your headache off on the user. You're co-opting his installation. Yeah, now you have two versions, but they're 99% the same. Build the normal Python one first and distribute it to the hackers. Then install some import and startup hooks, and distribute that to your "users". I, for one, still use Bobo, because Zope doesn't play fair and I have no need or desire for yet another Python installation. - Gordon From gmcm at hypernet.com Wed Sep 15 22:43:27 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 15 Sep 1999 16:43:27 -0400 Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <1274694439-31745225@hypernet.com> Marc-Andre wrote: > To get a little more constructive I've started hacking away on > Greg Stein's imputil.py to make it work with my DateTime package. > http://starship.skyport.net/~lemburg/imputil.py You (and Greg) are missing a rather important patch I submitted to Greg a long time ago (around line 62): def _reload_hook(self, module): # gmcm - Hmmm, reloading of a module may or may not be impossible, # (depending on the importer), but at least we can # look to see if it's ours to reload: if hasattr(module, '__importer__'): if getattr(module, '__importer__') == self: raise SystemError, "reload not yet implemented" return self.__chain_reload(module) - Gordon From jim at digicool.com Wed Sep 15 22:44:31 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 16:44:31 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274695692-31669872@hypernet.com> Message-ID: <37E0052F.A899E985@digicool.com> Gordon McMillan wrote: > > Jim Fulton wrote: > (snip) > Packages nest effectively. Subpackages just don't > masquerade as top level packages. They do currently when imported from the containing package. > [Barry] > > > A > > > user would drop the kjParsing directory into Shared, and then > > > all the import kjParsing.kjParser statements would Just Work. > > > > But then they'd be dumping kjParsing into the global package > > namespace. Eventually, there will be collisions. > > You are trying to do two incompatible things at once. You are > trying to be both a standalone application, and a Python > installation. You can be both, just not at the same time. We don't try to be a Python installation. We try to be a Python application that is a platform for Zope applications. > To be a standalone application, you can use tricks that Jim > Ahlstrom pointed out, or tricks from my Installer. The major > point being that sys.path has nothing to do with the > PYTHONPATH environment variable. You now have an > installation where you can be assured of no collisions, > because you control it completely. > > Of course, that will make liars out of the people at your > company who claim you can mix and match pieces of Zope > and your own stuff as you please. Zope reuses *alot* of code. I think Zope has a pretty good track record of working with existing code. > People who want to mix > and match need Zope to play by normal Python rules - and > that means packages are installed at the top level. Sorry, I just don't buy that. > Collisions > happen, but your audience is people who *want* to do things > differently and *want* control of their installations. Actually, many Zope user's come to Python by way of Zope. They want to be able to install Zope products without worrying about what their zope or python installation looks like. Zope developers want to be able to have their products drop into a Zope installation easily. > By creating "self contained" packages and putting them in a > normal Python installation, you're just pushing your headache > off on the user. You're co-opting his installation. No, putting packages in the python path coops their installation. Putting Zope-specific products that happen to include reusable Python packages in a Zope-specific package leaves their installation alone. > Yeah, now you have two versions, but they're 99% the same. > Build the normal Python one first and distribute it to the > hackers. Then install some import and startup hooks, and > distribute that to your "users". This is exactly what I want to prevent. I'd like package developers to develop their packages once without making them use esoteric import hooks. OTOH I'd like to be able to *use* their packages wherever I damn well please. I don't to be forced to put their packages in the python path. > I, for one, still use Bobo, because Zope doesn't play fair In what way does Zope not play fair? > and I > have no need or desire for yet another Python installation. Cool. You don't need to use another Python installation to use Zope. In fact, for development, we use a totally standard Python distribution. Developers here are not *allowed* to add anything special to the Python installations on the development machines here because we want to *make sure* that we can run with stock python installations. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From bwarsaw at cnri.reston.va.us Wed Sep 15 23:55:34 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 17:55:34 -0400 (EDT) Subject: [Python-Dev] Re: Path hacking References: <14303.59478.806707.880694@anthem.cnri.reston.va.us> <1274700606-31374305@hypernet.com> Message-ID: <14304.5590.405325.369903@anthem.cnri.reston.va.us> >>>>> "Gordo" == Gordon McMillan writes: Gordo> Hmm, I'm suspicious of the fact that no message from Barry Gordo> Warsaw ever gets "lost". Gordo> Stalin got started by being in charge of the Kremlin's Gordo> telephone system, you know... Well, I managed to get rid of Ken so my world domination plan is right on schedule! guido-may-be-benevolent-but-you-can-bet-i-won't-be-ly y'rs, -Barry From bwarsaw at cnri.reston.va.us Thu Sep 16 00:07:34 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Wed, 15 Sep 1999 18:07:34 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> Message-ID: <14304.6310.20571.487708@anthem.cnri.reston.va.us> >>>>> "JF" == Jim Fulton writes: JF> It actually does now, although that is bad. What if some JF> future python distribution has a Products package? Or a JF> Products.py? Wouldn't collide, because in my scheme, Shared isn't a package, it's a directory on sys.path. Everything inside Shared is a top-level package. You'd never "import Shared.DC.ZRDB" but instead you would "import DC.ZRDB". >> A user would drop the kjParsing directory into Shared, and then >> all the import kjParsing.kjParser statements would Just Work. JF> But then they'd be dumping kjParsing into the global package JF> namespace. Eventually, there will be collisions. Do you really think so? Someone else is going to write a kjParsing module? I think we're safe from this for the forseeable future through convention and peer pressure. Or is the concern that there will be two Products which use different versions of kjParsing? I'm also not concerned here because if my product needs a very specific version of kjParsing I can't see any way around shipping my own version, subpackaging it, and using an absolute package name to ensure that I'm getting at exactly the one I need. This is the approach Mailman takes. -Barry From guido at CNRI.Reston.VA.US Thu Sep 16 00:35:20 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 18:35:20 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Wed, 15 Sep 1999 14:29:38 EDT." <37DFE592.7FD384A2@interet.com> References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> Message-ID: <199909152235.SAA18693@eric.cnri.reston.va.us> [me] > > But the intention here is for the customization to be application > > specific (hence the Spam in the name). sitecustomize doesn't know > > whethere I need the Mailman or the Knowbot root added to my path. [JimA] > Ah, you have multiple scripts in one directory and multiple > Foo_path, Bar_path etc. I was thinking with my Windows head. > A commercial Windows app generally has its own exclusive > install directory, so I was thinking single directory so a single > sitecustomize.py. > > > Or do you mean to imply that we can do this with zero text added to > > the script, by simply dropping an appropriate sitecustomize.py in the > > script dir? > > Yes, that is exactly what I was thinking. > > > Unfortunately this does currently *not* work, because > > sys.path[0] is added after Py_Initialize() is run. > > Yikes! That kills using sitecustomize.py. Your Spam_path > still works because it is imported later, but requires an > import in each Python main script just as you said. Not too bad (who cares about one more line of boilerplate...). > Even worse, it means that exceptions.py and site.py can not > be found at all except using the normal PYTHONPATH, and > putting their path in Spam_path will *not* work. Why would you want your own exceptions.py and site.py? > > > > Because the script's directory is first on the default path, the Spam > > > > scripts will pick up Spam_path without any help from $PYTHONPATH. > > > > > > Hmmm. Is this really true? Nothing else, for example the registry, can > > > change sys.path[0]? Ever? Please say yes. > > > > Yes. (The registry can add module-specific paths, which will be > > searched before sys.path is even looked at, but this is only for > > specific modules. It cannot insert a general directory that is > > searched.) The only way this can fail is if an embedding app fails to > > call PySys_SetArgv(). > > Oh dear, I think I heard no instead of yes. Are you saying that if > someone else installs a Python app on my customer's machine after I do, > and sets a registry entry which sayes to use c:/other/path/to/site.py > for site.py (as he may very well want to do), then if my Python program > depends on getting my copy of site.py from my directory, it will then > use the other copy instead and may very well fail? Again - why would anyone register their own site.py? > > In any case, that's a separate issue -- I > > agree that if sys.path[0] is '' (as it often is) it's better for > > site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize > > it (and everything else on the path) so that it will still work if the > > app does a chdir(). > > Point on the curve: Windows apps generally start from an icon > which contains their path and current working directory, and > these are generally different. So a Windows app in general will > *never* have had a getcwd() equal to the path of either the > binary interpreter or the Python main script. You're lucky. It turns out that on Windows, under those circumstances at least, sys.path[0] is the absolute pathname of the directory. You only see '' if sys.argv[0] doesn't have any pathname information; that's only possible if the script *does* live in the current directory. > > > The files exceptions.py and site.py must be in all the bin > > > directories as well as sitecustomize.py because they are > > > automatically imported in Py_Initialize(). > > > > Yes. > > Well, *no* right? This fails unless the bin directories are in > fact on PYTHONPATH. The only way to get exceptions.py is by using > sys.path as it exists within Py_Initialize(). So there is no > hacked sys.path[0] equal to the script dir. And since the > path hacks in site.py haven't happened yet either, we have > an incomplete sys.path at that point. Sorry, I've lost track of what we were after here. Indeed the scripts' directory (which I presume you meant by the bin directory) indeed doesn't occur in sys.path until after Py_Initialize() has run. > > > added to sys). How about prepending the single directory sys.executable > > > to sys.path during Py_Initialize()? And demanding that modules > > > like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] > > > be placed there. > > > > On Unix, this is a bin directory and it is strongly discouraged to put > > non-program files there. > > Ok, point taken. > > > Is the full DLL path available at any point? This would certainly be > > a good starting point -- especially when the DLL is loaded implicitly > > as the result of some COM operation. > > I don't know about loading by COM, but if it is a file, its absolute > path is reliably known in sys, the code is identical to that currently > used for sys.executable (on Windows), and I have a patch if you want. I presume using GetModuleFileName()? Please send me the patch! > JimA's conjecture: It is currently impossible to > ship a Python app which can not be damaged by the installation of a > second Python app without using a hacked custom binary. Sounds right. All tricks to make the app unique require using a different registry key, which requires a change to the DLL. However, you can do this without recompiling! The version string is used is embedded in a resource, so you can patch it using some kind of resource editor. Mark Hammond planned it this way! --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at digicool.com Thu Sep 16 00:49:38 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 15 Sep 1999 18:49:38 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> Message-ID: <37E02282.BDF5FF21@digicool.com> "Barry A. Warsaw" wrote: > > >>>>> "JF" == Jim Fulton writes: > > JF> It actually does now, although that is bad. What if some > JF> future python distribution has a Products package? Or a > JF> Products.py? > > Wouldn't collide, because in my scheme, Shared isn't a package, it's a > directory on sys.path. Everything inside Shared is a top-level > package. You'd never "import Shared.DC.ZRDB" but instead you would > "import DC.ZRDB". I wasn't talking about your scheme. I was talking about how I thought it should be. > >> A user would drop the kjParsing directory into Shared, and then > >> all the import kjParsing.kjParser statements would Just Work. > > JF> But then they'd be dumping kjParsing into the global package > JF> namespace. Eventually, there will be collisions. > > Do you really think so? Yes. > Someone else is going to write a kjParsing > module? No, but sooner or later, two people are going to write two "Parsing" packages, or two "Oracle" packages or two .... Are you suggesting that package names should begin with some hash value to assure that they are unique? (What does kj stand for anyway? :) I guess we could use some form of GUID to be really safe ..... > I think we're safe from this for the forseeable future > through convention and peer pressure. 8^o (I feel like I should respond to this in some way but .... I just can't understand this kind of argument and don't know how to frame a response.) > Or is the concern that there will be two Products which use different > versions of kjParsing? No, at least not primarily. > I'm also not concerned here because if my > product needs a very specific version of kjParsing I can't see any way > around shipping my own version, subpackaging it, and using an absolute > package name to ensure that I'm getting at exactly the one I need. But even in this case, if you used a relative import, you *would* get the specific version you want. I give up. I'll just have to write some import hook to get the behavior I want in Zope. I will probably implement the second alternative I wrote about in my earlier document. I'll get Python to look at intermediate package paths before it looks at the main path. That way, any package that works as a top-level package will also work as a sub-package without modification. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From mhammond at skippinet.com.au Thu Sep 16 03:20:10 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 16 Sep 1999 11:20:10 +1000 Subject: [Python-Dev] License cleanup In-Reply-To: <199909151843.OAA18364@eric.cnri.reston.va.us> Message-ID: <004601beffe1$9edb01b0$0801a8c0@bobcat> > CNRI understands that open source (and now Open Source -- the OSI > board has approved the old Python license!) like Python requires > different licensing terms than a typical product developed solely by > CNRI. Actually, this is a good point. I know the issue of payment may get in the way, but it would make sense to have any future proposed licenses reviewed by a lawyer "on our side" - eg, someone whose mandate is to give a legal opinion on the risks and liabilities of the _user_ of the license. Obviously the CNRI lawers are protecting their (ie, CNRI's) interests, and everyone on this group is concerned about their own (ie, personally, their company, or companies they wish to introduct Python into) interests. If the legal jibberish can't be removed (which is likely with lawyers involved) I know I would personally feel much more comfortable with a legal opinion covering my interests.. But as I said, who will pay? If nothing else, we should ensure the OSI approves of the new license... Or maybe we can convince CNRI there is real and serious concern, and they could pay for an external IP lawyer? Mark. From gstein at lyra.org Thu Sep 16 03:37:17 1999 From: gstein at lyra.org (Greg Stein) Date: Wed, 15 Sep 1999 18:37:17 -0700 Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> Message-ID: <37E049CD.6B64DE00@lyra.org> Mark Hammond wrote: >... > Actually, this is a good point. I know the issue of payment may get > in the way, but it would make sense to have any future proposed > licenses reviewed by a lawyer "on our side" - eg, someone whose > mandate is to give a legal opinion on the risks and liabilities of the > _user_ of the license. Interesting thought! > Obviously the CNRI lawers are protecting their > (ie, CNRI's) interests, and everyone on this group is concerned about > their own (ie, personally, their company, or companies they wish to > introduct Python into) interests. Because of these varied interests, I don't think a review by a particular lawyer will be greatly helpful. The question will still remain: "did the lawyer review it from [my/our/company] perspective?" This will lead people back into the same review cycle. >... > But as I said, who will pay? If nothing else, we should ensure the > OSI approves of the new license... Or maybe we can convince CNRI > there is real and serious concern, and they could pay for an external > IP lawyer? Definitely having an OSI certification will be great (cool stuff on the cert for the existing license!). Having Bruce Perens review the license would also be a great boon (see www.perens.com for some of his writings; also see http://perens.com/Termination.html specifically). Licenses are a tough issue. I had to go through this entire morass when deciding what to do with mod_dav. There are a lot of varieties and issues and stuff to cover. I've read a bunch of license (not to mention a bazillion legal documents during the eShop/acquisition days). Not always exciting reading :-), but usually quite interesting. At this point, I think it is a great thing that CNRI is reviewing the license. Unfortunately, the license wasn't as non-controversial as it was thought to be :-(. I'm more than happy to wait and see where they go with the license. Cheers, -g -- Greg Stein, http://www.lyra.org/ From guido at CNRI.Reston.VA.US Thu Sep 16 05:32:18 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 15 Sep 1999 23:32:18 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Wed, 15 Sep 1999 14:10:17 EDT." <37DFE109.96C652B3@digicool.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> Message-ID: <199909160332.XAA19278@eric.cnri.reston.va.us> I don't like the pressure to allow relative imports and I don't like the solutions that those who insist on having it propose (they are all writing their own import hooks). The reasoning seems to be something like this: (a) If my package needs non-standard package X, I can't require that X be installed as a top-level package, because there might be another unrelated package also called X; therefore I need to install a copy of package X as a subpackage of my own package. (b) Because any non-standard package X now may be installed at the toplevel or as a subpackage of some other package, X must use relative imports exclusively to access subparts of itself. I find (b) unacceptable for a number of reasons, that have all been explained by others: duplicate installs (and hence possible imports) of the same package as a subpackage of various other packages, the requirement to be religious in the use of relative imports, the ugliness of the __ notation (I do *not* look forward to explaining that to 12-year-olds in the context of CP4E). I think that the fear expressed in (a) is an overreaction to some recent actual top-level name clashes. This was a real pain, but notice that it happened because several popular module collections weren't packagized at all! In the Java world, the right solution exist: a better top-level naming scheme. I forget what the exact rules are, but I seem to recall that Java has something of the same relative import rules af Python currently has: you can import a class from the same leaf package by its simple name, or you can import a class by its full name. As Tim explains, the occasional renames are easily handled with a global substitute in a smart editor. I don't want to do the same thing as Java and require everybody to use their reversed domain name as their package name. Of course you are free to do so: there are currently no toplevel modules or packages that I know of named org, com, edu, gov, mil or net. Two letter domains are all free except for re=Reunion (and ni=Nicaragua, but ni is expired). Oops, mx=Mexico! Well, Python package semantics are slightly different than Java semantics in that shared ownership of a subpackage of (e.g. :-) mx is harder in Python than in Java (Python requires that they are installed in the same directory; Java doesn't). In any case the proper thing to do is to pick a good unique toplevel name. Aaron Watters did a great job with kj (not a country code either!); Marc Andre also has the right spirit with mx. CNRI will probably use cnri as the toplevel package name for CNRI-developed packages that aren't distributed together with Python. If there are still worries about name choices, I volunteer to run a toplevel name registry. This has worked well e.g. for Apple where application developers had to pick a unique 4-letter code; certainly better than the Windows world where conflicting 3-letter extension definitions are common. In the Python world, I would recommend longer toplevel names, so it's more likely that you can use your company name (e.g. pythonware). Note that I still resist the urge to start renaming all the existing standard modules -- too much breakage. This will happen in 2.0. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one at email.msn.com Thu Sep 16 05:38:22 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 15 Sep 1999 23:38:22 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37DF65DB.46F0191A@lemburg.com> Message-ID: <000101befff4$ecc9bde0$712d153f@tim> Sure is a lot of traffic on this -- can't we go back to hating the new license ? [MAL] > ... > Sticking this technology into the import statement is really only > a more elegant approach. Nothing more. The whole thing already > works using the current Python implementation. Your point there has been made many times -- I'm not missing it. My point has also been made many times, so one more can't hurt : it still looks to me like bad practice, so I don't *want* (more, actively oppose) it to be made easier to spell. >> + JimF and MAL's shared notion that they're the only ones making significant >> use of packages is myopic. > Oh, c'mon Tim, we never did say that. We only mentioned having used > package for quite a while. This includes having seen all the different > pitfalls they have to offer. Well, JimF was laying it on a bit thick in two msgs, and sucked you in as his implicit conspirator. My apologies if you feel tarred unfairly. Yet you still can't resist implying that only you two are aware of the pitfalls . >> [on Dragon's "relentlessly straightforward" import policies] > This is perfectly ok if you're just using your own code, but it > gets a mess when third-party packages are involved simply because > you can't be sure they don't use import hooks, local imports or > hacks to __path__ etc. If you still want them to be useable, you'll > have to go down and dive into their structure. This takes time, is > error prone and not necessarily fun. While true, it's much safer to install a "cheating" package at top level, since the cheats presumably *assume* that's where the package is to be run from. If someone distributes a package that does not run correctly from top level, that's one hopeless package. Else it does run correctly from top level, and then the only person with a problem here is one who insists on running it from some embedded location. You then buy the need to dive into their code very carefully indeed -- relative imports don't save you from the huge variety of "clever tricks" they *may* be playing with hooks and path hacks. But note that, unlike Jim (& perhaps you too), I have almost no fear of top-level name collisions. If we do nothing about that, it will be years before it becomes a real problem; and several months before that , we can flock to Java's hokey but effective reverse-domain scheme (which I suspect you're already in favor of -- fine by me!). > Also, I don't know how "explicitly, long-windedly and straightforwardly" > writing module imports is any different from using relative imports. Hugely and profoundly different, as your example shows: > An example: > > mx > .DateTime > .ODBC.Windows > > In ODBC.Windows I would write: > > import __.__.DateTime > > which is just as explicit as writing > > import mx.DateTime > > except that the information about the top-level hierarchy is *not* > included in the import information. Exactly: now instead of a bulletproof absolute import in a self-contained importer, whose precise meaning is clear from casual static inspection, you've got a relative one whose meaning depends vitally on where the importer happens to live. As code moves around, the import may fail to work when you're lucky, or pick up an unintended DateTime package when you're not. The semantics of the import are indeed relative now: you see the opportunity to type three additional characters as a strength, and I see nothing but headaches masquerading as convenience. As further consequence, simple "search-and-replace" strategies are rendered at best useless. That is, a simple grep can no longer find all & only references to mx's flavor of DateTime: it picks up all references to all DateTime guys, with no way short of a structure-aware tree crawler to sort them out again. Complexity spreads. Of course, in the end, I lose no matter what. If *everyone* used dirt-dumb fully-qualified imports, all hassles with shifting package structure could be solved with a text editor in seconds. But that's not whiz-bang enough, so people *will* play endlessly dubious tricks with hooks, paths, relative imports, and metapackages too once Gordon gets around to them . Having an explicit way to spell relative imports would make my life easier because at least those are statically recognizable for what they are. I don't want them and won't use them, but it could ease the pain others will cause by using them. That's not enough to make me in favor of them, though; just enough so I won't pout if they do go in. > Anyway, I'm not too inclined continuing this discussion anymore. > I would never have thought that such a nifty little patch would > have stirred up so much negative response. I'll just hack up my > very own import mechanism using imputil and be done. So there! Don't feel picked on, M-A! Opposition is normal and healthy; and the last idea I got anywhere with was list.pop(). and-even-then-jimf-tried-to-stick-a-default-on-it-ly y'rs - tim From da at ski.org Thu Sep 16 06:19:32 1999 From: da at ski.org (David Ascher) Date: Wed, 15 Sep 1999 21:19:32 -0700 (Pacific Daylight Time) Subject: [Python-Dev] License cleanup In-Reply-To: <004601beffe1$9edb01b0$0801a8c0@bobcat> Message-ID: On Thu, 16 Sep 1999, Mark Hammond wrote: > But as I said, who will pay? If nothing else, we should ensure the > OSI approves of the new license... Or maybe we can convince CNRI > there is real and serious concern, and they could pay for an external > IP lawyer? Eric Raymond's wife is a lawyer, I believe =). From mhammond at skippinet.com.au Thu Sep 16 06:28:19 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 16 Sep 1999 14:28:19 +1000 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <000101befff4$ecc9bde0$712d153f@tim> Message-ID: <004e01befffb$e83a6d90$0801a8c0@bobcat> > Don't feel picked on, M-A! Opposition is normal and healthy; As a (hopefully welcome) tangent to this discussion, I am currently reading "Dynamics of Software Development" by Jim McCarty. He has an interesting section on exactly this. I will quote it verbatim (typing it in, so forgive typos). I present it without comment other than: It appears we are a healthy team! Mark. The Problem of Creativity: Creativity in a group is always limited by the groups defensiveness, and some amount of defensiveness is healthy. In the healthy team, then, no matter how highly valued creativity is ostensibly, change beyond a certain normative quantity or degree will be resisted. And even change within bounds will be accepted only insofar as it can seem to fall under the rubric of "improvement". Change must seem to build on the previously known and accepted reality. Even if there are only two steps involved in a change, with the second building on the first, in a single reality cycle - single product development cycle for us - the change is likely to be rejected by the healthy team. What you need for truly creative change, then, is an environment that transcends good health, an environment that not only accepts a continuum of change, which is normal, but one that positively engenders, nurtures and propels forward wholly new dynamics. The transcendent organization values radical or revolutionary change and esteems utterly new modes of thought. It's possible for a team to be healthy and not particularly creative, but this state of affairs is not especially desirable. What is desirable is team fecundity [MH - had to look this up - "intellectually productive or inventive to a marked degree" - synonym of fertile], the radiating of the new and the original from the normal and the healthy. This kind of creativity requires a flexibility and a courage beyond the reach of most of us most of the time It's ironic and worth remembering that the healthier the team the more effective the resistance to revolutionary thought will be. This is a natural consequence of the teams overall effectiveness on every relevant domain. -- end of quote-- From jcw at equi4.com Thu Sep 16 11:48:33 1999 From: jcw at equi4.com (Jean-Claude Wippler) Date: Thu, 16 Sep 1999 11:48:33 +0200 Subject: [Python-Dev] License cleanup References: <199909151843.OAA18364@eric.cnri.reston.va.us> Message-ID: <37E0BCF1.24FD7431@equi4.com> Guido van Rossum wrote: > I hereby withdraw the posted license. There still is the need for a > new license, but we need to go back to the drawing board for it. One more consideration: some people may compare the "scariness" of the Python license against, say, the Tcl license - and choose accordingly. It's not even about content: seeing that new license, or the Perl licenses for that matter, it sends out a strong message IMO: you are entering the world of lawyers. Proceed with caution. Guard dogs. -- Jean-Claude From mal at lemburg.com Thu Sep 16 12:44:34 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 16 Sep 1999 12:44:34 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> Message-ID: <37E0CA12.1E24EBF2@lemburg.com> Guido van Rossum wrote: > > I don't like the pressure to allow relative imports and I don't like > the solutions that those who insist on having it propose (they are all > writing their own import hooks). I'm wasn't planning to write my own hook until now. I need this feature to be able to organize my package internal stuff swiftly and easily, plus to be able to possibly put it under new top-level packages. Obviously I seem to be more or less alone with this need, so I'll stop argueing for inclusion of "something like relative imports" in the distribution. > The reasoning seems to be something like this: > > (a) If my package needs non-standard package X, I can't require that X > be installed as a top-level package, because there might be another > unrelated package also called X; therefore I need to install a copy of > package X as a subpackage of my own package. > > (b) Because any non-standard package X now may be installed at the > toplevel or as a subpackage of some other package, X must use relative > imports exclusively to access subparts of itself. > > I find (b) unacceptable for a number of reasons, that have all been > explained by others: duplicate installs (and hence possible imports) > of the same package as a subpackage of various other packages, the > requirement to be religious in the use of relative imports, the > ugliness of the __ notation (I do *not* look forward to explaining > that to 12-year-olds in the context of CP4E). (b) is (if at all) a problem only to be taken into account by the author of package X. He may or may not use rel. imports. A 12-year old probably won't (but then: you never know ;). BTW, what is this CP4E thing you're talking about. If it's an syntax aware editor, I have a friend who is really interested in these things... could you send me an URL that I can send him ? > I think that the fear expressed in (a) is an overreaction to some > recent actual top-level name clashes. This was a real pain, but > notice that it happened because several popular module collections > weren't packagized at all! > > In the Java world, the right solution exist: a better top-level naming > scheme. I forget what the exact rules are, but I seem to recall that > Java has something of the same relative import rules af Python > currently has: you can import a class from the same leaf package by > its simple name, or you can import a class by its full name. As Tim > explains, the occasional renames are easily handled with a global > substitute in a smart editor. > > I don't want to do the same thing as Java and require everybody to use > their reversed domain name as their package name. Of course you are > free to do so: there are currently no toplevel modules or packages > that I know of named org, com, edu, gov, mil or net. Two letter > domains are all free except for re=Reunion (and ni=Nicaragua, but ni > is expired). Oops, mx=Mexico! Oh well... I guess they'll have to use 'mex' if they decide to go the reverse domain way ;-) > Well, Python package semantics are > slightly different than Java semantics in that shared ownership of a > subpackage of (e.g. :-) mx is harder in Python than in Java (Python > requires that they are installed in the same directory; Java doesn't). True, perhaps we should lighten this requirement a little when we recode the import mechanism in Python ? E.g. if a local import fails continue the search with the fully qualified name and only if that fails, restart using the local name. This would need some kind of fastpath cache to make the search reasonably fast though. > In any case the proper thing to do is to pick a good unique toplevel > name. Aaron Watters did a great job with kj (not a country code > either!); Marc Andre also has the right spirit with mx. CNRI will > probably use cnri as the toplevel package name for CNRI-developed > packages that aren't distributed together with Python. > > If there are still worries about name choices, I volunteer to run a > toplevel name registry. This has worked well e.g. for Apple where > application developers had to pick a unique 4-letter code; certainly > better than the Windows world where conflicting 3-letter extension > definitions are common. In the Python world, I would recommend longer > toplevel names, so it's more likely that you can use your company name > (e.g. pythonware). > > Note that I still resist the urge to start renaming all the existing > standard modules -- too much breakage. This will happen in 2.0. Wouldn't it suffice to just put them into one package, e.g. 'python.' ? BTW, as Tim argued: the breakage an easily be leveraged by using a smart editor... ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 106 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Thu Sep 16 12:56:01 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 16 Sep 1999 12:56:01 +0200 Subject: [Python-Dev] Relative Package Imports References: <000101befff4$ecc9bde0$712d153f@tim> Message-ID: <37E0CCC1.6EE4E599@lemburg.com> Tim Peters wrote: > > Sure is a lot of traffic on this -- can't we go back to hating the new > license ? The license was withdrawn... I would prefer starting to talk about imputil.py. > [MAL] > > Also, I don't know how "explicitly, long-windedly and straightforwardly" > > writing module imports is any different from using relative imports. > > Hugely and profoundly different, as your example shows: > > > An example: > > > > mx > > .DateTime > > .ODBC.Windows > > > > In ODBC.Windows I would write: > > > > import __.__.DateTime > > > > which is just as explicit as writing > > > > import mx.DateTime > > > > except that the information about the top-level hierarchy is *not* > > included in the import information. > > Exactly: now instead of a bulletproof absolute import in a self-contained > importer, whose precise meaning is clear from casual static inspection, Ah, that's a new idea ! Package local absolute names... wow, that sounds weird. > you've got a relative one whose meaning depends vitally on where the > importer happens to live. As code moves around, the import may fail to work > when you're lucky, or pick up an unintended DateTime package when you're > not. The semantics of the import are indeed relative now: you see the > opportunity to type three additional characters as a strength, and I > see nothing but headaches masquerading as convenience. Tim, the intention was being able to move the mx level around, e.g. renaming it, eliminating it by sticking DateTime and ODBC under some other higher level package (yes, you'll have to watch out here because the higher level package might define DateTime and ODBC too -- but it's not all that likely), etc. The buzz word here is 'relocatable'. > As further consequence, simple "search-and-replace" strategies are rendered > at best useless. That is, a simple grep can no longer find all & only > references to mx's flavor of DateTime: it picks up all references to all > DateTime guys, with no way short of a structure-aware tree crawler to sort > them out again. Complexity spreads. > > Of course, in the end, I lose no matter what. If *everyone* used dirt-dumb > fully-qualified imports, all hassles with shifting package structure could > be solved with a text editor in seconds. But that's not whiz-bang enough, > so people *will* play endlessly dubious tricks with hooks, paths, relative > imports, and metapackages too once Gordon gets around to them . > Having an explicit way to spell relative imports would make my life easier > because at least those are statically recognizable for what they are. I > don't want them and won't use them, but it could ease the pain others will > cause by using them. That's not enough to make me in favor of them, though; > just enough so I won't pout if they do go in. > > > Anyway, I'm not too inclined continuing this discussion anymore. > > I would never have thought that such a nifty little patch would > > have stirred up so much negative response. I'll just hack up my > > very own import mechanism using imputil and be done. So there! > > Don't feel picked on, M-A! Opposition is normal and healthy; and the last > idea I got anywhere with was list.pop(). Yah well. Last thing I succeeded with was making Python methods wrap any callable type. Seems that we're a bit too healthy (see MarkH's post) sometimes, i.e. there isn't all that much room for experiments. Just think of cool developments like Chris' stackless python. Experience shows that these kind of things will never make it into the distribution. Unfortunately, maintaing patches to the dist across releases a real pain and much work, so these ideas will just sit there unused and untested. Much the same happened to gcc ... in the end corporate strength made egcs possible. Perhaps we need such a branch too ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 106 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein at lyra.org Thu Sep 16 13:00:50 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 04:00:50 -0700 Subject: [Python-Dev] Relative Package Imports References: <000101befff4$ecc9bde0$712d153f@tim> <37E0CCC1.6EE4E599@lemburg.com> Message-ID: <37E0CDE2.56A6C6E4@lyra.org> M.-A. Lemburg wrote: > ... > Seems that we're a bit too healthy (see MarkH's post) sometimes, > i.e. there isn't all that much room for experiments. Just think > of cool developments like Chris' stackless python. Experience shows > that these kind of things will never make it into the distribution. A portion of my threading patches went into 1.5. "Big" things can happen and they do... it's just a matter of passing muster with The Head Honcho. That, and the timing (e.g. the thread state changes wouldn't go into a 1.4.n release, but did go into 1.5). > Unfortunately, maintaing patches to the dist across releases a real > pain and much work, so these ideas will just sit there unused and > untested. Much the same happened to gcc ... in the end corporate > strength made egcs possible. Perhaps we need such a branch too ? NO -g -- Greg Stein, http://www.lyra.org/ From fredrik at pythonware.com Thu Sep 16 13:32:35 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 16 Sep 1999 13:32:35 +0200 Subject: [Python-Dev] CP4E References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> Message-ID: <004101bf0037$2dca30d0$f29b12c2@secret.pythonware.com> > BTW, what is this CP4E thing you're talking about. http://www.python.org/doc/essays/cp4e.html also see: http://www.deja.com/=dnc/thread/522068184 (on second thought, don't). > If it's an syntax aware editor, I have a friend who is really > interested in these things... aggressively python-aware editors are fun. want braces? sure, just change the style sheet ;-) From guido at CNRI.Reston.VA.US Thu Sep 16 13:49:44 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 07:49:44 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Thu, 16 Sep 1999 12:44:34 +0200." <37E0CA12.1E24EBF2@lemburg.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> Message-ID: <199909161149.HAA19650@eric.cnri.reston.va.us> > I'm wasn't planning to write my own hook until now. I need this > feature to be able to organize my package internal stuff swiftly > and easily, plus to be able to possibly put it under new top-level > packages. Obviously I seem to be more or less alone with this need, so > I'll stop argueing for inclusion of "something like relative imports" > in the distribution. I still hope against hoping to convince you not to do this. I think this adds lots of complexity for the reader of your package. I think of *all* software that I write as something that eventually someone is going to read and try to understand. I also think of it as something that eventually someone is going to port to a platform that wasn't even designed when I wrote the software. Tricks like depending on a custom import hook make reading a pain. For example, the reader who is just starting to browse a large package often needs to follow the thread of execution from one module to the next. If the import statements contain magic like __magic__.foo.bar (actually proposed magic renamed for generality :-), this reader will first need to understand the custom importer -- which is likely one of the hariest components of the package. The same thing applies with even more force to tools like package browsers. IDLE for example has a class browser which displays the true module and class name for base classes. E.g. if you have code that says from Tkinter import Widget class MyWidget(Widget): ... it correctly deduces that MyWidget's base class is Tkinter.Widget. (And in a future version it will probably allow you to click through to that class, show you its methods, etc.) Custom importers break this feature, and thus make the code harder to analyze for the reader. (Not all custom importers are bad; there are plenty of reasons to augment the standard importer. But, while custom importers make different interpretations of the namespace possible, I frown upon that particular use.) > (b) is (if at all) a problem only to be taken into account by > the author of package X. He may or may not use rel. imports. > A 12-year old probably won't (but then: you never know ;). Which severely limits your ability to do what you want with packages you didn't write. I know that this sounds politically incorrect in a radical free software world, but often in the end I find it more convenient to conform to the rest of the world and "fit in" than to be different in every little way. Note that my hair isn't blue :-); I've also replaced my "signature" glasses with a more conventional model. > BTW, what is this CP4E thing you're talking about. If it's an > syntax aware editor, I have a friend who is really interested > in these things... could you send me an URL that I can send him ? Where have you been? It's on the python.org homepage, has been discussed in c.l.py, c.l.tcl, even c.l.ada, in lwn, and on /.! The syntax aware editor (a proposed super-version of IDLE) is only a small part of it. See python.org/doc/essays/cp4e.html > Oh well... I guess they'll have to use 'mex' if they decide to go > the reverse domain way ;-) Or, in a worse-case scenario, the first Mexican developers using the reverse domain will probably not be aware of the mx toplevel package, and by the time their software hits the street it will be too late. > True, perhaps we should lighten this requirement a little when we > recode the import mechanism in Python ? E.g. if a local import > fails continue the search with the fully qualified name and only > if that fails, restart using the local name. This would need some > kind of fastpath cache to make the search reasonably fast though. Now this is something that I would consider. > Wouldn't it suffice to just put them into one package, e.g. > 'python.' ? And somehow make all user code live implicitly inside that package? I don't see how that solves anything. > BTW, as Tim argued: the breakage an easily be leveraged by using > a smart editor... ;-) What breakage? I think __ is a bit too hard to fix easily with a smart editor. And believe me, at some point *someone* is going to need to rip out your custom importer because it interferes with *his* custom importer, and he'll choose to replace all your __ imports with absolute package names, rather than trying to add the __ feature to his importer. --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Thu Sep 16 13:57:20 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 04:57:20 -0700 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <37E0DB20.20C72337@lyra.org> Guido van Rossum wrote: >... > What breakage? I think __ is a bit too hard to fix easily with a > smart editor. And believe me, at some point *someone* is going to > need to rip out your custom importer because it interferes with *his* > custom importer, and he'll choose to replace all your __ imports with > absolute package names, rather than trying to add the __ feature to > his importer. This is an aspect of Python's singular import hook. imputil is designed to allow for multiple Importers to be present and to work in harmony. As long as each Importer can identify that a particular import is part of its namespace (e.g. HTTPImporter recognizes _http_... as one of its own), then no conflicts will arise and everything will be hunky-dory. btw, it might be nice to allow a string to the import statement. I'd much rather say: import "http://www.lyra.org/greg/python/qp_xml.py" Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Thu Sep 16 14:00:06 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 05:00:06 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> Message-ID: <37E0DBC6.12D3EB06@lyra.org> Gordon McMillan wrote: >... > def _reload_hook(self, module): > # gmcm - Hmmm, reloading of a module may or may not > be impossible, > # (depending on the importer), but at least we can > # look to see if it's ours to reload: > if hasattr(module, '__importer__'): > if getattr(module, '__importer__') == self: > raise SystemError, "reload not yet implemented" > return self.__chain_reload(module) I've folded this in (finally). New imputil.py to be published in a bit... thx! -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Thu Sep 16 14:45:39 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 08:45:39 -0400 Subject: [Python-Dev] Relative Package Imports References: <1274695692-31669872@hypernet.com> <37E0052F.A899E985@digicool.com> Message-ID: <37E0E673.29A844BF@interet.com> Jim Fulton wrote: > > Gordon McMillan wrote: > > > > Jim Fulton wrote: > > > [Barry] > > > > A > > > > user would drop the kjParsing directory into Shared, and then > > > > all the import kjParsing.kjParser statements would Just Work. > > > > To be a standalone application, you can use tricks that Jim > > Ahlstrom pointed out, or tricks from my Installer. The major > > point being that sys.path has nothing to do with the > > PYTHONPATH environment variable. You now have an > > installation where you can be assured of no collisions, > > because you control it completely. JimF, I am still trying to understand this. Gordon, Barry and I think a good solution is to put "Shared" on a private sys.path, but you object to this and I don't understand. Specifically zope/ zope/Shared zope/Shared/mx/... zope/Shared/AnotherPackage/... zope/Shared/some_file.py zope/ZopePackage1/... zope/ZopePackage2/... zope/bin/python.exe Suppose zope users must use the special binary zope/bin/python.exe. Suppose further that this binary reliably sets its sys.path to the directories it needs, namely zope, zope/Package*, zope/Shared. Suppose further that all user's packages must go into zope/Shared either as files or package directories and so are at the top level as package imports have always assumed. Then it seems that your zope app would absolutely work, and no other Python installation could damage it by changing PYTHONPATH, and that your zope installation would not damage any other Python installation either. Is the reason this doesn't work the posibility of name collisions in zope/Shared? I thought before that you were worried about name collisions in another co-existing Python installation which may have unknown packages installed, conflicting packages, or packages with incompatible versions. If so, you are trying to solve a problem more severe than I thought. I have always assumed packages would have unique names and so a flat name space suffices. In particular, trying to design for conflicting names can not be supported by the current python *.pyl library file design, and in general would require a PYTHONPATH mechanism. Such PYTHONPATH mechanism would have to be "local" to packages. That is, PYTHONPATH would vary depending on the package doing the importing. Are you sure this problem needs a solution right now? Jim Ahlstrom From jim at interet.com Thu Sep 16 14:57:46 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 08:57:46 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> Message-ID: <37E0E94A.A2F0D37F@interet.com> Guido van Rossum wrote: > > Even worse, it means that exceptions.py and site.py can not > > be found at all except using the normal PYTHONPATH, and > > putting their path in Spam_path will *not* work. > > Why would you want your own exceptions.py and site.py? I don't. I never change Python library files. I am worried that they won't be found because I don't trust PYTHONPATH. > > Oh dear, I think I heard no instead of yes. Are you saying that if > > someone else installs a Python app on my customer's machine after I do, > > and sets a registry entry which sayes to use c:/other/path/to/site.py > > for site.py (as he may very well want to do), then if my Python program > > depends on getting my copy of site.py from my directory, it will then > > use the other copy instead and may very well fail? > > Again - why would anyone register their own site.py? I wouldn't, I am worried that someone else will break my installation. Remember that site.py was invented as a site-specific module, although that function moved to sitecustomize.py. > I presume using GetModuleFileName()? Please send me the patch! Yes, and OK. > > JimA's conjecture: It is currently impossible to > > ship a Python app which can not be damaged by the installation of a > > second Python app without using a hacked custom binary. > > Sounds right. All tricks to make the app unique require using a > different registry key, which requires a change to the DLL. However, > you can do this without recompiling! The version string is used is > embedded in a resource, so you can patch it using some kind of > resource editor. Mark Hammond planned it this way! I don't understand this. Is there documentation? Jim Ahlstrom From guido at CNRI.Reston.VA.US Thu Sep 16 15:49:16 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 09:49:16 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Thu, 16 Sep 1999 08:57:46 EDT." <37E0E94A.A2F0D37F@interet.com> References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> Message-ID: <199909161349.JAA19789@eric.cnri.reston.va.us> [me] > > Why would you want your own exceptions.py and site.py? [JimA] > I don't. I never change Python library files. I am worried > that they won't be found because I don't trust PYTHONPATH. Hmm... PYTHONPATH gets inserted in front of the default sys.path. (Many moons ago that was different. But it has been like this for a loooooong time.) So are you worried that someone put a *different* exceptions.py or site.py on their path? > > Again - why would anyone register their own site.py? > > I wouldn't, I am worried that someone else will break my installation. > Remember that site.py was invented as a site-specific module, although > that function moved to sitecustomize.py. Hm, I dug out the oldest site.py I have (used in Python 1.4), and it doesn't encourage editing it at all -- it tells you to use sitecustomize.py. I guess they could break your installation anyway, but only by messing with the general Python installation. > > Sounds right. All tricks to make the app unique require using a > > different registry key, which requires a change to the DLL. However, > > you can do this without recompiling! The version string is used is > > embedded in a resource, so you can patch it using some kind of > > resource editor. Mark Hammond planned it this way! > > I don't understand this. Is there documentation? The usual :-) Python/import.c shows that import calls PyWin_FindRegisteredModule() to find a registered module before looking in sys.path (but after checking for builtin and frozen modules). PC/import_nt.c shows that PyWin_FindRegisteredModule() uses a registry key of the form "Software\Python\PythonCore\\Modules\" where is the module name, is empty or "\Debug" depending on whether we are compiled with _DEBUG define. The resource value points to a file (either .py, .pyc/.pyo, .pyd or .dll; in fact any of the prefixes returned by imp.get_suffixes()). PC/dl_nt.c shows that PyWin_DLLVersionString is set to string 1000 loaded from the string resource table. PC/python_nt.rc shows that there's a stringtable with item 1000 being the MS_DLL_ID string, set to "1.5" in that file. Note that this value (PyWin_DLLVersionString) is also to Python code as sys.winver. I hope that Mark Hammond can point you to a tool that you can use to edit a string resource in an executable or DLL. --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Thu Sep 16 15:46:19 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 06:46:19 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <37E0F4AB.38EAB65F@lyra.org> M.-A. Lemburg wrote: >... > The original version of imputil I fetched from Greg's page > did work out of the box (from...import... hassles) and "did not work", I presume. From my original testing, I thought from...import worked. With more testing, I found that something of the form "from xml.dom import builder" did not work. I discovered why it failed (xml.dom was imported by Importer instance I1 but I2 thought it could handle the from...import, and this barfed a check). I've fixed this by delegating to the proper importer (I1 in my example) to complete the import. Your solution to check the __importer__ variable in the globals is probably incorrect. If I read/eval it correctly, that would mean that a module imported by IMP1 could not use modules imported by IMP2. In other words, a package module could not import a top-level module defined by a different importer. (note also that your globals.get() could fail if globals is None) > obviously did not support in-package shared libs. I've added I did not fold this in. Your change isn't "in the spirit" of the Importer mechanism. The "Right Way" to do this is to create a BuiltinImporter and add that to the chain of importers. The DirectoryImporter should only import from directories -- no reason for it to know about builtin stuff. As a result, I did not accept the new methods on Importer for handling builtins/special modules -- those would go in the BuiltinImporter. [BuiltinImporter should be written and included in imputil.py; I don't really have the time at the moment to write the thing... 7am and time for sleep...] However, your change here did raise a very important design issue: get_code() needs to be able to return a loaded module, rather than just a code object. I've folded in your patches for that. I also folded in many of your extended doc/comments (at least in concept; not necessarily verbatim). You and Gordon are recognized in the header now, and I've added a "proper" author notice and licensing (public domain). I did not include the "misses" feature that you added to the DirectoryImporter. I would hate to see a miss-cache get loaded, a module dropped into the filesystem, and the user never being able to import the thing. I didn't fold in your indentation changes or name changes. I liked mine :-). The __main__ thing at the bottom didn't make much sense to me, though, since the call to _test_dir() followed by an exit doesn't really do anything. And yes, I recognize that you can use "python -i imputil.py" but I'd rather just see "python" followed by "import imputil ; imputil._test_dir()". Of course, please feel free to generate a new patch if I've missed something (thinking about it, I missed the OSError thing). > both features so that the test script in DateTime can run > successfully. > > Things that remain are: > ? the win32 registry stuff (needs C code) And a new Importer to use it. > ? the Mac fork stuff (needs C code) Ditto. > ? a working __path__ implementation (is anyone using this attribute > which only is available in packages ?) Per the private mail that I sent to you: I explicitly punted on the __path__ attribute. It can lead to *way* too much confusion. It is also unavailable for frozen packages (boy oh boy did the win32com get some ugliness in there to compensate for being frozen w.r.t. its use of __path__). The DirectoryImporter can insert the attribute, but it definitely wouldn't go into the Importer itself. The __path__ attribute is specific to loading from a filesystem, yet Importer is generic. > ? probably a whole bunch of other quirks > ? some speedups (there currently are too many stat()s) Yes. I recognize that the "misses" feature was intended to remedy this. I don't have an immediate answer to the stat() issue. Does the Importer mechanism actually perform more stats on an import than Python itself? (it looks like it does one for the isdir() plus two for fetching file timestamps) And a big thanx: I appreciate the patches to imputil! The new module is now available in its "official" location at http://www.lyra.org/greg/python/ Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Thu Sep 16 15:56:00 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 09:56:00 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> Message-ID: <37E0F6F0.8FC49D4@interet.com> Greg Stein wrote: > > Gordon McMillan wrote: > >... > > def _reload_hook(self, module): > > # gmcm - Hmmm, reloading of a module may or may not > > be impossible, > > # (depending on the importer), but at least we can > > # look to see if it's ours to reload: > > if hasattr(module, '__importer__'): > > if getattr(module, '__importer__') == self: > > raise SystemError, "reload not yet implemented" > > return self.__chain_reload(module) > > I've folded this in (finally). > > New imputil.py to be published in a bit... As a result of all this import discussion I am a bit worried that the python library *.pyl file format may not be powerful enough. I have always thought in terms of unique top-level names and a format which supports import of modules and packages. But this does not support the full functionality of PYTHONPATH. For example, PYTHONPATH can (and is) used to select the correct plat-* directory files. And the format may not support Jim Fulton's fancy local import scheme. And what if someone invents a third thing to import besides a module or a package? PYTHONPATH is not going away nor should it. How about if the *.pyl file format is exactly a directory structure? I mean that the table of contents is limited to paths starting with a directory name only, and that the seperator is '/' instead of '.'. So a listing would be identical to the output of 'ls -R'. So: Lib/string.pyc Lib/exceptions.pyc Lib/plat-sunos4/... mx/__init__.pyc mx/... package2/... dir3/... ... The implied PYTHONPATH for this file is ["Lib", "."]. Since the format is exactly a directory tree, it is guaranteed that whatever PYTHONPATH or imports can do now or in the future with a directory tree, it can still do it with a *.pyl file. Jim Ahlstrom From gstein at lyra.org Thu Sep 16 16:05:42 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 07:05:42 -0700 Subject: [Python-Dev] Towards a Python based import scheme References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> <37E0F6F0.8FC49D4@interet.com> Message-ID: <37E0F936.6D32E983@lyra.org> James C. Ahlstrom wrote: >... > As a result of all this import discussion I am a bit worried that > the python library *.pyl file format may not be powerful enough. Background for the readers: .pyl is an extension that I used in my "small" distribution. I think Gordon uses it, too. In any case, it is effectively a concatenation of .pyc files along with a TOC mapping fully-qualified dotted module names to seek-positions within the file. [speaking of stat() calls: using a .pyl eliminates them quite nicely -- this may be part of Gordon's observed speed increase when using an archive] The .pyl format was discussed a bit on the distutils-sig list and "sort of" accepted as an okay format for jamming a bunch of modules into a single file. [by "sort of", I mean that the small group who participated in the discussion were okay with it :-); it is a great, minimalist format, so it probably won't please people who like a ton of features in a file format :-) ] >... > How about if the *.pyl file format is exactly a directory structure? > I mean that the table of contents is limited to paths starting with > a directory name only, and that the seperator is '/' instead of '.'. > So a listing would be identical to the output of 'ls -R'. So: > Lib/string.pyc > Lib/exceptions.pyc > Lib/plat-sunos4/... > mx/__init__.pyc > mx/... > package2/... > dir3/... > ... > > The implied PYTHONPATH for this file is ["Lib", "."]. Since the > format is exactly a directory tree, it is guaranteed that whatever > PYTHONPATH or imports can do now or in the future with a directory > tree, it can still do it with a *.pyl file. People import things using a dotted name. Therefore, I think it makes the most sense to map that straight to the resulting .pyc file. No reason to put directories into the file... they make no sense to the end user. During construction of the .pyl, you would walk the tree finding all the available modules (and their corresponding dotted name) and insert them. Note that you can distribute multiple .pyl files. There could be the Python standard lib in one file, the mx package in another, etc. As a module is searched for, the system just peeks into each .pyl in turn, looking for the module. Search order is currently defined by order of install() on the Importer instances. I believe the Right Way to do things is to create sys.importers (as a list of Importers) and deprecate the sys.path variable. Python could start up with an Importer than simply scanned sys.path as a backwards compat measure; it could also leave sys.path empty and create DirectoryImporters for each path component (this could cause problems, though, for some apps that believe sys.path shouldn't be empty, or that use it for magic-munging). I've search the standard lib in the past -- there are only a couple real uses of sys.path if I remember rightly (test package and the traceback module). Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein at lyra.org Thu Sep 16 16:19:41 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 07:19:41 -0700 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> <199909161349.JAA19789@eric.cnri.reston.va.us> Message-ID: <37E0FC7D.7494B8F7@lyra.org> Guido van Rossum wrote: > > [me] > > > Why would you want your own exceptions.py and site.py? > [JimA] > > I don't. I never change Python library files. I am worried > > that they won't be found because I don't trust PYTHONPATH. > > Hmm... PYTHONPATH gets inserted in front of the default sys.path. > (Many moons ago that was different. But it has been like this for a > loooooong time.) So are you worried that someone put a *different* > exceptions.py or site.py on their path? When you jam every file into a module archive, you still have to leave these two "outside" so that Python can find them when starting up. The problem then breaks down into two parts: 1) locating them 2) ensuring they're the right versions In my "small" distro, I relied on the current-dir always being in the path, so I could always find these. The two files were distributed as part of distro. Anyhow: JimA is saying that he doesn't trust PYTHONPATH. Not so much bad files/versions, but that he won't be able to find them because PYTHONPATH has been monkeyed in some way. > > > Again - why would anyone register their own site.py? > > > > I wouldn't, I am worried that someone else will break my installation. > > Remember that site.py was invented as a site-specific module, although > > that function moved to sitecustomize.py. > > Hm, I dug out the oldest site.py I have (used in Python 1.4), and it > doesn't encourage editing it at all -- it tells you to use > sitecustomize.py. I guess they could break your installation anyway, > but only by messing with the general Python installation. If the file exists, then somebody can mess with it. JimA is trying to create a bulletproof app here. To do this, you can distribute a python.exe, exceptions.py, site.py, and an archive of your other modules. site.py is scrapped in favor of installing an Importer to access the archive (this implies you also distribute imputil.py). These five files are the exact five in my "small" distro. It's pretty cool... no need for registry changes and a very small "file count" footprint. Gordon took this basis and added a bunch of features for bundling an application in there. JimA has also been mixing in some frozen modules (I forget exactly why/what). [oh, my small distro doesn't ship a python15.dll, although it easily could] >... > I hope that Mark Hammond can point you to a tool that you can use to > edit a string resource in an executable or DLL. The win32api module has resource manipulation functions such as BeginUpdateResource, UpdateResource, and EndUpdateResource. Write a Python script to modify your version string :-) A demo of resource munging can be seen in /win32/scripts/VersionStamp/verstamp.py. Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at digicool.com Thu Sep 16 16:56:53 1999 From: jim at digicool.com (Jim Fulton) Date: Thu, 16 Sep 1999 10:56:53 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <37DFFAFD.72F6A806@lemburg.com> Message-ID: <37E10535.CA96A77E@digicool.com> "M.-A. Lemburg" wrote: > > ? a working __path__ implementation (is anyone using this attribute > which only is available in packages ?) Yes. I use it for two things: - I modify it to allow a (logical) package to be spread over multiple physical locations. (In Zope, products can be installed in the Zope installation area or in Zope "instance" homes. - I use it to determine the location(s) of a package. Our packages usually contain many files, such as DTML source files, images, data files, etc., that are not Python modules. We have standard utilities for getting at these files in packages. This is extremely useful. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From jim at interet.com Thu Sep 16 16:58:48 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 10:58:48 -0400 Subject: [Python-Dev] Re: Path hacking References: <37DFBE91.C93816F1@interet.com> <199909151615.MAA17872@eric.cnri.reston.va.us> <37DFE592.7FD384A2@interet.com> <199909152235.SAA18693@eric.cnri.reston.va.us> <37E0E94A.A2F0D37F@interet.com> <199909161349.JAA19789@eric.cnri.reston.va.us> Message-ID: <37E105A8.A2B06A80@interet.com> Guido van Rossum wrote: > Hmm... PYTHONPATH gets inserted in front of the default sys.path. > (Many moons ago that was different. But it has been like this for a > loooooong time.) So are you worried that someone put a *different* > exceptions.py or site.py on their path? Yes, and/or (2) added a sitecustomize.py with their special import hook as has been proposed here over and over, or (3) PYTHONPATH is screwed up and doesn't find anything. Perhaps this is a prejudice of mine. I just look at "print sys.path" and marvel at what I don't understand. At least I can see it is not simple. I hate relying on thing that are not simple. And I hate custom import hooks. Unless they are mine of course ;-) > Hm, I dug out the oldest site.py I have (used in Python 1.4), and it > doesn't encourage editing it at all -- it tells you to use > sitecustomize.py. I guess they could break your installation anyway, > but only by messing with the general Python installation. Just adding sitecustomize.py would do it. And this is encouraged. I think Gordon put his finger on the issue. Either try to co-exist with other installed Python software and take the risk that everyone is playing by the rules, or build your own black-box self-contained Python world and duplicate storage. > The usual :-) > [Documentation goes here...] Thanks. This is very useful. But it doesn't help, perhaps as a result of more of my prejudices. This registry entry is meant to be used by a by-the-rules shared Python installation, so I must not change it. And I hate changing any registry entries at all. My commercial software keeps all its settings in a regular .ini file in its install directory, and makes only minimal and required registry entries. IMHO the Windows registry is a software catastrophe which ranks right up there with JCL (anyone else here old enough to remember that?). Anyone who doesn't agree should go with me to our money center banking clients, and sit there while they grill you on every registry entry and why it is required. Money center banking clients do not like their registry messed with. I do however see your point that I could change the version string to something non-standard and use the registry to control imports. I will think about this further. Maybe it would work. My current "solution" is to use freeze to create a black-box install, and worry about second Python installations and wasted storage when it happens. I was hoping that this thread whould result in a consensis of what to do, but it has not. So now I am hoping that Python library (jar) files will turn out to be a practical solution, so I am pestering Greg and Gordon. We'll see. Jim Ahlstrom From bwarsaw at cnri.reston.va.us Thu Sep 16 17:16:42 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 16 Sep 1999 11:16:42 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> Message-ID: <14305.2522.922445.932104@anthem.cnri.reston.va.us> >>>>> "JF" == Jim Fulton writes: >> I think we're safe from this for the forseeable future through >> convention and peer pressure. JF> 8^o JF> (I feel like I should respond to this in some way but .... JF> I just can't understand this kind of argument and don't JF> know how to frame a response.) Maybe I should have included half a smiley. Python may eventually need package naming rules as strict as Java's. I think it's a good idea now for those who are writing packages to consider trying to pick unique top-level package names. We might decide to release our stuff under the `cnri' top level package. That's probably unique enough. Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! Personally, I think `Mailman' as a top-level package name is pretty unique, so I don't plan to change what I'm doing there. -Barry From bwarsaw at cnri.reston.va.us Thu Sep 16 17:37:01 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 16 Sep 1999 11:37:01 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> <37E049CD.6B64DE00@lyra.org> Message-ID: <14305.3741.741370.948251@anthem.cnri.reston.va.us> >>>>> "GS" == Greg Stein writes: GS> Definitely having an OSI certification will be great (cool GS> stuff on the cert for the existing license!). Having Bruce GS> Perens review the license would also be a great boon (see GS> www.perens.com for some of his writings; also see GS> http://perens.com/Termination.html specifically). Interesting article, but IBM's termination clause was different than the JPython one. I fought hard on para7 because IIRC, RMS complained that an earlier version /could/ have been used to arbitrarily terminate. I think the current JPython para7 is better because /you/ have to materially breach, which seems like a much higher threshold. But it still may not be perfect. Aside: don't necessarily think I'm a grinning fan and defender of the JPython license. It's a huge win over what we had before, and I think it's good enough that nearly anybody who wants to do Real Things with JPython, now can. I've had only one question about the license since it was published and that was about the "displayed prominently" clause (i.e. was it okay to include the alternative handle text in an "about" menu pulldown? That seemed prominent enough to my nonlawyerly brain.) I'm glad to see the Python community push hard for the "other side's viewpoint" with reasoned and rational arguments. I think that such responses from Influential Python Users will provide us with useful ammunition when we re-evaluate the licenses. It means that ultimately we'll have the right license for Python (and JPython). Thanks, -Barry From skip at mojam.com Thu Sep 16 17:54:39 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 10:54:39 -0500 (CDT) Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37E0F6F0.8FC49D4@interet.com> References: <1274694439-31745225@hypernet.com> <37E0DBC6.12D3EB06@lyra.org> <37E0F6F0.8FC49D4@interet.com> Message-ID: <14305.4799.328120.623217@dolphin.mojam.com> Jim> As a result of all this import discussion I am a bit worried that Jim> the python library *.pyl file format may not be powerful enough. Not to rain on anyone's parade, but I want to remind the folks having this discussion that there are people reading this thread that while fairly well versed in Python have little idea what anyone is talking about anymore. (I don't know. Maybe I'm the only one.) python-dev is clearly the best place to discuss this in the short-term (anyone for an import SIG?), but whatever is implemented will have to be understood by lots of people on c.l.py to be of broad applicability. Perhaps I'm way off base and there are more than a handful of people who will ever run into the problems being solved here, but if we can partition the Python programming community into the package wizards and the mere import mortals, I worry that the potions concocted by the wizards will send a few of us import mortals to the hospital... The Java package scheme, while odious to some perhaps, is extremely easy to understand for anyone who's ever used Windows Explorer or execute "ls -R". just-a-cautionary-peanut-thrown-in-from-the-bleachers-ly y'rs Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From skip at mojam.com Thu Sep 16 19:01:58 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 12:01:58 -0500 (CDT) Subject: [Python-Dev] Relative Package Imports In-Reply-To: <14305.2522.922445.932104@anthem.cnri.reston.va.us> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> <14305.2522.922445.932104@anthem.cnri.reston.va.us> Message-ID: <14305.8838.329883.696911@dolphin.mojam.com> BAW> Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! Umm, that would be "Knowbots Over Everything" or "Knowledge Of Everyone"? ;-) Skip From guido at CNRI.Reston.VA.US Thu Sep 16 19:04:37 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 13:04:37 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: Your message of "Thu, 16 Sep 1999 12:01:58 CDT." <14305.8838.329883.696911@dolphin.mojam.com> References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <14303.59796.4588.287265@anthem.cnri.reston.va.us> <37DFF4B0.15390199@digicool.com> <14304.6310.20571.487708@anthem.cnri.reston.va.us> <37E02282.BDF5FF21@digicool.com> <14305.2522.922445.932104@anthem.cnri.reston.va.us> <14305.8838.329883.696911@dolphin.mojam.com> Message-ID: <199909161704.NAA01253@eric.cnri.reston.va.us> > BAW> Or `koe', or if we're really paranoid us.va.reston.cnri.koe :-! > Skip> Umm, that would be "Knowbots Over Everything" or "Knowledge Of Everyone"? Actually, "koe" is dutch for cow. I like the idea of cows over everything -- reminds me of the flying pig on an old Pink Floyd cover :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From gstein at lyra.org Thu Sep 16 19:27:49 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 10:27:49 -0700 Subject: [Python-Dev] License cleanup References: <004601beffe1$9edb01b0$0801a8c0@bobcat> <37E049CD.6B64DE00@lyra.org> <14305.3741.741370.948251@anthem.cnri.reston.va.us> Message-ID: <37E12895.443CC6F3@lyra.org> Barry A. Warsaw wrote: >... > Interesting article, but IBM's termination clause was different than > the JPython one. I fought hard on para7 because IIRC, RMS complained > that an earlier version /could/ have been used to arbitrarily > terminate. I think the current JPython para7 is better because /you/ > have to materially breach, which seems like a much higher threshold. > But it still may not be perfect. Yes, I was aware that it was a reactive termination, rather than arbitrary. That makes it quite acceptable, but it still isn't a desirable thing. Especially given some of the grey area in the license ("are we sure we aren't in breach of the license?"). Personally, I'd rather see a license without a termination clause. If it must be there, then I'd like to see it as tight as possible (see the IBM and Apple licenses: IIRC, they only kick in when the user initiates patent litigation against IBM/Apple; the termination cuts them off as an initial response to the suit). The other elements I raised actually caused me more anxiety than the termination. If CNRI finds it acceptable, I'd recommend they use an existing OSD license. They get immediate certfication and, more importantly, a builtin awareness in the open source community of what the license really means. Each time a new license arrives in the community, bunches of people have to go an figure it out; if the new license is the IBM Public License with a search/replace on the company and product name, then people go "oh. all righty. no problem." and move on to doing real stuff. Dang. I keep replying to this stuff. :-) I'm hoping that we wrap this up pending a new release. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gmcm at hypernet.com Thu Sep 16 19:38:03 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Towards a Python based import scheme In-Reply-To: <37E0F6F0.8FC49D4@interet.com> Message-ID: <1274619156-36273405@hypernet.com> Jim Ahlstrom wrote: > As a result of all this import discussion I am a bit worried that > the python library *.pyl file format may not be powerful enough. > I have always thought in terms of unique top-level names and a > format which supports import of modules and packages. But this > does not support the full functionality of PYTHONPATH. For > example, PYTHONPATH can (and is) used to select the correct > plat-* directory files. And the format may not support Jim > Fulton's fancy local import scheme. And what if someone invents > a third thing to import besides a module or a package? > PYTHONPATH is not going away nor should it. The central idea of imputil is that an importer is responsible for one little chunk of turf. If the desired module / package isn't "his", he just passes the request on to the next element in the chain. So I don't think there's a need for one canonical do-everything importer (or archive format). PYTHONPATH is outside any particular importer. Effectively, you can use a chain of importers to replace PYTHONPATH. So the platform specific modules might be found by one particular importer. In other words, I think it's more effective to specialize individual importers and chain them up than it is to try to create an overly-generalized importer. - Gordon From gmcm at hypernet.com Thu Sep 16 19:38:03 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E0FC7D.7494B8F7@lyra.org> Message-ID: <1274619162-36273021@hypernet.com> [Greg, replying to Guido's confusion on Jim's interest in site.py and exceptions.py] > When you jam every file into a module archive, you still have to > leave these two "outside" so that Python can find them when > starting up. In the soon-to-be-published beta version of my installer, I've got that down to exceptions.py. Background: my (Win32) installer has, as a stated goal, the ability to create quasi-frozen Python apps which won't interfere with (or be influenced by) existing Python installations (if any). And it doesn't require the programmer/user to have a compiler. Thanks to patches given me by Thomas Heller, my python.exe replacement is now a (minimal) embedding app, and I do the same things that Greg does in site.py directly from C code, (and turn off the SiteFlag, too). I believe that if I freeze in exceptions.py and tweak the resource in python15.dll (which is just the stock python15.dll), I can have a completely safe executable. I think the same techniques can be applied on *nix, (although I'm pretty sure programmer/users won't be able to get away without a compiler). This gives me a strong interest in import hooks for two distinct reasons: - I rely completely on Greg's imputil to make this work. - I rely on freeze's modulefinder to help build these things. Bizarre import hooks in a normal Python installation will fool modulefinder. Take a look at what Pmw does (fortunately, Pmw comes with it's own packager). So I'd like to see import hooks follow some sort of pattern that can be followed by a tool like modulefinder. I also want the hooks. Right now I use imputil with archives, but wouldn't it be cool if you could add another imputil importer that checks for more recent versions at some home site on the web and automatically updates the installation? Summary: I'm very interested in seeing import and import hooks get rationalized, and I think Greg's stuff goes a long, long way towards that goal. - Gordon From gmcm at hypernet.com Thu Sep 16 19:38:03 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Thu, 16 Sep 1999 13:38:03 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E105A8.A2B06A80@interet.com> Message-ID: <1274619161-36273101@hypernet.com> James C. Ahlstrom wrote: [Guido explains Windows registry usage] > Thanks. This is very useful. > > But it doesn't help, perhaps as a result of more of my > prejudices. This registry entry is meant to be used by a > by-the-rules shared Python installation, so I must not change it. Without trying it, I doubt you have to. It looks like you could set the resource to something that won't be found in the registry, and then just use normal Python mechanisms. > ... IMHO the Windows registry is a software catastrophe > which ranks right up there with JCL (anyone else here old enough > to remember that?). Ah. In a rush to get it down to ops, tripping at the head of the stairs and sending the card deck flying... Those were the days. - Gordon From skip at mojam.com Thu Sep 16 19:49:12 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 12:49:12 -0500 (CDT) Subject: [Python-Dev] Re: Path hacking In-Reply-To: <1274619162-36273021@hypernet.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> Message-ID: <14305.11672.687570.290221@dolphin.mojam.com> Gordon> In the soon-to-be-published beta version of my installer, I've Gordon> got that down to exceptions.py. Why not just run exceptions.py through Python2C, visually and experimentally verify that it works, then ship an exceptions.c as an optional module? People wanting to ship self-contained packages could then toss exceptions.py and build the C version of the exceptions module. Greg, is there anything in exceptions.py Python2C couldn't handle? Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From gstein at lyra.org Thu Sep 16 19:47:56 1999 From: gstein at lyra.org (Greg Stein) Date: Thu, 16 Sep 1999 10:47:56 -0700 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <37E12D4C.2394D311@lyra.org> Skip Montanaro wrote: > > Gordon> In the soon-to-be-published beta version of my installer, I've > Gordon> got that down to exceptions.py. > > Why not just run exceptions.py through Python2C, visually and experimentally > verify that it works, then ship an exceptions.c as an optional module? > People wanting to ship self-contained packages could then toss exceptions.py > and build the C version of the exceptions module. > > Greg, is there anything in exceptions.py Python2C couldn't handle? Nah, shouldn't have any problem at all. P2C will even create true class objects and expose them in the interface. I think a person might want to consider hand-tuning the output, though :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From jim at interet.com Thu Sep 16 20:03:13 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 14:03:13 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <37E130E1.F21CDE8B@interet.com> Skip Montanaro wrote: > Why not just run exceptions.py through Python2C, What is Python2C. Is it the same as freeze? Jim Ahlstrom From guido at CNRI.Reston.VA.US Thu Sep 16 20:08:01 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 14:08:01 -0400 Subject: [Python-Dev] Re: Path hacking In-Reply-To: Your message of "Thu, 16 Sep 1999 12:49:12 CDT." <14305.11672.687570.290221@dolphin.mojam.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> Message-ID: <199909161808.OAA01452@eric.cnri.reston.va.us> > Why not just run exceptions.py through Python2C, visually and experimentally > verify that it works, then ship an exceptions.c as an optional module? > People wanting to ship self-contained packages could then toss exceptions.py > and build the C version of the exceptions module. Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at mojam.com Thu Sep 16 20:34:47 1999 From: skip at mojam.com (Skip Montanaro) Date: Thu, 16 Sep 1999 13:34:47 -0500 (CDT) Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E130E1.F21CDE8B@interet.com> References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <37E130E1.F21CDE8B@interet.com> Message-ID: <14305.14407.189346.498215@dolphin.mojam.com> Jim> What is Python2C. Is it the same as freeze? Nope. Python in, compilable C out: http://www.mudlib.org/~rassilon/p2c/ Courtesy of Greg Stein and Bill Tutt. Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From jim at interet.com Thu Sep 16 20:37:58 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 14:37:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <199909161808.OAA01452@eric.cnri.reston.va.us> Message-ID: <37E13906.E2E8FB2E@interet.com> Guido van Rossum wrote: > Alternatively (and probably easier) it (and site.py) could be stored > as frozen modules. All it takes is some edits to Python/frozen.c. An excellent design, very simple. A large list of critical Python-language modules including a new importer, imputils.py and a new main could be reliably linked to their interpreter. It would open the door writing a larger part of Python in Python. But editing Python/frozen.c will kill the current freeze feature since a user currently replaces this pointer with their own. We can have a second _PyImport_InternalFrozenModules[] array which is searched FIRST, so that the existing frozen modules feature is retained. Current logic only allows one frozen module array. This is easy. A little harder is turning it off, which may be important to developers. They will probably want to use site.py etc. in PYTHONPATH directories. I would hate to use another command line option. Maybe another method in the imp module, say imp.EnablePyInternalLib(x) to turn on/off the internal frozen modules. A method to print the names in the list wouldn't hurt either. At one time I had code for multiple frozen modules which I may be able to find. Jim Ahlstrom From jim at interet.com Thu Sep 16 21:09:57 1999 From: jim at interet.com (James C. Ahlstrom) Date: Thu, 16 Sep 1999 15:09:57 -0400 Subject: [Python-Dev] Towards a Python based import scheme References: <1274619156-36273405@hypernet.com> Message-ID: <37E14085.FE46EB12@interet.com> Gordon McMillan wrote: > So I don't think there's a need for one canonical do-everything > importer (or archive format). PYTHONPATH is outside any > particular importer. Effectively, you can use a chain of > importers to replace PYTHONPATH. So the platform specific > modules might be found by one particular importer. In other > words, I think it's more effective to specialize individual > importers and chain them up than it is to try to create an > overly-generalized importer. Greg agrees with you so I defer to the experts on importers. The feature is meant to support a chain. Greg wrote: > The .pyl format was discussed a bit on the distutils-sig list and "sort > of" accepted as an okay format for jamming a bunch of modules into a > single file. > [by "sort of", I mean that the small group who participated in the > discussion were okay with it :-); it is a great, minimalist format, so > it probably won't please people who like a ton of features in a file >format :-) ] But I still disagree on the .pyl file format. If there is no Standard Format and everyone is linking in his own importer, then we will have exactly the same situation we have now with PYTHONPATH and novel import hooks. There should be a Standard Format to fix this problem. In particular, package authors should be able to publish packages as PYL files and expect them to be usable as is with no further effort. Sysadmins should be able to manage everything PYTHONPATH does with a small (one?) number of PYL files and in a standard way. Jim Ahlstrom From fredrik at pythonware.com Fri Sep 17 09:40:18 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 17 Sep 1999 09:40:18 +0200 Subject: [Python-Dev] Re: Path hacking References: <1274619162-36273021@hypernet.com> Message-ID: <00f801bf00df$e45361c0$f29b12c2@secret.pythonware.com> > Summary: I'm very interested in seeing import and import > hooks get rationalized, and I think Greg's stuff goes a long, > long way towards that goal. don't have time to say much more than: -- ME TOO! (we're also doing this in the pythonworks packager, and Greg's module is essential for that work. might as well turn it into a standard Python feature). From mhammond at skippinet.com.au Fri Sep 17 09:39:51 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 17 Sep 1999 17:39:51 +1000 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E0E94A.A2F0D37F@interet.com> Message-ID: <000501bf00df$d41cafa0$0801a8c0@bobcat> > > you can do this without recompiling! The version string is used is > > embedded in a resource, so you can patch it using some kind of > > resource editor. Mark Hammond planned it this way! > > I don't understand this. Is there documentation? Vague documentation at http://www.python.org/windows/python/registry.html Doesnt explicitely say much about this particular issue, and if I remember I will update it. VC will allow you to edit these resources without rebuilding - simply open the DLL, but select "Resources" as the file type. I havent investigated other resource editors as I have MSVC :-) Greg pointed you to the win32api functions - they do work, as Greg provided code that uses them (the code he pointed at was done by him) - although I never actually worked out how to make them work ( I spent about an hour trying to get code to re-stamp the Python15.dll with a new sys.winver, and gave up. Would be cool to get going tho... Mark. From mhammond at skippinet.com.au Fri Sep 17 09:48:26 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 17 Sep 1999 17:48:26 +1000 Subject: [Python-Dev] Re: Path hacking In-Reply-To: <37E105A8.A2B06A80@interet.com> Message-ID: <000601bf00e1$072a7d90$0801a8c0@bobcat> > But it doesn't help, perhaps as a result of more of my prejudices. > This registry entry is meant to be used by a by-the-rules shared > Python installation, so I must not change it. No - the point is that the "1.5" key is "reserved" by a standard install. Changing the string value actually allows you to have your own subtree, and you can assume you own that. For commercial apps I help with, re stick a "AppName" in this string (doesnt have to be a number) then simply use "HKLM\Software\Python\AppName" > And I hate changing any registry entries at all. Well, you should learn to get over it! That is what it is designed for. sure, we dont want the users to muck with the registry, but if you dont like your install scripts working with the registry, I definately feel you should get over this prejudice of yours. > only minimal and required registry entries. IMHO the Windows registry > is a software catastrophe which ranks right up there with JCL (anyone > else here old enough to remember that?). Anyone who doesn't agree > should > go with me to our money center banking clients, and sit there > while they > grill you on every registry entry and why it is required. > Money center > banking clients do not like their registry messed with. The registry is a huge bucket. Changing your own, custom kep under either HKLM\Software\ or HKCU\Software is perfectly reasonable. Many apps _require_ you to hit the registry - services, event log, performance monitor, etc. The fact the registry _also_ contains critical information is a problem, and sure, we dont want to mess with that. > My current "solution" is to use freeze to create a black-box install, > and worry about second Python installations and wasted storage when it > happens. well, IMO this is also the correct thing to do. any install that has >100 files is fragile. So I do both - freeze the app, _and_ a custom "sys.winver". > I was hoping that this thread whould result in a consensis of what > to do, but it has not. There is a consesus for people with the same problem. Different problems have different optimal solutions. Mark. From mal at lemburg.com Fri Sep 17 11:33:43 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 11:33:43 +0200 Subject: [Python-Dev] Relative Package Imports References: <1274889332-20022765@hypernet.com> <37DD166B.746B19AD@digicool.com> <199909142109.RAA15752@eric.cnri.reston.va.us> <37DFE109.96C652B3@digicool.com> <199909160332.XAA19278@eric.cnri.reston.va.us> <37E0CA12.1E24EBF2@lemburg.com> <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <37E20AF7.2D05F371@lemburg.com> Guido van Rossum wrote: > > > I'm wasn't planning to write my own hook until now. I need this > > feature to be able to organize my package internal stuff swiftly > > and easily, plus to be able to possibly put it under new top-level > > packages. Obviously I seem to be more or less alone with this need, so > > I'll stop argueing for inclusion of "something like relative imports" > > in the distribution. > > I still hope against hoping to convince you not to do this. I think > this adds lots of complexity for the reader of your package. I think > of *all* software that I write as something that eventually someone is > going to read and try to understand. I also think of it as something > that eventually someone is going to port to a platform that wasn't > even designed when I wrote the software. Tricks like depending on a > custom import hook make reading a pain. > > For example, the reader who is just starting to browse a large package > often needs to follow the thread of execution from one module to the > next. If the import statements contain magic like __magic__.foo.bar > (actually proposed magic renamed for generality :-), this reader will > first need to understand the custom importer -- which is likely one of > the hariest components of the package. While it will perhaps look funny to him at first sight, I'm sure that a proper name will make the idea clear to him: from __superpackage__.foo import bar or just from __super__.foo import bar [Super Foo -- sounds like a comic star ... there's even integrated fun in these names ;-)] > The same thing applies with even more force to tools like package > browsers. IDLE for example has a class browser which displays the > true module and class name for base classes. E.g. if you have code > that says > > from Tkinter import Widget > > class MyWidget(Widget): ... > > it correctly deduces that MyWidget's base class is Tkinter.Widget. > (And in a future version it will probably allow you to click through > to that class, show you its methods, etc.) > > Custom importers break this feature, and thus make the code harder to > analyze for the reader. That's why I tried argue for making it a standard: all tools will then know about the syntax used and take appropriate action. > (Not all custom importers are bad; there are plenty of reasons to > augment the standard importer. But, while custom importers make > different interpretations of the namespace possible, I frown upon that > particular use.) > > > (b) is (if at all) a problem only to be taken into account by > > the author of package X. He may or may not use rel. imports. > > A 12-year old probably won't (but then: you never know ;). > > Which severely limits your ability to do what you want with packages > you didn't write. > > I know that this sounds politically incorrect in a radical free > software world, but often in the end I find it more convenient to > conform to the rest of the world and "fit in" than to be different in > every little way. Note that my hair isn't blue :-); I've also > replaced my "signature" glasses with a more conventional model. Naa, I don't want to be different just to be plain silly different. The case here is one of maintenance. I have around 12 Python packages which I maintain, some of them public, others proprietary. Due to the large number I try to make them all as easy to handle as possible. Now I can't just go and rename the imports in case I want to put them under some other common package name for application A and then do the same with another package name for application B. I would have to do the same thing for every new release of the packages since there are also others who use the standard mx-based name. I just want the packages to work whereever I put them, e.g. I could link them into the new location using symlinks and be done with the work. Of course there are other ways to make this work (and I want to thank everybody who contributed to the sys.path ideas), but they just add omplexity to a whole different area. > > BTW, what is this CP4E thing you're talking about. If it's an > > syntax aware editor, I have a friend who is really interested > > in these things... could you send me an URL that I can send him ? > > Where have you been? It's on the python.org homepage, has been > discussed in c.l.py, c.l.tcl, even c.l.ada, in lwn, and on /.! The > syntax aware editor (a proposed super-version of IDLE) is only a small > part of it. See python.org/doc/essays/cp4e.html Nope, haven't heard anything about it -- perhaps I was on vacation. Sound interesting though. An editor where you can just type away (and the editor fills in the gaps) would be great fun. > > Oh well... I guess they'll have to use 'mex' if they decide to go > > the reverse domain way ;-) > > Or, in a worse-case scenario, the first Mexican developers using the > reverse domain will probably not be aware of the mx toplevel package, > and by the time their software hits the street it will be too late. Well, I guess then they'll have a good reason to upgrade their software. Just like all those companies who made extra money by sticking a "Y2000 + Euro-capable" button on their software. No, seriously, with the proposed extension (see below) this shouldn't be much of a problem because my packages all use capital first letters. The reverse domain name would use lower case ones, e.g. mx.pythonware.pilios or mx.zopistas... > > True, perhaps we should lighten this requirement a little when we > > recode the import mechanism in Python ? E.g. if a local import > > fails continue the search with the fully qualified name and only > > if that fails, restart using the local name. This would need some > > kind of fastpath cache to make the search reasonably fast though. > > Now this is something that I would consider. Should be easy to add to the imputil based core emulation. > > Wouldn't it suffice to just put them into one package, e.g. > > 'python.' ? > > And somehow make all user code live implicitly inside that package? I > don't see how that solves anything. No, to free the top-level namespace and make the origin of the imports clearer, e.g. import python.image import PIL.image import zope.image etc. > And believe me, at some point *someone* is going to > need to rip out your custom importer because it interferes with *his* > custom importer, and he'll choose to replace all your __ imports with > absolute package names, rather than trying to add the __ feature to > his importer. Note that with imputil doing its thing, he won't have to worry about adding code to his importer. It'll just pass on the import to my importer. That's the fun part about imputil. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Vladimir.Marangozov at inrialpes.fr Fri Sep 17 17:07:19 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Fri, 17 Sep 1999 16:07:19 +0100 (NFT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909151843.OAA18364@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 15, 99 02:43:56 pm" Message-ID: <199909171507.QAA33468@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > I hereby withdraw the posted license. There still is the need for a > new license, but we need to go back to the drawing board for it. So, in the end, am I still invited to sign & send the "wet" form or I'd better wait to let it dry? BTW, I'm surprised by the fact that in an Open Source world I'm asked to sign a licence agreement with CNRI or to send e-mails for contributed code. If Python or Linux had had such constraints from the start, they wouldn't have been what they are today. -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From guido at CNRI.Reston.VA.US Fri Sep 17 16:07:50 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 17 Sep 1999 10:07:50 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: Your message of "Fri, 17 Sep 1999 16:07:19 BST." <199909171507.QAA33468@pukapuka.inrialpes.fr> References: <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <199909171407.KAA00575@eric.cnri.reston.va.us> > So, in the end, am I still invited to sign & send the "wet" form or > I'd better wait to let it dry? Please send in the form -- the license was a totally separate issue that I shouldn't have brought up in the same mail (or at all, in this stage anyway -- we'll work this out with the Python consortium members first). > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. Unfortunately, that's the price we have to pay. What we get is legal protection from CNRI. In general CNRI has contributed a lot to Python; probably more than you realize. In any case, signing the form and including the email paragraphs is completely voluntary -- if you don't want to do it, just let me know. --Guido van Rossum (home page: http://www.python.org/~guido/) From Vladimir.Marangozov at inrialpes.fr Fri Sep 17 17:35:01 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Fri, 17 Sep 1999 16:35:01 +0100 (NFT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909171407.KAA00575@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 17, 99 10:07:50 am" Message-ID: <199909171535.QAA30684@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > Please send in the form -- the license was a totally separate issue > that I shouldn't have brought up in the same mail (or at all, in this > stage anyway -- we'll work this out with the Python consortium members > first). ok. > > > BTW, I'm surprised by the fact that in an Open Source world I'm asked > > to sign a licence agreement with CNRI or to send e-mails for contributed > > code. If Python or Linux had had such constraints from the start, they > > wouldn't have been what they are today. > > Unfortunately, that's the price we have to pay. What we get is legal > protection from CNRI. In general CNRI has contributed a lot to > Python; probably more than you realize. I realize that. But I also realize that in case of a problem, the wet form protects CNRI, not the contributor. Hm. And what happens if you get hit by a bus? Or in 100 years when we'll dance with angels in Paradise? Will Python stay bound to CNRI with little legal possibilities to detach it, in case our successors (or you) start working in another organization? IANAL and curious. -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From fdrake at acm.org Fri Sep 17 16:41:47 1999 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 17 Sep 1999 10:41:47 -0400 (EDT) Subject: [Python-Dev] License cleanup In-Reply-To: <199909171507.QAA33468@pukapuka.inrialpes.fr> References: <199909151843.OAA18364@eric.cnri.reston.va.us> <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <14306.21291.786632.89652@weyr.cnri.reston.va.us> Vladimir Marangozov writes: > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. You shouldn't be; the FSF certainly requires a signed copyright assignment from contributors. I had to sign one for a bunch of patches I made to oleo many years ago. It was a minor nuissance, but that's all. (The *sad* part is that there hasn't been a new release of oleo that could have included the patches for five years! ;) Just noticed that there is a release at ftp.gnu.org now; I'll have to take a look! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mal at lemburg.com Fri Sep 17 16:45:52 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 16:45:52 +0200 Subject: [Python-Dev] Methods and friends Message-ID: <37E25420.6C3F5E3D@lemburg.com> I was just looking into classobject.c again after probably more than a year and wondered... has anyone tried optimizing the method lookup and binding process recently ? It seems that for every method call the following happens: 1. the method is looked up in the instance dict; this fails 2. the method is looked up in the class dict; this succeeds and returns a function 3. the class then turns the function into a new unbound method 4. the instance sees the unbound method and creates a new bound method (deleting the unbound method) and returns it Two possible improvements: ? let the instance use a special class_getattr_ex function that does not do the extra unbound method step (3.) or ? make methods mutable and have step 4. insert the instance object into the method object to turn it into a bound method Has anyone tried this ? Does it produce anything noticeable w/r to method call speed ? BTW, why does the new module check for instance objects for the INSTANCE argument ? Methods can handle any object in this slot, just as they handle any callable object as "function". Could be put to some use when wrapping types into classes or vice versa... which is what motivated the above ;-) Just curious, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Fri Sep 17 17:09:40 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 17 Sep 1999 11:09:40 -0400 Subject: [Python-Dev] Methods and friends In-Reply-To: Your message of "Fri, 17 Sep 1999 16:45:52 +0200." <37E25420.6C3F5E3D@lemburg.com> References: <37E25420.6C3F5E3D@lemburg.com> Message-ID: <199909171509.LAA00729@eric.cnri.reston.va.us> > It seems that for every method call the following happens: > 1. the method is looked up in the instance dict; this fails > 2. the method is looked up in the class dict; this succeeds > and returns a function > 3. the class then turns the function into a new unbound method > 4. the instance sees the unbound method and creates a new > bound method (deleting the unbound method) and returns it Are you sure? As far as I know, steps 3 and 4 are combined when you do getattr on an instance: instance_getattr() calls instance_getattr1() which calls class_lookup(). The latter doesn't create an unbound method. instance_getattr1() then turns it into a bound method. --Guido van Rossum (home page: http://www.python.org/~guido/) From bwarsaw at cnri.reston.va.us Fri Sep 17 18:30:21 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Fri, 17 Sep 1999 12:30:21 -0400 (EDT) Subject: [Python-Dev] License cleanup References: <199909151843.OAA18364@eric.cnri.reston.va.us> <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <14306.27805.461540.758955@anthem.cnri.reston.va.us> >>>>> "VM" == Vladimir Marangozov writes: VM> BTW, I'm surprised by the fact that in an Open Source world VM> I'm asked to sign a licence agreement with CNRI or to send VM> e-mails for contributed code. If Python or Linux had had such VM> constraints from the start, they wouldn't have been what they VM> are today. Note that the FSF has been requiring signatures for a long time. Actually, their requirements are IMHO more onerous because they require you to assign your copyrights to the FSF. Their lawyers have told them that they cannot defend the copyright of, e.g. Emacs, unless they own the copyrights to the entire codebase (or at least, anything that they couldn't rip out, throw away, and not completely cripple the application). CNRI's viewpoint is less drastic, but still important. It means that you retain the copyright on the code (good for you), but you give us permission to use it as we see fit (good for us /and/ for you :). Otherwise, it would be possible for a malicious person to contribute something really vital and useful, wait for it to become indispensible, and then say, "oops, we really didn't mean to let you use that code, sorry!" -Barry From jim at interet.com Fri Sep 17 21:20:24 1999 From: jim at interet.com (James C. Ahlstrom) Date: Fri, 17 Sep 1999 15:20:24 -0400 Subject: [Python-Dev] Re: Path hacking References: <000601bf00e1$072a7d90$0801a8c0@bobcat> Message-ID: <37E29478.316C27C9@interet.com> Perhaps you have heard about the East coast US hurricane. It really hammered us here in New Jersey. I had trouble getting home last night due to high water. When I went to our usual Japanese lunch restaurant it had broken windows and was full of tree branches. Wow... Anyway, Mark Hammond wrote: > > No - the point is that the "1.5" key is "reserved" by a standard > install. Changing the string value actually allows you to have your > own subtree, and you can assume you own that. OK, now that I know the rules I will think about doing that. > > And I hate changing any registry entries at all. > > Well, you should learn to get over it! I will try. Of course, even if the registry helps ship Python apps on Windows, it is no help on Unix, and I care about that almost as much. > > My current "solution" is to use freeze to create a black-box > install, > > and worry about second Python installations and wasted storage when > it > > happens. > > well, IMO this is also the correct thing to do. any install that has > >100 files is fragile. So I do both - freeze the app, _and_ a custom > "sys.winver". Yes, but this requires a compiler. > > I was hoping that this thread whould result in a consensis of what > > to do, but it has not. > > There is a consesus for people with the same problem. Different > problems have different optimal solutions. The problem is that there is no reliable way to ship bullet-proof Python apps without recompiling and rebuilding Python. Each of us has his own pet hack to solve our own problem. Now there is talk of custom import hooks, and this is likely to result in each package requiring its own import hook! Aren't packages supposed to be software IC's? I hate to be a nag, but I will keep pushing for a single solution. Python is totally cross platform, and with all that machinery, there must be portable way to do that. I think this is good for Python. Please don't think I am trying to solve my own selfish problems. I have a compiler, I am happy using freeze, and I don't have any problems. Its just that Python would "sell" better and be more popular if a developer could read the documentation "How to ship and install your Python app in five minutes and make millions". This documentation currently reads "You can't". Jim Ahlstrom From gstein at lyra.org Sat Sep 18 00:14:04 1999 From: gstein at lyra.org (Greg Stein) Date: Fri, 17 Sep 1999 15:14:04 -0700 Subject: [Python-Dev] License cleanup References: <199909171507.QAA33468@pukapuka.inrialpes.fr> Message-ID: <37E2BD2C.72792E0F@lyra.org> Vladimir Marangozov wrote: >... > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for contributed > code. If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. Actually, this isn't surprising at all. The Free Software Foundation *requires* this kind of thing to be filed with them before you contribute code to the FSF. Essentially, it is a way for the FSF (and CNRI) to legally state that they own the copyright on the particular code. Without that, the contributor could come along later and claim a copyright on the code. The IBM folks who are working on Apache have provided legal releases to the Apache Software Foundation that basically states that IBM won't try to assume any rights under copyright law on the code they contribute to Apache. In fact, every time that I receive a patch for my mod_dav Apache module, the IBM guy attaches a release to the email that has the patch. In a pure, cooperative, world none of this would be necessary. However, the world simply doesn't work that way and all this stuff (licenses, copyrights, releases) is there to prevent Bad Things from happening. It isn't evil in itself, but simply a reflection of the business environment and the society that we're working within. Cheers, -g -- Greg Stein, http://www.lyra.org/ From tim_one at email.msn.com Sat Sep 18 08:38:27 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 02:38:27 -0400 Subject: [Python-Dev] License cleanup In-Reply-To: <14306.21291.786632.89652@weyr.cnri.reston.va.us> Message-ID: <000001bf01a0$6ae8cca0$022d153f@tim> [Vladimir Marangozov] > BTW, I'm surprised by the fact that in an Open Source world I'm asked > to sign a licence agreement with CNRI or to send e-mails for > contributed code. [Fred L. Drake, Jr.] > You shouldn't be; the FSF certainly requires a signed copyright > assignment from contributors. I had to sign one for a bunch of > patches I made to oleo many years ago. It was a minor nuissance, but > that's all. Except they add up: year after year, a new batch of stupid little requirements piles up on top of the last year's, and it's a ratchet effect -- always more, never less. The aggregate gets to be a real weariness on the soul. I had to laugh when Fran?ois Pinard happened to post this on c.l.py today: > ... > Would it be any volunteer, at least for taking care of filling the FSF > papers, if any are needed? I filled more than enough of those in my > life, I prefer to avoid the burden. Ask Barry how many years we've been trying to sign pymode over to the FSF <0.5 wink>. [Vlad] > If Python or Linux had had such constraints from the start, they > wouldn't have been what they are today. I sympathize, but that's really hard to say. You don't get pig-biting weary of this crap until you're my age . AFAIK, Berkeley has never beed sued over the BSD license, MIT over the X license, or the U of Arizona over the Icon license (none, really -- Icon is in the public domain). All the legal mumbo jumbo in the "modern" licenses is like wearing garlic around your neck to ward off vampires: the threat isn't real, and if it were it wouldn't do you any good anyway. a-wooden-stake-thru-the-heart-is-your-only-true-defense-ly y'rs - tim From Vladimir.Marangozov at inrialpes.fr Sat Sep 18 16:17:28 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Sat, 18 Sep 1999 15:17:28 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) Message-ID: <199909181417.PAA20866@pukapuka.inrialpes.fr> The relative import thread has hit me with full force. I'm not sure I understand all of what has been said, but since I know this is mainly a naming issue, I want to point you to one of the best CS papers I have had in my hands. It's not really my style to recommend references, but this one should be of major interest to this audience (and Python respectively): Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. It's not available online (it was written on a typewriter), so I'd be happy to send a hard copy of it to anyone who raises a hand in private mail (or cannot find LNCS 60). It's simply a jewel. This guy has understood everything (and I suspect Guido has read this one before implementing Python ;-) -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From tim_one at email.msn.com Sun Sep 19 01:44:49 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 19:44:49 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909160332.XAA19278@eric.cnri.reston.va.us> Message-ID: <000001bf022f$cc11cba0$072d153f@tim> [Guido] > ... > In the Java world, the right solution exist: a better top-level naming > scheme. I forget what the exact rules are, but I seem to recall that > Java has something of the same relative import rules af Python > currently has: you can import a class from the same leaf package by > its simple name, or you can import a class by its full name. You can import only by the full name. There are no shortcuts for anyone ever in explicit Java imports. The dodge is that to reference classes in exactly the same package, you don't need an import statement at all (you *can* have one, but then you need to use the fully qualified name). The other twist is that all Java code is in *some* package, and code that (wisely) doesn't want to live in the ill-defined unnamed package must contain a package statement declaring which package it's a part of, like package COM.dragonsys.justanexample; The periods there actually have no semantic significance, and package COM.dragonsys.notanexample has no more in common with the above than does package ORG.python.guido.hitmeagain. It's purely a conceptual thing that justanexample and notanexample are *called* "subpackages" of package COM.dragonsys in Java; although implementations are free (but not required) to make up mappings between dotted package names and directory trees. Anyway, the Java rules work fine wrt the issues that have been argued in this thread. Things that are worse than Python: + Magical implied import of same-package names is a PITA for readability and maintenance. + Since subpackages are an illusion, and even packages aren't objects in their own right, there's no possibility to create a shorthand local name for the first M chunks of an N-chunk dotted access path (with M < N). For example, you may use java.awt.Font and java.awt.TextArea a lot. You either spell those out in full all the time, or do import java.awt.Font; import java.awt.TextArea; and use just "Font" and "TextArea" later. Making up a local name for just the "java.awt" part isn't an option. Which leads to gross overuse of the following: + "import package.*" is Java's way of spelling "from package import *", and because of the preceding is the only way to get convenient local names for classes imported from other packages (note that can you can never import a package/module in Java; you can only import a type name). So most Java code will do the above as import java.awt.*; and end up importing a gazillion names. This sucks for the same reasons "import *" sucks in Python, although Java catches the name conflicts at compile time. Things that are better than Python: + Name conflicts are caught at compile time. + At least in theory, the names of packages aren't constrained by platform file name limitations or directory quirks. + Relatedly, explicitly naming the package a compilation unit belongs can be a Good Thing. + They have *an* effective scheme for preventing package-name collisions. > ... > Note that I still resist the urge to start renaming all the existing > standard modules -- too much breakage. This will happen in 2.0. At home, I dropped an empty __init__.py into the std Lib directory, so that I can do e.g. from Lib.random import randrange in my own packages with their own random modules. Works fine. For 1.6 you should pick some explicit way to *allow* explicit import of at least the std library modules. I suggest taking the name "Lib" . Java is very cute with its java.util.etc stuff, but longwindedness is not a virtue here -- "util" would have worked just as well and been just as clear. Nobody should write a package with a generic short name like "util" (or "Lib") anyway, so rather than waste this pleasant little corner of the namespace the language may as well reserve it for its own use. heck-it-already-stole-"and"-and-"or"-ly y'rs - tim From tim_one at email.msn.com Sun Sep 19 05:18:43 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:43 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37E20AF7.2D05F371@lemburg.com> Message-ID: <000001bf024d$ae271f00$f72d153f@tim> [MAL] > ... > Of course there are other ways to make this work (and I want > to thank everybody who contributed to the sys.path ideas), but > they just add complexity to a whole different area. The complexity is already in that area, and for all of us. It can be dealt with. Adding new complexity elsewhere doesn't decrease the complexity that's already there, it just adds more and new kinds of stuff to worry about. [Guido] >> ... >> And believe me, at some point *someone* is going to need to rip out >> your custom importer because it interferes with *his* custom importer, >> and he'll choose to replace all your __ imports with absolute package >> names, rather than trying to add the __ feature to his importer. > Note that with imputil doing its thing, he won't have to > worry about adding code to his importer. It'll just pass on the > import to my importer. That's the fun part about imputil. As I recall, the old ni took the initial "__" as referring to the *current* package. So where you intend to write import __.sibling.func it was written under ni as import __.__.sibling.func Since ni is semi-blessed prior art, some other joker is going to slam in an Importer to use those rules. I mix your packages with theirs, and then the meaning of "__" depends on whose Importer sees the damn thing first. Again complexity spreads. If Jim is wary of top-level name collisions now, wait until he thinks about naming-gimmick collisions . Guido is telling the truth: no package is going into Dragon's products unless it's rewritten to purge dicey name tricks. I think we're all in favor of Greg's imputil approach, but where it adds real value (primarily in allowing imports to get satisfied from sources other than direct file images). if-it's-in-a-file-you-just-import-it-by-name-ly y'rs - tim From tim_one at email.msn.com Sun Sep 19 05:18:52 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:52 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <37E0CCC1.6EE4E599@lemburg.com> Message-ID: <000201bf024d$b3213ae0$f72d153f@tim> [MAL] > ... > Seems that we're a bit too healthy (see MarkH's post) sometimes, > i.e. there isn't all that much room for experiments. The odds of a change making it into Python radically decreased when 1.0 hit the net, and have continued to decline (although slowly) since then. In recent years, Guido appears to me to have gotten ever more reluctant to entertain even 100% compatible changes to the internals, if they affect a delicate area of the implementation (ceval.c is the most obvious one there). But that's "normal & healthy" too. Languages & implementations get brittle with age, and it's eventually better to start over -- if Guido didn't have Python2 plans in mind, he'd be the first language designer ever to stop where he started! > Just think of cool developments like Chris' stackless python. Experience > shows that these kind of things will never make it into the distribution. Unfortunately, circumstances piled up and Chris got distracted from that, while nobody else made time to push it in his absence. Large changes have gone in, and even more may make it into the Python1 line, but it generally takes a large or "strategic" user base, and much persistence. GregS mentioned his massive work on threads (still not all in), and I'll add the NumPy extensions (which I wouldn't be surprised to see "mainstreamed"), BarryW's string methods, and DavidA's rich comparisons. > Unfortunately, maintaing patches to the dist across releases a real > pain and much work, so these ideas will just sit there unused and > untested. Much the same happened to gcc ... in the end corporate > strength made egcs possible. Perhaps we need such a branch too ? Don't tell, but I've always been surprised at how few people have tried to release a variant Python! The Alice version (case-insensitive names, and 1/2==0.5) is the only one that comes to mind, and the primary effect that had on today's Python is that raw expressions no longer print their value in non-interactive mode (before Alice, 1 + 2 on a line by itself caused "3" to get printed even in batch scripts; this interfered with the Alice team's favored object.method1().method2().method3() coding style, and Guido endured much pain to change "the real" Python to avoid a code split at that early stage of Python's life; ultimately futile, but then Alice Python didn't catch on anyway). So there's very little Python-related history to go on here. I don't mind seeing variants, but have to predict they won't get very far. Just picture what Python 1.6V would look like if its feature set were drawn from a consensus among you, me, Christian, Greg Ewing, John Skaller and Tom Christiansen . the-value-of-a-benevolent-dictator-is-easy-to-underestimate-ly y'rs - tim From tim_one at email.msn.com Sun Sep 19 05:18:47 1999 From: tim_one at email.msn.com (Tim Peters) Date: Sat, 18 Sep 1999 23:18:47 -0400 Subject: [Python-Dev] Relative Package Imports In-Reply-To: <199909161149.HAA19650@eric.cnri.reston.va.us> Message-ID: <000101bf024d$af986240$f72d153f@tim> [Guido] > I know that this sounds politically incorrect in a radical free > software world, but often in the end I find it more convenient to > conform to the rest of the world and "fit in" than to be different in > every little way. Note that my hair isn't blue :-); I've also > replaced my "signature" glasses with a more conventional model. !!! No wonder you tried to slam a conventional license down our throats. My advice is to get back the old glasses, and walk around the halls at CNRI counting backwards from 1000 by seventeens loudly and incessantly. you'd-be-amazed-at-how-few-lawyers-talk-to-me-ly y'rs - tim From bwarsaw at cnri.reston.va.us Sun Sep 19 09:01:29 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Sun, 19 Sep 1999 03:01:29 -0400 (EDT) Subject: [Python-Dev] Relative Package Imports References: <199909160332.XAA19278@eric.cnri.reston.va.us> <000001bf022f$cc11cba0$072d153f@tim> Message-ID: <14308.35401.20723.782353@anthem.cnri.reston.va.us> >>>>> "TP" == Tim Peters writes: TP> + "import package.*" is Java's way of spelling "from package TP> import *", and because of the preceding is the only way to get TP> convenient local names for classes imported from other TP> packages (note that can you can never import a package/module TP> in Java; you can only import a type name). So most Java code TP> will do the above as TP> import java.awt.*; TP> and end up importing a gazillion names. This sucks for the TP> same reasons "import *" sucks in Python, although Java catches TP> the name conflicts at compile time. The interesting this is that, while the Java developers did this at the language level, at the VM level, every class is "fully qualified"; you see the absolute path for every class name. import pkg.pkg.* is the reason why you still have to have unique named classes like PyThinging or JWiggie, and yup it sucks. -Barry From mal at lemburg.com Fri Sep 17 17:53:38 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 17 Sep 1999 17:53:38 +0200 Subject: [Python-Dev] Methods and friends References: <37E25420.6C3F5E3D@lemburg.com> <199909171509.LAA00729@eric.cnri.reston.va.us> Message-ID: <37E26402.3C519434@lemburg.com> Guido van Rossum wrote: > > > It seems that for every method call the following happens: > > 1. the method is looked up in the instance dict; this fails > > 2. the method is looked up in the class dict; this succeeds > > and returns a function > > 3. the class then turns the function into a new unbound method > > 4. the instance sees the unbound method and creates a new > > bound method (deleting the unbound method) and returns it > > Are you sure? As far as I know, steps 3 and 4 are combined when you > do getattr on an instance: instance_getattr() calls > instance_getattr1() which calls class_lookup(). The latter doesn't > create an unbound method. instance_getattr1() then turns it into a > bound method. Oops. Correct. I should have looked a little closer. So it already works in the first way I mentioned as improvement... your time machine again ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 105 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 20 10:59:58 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 20 Sep 1999 10:59:58 +0200 Subject: [Python-Dev] Relative Package Imports References: <000201bf024d$b3213ae0$f72d153f@tim> Message-ID: <37E5F78E.3859D12F@lemburg.com> Tim Peters wrote: > > [MAL] > > ... > > Seems that we're a bit too healthy (see MarkH's post) sometimes, > > i.e. there isn't all that much room for experiments. > > The odds of a change making it into Python radically decreased when 1.0 hit > the net, and have continued to decline (although slowly) since then. In > recent years, Guido appears to me to have gotten ever more reluctant to > entertain even 100% compatible changes to the internals, if they affect a > delicate area of the implementation (ceval.c is the most obvious one there). > > But that's "normal & healthy" too. Languages & implementations get > brittle with age, and it's eventually better to start over -- if Guido > didn't have Python2 plans in mind, he'd be the first language designer ever > to stop where he started! > > > Just think of cool developments like Chris' stackless python. Experience > > shows that these kind of things will never make it into the distribution. > > Unfortunately, circumstances piled up and Chris got distracted from that, > while nobody else made time to push it in his absence. Large changes have > gone in, and even more may make it into the Python1 line, but it generally > takes a large or "strategic" user base, and much persistence. GregS > mentioned his massive work on threads (still not all in), and I'll add the > NumPy extensions (which I wouldn't be surprised to see "mainstreamed"), > BarryW's string methods, and DavidA's rich comparisons. Plus the coercion stuff that's still sleeping in one of my project subdirs (I'll have to get this done *before* 1.6 hits the shelves). > > Unfortunately, maintaing patches to the dist across releases a real > > pain and much work, so these ideas will just sit there unused and > > untested. Much the same happened to gcc ... in the end corporate > > strength made egcs possible. Perhaps we need such a branch too ? > > Don't tell, but I've always been surprised at how few people have tried to > release a variant Python! The Alice version (case-insensitive names, and > 1/2==0.5) is the only one that comes to mind, and the primary effect that > had on today's Python is that raw expressions no longer print their value in > non-interactive mode (before Alice, > > 1 + 2 > > on a line by itself caused "3" to get printed even in batch scripts; this > interfered with the Alice team's favored > > object.method1().method2().method3() > > coding style, and Guido endured much pain to change "the real" Python to > avoid a code split at that early stage of Python's life; ultimately futile, > but then Alice Python didn't catch on anyway). > > So there's very little Python-related history to go on here. I don't mind > seeing variants, but have to predict they won't get very far. Just picture > what Python 1.6V would look like if its feature set were drawn from a > consensus among you, me, Christian, Greg Ewing, John Skaller and Tom > Christiansen . Actually, what I was thinking about here was a Python 2.0 branch starting now rather than in a year or so and thus leaving much room for experiments etc. The intention was the same as with egcs and gcc: to fold the enhancements back into the main branch in a few years. E.g. if Guido points us in the right direction, we could start hacking on that piece of revolutionary work now. BTW, I'd suggest using C++ with namespaces but without templates as target language. By the time Python 2.0 will hit the shelves this setup should have reached the same portability as C has now. Perhaps we could even use RTTI (run time type information) to implement optional type safety... ok, just dreaming a little ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 102 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From skip at mojam.com Mon Sep 20 17:29:55 1999 From: skip at mojam.com (Skip Montanaro) Date: Mon, 20 Sep 1999 10:29:55 -0500 Subject: [Python-Dev] Tackling circular dependencies in 2.0? Message-ID: <199909201529.KAA21605@dolphin.mojam.com> I think in Python 2.0 it would be nice to have some way to reclaim circular dependencies without the programmer explicitly having to do something like implement a destroy() method and requiring other programmers to (remember to) call it. I forget what the current state of affairs is w.r.t. future memory management in Python. Not knowing anything much about memory management, would it be possible to have a sort of mixed ref count/garbage collection system where you only use the gc stuff as a last resort? My thought is that it would be useful to use gc to find and reclaim circular garbage. can-you-tell-I-just-got-bitten-by-a-circular-reference?-ly y'rs Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From mal at lemburg.com Mon Sep 20 19:44:57 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 20 Sep 1999 19:44:57 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <37E67299.60899D38@lemburg.com> Skip Montanaro wrote: > > I think in Python 2.0 it would be nice to have some way to reclaim circular > dependencies without the programmer explicitly having to do something like > implement a destroy() method and requiring other programmers to (remember > to) call it. I forget what the current state of affairs is w.r.t. future > memory management in Python. Not knowing anything much about memory > management, would it be possible to have a sort of mixed ref count/garbage > collection system where you only use the gc stuff as a last resort? My > thought is that it would be useful to use gc to find and reclaim circular > garbage. > > can-you-tell-I-just-got-bitten-by-a-circular-reference?-ly y'rs If you don't want to wait until 2.0 becomes GA, you could try weak references: http://starship.skyport.net/~lemburg/mxProxy.html -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 102 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gward at cnri.reston.va.us Mon Sep 20 19:53:03 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Mon, 20 Sep 1999 13:53:03 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909201529.KAA21605@dolphin.mojam.com>; from Skip Montanaro on Mon, Sep 20, 1999 at 10:29:55AM -0500 References: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <19990920135302.B393@cnri.reston.va.us> > Not knowing anything much about memory > management, would it be possible to have a sort of mixed ref count/garbage > collection system where you only use the gc stuff as a last resort? My > thought is that it would be useful to use gc to find and reclaim circular > garbage. That's *sorta* what Perl does, 'though you can still be bitten by circular refs in a long-running process. Err, long-running thread. You see, Perl has a mark-and-sweep garbage collector which is run only on thread shutdown. From gward at cnri.reston.va.us Mon Sep 20 19:57:51 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Mon, 20 Sep 1999 13:57:51 -0400 Subject: [Python-Dev] Preview of some distutils news Message-ID: <19990920135750.D393@cnri.reston.va.us> Judging by the recent traffic on python-dev, lots of people here are interested in the problems of distributing and installing Python modules. So I thought y'all might want to know that I've finally got some interesting news to report on the distutils-sig, and will shortly be reporting it, soliciting feedback, testers, etc. If you're not already on that list, this might be a good time to join. (Oh, the news? Distutils can now compile and install NumPy, PIL, and mxDateTime (just because I happened to have those three distributions sitting around on my home PC). Of course there are all sorts of caveats and limitations, so c'mon over to the distutils-sig if you want to hear what they are and discuss how to fix them. Later this week, of course, after I've checked in all the relevant code.) Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From guido at CNRI.Reston.VA.US Mon Sep 20 20:07:03 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Mon, 20 Sep 1999 14:07:03 -0400 Subject: [Python-Dev] wet signature form clarification Message-ID: <199909201807.OAA07799@eric.cnri.reston.va.us> Several of you have asked me a clarification of the "contribution description" required on the wet signature form; given that you are all very prolific contributors with a weak memory, it would be nice if you could write something like "everything I've contributed until Sept. 18, 1999" in the space there rather than dig out all the specific module names. I got a call from CNRI's lawyer today, and she agreed that this was fine. Boy am I pleased today! I think she called it a "group description" or something like that. Anyway, if you were waiting for a ruling on this issue, please send in your signed form! (I'd appreciate it if the signature was dry by the time I got it. :-) The form is at http://python.org/1.5/wetsign.html --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one at email.msn.com Tue Sep 21 08:11:00 1999 From: tim_one at email.msn.com (Tim Peters) Date: Tue, 21 Sep 1999 02:11:00 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909201529.KAA21605@dolphin.mojam.com> Message-ID: <000101bf03f8$13ade740$ae2d153f@tim> [Skip Montanaro] > I think in Python 2.0 it would be nice to have some way to > reclaim circular dependencies without the programmer explicitly > having to do something ... This was debated (again) at great length on c.l.py just a few months ago. Guido chimed in with a proposal to keep track of only the dicts that have been allocated, and now and again mark everything reachable from the root set and nuke whatever dicts don't end up marked. Cycles involving dicts would get reclaimed this way, but not cycles not involving dicts. The approach to destructors for objects in cycles was "tough -- they don't get called". What to do about destructors for objects that are not themselves involved in cycles but are reachable only from dead cycles (so are in fact dead too) wasn't addressed. Seemed possible that stuff reachable from ordinary dicts (not in a cycle, and neither reachable from a cycle) would behave differently than today, since the "list of all dicts" may keep the stuff artificially alive until the next mark+sweep, even if the refcount on the stuff fell to zero; there's probably an OK way around that, though. Anyway, Guido was aiming for the minimal changes that could possibly do real good. It didn't pretend to reclaim all cycles, and was (IMO) too eager to punt on the hard issues (the combo of cycles, destructors and resurrection is a god-awful mess, even in theory; Scheme uses callbacks to dump the problems back on the users Java has incredibily elaborate rules that are both bulletproof and unusable; the Boehm collector lets objects with destructors that are in cycles simply leak, rather than do a wrong thing; Stroustrup has flip-flopped and most recently argued for Guido's "reclaim the memory but don't call the destructors" approach, but a member of the C++ committee told me he's overwhelmingly opposed on this one (I know I would oppose it)). In any case, nothing has come of it, and no easy principled solution is in sight. OTOH, if Guido balks at explaining what "__" means to 12-year-olds, wait until he tries to explain immortal cyclic trash . Perl scores points for its brute-force end-of-thread M&S, but I believe complex user-level data structures are much rarer in Perl, simply due to the clumsiness of the syntax and explicit reference model. Perl's version of "nested functions" don't actually nest (all "def"s are floated to the top level by the compiler, regardless of how deeply they're nested), so Perl's lexical closures don't create cyclic trash either (well, they do, but in the same sense there's a cycle between a Python module namespace and the functions in that module -- Perl also special-cases the snot out of those and busts those cycles by brute force). So there you go! It needs to be solved and nobody has a clue . if-java-hype-hadn't-suffered-exponential-decay-we-could-have- dumped-it-on-the-jvm-by-now-ly y'rs - tim From mal at lemburg.com Tue Sep 21 11:05:05 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 11:05:05 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> Message-ID: <37E74A41.107A1243@lemburg.com> Tim Peters wrote: > > [Skip Montanaro] > > I think in Python 2.0 it would be nice to have some way to > > reclaim circular dependencies without the programmer explicitly > > having to do something ... > > This was debated (again) at great length on c.l.py just a few months ago. > Guido chimed in with a proposal to keep track of only the dicts that have > been allocated, and now and again mark everything reachable from the root > set and nuke whatever dicts don't end up marked. Cycles involving dicts > would get reclaimed this way, but not cycles not involving dicts. The > approach to destructors for objects in cycles was "tough -- they don't get > called". What to do about destructors for objects that are not themselves > involved in cycles but are reachable only from dead cycles (so are in fact > dead too) wasn't addressed. Seemed possible that stuff reachable from > ordinary dicts (not in a cycle, and neither reachable from a cycle) would > behave differently than today, since the "list of all dicts" may keep the > stuff artificially alive until the next mark+sweep, even if the refcount on > the stuff fell to zero; there's probably an OK way around that, though. You could probably tackle the problem by doing local mark&sweep whenever the ref count on a dictionary falls down to 1 (meaning that it is only referenced from the list of all dicts). This is what I do in mxProxy's weak reference implementation and to my surprise it solved all those strange situations where objects are kept alive longer than they would have normally. > Anyway, Guido was aiming for the minimal changes that could possibly do real > good. It didn't pretend to reclaim all cycles, and was (IMO) too eager to > punt on the hard issues (the combo of cycles, destructors and resurrection > is a god-awful mess, even in theory; Scheme uses callbacks to dump the > problems back on the users Java has incredibily elaborate rules that are > both bulletproof and unusable; the Boehm collector lets objects with > destructors that are in cycles simply leak, rather than do a wrong thing; > Stroustrup has flip-flopped and most recently argued for Guido's "reclaim > the memory but don't call the destructors" approach, but a member of the C++ > committee told me he's overwhelmingly opposed on this one (I know I would > oppose it)). Not calling the destructor will cause leakage in all objects allocating extra storage, such as lists, instances and probably just about any dynamically sized object there is in Python... solving the problem only half way. Plus you will definitely run into trouble as soon as external resources are involved, e.g. open files or connections to databases. Perhaps we should give more power to the user instead of trying to give him fuzzy feelings about what's happening underneath the hood. Builtin weak references or other indirect ways of accessing objects (e.g. by giving unique names to the involved objects) can solve many of those circ. ref. problems. BTW, I usually use an instrumented Python interpreter to track down circular references: it uses a tracing hook in the allocation/deallocation code of Python instances which is used when Python is run in debugging mode (python -d). The hook calls a function sys.traceinstances (if present) which allows me to keep a track record of all allocated instances: def traceinstances(action,inst): """ Tracing hook. This is called whenever an instances is created and destroyed. action is either 'create' or 'delete'; inst points to the instance object. """ ... If anyone is interested I can post the patch (against Python 1.5). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Tue Sep 21 15:13:19 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 21 Sep 1999 09:13:19 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: Your message of "Tue, 21 Sep 1999 11:05:05 +0200." <37E74A41.107A1243@lemburg.com> References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> Message-ID: <199909211313.JAA09783@eric.cnri.reston.va.us> > Not calling the destructor will cause leakage in all objects > allocating extra storage, such as lists, instances and > probably just about any dynamically sized object there is in > Python... solving the problem only half way. Plus you will > definitely run into trouble as soon as external resources > are involved, e.g. open files or connections to databases. If I remember well, the only destructors not called would be __del__ methods, since the dependencies between to-be-deleted instances are unknown to the collector. Regular (C-level) destructors would of course be called. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at mojam.com Tue Sep 21 17:33:31 1999 From: skip at mojam.com (Skip Montanaro) Date: Tue, 21 Sep 1999 10:33:31 -0500 (CDT) Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <37E74A41.107A1243@lemburg.com> References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> Message-ID: <14311.42315.295460.174012@dolphin.mojam.com> Marc> BTW, I usually use an instrumented Python interpreter to track Marc> down circular references: ... Marc> If anyone is interested I can post the patch (against Python 1.5). That would be interesting to look at. I found my latest circular reference by building Python with Py_DEBUG defined and trudging through the output at the end. Skip Montanaro | http://www.mojam.com/ skip at mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented... From mal at lemburg.com Tue Sep 21 18:23:53 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 18:23:53 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> <14311.42315.295460.174012@dolphin.mojam.com> Message-ID: <37E7B119.5A695349@lemburg.com> Skip Montanaro wrote: > > Marc> BTW, I usually use an instrumented Python interpreter to track > Marc> down circular references: ... > > Marc> If anyone is interested I can post the patch (against Python 1.5). > > That would be interesting to look at. I found my latest circular reference > by building Python with Py_DEBUG defined and trudging through the output at > the end. Here it is: --- Objects/orig/classobject.c Thu Jan 1 20:39:04 1998 +++ Objects/classobject.c Sat Aug 8 21:38:57 1998 @@ -403,10 +588,37 @@ PyInstance_New(class, arg, kw) inst = NULL; } Py_DECREF(res); } } + /* sys.traceinstances hook */ + if (Py_DebugFlag && inst) { + PyObject *fct; + + fct = PySys_GetObject("traceinstances"); + if (fct) { + PyObject *v,*arg; + PyObject *error_type, *error_value, *error_traceback; + + /* Save and clear any exception */ + PyErr_Fetch(&error_type,&error_value, + &error_traceback); + PyErr_Clear(); + arg = Py_BuildValue("(sO)","create",(PyObject *)inst); + v = PyEval_CallObject(fct,arg); + Py_DECREF(arg); + if (!v) { + PyErr_Print(); + PyErr_Clear(); + } + else + Py_DECREF(v); + /* Restore exception state */ + PyErr_Restore(error_type,error_value, + error_traceback); + } + } return (PyObject *)inst; } /* Instance methods */ @@ -460,10 +672,31 @@ instance_dealloc(inst) } else Py_DECREF(res); Py_DECREF(del); } + /* sys.traceinstances hook */ + if (Py_DebugFlag) { + PyObject *fct; + + fct = PySys_GetObject("traceinstances"); + if (fct) { + PyObject *v,*arg; + + /* Clear any previous exception */ + PyErr_Clear(); + arg = Py_BuildValue("(sO)","delete",(PyObject *)inst); + v = PyEval_CallObject(fct,arg); + Py_DECREF(arg); + if (!v) { + PyErr_Print(); + PyErr_Clear(); + } + else + Py_DECREF(v); + } + } /* Restore the saved exception and undo the temporary revival */ PyErr_Restore(error_type, error_value, error_traceback); /* Can't use DECREF here, it would cause a recursive call */ if (--inst->ob_refcnt > 0) { #ifdef COUNT_ALLOCS -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Tue Sep 21 18:43:22 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 21 Sep 1999 18:43:22 +0200 Subject: [Python-Dev] Tackling circular dependencies in 2.0? References: <000101bf03f8$13ade740$ae2d153f@tim> <37E74A41.107A1243@lemburg.com> <199909211313.JAA09783@eric.cnri.reston.va.us> Message-ID: <37E7B5AA.6196A3B@lemburg.com> Guido van Rossum wrote: > > > Not calling the destructor will cause leakage in all objects > > allocating extra storage, such as lists, instances and > > probably just about any dynamically sized object there is in > > Python... solving the problem only half way. Plus you will > > definitely run into trouble as soon as external resources > > are involved, e.g. open files or connections to databases. > > If I remember well, the only destructors not called would be __del__ > methods, since the dependencies between to-be-deleted instances are > unknown to the collector. Regular (C-level) destructors would of > course be called. Ok, so low-stuff will not break. But what about e.g. wrappers around these low-level (C-level) objects written in Python, e.g. database abstraction classes ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 101 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From tim_one at email.msn.com Wed Sep 22 07:56:05 1999 From: tim_one at email.msn.com (Tim Peters) Date: Wed, 22 Sep 1999 01:56:05 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: <199909211313.JAA09783@eric.cnri.reston.va.us> Message-ID: <000801bf04bf$287e6560$342d153f@tim> [Guido van Rossum] > If I remember well, the only destructors not called would be __del__ > methods, since the dependencies between to-be-deleted instances are > unknown to the collector. Regular (C-level) destructors would of > course be called. Seems to me that the dependencies among to-be-deleted arbitrary C objects are equally unknown to the collector. Assuming there's a fundamental difference between objects implemented in Python and objects implemented in C seems shaky on the face of it. Or if it's not just a convenient assumption, on what is it based? and-what-about-objects-implemented-in-fortran?-ly y'rs - tim From mal at lemburg.com Wed Sep 22 14:47:03 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 22 Sep 1999 14:47:03 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports Message-ID: <37E8CFC7.673E040A@lemburg.com> Hi everybody, I'm currently busy converting my code to use mx.DateTime instead of just DateTime and must say that this is a real pain. Now instead of reviving the __ discussion I'd like to turn to what could be a workable compromise (I think Jim proposed something along these lines already and there was similar code in ni.py). The proposal is really only an addition to the lookup scheme used by the importer. No additional syntax is involved and best of all, it is backward compatible... The current lookup does the following if you want to import a module E from module A.B.C.D: 1. check A.B.C.E 2. check E 3. fail Now instead of failing we could add a lookup method that walks up the package structure: 3. check A.B.E 4. check A.E [5. check E -- already done] 6. fail so that the complete scheme looks like this: 1. check A.B.C.E 2. check E 3. check A.B.E 4. check A.E [5. check E -- already done] 6. fail That way I could leave intra-mx-package imports untouched and still have the convenience of achieving the goal of making my mx subpackages work in the mx context *plus* in the top-level context thus allowing a backward compatible and flexible setup for mx* users. Note that the scheme finds exactly the same modules it did previously, plus perhaps some more (which is intended), and it does not involve any search path hacks. How is that for a compromise ? [Ducking for cover ;-)] -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Wed Sep 22 15:12:40 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Wed, 22 Sep 1999 09:12:40 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37E8CFC7.673E040A@lemburg.com> Message-ID: <1274116686-66496233@hypernet.com> [M.-A. Lemburg] > The current lookup does the following if you want to import > a module E from module A.B.C.D: > > 1. check A.B.C.E > 2. check E > 3. fail > > Now instead of failing we could add a lookup method that > walks up the package structure: > > 3. check A.B.E > 4. check A.E > [5. check E -- already done] > 6. fail > > so that the complete scheme looks like this: > > 1. check A.B.C.E > 2. check E > 3. check A.B.E > 4. check A.E > [5. check E -- already done] > 6. fail > > That way I could leave intra-mx-package imports untouched and > still have the convenience of achieving the goal of making my mx > subpackages work in the mx context *plus* in the top-level > context thus allowing a backward compatible and flexible setup > for mx* users. Comment 1: You're just giving yourself headaches by allowing your users to install mx in anything other than the prescribed manner. Comment 2: I generally like this scheme, but think (for consistency and confusion-reduction) that it should go straight up the tree, instead of checking the root second. - Gordon From jim at interet.com Wed Sep 22 15:19:58 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 22 Sep 1999 09:19:58 -0400 Subject: [Python-Dev] Re: Path hacking References: <37E0FC7D.7494B8F7@lyra.org> <1274619162-36273021@hypernet.com> <14305.11672.687570.290221@dolphin.mojam.com> <37E12D4C.2394D311@lyra.org> Message-ID: <37E8D77E.2977C61B@interet.com> Greg Stein wrote: > > Skip Montanaro wrote: > > > > Gordon> In the soon-to-be-published beta version of my installer, I've > > Gordon> got that down to exceptions.py. > > > > Why not just run exceptions.py through Python2C, visually and experimentally > > verify that it works, then ship an exceptions.c as an optional module? Optional module... Good idea. But it is easier to use freeze within the optional module. So I wrote a "boot" built in module, where bootmodule.c is created by the Python program bootmake.py. All this lives in ./Modules. Boot incorporates exceptions.pyc etc. as frozen modules without breaking the current freeze feature. I changed pythonrun.c to load "boot" after sys but before any other imports, but only if it exists as a built in module, otherwise no error. Boot has methods to turn it on and off, and to print its contents. Please take a look at ftp://ftp.interet.com/pub/bootmodule.html I think this is a good solution for how to build in imputil.py. Jim Ahlstrom From guido at CNRI.Reston.VA.US Wed Sep 22 16:53:55 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 22 Sep 1999 10:53:55 -0400 Subject: [Python-Dev] Tackling circular dependencies in 2.0? In-Reply-To: Your message of "Wed, 22 Sep 1999 01:56:05 EDT." <000801bf04bf$287e6560$342d153f@tim> References: <000801bf04bf$287e6560$342d153f@tim> Message-ID: <199909221453.KAA13321@eric.cnri.reston.va.us> > [Guido van Rossum] > > If I remember well, the only destructors not called would be __del__ > > methods, since the dependencies between to-be-deleted instances are > > unknown to the collector. Regular (C-level) destructors would of > > course be called. [Tim] > Seems to me that the dependencies among to-be-deleted arbitrary C objects > are equally unknown to the collector. Assuming there's a fundamental > difference between objects implemented in Python and objects implemented in > C seems shaky on the face of it. Or if it's not just a convenient > assumption, on what is it based? My assumption was based on the standard Python objects. Cycles necessarily have to include dictionaries or lists (modules, classes and instances link to each other through dictionaries; ditto for function objects; i'm crossing my fingers here for stack frame and traceback objects :-) and I can do things to these to get rid of the links without getting rid of the objects: del L[:] or D.clear(). Third party C objects might have interdependencies similar to those found in Python instances, but my gut feeling is that these aren't as problematic -- e.g. interdependencies between C modules are rare (because the machinery is cumbersome) while they are common between Python modules; and C code isn't susceptible to the problems that Python destructors encounter when the modules they import have already been destroyed. I know, it's an unusual amount of handwaving... --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at lemburg.com Wed Sep 22 17:13:06 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 22 Sep 1999 17:13:06 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> Message-ID: <37E8F202.2A27508D@lemburg.com> Gordon McMillan wrote: > > [M.-A. Lemburg] > > The current lookup does the following if you want to import > > a module E from module A.B.C.D: > > > > 1. check A.B.C.E > > 2. check E > > 3. fail > > > > Now instead of failing we could add a lookup method that > > walks up the package structure: > > > > 3. check A.B.E > > 4. check A.E > > [5. check E -- already done] > > 6. fail > > > > so that the complete scheme looks like this: > > > > 1. check A.B.C.E > > 2. check E > > 3. check A.B.E > > 4. check A.E > > [5. check E -- already done] > > 6. fail > > > > That way I could leave intra-mx-package imports untouched and > > still have the convenience of achieving the goal of making my mx > > subpackages work in the mx context *plus* in the top-level > > context thus allowing a backward compatible and flexible setup > > for mx* users. > > Comment 1: You're just giving yourself headaches by allowing > your users to install mx in anything other than the prescribed > manner. Actually, I'm trying to provide them a way to smoothly switch from the old setup to the new one. This includes myself, of course ;-). > Comment 2: I generally like this scheme, but think (for > consistency and confusion-reduction) that it should go straight > up the tree, instead of checking the root second. That would probably break code because the search could find some other module having the same name as a top-level one. OTOH, perhaps that situation is not all the common to fear too much about it. Walking up all the way would certainly be easier to explain to a 12-year old ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Wed Sep 22 17:51:49 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Wed, 22 Sep 1999 11:51:49 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: Your message of "Wed, 22 Sep 1999 17:13:06 +0200." <37E8F202.2A27508D@lemburg.com> References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> Message-ID: <199909221551.LAA13721@eric.cnri.reston.va.us> > > Comment 2: I generally like this scheme, but think (for > > consistency and confusion-reduction) that it should go straight > > up the tree, instead of checking the root second. > > That would probably break code because the search could > find some other module having the same name as a top-level > one. OTOH, perhaps that situation is not all the common to > fear too much about it. > > Walking up all the way would certainly be easier to explain to > a 12-year old ;-) Yes, please. Do the long-term understandable thing here. I expect not too many packages have defined subpackages (or submodules) whose name conflicts with a standard library module, so you ought to be pretty safe here! --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at digicool.com Wed Sep 22 18:25:25 1999 From: jim at digicool.com (Jim Fulton) Date: Wed, 22 Sep 1999 12:25:25 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> Message-ID: <37E902F5.27630A90@digicool.com> Guido van Rossum wrote: > > > > Comment 2: I generally like this scheme, but think (for > > > consistency and confusion-reduction) that it should go straight > > > up the tree, instead of checking the root second. > > > > That would probably break code because the search could > > find some other module having the same name as a top-level > > one. OTOH, perhaps that situation is not all the common to > > fear too much about it. > > > > Walking up all the way would certainly be easier to explain to > > a 12-year old ;-) > > Yes, please. Do the long-term understandable thing here. I expect > not too many packages have defined subpackages (or submodules) whose > name conflicts with a standard library module, so you ought to be > pretty safe here! Walking straight up the tree is my preference. I think it is very natural. Jim -- Jim Fulton mailto:jim at digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. From gward at cnri.reston.va.us Wed Sep 22 19:26:12 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Wed, 22 Sep 1999 13:26:12 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <199909221551.LAA13721@eric.cnri.reston.va.us>; from Guido van Rossum on Wed, Sep 22, 1999 at 11:51:49AM -0400 References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> Message-ID: <19990922132611.C6623@cnri.reston.va.us> On 22 September 1999, Guido van Rossum said: > Yes, please. Do the long-term understandable thing here. I expect > not too many packages have defined subpackages (or submodules) whose > name conflicts with a standard library module, so you ought to be > pretty safe here! Especially since doing so doesn't work: for example, the distutils.errors module started life as distutils.exceptions. That changed pretty quickly, once I realized why putting import exceptions into other distutils modules didn't work -- obviously it didn't find my distutils.exceptions. Arguably I should have used an absolute import, but what the heck. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal at lemburg.com Fri Sep 24 11:10:59 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri, 24 Sep 1999 11:10:59 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1274116686-66496233@hypernet.com> <37E8F202.2A27508D@lemburg.com> <199909221551.LAA13721@eric.cnri.reston.va.us> <19990922132611.C6623@cnri.reston.va.us> Message-ID: <37EB4023.28DB43A3@lemburg.com> Greg Ward wrote: > > On 22 September 1999, Guido van Rossum said: > > Yes, please. Do the long-term understandable thing here. I expect > > not too many packages have defined subpackages (or submodules) whose > > name conflicts with a standard library module, so you ought to be > > pretty safe here! > > Especially since doing so doesn't work: for example, the > distutils.errors module started life as distutils.exceptions. That > changed pretty quickly, once I realized why putting > > import exceptions > > into other distutils modules didn't work -- obviously it didn't find my > distutils.exceptions. Arguably I should have used an absolute import, > but what the heck. Wow, so many positive answers -- not bad after that last round of relative imports ;-) Ok, then I'll use the walk-me-up approach. That'll be coded into a PathImporter class I'm writing for imputil which will try to mimic the standard behaviour as much as possible (to be released in a few weeks after my vacation). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 100 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mhammond at skippinet.com.au Fri Sep 24 14:36:10 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 24 Sep 1999 22:36:10 +1000 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37EB4023.28DB43A3@lemburg.com> Message-ID: <000501bf0689$63614af0$0801a8c0@bobcat> > try to mimic the standard behaviour as much as possible (to be > released in a few weeks after my vacation). no no Marc - you should know the rules by now - you release it mere hours _before_ your vacation :-) Enjoy! Mark. From mal at lemburg.com Sun Sep 26 21:34:15 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun, 26 Sep 1999 21:34:15 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <000501bf0689$63614af0$0801a8c0@bobcat> Message-ID: <37EE7537.1075369C@lemburg.com> Mark Hammond wrote: > > > try to mimic the standard behaviour as much as possible (to be > > released in a few weeks after my vacation). > > no no Marc - you should know the rules by now - you release it mere > hours _before_ your vacation :-) Enjoy! Ya, well ;-) I'll try my best... I already have something working but it doesn't do the win32 + mac magic yet because that'll require some additions to the builtin imp module. Also, I found that it is rather slow when compared to the builtin one. Caches can speed this up a little, but I still haven't achieved the same performance. BTW, while hacking along I found a few things that might be worth discussing w/r to a general import hook scheme: Currently, the imputil apporach uses a simple chaining technique. Unfortunately, it doesn't allow inspecting the chain for already loaded hooks, so the same type of hook could be loaded more than once. Also, there are at least two types of hooks: 1. hooks that redirect the import to some other data source 2. hooks that modify the way modules are searched Since the first variant may well also be suited to used by the second, the simple chaining method probably won't be powerful enough to handle it. I think what we really need is a set of register/deregister APIs + some framework to differentiate between the two hook types (and possibly other variants). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal at lemburg.com Mon Sep 27 12:19:25 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon, 27 Sep 1999 12:19:25 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <000501bf0689$63614af0$0801a8c0@bobcat> <37EE7537.1075369C@lemburg.com> Message-ID: <37EF44AD.7C99D9C5@lemburg.com> M.-A. Lemburg wrote: > > BTW, while hacking along I found a few things that might be worth > discussing w/r to a general import hook scheme: > > Currently, the imputil apporach uses a simple chaining technique. > Unfortunately, it doesn't allow inspecting the chain for already > loaded hooks, so the same type of hook could be loaded more than > once. > > Also, there are at least two types of hooks: > > 1. hooks that redirect the import to some other data source > > 2. hooks that modify the way modules are searched > > Since the first variant may well also be suited to used by > the second, the simple chaining method probably won't be > powerful enough to handle it. > > I think what we really need is a set of register/deregister > APIs + some framework to differentiate between the two > hook types (and possibly other variants). Another quirk that I think needs fixing: When I issues an import: import mx.DateTime the whole import is handled by the importer installed at the start of the import. It is not possible to install a different importer e.g. in mx/__init__.py to handle the rest of the import (in this case the import of subpackage DateTime). I think that the importer should honor the __importer__ function (this is set by imputil) if present to let it continue the import of subsequent elements in the dotted name. Aside: Perhaps this is getting too technical for this list... should I start an egroups mailing list for defining a new and more flexible import mechanism ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gmcm at hypernet.com Mon Sep 27 18:53:30 1999 From: gmcm at hypernet.com (Gordon McMillan) Date: Mon, 27 Sep 1999 12:53:30 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <37EF44AD.7C99D9C5@lemburg.com> Message-ID: <1273671699-5802298@hypernet.com> M.-A. Lemburg wrote: [msg 1] > Currently, the imputil apporach uses a simple chaining > technique. Unfortunately, it doesn't allow inspecting the chain > for already loaded hooks, so the same type of hook could be > loaded more than once. I was hoping Greg would jump in, but since he hasn't - You're associating the hook with the strategy. That's the old style. The imputil style is to associate the hook with the actual stuff being managed. The strategy is a property of the hook. > Also, there are at least two types of hooks: > > 1. hooks that redirect the import to some other data source > > 2. hooks that modify the way modules are searched > > Since the first variant may well also be suited to used by the > second, the simple chaining method probably won't be powerful > enough to handle it. The top level question is "is it mine to import?". Greg provides a framework that makes it easy to use alternate data sources, and alternate ways of finding things but that's not really the key thing. You're a "good" importer if you can (when appropriate) way "no it's not mine" efficiently. [msg 2] > Another quirk that I think needs fixing: > > When I issues an import: > > import mx.DateTime > > the whole import is handled by the importer installed at > the start of the import. It is not possible to install a > different importer e.g. in mx/__init__.py to handle the rest of > the import (in this case the import of subpackage DateTime). I > think that the importer should honor the __importer__ function > (this is set by imputil) if present to let it continue the import > of subsequent elements in the dotted name. Sure you can. Your first importer is the "mx" importer. It has a dict of sub-importers. When mx/DateTime/__init__.py runs, it puts itself into that dict. The importer chain is now a tree. This means, I think, that a "general" relative-path importer (ie, one that uses the default PYTHONPATH strategy), should be careful to install itself as the penultimate importer in the chain, (ie, the last before __builtin__.imp). But putting a relative-path search strategy into the "mx" importer is fine if it can quickly determine that the target is / is not a valid name in the "mx" namespace. - Gordon From mal at lemburg.com Tue Sep 28 00:59:57 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue, 28 Sep 1999 00:59:57 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1273671699-5802298@hypernet.com> Message-ID: <37EFF6ED.1693ACA3@lemburg.com> Gordon McMillan wrote: > > M.-A. Lemburg wrote: > [msg 1] > > Currently, the imputil apporach uses a simple chaining > > technique. Unfortunately, it doesn't allow inspecting the chain > > for already loaded hooks, so the same type of hook could be > > loaded more than once. > > I was hoping Greg would jump in, but since he hasn't - > > You're associating the hook with the strategy. That's the old > style. The imputil style is to associate the hook with the > actual stuff being managed. The strategy is a property of the > hook. I know, but there still is no way to query what kind of hooks are already loaded and what is worse, you cannot unload or reorder them. I'd suggest using a list of hooks which are then traversed in the order they appear in the list, e.g. __importers__ = [DirectoryImporter('/usr/local/lib/python1.5'), ArchiveImporter('/usr/local/lib/app.pyz'), WebImporter('http://www.python.org/pylib/'), PathImporter(('~/bin','~/lib'))] This also has the advantage of being able to easily query the importers during debugging and eliminates the need to have a predefined attribute naming scheme (such as the one imputil uses). > > Also, there are at least two types of hooks: > > > > 1. hooks that redirect the import to some other data source > > > > 2. hooks that modify the way modules are searched > > > > Since the first variant may well also be suited to used by the > > second, the simple chaining method probably won't be powerful > > enough to handle it. > > The top level question is "is it mine to import?". Greg provides > a framework that makes it easy to use alternate data sources, > and alternate ways of finding things but that's not really the > key thing. You're a "good" importer if you can (when > appropriate) way "no it's not mine" efficiently. It does a good job at this, but doesn't really separate lookup and loading of code too well. Everything is packaged into one single method (.get_code()) which is not always flexible enough, e.g. it wasn't possible to implement the walk-up-the-dotted-name scheme using modifications to .get_code() alone. Also, I can see many uses where you combine a lookup hook (e.g. for loading modules across the web) with a filtering hook (e.g. one which checks a module signature). This should go into the framework as well, IMHO... e.g. by having two methods .find_code() and .make_module() (like the builtin importer). > [msg 2] > > Another quirk that I think needs fixing: > > > > When I issues an import: > > > > import mx.DateTime > > > > the whole import is handled by the importer installed at > > the start of the import. It is not possible to install a > > different importer e.g. in mx/__init__.py to handle the rest of > > the import (in this case the import of subpackage DateTime). I > > think that the importer should honor the __importer__ function > > (this is set by imputil) if present to let it continue the import > > of subsequent elements in the dotted name. > > Sure you can. Your first importer is the "mx" importer. It has a > dict of sub-importers. When mx/DateTime/__init__.py runs, it > puts itself into that dict. The importer chain is now a tree. The problem is that the special importer has to be installed *prior* to doing the mx.DateTime import, because otherwise the importer will not take control over the DateTime subpackage (unless I tell it to do so explicitly, which is not really what I want to have to do). > This means, I think, that a "general" relative-path importer (ie, > one that uses the default PYTHONPATH strategy), should be > careful to install itself as the penultimate importer in the chain, > (ie, the last before __builtin__.imp). But putting a relative-path > search strategy into the "mx" importer is fine if it can quickly > determine that the target is / is not a valid name in the "mx" > namespace. Exactly... and this brings us back to the importer list I mentioned above. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From gstein at lyra.org Tue Sep 28 23:57:34 1999 From: gstein at lyra.org (Greg Stein) Date: Tue, 28 Sep 1999 14:57:34 -0700 (PDT) Subject: [Python-Dev] Alternative Approach to Relative Imports In-Reply-To: <1273671699-5802298@hypernet.com> Message-ID: On Mon, 27 Sep 1999, Gordon McMillan wrote: > M.-A. Lemburg wrote: > [msg 1] > > Currently, the imputil apporach uses a simple chaining > > technique. Unfortunately, it doesn't allow inspecting the chain > > for already loaded hooks, so the same type of hook could be > > loaded more than once. > > I was hoping Greg would jump in, but since he hasn't - I'm in a middle of a move back to CA. Unpacking now... > You're associating the hook with the strategy. That's the old > style. The imputil style is to associate the hook with the > actual stuff being managed. The strategy is a property of the > hook. Quite true. The chaining is simply an artifact of what has been installed as the import hook. I've always envisioned the potential for a "Importer Manager" that installs just like any other hook, but provides higher-level functions for importers to install themselves. The manager simply delegates the get_code() function to the sub-importers. Of course, the manager could use whatever technique to improve the speed of Importer selection. With respect to speed, I think the main point is to realize that the imputil technique is not inherently slow. It just depends on how you design your Importer subclasses -- do you install one or a hundred Importers? The imputil scheme is more about simplifying how people hook into the process (implement get_code() rather than a load/import combo). It also provides a simple capability (chaining) to allow *multiple* hooks to be installed. > > Also, there are at least two types of hooks: > > > > 1. hooks that redirect the import to some other data source > > > > 2. hooks that modify the way modules are searched Just one way -- your second is a variant of the first. "other data source" is a functional superset which includes searching. Importers don't simply alter searching -- they must perform the actual import (from wherever). This is the big change in mindset from the "ihooks" method -- find it and import it on the spot. The net effect is an Importer either imports a module or it doesn't (and the system can fallback to try another Importer). [ one the examples that people always like to specify was importing via URL which was actually quite difficult to use in the old scheme -- how do you separate an HTTP GET into a find/load step? Effectively, you had to double-fetch, or you had to place the whole module (which you retrieved during the find step) into your context for passing to the load. The other issue was the distinct semantics also implied that you could separate the functions -- I believe that to be quite unnecessary functionality. ] > > Since the first variant may well also be suited to used by the > > second, the simple chaining method probably won't be powerful > > enough to handle it. > > The top level question is "is it mine to import?". Greg provides > a framework that makes it easy to use alternate data sources, > and alternate ways of finding things but that's not really the > key thing. You're a "good" importer if you can (when > appropriate) way "no it's not mine" efficiently. Very true! > [msg 2] > > Another quirk that I think needs fixing: > > > > When I issues an import: > > > > import mx.DateTime > > > > the whole import is handled by the importer installed at > > the start of the import. It is not possible to install a > > different importer e.g. in mx/__init__.py to handle the rest of > > the import (in this case the import of subpackage DateTime). I > > think that the importer should honor the __importer__ function > > (this is set by imputil) if present to let it continue the import > > of subsequent elements in the dotted name. > > Sure you can. Your first importer is the "mx" importer. It has a > dict of sub-importers. When mx/DateTime/__init__.py runs, it > puts itself into that dict. The importer chain is now a tree. Gordon's on top of it here... :-) Yes, it is simply a matter of perspective on the import process. An importer does not have to be a static entity. It also can be much more than a way to search a path... it can be highly dynamic and flexible. Whatever you like. Just implement get_code() to map a module "mx.DateTime" to a code/module object. There are a bazillion ways to do that :-) > This means, I think, that a "general" relative-path importer (ie, > one that uses the default PYTHONPATH strategy), should be > careful to install itself as the penultimate importer in the chain, > (ie, the last before __builtin__.imp). But putting a relative-path > search strategy into the "mx" importer is fine if it can quickly > determine that the target is / is not a valid name in the "mx" > namespace. Part of the Importer work was done to satisfy importing modules from the COM+ namespace. I wanted to be able to say "import COM.foo.bar". The importer would handle all "COM." imports and delegate the "foo.bar" to the underlying Python/COM framework. In other words... yes, the Importer scheme should work *very* well for the "whatever...." type of module namespace. Cheers, -g -- Greg Stein, http://www.lyra.org/ From da at ski.org Wed Sep 29 02:02:20 1999 From: da at ski.org (David Ascher) Date: Tue, 28 Sep 1999 17:02:20 -0700 (Pacific Daylight Time) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) Message-ID: ---------- Forwarded message ---------- On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > It's not available online (it was written on a typewriter), so I'd be > happy to send a hard copy of it to anyone who raises a hand in private > mail (or cannot find LNCS 60). I then asked him for a copy stating: > If you send it to me, I can OCR it and make it available online. and he generously sent it to me. The problem is that I hadn't noticed the length of the manuscript. It's over a hundred pages, and the copy is nth generation, making OCR pretty much useless. So, if anyone wants a copy, I can make and send copies (which would make most sense for folks in the US -- sending things from France isn't cheap). So far, it's good reading. Funny to see "file name" in quotes in the text. --david From mal at lemburg.com Wed Sep 29 12:47:08 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed, 29 Sep 1999 12:47:08 +0200 Subject: [Python-Dev] Alternative Approach to Relative Imports References: Message-ID: <37F1EE2C.52F9A7E1@lemburg.com> Greg Stein wrote: > > On Mon, 27 Sep 1999, Gordon McMillan wrote: > > M.-A. Lemburg wrote: > > [msg 1] > > > Currently, the imputil apporach uses a simple chaining > > > technique. Unfortunately, it doesn't allow inspecting the chain > > > for already loaded hooks, so the same type of hook could be > > > loaded more than once. > > You're associating the hook with the strategy. That's the old > > style. The imputil style is to associate the hook with the > > actual stuff being managed. The strategy is a property of the > > hook. > > Quite true. The chaining is simply an artifact of what has been installed > as the import hook. I've always envisioned the potential for a "Importer > Manager" that installs just like any other hook, but provides higher-level > functions for importers to install themselves. The manager simply > delegates the get_code() function to the sub-importers. Of course, the > manager could use whatever technique to improve the speed of Importer > selection. > > With respect to speed, I think the main point is to realize that the > imputil technique is not inherently slow. It just depends on how you > design your Importer subclasses -- do you install one or a hundred > Importers? > > The imputil scheme is more about simplifying how people hook into the > process (implement get_code() rather than a load/import combo). It also > provides a simple capability (chaining) to allow *multiple* hooks to be > installed. As I wrote in my reply to Gordon, this setup has some drawbacks which an "Import Manager" could easily solve, e.g. by using a list of importers. > > > Also, there are at least two types of hooks: > > > > > > 1. hooks that redirect the import to some other data source > > > > > > 2. hooks that modify the way modules are searched > > Just one way -- your second is a variant of the first. "other data source" > is a functional superset which includes searching. Importers don't simply > alter searching -- they must perform the actual import (from wherever). Yes, I was just argueing for two types of functionality, not the old scheme. E.g. the Import Manager could provide a set of filters which implement signature checks or know how to un-gzip code plus a set of lookup functions for scanning directories or zip archives. I would like the importers to take advantage of such functionality. Of course, all of this could be implemented in form of classes which the importers then use as mixin classes. > This is the big change in mindset from the "ihooks" method -- find it and > import it on the spot. The net effect is an Importer either imports a > module or it doesn't (and the system can fallback to try another > Importer). > > [ one the examples that people always like to specify was importing via > URL which was actually quite difficult to use in the old scheme -- how do > you separate an HTTP GET into a find/load step? Effectively, you had to > double-fetch, or you had to place the whole module (which you retrieved > during the find step) into your context for passing to the load. The other > issue was the distinct semantics also implied that you could separate the > functions -- I believe that to be quite unnecessary functionality. ] .get_code() is fine for these kind of tasks, but there are some other areas (such as lazy imports) which work better using the split setup. This is pretty easy to implement btw, just have the Import Manager check whether the importer provides .get_code() and then have it revert to using .find_module(), .load_module() if it doesn't. The more I think about it, the more I like the idea of an Import Manager instead of the chaining approach. > > > Since the first variant may well also be suited to used by the > > > second, the simple chaining method probably won't be powerful > > > enough to handle it. > > > > The top level question is "is it mine to import?". Greg provides > > a framework that makes it easy to use alternate data sources, > > and alternate ways of finding things but that's not really the > > key thing. You're a "good" importer if you can (when > > appropriate) way "no it's not mine" efficiently. > > Very true! > > > [msg 2] > > > Another quirk that I think needs fixing: > > > > > > When I issues an import: > > > > > > import mx.DateTime > > > > > > the whole import is handled by the importer installed at > > > the start of the import. It is not possible to install a > > > different importer e.g. in mx/__init__.py to handle the rest of > > > the import (in this case the import of subpackage DateTime). I > > > think that the importer should honor the __importer__ function > > > (this is set by imputil) if present to let it continue the import > > > of subsequent elements in the dotted name. > > > > Sure you can. Your first importer is the "mx" importer. It has a > > dict of sub-importers. When mx/DateTime/__init__.py runs, it > > puts itself into that dict. The importer chain is now a tree. > > Gordon's on top of it here... :-) Yes, it is simply a matter of > perspective on the import process. An importer does not have to be a > static entity. It also can be much more than a way to search a path... it > can be highly dynamic and flexible. Whatever you like. Just implement > get_code() to map a module "mx.DateTime" to a code/module object. There > are a bazillion ways to do that :-) Except that they don't work due to the fact that the builtin importer is not recursively using __import__ for the imports. An Import Manager would help with this too :-) > > This means, I think, that a "general" relative-path importer (ie, > > one that uses the default PYTHONPATH strategy), should be > > careful to install itself as the penultimate importer in the chain, > > (ie, the last before __builtin__.imp). But putting a relative-path > > search strategy into the "mx" importer is fine if it can quickly > > determine that the target is / is not a valid name in the "mx" > > namespace. > > Part of the Importer work was done to satisfy importing modules from the > COM+ namespace. I wanted to be able to say "import COM.foo.bar". The > importer would handle all "COM." imports and delegate the "foo.bar" to the > underlying Python/COM framework. > > In other words... yes, the Importer scheme should work *very* well for > the "whatever...." type of module namespace. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From jim at interet.com Wed Sep 29 15:59:11 1999 From: jim at interet.com (James C. Ahlstrom) Date: Wed, 29 Sep 1999 09:59:11 -0400 Subject: [Python-Dev] Alternative Approach to Relative Imports References: <1273671699-5802298@hypernet.com> <37EFF6ED.1693ACA3@lemburg.com> Message-ID: <37F21B2F.DD65738F@interet.com> "M.-A. Lemburg" wrote: > > I know, but there still is no way to query what kind of hooks > are already loaded and what is worse, you cannot unload or reorder > them. I think this is a valid point. I am interested in hooks to read modules from a file archive. If you are a developer, it is necessary to turn this hook OFF, so that you can revert to the usual directory tree where your current source is. I am solving this by leaving a global variable "Importer" in sitecustomize, and calling sitecustomize.Importer.enable(0). This works, but it might be useful if imputil could de-install a hook as well as install it. Jim Ahlstrom From mhammond at skippinet.com.au Thu Sep 30 02:16:44 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 30 Sep 1999 10:16:44 +1000 Subject: [Python-Dev] getopt helper? Message-ID: <007d01bf0ad5$14b6cf60$0801a8c0@bobcat> A little thought I have had a few times now, so it is time to pass it on for comment. Basically _every_ time I use getopt, I write code like this: import getopt try: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") except getopt.error, why: print why print usage % (os.path.basename(sys.argv[0],)) sys.exit(1) Every single time. I have never used getopt without this code. How about we put this functionality into getopt itself? It could be triggered either by adding a new "usage" param defaulting to None, or by adding a new entry point. ie: opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:", usage=usage) or opts, args = getopt.getoptex(sys.argv[1:], "slc:d:e:", usage=usage) I know it is fairly trivial, but IMO is such a useful module and the pattern is used so regularly that is seems to make sense to add it. Any thoughts? If it is seen favourably, how should we spell it? Mark. From mal at lemburg.com Thu Sep 30 09:29:24 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 09:29:24 +0200 Subject: [Python-Dev] getopt helper? References: <007d01bf0ad5$14b6cf60$0801a8c0@bobcat> Message-ID: <37F31154.4FDA472C@lemburg.com> Mark Hammond wrote: > > A little thought I have had a few times now, so it is time to pass it > on for comment. > > Basically _every_ time I use getopt, I write code like this: > > import getopt > try: > opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:") > except getopt.error, why: > print why > print usage % (os.path.basename(sys.argv[0],)) > sys.exit(1) > > Every single time. I have never used getopt without this code. > > How about we put this functionality into getopt itself? > > It could be triggered either by adding a new "usage" param defaulting > to None, or by adding a new entry point. ie: > > opts, args = getopt.getopt(sys.argv[1:], "slc:d:e:", usage=usage) > or > opts, args = getopt.getoptex(sys.argv[1:], "slc:d:e:", usage=usage) > > I know it is fairly trivial, but IMO is such a useful module and the > pattern is used so regularly that is seems to make sense to add it. > > Any thoughts? If it is seen favourably, how should we spell it? Why not just add a higher level interface ? Something like CommandLine.py which is included in mxDateTime ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mhammond at skippinet.com.au Thu Sep 30 12:09:53 1999 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 30 Sep 1999 20:09:53 +1000 Subject: [Python-Dev] getopt helper? In-Reply-To: <37F31154.4FDA472C@lemburg.com> Message-ID: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> > > Basically _every_ time I use getopt, I write code like this: > Why not just add a higher level interface ? Something > like CommandLine.py which is included in mxDateTime ? Because _every_ time I use getopt, I write code like that :-) A higher level interface would maybe be handy, but it simply occurred to me that every time I used the module I used that pattern. I just got sick of typing it all the time and wondered if it struck a chord with anyone else (and I dont have or use a general purpose "mhutil" module :-) Im really just trying to save myself 10 lines of boilerplate coding, not introduce a new standard module :-) Mark. From Vladimir.Marangozov at inrialpes.fr Thu Sep 30 13:46:48 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Thu, 30 Sep 1999 12:46:48 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: from "David Ascher" at "Sep 28, 99 05:02:20 pm" Message-ID: <199909301146.MAA23138@pukapuka.inrialpes.fr> David Ascher wrote: > > ---------- Forwarded message ---------- > On Sat, 18 Sep 1999, Vladimir Marangozov wrote: > > > > Saltzer J., "Naming and Binding of Objects", in Bayer R., "Operating > > Systems - An Advanced Course", pp. 99-208, LNCS 60, 1978. > > > > It's not available online (it was written on a typewriter), so I'd be > > happy to send a hard copy of it to anyone who raises a hand in private > > mail (or cannot find LNCS 60). > > I then asked him for a copy stating: > > > If you send it to me, I can OCR it and make it available online. > > and he generously sent it to me. The problem is that I hadn't noticed the > length of the manuscript. It's over a hundred pages, and the copy is nth > generation, making OCR pretty much useless. If I find some spare time, I'll do it. This paper is a classic in Comp Sci that we kindly invite our students to read. In research language, we say that Saltzer has made "the turn around the clock" regarding this issue, putting a period on it (i.e. there's hardly something more to say). It's interesting, however, to see how the theory was applied on Python and to establish the fairly easy analogy of the binding model. This analogy proves the good design choices Guido has made, but also reveals some weaknesses or the incompleteness of the current implementation. I hope to discuss this for Python 2 in due time and perhaps settle on a compromise which trades genericity for performance. The naming/binding problem drives the whole implementation logic in Python (objects, classes, scopes, etc.). > > So, if anyone wants a copy, I can make and send copies (which would make > most sense for folks in the US -- sending things from France isn't cheap). I have 2 more copies lying around, ready to be sent. Give me an adress. The first 2 adresses win a hard copy, no matter the location (don't worry about mail costs.) -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From mal at lemburg.com Thu Sep 30 13:22:27 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 13:22:27 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> Message-ID: <37F347F3.6D18C285@lemburg.com> Mark Hammond wrote: > > > > Basically _every_ time I use getopt, I write code like this: > > > Why not just add a higher level interface ? Something > > like CommandLine.py which is included in mxDateTime ? > > Because _every_ time I use getopt, I write code like that :-) > > A higher level interface would maybe be handy, but it simply occurred > to me that every time I used the module I used that pattern. I just > got sick of typing it all the time and wondered if it struck a chord > with anyone else (and I dont have or use a general purpose "mhutil" > module :-) Im really just trying to save myself 10 lines of > boilerplate coding, not introduce a new standard module :-) Just a thought :-) I wrote the CommandLine.py for pretty much the same reason: I have quite a few command line apps lying in my bin/ dir and they all did some kind of getopt/sys.argv tricks to handle the input... way to confusing and not easy to maintain. So I decided to take an OO-approach to have them use the same interface with nice help output and to reduce the coding effort. As an example taken from mxDateTime: #!/usr/local/bin/python -u """ Simple Forking Alarm Sample Application for DateTime types and CommandLine. Only works on OSes which support os.fork(). Author: Marc-Andre Lemburg, mailto:mal at lemburg.com """ import time,sys,os from mx.DateTime import * from CommandLine import Application,ArgumentOption class Alarm(Application): header = "Simple Forking Alarm" options = [ArgumentOption('-s', 'set the alarm to now + arg seconds'), ArgumentOption('-m', 'set the alarm to now + arg minutes'), ArgumentOption('-a', 'set the alarm to ring at arg (hh:mm)'), ] version = '0.1' def main(self): atime = now() + (self.values['-s'] or self.values['-m'] * 60 or self.values['-h'] * 3600) * oneSecond abs = self.values['-a'] if abs: atime = strptime(abs,'%H:%M',today(second=0)) if atime < now(): print 'Alarm time has expired...' return print 'Alarm will ring at',atime if not os.fork(): time.sleep((atime - now()).seconds) alarm() os._exit(0) def alarm(): """ Ring alarm """ for i in range(10): sys.stdout.write('\007') sys.stdout.flush() time.sleep(0.2) if __name__ == '__main__': Alarm() Here's the help output this produces: /home/lemburg> alarm -h ------------------------------------------------------------------------ Simple Forking Alarm ------------------------------------------------------------------------ Synopsis: alarm [option] files... Options and default settings: -s arg set the alarm to now + arg seconds -m arg set the alarm to now + arg minutes -a arg set the alarm to ring at arg (hh:mm) -h show this help text --help show this help text --copyright show copyright --examples show examples of usage Version: 0.1 -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From guido at CNRI.Reston.VA.US Thu Sep 30 14:31:04 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 30 Sep 1999 08:31:04 -0400 Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: Your message of "Thu, 30 Sep 1999 12:46:48 BST." <199909301146.MAA23138@pukapuka.inrialpes.fr> References: <199909301146.MAA23138@pukapuka.inrialpes.fr> Message-ID: <199909301231.IAA11203@eric.cnri.reston.va.us> I've started reading the paper and agree that it's very good! > It's interesting, however, to see how the theory was applied on Python and > to establish the fairly easy analogy of the binding model. This analogy > proves the good design choices Guido has made, but also reveals some > weaknesses or the incompleteness of the current implementation. I hope > to discuss this for Python 2 in due time and perhaps settle on a compromise > which trades genericity for performance. The naming/binding problem drives > the whole implementation logic in Python (objects, classes, scopes, etc.). I'd like to hear what those weaknesses are in your eyes. I can think of two areas myself: (1) sys.path, $PYTHONPATH etc.; (2) class/instance attributes in the context of subclassing and evolution of the base class. (I don't expect the paper to take a stance on nested scopes.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at CNRI.Reston.VA.US Thu Sep 30 14:35:23 1999 From: guido at CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 30 Sep 1999 08:35:23 -0400 Subject: [Python-Dev] getopt helper? In-Reply-To: Your message of "Thu, 30 Sep 1999 13:22:27 +0200." <37F347F3.6D18C285@lemburg.com> References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> Message-ID: <199909301235.IAA11217@eric.cnri.reston.va.us> [Mark] > > > > Basically _every_ time I use getopt, I write code like this: [Marc-Andre] > > > Why not just add a higher level interface ? Something > > > like CommandLine.py which is included in mxDateTime ? [Mark] > > Because _every_ time I use getopt, I write code like that :-) [Marc-Andre] > I wrote the CommandLine.py for pretty much the same reason: Marc-Andre, you're not hearing what Mark is saying. He wants a change to the standard library, and he knows that small additions to existing modules there stand a better chance of adoption than new modules. I personally liked the idea of getoptex() best, except I would call it getopt_or_die(). If the usage message is omitted it can synthesize one from the (short and long) options arguments and sys.argv[0] (the latter being a bit controversial, but it's just a default). Hmm... Perhaps getopt_or_die() shouldn't take the args argument but extract sys.argv[1:] itself? --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Thu Sep 30 15:22:58 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 30 Sep 1999 15:22:58 +0200 Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> Message-ID: <007501bf0b46$eb2f38e0$f29b12c2@secret.pythonware.com> as reported on slashdot: http://www.perl.com/pub/1999/09/topaz.html (topaz is also the code name for our "rewrite Tkinter in Python", but that's another story ;-) From gward at cnri.reston.va.us Thu Sep 30 16:15:50 1999 From: gward at cnri.reston.va.us (Greg Ward) Date: Thu, 30 Sep 1999 10:15:50 -0400 Subject: [Python-Dev] getopt helper? In-Reply-To: <199909301235.IAA11217@eric.cnri.reston.va.us>; from Guido van Rossum on Thu, Sep 30, 1999 at 08:35:23AM -0400 References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <19990930101549.A10529@cnri.reston.va.us> On 30 September 1999, Guido van Rossum said: > Marc-Andre, you're not hearing what Mark is saying. He wants a change > to the standard library, and he knows that small additions to existing > modules there stand a better chance of adoption than new modules. > > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). Gasp! Guido hath drunk from the poisoned well of the Great Camel and it doth infest his thinking! "or die" indeed -- not an idiom I've seen since the last time I wrote some Perl code (umm, last week?). > If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? Whatever we call it, it should be able to take an explicit argument list, and only default to sys.argv[1:]. What if I want to parse options from an environment variable or a config file? I also like the "don't clobber sys.argv, but return the modified version instead" model -- it's nice to keep a pristine copy of the original argument list! Another problem with getopt is that it doesn't correlate long and short options. I wrote distutils.fancy_getopt (download your copy today! hurry, don't delay -- at this price, it WON'T LAST LONG!) to address this, and to maybe someday do something with help text. On the other hand, don't listen to me -- I tend to write mammoth, bloated, all-singing, all-dancing command-line parsing modules for every new language I encounter. They get more insane with each iteration. I have yet to top my Getopt::Tabular for Perl, though; see http://search.cpan.org/doc/GWARD/Getopt-Tabular-0.3/Tabular.pod if you've ever wondered how far this sort of thing can be taken in a high-level, dynamically typed language. Greg -- Greg Ward - software developer gward at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fredrik at pythonware.com Thu Sep 30 16:18:15 1999 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 30 Sep 1999 16:18:15 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <00b301bf0b4e$a3ac5cc0$f29b12c2@secret.pythonware.com> > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? sounds reasonable. what are you waiting for ;-) From mal at lemburg.com Thu Sep 30 17:28:31 1999 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu, 30 Sep 1999 17:28:31 +0200 Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> Message-ID: <37F3819F.2C713970@lemburg.com> Guido van Rossum wrote: > > [Mark] > > > > > Basically _every_ time I use getopt, I write code like this: > > [Marc-Andre] > > > > Why not just add a higher level interface ? Something > > > > like CommandLine.py which is included in mxDateTime ? > > [Mark] > > > Because _every_ time I use getopt, I write code like that :-) > > [Marc-Andre] > > I wrote the CommandLine.py for pretty much the same reason: > > Marc-Andre, you're not hearing what Mark is saying. He wants a change > to the standard library, and he knows that small additions to existing > modules there stand a better chance of adoption than new modules. Oh, I did get the idea... just wanted to plug my module here in a take-it-or-leave-it way ;-) I usually put such things into my lib/ dir for Python to find -- no need to make them a standard. > I personally liked the idea of getoptex() best, except I would call it > getopt_or_die(). If the usage message is omitted it can synthesize > one from the (short and long) options arguments and sys.argv[0] (the > latter being a bit controversial, but it's just a default). > > Hmm... Perhaps getopt_or_die() shouldn't take the args argument but > extract sys.argv[1:] itself? Better not: it's sometimes very useful to call the main(args) function of a script in interactive mode which then passes the args list to getopt(). How about adding something like: def getoptex(...,args=None,helptext='Read the source, Luke ;-)'): if args is None: args = sys.argv[1:] ... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 92 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From Vladimir.Marangozov at inrialpes.fr Thu Sep 30 21:24:53 1999 From: Vladimir.Marangozov at inrialpes.fr (Vladimir Marangozov) Date: Thu, 30 Sep 1999 20:24:53 +0100 (NFT) Subject: [Python-Dev] Naming and Binding of Objects (was: Relative Package Imports) (fwd) In-Reply-To: <199909301231.IAA11203@eric.cnri.reston.va.us> from "Guido van Rossum" at "Sep 30, 99 08:31:04 am" Message-ID: <199909301924.UAA32482@pukapuka.inrialpes.fr> Guido van Rossum wrote: > > > I've started reading the paper and agree that it's very good! > > > It's interesting, however, to see how the theory was applied on Python and > > to establish the fairly easy analogy of the binding model. This analogy > > proves the good design choices Guido has made, but also reveals some > > weaknesses or the incompleteness of the current implementation. I hope > > to discuss this for Python 2 in due time and perhaps settle on a compromise > > which trades genericity for performance. The naming/binding problem drives > > the whole implementation logic in Python (objects, classes, scopes, etc.). > > I'd like to hear what those weaknesses are in your eyes. I can think > of two areas myself: (1) sys.path, $PYTHONPATH etc.; (2) > class/instance attributes in the context of subclassing and evolution > of the base class. (2) subsumes (1). > > (I don't expect the paper to take a stance on nested scopes.) > But we can. I can't make the time for this right now and I apologize. This subject deserves more attention and I can't describe it quickly, so it has to wait. As I said, in due time ;-). I'm currently overloaded. (In my defense, and Barry will undesrand me very well, I'll say that among othger things I'm reworking the www.inrialpes.fr Web site, which is actually a shame -- now that I'm in charge, come and visit it in a month). Anyway, some quick general notes on what actually exists: Where Python (Guido) really strikes is that almost everything we have at the language level is interpreted as a name. Thus, "sharing can occur". These names are always resolved in some context associated with the object containing the named object. Since all we have is names, the risk of name conflicts is real, especially when half of the contexts against which name resolution occurs are implicit. Therefore, name resolution has to be done in a controlled way (what Guido has successfully tried to provide when designing the language). Simple example: >>> print o.attr This says: print the value of an object, whose name "attr" has to be resolved in the context associated with the object "o", this context being explicitely pointed out by a dot "." Another (not so obvious) one: >>> print 1 This says: print the value of an object, whose name "1" has to be resolved in the current (implicit) context. And this is exactly what happens inside the VM in terms of LOAD_CONST , then PRINT_ITEM. If you don't grasp this, try the same example: >>> a = 1 >>> print a This says: Resolve the name "1" in the current context (thus we reach the object in question) then assign a new binding ("a" -> the object) in the current context. Then for "print a", see "This says" of the previous example. A valuable thing happens in my last example: >>> print o.__dict__ This resolves the name "__dict__" in the context associated with the object named "o" (pointed by a dot "."), returning this same context! (or a fraction of it). Whether we have to get a portion of the context or the full context is debatable. It has been felt that with the dynamicity of Python, it's useful to get access to the context, then play with it in a controlled manner. So from here, I (and you) can deduce what happens in terms of naming and binding on function/class/... definitions, on module imports, on attr lookups, on "global" declarations, and so on, and when and where (and hopefully why) we get name conflicts or strange (implicit) name resolutions for nested functions. A last word towards classes: There's no difference between >>> o = C() # an instance of the class C >>> o.__class__ and >>> o = 1 >>> o.__class__ I'm prevented to type "1.__class__" only for syntactic reasons, but the context for the resolution of o.__class__ exists. What's missing is for a future mail. Later, -- Vladimir MARANGOZOV | Vladimir.Marangozov at inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 From bwarsaw at cnri.reston.va.us Thu Sep 30 22:29:55 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 30 Sep 1999 16:29:55 -0400 (EDT) Subject: [Python-Dev] getopt helper? References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <199909301235.IAA11217@eric.cnri.reston.va.us> <19990930101549.A10529@cnri.reston.va.us> Message-ID: <14323.51267.54862.538939@anthem.cnri.reston.va.us> I can sympathize with Mark, I have nearly the same code in every script I write. I once wrote a nice (IMO) class for doing all the common things I do with c.l. args, but I can't seem to dig it up at the moment. The idea was basically to have a base class that had all the machinery, while derived classes included specially named methods that did the app-specific option handling. It knew whether the method took 1 or zero arguments (not including self), glommed up the shortarg string and longarg list (with appropriate `:' and `=' thrown in), then parsed the args, dispatching to the handlers, e.g.: class MyOptions(getopt.Options): def handle_a(self): self.alpha = 1 handle_alpha = handle_a def handle_b(self, arg): self.beta = arg handle_beta = handle_b def handle_i(self, arg): try: self.integer = int(arg) except ValueError: self.usage() handle_integer = handle_i and could be used like so: #! /usr/bin/env python # # ... opts = MyOptions(sys.argv[1:], usage=__doc__ % globals()) if opts.alpha: do_my_alpha_thang() if opts.integer = 1: do_something_else() for file in opts.leftoverargs: process_file(file) While I liked this a lot, I seem to remember showing it to Guido, receiving the expected scoffing. So now, I just cut and paste the opt parsing stuff into every script I write :) I think it might be nice to add such a base class to the standard getopt module. -Barry From bwarsaw at cnri.reston.va.us Thu Sep 30 22:42:02 1999 From: bwarsaw at cnri.reston.va.us (Barry A. Warsaw) Date: Thu, 30 Sep 1999 16:42:02 -0400 (EDT) Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys References: <000001bf0b2b$f165e7b0$0801a8c0@bobcat> <37F347F3.6D18C285@lemburg.com> <007501bf0b46$eb2f38e0$f29b12c2@secret.pythonware.com> Message-ID: <14323.51994.307017.278161@anthem.cnri.reston.va.us> >>>>> "FL" == Fredrik Lundh writes: FL> as reported on slashdot: FL> http://www.perl.com/pub/1999/09/topaz.html FL> (topaz is also the code name for our "rewrite FL> Tkinter in Python", but that's another story ;-) Here's an interesting quote: When I was trying to figure out how to be persuasive on this subject, I finally realized that Perl may be competing with Java in the problem space, but when you're writing Perl, implementing the Perl runtime, really what you're doing is something equivalent to writing a JVM. You're writing the equivalent of a Java Virtual Machine. Now, would you write a JVM in Eiffel? I don't think so. No, so neither would you write the Perl runtime in Java or in Eiffel. In the context of Python, I disagree that it competes against Java; Python makes a nice complement to Java. And I obviously also think it makes perfect sense to write (a) Python runtime in Java, as JimH has so effectively proven. Another interesting tidbit we've addressed here: Was the syntax appropriate for declaring variables to give appropriate hints to a hypothetical compiler? That is to say MY INT $X or MY STR $Y -- and I thought that the INT and the STR and the NUM should be suffixes, something like MY $X:NUM-and, in fact, that suffix syntax is something that Larry officially has blessed, but just not for this purpose. -Barry From da at ski.org Thu Sep 30 23:30:20 1999 From: da at ski.org (David Ascher) Date: Thu, 30 Sep 1999 14:30:20 -0700 (Pacific Daylight Time) Subject: [Python-Dev] FYI: Perl 6, Topaz, and exploding kidneys In-Reply-To: <14323.51994.307017.278161@anthem.cnri.reston.va.us> Message-ID: On Thu, 30 Sep 1999, Barry A. Warsaw wrote: > Another interesting tidbit we've addressed here: > > Was the syntax appropriate for declaring variables to give > appropriate hints to a hypothetical compiler? That is to say MY > INT $X or MY STR $Y -- and I thought that the INT and the STR and > the NUM should be suffixes, something like MY $X:NUM-and, in fact, > that suffix syntax is something that Larry officially has blessed, > but just not for this purpose. What struck me about that paragraph is that if I understand the comment correctly, Larry is choosing a mapping from a specific syntax to some purpose (unspecified in the excerpt) while an otherwise well-informed Perl user (Chip) 'naturally' wanted to map said syntax to a different semantics. When this happens in Python-land, Guido says "nope, neither". --david