From zathras at thwackety.com Fri Oct 1 00:36:02 2004 From: zathras at thwackety.com (Michael Sparks) Date: Fri Oct 1 00:08:06 2004 Subject: [Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration In-Reply-To: <5.1.1.6.0.20040930153803.02bc0140@mail.telecommunity.com> Message-ID: On Thu, 30 Sep 2004, Phillip J. Eby wrote: ... > A mechanism to pass values or exceptions into generators [ Possibly somewhat off topic, and apologies if it is, and I'm positive someone's done something similar before, but I think it's relevant to the discussion in hand -- largely because the above use case *doesn't* require changes to python... ] A minimal(ish) example of a technique for doing this I presented last week at a streaming workshop looked like the following. (amongst other stuff...) Create a decorator that wraps the generator inside a class, derived from a supplied base class. (default to object) import copy def wrapgenerator(bases=object, **attrs): def decorate(func): class statefulgenerator(bases): __doc__ = func.__doc__ def __init__(self,*args): super(statefulgenerator, self).__init__(*args) self.func=func(self,*args) for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k]) self.next=self.__iter__().next def __iter__(self): return iter(self.func) return statefulgenerator return decorate Create a class to handle the behaviour you wish to use to communicate with the generator from outside: class component(object): def __init__(self, *args): # Default queues self.queues = {"inbox":[],"control":[],"outbox":[],"signal":[]} def send(self, box, object): self.queues[box].append(object) def dataReady(self,box): return len(self.queues[box])>0 def recv(self, box): # NB. Exceptions aren't caught X=self.queues[box][0] del self.queues[box][0] return X Then just use something like this: @wrapgenerator(component) def forwarderNay(self): "Simple data forwarding generator" while 1: if self.dataReady("inbox"): self.send("outbox",self.recv("inbox")+"Nay") elif self.dataReady("control"): if self.recv("control") == "shutdown": break yield 1 self.send("signal","shutdown") yield 0 In conjuction with a simplistic scheduler, and linkage functions this allows you to have something similar to CSP. I've come to the conclusion recently that the fact you can't* yield across multiple levels is actually beneficial because it encourages you to use many small components. Used standalone you can do this though: f=forwarderNay() for word in ["hello", "world", "test"]: f.send("inbox", word) f.next() print f.recv("outbox"), print Which of course outputs: * helloNay worldNay testNay For small scale things this amount of cruft is a bit of a pain. We're using *essentially* this approach to build network servers and it seems a rather satisfying way of doing so TBH. (not exactly this approach so if this looks odd, that's why - I'm not allowed to release the source for precisely what we're doing :-( but the above I was on the slides I *was* able to talk about... :-) Hope that's of some use... Best Regards, Michael. From pje at telecommunity.com Fri Oct 1 01:17:43 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Oct 1 01:17:44 2004 Subject: [Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration In-Reply-To: References: <5.1.1.6.0.20040930153803.02bc0140@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040930191017.03a5fc00@mail.telecommunity.com> At 11:36 PM 9/30/04 +0100, Michael Sparks wrote: >On Thu, 30 Sep 2004, Phillip J. Eby wrote: >... > > A mechanism to pass values or exceptions into generators > >[ Possibly somewhat off topic, and apologies if it is, and I'm positive > someone's done something similar before, but I think it's relevant to > the discussion in hand -- largely because the above use case *doesn't* > require changes to python... ] I know it doesn't; peak.events does this now, but in order to have a decent programmer interface, the implementation involves a fair amount of magic. Similarly, PEP 334 doesn't call for anything you can't do with a bit of work and magic. I was suggesting, however, that a true "simple co-routine" (similar to a generator but with bidirectional communication of values and exceptions) would be a valuable addition to the language, in the area of simplifying async programming in e.g. Twisted and peak.events. To put it another way: Slap the current title of PEP 334 onto the body of PEP 288, and change its syntax so you have a way to pass values and exceptions *in* to a suspended "coroutine" (a new and different animal from a generator), and I'm sold. From ncoghlan at email.com Fri Oct 1 03:09:02 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 03:09:08 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <5.1.1.6.0.20040930111348.038beb60@mail.telecommunity.com> References: <20040930135718.GA208@panix.com> <1096539066.415bdbbaaed43@mail.iinet.net.au> <20040930135718.GA208@panix.com> <5.1.1.6.0.20040930111348.038beb60@mail.telecommunity.com> Message-ID: <1096592942.415cae2e6cada@mail.iinet.net.au> Quoting "Phillip J. Eby" : > Using the C equivalent of 'imp.find_module()' should cover all these cases, > and any new forms of PY_SOURCE or PY_COMPILED that come up in future. That's essentially what the patch does. PyRun_SimpleModuleFlags uses _PyImport_FindModule to obtain the absolute file location, then uses PyRun_SimpleFileExFlags to actually run the script. _PyImport_FindModule is a trivial wrapper around import.c's find_module() (which, I believe, is what imp.find_module() invokes) When multiple versions exist, the patch operates on what find_module() returns (which ends up using the preference order '.py', '.pyw', '.py[co]') Cheers, Nick. -- Nick Coghlan Brisbane, Australia From ncoghlan at email.com Fri Oct 1 03:31:30 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 03:31:36 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <1096554430.20270.23.camel@geddy.wooz.org> References: <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <1096554430.20270.23.camel@geddy.wooz.org> Message-ID: <1096594290.415cb372f2f41@mail.iinet.net.au> Quoting Barry Warsaw : > > At 08:21 PM 9/30/04 +1000, Nick Coghlan wrote: > > >However, another possibility occurred to me: > > > > > >try: > > > # Do stuff > > >except sys.special_exceptions: > > > raise > > >except: > > > # Deal with all the mundane stuff > > +0, except that I'd rather see it put in the exceptions module and given > a name in builtins. Hmm, I forgot about the existence of the exceptions module. I agree that makes a more sensible location than sys. As for it being a builtin, I have no objections to that. I'll come up with two patches, though. One to create the tuple in exceptions, and one to give it a name in builtins (since the latter would presumably be more controversial, like any new builtin). My current list of exceptions for inclusion is KeyboardInterrupt, MemoryError, SystemExit & StopIteration. The inclusion of StopIteration is what makes me prefer 'special_exceptions' as the name of the tuple, rather than 'critical_exceptions'. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From bac at OCF.Berkeley.EDU Fri Oct 1 04:44:28 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 04:44:33 2004 Subject: [Python-Dev] Re: Dangerous exceptions (was Re: Another test_compilermystery) In-Reply-To: <1f7befae040907201426c33006@mail.gmail.com> References: <002d01c48083$9a89a6c0$5229c797@oemcomputer> <20040816112916.GA19969@vicky.ecs.soton.ac.uk> <1f7befae04090422024afaee58@mail.gmail.com> <413DBB19.40602@zope.com> <1094566431.8341.25.camel@geddy.wooz.org> <413DCFBD.7010306@theopalgroup.com> <1094572868.8342.43.camel@geddy.wooz.org> <1f7befae040907201426c33006@mail.gmail.com> Message-ID: <415CC48C.7050209@ocf.berkeley.edu> Tim Peters wrote: > After a bit more thought (and it's hard to measure how little), I'd > like to see "bare except" deprecated. That doesn't mean no way to > catch all exceptions, it means being explicit about intent. Only a > few of the bare excepts I've seen in my Python life did what was > actually intended, and there's something off in the design when the > easiest thing to say usually does a wrong thing. > Giving it same amount of thought as Tim, I like this idea as well. I don't think the burden of having to always specify an exception to catch is that much work and would definitely be more explicit. Plus it is not hard to teach to newbies; just tell them to catch the exception all of the "safe" exceptions inherit from. Is it PEP time? We have now had the idea of reorganizing the exception hierarchy come up in this thread and in early August (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is-an-exception-unless-it-doesn-t-inherit-from-exception for the thread on reorganizing for the purpose of using new-style classes and the idea of having all raisable objects inherit from a specific base class). It seems like a serious enough of an idea that it will happen for Python 3000 and thus should be discussed; never too early to start. But maybe it at least deserves to be mentioned in PEP 3000 (Guido?)? > Oh well. We should elect a benevolent dictator for Python! > Here, here! =) -Brett From bac at OCF.Berkeley.EDU Fri Oct 1 05:11:41 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 05:11:45 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-01 through 2004-09-15 [draft] Message-ID: <415CCAED.9050203@ocf.berkeley.edu> ===================== Summary Announcements ===================== Python 2.4a3 has been released. Go to http://www.python.org/2.4/ and give it twirl. Sorry for this summary being so short, but school has started back up again so I am in the middle of suddenly having to switch back into homework mode after spending the summer just having a 9:00-17:00 job. And since it is a new school year I am going to abuse this space and say that anyone in San Luis Obispo, including students, should join the `SLO Meetup`_ coming up on October 14. .. _SLO Meetup: http://python.meetup.com/95/ ========= Summaries ========= -------------------- Movement in PEP Land -------------------- `PEP 334`_ (Simple Coroutines via SuspendIteration) came into existence. `PEP 328`_ (Relative Imports) got some discussion on postponing making imports absolute instead of the relative/absolute semantics they have now. As it stands it looks like the changeover might get pushed off. `PEP 292`_ (Simpler String Substitutions) seems to finally be done and locked down. `PEP 335`_ (Overloadable Boolean Operators) came into existence. .. _PEP 334: http://www.python.org/peps/pep-0334.html .. _PEP 328: http://www.python.org/peps/pep-0328.html .. _PEP 292: http://www.python.org/peps/pep-0292.html .. _PEP 335: http://www.python.org/peps/pep-0335.html Contributing threads: - `PEP 334 - Simple Coroutines via SuspendIteration <>`__ - `PEP 328 - Relative Imports <>`__ - `Re: Alternative Implementation for PEP 292:Simple String Substitutions <>`__ - `ANN: PEP 335: Overloadable Boolean Operators <>`__ ------------------------------------------------------ __str__, __unicode__, and how to have them play nicely ------------------------------------------------------ Did you know that __str__ methods are allowed to return Unicode objects? Well, it turns out they can, but that str() (which calls PyObject_Str()) automatically tries to convert the value returned by __str__ into ASCII. Basically __str__ shouldn't return Unicode if you can help it and you should use __unicode__ instead and reserve __str__ to return str objects only. Contributing threads: - `unicode and __str__ <>`__ ---------------------- Backporting C APIs bad ---------------------- Somebody (*cough* Guido *cough*) asked if the datetime C API could be backported to 2.3 . The argument was that the only person who would probably use it is the person who asked for it, the author of cx_Oracle. Well, pretty much everyone spoke up against this. The argument went that adding an API would just be bad since there would suddenly be a point in the 2.3 releases where backwards compatibility was broken. People brought up the point of 2.2 where in 2.2.1 booleans were added and how that has caused compatibility headaches for some people. In the end the API was not backported. Contributing threads: - `Re: [Python-checkins] python/dist/src/Modules threadmodule.c, 2.56, 2.56.8.1 <>`__ ---------------------------------------------------------------------- Got to love race conditions thanks to the filesystem and external apps ---------------------------------------------------------------------- Tim Peters found a race condition one can have on Windows if you have an app that uses the low-level hooks into the filesystem. If you create a file, delete it, and then try to delete the directory the directory deletion will fail since the file is not deleted yet. What can happen is an indexing program can still be indexing the file before the filesystem is allowed to delete it and thus the directory is not truly empty when the directory deletion is executed. Fun stuff. Contributing threads: - `Coernic Desktop Search versus shutil.rmtree <>`__ ------------------------------- Python 2.4a3 is out the door!!! ------------------------------- Go to http://www.python.org/2.4/ , download it (using the bz2 version if possible so as to save on bandwidth), and run it against your code, run the test suite, put it on your head and sell yourself to an art gallery, etc. Contributing threads: - `RELEASED Python 2.4, alpha 3 <>`__ ---------------------------- Cleaning the Exception House ---------------------------- The idea of reorganizing the exceptions hierarchy came up again (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is-an-exception-unless-it-doesn-t-inherit-from-exception for a previous discussion on this). This time, the idea of separating the hierarchy into exceptions one would like to catch with a bare 'except' and those that you wouldn't was brought up. The idea is that some exceptions, such as MemoryError, one does not want to catch in a blanket statement usually. Chances of recovering from that kind of exception is low and should only be caught if you know what you are doing. So tweaking the exception hierarchy so that exceptions that were not near-catastrophic could inherit from an exception class that people could catch so as to allow the proper exceptions to propagate to the top-level without issue. Tim Peters even went as far as to suggest deprecating bare 'except' statements. This would force people to be explicit about what they want to catch, whether it be all "safe" exceptions or *all* exceptions. As it stands now no officially decision has been made for Python 3000 since that is about the only place this could happen. Contributing threads: - `Dangerous exceptions <>`__ --------------------------------------------------------------- Making decorators not look like decorators to the outside world --------------------------------------------------------------- Raymond Hettinger pointed out that a decorator does not, by default, look like the function that it is fiddling with (if that is the intent). Since most decorators will most likely be a wrapper function some things need to be set in the wrapper in order not to mask things in the wrapped function (doc string, argument parameters, etc.). So Raymond pointed out some things one can do. This also led to the suggestion of having a common name used to store a reference back to the wrapped function. There was also the mention that a decorator-oriented module in the stdlib will probably materialize in Python 2.5 . For now, though, stick recipes either in the Python Cookbook or in the Python wiki at http://www.python.org/cgi-bin/moinmoin/PythonDecoratorLibrary . Contributing threads: - `decorator support <>`__ =============== Skipped Threads =============== - random.py still broken wrt. urandom - random.py fixage - Re: [Python-checkins] python/dist/src/Lib/test test_compiler.py, 1.5, 1.6 test_decimal.py, 1.13, 1.14 - assert failure on obmalloc - Re: [Python-checkins] python/dist/src/Modules socketmodule.c, 1.304, 1.305 - Install-on-first-use vs. optional extensions - Console vs. GUI applications - Adding status code constants to httplib - PEP 292: method names - PEP 265 - Sorting dicts by value - httplib is not v6 compatible, is this going to be fixed? - OT: Unicode history - --with-tsc compile fails - tempfile.TemporaryFile on Windows NT - PyExc_UnicodeDecodeError - urllib.urlopen() vs IDNs, percent-encoded hosts, ':' - tabs in httplib.py and test_httplib.py There is now a vimrc file in the Misc directory that sets things up to follow PEPs 7 & 8 - strawman decision: @decorator won't change - unicode inconsistency? From ncoghlan at email.com Fri Oct 1 06:47:15 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 06:47:22 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <1096539066.415bdbbaaed43@mail.iinet.net.au> References: <1096539066.415bdbbaaed43@mail.iinet.net.au> Message-ID: <1096606035.415ce15346b75@mail.iinet.net.au> Quoting Nick Coghlan : > - run the located script as __main__ (note that containing packages are NOT > imported first - it's as if the relevant module was executed directly from > the > command line) I've realised that this means that the '-m' option doesn't understand packages that modify __path__ in their __init__.py scripts. What do people think of semantics which say "python -m some.package.module" means that "some.package" gets imported before "module" gets executed as "__main__"? The advantages are that __path__ will be interpreted correctly and package initialisation code will be invoked before the module is executed. It seems slightly odd to be importing things before the script starts executing, but these semantics seem to be more in line with the behaviour of the rest of Python's import machinery. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From pje at telecommunity.com Fri Oct 1 06:57:45 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Oct 1 06:57:48 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <1096606035.415ce15346b75@mail.iinet.net.au> References: <1096539066.415bdbbaaed43@mail.iinet.net.au> <1096539066.415bdbbaaed43@mail.iinet.net.au> Message-ID: <5.1.1.6.0.20041001005128.02b54750@mail.telecommunity.com> At 02:47 PM 10/1/04 +1000, Nick Coghlan wrote: >Quoting Nick Coghlan : > > - run the located script as __main__ (note that containing packages are NOT > > imported first - it's as if the relevant module was executed directly from > > the > > command line) > >I've realised that this means that the '-m' option doesn't understand packages >that modify __path__ in their __init__.py scripts. > >What do people think of semantics which say "python -m some.package.module" >means that "some.package" gets imported before "module" gets executed as >"__main__"? > >The advantages are that __path__ will be interpreted correctly and package >initialisation code will be invoked before the module is executed. > >It seems slightly odd to be importing things before the script starts >executing, >but these semantics seem to be more in line with the behaviour of the rest of >Python's import machinery. Although it would be nice if the new functionality supported existing scripts, I'd almost rather see the semantics of '-m foo.bar' be the same as '-c "from foo.bar import __main__; __main__()"', since the latter's semantics are much more well-defined. Alternatively, one could use '-m foo.bar.__main__' or '-m timeit.main' or '-m pydoc.cli' or '-m unittest.main', and thus be explicit about exactly what will be run. In either case, though, I think import semantics are easier to explain/understand than __name__=='__main__' semantics, especially in situations where the "script" may be re-imported by other code it imports (e.g. the unittest module). From ncoghlan at email.com Fri Oct 1 08:03:02 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 08:03:08 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <1096594290.415cb372f2f41@mail.iinet.net.au> References: <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <1096554430.20270.23.camel@geddy.wooz.org> <1096594290.415cb372f2f41@mail.iinet.net.au> Message-ID: <1096610582.415cf316c0dfe@mail.iinet.net.au> A candidate implementation is up at SF. Patch # 1038256. It uses the name 'special_exceptions' and includes SystemExit, MemoryError, KeyboardInterrupt and StopIteration (adding/removing exceptions from the list is easy, as is changing the name). One version of the patch creates 'exceptions.special_exceptions' only, and the other version also creates a builtin. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From python at rcn.com Fri Oct 1 08:28:17 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 1 08:29:47 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <1096610582.415cf316c0dfe@mail.iinet.net.au> Message-ID: <003201c4a77f$da68c140$e841fea9@oemcomputer> > A candidate implementation is up at SF. Patch # 1038256. > > It uses the name 'special_exceptions' and includes SystemExit, > MemoryError, > KeyboardInterrupt and StopIteration (adding/removing exceptions from the > list is > easy, as is changing the name). > > One version of the patch creates 'exceptions.special_exceptions' only, and > the > other version also creates a builtin. Since we're trying to catch anything *not* special, is the intended usage something like this: try: func() except special_exceptions: raise except: altfunc() # handle non-special exceptions Raymond From ncoghlan at email.com Fri Oct 1 08:36:27 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 08:36:33 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <5.1.1.6.0.20041001005128.02b54750@mail.telecommunity.com> References: <1096539066.415bdbbaaed43@mail.iinet.net.au> <1096539066.415bdbbaaed43@mail.iinet.net.au> <5.1.1.6.0.20041001005128.02b54750@mail.telecommunity.com> Message-ID: <1096612587.415cfaeb7ee5d@mail.iinet.net.au> Quoting "Phillip J. Eby" : > Although it would be nice if the new functionality supported existing > scripts, I'd almost rather see the semantics of '-m foo.bar' be the same as > '-c "from foo.bar import __main__; __main__()"', since the latter's > semantics are much more well-defined. > > Alternatively, one could use '-m foo.bar.__main__' or '-m timeit.main' or > '-m pydoc.cli' or '-m unittest.main', and thus be explicit about exactly > what will be run. > > In either case, though, I think import semantics are easier to > explain/understand than __name__=='__main__' semantics, especially in > situations where the "script" may be re-imported by other code it imports > (e.g. the unittest module). Jim suggested something similar. The thing is, such idioms are already quite easy to handle using '-c' (see above for an example ;). What isn't possible is invoking the "if __name__ == '__main__':" idiom without knowing the location of a module in the file system. About the closest we get for Unix'y platforms is to run 'python `python -c "import inspect; import x; print inspect.getsourcefile(x)"`'. (if the quotes aren't clear, those are backticks around the python -c invocation). Anyway, I think -m really only makes sense if it gives us the power to invoke a module as "__main__". Otherwise we might as well stick to using -c. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From ncoghlan at email.com Fri Oct 1 08:51:07 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 08:51:14 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <003201c4a77f$da68c140$e841fea9@oemcomputer> References: <003201c4a77f$da68c140$e841fea9@oemcomputer> Message-ID: <1096613467.415cfe5bcf784@mail.iinet.net.au> Quoting Raymond Hettinger : > Since we're trying to catch anything *not* special, is the intended > usage something like this: > > try: > func() > except special_exceptions: > raise > except: > altfunc() # handle non-special exceptions Yep. It's essentially a workaround for the fact that we can't do anything too drastic to the exception heirarchy without serious backwards compatibility problems, but have some 'critical' exceptions that most code shouldn't be suppressing. At the moment, 'except:' is bad, and in most cases 'except Exception:' isn't any better (which surprised me - I didn't realise this until Tim brought it up recently). Tim suggested 'except StandardError:' which is an improvement but still not quite right (SystemExit and StopIteration make it through then, but MemoryError and KeyboardInterrupt still get eaten). I'm dreaming of the day when I can hit 'Ctrl-C' on any Python script and actually have the darn thing stop without hitting it 10 more times ;) Cheers, Nick. -- Nick Coghlan Brisbane, Australia From mwh at python.net Fri Oct 1 13:43:56 2004 From: mwh at python.net (Michael Hudson) Date: Fri Oct 1 13:43:58 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <5.1.1.6.0.20040930153454.02bc04c0@mail.telecommunity.com> (Phillip J. Eby's message of "Thu, 30 Sep 2004 15:37:17 -0400") References: <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <1096539707.415bde3ba1425@mail.iinet.net.au> <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <5.1.1.6.0.20040930153454.02bc04c0@mail.telecommunity.com> Message-ID: <2mwtyapssj.fsf@starship.python.net> "Phillip J. Eby" writes: > At 12:52 PM 9/30/04 -0300, Lalo Martins wrote: >>On Thu, Sep 30, 2004 at 10:19:22AM -0400, Phillip J. Eby wrote: >> > >> > Also, maybe in 2.5 we could begin warning about bare excepts that aren't >> > preceded by non-bare exceptions. >> >>try: >> foo() >>except: >> print_or_log_exception_in_a_way_that_is_meaningful() >> raise >> >>doesn't seem to be incorrect to me. For example, if the program >>is a daemon, I want the exception logged somewhere so that I can >>see it later, because I won't be watching stderr. > > 1. If the exception raised is a MemoryError, your daemon is in trouble. Not necessarily. Typing 'range(sys.maxint)' into the interactive interpreter gives a fairly harmless MemoryError (the only exception PyRepl doesn't catch is SystemExit). > 2. I said *warn*, and it'd be easy to suppress the warning using > except Exception:', if that's what you really mean Well, apart from the fact that this doesn't catch, uh, exceptions that don't derive from Exception? Until/unless that's enforced (something that gets thrashed around every other month) there's still a place for 'bare except:'. Cheers, mwh -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods From goodger at python.org Fri Oct 1 16:24:39 2004 From: goodger at python.org (David Goodger) Date: Fri Oct 1 16:26:05 2004 Subject: [Python-Dev] ConfigParser patches Message-ID: <415D68A7.3080702@python.org> I have patches up on SF for two ConfigParser bugs, which I'd like to check in ASAP. Any objections? http://www.python.org/sf/1017864: Case sensitivity bug in ConfigParser ====================================================================== This is a simple bug relating to default values passed in as dictionaries, illustrated here: >>> import ConfigParser >>> cp = ConfigParser.ConfigParser({"foo": "Bar"}) >>> cp.get("DEFAULT", "Foo") 'Bar' >>> cp = ConfigParser.ConfigParser({"FOO": "Bar"}) # <-- note CAPS >>> cp.get("DEFAULT", "FOO") # same for "foo", "Foo", etc. Traceback (most recent call last): ... ConfigParser.NoOptionError: No option 'foo' in section: 'DEFAULT' The patch applies ConfigParser.optionxform to keys of defaults when supplied, to be consistent with the handling of keys of config file entries and runtime-set options. A test patch is also included. http://www.python.org/sf/997050: ConfigParser behavior change ============================================================= This may be more controversial. In Python 2.3 and earlier, ConfigParser implicitly allowed non-string values to be set. http://python.org/sf/810843 asked for clarification, and in rev 1.65 an explicit type check was added that raises TypeError. The problem is that this breaks Docutils code, and I suspect it will break other code as well, which we won't hear about until after Python 2.4 final is released. Setting non-string values worked just fine for application-internal use, as long as interpolation is turned off (RawConfigParser is used or ConfigParser with "raw" parameter set) and config files aren't written. A new SafeConfigParser class was added in Python 2.3: New applications should prefer this version if they don't need to be compatible with older versions of Python. My solution is to add the new string-only restriction to the SafeConfigParser class, leave existing ConfigParser & RawConfigParser behavior alone, and document the conditions under which non-string values work. Code, doc, and test patches included. What say you? -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/d193ff4f/signature.pgp From python at rcn.com Fri Oct 1 22:07:44 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 1 22:09:03 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415D68A7.3080702@python.org> Message-ID: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> [David Goodger] > In Python 2.3 and earlier, > ConfigParser implicitly allowed non-string values to be set. > http://python.org/sf/810843 asked for clarification, and in rev 1.65 > an explicit type check was added that raises TypeError. Instead of an explicit type check, it may be better to wrap the interpolation step in a try/except. For non-string objects, raise a TypeError with an informative error message. This would meet the OP's need for an informative error message while leaving the module backwards compatible for internal uses of ConfigParser. Ideally, the docs should discourage further non-string uses and advise that ConfigParser will be string only for Py3.0. Raymond From fdrake at acm.org Fri Oct 1 22:37:57 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 1 22:38:06 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> Message-ID: <200410011637.57629.fdrake@acm.org> On Friday 01 October 2004 04:07 pm, Raymond Hettinger wrote: > Ideally, the docs should discourage further non-string uses and advise > that ConfigParser will be string only for Py3.0. Yikes! Can't we just toss it for Py3K? This module just hasn't held up, and exposes a really poor model even for .ini-style configuration files. -Fred -- Fred L. Drake, Jr. From bac at OCF.Berkeley.EDU Fri Oct 1 23:45:39 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 23:48:14 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-01 through 2004-09-15 [draft] In-Reply-To: <415CCAED.9050203@ocf.berkeley.edu> References: <415CCAED.9050203@ocf.berkeley.edu> Message-ID: <415DD003.1040000@ocf.berkeley.edu> I forgot to say that I plan to send out this summary some time between Saturday evening and Monday night. -Brett From bac at OCF.Berkeley.EDU Fri Oct 1 23:52:09 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 23:53:12 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <200410011637.57629.fdrake@acm.org> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> Message-ID: <415DD189.7000900@ocf.berkeley.edu> Fred L. Drake, Jr. wrote: > On Friday 01 October 2004 04:07 pm, Raymond Hettinger wrote: > > Ideally, the docs should discourage further non-string uses and advise > > that ConfigParser will be string only for Py3.0. > > Yikes! Can't we just toss it for Py3K? This module just hasn't held up, and > exposes a really poor model even for .ini-style configuration files. > Ditto from me. Personally I would want something that used XML (like property lists on OS X), but I realize people still want a config file style that is easy to modify in a text editor, so I don't see the .ini style going away. But we could stand to come up with a uniform interface that both an XML and .ini config file parsers could use for consistency and thus support both styles. Wasn't there talk for a while of doing an shootout of config file packages like we did for optparse? -Brett From gvanrossum at gmail.com Sat Oct 2 00:09:02 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 00:09:05 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DD189.7000900@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: > > > Ideally, the docs should discourage further non-string uses and advise > > > that ConfigParser will be string only for Py3.0. > > > > Yikes! Can't we just toss it for Py3K? This module just hasn't held up, and > > exposes a really poor model even for .ini-style configuration files. Hear hear. > Ditto from me. Personally I would want something that used XML (like property > lists on OS X), but I realize people still want a config file style that is > easy to modify in a text editor, so I don't see the .ini style going away. But > we could stand to come up with a uniform interface that both an XML and .ini > config file parsers could use for consistency and thus support both styles. Well, for me personally, .ini style config files still win over XML every day. And I now have significant experience with both here at ESI. What sucks (relatively) is the specific way that ConfigParser provides access to .ini files; I always end up writing a wrapping layer around it. > Wasn't there talk for a while of doing an shootout of config file packages like > we did for optparse? There are two different axes here: the config file format and the API for accessing it. I'm not sure that attempting to provide a single API for both XML and .ini files will work; the API should reflect the structure of the file (to some extent) and that's just too different. But I may be wrong; I do see some similarities: What I have come to like, both for dealing with XML and with .ini files, is something that lets you map the configuration values (or whatever it is that you're parsing, really) to Python attributes. This is really some kind of simplified DTD support, and there are different ways to go about it; but the net result is that you end up writing e.g. options.client.max_retries (which is an int with a default value) rather than options.getint("client", "max-retries"). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From goodger at python.org Sat Oct 2 00:27:59 2004 From: goodger at python.org (David Goodger) Date: Sat Oct 2 00:28:11 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> Message-ID: <415DD9EF.7000606@python.org> [Raymond Hettinger] > Instead of an explicit type check, it may be better to wrap the > interpolation step in a try/except. Problem with this approach: the interpolation is done at .get() time, so any exception raised would be far from the culprit, .set(). > Ideally, the docs should discourage further non-string uses My doc patch makes clear that non-string uses don't work with interpolation & file output. > and advise that ConfigParser will be string only for Py3.0. Are there any other Python 3.0 advisories in the docs now? Seeing as how there's no definite schedule or plan for Python 3.0, indeed it's "a hypothetical future release" (PEP 3000), that may not be a precedent we want to set. In any case, I don't know that ConfigParser *should* be declared string-only at all. Docutils uses it only to read config files; the non-string values come from interpreting those values. It's quite convenient to plug the interpreted values back into the ConfigParser dicts and use those. -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/cc8fc5fc/signature.pgp From tim.peters at gmail.com Sat Oct 2 00:29:49 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sat Oct 2 00:29:53 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: <1f7befae0410011529587160e2@mail.gmail.com> [Guido] ... > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). LOL. I wrote a Config wrapper for SpamBayes like that. Then I stopped paying attention, and next thing I knew all the option-accessing code had been changed to look like SB's current prob = options["Classifier", "unknown_word_prob"] instead. But even that's better than options.getint("client", "max-retries")! Encoding the return type at every access point is Plain Wrong. From bac at OCF.Berkeley.EDU Sat Oct 2 00:30:44 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 00:30:55 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: <415DDA94.4060505@ocf.berkeley.edu> Guido van Rossum wrote: [SNIP] >>Ditto from me. Personally I would want something that used XML (like property >>lists on OS X), but I realize people still want a config file style that is >>easy to modify in a text editor, so I don't see the .ini style going away. But >>we could stand to come up with a uniform interface that both an XML and .ini >>config file parsers could use for consistency and thus support both styles. > > > Well, for me personally, .ini style config files still win over XML > every day. And I now have significant experience with both here at > ESI. > OK. Do realize that plists are basically .ini style just expressed in XML:: Section key value key2 value2 I am not thinking of anything fancy or beyond something like this; .ini files expressed in XML. Just thinking that XML might be nice since all of those poor souls who don't use Python have easy access to an XML parser but not necessarily a .ini file parser. > What sucks (relatively) is the specific way that ConfigParser provides > access to .ini files; I always end up writing a wrapping layer around > it. > > >>Wasn't there talk for a while of doing an shootout of config file packages like >>we did for optparse? > > > There are two different axes here: the config file format and the API > for accessing it. > > I'm not sure that attempting to provide a single API for both XML and > .ini files will work; the API should reflect the structure of the file > (to some extent) and that's just too different. But I may be wrong; I > do see some similarities: > For free-wheeling XML, yes, I would agree, but if we restrict the DTD/schema/spec to be fairly similar to .ini files then I don't think the structure of the files will be that drastic. And as long as people use the API to create/edit/save the files then we can enforce universal rules that makes sure that the difference is really just in the structure of the output. > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). > That is what I am thinking. You have sections and then key/value pairs within the sections in both the XML and the .ini format. We then just agree on how case-sensitivity, naming, etc. works within the file formats and then provide a common-place interface much like the one you lay out above. Is this worth working on now or wait until Py3k? It probably seems like I am playing the PEP drum (not a fan of the harp =) a lot lately in regards to stuff that is going to not be an issue until Py3k, but the longer we have to work on something the better chance we have of having something that everyone likes. Plus for things like modules we can provide it for people now as an external module for experimentation and thus be a little willy-nilly with the API initially until we have it nailed and either integrate into the stdlib beofre Py3k or just wait until then. -Brett From janssen at parc.com Sat Oct 2 00:31:14 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Oct 2 00:31:38 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: Your message of "Fri, 01 Oct 2004 15:09:02 PDT." Message-ID: <04Oct1.153122pdt."58617"@synergy1.parc.xerox.com> > Well, for me personally, .ini style config files still win over XML > every day. And I now have significant experience with both here at > ESI. > > What sucks (relatively) is the specific way that ConfigParser provides > access to .ini files; I always end up writing a wrapping layer around > it. Me too. I like the .ini file format (mail header formats will always win over XML for config files, I suspect), but had to wrap it for my latest project. Mainly for two reasons: to describe a hierarchy of config files to process when called, and to provide getint, getbool, and getfloat methods. Bill From goodger at python.org Sat Oct 2 00:29:22 2004 From: goodger at python.org (David Goodger) Date: Sat Oct 2 00:32:37 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: <415DDA42.3010904@python.org> [Guido van Rossum] > What sucks (relatively) is the specific way that ConfigParser > provides access to .ini files; I always end up writing a wrapping > layer around it. Unfortunately ConfigParser discourages such wrappers by hiding too much of its implementation as _private attributes (not __class_privates, so they *are* accessible, but even single-underscore _privacy screams "don't touch this -- subject to change without notice"). Back on topic though, I'd like to see +/-'s on the two patches. The first one (http://www.python.org/sf/1017864) is IMO a no-brainer. The second (http://www.python.org/sf/997050) is what I'm really interested in: do we maintain the existing API and functionality, or is it OK to break existing code by *removing* functionality? -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/c85cd3b4/signature.pgp From barry at python.org Sat Oct 2 00:38:48 2004 From: barry at python.org (Barry Warsaw) Date: Sat Oct 2 00:38:55 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: <1096670328.8358.46.camel@geddy.wooz.org> On Fri, 2004-10-01 at 18:09, Guido van Rossum wrote: > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). An enthusiastic +1. I'll just note that I often find that generally useful when dealing with XML files, not just those that represent configuration values. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/4330b42b/attachment.pgp From gvanrossum at gmail.com Sat Oct 2 00:43:35 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 00:43:42 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DDA94.4060505@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: > > Well, for me personally, .ini style config files still win over XML > > every day. And I now have significant experience with both here at > > ESI. > > OK. Do realize that plists are basically .ini style just expressed in XML:: > > > "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> > > > Section > > key > value > key2 > value2 > > > > > I am not thinking of anything fancy or beyond something like this; .ini files > expressed in XML. Just thinking that XML might be nice since all of those poor > souls who don't use Python have easy access to an XML parser but not > necessarily a .ini file parser. This reveals IMO a big mistake in thinking about configuration files. The most important user of a config file is not the programmer who has to get data out of it; the most important user is the user who has to edit the config file. The outrageous verbosity of XML makes the above example a complete usability liability. Now, if you're talking about config files that represent options that the user edits in a convenient application-specific options dialog, that's a different story; I think XML is well-suited for that; but I'm talking about the classic configuration file pattern where you use your favorite flat-file text editor to edit the options file. In that situation, using XML is insane. > Is this worth working on now or wait until Py3k? I see no advantage in waiting until Py3K; this is not a language issue and there is no problem with having several library modules (as long as it's clear which one is deprecated). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Oct 2 01:18:28 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 01:18:36 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <415DE5C4.9080107@ocf.berkeley.edu> Guido van Rossum wrote: [SNIP] > > This reveals IMO a big mistake in thinking about configuration files. > The most important user of a config file is not the programmer who has > to get data out of it; the most important user is the user who has to > edit the config file. The outrageous verbosity of XML makes the above > example a complete usability liability. > > Now, if you're talking about config files that represent options that > the user edits in a convenient application-specific options dialog, > that's a different story; I think XML is well-suited for that; but I'm > talking about the classic configuration file pattern where you use > your favorite flat-file text editor to edit the options file. In that > situation, using XML is insane. > I am thinking of the app-specific options dialog situation for XML and I do think that using XML for the classic style of editing the flat file by hand is insane. My thinking on the XML idea is that people do write apps where the whole thing is behind a GUI and thus XML makes good sense. So why not make there lives a little easier by giving them a basic API? But it is not a make-or-break thing for me so I am willing to let it go if it is deemed not worth the effort. > >>Is this worth working on now or wait until Py3k? > > > I see no advantage in waiting until Py3K; this is not a language > issue and there is no problem with having several library modules (as > long as it's clear which one is deprecated). > OK, so how do people want to proceeed with this? PEP? Shootout? -Brett From gvanrossum at gmail.com Sat Oct 2 01:22:57 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 01:23:06 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DE5C4.9080107@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <415DE5C4.9080107@ocf.berkeley.edu> Message-ID: > My thinking on the XML idea is that people do write apps where the whole thing > is behind a GUI and thus XML makes good sense. So why not make there lives a > little easier by giving them a basic API? People who want to use XML already have all the rope they need. I would be in for adding something like my simplified XML handling class to the library, but I first have to lobby my employer to open-source it (not a problem, I just have to time the request right so it won't be in 2.4). Anyway, this needn't and shouldn't be specific to the problem of storing option values. > OK, so how do people want to proceeed with this? PEP? Shootout? I'm all for working code in this case, so let's do a shootout. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Oct 2 01:31:39 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 01:31:44 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <415DE5C4.9080107@ocf.berkeley.edu> Message-ID: <415DE8DB.4060908@ocf.berkeley.edu> Guido van Rossum wrote: >>My thinking on the XML idea is that people do write apps where the whole thing >>is behind a GUI and thus XML makes good sense. So why not make there lives a >>little easier by giving them a basic API? > > > People who want to use XML already have all the rope they need. I > would be in for adding something like my simplified XML handling class > to the library, but I first have to lobby my employer to open-source > it (not a problem, I just have to time the request right so it won't > be in 2.4). Anyway, this needn't and shouldn't be specific to the > problem of storing option values. > OK. > >>OK, so how do people want to proceeed with this? PEP? Shootout? > > > I'm all for working code in this case, so let's do a shootout. > OK. Should we do an announcement to python-announce and c.l.py saying we are looking for a new API for .ini file editing/accessing? -Brett From fperez528 at yahoo.com Sat Oct 2 01:34:20 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Sat Oct 2 01:34:25 2004 Subject: [Python-Dev] Re: ConfigParser patches References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: Guido van Rossum wrote: > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). That's exactly what I wrote for ipython's option handler, with a fairly complex (and brittle) code to allow recursive loading of config files, since ipython has the (very useful, it turns out) concept of configuration 'profiles' which can include generic defaults and override them for specialized tasks. This means that options files can specify include statements to pull in other files, recursively. This has proven to be very useful, the rc format is a plain text key/value pair, and internally the options structure is (almost) a simple class with attributes for the options. Because I wrote this in my second week of learning python, the actual implementation is an atrocious mess, and the options class unfortunately has its own methods, which prevents having options with certain reserved names (basically all the methods of a dict). This last part was obviously a big mistake. But the overall idea, despite the shortcomings of my implementation, I think satisfies multiple important (for me) requirements: - flat text for users to edit without pulling them into the mudpit of XML. I simply refuse to ask a user to hand-edit XML, period. From an aesthetic standpoint, I think it would be borderline criminal to do so. - recursive inclusion for handling complex configuration situations. - exposes the options as a simple object with attributes accessible as rc.opt1, rc.opt2, etc. This makes the code which uses all these options (IPython has a _ton_ of cmd line flags) as readable as possible. Anyway, this is just $.02 from yet another one who reinvented this particular wheel... Best, f. ps. On the topic of @decorators and IPython, after some consultations with the ipython user community, the next release will use % for magic functions. This should be out in a few weeks, far before 2.4 is officially out. From janssen at parc.com Sat Oct 2 03:03:35 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Oct 2 03:04:07 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: Your message of "Fri, 01 Oct 2004 15:30:44 PDT." <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <04Oct1.180337pdt."58617"@synergy1.parc.xerox.com> > Just thinking that XML might be nice since all of those poor > souls who don't use Python have easy access to an XML parser but not > necessarily a .ini file parser. Sigh. And here I just wrote a .ini parser for Java... :-). Bill From pinard at iro.umontreal.ca Sat Oct 2 02:43:54 2004 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Sat Oct 2 03:29:10 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <20041002004354.GA24775@titan.progiciels-bpi.ca> [Guido van Rossum] > The outrageous verbosity of XML makes the above example a complete > usability liability. [...] I'm talking about the classic configuration > file pattern where you use your favorite flat-file text editor to edit > the options file. In that situation, using XML is insane. XML may sometimes be useful _when_ a configuration _has_ to be a tree of thick (contents heavy) nodes. For simpler configurations files, and this covers a flurry of cases, Guido above expresses my feelings exactly. There are XML-fanatics -- the same as there are Unicode-fanatics :-). Not so long ago, I've seen a pushy trend by which `/etc/passwd' and a lot of other simple system files would be reformulated as XML, to be XML all over. (Maybe after an announcement of Microsoft -- real or made-up, I do not know -- that they aim such a thing for the next "Windows".) We should not go overboard with XML. Configuration files with lines and words for structuring of a two-level hierarchy have long proven their value. ConfigParser `.ini' files add a third level to these. and with good sense, we can still go another long way without resorting to XML. Besides, when configuration files are going to have some more complex structure, but explicitly made to drive big applications written in Python -- a common case for most of us presumably -- I found out that Python itself is a very convenient and powerful tool for expressing such parameterisation. Users can easily modify a carefully designed Python-written configuration template without knowing a bit about Python itself. If they know the concepts of the applications they are configuring, in my experience at least, they quickly get what needs to be done, and even grasp by mere mimetism, without much explanations, the bits of syntax to respect and follow. With proper care, this can be made incredibly powerful, while staying _way_ more legible than XML. Also compare speed and simplicity, for the application programmer, for bringing such configuration in memory, type concerns already addressed. Of course, XML better inter-operates with foreign programming languages and systems, and would undoubtedly allow easier communication with alien races from neighbouring galaxies :-). But deep down, I do not think that this need in configuration neutrality is common enough to warrant the added complexity, and loss of legibility, for most practical cases. -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From gvanrossum at gmail.com Sat Oct 2 03:40:25 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 03:40:31 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DE8DB.4060908@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <415DE5C4.9080107@ocf.berkeley.edu> <415DE8DB.4060908@ocf.berkeley.edu> Message-ID: > OK. Should we do an announcement to python-announce and c.l.py saying we are > looking for a new API for .ini file editing/accessing? I don't see the point -- it's not like this is the most important issue of the year. I would rather just see if any of the readers here happen to have code lying around that they like. It sounds like maybe the IPython stuff could be used after a thorough rewrite. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From just at letterror.com Sat Oct 2 11:16:29 2004 From: just at letterror.com (Just van Rossum) Date: Sat Oct 2 11:16:37 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: Message-ID: Guido van Rossum wrote: > > > Well, for me personally, .ini style config files still win over > > > XML every day. And I now have significant experience with both > > > here at ESI. > > > > OK. Do realize that plists are basically .ini style just expressed > > in XML:: > > > > > > > "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> > > > > > > Section > > > > key > > value > > key2 > > value2 > > > > > > > > > > I am not thinking of anything fancy or beyond something like this; > > .ini files expressed in XML. Just thinking that XML might be nice > > since all of those poor souls who don't use Python have easy access > > to an XML parser but not necessarily a .ini file parser. > > This reveals IMO a big mistake in thinking about configuration files. > The most important user of a config file is not the programmer who has > to get data out of it; the most important user is the user who has to > edit the config file. The outrageous verbosity of XML makes the above > example a complete usability liability. Plist is handy when: - you need a simple storage format for simple data structures - you want to be able to edit a file manually in exceptional situations If you need to *routinely* edit these files by hand than plist is indeed not the right choice. It's not a pretty format, but it's well defined. I find it extremely handy and use it all over the place. I'd be happy to contribute plistlib.py to the std library (ie. move it one level up from Lib/plat-mac/ ;-), but I doubt there's enough interest. Just From arigo at tunes.org Sat Oct 2 11:18:46 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat Oct 2 11:24:04 2004 Subject: [Python-Dev] [OT] stats on type-inferencing atomic types in local variables in the stdlib In-Reply-To: <415C665A.4060706@ocf.berkeley.edu> References: <415C665A.4060706@ocf.berkeley.edu> Message-ID: <20041002091846.GA16603@vicky.ecs.soton.ac.uk> Hello Brett, On Thu, Sep 30, 2004 at 01:02:34PM -0700, Brett C. wrote: > (101, ('BINARY_MULTIPLY', (8, 4))), > (106, ('BINARY_SUBSCR', 128)), > (118, ('GET_ITER', 128)), > (124, ('BINARY_MODULO', None)), > (195, ('meth_join', 4)), > (204, ('BINARY_ADD', (8, 8))), > (331, ('BINARY_ADD', (4, 4))), > (513, ('BINARY_LSHIFT', (8, 8))), > (840, ('meth_append', 128)), > (1270, ('PRINT_ITEM', 4)), > (1916, ('BINARY_MODULO', 4)), > (12302, ('STORE_SUBSCR', 64))] I was surprized at first to see so few operations involving integers. There isn't even LOAD_SUBSCR for a list and an integer, though I believe it to be a very common operation. It makes me guess that your type-inferencer does not recognize 'i' to be an integer in constructions like for i in range(...) ? If so, it might be worth considering that some built-ins (most likely range(), len(), enumerate()) should be treated specially. Remember there was also discussion in this list at some point about the compiler producing opcodes like BUILTIN_LEN. This means that it might be acceptable to break the precise semantics (which involve looking up the name 'len' or 'range' in the globals first) and just special-case these common built-ins. Armin From anthony at interlink.com.au Sat Oct 2 14:04:57 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Oct 2 14:07:43 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <415E9969.1040207@interlink.com.au> All this talk of a replacement for ConfigParser is all well and good (and I agree that a more pythonic interface would be nice)... but. We're under 2 weeks from beta1, and I can't see this new module being designed, implemented, and committed before then. Remember, once b1 is out, we're stuck with the API that we have come up with. I'd much rather see one or more packages developed alongside Python - we can then take the best of them (or maybe a merger of the best ideas for them) for 2.5. Which leaves David's original question about the two patches. While CP isn't perfect, and it would be nice to replace it, for 2.4, I don't think these patches make things any worse, and they add useful functionality (back, in one case) so I'm inclined towards accepting them. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From gvanrossum at gmail.com Sat Oct 2 18:22:42 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 18:22:54 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415E9969.1040207@interlink.com.au> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <415E9969.1040207@interlink.com.au> Message-ID: > All this talk of a replacement for ConfigParser is all well > and good (and I agree that a more pythonic interface would > be nice)... but. We're under 2 weeks from beta1, and I can't > see this new module being designed, implemented, and committed > before then. Remember, once b1 is out, we're stuck with the > API that we have come up with. Of course. Other recent proposals for brand new stuff also seem poorly timed. > I'd much rather see one or more packages developed alongside > Python - we can then take the best of them (or maybe a merger > of the best ideas for them) for 2.5. Right. So it went for optparse and logging. > Which leaves David's original question about the two patches. > While CP isn't perfect, and it would be nice to replace it, for > 2.4, I don't think these patches make things any worse, and they > add useful functionality (back, in one case) so I'm inclined > towards accepting them. If you think so, go ahead. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Oct 2 20:26:17 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 20:26:29 2004 Subject: [Python-Dev] [OT] stats on type-inferencing atomic types in local variables in the stdlib In-Reply-To: <20041002091846.GA16603@vicky.ecs.soton.ac.uk> References: <415C665A.4060706@ocf.berkeley.edu> <20041002091846.GA16603@vicky.ecs.soton.ac.uk> Message-ID: <415EF2C9.20400@ocf.berkeley.edu> Armin Rigo wrote: > Hello Brett, > > On Thu, Sep 30, 2004 at 01:02:34PM -0700, Brett C. wrote: > >> (101, ('BINARY_MULTIPLY', (8, 4))), >> (106, ('BINARY_SUBSCR', 128)), >> (118, ('GET_ITER', 128)), >> (124, ('BINARY_MODULO', None)), >> (195, ('meth_join', 4)), >> (204, ('BINARY_ADD', (8, 8))), >> (331, ('BINARY_ADD', (4, 4))), >> (513, ('BINARY_LSHIFT', (8, 8))), >> (840, ('meth_append', 128)), >> (1270, ('PRINT_ITEM', 4)), >> (1916, ('BINARY_MODULO', 4)), >> (12302, ('STORE_SUBSCR', 64))] > > > I was surprized at first to see so few operations involving integers. There > isn't even LOAD_SUBSCR for a list and an integer, though I believe it to be a > very common operation. It's picked up and listed above; 106 times. I already type check that lists are being indexed by integer or an object so I didn't bother to keep separate stats for those two cases. > It makes me guess that your type-inferencer does not > recognize 'i' to be an integer in constructions like > > for i in range(...) > > ? Right, I don't infer those situations. > If so, it might be worth considering that some built-ins (most likely > range(), len(), enumerate()) should be treated specially. Remember there was > also discussion in this list at some point about the compiler producing > opcodes like BUILTIN_LEN. This means that it might be acceptable to break the > precise semantics (which involve looking up the name 'len' or 'range' in the > globals first) and just special-case these common built-ins. > Not going to break semantics for my thesis (the whole was not to), but this is all going to be mentioned in the Future Work section on what can be done to improve the situation for type inferencing. It would definitely help if built-ins were known at compile-time, but since I have limited time I had to limit myself to what could be done without adding features to the compiler beyond just type inferencing. -Brett From ncoghlan at email.com Sat Oct 2 22:55:18 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sat Oct 2 22:55:25 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <415E9969.1040207@interlink.com.au> Message-ID: <1096750518.415f15b65512d@mail.iinet.net.au> Quoting Guido van Rossum : > Of course. Other recent proposals for brand new stuff also seem poorly > timed. > I may be guilty of that. Would it be worth adding a Python 2.5 group to the SF Patch tracker for things which shouldn't be considered seriously until after 2.4 final is out the door? Cheers, Nick. -- Nick Coghlan Brisbane, Australia From python at rcn.com Sun Oct 3 00:22:48 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 00:24:09 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <1096750518.415f15b65512d@mail.iinet.net.au> Message-ID: <002001c4a8ce$5b72dc40$e841fea9@oemcomputer> [Guido] > > Of course. Other recent proposals for brand new stuff also seem poorly > > timed. [Nick] > I may be guilty of that. Would it be worth adding a Python 2.5 group to > the SF > Patch tracker for things which shouldn't be considered seriously until > after 2.4 > final is out the door? Though ill timed, the -m idea is honking good. If it can be reliably worked out in its simplest form (not trying to be all things to all packages), it's worth going ahead for Py2.4. Command line guys like me need this every day. His patch will address a continual source of irritation and make the tool more pleasant to use. Raymond From python at rcn.com Sun Oct 3 00:41:25 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 00:42:43 2004 Subject: [Python-Dev] Py2.5 tracker group In-Reply-To: <1096750518.415f15b65512d@mail.iinet.net.au> Message-ID: <002601c4a8d0$f30d1000$e841fea9@oemcomputer> > Would it be worth adding a Python 2.5 group to > the SF > Patch tracker for things which shouldn't be considered seriously until > after 2.4 > final is out the door? Done. Raymond From gvanrossum at gmail.com Sun Oct 3 02:45:54 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 3 02:46:01 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <002001c4a8ce$5b72dc40$e841fea9@oemcomputer> References: <1096750518.415f15b65512d@mail.iinet.net.au> <002001c4a8ce$5b72dc40$e841fea9@oemcomputer> Message-ID: > Though ill timed, the -m idea is honking good. If it can be reliably > worked out in its simplest form (not trying to be all things to all > packages), it's worth going ahead for Py2.4. Command line guys like me > need this every day. His patch will address a continual source of > irritation and make the tool more pleasant to use. I was in fact thinking of the -m proposal when I wrote that... It's so easy to define an alias or use a one-line shell script for invoking Python with a full pathname that I'm really not sure I like the -m idea, and I worry that it would be done the wrong way because it's so close to beta. Half the time the problem is getting the path set in the first place, and there -m doesn't help you. Supporting it with packages seems insanity. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Sun Oct 3 03:14:07 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 03:16:10 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: Message-ID: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> > I was in fact thinking of the -m proposal when I wrote that... The timing does suck. > It's so easy to define an alias or use a one-line shell script for > invoking Python with a full pathname that I'm really not sure I like > the -m idea, My understanding was that it wasn't about a full pathname to python, it was about searching sys.path for the darned script so us poor Windows guys don't have to change directories a million times a day (no aliases or shebangs for us either). I have a big pile of batch files just to invoke timeit, texchecker, profile, etc. It bites the big one. If the patch doesn't get worked out for Py2.4, I hope it doesn't get abandoned. > Supporting it with packages seems insanity. No doubt. Raymond From ncoghlan at email.com Sun Oct 3 05:27:10 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sun Oct 3 05:27:16 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> Message-ID: <1096774030.415f718ed9a87@mail.iinet.net.au> Quoting Raymond Hettinger : > > It's so easy to define an alias or use a one-line shell script for > > invoking Python with a full pathname that I'm really not sure I like > > the -m idea, > > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). Correct. Ilya's original example was wanting to invoke pdb's script behaviour in order to debug another script. At the moment, you have to run: python -c "from inspect import getsourcefile; import pdb; print getsourcefile(pdb)" or python -c "from imp import find_module; print find_module("pdb")[1]" to find out where the file is, and then run python normally with the filename that gets printed by one of the above scripts. (With a Unix-style shell, backticks make that easier. Doesn't help on Windows though). The "-m" option aims to automate this process (using the latter 'find_module' approach in C code) > > Supporting it with packages seems insanity. > > No doubt. I'm starting to lean that way myself. The semantics for a straight module are easy, but the package version is horrible (Either we have a package getting imported before "__main__" starts running, or else we don't properly support package __path__ variables. I don't find either option appealing) I think I'll simply drop all package support from the patch and allow modules only. That will still cover the initial use cases (pdb, profile and the like) Adding support for packages later (in a different patch) would still be possible if there was interest. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From barry at python.org Sun Oct 3 05:43:06 2004 From: barry at python.org (Barry Warsaw) Date: Sun Oct 3 05:43:09 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096774030.415f718ed9a87@mail.iinet.net.au> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> <1096774030.415f718ed9a87@mail.iinet.net.au> Message-ID: <1096774986.7313.72.camel@geddy.wooz.org> On Sat, 2004-10-02 at 23:27, Nick Coghlan wrote: > I'm starting to lean that way myself. The semantics for a straight module are > easy, but the package version is horrible (Either we have a package getting > imported before "__main__" starts running, or else we don't properly support > package __path__ variables. I don't find either option appealing) > > I think I'll simply drop all package support from the patch and allow modules > only. That will still cover the initial use cases (pdb, profile and the like) > Adding support for packages later (in a different patch) would still be possible > if there was interest. +1 and I'd still love to see this in Python 2.4. Could you please assign the patch to someone other than me though? I don't honestly think I'll have time to review it before 2.4b1. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041002/4475e5a7/attachment-0001.pgp From fperez528 at yahoo.com Sun Oct 3 06:28:48 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Sun Oct 3 06:28:23 2004 Subject: [Python-Dev] RE: ConfigParser patches References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> Message-ID: Raymond Hettinger wrote: > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). > > I have a big pile of batch files just to invoke timeit, texchecker, > profile, etc. It bites the big one. Sorry for the plug, but I hope you'll forgive it if it actually turns out to be useful. You may want to have a look at ipython (http://ipython.scipy.org). IPython has a 'magic' run command, which allows you to run any python script from its command line: In [1]: cat argv.py #!/usr/bin/env python import sys print 'argv:',sys.argv In [2]: run argv argv: ['argv.py'] In its current form, it is just an execfile() on steroids. But it would be trivial to add to it a -m option so that it would search in sys.path if nothing in the current directory is found matching the given name. Ipython also gives you aliases and 'bookmarks', a persistent list of directories you can change to with a single 'cd' command: ############### 'screenshot' In [1]: pdoc alias Define an alias for a system command. '@alias alias_name cmd' defines 'alias_name' as an alias for 'cmd' Then, typing 'alias_name params' will execute the system command 'cmd params' (from your underlying operating system). [... snip. It's a long docstring] In [6]: pdoc bookmark Manage IPython's bookmark system. @bookmark - set bookmark to current dir @bookmark - set bookmark to @bookmark -l - list all bookmarks @bookmark -d - remove bookmark @bookmark -r - remove all bookmarks You can later on access a bookmarked folder with: @cd -b or simply '@cd ' if there is no directory called AND there is such a bookmark defined. ################ /screenshot Using ipython's profile system (referred to earlier in this thread), you can use it as a quasi-system shell. The provided 'pysh' profile loads all of your $PATH (with proper extensions in Windows) as ipython aliases, and you can do regular system things, but retaining python syntax. It has also a mechanism for capturing shell output into python variables and expanding back to the shell: ############### 'screenshot' fperez@maqroll[~/test]|4> print 'Yes, this is still %s !' % 'Python' Yes, this is still Python ! fperez@maqroll[~/test]|5> $$files=ls fperez@maqroll[~/test]|6> type files ------------------------> type(files) <6> fperez@maqroll[~/test]|7> len files ------------------------> len(files) <7> 31 fperez@maqroll[~/test]|8> for f in files: |.> if len(f)>10: |.> wc -l $f |.> 4 inspectbug.py 73 ramptest.py fperez@maqroll[~/test]|9> run argv.py argv: ['argv.py'] ################ /screenshot Note that the 'run' command optionally takes a -p switch, to run code under the profiler's control. Ipython can also automatically activate the pdb debugger inside any uncaught exception, thanks to the @pdb magic. I simply point these things in the hope that they are useful to you. From what I've read on this thread, ipython seems to already give most of the motivations for the -m switch, and it works today with python >= 2.2. Try it, you might like it :) (I'm not trying to say the -m switch is a bad idea, simply that ipython may provide a useful alternative today). Ok, end of blatant plug. Sorry if it was out of line. Cheers, f From gvanrossum at gmail.com Sun Oct 3 06:33:09 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 3 06:33:12 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096774030.415f718ed9a87@mail.iinet.net.au> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> <1096774030.415f718ed9a87@mail.iinet.net.au> Message-ID: > Correct. Ilya's original example was wanting to invoke pdb's script behaviour in > order to debug another script. At the moment, you have to run: > > python -c "from inspect import getsourcefile; import pdb; print getsourcefile(pdb)" > > or > > python -c "from imp import find_module; print find_module("pdb")[1]" > > to find out where the file is, and then run python normally with the filename > that gets printed by one of the above scripts. (With a Unix-style shell, > backticks make that easier. Doesn't help on Windows though). > > The "-m" option aims to automate this process (using the latter 'find_module' > approach in C code) I'm not going to stop you from adding this, but I do think you're overlooking the fact that if you need this a lot, it's trivial to write a tiny Python script that takes the place of "python -m", and putting that python script in your PATH. Something like: #!/usr/bin/env python import sys, imp fn = imp.find_module(sys.argv[1])[1] del sys.argv[0] sys.argv[0] = fn execfile(fn) seems do the trick (though it could use nicer error handling). Wouldn't it be easier to add *this* to the Tools/scripts directory (and the list of scripts that are installed by "make install" or the Windows equivalent) than to burden the Python interpreter with C code to do the same? Why does every handy thing that anyone ever thought of have to be a Python command line option? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Sun Oct 3 10:29:46 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Oct 3 10:37:17 2004 Subject: [Python-Dev] proposed struct module format code addition Message-ID: <20041003012803.FEA1.JCARLSON@uci.edu> Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah From bob at redivi.com Sun Oct 3 10:58:11 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Oct 3 10:58:45 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <20041003012803.FEA1.JCARLSON@uci.edu> References: <20041003012803.FEA1.JCARLSON@uci.edu> Message-ID: <5A34A47C-151A-11D9-AFF0-000A95686CD8@redivi.com> On Oct 3, 2004, at 4:29 AM, Josiah Carlson wrote: > I have produced a patch against the latest CVS to add support for two > new formatting characters in the struct module. It is currently an > RFE, > which I include a link to at the end of this post. Please read the > email before you respond to it. -- > I had proposed two new formatting characters, 'g' and 'G' (for biGint > or > lonG int). > > There was one primary purpose, to offer users the opportunity to > specify > their own integer lengths (very useful for cryptography, and real-world > applications that involve non-standard sized integers). Current > solutions involve shifting, masking, and multiple passes over data. I'm +1 on this.. I've dealt with 24, 48, and 128 bit integers before and it's always been a pain with the struct module. Another addition I'd like to see is bit length struct fields, but that opens an entirely different can of worms. -bob From python at rcn.com Sun Oct 3 17:56:07 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 17:57:28 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <20041003012803.FEA1.JCARLSON@uci.edu> Message-ID: <001f01c4a961$7f564aa0$b529cb97@oemcomputer> > If you believe this functionality is useful, or even if you think that I > am full of it, tell us: http://python.org/sf/1023290 The suggested alternative is adding two functions to binascii that parallel the existing hexlify() and unhexlify(). From gvanrossum at gmail.com Sun Oct 3 18:06:47 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 3 18:06:53 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <001f01c4a961$7f564aa0$b529cb97@oemcomputer> References: <20041003012803.FEA1.JCARLSON@uci.edu> <001f01c4a961$7f564aa0$b529cb97@oemcomputer> Message-ID: > > If you believe this functionality is useful, or even if you think that > I > > am full of it, tell us: http://python.org/sf/1023290 > > The suggested alternative is adding two functions to binascii that > parallel the existing hexlify() and unhexlify(). Those functions ought to exist whether or not this RFE is accepted. Here's the crux I think. Is this used often enough in a context where (a) the length of the number is fixed (not determined by a count in a previous field) and (b) preceded or followed by other fixed-length fields so that it makes sense to use the struct module for parsing or formatting those other fields? I have often found that amongst less-experienced programmers there is a great mystery about the correspondence between the "binary" representation of numbers (in strings of bytes) and the numeric objects that Python makes available (int, lont). Often the struct module is considered the only way to cross this boundary, while in fact there are many other approaches; often using the built-in functions ord() or chr() and shifting and masking works just as well, but you have to think about it the right way. I apologize for not having read the entire post before responding; in case the motivation is already there, that's great, and let it be a response to Raymond. If it is not there, I like Raymond's proposal better. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From goodger at python.org Sun Oct 3 18:09:04 2004 From: goodger at python.org (David Goodger) Date: Sun Oct 3 18:09:07 2004 Subject: [Python-Dev] Re: ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <415E9969.1040207@interlink.com.au> Message-ID: <41602420.1020002@python.org> [Anthony Baxter] >> Which leaves David's original question about the two patches. >> While CP isn't perfect, and it would be nice to replace it, for >> 2.4, I don't think these patches make things any worse, and they >> add useful functionality (back, in one case) so I'm inclined >> towards accepting them. [Guido van Rossum] > If you think so, go ahead. Taking this as approval, and since it's easier to ask forgiveness (and probably would've been, in the first place ;-), I've gone ahead and checked in the patches to SF bugs 1017864 & 997050, closed and marked the bug reports "accepted". -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041003/816d5dec/signature.pgp From tim.peters at gmail.com Sun Oct 3 20:14:35 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 3 20:15:10 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <20041003012803.FEA1.JCARLSON@uci.edu> References: <20041003012803.FEA1.JCARLSON@uci.edu> Message-ID: <1f7befae0410031114724f5f2c@mail.gmail.com> [Josiah Carlson] > ... > The 'p' format specifier is also not a standard C type, and yet it > is included in struct, specifically because it is useful. I never used it . > Argument: > You can already do the same thing with: > pickle.encode_long(long_int) > pickle.decode_long(packed_long) That isn't an argument I've seen before, although I pointed out these pickle functions in the tracker item. The argument there is that pickle demonstrates actual long<->bytes use cases in the core, implemented painfully in Python, and that the struct patch wouldn't make them easier. The argument was not that you can already use the pickle functions, it was that the pickle functions shouldn't need to exist -- they're isolated hacks that address only one part of the whole problem, and even that part is addressed painfully now. The workalike implementation in cPickle.c was actually easier than the Python implementation in pickle.py, because the former get to use the flexible C API functions I added for "this kind of thing". > and some likely soon-to-be included additions to the binascii module. The pickle functions *may* argue in favor of binascii functions, if those were actually specified. I'm not sure Raymond has ever spelled out what their signatures would be, so I don't know. If suitable functions in binascii will exist, then that (as Guido said) raises the bar for adding similar functionality to struct too, but does not (as Guido also said) preclude adding similar functionality to struct. > My Response: > That is not the same. Nontrivial problems require multiple passes > over your data with multiple calls. A simple: > struct.unpack('H3G3G', st) My problem with that use case is that I've never had it, and have never seen an app that had it. Since I've been around for decades, that triggers a suspicion that it's not a common need. ... > Argument: > The struct module has a steep learning curve already, and this new > format specifier doesn't help it. That's another argument I haven't seen before, but bears an hallucinatory resemblance to one I made. People have wondered how to convert between long and bytestring in Python for years, and prior to this iteration, they have always asked whether there's "a function" to do it. Like all the use cases I ever had, they have one long to convert, or one bytestring, at a time. "Ah, see the functions in binascii" would be a direct answer to their question, and one that wouldn't require that they climb any part of struct's learning curve. IOW, if it *could* be done without struct, I'm sure that would make life easier for most people who ever asked about it. For people who already know struct, the marginal learning burden of adding another format code is clearly minor. > I can't see how a new format specifier would necessarily make the > learning curve any more difficult, Neither can I, for people who already know struct. > if it was even difficult in the first place. It is difficult. There are many format codes, they don't all work the same way, and there are distinctions among: - native, standard, and forced endianness - native and standard (== no) alignment - native and standard sizes for some types Newbie confusion about how to use struct is common on c.l.py, and is especially acute among those who don't know C (just *try* to read the struct docs from the POV of someone who hasn't used C). > ... > If you believe this functionality is useful, or even if you think that I > am full of it, tell us: http://python.org/sf/1023290 I certainly would like to see more people say they'd *use* the g and G codes in struct even if "one shot" functions in binascii were available. I'd also like to see a specific design for binascii functions. I don't think "simple" would be an accurate description of those, if they're flexible enough to handle the common use cases I know about. They'd be more like long2bytes(n, length=None, unsigned=False, msdfirst=False) where, by default (length is None), long2bytes(n) is the same as pickle.encode_long(n), except that long2bytes(0) would produce '\0' instead of an empty string. Specifying length <= 0 is a ValueError. length > 0 would be akin to the C API _PyLong_AsByteArray(n, buf, length, not msdfirst, not unsigned) ValueError if n < 0 and unsigned=True. OverflowError if length > 0 and n's magnitude is too large to fit in length bytes. bytes2long(bytes, unsigned=False, msdfirst=False) would be akin to the C API _PyLong_FromByteArray(bytes, len(bytes), not msdfirst, not unsigned) except that bytes=="" would raise ValueError instead of returning 0. From jcarlson at uci.edu Sun Oct 3 20:22:30 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Oct 3 20:29:40 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: References: <001f01c4a961$7f564aa0$b529cb97@oemcomputer> Message-ID: <20041003110405.FEB5.JCARLSON@uci.edu> > > The suggested alternative is adding two functions to binascii that > > parallel the existing hexlify() and unhexlify(). > > Those functions ought to exist whether or not this RFE is accepted. No argument here. I believe the binascii and struct additions have orthogonal use-cases. > Here's the crux I think. Is this used often enough in a context where > (a) the length of the number is fixed (not determined by a count in a > previous field) and All use-cases I have for it, yes. Seemingly this is a yes with Bob Ippolito as well (he stated he uses 3, 6 and 16 byte integers, which certainly seem fixed). > (b) preceded or followed by other fixed-length > fields so that it makes sense to use the struct module for parsing or > formatting those other fields? In my case, yes. As provided in the email, two that I use right now are 'H3G3G' and '3G5G'. > I have often found that amongst less-experienced programmers there is > a great mystery about the correspondence between the "binary" > representation of numbers (in strings of bytes) and the numeric > objects that Python makes available (int, lont). Often the struct > module is considered the only way to cross this boundary, while in > fact there are many other approaches; often using the built-in > functions ord() or chr() and shifting and masking works just as well, > but you have to think about it the right way. Indeed, this /can/ be the case, but it is not in my case. Before I knew of struct, I had written my own binary packers and unpackers using ord and chr. While it worked, I nearly wept with joy when I discovered struct, which did 90% of what I needed it to do, in a much cleaner fashion. > I apologize for not having read the entire post before responding; in Yeah, it was a bit long, I believed there was a lot to say. - Josiah From shane.holloway at ieee.org Sun Oct 3 21:28:30 2004 From: shane.holloway at ieee.org (Shane Holloway (IEEE)) Date: Sun Oct 3 21:29:02 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <1f7befae0410031114724f5f2c@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> Message-ID: <416052DE.3020809@ieee.org> Since "long" is supposed to be a full-fledged member of the python building blocks, I'm +1 for functions being added in both binascii and struct. One of the greatest things I use struct is for is packing (and unpacking) the python building blocks for "external use" -- network, database, and (usually C) libraries. I think it would be best if all the building blocks could be packed and unpacked from one module. The additions to binascii would be more convenient to use of the two additions. But truth to tell, I rarely use binascii. I tend to prefer struct.pack with str.encode. What do you think about adding long.tobytes()/long.frombytes() to go with the new bytes() type? -Shane Holloway From jcarlson at uci.edu Sun Oct 3 21:48:43 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Oct 3 21:56:42 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <1f7befae0410031114724f5f2c@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> Message-ID: <20041003112337.FEB8.JCARLSON@uci.edu> > > The 'p' format specifier is also not a standard C type, and yet it > > is included in struct, specifically because it is useful. > > I never used it . If you have used PostgreSQL, all strings are a variant of pascal strings. It may be the case in other databases, but I have little experience with them. > > My Response: > > That is not the same. Nontrivial problems require multiple passes > > over your data with multiple calls. A simple: > > struct.unpack('H3G3G', st) > > My problem with that use case is that I've never had it, and have > never seen an app that had it. Since I've been around for decades, > that triggers a suspicion that it's not a common need. It may be domain specific. I've only been using Python for 4 1/2 years, yet I have used such structs to build socket protocols, databases and search engines, for both class and contract. Heck, I find it useful enough that I have considered donating to the PSF just to see this feature included. > > Argument: > > The struct module has a steep learning curve already, and this new > > format specifier doesn't help it. > > That's another argument I haven't seen before, but bears an > hallucinatory resemblance to one I made. People have wondered how to This was one argument that Raymond has offered a few times. In the case of native alignment issues that seem to be the cause of much frustration among new struct learners, this particular format specifier doesn't much apply; it is not native. > binascii" would be a direct answer to their question, and one that ... So there is no confusion, I agree with Raymond, Guido, and you, that the binascii function additions should occur. > Newbie confusion about how to use struct is common on c.l.py, and is > especially acute among those who don't know C (just *try* to read the > struct docs from the POV of someone who hasn't used C). Good point about the docs regarding not using C. Does it make sense to include documentation regarding C structs (perhaps in an appendix) to help those who have otherwise not experienced C? > I certainly would like to see more people say they'd *use* the g and G > codes in struct even if "one shot" functions in binascii were > available. > I'd also like to see a specific design for binascii functions. I > don't think "simple" would be an accurate description of those, if > they're flexible enough to handle the common use cases I know about. > They'd be more like [snip conversion operations] Those would work great for packing and unpacking of single longs. - Josiah From lars.blockken at telenet.be Sun Oct 3 22:27:12 2004 From: lars.blockken at telenet.be (Lars Blockken) Date: Sun Oct 3 22:27:13 2004 Subject: [Python-Dev] introducing Message-ID: <002101c4a987$5d5d9630$e701a8c0@lars> Hello, I'm Lars and i'm knew for python so i hope you can answer my questions in the future. Greetz Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041003/837912ce/attachment-0001.htm From carribeiro at gmail.com Sun Oct 3 22:52:12 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sun Oct 3 22:52:14 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <416052DE.3020809@ieee.org> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <416052DE.3020809@ieee.org> Message-ID: <864d370904100313524c83ef7@mail.gmail.com> On Sun, 03 Oct 2004 13:28:30 -0600, Shane Holloway (IEEE) wrote: > One of the greatest things I use struct is for is packing (and > unpacking) the python building blocks for "external use" -- network, > database, and (usually C) libraries. I think it would be best if all > the building blocks could be packed and unpacked from one module. > > The additions to binascii would be more convenient to use of the two > additions. But truth to tell, I rarely use binascii. I tend to prefer > struct.pack with str.encode. > > What do you think about adding long.tobytes()/long.frombytes() to go > with the new bytes() type? Sorry for introducing my not-very-qualified words on this topic, but... I've read the thread up to this point wondering why the bytes() type were not being thought of as a clean and definitive solution to this problem. It would allow to greatly simplify everything regarding struct, binascii and arbitrary low level data manipulation for networking and similar stuff. I also agree with Tim Peters comments regarding struct's C heritage -- I never really liked C even when I *had* to use it daily, and the struct syntax still reads alien to me. I know this is another timeframe entirely, but *if* my vote counted, I would be +1 for a future struct implementation tightly integrated with the bytes() type. But that's me anyway. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From martin at v.loewis.de Sun Oct 3 23:18:20 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Oct 3 23:18:22 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <864d370904100313524c83ef7@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <416052DE.3020809@ieee.org> <864d370904100313524c83ef7@mail.gmail.com> Message-ID: <41606C9C.5060908@v.loewis.de> Carlos Ribeiro wrote: > Sorry for introducing my not-very-qualified words on this topic, > but... I've read the thread up to this point wondering why the bytes() > type were not being thought of as a clean and definitive solution to > this problem. It would allow to greatly simplify everything regarding > struct, binascii and arbitrary low level data manipulation for > networking and similar stuff. No, it wouldn't. If you have a 'long' value, and you want to convert it to 'bytes', how exactly would you do that? Two's complement, I suppose - but that would close out people who want unsigned numbers. Also, do you want big-endian or little-endian? What about a minimum width, what about overflows? Tim has proposed a signature for binascii that covers all these scenarios, and I doubt it could get simpler then that and still useful. > I also agree with Tim Peters comments regarding struct's C heritage -- > I never really liked C even when I *had* to use it daily, and the > struct syntax still reads alien to me. I know this is another > timeframe entirely, but *if* my vote counted, I would be +1 for a > future struct implementation tightly integrated with the bytes() type. I think you will find that the struct module *already* supports the bytes type. The bytes type will be just a synonym for the current string type, except that people will stop associating characters with the individual bytes; plus the bytes type will be possibly mutable. As the struct module creates (byte) strings today, it will trivially support the bytes type. Regards, Martin From martin at v.loewis.de Sun Oct 3 23:20:47 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Oct 3 23:20:51 2004 Subject: [Python-Dev] introducing In-Reply-To: <002101c4a987$5d5d9630$e701a8c0@lars> References: <002101c4a987$5d5d9630$e701a8c0@lars> Message-ID: <41606D2F.6070004@v.loewis.de> Lars Blockken wrote: > Hello, > I'm Lars and i'm knew for python so i hope you can answer my questions > in the future. Hello Lars, If you are new to Python, python-dev is not the list you should use; it is meant for the development *of* Python, not for the development *with* Python. You can use the python-tutor list if you want to actively learn Python, or python-list (aka news:comp.lang.python) for general discussions and questions. Regards, Martin From carribeiro at gmail.com Sun Oct 3 23:35:41 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sun Oct 3 23:35:44 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <41606C9C.5060908@v.loewis.de> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <416052DE.3020809@ieee.org> <864d370904100313524c83ef7@mail.gmail.com> <41606C9C.5060908@v.loewis.de> Message-ID: <864d3709041003143579759fcd@mail.gmail.com> On Sun, 03 Oct 2004 23:18:20 +0200, Martin v. L?wis wrote: > Carlos Ribeiro wrote: > > Sorry for introducing my not-very-qualified words on this topic, > > but... I've read the thread up to this point wondering why the bytes() > > type were not being thought of as a clean and definitive solution to > > this problem. It would allow to greatly simplify everything regarding > > struct, binascii and arbitrary low level data manipulation for > > networking and similar stuff. > > No, it wouldn't. If you have a 'long' value, and you want to convert > it to 'bytes', how exactly would you do that? Two's complement, I > suppose - but that would close out people who want unsigned numbers. > Also, do you want big-endian or little-endian? What about a minimum > width, what about overflows? Well, I don't intend to get way too off topic. But in my mind, it makes sense to have a few methods to allow any struct-type hack to work *directly* with the bytes() type. For example: the bytes() type could have a constructor that would take a struct-type string, as in: bytes(data, 'format-string-in-struct-format') or bytes.fromstruct(data, 'format-string-in-struct-format') Alternatively, an append method could take two parameters, one being the data to be appended, and the other the 'transformation rule' -- big endian, little endian, etc: bytes.append(data, 'format-string-in-struct-format') The interface for the data specification could also be a little bit cleaner; I don't see great value at specifying everything with single-character codes. It may look obvious to old C coders, but it doesn't mean that it's the only, or the better, way to do it. Besides concatenation, a few other transformations are useful for bytes -- shifting and rotation in both directions (working at bit level, perhaps?). That's how I thought it should work. (... and, as far as binascii is concerned, I see it more as a way to encode/decode binary data in true string formats than anything else.) > Tim has proposed a signature for binascii that covers all these > scenarios, and I doubt it could get simpler then that and still useful. > > > I also agree with Tim Peters comments regarding struct's C heritage -- > > I never really liked C even when I *had* to use it daily, and the > > struct syntax still reads alien to me. I know this is another > > timeframe entirely, but *if* my vote counted, I would be +1 for a > > future struct implementation tightly integrated with the bytes() type. > > I think you will find that the struct module *already* supports > the bytes type. The bytes type will be just a synonym for the current > string type, except that people will stop associating characters > with the individual bytes; plus the bytes type will be possibly mutable. > As the struct module creates (byte) strings today, it will trivially > support the bytes type. As I've explained above, it's a good first step, but a true bytes() type could have a little bit more functionality than char strings have. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From adurdin at gmail.com Mon Oct 4 00:53:10 2004 From: adurdin at gmail.com (Andrew Durdin) Date: Mon Oct 4 00:53:13 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <1f7befae0410031114724f5f2c@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> Message-ID: <59e9fd3a041003155370ea8761@mail.gmail.com> On Sun, 3 Oct 2004 14:14:35 -0400, Tim Peters wrote: > > I certainly would like to see more people say they'd *use* the g and G > codes in struct even if "one shot" functions in binascii were > available. I have an application where I have to read and write a series of 24-bit integers in a binary file. The g and G codes would make this more convenient, as well as making all the reading and writing code more consistent (as the rest of it uses struct). From skip at pobox.com Sat Oct 2 19:03:14 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Oct 4 04:00:35 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <16734.57170.116719.419227@montanaro.dyndns.org> Guido> This reveals IMO a big mistake in thinking about configuration Guido> files. The most important user of a config file is not the Guido> programmer who has to get data out of it; the most important user Guido> is the user who has to edit the config file. The outrageous Guido> verbosity of XML makes the above example a complete usability Guido> liability. Agreed. What about YaML? It's human readable/editable and uses indentation to denote structure instead of . I'd Google for a reference but I'm off-net at the moment. Skip From pje at telecommunity.com Mon Oct 4 04:44:31 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Oct 4 04:44:23 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <16734.57170.116719.419227@montanaro.dyndns.org> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <5.1.1.6.0.20041003224007.036bf4b0@mail.telecommunity.com> At 12:03 PM 10/2/04 -0500, Skip Montanaro wrote: > Guido> This reveals IMO a big mistake in thinking about configuration > Guido> files. The most important user of a config file is not the > Guido> programmer who has to get data out of it; the most important user > Guido> is the user who has to edit the config file. The outrageous > Guido> verbosity of XML makes the above example a complete usability > Guido> liability. > >Agreed. > >What about YaML? It's human readable/editable and uses indentation to >denote structure instead of . I'd Google for a reference but I'm >off-net at the moment. YaML isn't very friendly to non-techies, IMO. I consider it a very good human *readable* format, but not a human *writable* format. I honestly have an easier time writing XML than YaML, because there are fewer symbols to remember, the format is more regular, and I don't have to think as hard about what the processor is looking for. Of course, I *definitely* prefer .ini files to XML, if they are sufficient for the use case. From carribeiro at gmail.com Mon Oct 4 04:53:44 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 4 04:53:50 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <5.1.1.6.0.20041003224007.036bf4b0@mail.telecommunity.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <415DDA94.4060505@ocf.berkeley.edu> <16734.57170.116719.419227@montanaro.dyndns.org> <5.1.1.6.0.20041003224007.036bf4b0@mail.telecommunity.com> Message-ID: <864d370904100319535d14ce67@mail.gmail.com> On Sun, 03 Oct 2004 22:44:31 -0400, Phillip J. Eby wrote: > Of course, I *definitely* prefer .ini files to XML, if they are sufficient > for the use case. Which means almost always, as far as text configuration files are concerned. (if the configuration is complex enough as to *require* XML, then it's probably better to provide a full fledged user interface for the customization anyway). -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From ncoghlan at email.com Mon Oct 4 05:28:06 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 05:28:16 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> <1096774030.415f718ed9a87@mail.iinet.net.au> Message-ID: <1096860486.4160c346380a5@mail.iinet.net.au> Quoting Guido van Rossum : > #!/usr/bin/env python > import sys, imp > fn = imp.find_module(sys.argv[1])[1] > del sys.argv[0] > sys.argv[0] = fn > execfile(fn) > > seems do the trick (though it could use nicer error handling). And is basically identical to the C version of this code (which uses _PyImp_FindModule and PyRun_SimpleFileExFlags). I like this idea, as it should automatically work with other versions of the interpreter like Jython and IronPython, whereas the command line option would required reimplementation in each interpreter. The hardest part would be to come up with a name for the script (pymod?, pymodule?, pyrun?, runpy?). It will probably be slightly more verbose to invoke than '-m' would be (particularly on an uninstalled development build), but also easier to enhance (e.g supporting modules inside packages). Since I'm on Linux at the moment, and can't recall how the Windows installer handles things like pydoc, I'll wait and see what Raymond and others think about how well this would work on Windows. Unless they think it won't work for other platforms, I'm inclined to reject my own patch and look at a script-based approach. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From martin at v.loewis.de Mon Oct 4 08:16:33 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Oct 4 08:16:36 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <59e9fd3a041003155370ea8761@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <59e9fd3a041003155370ea8761@mail.gmail.com> Message-ID: <4160EAC1.4010904@v.loewis.de> Andrew Durdin wrote: > I have an application where I have to read and write a series of > 24-bit integers in a binary file. The g and G codes would make this > more convenient, as well as making all the reading and writing code > more consistent (as the rest of it uses struct). Out of curiosity: is this a fixed number of of integers in that series? If so, how many? If not, it seems you might be better served if this was an extension to the array module. Regards, Martin From python at rcn.com Mon Oct 4 09:03:52 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Oct 4 09:05:14 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: Message-ID: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> [Guido] > #!/usr/bin/env python > import sys, imp > fn = imp.find_module(sys.argv[1])[1] > del sys.argv[0] > sys.argv[0] = fn > execfile(fn) > > seems do the trick (though it could use nicer error handling). > > Wouldn't it be easier to add *this* to the Tools/scripts directory Though not as clean as -m, this script helps. For my own insufficiently advanced windows box, I added a launcher batch file, pythonm.bat: @python \py24\tools\scripts\runpy.py %1 %2 %3 %4 %5 %6 %7 %8 %9 It runs pretty well: C:\tmp> pythonm timeit -s "a=range(20)" "a.append(1)" "a.pop()" 100000 loops, best of 3: 1.75 usec per loop C:\tmp> pythonm regrtest test_string test_string 1 test OK. The combination of batch file and run.py isn't as good as the -m option, but it works and mostly meets my needs. The batch file is rather dumb, doesn't work with other python command line options, and won't work pipes. > Why does every handy thing that anyone ever thought of > have to be a Python command line option? Could Nick's idea be done without an -m option? If a user types, "python abc.py" and "abc.py" is not found, before aborting, try looking for it on sys.path. Raymond From adurdin at gmail.com Mon Oct 4 13:21:48 2004 From: adurdin at gmail.com (Andrew Durdin) Date: Mon Oct 4 13:21:50 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <4160EAC1.4010904@v.loewis.de> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <59e9fd3a041003155370ea8761@mail.gmail.com> <4160EAC1.4010904@v.loewis.de> Message-ID: <59e9fd3a04100404213f20ffb6@mail.gmail.com> On Mon, 04 Oct 2004 08:16:33 +0200, Martin v. L?wis wrote: > Andrew Durdin wrote: > > I have an application where I have to read and write a series of > > 24-bit integers in a binary file. The g and G codes would make this > > more convenient, as well as making all the reading and writing code > > more consistent (as the rest of it uses struct). > > Out of curiosity: is this a fixed number of of integers in that series? > If so, how many? > > If not, it seems you might be better served if this was an extension > to the array module. There are a fixed number of them -- though it's somewhere in the thousands range... The array module would handle them quite nicely if it supported 3-byte integers; but in general, a more generic struct module will be handier than a more generic array module (I've never dealt with a tuple with thousands of entries before -- is it likely to be a problem? Anyway, wrapping it in a function to read all the ints in blocks and put them in a list is very little trouble). From ncoghlan at email.com Mon Oct 4 13:43:30 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 13:43:36 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> Message-ID: <1096890210.4161376227c2d@mail.iinet.net.au> Quoting Raymond Hettinger : > The combination of batch file and run.py isn't as good as the -m option, > but it works and mostly meets my needs. The batch file is rather dumb, > doesn't work with other python command line options, and won't work > pipes. Hmm, that point about command line options is a good one. I know I'd find it handy to be able to throw in a '-i' when I wanted one. Then again for commands with minor variations, I tend to rely on command line history rather than using a batch file (that is, I'll type out a long invocation sequence once a session, then edit it as needed) > Could Nick's idea be done without an -m option? To be fair to Ilya - his idea, my implementation :) > If a user types, > "python abc.py" and "abc.py" is not found, before aborting, try looking > for it on sys.path. That's do-able, but the command line option seems cleaner if the interpreter is going to provide C-level support for this. I interpreted Guido's last message as saying he's prepared to tolerate the command line option if there's sufficient support for it. To summarise the differences between the two approaches: Pros (-m over runpy.py, in order of significance as I see it): - easy to combine with other Python command line options - OS & environment independent - easier to use with a non-installed interpreter - no work for Python packaging folks - more concise to invoke - no namespace issues with naming a script - C API for those embedding python Cons: - YACLO (Yet Another Command Line Option) - CPython specific (other interpreters could choose to reimplement) With YACLO seeming to be Guido's main reason for being -0 on the idea. It looks like the script option would require input from those with better knowledge than me of the packaging set up for Windows, *nix and Mac as to how it could be made to work 'nicely' in each environment. And the issue of combining it with other command line options would remain. It's all very well to have "runpy.py" as an executable script, but providing extra options to the interpreter would require: 1. Locate runpy.py (e.g. /usr/bin/local/runpy.py, /usr/bin/runpy.py) 2. Invoke python /runpy.py Which means we're back to having to find out where at least part of the Python installation lives on each machine (only once per machine, which is an improvement over the status quo, but still not as tidy as a single command line option). I find that persuasive enough that I think it's worthwhile to finish the '-m' patch, and add a documentation patch. Regards, Nick. -- Nick Coghlan Brisbane, Australia From bhvijaykumar at lucent.com Mon Oct 4 14:03:28 2004 From: bhvijaykumar at lucent.com (B H, Vijaya Kumar (Vijaya)) Date: Mon Oct 4 14:03:35 2004 Subject: [Python-Dev] new member Message-ID: <6733C768256DEC42A72BAFEFA9CF06D20D04C980@ii0015exch002u.iprc.lucent.com> Hi all My name is Vijay I am new member of this group I have been working on python from last 1 year , i love this language for its simpicity , i also worked on jython which i used in few of my automation tasks Regards Vijaya Kumar B.H. From aahz at pythoncraft.com Mon Oct 4 16:01:10 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Oct 4 16:01:47 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> Message-ID: <20041004140109.GA22637@panix.com> On Mon, Oct 04, 2004, Raymond Hettinger wrote: > > Could Nick's idea be done without an -m option? If a user types, > "python abc.py" and "abc.py" is not found, before aborting, try looking > for it on sys.path. -1 -- too much magic "Explicit is better than implicit" -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines." --Ralph Waldo Emerson From carribeiro at gmail.com Mon Oct 4 16:29:41 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 4 16:29:44 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <20041004140109.GA22637@panix.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <20041004140109.GA22637@panix.com> Message-ID: <864d3709041004072960b5a213@mail.gmail.com> On Mon, 4 Oct 2004 10:01:10 -0400, Aahz wrote: > On Mon, Oct 04, 2004, Raymond Hettinger wrote: > > > > Could Nick's idea be done without an -m option? If a user types, > > "python abc.py" and "abc.py" is not found, before aborting, try looking > > for it on sys.path. > > -1 -- too much magic It's exactly what virtually every shell does. I sincerely don't think this would come up as surprise. On the other hand, I've often became frustrated to have to type long (I mean looooong) paths by hand, without autocompletion to help (welcome to the Windows DOS box!), just to discover that the script wasn't exactly at *that* point, but still it could be found somewhere down the path. (Just to mention, and for the sake of completeness: instead of sys.path, it could search on the shell path... but that's not a good solution either, it seems even more arbitrary) OTOH, I'm also a little bit concerned about YACLO, because there's always more cruft to add. And finally, why '-m'? Just because this letter was available ;-) ? I think that for some stuff, long names are better, even if I have to type them sometimes, and specially if I can have the default stored somewhere. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From gvanrossum at gmail.com Mon Oct 4 17:05:01 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 4 17:05:07 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096890210.4161376227c2d@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: > Pros (-m over runpy.py, in order of significance as I see it): > - easy to combine with other Python command line options The script approach can do this too > - OS & environment independent So is the script -- probably more so, since the script can use Python's OS independence layer. > - easier to use with a non-installed interpreter > - no work for Python packaging folks No contest. > - more concise to invoke Depends on the length of the name of the script. :-) > - no namespace issues with naming a script Actually, one-letter options are a much scarcer resource than script names. > - C API for those embedding python And who needs that? > Cons: > - YACLO (Yet Another Command Line Option) > - CPython specific (other interpreters could choose to reimplement) Additional Pros for using a script: - less code to maintain - can work with older Python versions - shows users how to do a similar thing themselves with additional features (e.g. a common addition I expect will be to hardcode a personal sys.path addition) PS a big honking -1000 on Carlos Ribeiro's suggestion of doing this automatically if the filename path isn't found. Too many chances for accidentally invoking the wrong thing -- including mysyerious silences when the named module happens to exist but doesn't do anything when run as a script (i.e. the majority of modules). (As an aside, I wonder why every single of Carlos' interactions starts of with an idea that is completely misguided and then ends with him vehemently defending it against all opposition. Sounds like a recipe for flame wars to me.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Mon Oct 4 17:30:35 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Mon Oct 4 17:29:24 2004 Subject: [Python-Dev] Re: '-m' option (was RE: ConfigParser patches) In-Reply-To: <864d3709041004072960b5a213@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <20041004140109.GA22637@panix.com> <864d3709041004072960b5a213@mail.gmail.com> Message-ID: Carlos Ribeiro wrote: > ... I've often became frustrated to have to type long (I mean looooong) > paths by hand, without autocompletion to help (Windows DOS box!)... Carlos, if you are using CMD.EXE (that is, on Win2K or XP), try running CMD /F:ON (I have a desktop link to cmd.exe that does this). This gives Ctrl-F to complete filenames and Ctrl-D for directories. You can change the keys used by tweaking the registry, but I've never felt the need. You'll get a description from "help cmd". -- -- Scott David Daniels Scott.Daniels@Acm.Org From carribeiro at gmail.com Mon Oct 4 18:30:31 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 4 18:30:36 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: <864d370904100409307d342465@mail.gmail.com> On Mon, 4 Oct 2004 08:05:01 -0700, Guido van Rossum wrote: > PS a big honking -1000 on Carlos Ribeiro's suggestion of doing this > automatically if the filename path isn't found. Too many chances for > accidentally invoking the wrong thing -- including mysyerious silences > when the named module happens to exist but doesn't do anything when > run as a script (i.e. the majority of modules). (As an aside, I wonder > why every single of Carlos' interactions starts of with an idea that > is completely misguided and then ends with him vehemently defending it > against all opposition. Sounds like a recipe for flame wars to me.) Sorry. Really. I may have expressed myself VERY badly on this topic. Please, let me defend myself as I don't like to leave this impression as I'm just starting here: -- The original idea was not mine -- I was just commenting because it seemed good to me, as I'm limited to a DOS box most of the time, and searching modules by hand tend to become an issue in this enviroment; -- I really meant to say this (regarding YACLO): I *don't* think that adding command line options for everything is a good idea for any software. Also, I think that single line options are confusing, and longer option names are better/more readable. It seems that the message was read with the opposite meaning. -- As far as the rest of the comments are concerned, I apologize if I tend to sound so 'flamish'. Actually, I have a tendency to think aloud sometimes, and to explore different ideas. Sometimes it makes for amusing comments, sometimes it's weird, and sometimes it just sounds dumb. Sometimes I get it right. But overall it's not the right thing to do, specially as I'm yet to make some useful contribution. In the end, I think I should learn to keep my mouth/fingers shut. Lesson taken, and really, really, sorry. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From ilya at bluefir.net Mon Oct 4 19:33:26 2004 From: ilya at bluefir.net (Ilya Sandler) Date: Mon Oct 4 19:31:55 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096890210.4161376227c2d@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: > To summarise the differences between the two approaches: > > Pros (-m over runpy.py, in order of significance as I see it): > - easy to combine with other Python command line options > - OS & environment independent > - easier to use with a non-installed interpreter > - no work for Python packaging folks > - more concise to invoke > - no namespace issues with naming a script > - C API for those embedding python There is one more small pro: easier to version: e.g if there are multiple installed versions of python on your machine, you could could invoke a module from a particular python version: python2.4 -m pydoc with a wrapper script you will need a wrapper per version...(Or some way to find/invoke the right version of python from the wrapper) Ilya > > Cons: > - YACLO (Yet Another Command Line Option) > - CPython specific (other interpreters could choose to reimplement) > > With YACLO seeming to be Guido's main reason for being -0 on the idea. > > It looks like the script option would require input from those with better > knowledge than me of the packaging set up for Windows, *nix and Mac as to how it > could be made to work 'nicely' in each environment. > > And the issue of combining it with other command line options would remain. It's > all very well to have "runpy.py" as an executable script, but providing extra > options to the interpreter would require: > > 1. Locate runpy.py (e.g. /usr/bin/local/runpy.py, /usr/bin/runpy.py) > 2. Invoke python /runpy.py > > Which means we're back to having to find out where at least part of the Python > installation lives on each machine (only once per machine, which is an > improvement over the status quo, but still not as tidy as a single command line > option). > > I find that persuasive enough that I think it's worthwhile to finish the '-m' > patch, and add a documentation patch. > > Regards, > Nick. > > -- > Nick Coghlan > Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ilya%40bluefir.net > From p.f.moore at gmail.com Mon Oct 4 20:15:52 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Mon Oct 4 20:16:01 2004 Subject: [Python-Dev] Utility scripts (Was: '-m' option) In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: <79990c6b04100411157d14cf43@mail.gmail.com> The problem here is really one of packaging. How should utility scripts be distributed, either with Python, or with extensions? There are many different approaches: 1. a library module with useful behaviour in the __main__ block. 2. a script in sys.prefix + '/scripts' with no extension and a #! line 3. a script as above, but with a .py extension 4. a script as (2) plus a .bat wrapper for Windows 5. a script as (3) plus a .bat wrapper for Windows However, *none* of these give the best behaviour, uniformly across all Windows platforms (the Unix story is far better, as there is a consistent and useful behaviour across all shells). While case (1) is a bit of an anomaly, Guido's script would probably cover it, *if* a really acceptable solution for scripts could be found. Accepting that Unix is generally not an issue, let's look at Windows. On cmd.exe, at least on Windows 2000 and up, the following makes .py files "executable" by python: assoc .py=Python.File ftype Python.File=C:\Python23\python.exe "%1" %* set PATHEXT=%PATHEXT%;.py Can anyone test this on COMMAND.COM on Win9x? It works using the COMMAND.COM supplied with Windows 2000, whatever that proves :-) Of these, the first two are set by the standard Windows install. The final one could be. With that change, simply putting a .py script into C:\Python23\Scripts would be enough (assuming C:\Python23\Scripts was on the user's PATH, which it isn't by default, but the user can add it). Proposal: - Change the Windows Python binary to add .py (and .pyw, maybe) to PATHEXT - Add a variation of Guido's script to the standard install, as something like runmod.py - Document that the way for extensions to include scripts is as .py files in the sys.prefix+'/scripts' directory, with a #! line for Unix - For Unix users who don't like extensions, suggest (again in the docs) that a hardlink can be added without the extension. Maybe include a suitable setup.py snippet to show how. - Possibly add sys.path+'/scripts' to the PATH - I'm not sure about this, as on Windows the installer seems to try hard to *not* modify PATH, so maybe this is a policy issue. I'm punting on the issue of python.exe flags. If someone wants to run a supplied script as python -E -S -O script.py, I don't have a solution (other than doing so "longhand"). But I don't have a use case, either, so I'll leave that one for others... Paul From tim.peters at gmail.com Mon Oct 4 20:49:43 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 4 20:49:49 2004 Subject: [Python-Dev] Utility scripts (Was: '-m' option) In-Reply-To: <79990c6b04100411157d14cf43@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <79990c6b04100411157d14cf43@mail.gmail.com> Message-ID: <1f7befae04100411492fc43961@mail.gmail.com> [Paul Moore] > ... > Accepting that Unix is generally not an issue, let's look at Windows. > On cmd.exe, at least on Windows 2000 and up, the following makes .py > files "executable" by python: > > assoc .py=Python.File > ftype Python.File=C:\Python23\python.exe "%1" %* > set PATHEXT=%PATHEXT%;.py > > Can anyone test this on COMMAND.COM on Win9x? PATHEXT has no special meaning on 9x; you can set PATHEXT on 9x, but it has no effect. ... > - Change the Windows Python binary to add .py (and .pyw, maybe) to > PATHEXT I don't like that. The Python Windows installer doesn't add, or modify, any environment variables now. It's anti-social to muck with them. Anyone using "a DOS box" should know how to do that themself -- if that's what they want. I'll note that I spend much of my life in a WinXP DOS box running Python programs, but I haven't set PATHEXT. Since cmd.exe's path completion fills in the trailing .py on a .py file all by itself, setting PATHEXT wouldn't save me any typing. From gvanrossum at gmail.com Mon Oct 4 21:24:43 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 4 21:24:46 2004 Subject: [Python-Dev] Utility scripts (Was: '-m' option) In-Reply-To: <1f7befae04100411492fc43961@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: > > - Change the Windows Python binary to add .py (and .pyw, maybe) to > > PATHEXT > > I don't like that. The Python Windows installer doesn't add, or > modify, any environment variables now. It's anti-social to muck with > them. Anyone using "a DOS box" should know how to do that themself -- > if that's what they want. I'll note that I spend much of my life in a > WinXP DOS box running Python programs, but I haven't set PATHEXT. > Since cmd.exe's path completion fills in the trailing .py on a .py > file all by itself, setting PATHEXT wouldn't save me any typing. Right. Typing the .py is usually the right thing to do. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From p.f.moore at gmail.com Mon Oct 4 22:03:24 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Mon Oct 4 22:11:07 2004 Subject: [Python-Dev] Re: Utility scripts References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: Tim Peters writes: > [Paul Moore] >> ... >> Can anyone test this on COMMAND.COM on Win9x? > > PATHEXT has no special meaning on 9x; you can set PATHEXT on 9x, but > it has no effect. Right. That means that this isn't a general Windows solution. On the other hand, further experimentation seems to imply that all that PATHEXT actually does is allow the ".py" to be omitted - I hadn't realised that. So as things stand, putting a ".py" script, with a #! line, into sys.prefix+'/scripts', and setting the executable bit on Unix, is already a general way of installing a command - with two provisos: 1. The user must add the directory to his PATH 2. The command must be typed with the .py extension specified So why do existing extensions not do this? Things like wxPython and Twisted seem to jump through a lot of hoops for little benefit beyond allowing users to omit the .py extension... Maybe it would be worth adding a paragraph to section 3.4 of "Distributing Python Modules" (the "Installing Scripts" section) saying something like: Scripts should be given names with a .py extension. This allows them to be executed on both Unix and Windows systems (Windows uses the extension to locate the correct interpreter, in this case Python, in much the same way that Unix uses the #! line). Of course, this will enrage Unix users who dislike file extensions :-) Maybe having a user-specifiable option, something like --strip-ext, to strip the .py extension from scripts when installing, would be useful. (I recall something like this being discussed on the distutils-sig a while back. Maybe this discussion should move there.) >> - Change the Windows Python binary to add .py (and .pyw, maybe) to >> PATHEXT > > I don't like that. The Python Windows installer doesn't add, or > modify, any environment variables now. It's anti-social to muck with > them. OK, that's the policy I was thinking of. Paul. -- Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind. -- Dr. Seuss From fdrake at acm.org Mon Oct 4 22:45:48 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Oct 4 22:45:57 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: <200410041645.48976.fdrake@acm.org> On Monday 04 October 2004 04:03 pm, Paul Moore wrote: > Of course, this will enrage Unix users who dislike file extensions :-) Yes, indeed it will, as well it should. > Maybe having a user-specifiable option, something like --strip-ext, to > strip the .py extension from scripts when installing, would be useful. > (I recall something like this being discussed on the distutils-sig a > while back. Maybe this discussion should move there.) We discussed something very similar to this, but it wasn't so much a per-user distinction. As a package author, the name of the executables that get installed as part of my package are part of the user interface; I should be abe to control them on each platform I support. There are enough twists to this that we didn't come up with a general solution at the time, and I've not had time to revisit the topic since then, though I'd like to get something more workable over the next few months. -Fred -- Fred L. Drake, Jr. From ncoghlan at email.com Mon Oct 4 23:07:22 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 23:07:28 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: <1096924042.4161bb8a0a35a@mail.iinet.net.au> Aside for Carlos: 'm' is short for 'module', and beefing up Python's option parsing to handle things like '--module' is a whole 'nother story. 'python -h' gives a decent description of all the options, though. Guido van Rossum wrote: >>Pros (-m over runpy.py, in order of significance as I see it): >> - easy to combine with other Python command line options > > The script approach can do this too Only by spelling out the invocation if we want something non-standard. Using the laptop I'm typing on as an example. . . Python 2.3 is the installed version, so "#!/usr/bin/env python" in a script will invoke Py 2.3 without any extra options If I (or a Python installer) were to create "/usr/local/bin/runpy.py" with that shebang line (i.e. the same as the one you posted), we have the following for profiling a sample script in my current directory with different versions of the interpreter: Installed: runpy.py profile demo.py Prompt After: python -i /usr/local/bin/runpy.py profile demo.py Alt install: python2.4 /usr/local/bin/runpy.py profile demo.py Build dir: ./python /usr/local/bin/runpy.py profile demo.py If we wanted to use the version of the script that came with the relevant version of python, those last two would become: Alt install*: python2.4 /usr/local/lib/python2.4/runpy.py profile demo.py Build dir: ./python Tools/scripts/runpy.py profile demo.py (* This is based on what happens to pydoc under 'make altinstall'. The shebang line is version agnostic, so it tries to use the Py2.3 interpreter with the Py2.4 library modules and it all falls apart. So to run a library module of the altinstall, this is how I have to do it. And this is assuming the script would get installed at all, which it may not, as it isn't actually a library module, unlike pydoc) Using -m, those become: Installed: python -m profile demo.py Prompt After: python -i -m profile demo.py Alt install: python24 -m profile demo.py Build dir: ./python -m profile demo.py >> - OS & environment independent > > So is the script -- probably more so, since the script can use > Python's OS independence layer. Paul's message goes into detail on what I meant here. The script itself is highly portable, the mechanism for invoking it really isn't. The C code in the patch is platform independent - _PyImport_FindModule (a trivial wrapper around the existing find_module in import.c) and PyRun_SimpleFileExFlags do all the heavy lifting. In fact, sans error-checking, the guts of PyRun_SimpleModuleFlags looks remarkably similar to the Python code in your script. (I initially thought the Python version might handle zip imports, while the C version didn't. However, a quick experiment shows that *neither* of them can handle zip imports. And the source code confirms it - imp.find_module and imp.load_module don't allow zip imports, and PyRun_Module in the patch doesn't allow them either. Amusingly, the current limitations of the imp module make it easier to support zip imports with the *C* version. Allowing imp.find_module to return a 4th value for the module loader would require adding an optional boolean argument to avoid breaking existing code, whereas the patch's new private C API for _PyImport_FindModule already exposes the loader argument) >> - more concise to invoke > Depends on the length of the name of the script. :-) See the examples above for what I meant with this one. For the vanilla case you're right, but as soon as we do anything slightly different, the story changes. >> - no namespace issues with naming a script > Actually, one-letter options are a much scarcer resource than script names. Well, with '-m' in place, we'd be using 17 out of the available 62 (upper & lower alpha, plus digits). The difference is that we're only competing with ourselves and the other Python interpreter authors for characters to use, whereas we're competing with all and sundry for unique executable names. (Windows isn't immune, either, given enough applications with directories on PATH. Although retaining the '.py' helps a lot there) (I have checked that Jython at least doesn't use '-m' for anything. I don't know about other interpreters) >> - C API for those embedding python > And who needs that? There's a reason this one was last on my list :) > Additional Pros for using a script: > - less code to maintain Once we factor in the additional packaging requirements to make the script as easy to use on all target platforms as -m would be, I think this one is at least arguable (script + packaging vs option-parsing and C function). > - can work with older Python versions > - shows users how to do a similar thing themselves with additional > features Certainly, dropping a version of this script into Tools/scripts in CVS couldn't hurt, regardless of whether or not '-m' gets adopted. The same would go for an entry in the Python cookbook. > (e.g. a common addition I expect will be to hardcode a > personal sys.path addition) Except that this feature isn't so much for your *own* scripts, as it is for installed modules that are also usable as scripts (like profile and pdb). In the former case, you *know* where those scripts are (for me ~/script_name usually does the trick on *nix). In the latter case, though, it'd be nice to be able to use these things easily 'out of the box'. For those who want to tweak the search behaviour, all the usual environment variables apply (PYTHONPATH in particular). Heck, there's nothing to stop someone from doing something like the following if they really want to: python -m my_run_module_script some_other_module The command line interface is one of the major holdouts in Python where we really need to care where the source file for a module lives. It'd be nice to change that. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From ncoghlan at email.com Mon Oct 4 23:21:41 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 23:25:32 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096924042.4161bb8a0a35a@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <1096924042.4161bb8a0a35a@mail.iinet.net.au> Message-ID: <1096924901.4161bee5b2cec@mail.iinet.net.au> Quoting Nick Coghlan : > Using -m, those become: > > Installed: python -m profile demo.py OK, of course Python 2.3 doesn't have a '-m' switch. Consider the example to start with the first version to ship with '-m' installed. For actual Python 2.3, this would obviously have to use the absolute path, or some flavour of Guido's script (probably ~/runpy, since that is handiest). Cheers, Nick. -- Nick Coghlan Brisbane, Australia From tim.peters at gmail.com Mon Oct 4 23:29:28 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 4 23:29:36 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: <1f7befae0410041429dfb729b@mail.gmail.com> [Paul Moore] ... > So why do existing extensions not do this? Things like wxPython > and Twisted seem to jump through a lot of hoops for little benefit > beyond allowing users to omit the .py extension... ... > Of course, this will enrage Unix users who dislike file extensions :-) Bingo -- that's why. There's no other reason. ... > (I recall something like this being discussed on the distutils-sig a > while back. Yes. > Maybe this discussion should move there.) You're batting 1000 . From dubnerm-news at mail.ru Mon Oct 4 23:44:58 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Mon Oct 4 23:44:33 2004 Subject: [Python-Dev] Guidelines for Logging Usage Message-ID: <4161C45A.4050205@mail.ru> Hello, I beg your pardon in advance for my English, which isn't my native language. I'm posting PEProposal for discussion. Last part is reasoning for posting it in this hot time. I'm ready to implement it all by myself... :-) -- Best regards, Michael Dubner (dubnerm@mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English -------------- next part -------------- PEP: XXX Title: Guidelines for Logging Usage Version: $Revision: $ Last-Modified: $Date: $ Author: Michael P. Dubner Status: Draft Type: Standards Track Content-Type: text/plain Created: 02-Oct-2004 Python-Version: 2.5 Post-History: Abstract This PEP defines guidelines for using logging system (PEP 282 [1]) in standard library. Implementing this PEP will simplify development of daemon applications. As a downside this PEP requires to slightly modify (however in backportable way) large number of standard modules. After implementing this PEP one can use following filtering scheme:: logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) Rationale There are couple of situations when output to stdout or stderr is very incomfortable. - Daemon application where framework doesn't allows to redirect standard output to some file, but assumes use of some other way of logging. Examples are syslog under *nix'es and EventLog under WinNT+. - GUI application which want to output every new log entry in separate popup window (i.e. fading OSD). Also sometimes application want to filter output enties based on it's source or severity. This requirement can't be implemented using simple redirection. At last sometimes output need to be marked with time of event, which can be acqured with logging system with ease. Proposal Every module usable for daemon and GUI applications should be rewritten to use logging system instead of 'print' or 'sys.stdout.write'. There should be code like this included in the beginning of every modified module:: import logging _log = logging.getLogger('stdlib.') Prefix of 'stdlib.' must be used by all modules included in standard library distributed along with Python, and only by such modules (unverifiable). Using of "_log" is intentional as we don't want to auto-export it. For modules that using log only in one class logger can be created inside class definition as following:: class XXX: __log = logging.getLogger('stdlib.') Then this class can create access methods to log to this private logger. So print's and "sys.std{out|err}.write" should be replaced with "_log.{debug|info}" and "traceback.print_exception" with "_log.exception" or sometimes "_log.debug('...',exc_info=1)". Module List Here is (possibly incomplete) list of modules to be reworked: - asyncore (dispatcher.log, dispatcher.log_info) - BaseHTTPServer (BaseHTTPRequestHandler.log_request, BaseHTTPRequestHandler.log_error, BaseHTTPRequestHandler.log_message) - cgi (possibly - is cgi.log used by somebody?) - ftplib (if FTP.debugging) - gopherlib (get_directory) - httplib (HTTPResponse, HTTPConnection) - ihooks (_Verbose) - imaplib (IMAP4._mesg) - mhlib (MH.error) - nntplib (NNTP) - pipes (Template.makepipeline) - pkgutil (extend_path) - platform (_syscmd_ver) - poplib (if POP3._debugging) - profile (if Profile.verbose) - robotparser (_debug) - smtplib (if SGMLParser.verbose) - shlex (if shlex.debug) - smtpd (SMTPChannel/PureProxy where print >> DEBUGSTREAM) - smtplib (if SMTP.debuglevel) - SocketServer (BaseServer.handle_error) - telnetlib (if Telnet.debuglevel) - threading? (_Verbose._note, Thread.__bootstrap) - timeit (Timer.print_exc) - trace - uu (decode) Additionaly there are couple of modules with commented debug output or modules where debug output should be added. For example: - urllib At last possibly some modules should be extended to provide more debug information. Doubtful Modules Here should be placed modules that community will propose for addition to module list and modules that community say should be removed from module list. - tabnanny (check) Guidelines for Logging Usage Also we can provide some recommendation to authors of library modules so they all follow same format of naming loggers. I propose that non-standard library modules should use logger named after their full names, so module "spam" in sub-package "junk" of package "dummy" will be named "dummy.junk.spam" and, of cause, "__init__" module of same package will have logger "dummy.junk". Implementation Schedule Proposal As one can see whole bunch of changes required to fullfil this proposal is rather large. I propose to delay these changes until after 2.4 release, and change only modules critical for server applications: - BaseHTTPServer - ftplib - httplib - poplib - smtpd - smtplib - SocketServer This can be done rather fast and reliable. References [1] PEP 282, A Logging System, Vinay Sajip, Trent Mick http://www.python.org/peps/pep-0282.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From ncoghlan at email.com Mon Oct 4 23:51:08 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 23:51:14 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <1096924042.4161bb8a0a35a@mail.iinet.net.au> Message-ID: <1096926668.4161c5cc9c49e@mail.iinet.net.au> Quoting Paramjit Oberoi : > With a slightly longer script, and a little work in the installer, > these could be written as: > > Installed: runpy.py profile demo.py > Prompt After: runpy.py -i profile demo.py > Alt install: runpy2.4.py profile demo.py > Build dir: ./runpy.py profile demo.py > - or - ./scripts/runpy.py profile demo.py Indeed, it may be possible to get those to *run*, but the problem is that the semantics of the following two commands are likely to differ: python runpy.py module python runpy.py module References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <1096924042.4161bb8a0a35a@mail.iinet.net.au> <1096926668.4161c5cc9c49e@mail.iinet.net.au> Message-ID: I'm going to drop out of this discussion; I'm still thinking this is just code bloat, but I won't stop you from adding it anyway. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From exarkun at divmod.com Tue Oct 5 00:39:24 2004 From: exarkun at divmod.com (exarkun@divmod.com) Date: Tue Oct 5 00:39:29 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: Message-ID: <20041004223924.29723.995823090.divmod.quotient.9779@ohm> On Mon, 04 Oct 2004 21:03:24 +0100, Paul Moore wrote: > [snip] > So why do existing extensions not do this? Things like wxPython > and Twisted seem to jump through a lot of hoops for little benefit > beyond allowing users to omit the .py extension... Twisted jumps through some hoops to let developers run uninstalled versions. The lack of .py extensions is incidental and mostly unimportant. It also does things differently on UNIX than on Windows (it sets things up on Windows so uses can just type "mktap", "twistd", etc (as opposed to path\to\mktap, not as opposed to mktap.py)). Jp From garthk at gmail.com Tue Oct 5 03:48:01 2004 From: garthk at gmail.com (Garth T Kidd) Date: Tue Oct 5 03:48:04 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: <8aeed5d0041004184865e697ff@mail.gmail.com> >>> - Change the Windows Python binary to add .py (and .pyw, maybe) to >>> PATHEXT >> >> I don't like that. The Python Windows installer doesn't add, or >> modify, any environment variables now. It's anti-social to muck with >> them. Half the software on my workstation has changed environment variables, social or not. To me, it's less social to leave me to manually change PATHEXT. I'd much rather have Python add its extensions to PATHEXT -- and also have distutils add .py to the extension of Python scripts dropped into the Scripts directory. I used to be a control freak, but now I'm a control freak with kids, and the utter lack of spare time really changes one's perspective on such things. :) Regards, Garth. From paul at pfdubois.com Tue Oct 5 04:02:19 2004 From: paul at pfdubois.com (Paul F. Dubois) Date: Tue Oct 5 04:02:24 2004 Subject: [Python-Dev] grant applications Message-ID: <416200AB.6000006@pfdubois.com> Dear devs, You'll be pleased to learn that the PSF grants process brought in sixty-four proposals. I think that is a stunning success, and the wide variety of these proposals is a testimony to the breadth and strength of the community. We should seek some branding sponsorship in the future so we can accept more of these. I'm thinking along the likes of the "Kraft Cottage Cheese Grant". (:-> Picture of smiling Guido enjoying the product at his terminal. From tim.peters at gmail.com Tue Oct 5 04:08:12 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 5 04:08:21 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: <8aeed5d0041004184865e697ff@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> <8aeed5d0041004184865e697ff@mail.gmail.com> Message-ID: <1f7befae041004190839b5921f@mail.gmail.com> [Garth T Kidd] > Half the software on my workstation has changed environment variables, > social or not. To me, it's less social to leave me to manually change > PATHEXT. I'd much rather have Python add its extensions to PATHEXT -- > and also have distutils add .py to the extension of Python scripts > dropped into the Scripts directory. So contribute installer code to do so -- and don't forget to make the uninstaller smart enough to remove them again, without damaging what other installers may have added in the meantime. Ah, you don't want it enough to do anything to get it . That's OK. What good would it do you if it were there? As was made plain in the rest of this thread, PATHEXT does nothing except on cmd.exe systems. On cmd.exe systems, it only allows to leave the trailing ".py" off, and since cmd.exe's path completion fills in the ".py" for you, you then have to manually delete the ".py" in order to get a benefit from PATHEXT. Unless you add directories containing Python scripts to your PATH envar too -- in which case you can't credibly claim that adding .py to PATHEXT is beyond your abilities or time. Adding ".py" to scripts is a different issue, but that belongs on the distutils list. From dubnerm-news at mail.ru Tue Oct 5 08:18:18 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Tue Oct 5 08:17:24 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <8aeed5d0041004191553da8d0@mail.gmail.com> References: <4161C45A.4050205@mail.ru> <8aeed5d0041004191553da8d0@mail.gmail.com> Message-ID: <41623CAA.9070207@mail.ru> Garth T Kidd wrote: >Strikes me that not much work could let someone keep all of their >existing print statements more or less intact and use the logging >module properly, too: > > import logging > > class LoggingHandle: > def __init__(self, wrappedmethod): > self.wrappedmethod = wrappedmethod > def write(self, msg): > msg = msg.rstrip() > if msg: > self.wrappedmethod(msg.rstrip()) > > logging.basicConfig() > log = logging.getLogger('whatever') > stderr = LoggingHandle(log.error) > stdout = LoggingHandle(log.info) > > print >> stdout, "This is informational." # logged as INFO > print >> stderr, "This is an error." # guess. > >Add some basic infrastructure to logging, and you're done. The >Extremely Brave could even alter sys.stderr and sys.stdout. > > Main purpose of logging module is not redirection, but filtering capability. -- Best regards, Michael Dubner (dubnerm@mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From vinay_sajip at red-dove.com Tue Oct 5 15:03:07 2004 From: vinay_sajip at red-dove.com (Vinay Sajip at Red Dove) Date: Tue Oct 5 15:03:26 2004 Subject: [Python-Dev] Guidelines for Logging Usage Message-ID: <000501c4aadb$ae9f1f00$0400a8c0@DELTA> [Michael Dubner] > Main purpose of logging module is not redirection, but filtering capability. I don't agree. In general, both the redirection capability (targeting events at a particular audience) and the filtering capability (verbosity control) are important. Regards, Vinay Sajip From dubnerm-news at mail.ru Tue Oct 5 23:48:37 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Tue Oct 5 23:46:46 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <000501c4aadb$ae9f1f00$0400a8c0@DELTA> References: <000501c4aadb$ae9f1f00$0400a8c0@DELTA> Message-ID: <416316B5.3080905@mail.ru> Vinay Sajip at Red Dove wrote: >[Michael Dubner] > > >>Main purpose of logging module is not redirection, but filtering >> >> >capability. > >I don't agree. In general, both the redirection capability (targeting events >at a particular audience) and the filtering capability (verbosity control) >are important. > > I mean "purpose of logging module" only in content of PEP. -- Best regards, Michael Dubner (dubnerm@mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From garthk at gmail.com Wed Oct 6 00:28:13 2004 From: garthk at gmail.com (Garth T Kidd) Date: Wed Oct 6 00:28:24 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <41623CAA.9070207@mail.ru> References: <4161C45A.4050205@mail.ru> <8aeed5d0041004191553da8d0@mail.gmail.com> <41623CAA.9070207@mail.ru> Message-ID: <8aeed5d004100515283cf64a93@mail.gmail.com> It since occurred to me that people might want the standard library to be the canonical example of Good Python rather than some messy kludge. :) So, replacing print with appropriate direct calls to logging or warnings would be more appropriate than hooking print into those frameworks by futzing around with handles. Speaking of which: any thoughts on having warnings use logging rather than print? I'm showing 224 files in Lib\ without 'test' in the name and with print statements, though some of those are sensible (like zipfile.ZipFile.printdir) and some look like internal tests. Only cookielib, _LWPCookieJar.py and _MozillaCookieJar seem to import the logging module. So, you've got some real work on your hands updating everything to use logging. Cookie, cookielib, distutils, email, filecmp, idlelib, pickle, plat-mac, posixfile, pydoc, random, reconvert, regsub, shelve, statcache, tempfile, tzparse, whrandom and xmllib all import warnings. I'm not sure if it's appropriate to timestamp the root logger going to console by default; WARNING:module:message is messy enough without a timestamp. Regards, Garth. On Tue, 05 Oct 2004 10:18:18 +0400, Michael P. Dubner wrote: > > > Garth T Kidd wrote: > > >Strikes me that not much work could let someone keep all of their > >existing print statements more or less intact and use the logging > >module properly, too: > > > > import logging > > > > class LoggingHandle: > > def __init__(self, wrappedmethod): > > self.wrappedmethod = wrappedmethod > > def write(self, msg): > > msg = msg.rstrip() > > if msg: > > self.wrappedmethod(msg.rstrip()) > > > > logging.basicConfig() > > log = logging.getLogger('whatever') > > stderr = LoggingHandle(log.error) > > stdout = LoggingHandle(log.info) > > > > print >> stdout, "This is informational." # logged as INFO > > print >> stderr, "This is an error." # guess. > > > >Add some basic infrastructure to logging, and you're done. The > >Extremely Brave could even alter sys.stderr and sys.stdout. > > Main purpose of logging module is not redirection, but filtering capability. From jcarlson at uci.edu Sun Oct 3 03:08:57 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 6 03:48:56 2004 Subject: [Python-Dev] proposed struct module format code addition Message-ID: <415F5129.3040006@uci.edu> Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah From jcarlson at uci.edu Wed Oct 6 04:09:47 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 6 04:17:27 2004 Subject: [Python-Dev] proposed struct module format code addition Message-ID: <20041005190838.ECF7.JCARLSON@uci.edu> If the proposed struct module post seemed to have been a dupe, it is because it sat in a moderator queue for a few days. Sorry about that. - Josiah From kbk at shore.net Wed Oct 6 06:02:47 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Oct 6 06:02:53 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200410060402.i9642lO6022228@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 241 open ( +6) / 2640 closed ( +3) / 2881 total ( +9) Bugs : 766 open ( -2) / 4492 closed (+12) / 5258 total (+10) RFE : 154 open ( +2) / 131 closed ( +0) / 285 total ( +2) New / Reopened Patches ______________________ ftplib PASV error bug (2004-09-30) http://python.org/sf/1037516 opened by Tim Nelson urllib2 HTTP digest authentication fix (2004-09-30) http://python.org/sf/1037974 opened by Mathieu Fenniak Provide tuple of "special" exceptions (2004-10-01) http://python.org/sf/1038256 opened by Nick Coghlan __main__ for whichdb.py (2004-10-01) http://python.org/sf/1038388 opened by Oleg Broytmann Fix struct.pack on 64-bit archs (broken on 2.*) (2004-10-02) http://python.org/sf/1038854 opened by Stefan Grundmann pydoc method documentation lookup enhancement (2004-10-02) http://python.org/sf/1038909 opened by Alexander Schmolck OS X: Can't use #!/usr/bin/pythonw (2004-10-02) http://python.org/sf/1038911 opened by Nathan Gray repair typos (2004-10-02) CLOSED http://python.org/sf/1038917 opened by George Yoshida SimpleXMLRPCServer optional allow_none / encoding arguments (2004-10-02) http://python.org/sf/1039083 opened by Guillermo M. Narvaja Patches Closed ______________ repair typos (2004-10-02) http://python.org/sf/1038917 closed by rhettinger doctest: allow custom matchers for testing if got=want (2004-04-10) http://python.org/sf/932935 closed by edloper doctest: Add Tester params to DocTestSuite (2004-04-10) http://python.org/sf/932932 closed by edloper New / Reopened Bugs ___________________ len(str(x)) in decimal.py is not Python 2.3 compatible (2004-09-30) CLOSED http://python.org/sf/1037373 opened by Nick Coghlan textReader: reading after close is a seg fault (2004-09-30) http://python.org/sf/1038013 opened by Jyrki Alakuijala silent installation on windows: no drive in registry values (2004-10-01) CLOSED http://python.org/sf/1038410 opened by Matteo Gallivanoni Python 2.3+ socket._fileobject handles EAGAIN with data loss (2004-10-01) http://python.org/sf/1038591 opened by Jon Nelson bad link in Tkinter docs (2004-10-01) http://python.org/sf/1038693 opened by Brett Cannon Typo "signiture" in HTML generated by DocXMLRPCServer (2004-10-02) CLOSED http://python.org/sf/1038935 opened by Malte Helmert apply.__doc__ says "Deprecated since release 2.3" (2004-10-02) CLOSED http://python.org/sf/1039092 opened by Jp Calderone time zone tests fail on Windows (2004-10-03) CLOSED http://python.org/sf/1039270 opened by George Yoshida os.times() is bogus (2004-10-04) http://python.org/sf/1040026 opened by Guido van Rossum Missing documentation on How To Link With libpython (2004-10-05) http://python.org/sf/1040439 opened by Sir Raorn Bugs Closed ___________ Odd behavior with unicode.translate on OSX. (2004-09-22) http://python.org/sf/1032615 closed by doerwalter Can't inherit slots from new-style classes implemented in C (2004-09-24) http://python.org/sf/1034178 closed by philthompson10 len(str(x)) in decimal.py is not Python 2.3 compatible (2004-09-30) http://python.org/sf/1037373 closed by ncoghlan printf() in dynload_shlib.c should be PySys_WriteStderr (2004-09-29) http://python.org/sf/1036752 closed by loewis hex() and oct() documentation is incorrect (2004-09-27) http://python.org/sf/1035279 closed by rhettinger silent installation on windows: no drive in registry values (2004-10-01) http://python.org/sf/1038410 closed by theller Typo "signiture" in HTML generated by DocXMLRPCServer (2004-10-02) http://python.org/sf/1038935 closed by bcannon apply.__doc__ says "Deprecated since release 2.3" (2004-10-02) http://python.org/sf/1039092 closed by rhettinger Message.get_filename() problem (2003-12-04) http://python.org/sf/854102 closed by bwarsaw time zone tests fail on Windows (2004-10-02) http://python.org/sf/1039270 closed by bcannon Case sensitivity bug in ConfigParser (2004-08-27) http://python.org/sf/1017864 closed by goodger ConfigParser behavior change (2004-07-24) http://python.org/sf/997050 closed by goodger New / Reopened RFE __________________ win32 silent installation: allow MAINDIR on command line (2004-10-04) http://python.org/sf/1039857 opened by Matteo Gallivanoni os.run function for convinience and robustness (2004-10-04) http://python.org/sf/1040267 opened by Alexander Schmolck From amk at amk.ca Wed Oct 6 21:57:05 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Oct 6 21:58:51 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <4161C45A.4050205@mail.ru> References: <4161C45A.4050205@mail.ru> Message-ID: <20041006195705.GB17032@rogue.amk.ca> On Tue, Oct 05, 2004 at 01:44:58AM +0400, Michael P. Dubner wrote: > I'm posting PEProposal for discussion. Last part is reasoning for > posting it in this hot time. I like this proposal; it begins to impose some degree of sanity on the standard library. (If you want editing help with the grammar and text of the PEP, let me know; I'll happily polish the wording.) > logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) There's been discussion in the past of having a name for the Python standard library, so that 'from import httplib' would always get the standard httplib module. This name should match whatever's using in the logging, so the logging should use a name everyone is happy with using in import statements. Whether that's stdlib or std or __std__ or Lib isn't important, as long as the logging matches. PEP 8 should also gain some text encouraging the use of the logging module in code and describing the dummy.junk convention. --amk From barry at python.org Wed Oct 6 22:16:22 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 6 22:16:30 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <20041006195705.GB17032@rogue.amk.ca> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> Message-ID: <1097093782.20140.39.camel@geddy.wooz.org> On Wed, 2004-10-06 at 15:57, A.M. Kuchling wrote: > There's been discussion in the past of having a name for the Python > standard library, so that 'from import httplib' would always > get the standard httplib module. This name should match whatever's > using in the logging, so the logging should use a name everyone is > happy with using in import statements. Whether that's stdlib or std > or __std__ or Lib isn't important, as long as the logging matches. +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041006/84bdc3bb/attachment.pgp From ncoghlan at email.com Thu Oct 7 00:18:39 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Thu Oct 7 00:18:43 2004 Subject: [Python-Dev] Generator expression in _strptime.py TimeRE.__init__ Message-ID: <1097101119.41646f3f1177a@mail.iinet.net.au> This is currently breaking the tests for me, as [:] doesn't work for generator expressions. Either TimeRE.__init__ needs to go back to using a list comp, or it needs to use list() in __seqToRE instead of [:]. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From bac at OCF.Berkeley.EDU Thu Oct 7 00:51:08 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Oct 7 00:51:20 2004 Subject: [Python-Dev] Generator expression in _strptime.py TimeRE.__init__ In-Reply-To: <1097101119.41646f3f1177a@mail.iinet.net.au> References: <1097101119.41646f3f1177a@mail.iinet.net.au> Message-ID: <416476DC.90808@ocf.berkeley.edu> Nick Coghlan wrote: > This is currently breaking the tests for me, as [:] doesn't work for generator > expressions. > > Either TimeRE.__init__ needs to go back to using a list comp, or it needs to use > list() in __seqToRE instead of [:]. > Damn, sorry everyone. Fixed now. I would have sworn that I ran the regression tests before I did the commit, but I guess I didn't between the last commit and the previous one. But it is fixed now. -Brett From pythondev-dang at lazytwinacres.net Thu Oct 7 01:48:59 2004 From: pythondev-dang at lazytwinacres.net (Daniel 'Dang' Griffith) Date: Thu Oct 7 01:49:04 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <20041003034331.46FB91E400A@bag.python.org> References: <20041003034331.46FB91E400A@bag.python.org> Message-ID: <4164846B.3040205@lazytwinacres.net> At 11:43 PM 10/2/2004, python-dev-request@python.org wrote: > Date: Sat, 2 Oct 2004 21:14:07 -0400 > From: "Raymond Hettinger" > Subject: RE: [Python-Dev] ConfigParser patches > To: > Cc: python-dev@python.org > Message-ID: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> > Content-Type: text/plain; charset="us-ascii" > > > I was in fact thinking of the -m proposal when I wrote that... > > The timing does suck. > > > It's so easy to define an alias or use a one-line shell script for > > invoking Python with a full pathname that I'm really not sure I like > > the -m idea, > > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). > > I have a big pile of batch files just to invoke timeit, texchecker, > profile, etc. It bites the big one. Am I missing something? I run python scripts all the time from the command line in Windows. Modify your PATHEXT environment variable: PATHEXT=.PY;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH Then put your script directory into your PATH: PATH=C:\Python23\lib;%PATH% The first time you run timeit (or whatever.py), Windows will prompt you to associate the .py extension with a program. Pick your favorite python.exe. You can also do this manually before running your script. From then on, you're set: C:\>timeit -s "a=range(20)" "a.append(1)" "a.pop()" 1000000 loops, best of 3: 1.13 usec per loop --dang p.s. You can put .pyc in your path too, but then you'll get the compiled version even if the uncompiled version is newer. But you can always compile them. From pje at telecommunity.com Thu Oct 7 01:51:21 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Oct 7 01:51:07 2004 Subject: [Python-Dev] Status of undocumented distutils features in 2.4? Message-ID: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> I was looking up something in the distutils source today, and I noticed that there are some new, undocumented, and not quite complete features that were added earlier this year. Specifically, the 'requires' and 'provides' arguments to 'setup()', and the 'checkdep' command. Should these be removed for 2.4? It doesn't seem likely they'd be able to be fleshed out before the beta, especially since as far as I can recall they haven't been discussed on the Distutils-SIG yet, and some of the features don't seem to be in alignment with the relevant PEPs (e.g. PEP 262's definition of "requires" and "provides"). And, the actual current implementation of the dependency checking is (per the author's comments) "kinda hacky". From fdrake at acm.org Thu Oct 7 03:40:34 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Oct 7 03:40:47 2004 Subject: [Python-Dev] Status of undocumented distutils features in 2.4? In-Reply-To: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> References: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> Message-ID: <200410062140.34725.fdrake@acm.org> On Wednesday 06 October 2004 07:51 pm, Phillip J. Eby wrote: > I was looking up something in the distutils source today, and I noticed > that there are some new, undocumented, and not quite complete features > that were added earlier this year. Specifically, the 'requires' and > 'provides' arguments to 'setup()', and the 'checkdep' command. Anthony and I threw those together pretty quickly at PyCon; it's not clear that they're useful. For Zope Corp., they won't be. We've decided that we will not be adding __version__ assignments into the codebase, because there lies a stupid maintenance hassle. I have no intention of documenting that stuff myself. > Should these be removed for 2.4? Yes. > It doesn't seem likely they'd be able to > be fleshed out before the beta, especially since as far as I can recall > they haven't been discussed on the Distutils-SIG yet, and some of the Frankly, I have no confidence that we'll ever end up with an agreement on something that's sufficient and still easy to use. Versioning simply isn't any easy aspect of packaging. > features don't seem to be in alignment with the relevant PEPs (e.g. PEP > 262's definition of "requires" and "provides"). And, the actual current > implementation of the dependency checking is (per the author's comments) > "kinda hacky". You're being generous. ;-) It's unfortunate that PEP 262 was deferred, but that's what happens when no one has time to work on the beast. -Fred -- Fred L. Drake, Jr. From garthk at gmail.com Thu Oct 7 04:08:43 2004 From: garthk at gmail.com (Garth T Kidd) Date: Thu Oct 7 04:08:46 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <20041006195705.GB17032@rogue.amk.ca> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> Message-ID: <8aeed5d004100619083a1b8bcf@mail.gmail.com> > PEP 8 should also gain some text encouraging the use of the logging > module in code and describing the dummy.junk convention. Index: pep-0008.txt =================================================================== RCS file: /cvsroot/python/python/nondist/peps/pep-0008.txt,v retrieving revision 1.25 diff -c -r1.25 pep-0008.txt *** pep-0008.txt 6 Aug 2004 18:47:26 -0000 1.25 --- pep-0008.txt 7 Oct 2004 02:07:02 -0000 *************** *** 600,605 **** --- 600,608 ---- No: if greeting == True: Yes: if greeting: + - Use the print statement only where the user's objective is + to print. Use the logging method to deliver status information + to users on the fly. For more information, see PEP xxxx [6]. References *************** *** 614,619 **** --- 617,624 ---- [5] Barry's GNU Mailman style guide http://barry.warsaw.us/software/STYLEGUIDE.txt + [6] PEP xxxx, Guidelines for Logging Usage + Copyright From anthony at interlink.com.au Thu Oct 7 05:11:24 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Oct 7 05:11:33 2004 Subject: [Python-Dev] Status of undocumented distutils features in 2.4? In-Reply-To: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> References: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> Message-ID: <4164B3DC.1040804@interlink.com.au> Phillip J. Eby wrote: > I was looking up something in the distutils source today, and I noticed > that there are some new, undocumented, and not quite complete features > that were added earlier this year. Specifically, the 'requires' and > 'provides' arguments to 'setup()', and the 'checkdep' command. They were something we worked on at PyCon. In hindsight, they're probably not the right way to go about it, and should be removed. If you want to remove them, great!, otherwise I'll do it in the next day or so. -- Anthony Baxter It's never too late to have a happy childhood. From mwh at python.net Thu Oct 7 12:16:24 2004 From: mwh at python.net (Michael Hudson) Date: Thu Oct 7 12:16:25 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Doc/tut tut.tex, 1.253, 1.254 In-Reply-To: (rhettinger@users.sourceforge.net's message of "Wed, 06 Oct 2004 23:46:55 -0700") References: Message-ID: <2mlleikf47.fsf@starship.python.net> rhettinger@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Doc/tut > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv27294/Doc/tut > > Modified Files: > tut.tex > Log Message: > SF patch #1035498: -m option to run a module as a script > (Contributed by Nick Coghlan.) Does anyone know how to edit Misc/python.man to reflect this change? Heck, does anyone know enough Nroff or whatever to check that python.man is even vaguely up to date? I see it mentions -E, so it can't be *that* stale... Cheers, mwh -- what proportion of Swedes speak english? xyld: all those with teeth From amk at amk.ca Thu Oct 7 14:40:32 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Oct 7 14:42:19 2004 Subject: [Python-Dev] Man pages In-Reply-To: <2mlleikf47.fsf@starship.python.net> References: <2mlleikf47.fsf@starship.python.net> Message-ID: <20041007124032.GA20713@rogue.amk.ca> On Thu, Oct 07, 2004 at 11:16:24AM +0100, Michael Hudson wrote: > Does anyone know how to edit Misc/python.man to reflect this change? Done. The man page seems up to date; it even lists a few environment variables that 'python -h' doesn't. This reminds me: the Debian packaging of Python includes a man page for pydoc, written by Moshe Zadka; I think there's another added man page or two. It would be nice to include these pages with the main Python release, but I can't figure out what license Debian's additions are under. Is anyone here from the debian-python mailing list who wants to pursue this? --amk From amk at amk.ca Thu Oct 7 18:31:56 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Oct 7 18:34:11 2004 Subject: [Python-Dev] Version # for PSF license? Message-ID: <20041007163156.GB21908@rogue.amk.ca> DOAP (http://usefulinc.com/doap) is yet another schema for recording data about software projects. I'd like to add support for it to PyPI; however, before doing that a URL needs to be allocated for the PSF license in the listing at http://usefulinc.com/doap/licenses . Question: does the PSF license have a version number of its own or do we have to say 'PSF license for Python 2.2' (2.3, 2.4, etc)? --amk From vegeta.z at gmail.com Thu Oct 7 19:25:20 2004 From: vegeta.z at gmail.com (daniel narf) Date: Thu Oct 7 19:31:28 2004 Subject: [Python-Dev] Toward Python's future article Message-ID: Hi i am sure most of you have read the article of Andrew Kuchling about focusing more in the standart library than language newFeatures/tweaking and probably i as many others would like to know what the python hackers/developers think about this proposal.Maybe this post is out of place but oh well.. i am personaly very interested in improving the stdlib which is very messy in my opinion right now. the article(several comments): http://www.amk.ca/diary/archives/cat_python.html#003382 From python at rcn.com Fri Oct 8 01:55:03 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 8 01:56:30 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: Message-ID: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> > Hi i am sure most of you have read the article of Andrew Kuchling about > focusing > more in the standart library than language newFeatures/tweaking and > probably i > as many others would like to know what the python hackers/developers think > about > this proposal. There is a great deal of merit to efforts for cataloging resources and thinking about ways to improve the standard library. Whether successful or not likely depends on who wants to volunteer their time for what. Where there are multiple, competing third party solutions, Guido has historically resisted ending exploration by locking in a single solution. Instead, he said he prefers "category killers" such as the unittest and logging modules. Martin has also been clear that community acceptance and a commitment to on-going maintenance are also important considerations. For the core, python-dev discussions indicate that several things are still in the works and will probably happen depending on who has the time, interest, and ability: * Someday, decimal may become a built-in type. * The AST version of the compiler may get finished. * A mutable bytes type may emerge on the road to all strings being Unicode. * Someday, C99 may rule the roost and cmath will be updated. * One of three proposals may be accepted for optimized attribute lookup. * A bytecode verifier seems to have a chance. * reST support may be added when it becomes mature enough. * The project to transition to unittests and increase coverage is continuing. * If the class/instance semantics get worked out, exceptions may become new style classes along the road to Py3.0. FWIW, there is a trend toward providing pure python equivalents to CPython implementations (such as that for compiler, sets, bisect, heapq, deques, itertools, decimal, etc). The thought is that these may outlive their C counterparts. > i am personaly very interested in improving the stdlib which is very messy > in my > opinion right now. It's not clear whether you're volunteering or just making a vague blanket request/criticism. If it is the former, then the backlog of bug reports and patch reviews is a good place to start. Raymond From jepler at unpythonic.net Fri Oct 8 02:15:54 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Fri Oct 8 02:15:54 2004 Subject: [Python-Dev] Python Bug Day this month? Message-ID: <20041008001554.GC6737@unpythonic.net> I thought the Python Bug Day was going to return this month, and I was hoping to spare some of my time to help out (on one previous bug day I reviewed some doc patches), but I haven't seen anything about it. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041007/0f14311b/attachment.pgp From carribeiro at gmail.com Fri Oct 8 02:24:01 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Fri Oct 8 02:24:10 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> References: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> Message-ID: <864d370904100717246f51e6fa@mail.gmail.com> On Thu, 7 Oct 2004 19:55:03 -0400, Raymond Hettinger wrote: > Where there are multiple, competing third party solutions, Guido has > historically resisted ending exploration by locking in a single > solution. Instead, he said he prefers "category killers" such as the > unittest and logging modules. Martin has also been clear that community > acceptance and a commitment to on-going maintenance are also important > considerations. I see the reasoning behind this position. But I fear it can lead to lack of standardization in some areas. There is at least one area where a firm position would greatly clarify things: standard interfaces for modules where multiple providers are always the norm, such as the DB API and WSGI. WSGI is working is way out, but it's still too early to tell if they will be successful. But the DB API could really take advantage of a BDFL-style intervention to settle things. (it does not *need* to be Guido, but it has to be someone with power enough to take a firm stand and enforce it; but I don't know anyone who could assume this role as the undisputed leader of the DB-API effort). > For the core, python-dev discussions indicate that several things are > still in the works and will probably happen depending on who has the > time, interest, and ability: > > * Someday, decimal may become a built-in type. > * The AST version of the compiler may get finished. > * A mutable bytes type may emerge on the road to all strings being > Unicode. > * Someday, C99 may rule the roost and cmath will be updated. > * One of three proposals may be accepted for optimized attribute lookup. > * A bytecode verifier seems to have a chance. > * reST support may be added when it becomes mature enough. > * The project to transition to unittests and increase coverage is > continuing. > * If the class/instance semantics get worked out, exceptions may become > new style classes along the road to Py3.0. Nice summary. > FWIW, there is a trend toward providing pure python equivalents to > CPython implementations (such as that for compiler, sets, bisect, heapq, > deques, itertools, decimal, etc). The thought is that these may outlive > their C counterparts. That's great -- I mean, taken to its maximum expression, Python would be able to bootstrap itself this way (a Python VM written in Python?). It's a sign of maturity, don't you think? > > i am personaly very interested in improving the stdlib which is very > messy > > in my > > opinion right now. > > It's not clear whether you're volunteering or just making a vague > blanket request/criticism. If it is the former, then the backlog of bug > reports and patch reviews is a good place to start. I still need to find my way there (docs/sourceforge stuff). Depending on the way my current projects go, I may still be able to dedicate some time to it. I'm really looking forward to it... -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From mal at egenix.com Fri Oct 8 10:54:34 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Oct 8 10:55:18 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <864d370904100717246f51e6fa@mail.gmail.com> References: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> <864d370904100717246f51e6fa@mail.gmail.com> Message-ID: <416655CA.1020006@egenix.com> Carlos Ribeiro wrote: > On Thu, 7 Oct 2004 19:55:03 -0400, Raymond Hettinger wrote: > >>Where there are multiple, competing third party solutions, Guido has >>historically resisted ending exploration by locking in a single >>solution. Instead, he said he prefers "category killers" such as the >>unittest and logging modules. Martin has also been clear that community >>acceptance and a commitment to on-going maintenance are also important >>considerations. > > > I see the reasoning behind this position. But I fear it can lead to > lack of standardization in some areas. There is at least one area > where a firm position would greatly clarify things: standard > interfaces for modules where multiple providers are always the norm, > such as the DB API and WSGI. WSGI is working is way out, but it's > still too early to tell if they will be successful. But the DB API > could really take advantage of a BDFL-style intervention to settle > things. (it does not *need* to be Guido, but it has to be someone with > power enough to take a firm stand and enforce it; but I don't know > anyone who could assume this role as the undisputed leader of the > DB-API effort). I don't know anything about the way the WSGI specification is developed. The DB-API specification was developed outside the Python development process and has so far been very successful at that. The DB-SIG which manages the specification is in charge of the process and will continue to work together with module authors and users to accommodate for new features that may become generally available in databases or their interfaces. Note that there is no need to "settle things": the DB-API spec has been around for many years, is very stable, widely used and still flexible enough to meet requirements of various different database backends. If you feel that a certain feature is missing, I'd suggest you direct your constructive criticism to the db-sig@python.org. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From astrand at lysator.liu.se Fri Oct 8 13:07:02 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Fri Oct 8 13:07:13 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: [From Mail to python-announce list] > I'd like to give you an updated status report of the popen5 project. Since > my last post at 2004-06-04, this has happened: ... > With these changes, the subprocess module now feels quite complete. In the > near future, I will focus on getting the module included in the standard > library. I've recieved very positive feedback on my module. Many users are also asking me if this module will be included in the standard library. That is, of course, my wish as well. So, can the subprocess module be accepted? If not, what needs to be done? /Peter ?strand From walter at livinglogic.de Fri Oct 8 13:35:41 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri Oct 8 13:35:44 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> References: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> Message-ID: <41667B8D.8090406@livinglogic.de> Raymond Hettinger wrote: > [...] > * If the class/instance semantics get worked out, exceptions may become > new style classes along the road to Py3.0. So should we switch from a PendingDeprecationWarning to a DeprecationWaring for string exceptions in 2.4? Bye, Walter D?rwald From amk at amk.ca Fri Oct 8 14:19:47 2004 From: amk at amk.ca (A.M. Kuchling) Date: Fri Oct 8 14:21:38 2004 Subject: [Python-Dev] Python Bug Day this month? In-Reply-To: <20041008001554.GC6737@unpythonic.net> References: <20041008001554.GC6737@unpythonic.net> Message-ID: <20041008121947.GA27702@rogue.amk.ca> On Thu, Oct 07, 2004 at 07:15:54PM -0500, Jeff Epler wrote: > I thought the Python Bug Day was going to return this month, and I was > hoping to spare some of my time to help out (on one previous bug day I > reviewed some doc patches), but I haven't seen anything about it. I don't have time to do the organization for it or to spare an entire day for it. If someone else wants to make arrangements, feel free! --amk From theller at python.net Fri Oct 8 16:30:49 2004 From: theller at python.net (Thomas Heller) Date: Fri Oct 8 16:30:38 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: (Peter Astrand's message of "Fri, 8 Oct 2004 13:07:02 +0200 (MEST)") References: Message-ID: Peter Astrand writes: > [From Mail to python-announce list] > >> I'd like to give you an updated status report of the popen5 project. Since >> my last post at 2004-06-04, this has happened: > ... >> With these changes, the subprocess module now feels quite complete. In the >> near future, I will focus on getting the module included in the standard >> library. > > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. That > is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be done? I would suggest to replace the _subprocess extension module by a Python implementation based on ctypes (and include ctypes in 2.4, at least for Windows). How many _xxxx.pyd windows specific modules do we need? Thomas From carribeiro at gmail.com Fri Oct 8 17:46:57 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Fri Oct 8 17:47:00 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <416655CA.1020006@egenix.com> References: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> <864d370904100717246f51e6fa@mail.gmail.com> <416655CA.1020006@egenix.com> Message-ID: <864d3709041008084665e0956f@mail.gmail.com> Hello all, There must be something about the way I write my comments that may be making them sound worst than I want. I have _no_ intention to start a flame war... and I really would like to make it sound constructive. I apologize if it sounded otherwise. (I have a few comments that still belong here, if only to clarify things) On Fri, 08 Oct 2004 10:54:34 +0200, M.-A. Lemburg wrote: > Note that there is no need to "settle things": the DB-API > spec has been around for many years, is very stable, > widely used and still flexible enough to meet requirements > of various different database backends. There are several packages around that don't implement the DB API correctly, or implement a completely different API. This is clearly not a fault of the DB API itself, or of its developers, but it's an issue for its users. It's a difficult situation, because nobody is really in a position to enforce compliance. In this sense, bringing the issue to the standard library _could_ be helpful. One suggestion is to have a reference implementation -- for example, a simple DB API module, or a library that makes use of DB-API compliant modules. Of course, it's open to discussion whether this is feasible or not; for now it's only a personal opinion. > If you feel that a certain feature is missing, I'd suggest > you direct your constructive criticism to the db-sig@python.org. I think the DB API itself is fine. What is lacking is some way to make sure that everyone _comply_ to it. If that's can be done within the DB-SIG group alone, fine. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From theller at python.net Fri Oct 8 20:09:30 2004 From: theller at python.net (Thomas Heller) Date: Fri Oct 8 20:09:19 2004 Subject: [Python-Dev] HAVE_USABLE_WCHAR_T Message-ID: The Include/unicodeobject.h file says (line 103): /* If the compiler provides a wchar_t type we try to support it through the interface functions PyUnicode_FromWideChar() and PyUnicode_AsWideChar(). */ This isn't true - grepping the CVS sources for this symbol shows that it is used in these ways: - When defined together with the WANT_WCTYPE_FUNCTIONS symbol, the compiler's wctype.h functions are used instead of the ones supplied with Python. Include/unicodeobject.h, line 294. - When defined together with MS_WINDOWS, it makes available mbcs_enocde and mbcs_decode functions (in Modules/_codecsmodule.c), plus the PyUnicode_DecodeMBCS and PyUnicode_AsMBCSString functions in Objects/unicodeobject.c. - Contrary to the comment at the top of this message, the PyUnicode_FromWideChar and PyUnicode_AsWideChar functions are compiled when HAVE_WCHAR_H is defined. The HAVE_USABLE_WCHAR_T symbol is only used to determine whether memcpy is used, or the unicode characters are copied one by one. - Finally, again when defined together with MS_WINDOWS, it sets the filesystem encoding to mbcs. So, it seems that the HAVE_USABLE_WCHAR_T symbol doesn't play any role for the extension programmer *at all*. The preprocessor flag that plays a role for extensions seem to be HAVE_WCHAR_H since they mark whether the PyUnicode_FromWideChar and PyUnicode_AsWideChar are available or not. This has caused me quite some confusion, and so I suggest the comment above in the Include/unicodeobject.h file should be fixed. Finally, the docs also seem to get it wrong (although I haven't followed that in detail). Can't reach python.org at the moment, but Python C/api manual, section 7.3.2, unicode objects says: Py_UNICODE This type represents a 16-bit unsigned storage type which is used by Python internally as basis for holding Unicode ordinals. On platforms where wchar_t is available and also has 16-bits, Py_UNICODE is a typedef alias for wchar_t to enhance native platform compatibility. On all other platforms, Py_UNICODE is a typedef alias for unsigned short. Isn't the size 32 bits for wide unicode builds? Please, please fix this - unicode is already complicated enough even without this confusion! Thomas From lunz at falooley.org Fri Oct 8 20:16:20 2004 From: lunz at falooley.org (Jason Lunz) Date: Fri Oct 8 20:16:24 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: astrand@lysator.liu.se said: > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. > That is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be > done? I've been watching the progression of subprocess with some interest. It looks encouraging, and is exactly the sort of thing I need for my work. One small nit I've noticed: aren't the names of subprocess.call() and subprocess.callv() reversed? If you look at unix execl() and execv(), execl() takes a variable-length argument list and execv() takes a list (vector?) of arguments. But it's the opposite for subprocess -- callv() takes a variable-length arg list and call() takes a list of args. Am I missing something? Can these be renamed now before it gets standardized? Jason From mal at egenix.com Fri Oct 8 21:30:11 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Oct 8 21:30:15 2004 Subject: [Python-Dev] HAVE_USABLE_WCHAR_T In-Reply-To: References: Message-ID: <4166EAC3.7080708@egenix.com> Thomas Heller wrote: > The Include/unicodeobject.h file says (line 103): > > /* If the compiler provides a wchar_t type we try to support it > through the interface functions PyUnicode_FromWideChar() and > PyUnicode_AsWideChar(). */ > > This isn't true - grepping the CVS sources for this symbol shows that it > is used in these ways: > > - When defined together with the WANT_WCTYPE_FUNCTIONS symbol, the > compiler's wctype.h functions are used instead of the ones supplied with > Python. Include/unicodeobject.h, line 294. > > - When defined together with MS_WINDOWS, it makes available mbcs_enocde > and mbcs_decode functions (in Modules/_codecsmodule.c), plus the > PyUnicode_DecodeMBCS and PyUnicode_AsMBCSString functions in > Objects/unicodeobject.c. > > - Contrary to the comment at the top of this message, the > PyUnicode_FromWideChar and PyUnicode_AsWideChar functions are compiled > when HAVE_WCHAR_H is defined. The HAVE_USABLE_WCHAR_T symbol is only > used to determine whether memcpy is used, or the unicode characters are > copied one by one. > > - Finally, again when defined together with MS_WINDOWS, it sets the > filesystem encoding to mbcs. > > > So, it seems that the HAVE_USABLE_WCHAR_T symbol doesn't play any role > for the extension programmer *at all*. That symbol is defined by the configure script for use in the interpreter - why did you think it is usable for extensions ? The HAVE_USABLE_WCHAR_T symbol only means that we can use wchar_t as synonym for Py_UNICODE and thus makes some APIs more efficient, e.g. on Windows - nothing more. > The preprocessor flag that plays > a role for extensions seem to be HAVE_WCHAR_H since they mark whether > the PyUnicode_FromWideChar and PyUnicode_AsWideChar are available or > not. Right, since wchar.h is the include file that defines the wchar_t type. > This has caused me quite some confusion, and so I suggest the comment > above in the Include/unicodeobject.h file should be fixed. > > Finally, the docs also seem to get it wrong (although I haven't followed > that in detail). Can't reach python.org at the moment, but Python C/api > manual, section 7.3.2, unicode objects says: > > Py_UNICODE > > This type represents a 16-bit unsigned storage type which is used by > Python internally as basis for holding Unicode ordinals. On platforms > where wchar_t is available and also has 16-bits, Py_UNICODE is a > typedef alias for wchar_t to enhance native platform compatibility. On > all other platforms, Py_UNICODE is a typedef alias for unsigned short. > > Isn't the size 32 bits for wide unicode builds? Yes. > Please, please fix this - unicode is already complicated enough even > without this confusion! Please add a bug report to SF for this. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From vegeta.z at gmail.com Fri Oct 8 21:32:03 2004 From: vegeta.z at gmail.com (daniel narf) Date: Fri Oct 8 21:32:04 2004 Subject: [Python-Dev] Re: Toward Python's future article References: Message-ID: i know there are a lot of volunters,fans,developers who are very actively developing/improving the library but the main point of the argument(i think),is to realy focus on the stdlib improvements,not implying that the interpreter improvements are less important,just that the stdlib is currently lacking a bit in certain aspects already mentioned,to try to balance the ecosystem. A more formal,focused,organized aproach to improve the stdlib is the point of the proposal,some sort of plan,roadmap is what is needed(in my opinion). Some things that comes to mind and others based on feedback about the AFK non formal proposal are: to refactor(someway) stdlib parts that are a bit inconsistent,as duplicated functionality of some modules,modules with just one function,obscure ways of working of some of them,to stablish realy strong naming,style coding conventions ie: modules names with get_type,getType,gettype. To realy start a stdlib documentation effort,aspiring it to be the best,clearest documented stdlib out there. To develop the greatest,more advanced library packaging system out there,taking the strengh and weaknesses of others(as CPAM) as a guide. To start a planed and formal draft for py 3.0,some similar system as the iniciative from zope,the zope X3 project and maybe to #adelantar a litle python 3.0 realease(to jump from python 2.7 to python 3000 ^_^). And many other great ideas to start making python the best language ever. From anthony at interlink.com.au Fri Oct 8 22:13:06 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Oct 8 22:18:38 2004 Subject: [Python-Dev] Re: Toward Python's future article In-Reply-To: References: Message-ID: <4166F4D2.4070107@interlink.com.au> daniel narf wrote: > i know there are a lot of volunters,fans,developers who are very actively > developing/improving the library but the main point of the argument(i > think),is to realy focus on the stdlib improvements,not implying that the > interpreter improvements are less important,just that the stdlib is > currently lacking a bit in certain aspects already mentioned,to try to > balance the ecosystem. As I said in a comment to Andrew's piece - the _best_ way is to pick up a piece of the stdlib that annoys you, and fix it. Attempting to setup a roadmap, or a plan, or whatever, will only mean that nothing ever gets done. For instance, look at the long and sorry history before PyPI. Many, many attempts to do some big picture thing, and no result. What we have now isn't perfect, but it's a start. And it was accomplished by one person (Richard) getting annoyed enough to just roll up the sleeves and do it. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fredrik at pythonware.com Fri Oct 8 22:24:23 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Oct 8 22:22:19 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: Peter Astrand wrote: > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. That > is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be done? just check it in, already. (can os.popen and popen2 be refactored to use subprocess, btw?) From dubnerm-news at mail.ru Sat Oct 9 00:18:44 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Sat Oct 9 00:16:56 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <20041006195705.GB17032@rogue.amk.ca> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> Message-ID: <41671244.5070003@mail.ru> A.M. Kuchling wrote: >I like this proposal; it begins to impose some degree of sanity on the >standard library. > > I was the reason behind this PEP. :-) >(If you want editing help with the grammar and text of the PEP, let me >know; I'll happily polish the wording.) > > I will be happy to get such help. I haven't grammar checker handy, so as I recall I've only checked spelling once (and possibly not last edition). Sorry to native English speakers. >> logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) >> >> >There's been discussion in the past of having a name for the Python >standard library, so that 'from import httplib' would always >get the standard httplib module. This name should match whatever's >using in the logging, so the logging should use a name everyone is >happy with using in import statements. Whether that's stdlib or std >or __std__ or Lib isn't important, as long as the logging matches. > > I'm not really sure that these names MUST be equal. Name to use for import statement can be __stdlib__ or ever __python_library__ to ensure uniqueness and specific name MUST draw attention to this place. But for logger name this prefix should be as short as possible because it's to be displayed, and sometime logger plus time, level and message text must fit 80 character line... -- Best regards, Michael Dubner (dubnerm.at.mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From ncoghlan at email.com Sat Oct 9 02:04:01 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sat Oct 9 02:04:07 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <41671244.5070003@mail.ru> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> <41671244.5070003@mail.ru> Message-ID: <1097280241.41672af18e6be@mail.iinet.net.au> Quoting "Michael P. Dubner" : > >There's been discussion in the past of having a name for the Python > >standard library, so that 'from import httplib' would always > >get the standard httplib module. This name should match whatever's > >using in the logging, so the logging should use a name everyone is > >happy with using in import statements. Whether that's stdlib or std > >or __std__ or Lib isn't important, as long as the logging matches. > > > > > I'm not really sure that these names MUST be equal. Name to use for > import statement can be __stdlib__ > or ever __python_library__ to ensure uniqueness and specific name MUST > draw attention to this place. But > for logger name this prefix should be as short as possible because it's > to be displayed, and sometime logger > plus time, level and message text must fit 80 character line... I agree with Andrew that the names should be equal - consistency is good, and brevity is a benefit for 'from x import' as well. I'd propose taking a tip from Java and using "py." as the prefix for the stdlib. Short, to the point, and I'd be surprised if anyone was using a bare "py" as a package or module name. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From fredrik at pythonware.com Sat Oct 9 08:07:20 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 9 08:05:16 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: Message-ID: Thomas Heller wrote: > How many _xxxx.pyd windows specific modules do we need? as many as it takes to make Python an excellent tool. it's not like one more entry in the makefile, and 20k in the binary distribution will cause problem for anyone. if you want to stop contributions to Python's standard library unless they use your tools, you have a serious problem. From astrand at lysator.liu.se Sat Oct 9 12:08:20 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 12:30:33 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: > I've been watching the progression of subprocess with some interest. It > looks encouraging, and is exactly the sort of thing I need for my work. > > One small nit I've noticed: aren't the names of subprocess.call() and > subprocess.callv() reversed? If you look at unix execl() and execv(), > execl() takes a variable-length argument list and execv() takes a list > (vector?) of arguments. But it's the opposite for subprocess -- callv() > takes a variable-length arg list and call() takes a list of args. Oh. Yes, you are right. > Am I missing something? Can these be renamed now before it gets > standardized? I'd prefer not to rename the call() function. The name is short and simple, and the function is very much used. I'm positive to renaming the callv() function, though. One obvious name would be "calll", but that's quite ugly. How about "lcall"? Then we can keep the "callv" name for backwards compatibility. Or, we could just keep the "callv" name, and pretend that "v" stands for "variable number of arguments". /Peter ?strand From fredrik at pythonware.com Sat Oct 9 13:06:36 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 9 13:04:33 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: Message-ID: Peter Astrand wrote: > > Am I missing something? Can these be renamed now before it gets > > standardized? > > I'd prefer not to rename the call() function. The name is short and > simple, and the function is very much used. I'm positive to renaming the > callv() function, though. One obvious name would be "calll", but that's > quite ugly. How about "lcall"? Then we can keep the "callv" name for > backwards compatibility. do we need both? you could rename "callv" to "call", and let people type an extra "*" if they want to pass in a list of arguments: subprocess.call("stty", "sane", "-F", device) subprocess.call(*["stty", "sane", "-F", device]) or, more likely: args = ["somecommand"] # several lines of code to add options and parameters # to the args list subprocess.call(*args) > Or, we could just keep the "callv" name, and pretend that "v" stands for > "variable number of arguments". I have no problem with that... From astrand at lysator.liu.se Sat Oct 9 14:04:33 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 14:29:47 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: On Sat, 9 Oct 2004, Fredrik Lundh wrote: > > I'd prefer not to rename the call() function. The name is short and > > simple, and the function is very much used. I'm positive to renaming the > > callv() function, though. One obvious name would be "calll", but that's > > quite ugly. How about "lcall"? Then we can keep the "callv" name for > > backwards compatibility. > > do we need both? you could rename "callv" to "call", and let people type > an extra "*" if they want to pass in a list of arguments: > > subprocess.call("stty", "sane", "-F", device) > subprocess.call(*["stty", "sane", "-F", device]) I'd like "call" to stay as it is. It's interface is pretty clean, and maps directly to the Popen constructor. "callv" is only syntactic sugar. It's for people that thinks that: subprocess.call(["ls", "-l", "/foo/bar"]) ...is to ugly compared to: os.system("ls -l /foo/bar") With callv, it is: subprocess.callv("ls", "-l", "/foo/bar") ...which is slighly nicer. The drawback with callv is that it does not allow specifying the program and it's arguments as a whitespace-separated string: The entire (first) string would be intepreted as the executable. So, you cannot do: subprocess.callv("somewindowsprog.exe some strange command line") because then the system would try to execute a program called "somewindowsprog.exe some strange command line", which doesn't exist. You cannot do this either: subprocess.callv("somewindowsprog.exe", "some", "strange", "command", "line") ...if somewindowsprog.exe doesn't use the MS C runtime argument rules. To summarize: call() must stay as it is for completeness. callv() is just syntactic sugar, but probably deserves to stay as well. /Peter ?strand From lunz at falooley.org Sat Oct 9 15:28:29 2004 From: lunz at falooley.org (Jason Lunz) Date: Sat Oct 9 15:28:33 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: astrand@lysator.liu.se said: > I'd prefer not to rename the call() function. The name is short and > simple, and the function is very much used. That's understandable. Though if people are going to go through the pain of changing it, it's better that it happen before it becomes a standard part of python. > I'm positive to renaming the callv() function, though. One obvious > name would be "calll", but that's quite ugly. How about "lcall"? Then > we can keep the "callv" name for backwards compatibility. How recently was callv added? I'd prefer not to have a callv at all than to have a call/callv pair that don't map naturally to execl/execv. > Or, we could just keep the "callv" name, and pretend that "v" stands for > "variable number of arguments". I really don't want to do this. I can tell already I'll be forever forgetting which one I need, and probably anyone else with C/unix experience will be in the same boat. It's the kind of irritant I'd like to wipe out now while there's still the opportunity. Jason From lunz at falooley.org Sat Oct 9 15:54:22 2004 From: lunz at falooley.org (Jason Lunz) Date: Sat Oct 9 15:54:26 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: astrand@lysator.liu.se said: > ...which is slighly nicer. The drawback with callv is that it does not > allow specifying the program and it's arguments as a whitespace-separated > string: The entire (first) string would be intepreted as the executable. > So, you cannot do: > > subprocess.callv("somewindowsprog.exe some strange command line") > > because then the system would try to execute a program called > "somewindowsprog.exe some strange command line", which doesn't exist. You > cannot do this either: > > subprocess.callv("somewindowsprog.exe", "some", "strange", "command", "line") > > ...if somewindowsprog.exe doesn't use the MS C runtime argument rules. I'm not sure I understand what the MSC runtime has to do with the naming of call/callv. Your examples don't work with call either, right? Their call() equivalents: subprocess.call(["somewindowsprog.exe some strange command line"]) subprocess.call(["somewindowsprog.exe", "some", "strange", "command", "line"]) are just as broken, no? Overall, I agree that callv() is superfluous. In my programming, I always end up using the "v" variants of exec functions, because there's always _something_ you do to the command line first, and it's easier to handle arguments as a list. [The above paragraph makes my point: "I always use execv(), so we should drop subprocess.callv()?" The naming hurts my poor brain.] Jason From astrand at lysator.liu.se Sat Oct 9 15:57:50 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 16:26:16 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: On Sat, 9 Oct 2004, Jason Lunz wrote: > > I'm positive to renaming the callv() function, though. One obvious > > name would be "calll", but that's quite ugly. How about "lcall"? Then > > we can keep the "callv" name for backwards compatibility. > > How recently was callv added? I'd prefer not to have a callv at all than > to have a call/callv pair that don't map naturally to execl/execv. callv has been around even longer than call actually, although callv was earlier called "run". > > Or, we could just keep the "callv" name, and pretend that "v" stands for > > "variable number of arguments". > > I really don't want to do this. I can tell already I'll be forever > forgetting which one I need, and probably anyone else with C/unix > experience will be in the same boat. It's the kind of irritant I'd like > to wipe out now while there's still the opportunity. I don't have a very strong opinion about callv, so if the general opinion wants to remove it, that's OK with me. /Peter ?strand From astrand at lysator.liu.se Sat Oct 9 16:06:23 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 16:47:55 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: > > subprocess.callv("somewindowsprog.exe", "some", "strange", "command", "line") > > > > ...if somewindowsprog.exe doesn't use the MS C runtime argument rules. > > I'm not sure I understand what the MSC runtime has to do with the naming > of call/callv. In that case, my explanation wasn't good enough :) It's somewhat complicated. Most people will never have any problems with these issues, but I've taken care so that the API should support all cornercases. >Your examples don't work with call either, right? They work with call if you use a string argument. That's the core of the problem: The callv function doesn't support passing a string-type args argument to the Popen constructor. >Their > call() equivalents: > > subprocess.call(["somewindowsprog.exe some strange command line"]) > subprocess.call(["somewindowsprog.exe", "some", "strange", "command", "line"]) > > are just as broken, no? Yes. You'll need to do: subprocess.call("somewindowsprog.exe some strange command line") /Peter ?strand From ncoghlan at email.com Sat Oct 9 18:24:00 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sat Oct 9 18:50:38 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: <1097339040.416810a0f3a95@mail.iinet.net.au> Quoting Peter Astrand : > >Your examples don't work with call either, right? > > They work with call if you use a string argument. That's the core of the > problem: The callv function doesn't support passing a string-type args > argument to the Popen constructor. > > > >Their > > call() equivalents: > > > > subprocess.call(["somewindowsprog.exe some strange command line"]) > > subprocess.call(["somewindowsprog.exe", "some", "strange", "command", > "line"]) > > > > are just as broken, no? > > Yes. You'll need to do: > > subprocess.call("somewindowsprog.exe some strange command line") Given that call is only a shortcut function, wouldn't the following suffice?: def call(*args, **kwds): if len(args) <= 1: return Popen(*args, **kwds) else: return Popen(args, **kwds) With that implementation, a single string, a list, a sequence of strings and Popen keywords only would all work as arguments to call. That is: call("ls -l") -> Popen("ls -l") call("ls", "-l") -> Popen(["ls", "-l"]) call(["ls", "-l"]) -> Popen(["ls", "-l"]) call(args="ls -l") -> Popen(args="ls -l") All it would mean is that if you want to use the optional arguments to Popen, you either don't use call, or you use keywords. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From erik at heneryd.com Sat Oct 9 19:46:38 2004 From: erik at heneryd.com (Erik Heneryd) Date: Sat Oct 9 19:46:43 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: <416823FE.6080206@heneryd.com> Haven't really read the PEP/tried the module, so I can't comment on it specifically, but +1 on a proper subprocess module. Implemented something like this myself some time ago. Peter Astrand wrote: >>Am I missing something? Can these be renamed now before it gets >>standardized? > > > I'd prefer not to rename the call() function. The name is short and > simple, and the function is very much used. I'm positive to renaming the > callv() function, though. One obvious name would be "calll", but that's > quite ugly. How about "lcall"? Then we can keep the "callv" name for > backwards compatibility. Don't think backwards compatibility is that much of an issue. Since you're renaming it subprocess (+1 on the name) old code will have to be updated anyway. -1 on function names conflicting with the exec/spawn way of naming things. Erik From lunz at falooley.org Sat Oct 9 20:01:13 2004 From: lunz at falooley.org (Jason Lunz) Date: Sat Oct 9 20:01:23 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: astrand@lysator.liu.se said: > They work with call if you use a string argument. That's the core of the > problem: The callv function doesn't support passing a string-type args > argument to the Popen constructor. OK, I read the code again and I see what you mean. So yes, this argues even more against the existence of callv(). I would have expected that it would always be possible to translate a call() invocation to its callv() equivalent and vice-versa. That turns out not to be true in the case of call() users who want MSC-style command-line parsing done on a bare string (whether by CreateProcess() on windows or cmdline2list() on unix), because the _execute_child() implementations need to know whether args was originally a string or a list, and this information is hidden by callv()'s list encapsulation of the args. This all makes me think there could be a better approach to cross-platform handling of command-line arguments. When is anyone ever going to want the braindamaged MS cmdline expansion done while they're trying to invoke a subprocess on unix? I don't see that getting used a lot. I think I need to understand better the division of labor between parent and child on Windows when it comes to passing the command line during process creation. I'd like to think that unix execv() is a superset of the interface offered by CreateProcess(), in that you can initialize your child's argv however you like without regard to whitespace or quoting. So it would be best if it were possible to offer the execv() interface on both platforms if that's possible. Jason From pf_moore at yahoo.co.uk Sat Oct 9 21:18:37 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sat Oct 9 21:18:41 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: Jason Lunz writes: > I think I need to understand better the division of labor between parent > and child on Windows when it comes to passing the command line during > process creation. It's simple. The parent passes a *command line*, and the child retrieves it using the GetCommandLine API. There is no concept of argv at the Windows level - it is implemented by the C runtime parsing the return value of GetCommandLine() and passing the resulting arguments to main(). Hence on Windows, a command line is the fundamental unit, whereas on Unix an argument list is fundamental. The biggest problem on Windows is that not all executables use the Microsoft C runtime. Some use other C runtimes, others parse the command line directly and don't use argv at all. > I'd like to think that unix execv() is a superset of the interface > offered by CreateProcess(), in that you can initialize your child's > argv however you like without regard to whitespace or quoting. It would be nice to think that, but it ain't true . > So it would be best if it were possible to offer the execv() > interface on both platforms if that's possible. The unix execv is just *different*. Both the Windows and the Unix interfaces have capabilities the other doesn't offer. And as a result, it's not possible to offer an execv interface on Windows, at least not without running the risk of garbling some commands. Luckily, the oddball cases are rare. So 99% of the time, either interface will do OK (os.system on Unix, os.spawnv on Windows). But the 1% can be crucial - shell-exploit style security holes on Unix, garbled commands on Windows. I think Peter's approach of supporting both forms - a single string as a command line, and list of strings as an argv list, and converting both to the more natural OS-native form as needed, is sensible (I would, I argued for it when he was developing it!) Paul -- Ooh, how Gothic. Barring the milk. From nhodgson at bigpond.net.au Sun Oct 10 01:06:48 2004 From: nhodgson at bigpond.net.au (Neil Hodgson) Date: Sun Oct 10 01:06:53 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module Message-ID: <015001c4ae54$a7b32d40$a44a8890@neil> Fredrik Lundh: > Thomas Heller wrote: > > > How many _xxxx.pyd windows specific modules do we need? > > as many as it takes to make Python an excellent tool. it's not like one > more entry in the makefile, and 20k in the binary distribution will cause > problem for anyone. Requiring windows interface modules to be written in C rather than Python reduces the number of Python users that can add, modify and fix these modules. It also increases the amount of effort involved in working on these modules. Neil From gvanrossum at gmail.com Sun Oct 10 01:18:29 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 10 01:18:36 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module In-Reply-To: <015001c4ae54$a7b32d40$a44a8890@neil> References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: > Requiring windows interface modules to be written in C rather than Python > reduces the number of Python users that can add, modify and fix these > modules. It also increases the amount of effort involved in working on these > modules. I don't know. Writing Windows interface modules is a highly specialized form of torture, requiring arcane expertise. Most Python users don't have that expertise, and could do more damage than good. I'd expect those folks that *do* have the expertise to have learned what they know by coding in C/C++, so I don't see that requiring the use of C is any particular burden. Rather, having to be familiar with the intricate details of the mapping between C and Python is likely to trip up folks occasionally. And what about all the stuff that's defined in header files? In the past, in the Unix world, I experimented with translating .h files to Python modules to save me from the burden of having to add large numbers of symbol definitions to C extensions. (Look for files named Lib/plat-XXX/regen in the source tree.) One by one, that approach has proven to be problematic, and nearly all of those have eventually been turned into systematic lists of symbol definitions in C code. See for example the posix, socket, fcntl, and signal modules. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pgimpelj at sympatico.ca Sun Oct 10 02:49:18 2004 From: pgimpelj at sympatico.ca (Paul Gimpelj) Date: Sun Oct 10 02:48:16 2004 Subject: [Python-Dev] scropion References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: <03c701c4ae62$facd40c0$c901010a@zoom> hello, I am new to python, and I have found a robot vision system that uses python. It looks pretty good. Its called scorpion. It runs on xp platform. This is bringing me back to python. I was worried you would change the architecture. My question is for performance: isn't binary code faster than p code? So If I can vote, I would vote for c or c++ in the numerics functions and data transfer areas. even assembler. The quicker the better. Thanks Paul From nhodgson at bigpond.net.au Sun Oct 10 04:02:59 2004 From: nhodgson at bigpond.net.au (Neil Hodgson) Date: Sun Oct 10 04:03:09 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: <027b01c4ae6d$446deea0$a44a8890@neil> Guido van Rossum: > I don't know. Writing Windows interface modules is a highly > specialized form of torture, requiring arcane expertise. Most Python > users don't have that expertise, and could do more damage than good. > I'd expect those folks that *do* have the expertise to have learned > what they know by coding in C/C++, so I don't see that requiring the > use of C is any particular burden. Visual Basic programmers often write to the Win32 API without knowing enough C to write an extension module. They can declare a function and then call it: Declare Function GetComputerName Lib "kernel32" Alias "GetComputerNameA" (ByVal sBuffer As String, lSize As Long) As Long strString = String(255, Chr$(0)) GetComputerName strString, 255 The incantations are a little more arcane than for C but there is quite a lot of documentation available about how to do this so VB programmers often don't need to read the C header files. One source is win32api.txt which lists the structs, constants, and functions in the Win32 API as VB declarations. > And what about all the stuff that's defined in header files? ctypes doesn't yet have a good solution to header files. http://starship.python.net/crew/theller/moin.cgi/WrappingWin32Api > In the > past, in the Unix world, I experimented with translating .h files to > Python modules to save me from the burden of having to add large > numbers of symbol definitions to C extensions. (Look for files named > Lib/plat-XXX/regen in the source tree.) One by one, that approach has > proven to be problematic, and nearly all of those have eventually been > turned into systematic lists of symbol definitions in C code. > See for example the posix, socket, fcntl, and signal modules. Where the problems are caused by multiple vendor or version variations then Windows is much more stable with additions occurring often but amendments, removals and world changes (Win16->Win32) being very rare. Looking at the all_ins function in posix, it seems to have been made by hand. This wouldn't be a great approach to a large API (there are thousands of values #defined in Win32) where finding the names of the constants (and not, for example, function names which are often #defined to either narrow or wide functions) is as difficult as finding their values. When coding an interface module in Python, it would be fastest to copy just the relevant declarations from the headers or win32api.txt and convert to Python syntax by hand. Neil From dubnerm-news at mail.ru Sun Oct 10 04:27:29 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Sun Oct 10 04:25:12 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <1097280241.41672af18e6be@mail.iinet.net.au> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> <41671244.5070003@mail.ru> <1097280241.41672af18e6be@mail.iinet.net.au> Message-ID: <41689E11.10106@mail.ru> Nick Coghlan wrote: >I agree with Andrew that the names should be equal - consistency is good, and >brevity is a benefit for 'from x import' as well. > >I'd propose taking a tip from Java and using "py." as the prefix for the stdlib. >Short, to the point, and I'd be surprised if anyone was using a bare "py" as a >package or module name. > > +1 -- Best regards, Michael Dubner (dubnerm.at.mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From fdrake at acm.org Sun Oct 10 05:14:55 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun Oct 10 05:15:09 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <1097280241.41672af18e6be@mail.iinet.net.au> References: <4161C45A.4050205@mail.ru> <41671244.5070003@mail.ru> <1097280241.41672af18e6be@mail.iinet.net.au> Message-ID: <200410092314.55154.fdrake@acm.org> On Friday 08 October 2004 08:04 pm, Nick Coghlan wrote: > I agree with Andrew that the names should be equal - consistency is good, > and brevity is a benefit for 'from x import' as well. Yes, and yes. > I'd propose taking a tip from Java and using "py." as the prefix for the > stdlib. Short, to the point, and I'd be surprised if anyone was using a > bare "py" as a package or module name. Works for me! +1. -Fred -- Fred L. Drake, Jr. From rkern at ucsd.edu Sun Oct 10 05:32:22 2004 From: rkern at ucsd.edu (Robert Kern) Date: Sun Oct 10 05:40:56 2004 Subject: [Python-Dev] Re: Guidelines for Logging Usage In-Reply-To: <1097280241.41672af18e6be@mail.iinet.net.au> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> <41671244.5070003@mail.ru> <1097280241.41672af18e6be@mail.iinet.net.au> Message-ID: Nick Coghlan wrote: > I'd propose taking a tip from Java and using "py." as the prefix for the stdlib. > Short, to the point, and I'd be surprised if anyone was using a bare "py" as a > package or module name. Well, there's Py, n?e PyCrust, but it's now buried under the main wxPython package. >>> from wx import py http://www.wxpython.org/py.php -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From tim.peters at gmail.com Sun Oct 10 07:19:19 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 10 07:19:24 2004 Subject: [Python-Dev] assert failure on obmalloc In-Reply-To: References: Message-ID: <1f7befae04100922196d299c3e@mail.gmail.com> [Jeremy Hylton, on Tue, 7 Sep 2004] > Failure running the test suite today with -u compiler enabled on Windows XP. > > test_logging > Assertion failed: bp != NULL, file > \code\python\dist\src\Objects\obmalloc.c, line 604 > > The debugger says the error is here: > msvcr71d.dll!_assert(const char * expr=0x1e22bcc0, const char * > filename=0x1e22bc94, unsigned int lineno=604) Line 306 C > python24_d.dll!PyObject_Malloc(unsigned int nbytes=100) Line 604 + 0x1b C > python24_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=84) Line > 1014 + 0x9 C > python24_d.dll!PyThreadState_New(_is * interp=0x00951028) Line 136 + 0x7 C > python24_d.dll!PyGILState_Ensure() Line 430 + 0xc C > python24_d.dll!t_bootstrap(void * boot_raw=0x02801d48) Line 431 + 0x5 C > python24_d.dll!bootstrap(void * call=0x04f0d264) Line 166 + 0x7 C > msvcr71d.dll!_threadstart(void * ptd=0x026a2320) Line 196 + 0xd C > > I've been seeing this sort of error on-and-off for at least a year > with my Python 2.3 install. It's the usual reason my spambayes > popproxy dies. I can't recell seeing it before on Windows or while > running the test suite. I expect bug 1041645 is relevant. That was suffering the debug-build problem identified later in this thread by Michael Hudson, and also from that the internal TLS function find_key() was thread-insane (regardless of build type). Those should all be fixed now, so gripe if you see this again after updating. BTW, those are all critical bugfixes, but I don't have time to backport them. If anyone does, there was one checkin in the chain that changed when PyGILState_Release() deleted its TLS key, but I'm pretty sure there was no actual need for that change. From fredrik at pythonware.com Sun Oct 10 08:42:01 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun Oct 10 08:40:02 2004 Subject: [Python-Dev] Re: Re: Re: subprocess - updated popen5 module References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: Guido wrote: > I don't know. Writing Windows interface modules is a highly > specialized form of torture, requiring arcane expertise. Despite that, some of us can do this in our sleep (including writing the bindings). I find it somewhat sad that some people don't want us to contribute to Python. While we have you on line, can you pronounce on the subprocess module? The code is written by highly experienced programmers, it has been widely tested, it's entirely self-contained, builds and runs under 2.2 and later, has an extensive test suite and a PEP (which can be quickly turned into a library reference section), and replaces several fragile API:s in the current library with a single, well-defined construct. If this isn't good enough for the standard library, I don't see what is. From gjc at inescporto.pt Sun Oct 10 13:41:08 2004 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Sun Oct 10 13:41:20 2004 Subject: [Python-Dev] Patch to make GIL API calls noops in single threaded applications Message-ID: <1097408468.2328.6.camel@emperor> Can someone review my patch[1]? It makes GIL API calls noops in single threaded applications, i.e. if PyEval_InitThreads hasn't been called. It's a very simple patch, but it makes a lot of difference if you want to make a Python extension module thread-aware without incurring an extra (unnecessary) performance penalty in single-threaded programs. Without this in Python, each extension library has to keep track of thread state (enabled or not), and one extension library doesn't work well with another without some explicit synchronization between the two. Please, we should fix this before Python 2.4 final. Regards. [1] https://sourceforge.net/tracker/? func=detail&aid=1011380&group_id=5470&atid=105470 -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic From lunz at falooley.org Sun Oct 10 14:49:55 2004 From: lunz at falooley.org (Jason Lunz) Date: Sun Oct 10 14:49:59 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: fredrik@pythonware.com said: > While we have you on line, can you pronounce on the subprocess module? > The code is written by highly experienced programmers, it has been > widely tested, it's entirely self-contained, builds and runs under 2.2 > and later, has an extensive test suite and a PEP (which can be quickly > turned into a library reference section), and replaces several fragile > API:s in the current library with a single, well-defined construct. > If this isn't good enough for the standard library, I don't see what > is. hear, hear! Jason From lunz at falooley.org Sun Oct 10 15:17:41 2004 From: lunz at falooley.org (Jason Lunz) Date: Sun Oct 10 15:17:46 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: pf_moore@yahoo.co.uk said: > Hence on Windows, a command line is the fundamental unit, whereas on > Unix an argument list is fundamental. Yes, you're right. I read up on CreateProcess(), GetCommandLine(), and CommandLineToArgvW() after posting. Interestingly, MS's sample code for CommandLineToArgvW is buggy because of confusion between the two interfaces. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/commandlinetoargvw.asp Also, it can fail. > The biggest problem on Windows is that not all executables use the > Microsoft C runtime. Some use other C runtimes, others parse the > command line directly and don't use argv at all. So why does subprocess use cmdline2list() in the parent on unix to emulate the way a child subprocess might parse the string on windows? (But only if it's written in C, uses the MSC runtime, and uses the argv/argc handed to main() rather than calling GetCommandLine() itself). Why not emulate CommandLineToArgvW()? or something else entirely? I think it would be cleaner not to emulate anything at all. > The unix execv is just *different*. Both the Windows and the Unix > interfaces have capabilities the other doesn't offer. Well, the windows interface is a subset of the unix one. The length of argv on windows is limited to 1. > I think Peter's approach of supporting both forms - a single string as > a command line, and list of strings as an argv list, and converting > both to the more natural OS-native form as needed, is sensible (I > would, I argued for it when he was developing it!) I can see that it's trying to be symmetric and orthogonal, but I don't think that the result is worth it in this case. In what scenario is the use of cmdline2list() really useful? Jason From jasone at canonware.com Sun Oct 10 21:39:58 2004 From: jasone at canonware.com (Jason Evans) Date: Sun Oct 10 21:39:40 2004 Subject: [Python-Dev] Cyclic GC issues Message-ID: <20041010193958.GF341@canonware.com> Since the spring of 2003, I have been developing Crux, which is a computer program for phylogenetic inferencing (bioinformatics) research. In March of 2004, I switched Crux to using Python from having used a different embeddable interpreter. For the most part, I have been very happy with Python, but Python's garbage collector has been a major source of frustration. Below, I describe my trials and tribulations with Python's GC. I also offer some suggestions for changes to Python; if any of the proposed changes receive favorable feedback, I am willing to help develop patches to Python. Naturally, if I am somehow abusing Python, and there are better ways to do things, I'd be happy to hear how to improve Crux. The important aspect of Crux is that it deals with trees. These trees are unrooted (there is no up or down), and multifurcating (nodes can have an arbitrary number of neighboring nodes). Thus, the trees are self-referential, and without the cyclic GC capabilities of Python, there would be little hope of making these trees integrate well with Python. Following is a diagram that illustrates the linkage between various objects for a simple tree. Crux exposes all of the components of the trees as Python objects. All lines in the diagram represent bi-directional references (except for the T-->N line). Every object refers to the tree object; those lines are left out in order to reduce clutter. T: Tree N N N: Node \ / E: Edge R R R: Ring \ / E E \ / R---------R / \ / \ / \ / \ | \ / | | \ / | | T--->N | | | | \ | / \ | / \----R----/ | E | R | N At the C (not Python object) level, the R-E-R construct is actually a set of structures that are allocated/deallocated as a single unit. Edges are *always* connected to two rings, so there's no point in allocating these separately. Also, lone ring objects actually are rings with one member; they refer to themselves (prev and next pointers). That should be enough information to understand the problems I encountered. 1) I originally had lone rings refer to themselves (ob_refcnt started out at 3; 2 self-references and one reference held by the associated edge). This didn't work. It appears that the cyclic GC does not actually calculate the number of live references to objects (references that are reachable by traversing all objects accessible from the root set); instead it assumes that if tp_clear() doesn't drop the reference count to a number that equals the number of references from live objects, there must still be references from live objects. Unfortunately, visit()ing self does not work, so there is no way to convince Python that all references are from unreachable objects. Working around this in Crux requires a lot of extra reference counting complexity, because there are three different cases for reference counts, depending on how many members there are in a ring (1, 2, or 3+ members). 2) This issue is really just a different manifestation of issue (1). At the C (not Python object) level, each node only actually stores a pointer to a single member of the associated ring. Given a single ring member, it is possible to traverse the ring and reach all other ring members. As mentioned in issue (1), the cyclic GC expects tp_traverse() to call visit() once for each reference held. It is not enough for a node to visit() one ring member; it must visit() all ring members, in order for the GC to count how many references are from unreachable objects, versus reachable from the root set. In summary, issues (1) and (2) are due to how the cyclic GC does the "marking" phase of mark/sweep GC. My expectation of mark/sweep GC is that it should be sufficient to assure that all objects reachable from the root set are visit()ed at least once; it should not be important how many times each unreachable object is visit()ed. I don't have a deep enough understanding of the Python interpreter to give a detailed suggestion for improvement. I suspect that determining the root set is not an easy operation; if this is the case, then I think we're stuck with the current design. If determining the root set *is* easy (or even possible), then I would suggest using a standard mark/sweep collector, where unreachable objects are scheduled for destruction. tp_traverse(), tp_clear(), and tp_dealloc() would retain the same structure; the only difference would be the logic that determines which objects can be destroyed. 3) A strange thing can happen when tp_clear() is called. Suppose that an edge object is being cleared, and it drops its references to the associated rings. If ob_refcnt of one of the rings drops to 0 as a result, Python will tp_dealloc() the ring *right* *now*, without ever calling tp_clear() on the ring. That means that I have to keep track of whether tp_clear() has been called on objects, if it is at all important that tp_clear() be called, so that I can manually do so in tp_dealloc(), if necessary. It is in my opinion reasonable to have cleanup code in tp_clear(), with the assumption that it will be called precisely once, but Python makes me do extra work to make sure that this happens. This should be pretty easy to change. A single bit per object is needed to keep track of whether tp_clear() has been called. I think this only needs to be done for object types that support cyclic GC. 4) There is no way to control the order in which objects are tp_dealloc()ed. This is a problem for the R-E-R construct, since at a low level, these objects are always linked together. What I would like to do is defer tp_dealloc() on the edge until after both rings have been deleted. Instead, I am forced to use a reference-counted deletion function. Not calling self->ob_type->tp_free() on the edge in tp_dealloc() until later is not a reasonable option, because this defers deletion of the edge until a later round of garbage collection. This could be addressed in the Python interpreter by paying heed to the return value of tp_dealloc(). If the return value is non-zero, move the object to the end of the list of objects to be destroyed, so that destruction is tried later. This allows the module to impose its own destruction ordering. I look forward to feedback. Thank you, Jason Evans From martin at v.loewis.de Sun Oct 10 23:36:11 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Oct 10 23:36:14 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041010193958.GF341@canonware.com> References: <20041010193958.GF341@canonware.com> Message-ID: <4169AB4B.9030605@v.loewis.de> > 1) I originally had lone rings refer to themselves (ob_refcnt started out > at 3; 2 self-references and one reference held by the associated edge). I would have thought the refcount should have been 4: two self references, plus one from E, plus one from N (maybe I don't understand what make a ring "lone", though). > This didn't work. It appears that the cyclic GC does not actually > calculate the number of live references to objects (references that are > reachable by traversing all objects accessible from the root set); That's correct. It doesn't need to. > instead it assumes that if tp_clear() doesn't drop the reference count > to a number that equals the number of references from live objects, > there must still be references from live objects. That's also correct. tp_clear is called for objects that are not reachable from the root set, anymore, so it *should* break all non-reachable cycles. > Unfortunately, > visit()ing self does not work, so there is no way to convince Python > that all references are from unreachable objects. Wrong. If tp_clear is called, you already know for certain that "self" is garbage. So you need to clear all references - regardless of whether you think you should: trust us, we know what we do. > Working around this in Crux requires a lot of extra reference counting > complexity, because there are three different cases for reference > counts, depending on how many members there are in a ring (1, 2, or 3+ > members). Why is that? Just DECREF every PyObject* in "self", then set the PyObject* to NULL. No need for any additional reference counts. > 2) This issue is really just a different manifestation of issue (1). > > At the C (not Python object) level, each node only actually stores a > pointer to a single member of the associated ring. Given a single ring > member, it is possible to traverse the ring and reach all other ring > members. As mentioned in issue (1), the cyclic GC expects tp_traverse() > to call visit() once for each reference held. It is not enough for a > node to visit() one ring member; it must visit() all ring members, in > order for the GC to count how many references are from unreachable > objects, versus reachable from the root set. Hmmm. Even if a ring member holds only a single reference to another ring member, would not calling visit() on this member in turn invoke visit() on the other one, which would, in turn, invoke visit() on the next one, and so on? > In summary, issues (1) and (2) are due to how the cyclic GC does the > "marking" phase of mark/sweep GC. My expectation of mark/sweep GC is > that it should be sufficient to assure that all objects reachable from > the root set are visit()ed at least once; it should not be important how > many times each unreachable object is visit()ed. Correct. Indeed, cyclic GC makes sure visit is not called more often than needed. > I don't have a deep enough understanding of the Python interpreter to > give a detailed suggestion for improvement. I suspect that determining > the root set is not an easy operation; if this is the case, then I think > we're stuck with the current design. If determining the root set *is* > easy (or even possible), then I would suggest using a standard > mark/sweep collector, where unreachable objects are scheduled for > destruction. That is already the case. tp_clear is the procedure that performs the destruction. > tp_traverse(), tp_clear(), and tp_dealloc() would retain > the same structure; the only difference would be the logic that > determines which objects can be destroyed. I don't see a need for change. The logic that determines the objects to be destroyed is already "precise", determining only unreachable objects. > 3) A strange thing can happen when tp_clear() is called. Suppose that an > edge object is being cleared, and it drops its references to the > associated rings. If ob_refcnt of one of the rings drops to 0 as a > result, Python will tp_dealloc() the ring *right* *now*, without ever > calling tp_clear() on the ring. That means that I have to keep track of > whether tp_clear() has been called on objects, if it is at all important > that tp_clear() be called, so that I can manually do so in tp_dealloc(), > if necessary. It is in my opinion reasonable to have cleanup code in > tp_clear(), with the assumption that it will be called precisely once, > but Python makes me do extra work to make sure that this happens. Your expectation is wrong. tp_clear is not the place to perform cleanup. It is the place to break cycles. tp_dealloc is the place to perform cleanup. > 4) There is no way to control the order in which objects are > tp_dealloc()ed. Correct. Unfortunately, there is no way to change that. For cyclic structures, there is no natural "starting point". Python picks an arbitrary starting point, assuming that the order does not matter. If class objects have an __del__, it assumes that the order does matter, and gives up - putting the objects into gc.garbage. > This could be addressed in the Python interpreter by paying heed to the > return value of tp_dealloc(). If the return value is non-zero, move the > object to the end of the list of objects to be destroyed, so that > destruction is tried later. This allows the module to impose its own > destruction ordering. I don't understand. tp_dealloc does not have a return value, so what does "pay heed" mean? Regards, Martin From tim.peters at gmail.com Mon Oct 11 00:45:00 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 00:45:10 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041010193958.GF341@canonware.com> References: <20041010193958.GF341@canonware.com> Message-ID: <1f7befae04101015454de20b5e@mail.gmail.com> [Jason Evans] > Since the spring of 2003, I have been developing Crux, which is a computer > program for phylogenetic inferencing (bioinformatics) research. In March > of 2004, I switched Crux to using Python from having used a different > embeddable interpreter. For the most part, I have been very happy with > Python, but Python's garbage collector has been a major source of > frustration. Then you're not using it correctly -- it's easy if you are. You didn't show actual code, though, and I don't have time to guess what you might have done. > ... > The important aspect of Crux is that it deals with trees. The details shouldn't matter; Python's cyclic gc handles arbitrary graph structures, including multiple self-loops if you want them. The Python core creates graph structures of boggling complexity, so it might help you to study how core objects implement their tp_traverse and tp_clear methods. They're mostly very simple. Where they're not very simple, it's usually because someone had a silly idea of "optimization" in mind. ... > 1) I originally had lone rings refer to themselves (ob_refcnt started out > at 3; 2 self-references and one reference held by the associated edge). > This didn't work. "Didn't work" leaves the reader guessing about everything. Showing code, and being specific about both what you expected and what you observed, are necessary. python-dev isn't really the right place for that, though. > It appears that the cyclic GC does not actually calculate the number of live > references to objects (references that are reachable by traversing all objects > accessible from the root set); Python's gc has no "root set" concept, so thinking in those terms won't help. It won't grow a root-set concept, either: "foreign" extension modules (like Tcl/Tk, or library packages written in Fortran) can hold what would normally be considered root-set objects for Python, and that's fine. They're not required to cooperate with Python's memory management beyond respecting Python's refcount rules. Forcing modules (via some new API) to identify root-set objects they own would be backward-incompatible (not to mention a major new burden on extension authors and users), so won't happen. > instead it assumes that if tp_clear() doesn't drop the reference count > to a number that equals the number of references from live objects, > there must still be references from live objects. Unfortunately, > visit()ing self does not work, so there is no way to convince Python > that all references are from unreachable objects. Again I can't guess what "does not work" means. But I can assure you it "would work" if I wrote it . It's easy: an object's tp_traverse must call the visit() callback exactly once for each non-NULL PyObject* *directly* (in one step) reachable from the object, and that's all. If self is reachable from self via a refcounted pointer held by self, then self's tp_traverse() implementation must call visit(self, arg). But I think you're getting off track even *caring* what your pointers point at. If some object type T has three Py_Object* pointers pa, pb, pc, then using Python 2.4's Py_VISIT macro, T's tp_traverse should be coded static int T_traverse(T *self, visitproc visit, void *arg) { Py_VISIT(self->pa); Py_VISIT(self->pb); Py_VISIT(self->pc); return 0; } It doesn't matter what pa (etc) points at. If pa points to self, fine; if it doesn't, also fine. > Working around this in Crux requires a lot of extra reference counting > complexity, because there are three different cases for reference > counts, depending on how many members there are in a ring (1, 2, or 3+ > members). Please show code. If, e.g., you have a vector v of length n holding the PyObject* "members", then int i; for (i = 0; i < self->n; ++i) Py_VISIT(self->v[i]); return 0; is an appropriate tp_traverse(). For example, that's exactly what the tp_traverse() for Python's builtin list objects does, and a Python list L can certainly be an element of itself: >>> L = [] >>> L.append(L) >>> L[0] is L True >>> There's no special-casing needed for the number of containees, or for the nature of what they do or don't point at. It's also irrelevant what can be reached *from* the direct containees. > 2) This issue is really just a different manifestation of issue (1). > > At the C (not Python object) level, each node only actually stores a > pointer to a single member of the associated ring. Please show code. If the C object contains exactly one pointer, then its tp_traverse should call visit() exactly once; etc. The correct C-level code has entirely to do with what the C struct contains, and nothing to do with the view presented at the Python level. > Given a single ring member, it is possible to traverse the ring and reach all > other ring members. That's irrelevant to gc. tp_traverse() is only responsible for revealing the objects *directly* reachable from self, or, more precisely, for revealing the PyObject* pointers on which self owns a reference. gc computes the transitive closure itself; tp_traverse() isn't responsible for that. > As mentioned in issue (1), the cyclic GC expects tp_traverse() > to call visit() once for each reference held. Yes. > It is not enough for a node to visit() one ring member; Don't know what it means for "a node to visit()". If the C struct has exactly one pointer, then it is enough for visit() to be invoked exactly once with that pointer. It would be incorrect not to visit() that pointer, and it would be incorrect to visit() more pointers than just that one. > it must visit() all ring members, If and only if for each ring member M, self contains a pointer directly to M. Showing code is much better than English. View your C-level objects as a directed graph. tp_traverse at a node `self` is reponsible for revealing self's outgoing edges, no less and no more than that. > in order for the GC to count how many references are from unreachable > objects, versus reachable from the root set. > > In summary, issues (1) and (2) are due to how the cyclic GC does the > "marking" phase of mark/sweep GC. It's not a mark/sweep collector. > My expectation of mark/sweep GC is You expectation is irrelevant , since it's not a mark/sweep collector. > that it should be sufficient to assure that all objects reachable from > the root set are visit()ed at least once; it should not be important how > many times each unreachable object is visit()ed. Forget roots sets and mark/sweep collectors. If you want to know how Python's gc actually works, read the comments in gcmodule.c's collect() function, and drill down from there as deep as your curiosity takes you. > I don't have a deep enough understanding of the Python interpreter to > give a detailed suggestion for improvement. I suspect that determining > the root set is not an easy operation; if this is the case, then I think > we're stuck with the current design. We are, but I don't feel "stuck" with it -- it's a very good design. > If determining the root set *is* easy (or even possible), then I would suggest > using a standard mark/sweep collector, where unreachable objects are > scheduled for destruction. The root set is impossible to determine without active cooperation from extension modules. Python's gc can infer the existence of roots, but not their identies; it can infer the transitive closure of what's reachable from the root set intersected with the set of objects that have registered with gc, but knows nothing about the roots themselves. > tp_traverse(), tp_clear(), and tp_dealloc() would retain > the same structure; the only difference would be the logic that > determines which objects can be destroyed. > > 3) A strange thing can happen when tp_clear() is called. Suppose that an > edge object is being cleared, and it drops its references to the > associated rings. If ob_refcnt of one of the rings drops to 0 as a > result, Python will tp_dealloc() the ring *right* *now*, without ever > calling tp_clear() on the ring. That depends on how the tp_dealloc() for the ring is coded, so is up to you. The only *necessary* thing for tp_clear() to do is to clear enough pointers to break cycles. Some people do only that much. For example, Python tuples don't even implement tp_clear, because a cycle involving tuples necessarily involves non-tuple objects too. Other people write tp_clear() so that it can be called from their tp_dealloc() function too. That's up to them. If you want to do the latter, it's good to write tp_clear() in such a way that calling it more than once is harmless; e.g., if (self->pa) { Py_DECREF(self->pa); self->pa = NULL; } etc or, using 2.4's Py_CLEAR macro, Py_CLEAR(self->pa); etc. > That means that I have to keep track of whether tp_clear() has been called > on objects, if it is at all important that tp_clear() be called, so that I can > manually do so in tp_dealloc(), As above, make your tp_clear() idempotent and then you don't have to keep track of anything. It's usually good practice for a tp_clear() to be robust against NULL pointers anyway. > if necessary. It is in my opinion reasonable to have cleanup code in > tp_clear(), with the assumption that it will be called precisely once, > but Python makes me do extra work to make sure that this happens. You won't get a guarantee that tp_clear() will be called exactly once. You have a guarantee that tp_dealloc() will be called no more than once. It's actually unusual to find a tp_dealloc() that calls its type's tp_clear(), but I think it's a nice pattern when you can get away with it. In such cases, tp_clear() should be coded so that it can be called multiple times, and that's generally very easy to do. > This should be pretty easy to change. A single bit per object is needed > to keep track of whether tp_clear() has been called. I think this only > needs to be done for object types that support cyclic GC. There's no such thing as "one bit" in reality: because of malloc memory padding, "one more bit" would require adding 4 bytes to every gc'able object. That's an extraordinary expense. Luckily, it's not needed. > 4) There is no way to control the order in which objects are > tp_dealloc()ed. It's true that there's no direct way to control the order. > This is a problem for the R-E-R construct, since at a low level, these objects > are always linked together. Many things are always linked together in the core too. Why that's "a problem" for you isn't clear. It's not a problem in the core, for example. > What I would like to do is defer tp_dealloc() on the edge until after both > rings have been deleted. Instead, I am forced to use a reference-counted > deletion function. Sorry, don't know what that means, and (as above) don't know why you care. > Not calling self->ob_type->tp_free() on the edge in tp_dealloc() until > later is not a reasonable option, because this defers deletion of the > edge until a later round of garbage collection. > > This could be addressed in the Python interpreter by paying heed to the > return value of tp_dealloc(). If the return value is non-zero, move the > object to the end of the list of objects to be destroyed, Except there is no list of objects to be destroyed. > so that destruction is tried later. This allows the module to impose its own > destruction ordering. That part would need clearer motivation, preferably in the form of a PEP. From tim.peters at gmail.com Mon Oct 11 01:18:51 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 01:19:00 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <4169AB4B.9030605@v.loewis.de> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> Message-ID: <1f7befae04101016183eeb535d@mail.gmail.com> [Martin v. L?wis] > Hmmm. Even if a ring member holds only a single reference to another > ring member, would not calling visit() on this member in turn invoke > visit() on the other one, which would, in turn, invoke visit() on > the next one, and so on? Yes, but not recursively (which that description appears to imply), and not because of what's passed *to* visit(). More precisely, visit(some_pointer) does different things during different phases of gc (different phases pass different gcmodule.c callback functions to tp_traverse() implementations). None of them lead to recursive calls to the callback. Maybe a bit ironically, refcount-based deletion can require unbounded stack depth (hence the "trashcan" mechanism to avoid blowing the C stack), but Python's cyclic gc isn't recursive at all. I'm guessing that Jason is confusing the update_refs() and subtract_refs() phases of Python's gc with the "mark" phase of a mark/sweep collector. If so, the visit() callbacks used by those are actually irrelevant to the set of objects whose tp_traverse() is called during those phases. The tp_traverse() of *every* object in the generation being collected is invoked by that pair of phases, and the set of objects passed *to* the visit callbacks during those phases has no effect on the set of objects whose tp_traverse() gets called during those phases. So one of Jason's objects will get its tp_traverse() called by those phases if and only if it's in the generation currently being collected, and it's currently tracked by gc. It doesn't matter whether, e.g., anything points at it, or whether it points at anything (although if nothing anywhere points at it, it would already have been destroyed via refcounting). From theller at python.net Mon Oct 11 08:51:16 2004 From: theller at python.net (Thomas Heller) Date: Mon Oct 11 08:51:00 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: (Thomas Heller's message of "Fri, 08 Oct 2004 16:30:49 +0200") References: Message-ID: "Fredrik Lundh" writes: > Thomas Heller wrote: > >> How many _xxxx.pyd windows specific modules do we need? > > as many as it takes to make Python an excellent tool. it's not like one > more entry in the makefile, and 20k in the binary distribution will cause > problem for anyone. > > if you want to stop contributions to Python's standard library unless they > use your tools, you have a serious problem. Sorry if you misunderstood me, I didn't want to stop this contribution. Here is what I wrote, again: > I would suggest to replace the _subprocess extension module by a Python > implementation based on ctypes (and include ctypes in 2.4, at least for > Windows). > Thomas From jasone at canonware.com Mon Oct 11 08:59:52 2004 From: jasone at canonware.com (Jason Evans) Date: Mon Oct 11 08:59:30 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <4169AB4B.9030605@v.loewis.de> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> Message-ID: <20041011065952.GH341@canonware.com> On Sun, Oct 10, 2004 at 11:36:11PM +0200, "Martin v. L?wis" wrote: > >1) I originally had lone rings refer to themselves (ob_refcnt started out > > at 3; 2 self-references and one reference held by the associated edge). > > I would have thought the refcount should have been 4: two self > references, plus one from E, plus one from N (maybe I don't understand > what make a ring "lone", though). You understand correctly, except that the R-E-R construct starts out not being attached to any nodes. The "lone" ring term was my attempt to differentiate this: R from these: R-R R---R R-R \ / | | R R-R > > Working around this in Crux requires a lot of extra reference counting > > complexity, because there are three different cases for reference > > counts, depending on how many members there are in a ring (1, 2, or 3+ > > members). > > Why is that? Just DECREF every PyObject* in "self", then set the > PyObject* to NULL. No need for any additional reference counts. The low level tree code is implemented completely separately from the Python object wrapper code. This means that, for example, the Python "Node" object does not actually store PyObject* pointers to Ring objects; instead it uses the low level tree code to find out what Ring objects are attached to it. Crux was designed this way in order to be able to implement various low level algorithms such that they never have to call interpreter-related code. As such, reference breaking code must actually tear down the low level tree in such a way that it is always "consistent" (simply setting pointers to NULL doesn't fit well with this way of doing things). > Hmmm. Even if a ring member holds only a single reference to another > ring member, would not calling visit() on this member in turn invoke > visit() on the other one, which would, in turn, invoke visit() on > the next one, and so on? Yes, this is true. The problem I ran into had to do with the node logically holding references to all the ring objects, but only reporting one of them. That is, the node's tp_traverse function was not reporting all references, though it was reporting enough of them to guarantee that all reachable objects were visited. At this point, it is clear to me that Python's cyclic GC does not do mark/sweep GC, and that this won't change, so there's probably not much point in discussing this issue further. > >3) A strange thing can happen when tp_clear() is called. Suppose that an > > edge object is being cleared, and it drops its references to the > > associated rings. If ob_refcnt of one of the rings drops to 0 as a > > result, Python will tp_dealloc() the ring *right* *now*, without ever > > calling tp_clear() on the ring. That means that I have to keep track of > > whether tp_clear() has been called on objects, if it is at all important > > that tp_clear() be called, so that I can manually do so in tp_dealloc(), > > if necessary. It is in my opinion reasonable to have cleanup code in > > tp_clear(), with the assumption that it will be called precisely once, > > but Python makes me do extra work to make sure that this happens. > > Your expectation is wrong. tp_clear is not the place to perform cleanup. > It is the place to break cycles. tp_dealloc is the place to perform > cleanup. What if breaking cycles and cleanup are the same thing? In the case of Crux, they are one and the same, since the low level tree must be torn down one piece at a time. Apparently I was mistaken in viewing tp_clear and tp_dealloc as two stages of destruction. > >4) There is no way to control the order in which objects are > > tp_dealloc()ed. > > Correct. Unfortunately, there is no way to change that. For cyclic > structures, there is no natural "starting point". Python picks an > arbitrary starting point, assuming that the order does not matter. > If class objects have an __del__, it assumes that the order does > matter, and gives up - putting the objects into gc.garbage. > > > This could be addressed in the Python interpreter by paying heed to the > > return value of tp_dealloc(). If the return value is non-zero, move the > > object to the end of the list of objects to be destroyed, so that > > destruction is tried later. This allows the module to impose its own > > destruction ordering. > > I don't understand. tp_dealloc does not have a return value, so what > does "pay heed" mean? I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), the clear() call does not pay attention to the return value. It would be possible to check the return value, and defer deallocation if the return value were non-zero, and ob_refcnt were 1. (However, it would probably have to be a separate list of objects, contrary to what I claimed before.) Thanks for your comments, Martin. I'm starting to understand the Python GC world view a bit better. Jason From fredrik at pythonware.com Mon Oct 11 09:13:23 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon Oct 11 09:11:22 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: Thomas Heller wrote: >> if you want to stop contributions to Python's standard library unless they >> use your tools, you have a serious problem. > > Sorry if you misunderstood me, I didn't want to stop this contribution. > Here is what I wrote, again: > >> I would suggest to replace the _subprocess extension module by a Python >> implementation based on ctypes (and include ctypes in 2.4, at least for >> Windows). if you're suggesting a rewrite for political reasons, you're effectively blocking this contribution. don't do that; python-dev have enough problems getting non- core contributions as it is. From theller at python.net Mon Oct 11 09:26:35 2004 From: theller at python.net (Thomas Heller) Date: Mon Oct 11 09:26:20 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module In-Reply-To: (Guido van Rossum's message of "Sat, 9 Oct 2004 16:18:29 -0700") References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: <8yadd8b8.fsf@python.net> Guido van Rossum writes: > And what about all the stuff that's defined in header files? In the > past, in the Unix world, I experimented with translating .h files to > Python modules to save me from the burden of having to add large > numbers of symbol definitions to C extensions. (Look for files named > Lib/plat-XXX/regen in the source tree.) One by one, that approach has > proven to be problematic, and nearly all of those have eventually been > turned into systematic lists of symbol definitions in C code. See for > example the posix, socket, fcntl, and signal modules. I have two solutions here, both based on gccxml (see http://www.gccxml.org/). This is the C++ parser part of gcc, with a backend that turns the declaration in C header files into xml. gccxml runs on windows too, and since the installer even does some small patches to the MS header files to make them compliant, it parses the windows H files without a problem. You can even request it to simulate MSVC 6, MSVC 7.0, or MSVC 7.1. Now I have written an XML parser/Python code generator combo which generates a valid Python module from header files. This includes enums, structures, unions, and exported functions, but not #define symbol definitions, of course. You could even specify the symbols you want to be generated, it handles dependencies correctly and will include all that is needed. For the #define symbols, I let gccxml dump the preprocessor definitions after running it on the header files. This is to find the defined names. The actual code generation into Python code is done by a set of C++ function templates. This works better than Tools/scripts/h2py.py. Both things are still work in progress, but they do work. Thomas From astrand at lysator.liu.se Mon Oct 11 09:50:04 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Mon Oct 11 10:14:59 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: On Sun, 10 Oct 2004, Jason Lunz wrote: > > The biggest problem on Windows is that not all executables use the > > Microsoft C runtime. Some use other C runtimes, others parse the > > command line directly and don't use argv at all. > > So why does subprocess use cmdline2list() in the parent on unix to > emulate the way a child subprocess might parse the string on windows? > (But only if it's written in C, uses the MSC runtime, and uses the > argv/argc handed to main() rather than calling GetCommandLine() itself). > Why not emulate CommandLineToArgvW()? or something else entirely? I > think it would be cleaner not to emulate anything at all. One goal with subprocess is being able to write cross-platform applications. For example, it should be possible to open up www.python.org in Mozilla. The best way to write this is: subprocess.call(["mozilla", "http://www.python.org"]) In this case, the list form is translated to the string form when running on Windows. Why allow the string form on UNIX? Answer: Symmetri, plus that some users that has been using os.system() for a long time thinks it's nice to be able to do: subprocess.call("ls -l /foo/bar") There's a risk that UNIX users might expect UNIX shell-like quoting support rather than the MSC one, though. > > The unix execv is just *different*. Both the Windows and the Unix > > interfaces have capabilities the other doesn't offer. > > Well, the windows interface is a subset of the unix one. The length of > argv on windows is limited to 1. True, if we are talking about the UNIX exec functions. When executing through a UNIX shell, the native interface is a string. > > I think Peter's approach of supporting both forms - a single string as > > a command line, and list of strings as an argv list, and converting > > both to the more natural OS-native form as needed, is sensible (I > > would, I argued for it when he was developing it!) > > I can see that it's trying to be symmetric and orthogonal, but I don't > think that the result is worth it in this case. In what scenario is the > use of cmdline2list() really useful? I don't really have a good example. If we should remove the cmdline2list stuff, what should happen if the users passes a string on UNIX? Do you prefer: 1) Run through the shell (automatic shell=True). or 2) A ValueError raised. I guess alternative 1 is most intuitive. That would line up with popen2 as well. Anyone objecting to this change? /Peter ?strand From anthony at interlink.com.au Mon Oct 11 15:09:49 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Oct 11 15:10:18 2004 Subject: [Python-Dev] compiler bug #1042238 is blocking the 2.4b1 release Message-ID: <416A861D.5020804@interlink.com.au> Anyone with knowledge of Lib/compiler is urged to examine www.python.org/sf/1042238. It's breaking on some genexprs, in particular in Lib/_strptime.py The failing code, distilled down, is: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'', 'exec') It's the combination of the genexps and the function call with another argument. This bug is bad enough (IMHO) to block 2.4b1. Please, if you know about this, have a look at the failure. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From lunz at falooley.org Mon Oct 11 16:57:07 2004 From: lunz at falooley.org (Jason Lunz) Date: Mon Oct 11 16:57:16 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: Message-ID: astrand@lysator.liu.se said: > One goal with subprocess is being able to write cross-platform > applications. For example, it should be possible to open up www.python.org > in Mozilla. The best way to write this is: > > subprocess.call(["mozilla", "http://www.python.org"]) > > In this case, the list form is translated to the string form when running > on Windows. understood. I personally have a definite need for this in my programs, so I know what's involved. > There's a risk that UNIX users might expect UNIX shell-like quoting > support rather than the MSC one, though. exactly. I just see it confusing people. If someone wants simple string handling on unix, then shell=True. problem solved. If they need the type of portability referred to with your mozilla example, then they should use the list form and let windows do list2cmdline() on it. The converse case of using a string for windows and using cmdline2list() on it for unix is a good try, but there's no guarantee that the actual windows subprocess will even do the MSVCRT-compatible-cmdline2list() conversion. Which leaves you in a very confusing situation indeed. >> I can see that it's trying to be symmetric and orthogonal, but I don't >> think that the result is worth it in this case. In what scenario is the >> use of cmdline2list() really useful? > > I don't really have a good example. > > If we should remove the cmdline2list stuff, what should happen if the > users passes a string on UNIX? Do you prefer: > > 1) Run through the shell (automatic shell=True). > or > 2) A ValueError raised. > > I guess alternative 1 is most intuitive. That would line up with > popen2 as well. Use of the shell should be explicit, not automatic, because of the usual shell metacharacter security concerns. unix programmers used to doing os.system('ls -l') will quickly learn that the subprocess way of doing the same is subprocess.call('ls -l', shell=True). This has the added benifit of making it obvious exactly what's happening. I don't think that the only alternative to number 1) is to raise a ValueError. What do you think of the below patch? Just listify bare strings on unix. This does exactly what it should when the string actually references a binary, and gives a meaningful error when it doesn't. Even if the filename has strange characters of some kind. $ echo '#! /bin/sh' > 'ls -l' $ echo 'echo foo' >> 'ls -l' $ cat 'ls -l' #! /bin/sh echo foo $ python Python 2.3.4 (#2, Sep 24 2004, 08:39:09) >>> from subprocess import call >>> call('./ls -l') Traceback (most recent call last): File "", line 1, in ? File "subprocess.py", line 441, in call return Popen(*args, **kwargs).wait() File "subprocess.py", line 524, in __init__ errread, errwrite) File "subprocess.py", line 942, in _execute_child raise child_exception OSError: [Errno 13] Permission denied >>> $ chmod +x 'ls -l' $ python Python 2.3.4 (#2, Sep 24 2004, 08:39:09) >>> from subprocess import call >>> call('./ls -l') foo 0 >>> call('./ls -blah') Traceback (most recent call last): File "", line 1, in ? File "subprocess.py", line 441, in call return Popen(*args, **kwargs).wait() File "subprocess.py", line 524, in __init__ errread, errwrite) File "subprocess.py", line 942, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory Nice, straightforward, easy to understand. And look how much code is removed -- I haven't even cleaned up the corresponding docs yet. Index: subprocess.py =================================================================== RCS file: /cvsroot/python-popen5/popen5/subprocess.py,v retrieving revision 1.15 diff -u -p -r1.15 subprocess.py --- subprocess.py 9 Oct 2004 10:11:06 -0000 1.15 +++ subprocess.py 11 Oct 2004 14:56:26 -0000 @@ -854,11 +854,11 @@ class Popen: errread, errwrite): """Execute program (POSIX version)""" + if isinstance(args, types.StringTypes): + args = [args] + if shell: args = ["/bin/sh", "-c"] + args - else: - if isinstance(args, types.StringTypes): - args = self.cmdline2list(args) if executable == None: executable = args[0] @@ -1051,93 +1051,6 @@ class Popen: self.wait() return (stdout, stderr) - - - def cmdline2list(cmdline): - """ - Translate a command line string into a list of arguments, using - using the same rules as the MS C runtime: - - 1) Arguments are delimited by white space, which is either a - space or a tab. - - 2) A string surrounded by double quotation marks is - interpreted as a single argument, regardless of white space - contained within. A quoted string can be embedded in an - argument. - - 3) A double quotation mark preceded by a backslash is - interpreted as a literal double quotation mark. - - 4) Backslashes are interpreted literally, unless they - immediately precede a double quotation mark. - - 5) If backslashes immediately precede a double quotation mark, - every pair of backslashes is interpreted as a literal - backslash. If the number of backslashes is odd, the last - backslash escapes the next double quotation mark as - described in rule 3. - """ - - # See - # http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp - - # Step 1: Translate all literal quotes into QUOTE. Justify number - # of backspaces before quotes. - tokens = [] - bs_buf = "" - QUOTE = 1 # \", literal quote - for c in cmdline: - if c == '\\': - bs_buf += c - elif c == '"' and bs_buf: - # A quote preceded by some number of backslashes. - num_bs = len(bs_buf) - tokens.extend(["\\"] * (num_bs//2)) - bs_buf = "" - if num_bs % 2: - # Odd. Quote should be placed literally in array - tokens.append(QUOTE) - else: - # Even. This quote serves as a string delimiter - tokens.append('"') - - else: - # Normal character (or quote without any preceding - # backslashes) - if bs_buf: - # We have backspaces in buffer. Output these. - tokens.extend(list(bs_buf)) - bs_buf = "" - - tokens.append(c) - - # Step 2: split into arguments - result = [] # Array of strings - quoted = False - arg = [] # Current argument - tokens.append(" ") - for c in tokens: - if c == '"': - # Toggle quote status - quoted = not quoted - elif c == QUOTE: - arg.append('"') - elif c in (' ', '\t'): - if quoted: - arg.append(c) - else: - # End of argument. Output, if anything. - if arg: - result.append(''.join(arg)) - arg = [] - else: - # Normal character - arg.append(c) - - return result - - cmdline2list = staticmethod(cmdline2list) def list2cmdline(seq): From seojiwon at gmail.com Mon Oct 11 17:01:24 2004 From: seojiwon at gmail.com (Jiwon Seo) Date: Mon Oct 11 17:02:05 2004 Subject: [Python-Dev] compiler bug #1042238 is blocking the 2.4b1 release In-Reply-To: <416A861D.5020804@interlink.com.au> References: <416A861D.5020804@interlink.com.au> Message-ID: Uploaded patch for the bug. The bug was due to a wrong condition check for allowing f(x for x in y) but rejecting f(x for x in y, 1). The patch fixes the bug alright, but you might want to test more. =) Jiwon. On Mon, 11 Oct 2004 23:09:49 +1000, Anthony Baxter wrote: > Anyone with knowledge of Lib/compiler is urged to examine > www.python.org/sf/1042238. It's breaking on some genexprs, > in particular in Lib/_strptime.py > > The failing code, distilled down, is: > > import compiler > a="""def foo(*x): return x > foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" > compiler.compile(a,'', 'exec') > > It's the combination of the genexps and the function call > with another argument. > > This bug is bad enough (IMHO) to block 2.4b1. Please, > if you know about this, have a look at the failure. > > Anthony > > -- > Anthony Baxter > It's never too late to have a happy childhood. > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/seojiwon%40gmail.com > From mwh at python.net Mon Oct 11 17:12:53 2004 From: mwh at python.net (Michael Hudson) Date: Mon Oct 11 17:12:55 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS Message-ID: <2mfz4lgufe.fsf@starship.python.net> As subject. Is this deliberate? It breaks test_compiler, though that's possibly a bug because test_compiler should be using universal newlines modes when opening the file, but still... Cheers, mwh -- The word "Fascism" has now no meaning except in so far as it signifies 'something not desirable'. -- George Orwell in "Politics and the English Language" From gvanrossum at gmail.com Mon Oct 11 17:17:51 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 11 17:17:56 2004 Subject: [Python-Dev] Re: Re: Re: subprocess - updated popen5 module In-Reply-To: References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: [Guido] > > I don't know. Writing Windows interface modules is a highly > > specialized form of torture, requiring arcane expertise. [Fredrik] > Despite that, some of us can do this in our sleep (including writing the bindings). > I find it somewhat sad that some people don't want us to contribute to Python. Hey, I wrote that in *support* of you! > While we have you on line, can you pronounce on the subprocess module? The > code is written by highly experienced programmers, it has been widely tested, > it's entirely self-contained, builds and runs under 2.2 and later, has an extensive > test suite and a PEP (which can be quickly turned into a library reference section), > and replaces several fragile API:s in the current library with a single, well-defined > construct. If this isn't good enough for the standard library, I don't see what is. I expect it is, and I value your judgement. I have recently experimented with a cross-platform process management abstraction myself, and I'd like to check out the subprocess module to make sure that it can do everything that I put in my own code (not very much but some specific requirements about I/O redirection, current directory, and setting the priority. I hope to get back to this later today. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Mon Oct 11 17:21:10 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 17:21:18 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <2mfz4lgufe.fsf@starship.python.net> References: <2mfz4lgufe.fsf@starship.python.net> Message-ID: <1f7befae04101108212f0a528@mail.gmail.com> [Michael Hudson] > As subject. Is this deliberate? Don't know, but guess so: it contains bytes outside the set ANSI C says can be used portably in text files. > It breaks test_compiler, That must be platform-specific damage, but you haven't identified your platform. > though that's possibly a bug because test_compiler should be using universal > newlines modes when opening the file, Probably, yes. > but still... So stop whining and fix it . From mwh at python.net Mon Oct 11 17:25:21 2004 From: mwh at python.net (Michael Hudson) Date: Mon Oct 11 17:25:23 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <1f7befae04101108212f0a528@mail.gmail.com> (Tim Peters's message of "Mon, 11 Oct 2004 11:21:10 -0400") References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> Message-ID: <2m8yadgtum.fsf@starship.python.net> Tim Peters writes: > [Michael Hudson] >> As subject. Is this deliberate? > > Don't know, but guess so: it contains bytes outside the set ANSI C > says can be used portably in text files. > >> It breaks test_compiler, > > That must be platform-specific damage, but you haven't identified your platform. Well, yes, but this information isn't especially relavent. It's obviously a different platform from whoever edited the file last... I'm on linux, though. >> though that's possibly a bug because test_compiler should be using universal >> newlines modes when opening the file, > > Probably, yes. > >> but still... > > So stop whining and fix it . I have done and am waiting for test_compiler to finish ... that allows for a lot of whining :) Cheers, mwh -- There are two kinds of large software systems: those that evolved from small systems and those that don't work. -- Seen on slashdot.org, then quoted by amk From sjoerd at acm.org Mon Oct 11 18:00:03 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Mon Oct 11 18:00:39 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <1f7befae04101108212f0a528@mail.gmail.com> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> Message-ID: <416AAE03.9020209@acm.org> Tim Peters wrote: > [Michael Hudson] > >>As subject. Is this deliberate? > > > Don't know, but guess so: it contains bytes outside the set ANSI C > says can be used portably in text files. However, this is not necessarily enough reason to use -kb. The only things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for updates, and not expanding $ keywords. None of those matter in this file. > >>It breaks test_compiler, > > > That must be platform-specific damage, but you haven't identified your platform. > > >>though that's possibly a bug because test_compiler should be using universal >>newlines modes when opening the file, > > > Probably, yes. > > >>but still... > > > So stop whining and fix it . > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sjoerd%40acm.org -- Sjoerd Mullender -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 374 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041011/cb1df212/signature-0001.pgp From tim.peters at gmail.com Mon Oct 11 19:32:34 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 19:32:55 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416AAE03.9020209@acm.org> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> <416AAE03.9020209@acm.org> Message-ID: <1f7befae04101110325f8f0189@mail.gmail.com> [Sjoerd Mullender] > However, this is not necessarily enough reason to use -kb. The only > things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for > updates, and not expanding $ keywords. None of those matter in this file. So fix it. From nbastin at opnet.com Mon Oct 11 21:14:52 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Oct 11 21:15:14 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041011065952.GH341@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> Message-ID: On Oct 11, 2004, at 2:59 AM, Jason Evans wrote: >>> Working around this in Crux requires a lot of extra reference >>> counting >>> complexity, because there are three different cases for reference >>> counts, depending on how many members there are in a ring (1, 2, >>> or 3+ >>> members). >> >> Why is that? Just DECREF every PyObject* in "self", then set the >> PyObject* to NULL. No need for any additional reference counts. > > The low level tree code is implemented completely separately from the > Python object wrapper code. This means that, for example, the Python > "Node" object does not actually store PyObject* pointers to Ring > objects; > instead it uses the low level tree code to find out what Ring objects > are > attached to it. Crux was designed this way in order to be able to > implement various low level algorithms such that they never have to > call > interpreter-related code. As such, reference breaking code must > actually > tear down the low level tree in such a way that it is always > "consistent" > (simply setting pointers to NULL doesn't fit well with this way of > doing > things). You've now reached the point where the Python-GC-ease-of-implementation breaks down. We have encountered the same problem - the GC is fine as long as you're solely extending Python, but if you're embedding it, chances are you'll encounter some interesting issues like this along the way. It's not that you can't make it work - you can, but need to do a lot more work yourself. Basically, if I understand you correctly (and I may not be), there are times when you do not want tp_dealloc to actually destroy the low-level data structure, because somebody else (not living in the Python reference-counted world) isn't done with it yet. The problem is in knowing when those times are, and when they aren't. In this case, you may need to implement basic reference counting in your own low-level data structures, so you know when you're really done with a ring and ready to destroy it. The other option is to attempt to make sure that you're not creating cycles (in python) and avoid interfacing with GC entirely, but that may not be possible in your case. -- Nick From martin at v.loewis.de Mon Oct 11 22:50:21 2004 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon Oct 11 22:50:22 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416AAE03.9020209@acm.org> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> <416AAE03.9020209@acm.org> Message-ID: <416AF20D.6010706@v.loewis.de> Sjoerd Mullender wrote: >> Don't know, but guess so: it contains bytes outside the set ANSI C >> says can be used portably in text files. > > > However, this is not necessarily enough reason to use -kb. The only > things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for > updates, and not expanding $ keywords. That is not true. On Apple computers, it also avoids conversion from Latin-1 to Mac-Roman, which Mac CVS does by default for text files. Making the files binary is the only way to avoid this conversion, and that is precisely the reason why the file is binary. You may argue that this is a bug in Mac CVS, and I would agree. However, that specific bug has -kb as a known work-around, and the issue reported here points to a bug in the compiler packages which should be fixed rather than worked-around. Regards, Martin From astrand at lysator.liu.se Mon Oct 11 22:41:38 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Mon Oct 11 23:04:01 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: On Mon, 11 Oct 2004, Jason Lunz wrote: > > If we should remove the cmdline2list stuff, what should happen if the > > users passes a string on UNIX? Do you prefer: > > > > 1) Run through the shell (automatic shell=True). > > or > > 2) A ValueError raised. > > > > I guess alternative 1 is most intuitive. That would line up with > > popen2 as well. > > Use of the shell should be explicit, not automatic, because of the usual > shell metacharacter security concerns. unix programmers used to doing > os.system('ls -l') will quickly learn that the subprocess way of doing > the same is subprocess.call('ls -l', shell=True). This has the added > benifit of making it obvious exactly what's happening. Good points. > I don't think that the only alternative to number 1) is to raise a ValueError. > > What do you think of the below patch? Just listify bare strings on unix. This > does exactly what it should when the string actually references a binary, and > gives a meaningful error when it doesn't. Even if the filename has strange > characters of some kind. I like it. I've submitted your patch (but with documentation updates as well). /Peter ?strand From gvanrossum at gmail.com Mon Oct 11 23:18:49 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 11 23:18:52 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. That > is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be done? FWIW, I've given Peter some feedback, but in general I think this code is ready for inclusion into 2.4. I hope it's not too late. (My recommendation on the call/callv issue is to get rid of callv(), and instead let call() interpret multiple string args as such; a single string arg will be interpreted as a whole command line, per Popen(); if you want a single file arg you have to enclose it in [...].) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Tue Oct 12 00:34:58 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Oct 12 00:48:00 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: References: Message-ID: <20041011152210.08C1.JCARLSON@uci.edu> [snip Guido's mention of subprocess being included in Python 2.4, if at all possible] Speaking of inclusion, we seemed to have come upon a stalemate with the 'gG' format code additions. Some users (without commit access to Python CVS) have stated they would use it (and currently write their own packing and unpacking functions), but some developers (with commit access to Python CVS) have stated they would never use it. At that point, the discussion pretty much ended. I know that the request is coming in a bit late in the 2.4 development cycle, but waiting until Python 2.5 seems like a bit much, considering that the functionality is so simple it can be proven correct quite easily (if it is to be included). A pronouncement would be nice, but I understand that I am in no position to ask/beg for a (yes, we'll add it) response. <.5 wink> Thank you, - Josiah P.S. Note that everyone involved believes that binascii.from/to_long should be added (I think it was even pronounced by Guido). From gvanrossum at gmail.com Tue Oct 12 01:12:38 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Oct 12 01:12:40 2004 Subject: [Python-Dev] Re: Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011152210.08C1.JCARLSON@uci.edu> References: <20041011152210.08C1.JCARLSON@uci.edu> Message-ID: > [snip Guido's mention of subprocess being included in Python 2.4, if at > all possible] > > Speaking of inclusion, we seemed to have come upon a stalemate with the > 'gG' format code additions. Some users (without commit access to Python > CVS) have stated they would use it (and currently write their own > packing and unpacking functions), but some developers (with commit > access to Python CVS) have stated they would never use it. > > At that point, the discussion pretty much ended. > > I know that the request is coming in a bit late in the 2.4 development > cycle, but waiting until Python 2.5 seems like a bit much, considering > that the functionality is so simple it can be proven correct quite > easily (if it is to be included). > > A pronouncement would be nice, but I understand that I am in no position > to ask/beg for a (yes, we'll add it) response. <.5 wink> > > Thank you, > - Josiah > > P.S. > Note that everyone involved believes that binascii.from/to_long should > be added (I think it was even pronounced by Guido). I'm in favor of adding the latter only. Is there a patch? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Tue Oct 12 01:36:32 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 12 01:37:34 2004 Subject: [Python-Dev] Re: Struct module format code addition (previously was in the subprocess thread) In-Reply-To: References: <20041011152210.08C1.JCARLSON@uci.edu> Message-ID: <1f7befae04101116361b3a5185@mail.gmail.com> [Josiah Carlson] >> Note that everyone involved believes that binascii.from/to_long should >> be added (I think it was even pronounced by Guido). [Guido] > I'm in favor of adding the latter only. Is there a patch? No. It would be easy enough, but nobody had time for it. AFAICT, not even the signatures for such functions have been fully worked out yet. Since it would be a small and self-contained addition, I wouldn't worry about adding it after the beta. I wouldn't worry much more about adding g/G after either, for that matter. From tim.peters at gmail.com Tue Oct 12 03:21:57 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 12 03:22:07 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041011065952.GH341@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> Message-ID: <1f7befae041011182151f7b219@mail.gmail.com> [Jason Evans] ... > The low level tree code is implemented completely separately from the > Python object wrapper code. This means that, for example, the Python > "Node" object does not actually store PyObject* pointers to Ring objects; > instead it uses the low level tree code to find out what Ring objects are > attached to it. I think you're in for a rude awakening if you expect any language's gc support to help you manage arbitrary blobs of memory obtained from the system malloc(). If all the nodes in your object graph derived from PyObject, Python's gc would do it all for you. If you want to plant pointers in and among arbitrary C structs, then you'll have to implement your own gc, or hope that you can bash a general gc thingie (like BDW -- and I'm not sure there is anything else of that sort usefully available, so "like" is probably optimistic) into guessing the layout of your structs. > Crux was designed this way in order to be able to implement various low level > algorithms such that they never have to call interpreter-related code. I don't know why that was thought to be desirable, or even exactly what it means. One plausible meaning is that you don't want to tie the "various low level algorithms" to Python, because you hope to reuse them in other, non-Python contexts. And, in addition to that, you don't want, or don't know how, to abstract the necessary operations into macros or functions that are pluggable for the different contexts you have in mind. As a very simple example, using Py_INCREF in a function doesn't tie that function to Python -- if you want to use it in a non-Python context, #define Py_INCREF(x) gets rid of such calls. But I'm guessing too much about what you might mean, so I'll stop there. > As such, reference breaking code must actually tear down the low level tree in > such a way that it is always "consistent" (simply setting pointers to NULL > doesn't fit well with this way of doing things). Martin had in mind that the pointers were to properly refcounted PyObject*, in which case his advice was sound. [Martin] >> Hmmm. Even if a ring member holds only a single reference to another >> ring member, would not calling visit() on this member in turn invoke >> visit() on the other one, which would, in turn, invoke visit() on >> the next one, and so on? [Jason] > Yes, this is true. It's false, but I covered that one yesterday. What's passed back to visit() has nothing to do with which objects' tp_traverse()s get called. Registering ("tracking") PyObject*s of gc'able types is what determines which objects' tp_traverse()s get called. Python's gc doesn't try to determine what's reachable, it determines what isn't reachable from pointers it *hasn't* been told about. It infers "reachable from outside" from the refcounts: if all the refcounts on an object are not accounted for by the objects gc *has* been told about, then that object is directly reachable from something gc hasn't been told about, so gc dare not touch it. The transitive closure of what's reachable from such objects is the set of all objects gc knows about that are reachable from outside. gc leaves those alone. Everything else gc knows about is necessarily cyclic trash, or reachable only from cyclic trash. It can't know anything about objects that aren't tracked by gc, so in particular can't know anything about pointers that don't point at a PyObject. > The problem I ran into had to do with the node logically holding references to all > the ring objects, but only reporting one of them. If a ring object doesn't derive from PyObject, then visit() must not be passed a pointer to a ring object. Doing so would account for many catastrophic meanings of "doesn't work" . > That is, the node's tp_traverse function was not reporting all references, though > it was reporting enough of them to guarantee that all reachable objects were > visited. No, the visit() callback has nothing to do with which objects are traversed. gc determines the set of objects to traverse, before any object's tp_traverse() has been called, as a "generational" subset of all objects that have registered with gc. The pointers passed back to visit() callbacks have no effect on that set: they can't add anything to, or remove anything from, that set. The set of objects to traverse is predetermined. > At this point, it is clear to me that Python's cyclic GC does not do > mark/sweep GC, RIght, although we're only talking about CPython here (a particular implementation of Python), > and that this won't change, In CPython that's a safe bet. > so there's probably not much point in discussing this issue further. We can help you use Python's gc, but not if you're unwilling or unable to change anything about what you do. > ... > What if breaking cycles and cleanup are the same thing? In the case of > Crux, they are one and the same, since the low level tree must be torn down > one piece at a time. Again Martin had in mind that all your objects derived from PyObject. If that were so, then Python's gc would guarantee not to tear down anything until every cycle it was part of became unreachable, in which case internal invariants no longer matter (since they're no longer reachable form anything except other unreachable trash). Python's gc is very nice to work with, for Python objects. If you can't make your objects PyObjects, then you get all the pain of implementing a form of gc that does work with your objects. You really should look at how core objects implement tp_traverse() and tp_clear(), to see how easy life *can* be, even in the presence of much hairier object graphs than you're working with. > I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), > the clear() call does not pay attention to the return value. Heh. So true! See http://mail.python.org/pipermail/python-dev/2003-April/034435.html > It would be possible to check the return value, and defer deallocation if the > return value were non-zero, and ob_refcnt were 1. (However, it would probably > have to be a separate list of objects, contrary to what I claimed before.) I'm not sure, but I think your desire for this was still based on misunderstanding of what Python's gc does. From jasone at canonware.com Tue Oct 12 04:25:47 2004 From: jasone at canonware.com (Jason Evans) Date: Tue Oct 12 04:25:23 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <1f7befae041011182151f7b219@mail.gmail.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> Message-ID: <20041012022547.GC27613@canonware.com> On Mon, Oct 11, 2004 at 09:21:57PM -0400, Tim Peters wrote: > [Jason Evans] > > I think you're in for a rude awakening if you expect any language's gc > support to help you manage arbitrary blobs of memory obtained from the > system malloc(). I don't expect that. What I do expect is for there to be clean and efficient mechanisms for integrating custom Python object types into Python. Python more or less meets this expectation. > If all the nodes in your object graph derived from PyObject, Python's gc > would do it all for you. If you want to plant pointers in and among > arbitrary C structs, then you'll have to implement your own gc, or hope > that you can bash a general gc thingie (like BDW -- and I'm not sure > there is anything else of that sort usefully available, so "like" is > probably optimistic) into guessing the layout of your structs. No, this is not necessary. What is necessary is for my code to accurately maintain and report all references to Python objects during traversal. The Crux code does so, and works as designed. > > Crux was designed this way in order to be able to implement various low level > > algorithms such that they never have to call interpreter-related code. > > I don't know why that was thought to be desirable, or even exactly > what it means. One plausible meaning is that you don't want to tie > the "various low level algorithms" to Python, because you hope to > reuse them in other, non-Python contexts. And, in addition to that, > you don't want, or don't know how, to abstract the necessary > operations into macros or functions that are pluggable for the > different contexts you have in mind. First, "hoping" to be able use my code in other contexts is what allowed me to transition from using a different embeddable interpreter to using Python inside of two days. Had I melded all of the scripting glue code with the low level tree code, it would not have been nearly as easy to transition to using Python. Second, I have the scripting language glue and low level tree code separated for two reasons: 1) performance, and 2) modularization. There's no question of whether I want to use cpp magic to be able to munge two large piles of code into one huge pile of code, nor whether I'm capable of doing so. To do so would simply take the code in a bad direction. > [Martin] > >> Hmmm. Even if a ring member holds only a single reference to another > >> ring member, would not calling visit() on this member in turn invoke > >> visit() on the other one, which would, in turn, invoke visit() on > >> the next one, and so on? > > [Jason] > > Yes, this is true. > > It's false, but I covered that one yesterday. It is true, given my interpretation of the original context of the dialogue. We were discussing whether it would work for a node object to hold a reference to a single ring object, and rely on the ring objects to report their neighbors. Indeed this would work. > > The problem I ran into had to do with the node logically holding references to all > > the ring objects, but only reporting one of them. > > If a ring object doesn't derive from PyObject, then visit() must not > be passed a pointer to a ring object. Doing so would account for many > catastrophic meanings of "doesn't work" . If you want to see the Crux source code, I'll be happy to provide you with a source archive. In the meanwhile, please rest assured that the code doesn't suck nearly as badly as you are insinuating here. > We can help you use Python's gc, but not if you're unwilling or unable > to change anything about what you do. Prior to switching to using Python, Crux integrated with a language that implemented atomic mark and sweep garbage collection. I have had to change a tremendous amount about Crux in order to use a reference counted interpreter. In fact, I made some changes to Crux this morning that were a direct result of your original response to my email (ring objects now report references to themselves, which makes reference counting of ring objects much simpler). Many of the issues I ran into had to do with the impedence mismatch between my previous experience and the way Python works. I appreciate you taking the time to respond to me, especially since you've helped me gain a deeper understanding of how Python's GC works. However, please get it out of your mind that I am incompetent and/or set in my ways. > > I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), > > the clear() call does not pay attention to the return value. > > Heh. So true! See > > http://mail.python.org/pipermail/python-dev/2003-April/034435.html > > > It would be possible to check the return value, and defer deallocation if the > > return value were non-zero, and ob_refcnt were 1. (However, it would probably > > have to be a separate list of objects, contrary to what I claimed before.) > > I'm not sure, but I think your desire for this was still based on > misunderstanding of what Python's gc does. Of the original issues I brought up, I think this one potentially still has merit, though I'm not sure that it's of sufficiently general utility. The idea is to allow extension modules some control over the order in which objects are deallocated, so that it is possible to tear down interconnections in a particular order. However, it occurs to me that since tp_clear can leave references in place, it might be possible to control the object destruction order by leaving a DAG of references in place, and letting Python deallocate objects in dependency order. Does this work? Thanks, Jason From tim.peters at gmail.com Tue Oct 12 05:34:34 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 12 05:34:43 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041012022547.GC27613@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> Message-ID: <1f7befae041011203471688554@mail.gmail.com> [Tim Peters] >> I think you're in for a rude awakening if you expect any language's gc >> support to help you manage arbitrary blobs of memory obtained from the >> system malloc(). [Jason Evans] > I don't expect that. What I do expect is for there to be clean and > efficient mechanisms for integrating custom Python object types into > Python. Python more or less meets this expectation. OK ... but you must want more than just that, else this thread wouldn't have started . >> If all the nodes in your object graph derived from PyObject, Python's gc >> would do it all for you. If you want to plant pointers in and among >> arbitrary C structs, then you'll have to implement your own gc, or hope >> that you can bash a general gc thingie (like BDW -- and I'm not sure >> there is anything else of that sort usefully available, so "like" is >> probably optimistic) into guessing the layout of your structs. > No, this is not necessary. What is necessary is for my code to accurately > maintain and report all references to Python objects during traversal. The > Crux code does so, and works as designed. Isn't it also necessary for your code to clean up object graphs where the nodes are *not* Python objects? That's what I thought you were saying to Martin, If I misunderstood that, I apologize. Almost everything else I said was based on that belief. >>> Crux was designed this way in order to be able to implement various low level >>> algorithms such that they never have to call interpreter-related code. >> I don't know why that was thought to be desirable, or even exactly >> what it means. One plausible meaning is that you don't want to tie >> the "various low level algorithms" to Python, because you hope to >> reuse them in other, non-Python contexts. And, in addition to that, >> you don't want, or don't know how, to abstract the necessary >> operations into macros or functions that are pluggable for the > > different contexts you have in mind. > First, "hoping" to be able use my code in other contexts is what allowed me > to transition from using a different embeddable interpreter to using Python > inside of two days. Had I melded all of the scripting glue code with the > low level tree code, it would not have been nearly as easy to transition to > using Python. So the "you don't want to tie ... to Python" guess was right (please realize that you haven't *said* so before), and the "don't want to abstract ... into macros or functions" part of the other guess was on target (ditto). I think you're reading judgments into my attempts to extract answers. It's fine by me if you don't want to make everything a PyObject, but then I'm also trying to explain consequences of that choice wrt what Python's gc will and won't do for you. > Second, I have the scripting language glue and low level tree code > separated for two reasons: 1) performance, and 2) modularization. There's > no question of whether I want to use cpp magic to be able to munge two > large piles of code into one huge pile of code, nor whether I'm capable of > doing so. To do so would simply take the code in a bad direction. Maybe. I resist some here because adding just enough PyObject hair to enable a C struct to work smoothly with Python's native gc doesn't usually count as a "large pile" of code by any metric I use, and really does make gc life easy. [Martin] >>>> Hmmm. Even if a ring member holds only a single reference to another >>>> ring member, would not calling visit() on this member in turn invoke >>>> visit() on the other one, which would, in turn, invoke visit() on >>>> the next one, and so on? [Jason] >>> Yes, this is true. >> It's false, but I covered that one yesterday. > It is true, given my interpretation of the original context of the > dialogue. We were discussing whether it would work for a node object to > hold a reference to a single ring object, and rely on the ring objects to > report their neighbors. Indeed this would work. Regardless of context, in Python's gc there is no sense in which calling a visit() callback "in turn" invokes visit() on anything. If there is a sense here in which that is true, the context must be something other than Python's gc. If the ring objects are not PyObject themselves (I thought it was clear that they weren't, but maybe that was wrong), then Python's gc gives them no way to "report their neighbors". >>> The problem I ran into had to do with the node logically holding references to >>> all the ring objects, but only reporting one of them. >> If a ring object doesn't derive from PyObject, then visit() must not >> be passed a pointer to a ring object. Doing so would account for many >> catastrophic meanings of "doesn't work" . > If you want to see the Crux source code, I'll be happy to provide you with > a source archive. I'd like to say "yes", but I'll never make time to look at it. I would have been happier here if you had confirmed or denied that ring objects don't derive from PyObject. I pressed before on that "doesn't work" isn't helpful, and that never got clarified. Possible meanings include segfaults, error messages, Python exceptions. and memory leaks. > In the meanwhile, please rest assured that the code doesn't suck nearly as > badly as you are insinuating here. I'm not insinuating anything about the code, I'm trying to guess what you meant in your original post by "doesn't work" this-and-that. Passing a non-PyObject* to visit() remains a possible cause for "doesn't work" in my head, because you haven't said anything to rule it out. If you in fact *had* been passing a non-PyObject* to visit(), then the paragraph above could have been helpful ("ah, that's why it didn't work!"). >> We can help you use Python's gc, but not if you're unwilling or unable >> to change anything about what you do. > Prior to switching to using Python, Crux integrated with a language that > implemented atomic mark and sweep garbage collection. I have had to change > a tremendous amount about Crux in order to use a reference counted > interpreter. In fact, I made some changes to Crux this morning that were a > direct result of your original response to my email (ring objects now > report references to themselves, which makes reference counting of ring > objects much simpler). So Ring objects do derive from PyObject*. Forget everything I said then . > Many of the issues I ran into had to do with the impedence mismatch between > my previous experience and the way Python works. I sure believe that. Python's cyclic gc is unusual in several respects. > I appreciate you taking the time to respond to me, especially since you've > helped me gain a deeper understanding of how Python's GC works. However, > please get it out of your mind that I am incompetent and/or set in my ways. Those aren't in my head. I do think you're so steeped in the details of your code that you're not nearly as successful at communicating the *salient* points as you think you are, though. I'm so steeped in the details of Python's gc that a similar claim could easily be made of me. I don't expect it to be clear the first time, though, so play 20-questions trying to tease out what you really need to know about it. If it's any consolation, this process has been painful on both sides . >>> I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), >>> the clear() call does not pay attention to the return value. >> Heh. So true! See >> >> http://mail.python.org/pipermail/python-dev/2003-April/034435.html >>> It would be possible to check the return value, and defer deallocatin if the >>> return value were non-zero, and ob_refcnt were 1. (However, it would probably >>> have to be a separate list of objects, contrary to what I claimed before.) >> I'm not sure, but I think your desire for this was still based on >> misunderstanding of what Python's gc does. > Of the original issues I brought up, I think this one potentially still has > merit, though I'm not sure that it's of sufficiently general utility. The > idea is to allow extension modules some control over the order in which > objects are deallocated, so that it is possible to tear down > interconnections in a particular order. Well, think about it some more then. "A separate list of objects" is very easy to create efficiently in Python's gc internals -- each gc'able object is in *some* doubly-linked list, so the cost of a new list is just a new header node to identify it, and moving a gc'able object from one list to another is 6 pointer stores. > However, it occurs to me that since tp_clear can leave references in place, > it might be possible to control the object destruction order by leaving a > DAG of references in place, and letting Python deallocate objects in > dependency order. Does this work? It could. Here it should help to know that Python's cyclic gc never calls an object's tp_dealloc directly. The only way tp_dealloc ever triggers in Python is as a side effect of a refcount falling to zero -- no exceptions. So, indeed, "all" cyclic gc does is break cycles, and the only places garbage actually gets recycled in gcmodule.c are as side effects of the Py_DECREF(op); lines in delete_garbage() and release_weakrefs(). Weakrefs are probably irrelevant to you at this point, so there's only one Py_DECREF() in the codebase you care about. So, e.g, if you have a simple cycle like A -> B -> C -> ... -> Z -> A and the tp_clear() calls break only the link from Z to A, that's all they *need* to do, and then A's tp_dealloc() will be called first, then B's tp_dealloc, ..., and finally Z's tp_dealloc. You should be aware that long chains of that sort can end up overflowing the C stack, depeding on how tp_dealloc is coded. That's in CPython, of course. Jython uses Java's native gc, and I suppose (but don't know that) IronPython uses .NET's. From martin at v.loewis.de Tue Oct 12 06:13:35 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 12 06:13:37 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011152210.08C1.JCARLSON@uci.edu> References: <20041011152210.08C1.JCARLSON@uci.edu> Message-ID: <416B59EF.8010403@v.loewis.de> Josiah Carlson wrote: > A pronouncement would be nice, but I understand that I am in no position > to ask/beg for a (yes, we'll add it) response. <.5 wink> Well, I can declare, in public :-), that I personally won't add that feature to Python 2.4. There are so many other things to do, and I don't consider this feature (in whatever form) important, as this can be easily done in pure Python for those who need it. Regards, Martin From jcarlson at uci.edu Tue Oct 12 09:00:48 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Oct 12 09:08:16 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <416B59EF.8010403@v.loewis.de> References: <20041011152210.08C1.JCARLSON@uci.edu> <416B59EF.8010403@v.loewis.de> Message-ID: <20041011232519.08C8.JCARLSON@uci.edu> Martin v. L?wis wrote: > Josiah Carlson wrote: > > A pronouncement would be nice, but I understand that I am in no position > > to ask/beg for a (yes, we'll add it) response. <.5 wink> > > Well, I can declare, in public :-), that I personally won't add that > feature to Python 2.4. Scratch chin. Then I suppose it is a good thing that you aren't the only person with commit access then . > There are so many other things to do, and I > don't consider this feature (in whatever form) important, as this > can be easily done in pure Python for those who need it. The same thing can be said for some syntactic changes to Python over the years; yet that didn't stop list comprehensions, rich comparisons, iterators (generators being the language change that made them easier), etc., from changing the language. I'm not even asking to change the language, I'm offering a patch that implements a functionality desireable to a group of users by making a module more flexible. Since Tim has said that the change is so minor as to be all right to wait until after the beta, I'll keep out of everyone's hair until then. - Josiah From astrand at lysator.liu.se Tue Oct 12 09:25:14 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Tue Oct 12 09:25:23 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: On Mon, 11 Oct 2004, Guido van Rossum wrote: > > So, can the subprocess module be accepted? If not, what needs to be done? > > FWIW, I've given Peter some feedback, but in general I think this code > is ready for inclusion into 2.4. I hope it's not too late. Sounds great. >(My > recommendation on the call/callv issue is to get rid of callv(), callv() has now been removed. >and > instead let call() interpret multiple string args as such; a single > string arg will be interpreted as a whole command line, per Popen(); if > you want a single file arg you have to enclose it in [...].) I haven't done any changes to call(): it passes the arguments directly to Popen(), just like before. I would prefer if we could keep it that way; it's so nice to have a short and simple description of the function: Run command with arguments. Wait for command to complete, then return the returncode attribute. If we should do any more changes to handling of multiple args/strings/sequences, we should probably do it to the Popen class itself. But I think we have found a good API now. Here's the documentation: " args should be a string, or a sequence of program arguments. The program to execute is normally the first item in the args sequence or string, but can be explicitly set by using the executable argument. On UNIX, with shell=False (default): In this case, the Popen class uses os.execvp() to execute the child program. args should normally be a sequence. A string will be treated as a sequence with the string as the only item (the program to execute). On UNIX, with shell=True: If args is a string, it specifies the command string to execute through the shell. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments. On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline method. Please note that not all MS Windows applications interpret the command line the same way: The list2cmdline is designed for applications using the same rules as the MS C runtime. " /Peter ?strand From sjoerd at acm.org Tue Oct 12 09:39:52 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Tue Oct 12 09:40:01 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416AF20D.6010706@v.loewis.de> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> <416AAE03.9020209@acm.org> <416AF20D.6010706@v.loewis.de> Message-ID: <416B8A48.6010500@acm.org> Martin v. L?wis wrote: > Sjoerd Mullender wrote: > >>> Don't know, but guess so: it contains bytes outside the set ANSI C >>> says can be used portably in text files. >> >> >> >> However, this is not necessarily enough reason to use -kb. The only >> things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for >> updates, and not expanding $ keywords. > > > That is not true. On Apple computers, it also avoids conversion from > Latin-1 to Mac-Roman, which Mac CVS does by default for text files. > Making the files binary is the only way to avoid this conversion, and > that is precisely the reason why the file is binary. I didn't know this. > You may argue that this is a bug in Mac CVS, and I would agree. However, > that specific bug has -kb as a known work-around, and the issue reported > here points to a bug in the compiler packages which should be fixed > rather than worked-around. I agree that this sounds very much like a MacCVS bug, but it also sounds like an excellent reason to leave this file alone. And the compiler issue should be (and has been, I saw) fixed. -- Sjoerd Mullender -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 374 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041012/577896c2/signature.pgp From anthony at interlink.com.au Tue Oct 12 09:57:39 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Oct 12 09:57:57 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011232519.08C8.JCARLSON@uci.edu> References: <20041011152210.08C1.JCARLSON@uci.edu> <416B59EF.8010403@v.loewis.de> <20041011232519.08C8.JCARLSON@uci.edu> Message-ID: <416B8E73.2090304@interlink.com.au> Josiah Carlson wrote: > Since Tim has said that the change is so minor as to be all right to > wait until after the beta, I'll keep out of everyone's hair until then. I guess this _is_ pretty minor, but please don't assume that this means I'll be letting all sorts of new things in past b1 (at least, not without a lot of complaining on my part ;) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From just at letterror.com Tue Oct 12 10:18:14 2004 From: just at letterror.com (Just van Rossum) Date: Tue Oct 12 10:18:20 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416B8A48.6010500@acm.org> Message-ID: Sjoerd Mullender wrote: > >> However, this is not necessarily enough reason to use -kb. The > >> only things -kb does are LF -> CRLF / LF -> CR mapping, not using > >> diff for updates, and not expanding $ keywords. > > > > That is not true. On Apple computers, it also avoids conversion > > from Latin-1 to Mac-Roman, which Mac CVS does by default for text > > files. Making the files binary is the only way to avoid this > > conversion, and that is precisely the reason why the file is binary. > > I didn't know this. > > > You may argue that this is a bug in Mac CVS, and I would agree. > > However, that specific bug has -kb as a known work-around, and the > > issue reported here points to a bug in the compiler packages which > > should be fixed rather than worked-around. > > I agree that this sounds very much like a MacCVS bug, but it also > sounds like an excellent reason to leave this file alone. And the > compiler issue should be (and has been, I saw) fixed. On the other hand, I hardly know anyone who uses MacCVS anymore. Before OSX you _had_ to use some Mac port of CVS (there were/are many), but on OSX most people simply use cvs on the command line, which is the standard unix version. Still, you can configure MacCVS to not convert the encoding. In other words: I wouldn't worry about MacCVS anymore, at all. Just From mcherm at mcherm.com Tue Oct 12 16:27:03 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Oct 12 16:26:03 2004 Subject: [Python-Dev] scropion Message-ID: <1097591223.416be9b73a9a4@mcherm.com> > [Paul introduces himself] > My question is for performance: isn't binary code faster than p code? > So If I can vote, I would vote for c or c++ in the numerics functions and > data transfer areas. > even assembler. The quicker the better. Hello, and welcome. Thanks for your input on what kinds of things you value in Python. (The Python developers, who use the python-dev list) DO pay attention to such opinion (although don't expect sudden changes... you are one in a sea of users). You are not alone in caring about performance -- it's been a popular topic lately. I'll comment briefly on your specific suggestions. Binary code is faster than "p code" usually, but in some cases the "p code" is more ammenable to automated optimization. That is why certain kinds of code can run FASTER in java than in C. Python code too will often run faster than the equivalent C code... not because of a powerful just-in-time optimizer, but because the Python code is written using better algorithms and data structures than one would bother with in C. But you specifically mention numerics functions and data transfer... two areas where well-written C code is likely to be much better performing. We certainly share the goal of better performance (so long as it doesn't come at the cost of other things like usability and maintainability). If you know of some specific areas where we could improve, please submit a patch (through SourceForge's patch manager) -- we'd love to have the help! Finally, you mention the possibility of dropping into assembler. Unfortunately, this is sure to be rejected because it conflicts with an important design principle -- Python is designed to be VERY portable, and should compile on just about any system with a standards-compliant C compiler. -- Michael Chermside From anthony at interlink.com.au Tue Oct 12 19:15:55 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Oct 12 16:31:51 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: <416C114B.8000906@interlink.com.au> Guido van Rossum wrote: > FWIW, I've given Peter some feedback, but in general I think this code > is ready for inclusion into 2.4. I hope it's not too late. (My > recommendation on the call/callv issue is to get rid of callv(), and > instead let call() interpret multiple string args as such; a single > string arg will be interpreted as a whole command line, per Popen(); > if you want a single file arg you have to enclose it in [...].) 2.4b1 is planned for this Thursday. I'm taking Wednesday off work to commit a backlog of patches I want to see in, then plan to cut the release Thursday. I can probably move this to Friday, if it would help this get in. I _really_ don't want this landing after b1, tho - that kinda makes the entire release process pointless if we're landing stuff that late. I'm off to sleep now - could Peter or whoever else is likely to do the work let me know? If it's just a matter of doing the checkin, and making sure all the fiddly bits are done, you could also point me at a patch and I'll do it. From fredrik at pythonware.com Tue Oct 12 16:50:23 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 12 16:48:19 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <416C114B.8000906@interlink.com.au> Message-ID: Anthony Baxter wrote > I'm off to sleep now - could Peter or whoever else is likely to do > the work let me know? If it's just a matter of doing the checkin, > and making sure all the fiddly bits are done, you could also point > me at a patch and I'll do it. I can check in the source files; someone with a proper Windows setup will have to help us out with the makefile and installer issues. Stay tuned. From fredrik at pythonware.com Tue Oct 12 17:07:17 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 12 17:05:12 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: <416C114B.8000906@interlink.com.au> Message-ID: Anthony Baxter wrote: > I'm off to sleep now - could Peter or whoever else is likely to do > the work let me know? If it's just a matter of doing the checkin, > and making sure all the fiddly bits are done, you could also point > me at a patch and I'll do it. I'll check in the source files shortly. Someone with a proper Windows setup will have to help out with makefile and installer tweaks for the PC/_subprocess.c module; drop me a line if you need info. From fredrik at pythonware.com Tue Oct 12 18:16:14 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 12 18:14:10 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 References: Message-ID: > +\begin{seealso} > +\seepep{324}{subprocess - New process module}{Written and implemented by Peter Astrand, with > assistance from Fredrik Lundh and others.} > +\end{seealso} I don't know how to add ISO-8859-1 characters to a Latex document, but in HTML notation, Peter's last name should be Åstrand. cheers /F From aparente at adobe.com Tue Oct 12 18:17:49 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Tue Oct 12 18:18:29 2004 Subject: [Python-Dev] Python on Windows 64bits? Message-ID: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> Sorry if I'm asking something that has been answered before, but I could not find a clear answer anywhere. Is there a project page for Windows 64bits? Is it already on CVS? My company tries to assess if we can plan on using python on Windows 64 bits. We want to do it natively (i.e. no 32b emulation), because we have some python extensions we want to use in the process. Thanks! alex. From gvanrossum at gmail.com Tue Oct 12 19:50:56 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Oct 12 19:51:17 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: References: <416B8A48.6010500@acm.org> Message-ID: In any case, the necessity of -kb for this file (if it is still necessary) should be documented in a comment inside the file, to prevent a repeat of this discussion three years from now. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Tue Oct 12 20:15:00 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Tue Oct 12 20:13:47 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 In-Reply-To: References: Message-ID: Fredrik Lundh wrote: >>+\seepep{324}{subprocess - ...}{Written ... Peter Astrand, ... > > I don't know how to add ISO-8859-1 characters to a Latex document, > but in HTML notation, Peter's last name should be Åstrand. > > cheers /F I think that should be: +\seepep{324}{subprocess - ...}{Written ... Peter \AA strand, ... -- Scott David Daniels Scott.Daniels@Acm.Org From abingham at alumni.sfu.ca Tue Oct 12 20:39:30 2004 From: abingham at alumni.sfu.ca (Aaron Bingham) Date: Tue Oct 12 20:48:56 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 In-Reply-To: References: Message-ID: <416C24E2.4020304@alumni.sfu.ca> Fredrik Lundh wrote: >>+\begin{seealso} >>+\seepep{324}{subprocess - New process module}{Written and implemented by Peter Astrand, with >>assistance from Fredrik Lundh and others.} >>+\end{seealso} >> >> > >I don't know how to add ISO-8859-1 characters to a Latex document, >but in HTML notation, Peter's last name should be Åstrand. > > I don't know if it's allowed for for Python docs, but the easiest way I know of is to put \usepackage{isolatin1} in the prelude and just use regular ISO-8859-1 characters in the file. Aaron From barry at python.org Tue Oct 12 21:31:55 2004 From: barry at python.org (Barry Warsaw) Date: Tue Oct 12 21:32:05 2004 Subject: [Python-Dev] subprocess module test failure when --with-pydebug Message-ID: <1097609515.8573.50.camel@geddy.wooz.org> The new subprocess module test is not compatible with a Python built --with-pydebug because the [xxxx refs] output breaks the comparison. I've recently been thinking that it would be nice if that [xxxx refs] output could be suppressed with an envar or a command line switch. FWIW, I generally build our embedding product with a --with-pydebug built Python and that output breaks our unit tests too. I work around this by suppressing the stderr comparison in our testing framework. Once we decide to do it, and what to wrap that call in, it's likely a one or two line change to pythonrun.c. Does one of the existing envars make sense to use? Barring any change here, we need to at least make sure the subprocess test doesn't fail for a debug build. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041012/f96bafc2/attachment.pgp From janssen at parc.com Tue Oct 12 21:31:58 2004 From: janssen at parc.com (Bill Janssen) Date: Tue Oct 12 21:33:32 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: Your message of "Tue, 12 Oct 2004 01:18:14 PDT." Message-ID: <04Oct12.123203pdt."58617"@synergy1.parc.xerox.com> Just van Rossum writes: > On the other hand, I hardly know anyone who uses MacCVS anymore. Before > OSX you _had_ to use some Mac port of CVS (there were/are many), but on > OSX most people simply use cvs on the command line, which is the > standard unix version. Still, you can configure MacCVS to not convert > the encoding. In other words: I wouldn't worry about MacCVS anymore, at > all. Agreed. Bill From aahz at pythoncraft.com Tue Oct 12 21:36:59 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Oct 12 21:37:08 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> Message-ID: <20041012193659.GA26457@panix.com> On Tue, Oct 12, 2004, Alexandre Parenteau wrote: > > Sorry if I'm asking something that has been answered before, but I could > not find a clear answer anywhere. > > Is there a project page for Windows 64bits? Is it already on CVS? > > My company tries to assess if we can plan on using python on Windows 64 > bits. > > We want to do it natively (i.e. no 32b emulation), because we have some > python extensions we want to use in the process. In theory, it should "just work". Python is already 64-bit compliant; any issues are likely to be problems with Microsoft compilers. You should probably ask on comp.lang.python unless you uncover specific problems that you believe are caused by Python itself. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From abkhd at hotmail.com Wed Oct 13 04:47:59 2004 From: abkhd at hotmail.com (Khalid A. B.) Date: Wed Oct 13 04:48:05 2004 Subject: [Python-Dev] test_cwd of test_subprocess.py fails on Win98 Message-ID: Hello. The test fails because it seems to be comparing "C:\WINDOWS\TEMP" with "C:\WINDOWS\Temp" and even though they are the same thing in Win98, they are of cource not equal. Details follow: $ python -i Python 2.4a3 (#56, Oct 13 2004, 02:29:05) [GCC 3.4.1 (mingw special)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>import os, sys, subprocess >>>tmpdir = os.getenv("TEMP", "/tmp") >>>tmpdir = os.path.realpath(tmpdir) >>>tmpdir 'C:\\WINDOWS\\TEMP' >>>p = subprocess.Popen([sys.executable, "-c", ... 'import sys,os;' \ ... 'sys.stdout.write(os.getcwd())'], ... stdout=subprocess.PIPE, ... cwd=tmpdir) >>>subProcessTmpDir = p.stdout.read() >>>subProcessTmpDir 'C:\\WINDOWS\\Temp' >>>tmpdir == subProcessTmpDir False >>> Although I don't know if this will work in other versions of Windows, here is what fixed it for me: Index: python/dist/src/Lib/test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.4 diff -u -d -r1.4 test_subprocess.py --- python/dist/src/Lib/test/test_subprocess.py 12 Oct 2004 22:29:54 -0000 1.4 +++ python/dist/src/Lib/test/test_subprocess.py 13 Oct 2004 02:27:13 -0000 @@ -192,9 +192,16 @@ def test_cwd(self): tmpdir = os.getenv("TEMP", "/tmp") tmpdir = os.path.realpath(tmpdir) - p = subprocess.Popen([sys.executable, "-c", - 'import sys,os;' \ - 'sys.stdout.write(os.getcwd())'], + commandStr = [sys.executable, "-c"] + if mswindows: + tmpdir = tmpdir.upper() + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd().upper())') + else: + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd())') + + p = subprocess.Popen(commandStr, stdout=subprocess.PIPE, cwd=tmpdir) self.assertEqual(p.stdout.read(), tmpdir) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/ From erik at heneryd.com Wed Oct 13 07:09:22 2004 From: erik at heneryd.com (Erik Heneryd) Date: Wed Oct 13 07:15:01 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: References: Message-ID: <416CB882.3050308@heneryd.com> Quick thought: The subprocess class is still called Popen. Any reason not to drop this last reference to the unix clib function and rename it Subprocess? Erik From martin at v.loewis.de Wed Oct 13 07:22:56 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 07:22:57 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011232519.08C8.JCARLSON@uci.edu> References: <20041011152210.08C1.JCARLSON@uci.edu> <416B59EF.8010403@v.loewis.de> <20041011232519.08C8.JCARLSON@uci.edu> Message-ID: <416CBBB0.9040809@v.loewis.de> Josiah Carlson wrote: >>There are so many other things to do, and I >>don't consider this feature (in whatever form) important, as this >>can be easily done in pure Python for those who need it. > > > The same thing can be said for some syntactic changes to Python over the > years; yet that didn't stop list comprehensions, rich comparisons, > iterators (generators being the language change that made them easier), > etc., from changing the language. Indeed. I wish some of the changes would not have been made (although the changes on your list are all fine). For those features, the potential user community is *much* larger than for the feature under discussion, and I feel I would waste my time for adding a feature that only few users ever need. If you would like to debate the size of the user community: any significanly-large user community should have come up with a standard, pure-Python solution to the problem by now which would just wait for integration. This is IMO the process that should be followed for all library extensions: the library should be developed elsewhere, and wait some time for API and implementation stabilization. Only *then* it become candidate for inclusion into Python. Regards, Martin From martin at v.loewis.de Wed Oct 13 07:36:36 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 07:36:36 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> Message-ID: <416CBEE4.5040908@v.loewis.de> Alexandre Parenteau wrote: > Sorry if I'm asking something that has been answered before, but I could > not find a clear answer anywhere. > > Is there a project page for Windows 64bits? Is it already on CVS? > > My company tries to assess if we can plan on using python on Windows 64 > bits. > > We want to do it natively (i.e. no 32b emulation), because we have some > python extensions we want to use in the process. Itanium or AMD64? For Itanium, we have installers (of Python 2.4) available: http://www.python.org/2.4/ For AMD64, you need to compile it yourself. Regards, Martin From astrand at lysator.liu.se Wed Oct 13 07:38:31 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 07:38:39 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <416CB882.3050308@heneryd.com> References: <416CB882.3050308@heneryd.com> Message-ID: On Wed, 13 Oct 2004, Erik Heneryd wrote: > Quick thought: The subprocess class is still called Popen. Any reason > not to drop this last reference to the unix clib function and rename it > Subprocess? It's shorter... :) Also, I think it's good that the name hints about "process open"; that you can read and write to the process. /Peter ?strand From fredrik at pythonware.com Wed Oct 13 08:56:24 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 08:54:18 2004 Subject: [Python-Dev] Re: test_cwd of test_subprocess.py fails on Win98 References: Message-ID: "Khalid A. B." wrote: > The test fails because it seems to be comparing "C:\WINDOWS\TEMP" with > "C:\WINDOWS\Temp" and even though they are the same thing in Win98, > they are of cource not equal. I just applied the patch below, which I think does the same thing in a slightly more convenient way. --- Lib/test/test_subprocess.py 13 Oct 2004 04:07:12 -0000 1.9 +++ Lib/test/test_subprocess.py 13 Oct 2004 06:52:56 -0000 @@ -216,7 +216,8 @@ 'sys.stdout.write(os.getcwd())'], stdout=subprocess.PIPE, cwd=tmpdir) - self.assertEqual(p.stdout.read(), tmpdir) + normcase = os.path.normcase + self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir)) def test_env(self): newenv = os.environ.copy() From fredrik at pythonware.com Wed Oct 13 09:35:36 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 09:33:30 2004 Subject: [Python-Dev] latex help wanted! (Re: subprocess - updated popen5 module) References: <416C114B.8000906@interlink.com.au> Message-ID: Fredrik Lundh wrote: > I'll check in the source files shortly. Someone with a proper Windows setup > will have to help out with makefile and installer tweaks for the PC/_subprocess.c > module; drop me a line if you need info. Tim fixed the windows issues (and got rid of the banana), but I just realized that we could need some help with converting the documentation from PEP/docstring format to latex... Any volunteers? (the PEP in nondist/peps/pep-0324.txt contains extensive documentation, but it can also be good to check the Lib/subprocess.py docstring for additional examples) (Now, if only someone could convert the docs to "html body + semantic classes" so people with ordinary skills can work on it, without having to learn any new syntaxes or install any new tools, besides python and a web browser...) From jcarlson at uci.edu Wed Oct 13 09:26:38 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 13 09:34:06 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <416CBBB0.9040809@v.loewis.de> References: <20041011232519.08C8.JCARLSON@uci.edu> <416CBBB0.9040809@v.loewis.de> Message-ID: <20041012232318.091E.JCARLSON@uci.edu> > For those features, the > potential user community is *much* larger than for the feature under > discussion, and I feel I would waste my time for adding a feature that > only few users ever need. And yet you continue the discussion? I talk about it because /I/ find the functionality useful, I believe /others/ would find it useful if they were put in similar situations as I, and I believe it adds flexibility to a module (which I think is positive). > If you would like to debate the size of the user community: any > significanly-large user community should have come up with a standard, > pure-Python solution to the problem by now which would just wait for > integration. This is IMO the process that should be followed for all > library extensions: the library should be developed elsewhere, and > wait some time for API and implementation stabilization. Only *then* > it become candidate for inclusion into Python. If every modification to Python required a community of people, who all joined together to advocate something, then nothing would ever get done. People would be spending all their time trying to gather groups of people to agree with them that functionality K is necessary. Thankfully there has been cases in the past where someone has said, "hey, this thing could really use X", some people agree, some people disagree, a sample implementation is done, and sometimes it is accepted. I'm not even saying that we should add a new module. I'm not even saying we should add a new function to a module. Heck, I'm not even asking for a new argument to a function that previously existed. I am, quite literally, asking for the equivalent of less than one bit in a flag (there are currently 22 format/endian characters, which are all orthogonal, which would require 5 bits, if they were flags). The API already exists. The framework already exists. I'm not asking for Python to interpret something that was valid before as something different now. I'm asking for the equivalent of a previously invalid flag, to become valid, in order to expose previously existing translation mechanisms, whose use can be found in databases, network protocols, encryption, etc. Try to read the above paragraph outside of the context of struct and the RFE. Does it make sense to include the change now? If every request to interpret a new flag required significant community involvement, goodness, would it take an act of Guido to get a commit done? Have a good day Martin, - Josiah From theller at python.net Wed Oct 13 09:41:45 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 09:41:29 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib subprocess.py, NONE, 1.1 In-Reply-To: (effbot@users.sourceforge.net's message of "Tue, 12 Oct 2004 08:26:29 -0700") References: Message-ID: effbot@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv24287/Lib > > Added Files: > subprocess.py > Log Message: > Added Peter Astrand's subprocess module. I suggest to remove the imports of the pywin32 stuff, it's not needed and confuses tools like freeze or py2exe. if mswindows: import threading import msvcrt try: from _subprocess import * class STARTUPINFO: dwFlags = 0 hStdInput = None hStdOutput = None hStdError = None class pywintypes: error = IOError except ImportError: import pywintypes from win32api import GetStdHandle, STD_INPUT_HANDLE, \ STD_OUTPUT_HANDLE, STD_ERROR_HANDLE from win32api import GetCurrentProcess, DuplicateHandle, \ GetModuleFileName, GetVersion from win32con import DUPLICATE_SAME_ACCESS from win32pipe import CreatePipe from win32process import CreateProcess, STARTUPINFO, \ GetExitCodeProcess, STARTF_USESTDHANDLES, \ CREATE_NEW_CONSOLE from win32event import WaitForSingleObject, INFINITE, WAIT_OBJECT_0 From fredrik at pythonware.com Wed Oct 13 09:48:48 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 09:46:48 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) References: <416C114B.8000906@interlink.com.au> Message-ID: can bug categories be associated with developers? if so, would it perhaps make sense to add Peter as a developer, and add a category for the subprocess system? From fredrik at pythonware.com Wed Oct 13 09:54:15 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 09:52:10 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 References: Message-ID: Thomas Heller wrote: > I suggest to remove the imports of the pywin32 stuff, it's not needed > and confuses tools like freeze or py2exe. That's really a problem with the tools, not with the module. Demanding that users of a dynamic language use only static linking is really silly. Wouldn't it be better to invent a (pragma-level) syntax so that a module can inform the tool about its intentions? (removing the pywin stuff from the module would hamper development of new features, and also make it harder to verify that the _subprocess module works as it should). From theller at python.net Wed Oct 13 10:04:19 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 10:04:04 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: (Fredrik Lundh's message of "Wed, 13 Oct 2004 09:54:15 +0200") References: Message-ID: "Fredrik Lundh" writes: > Thomas Heller wrote: > >> I suggest to remove the imports of the pywin32 stuff, it's not needed >> and confuses tools like freeze or py2exe. > > That's really a problem with the tools, not with the module. Yep. > Demanding that users of a dynamic language use only static linking is > really silly. > > Wouldn't it be better to invent a (pragma-level) syntax so that a module can > inform the tool about its intentions? Yes. But I don't think I have the energy to discuss that here, and don't feel it would be easy to get this into the core. > (removing the pywin stuff from the module would hamper development of new > features, and also make it harder to verify that the _subprocess module works > as it should). Hm, that what the tests are for, and not code that is never executed. A working solution would be to comment out this stuff, or put it into an 'if 0:' block, in both cases one could easily activate it again for experimentation. Thomas From fredrik at pythonware.com Wed Oct 13 10:19:26 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 10:17:21 2004 Subject: [Python-Dev] Re: Re: python/dist/src/Lib subprocess.py,NONE,1.1 References: Message-ID: Thomas Heller wrote: >> (removing the pywin stuff from the module would hamper development of new >> features, and also make it harder to verify that the _subprocess module works >> as it should). > > Hm, that what the tests are for, and not code that is never executed. if you remove the _subprocess driver, it is executed. _subprocess is just a win32all emulator... > A working solution would be to comment out this stuff, or put it into an > 'if 0:' block, in both cases one could easily activate it again for > experimentation. does py2exe support "if 0" blocks? From theller at python.net Wed Oct 13 10:33:36 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 10:33:20 2004 Subject: [Python-Dev] Re: Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: (Fredrik Lundh's message of "Wed, 13 Oct 2004 10:19:26 +0200") References: Message-ID: "Fredrik Lundh" writes: > Thomas Heller wrote: > >>> (removing the pywin stuff from the module would hamper development of new >>> features, and also make it harder to verify that the _subprocess module works >>> as it should). >> >> Hm, that what the tests are for, and not code that is never executed. > > if you remove the _subprocess driver, it is executed. _subprocess is just > a win32all emulator... To be honest, I'm against code in the core that depends on whether pywin32 is installed or not. >> A working solution would be to comment out this stuff, or put it into an >> 'if 0:' block, in both cases one could easily activate it again for >> experimentation. > > does py2exe support "if 0" blocks? py2exe uses modulefinder to scan the byte code of compiled modules, and Python doesn't generate code for 'if 0' blocks, so, yes. Thomas From python at rcn.com Wed Oct 13 10:56:53 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 13 10:58:31 2004 Subject: [Python-Dev] Struct module format code addition In-Reply-To: <20041012232318.091E.JCARLSON@uci.edu> Message-ID: <001501c4b102$96de0480$e841fea9@oemcomputer> [Josiah] > I'm asking for the equivalent of a previously invalid > flag, to become valid, ... IMO, there was an early transition from asking to demanding. Negative comments by some of the top developers did not dissuade you in the least. > If every request to interpret a new flag required significant community > involvement, goodness, would it take an act of Guido to get a commit > done? Yes, if a proposal doesn't win developer support, then having Guido as a champion is the best bet. If you are opinion shopping, then the best so far is Tim's comment which seems to equate to a +0. That would be sufficient only if he cares enough to review the patch, write the docs, test it, maintain it, etc. Perhaps he will, perhaps he won't. FWIW, last year I had dropped one of my own proposals (relating to a non-unicode use for encode/decode) simply based on respect for Martin's -1. For me, that was more important than the +0 to +1 comments from others, more important than my own use cases, and more important than feeling like I was right. In the end, there was an easy pure python equivalent and life went on. Looking at the struct feature request, I think it would be harmless to add it. OTOH, it is easily handled in python and will be easier still when the binascii functions get put in. Arguably, the feature is somewhat trivial and won't change anyone's life for good or ill. So, you have to ask yourself whether it is worth jamming down everyone's throats to get it in after the beta goes out. For my money, the feature request has already consumed more developer resources than it could ever save. Raymond From fredrik at pythonware.com Wed Oct 13 11:57:12 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 11:57:18 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 References: Message-ID: Thomas Heller wrote: > To be honest, I'm against code in the core that depends on whether > pywin32 is installed or not. >> does py2exe support "if 0" blocks? > > py2exe uses modulefinder to scan the byte code of compiled modules, and > Python doesn't generate code for 'if 0' blocks, so, yes. just one more question: is there a specific problem with win32all? if this is a general problem, how do you deal with other optional modules in the standard library (e.g. _xmlrpclib) ? From theller at python.net Wed Oct 13 14:01:49 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 14:01:33 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: (Fredrik Lundh's message of "Wed, 13 Oct 2004 11:57:12 +0200") References: Message-ID: "Fredrik Lundh" writes: > Thomas Heller wrote: > >> To be honest, I'm against code in the core that depends on whether >> pywin32 is installed or not. > >>> does py2exe support "if 0" blocks? >> >> py2exe uses modulefinder to scan the byte code of compiled modules, and >> Python doesn't generate code for 'if 0' blocks, so, yes. > > just one more question: is there a specific problem with win32all? if this > is a general problem, how do you deal with other optional modules in the > standard library (e.g. _xmlrpclib) ? If _xmlrpclib is installed, py2exe will find it if the script uses xmlrpclib. I guess that is what the user expects. If _xmlrpclib is not installed, py2exe will warn that it is missing (this warning is probably not very useful, and it seems Bob has supressed this warning in py2app). The point with win32all (now pywin32) is that most windows users will have it installed, for good reasons. py2exe will find it, and include it if the subprocess module is pulled in - whether _subprocess is available or not. But subprocess, the Python 2.4 version, will *never* use it, because _subprocess is always available (isn't this now even a builtin module, in python24.dll?). Hope that answers your question, Thomas From fdrake at acm.org Wed Oct 13 15:26:18 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Oct 13 15:26:26 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/distutils/command __init__.py, 1.19, 1.20 install.py, 1.70, 1.71 checkdep.py, 1.1, NONE In-Reply-To: References: Message-ID: <200410130926.18709.fdrake@acm.org> On Wednesday 13 October 2004 08:35 am, Anthony Baxter wrote: > Backing out the basic dependency checking (from pycon sprint). > This support was only a first cut, and doesn't deserve to be in > a released version (where we have to support it in an ongoing > manner) Thanks, Anthony! -Fred -- Fred L. Drake, Jr. From anthony at interlink.com.au Wed Oct 13 16:24:46 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 13 16:25:02 2004 Subject: [Python-Dev] test_subprocess fails on OSX In-Reply-To: References: Message-ID: <416D3AAE.6090907@interlink.com.au> On OSX 10.3.4, test_subprocess is failing intermittently when run from regrtest: test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/Users/anthonybaxter/pyhead/src/Lib/test/test_subprocess.py", line 434, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '4' != '3' Running just the test by itself works. I'm guessing it could be part of the test suite leaking a FD or something? Anyone know of a tool on OSX that I can run to examine open FDs for a process? -- Anthony Baxter It's never too late to have a happy childhood. From bob at redivi.com Wed Oct 13 16:45:41 2004 From: bob at redivi.com (Bob Ippolito) Date: Wed Oct 13 16:46:19 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: References: Message-ID: <8DF3F5EC-1D26-11D9-AAC5-000A95686CD8@redivi.com> On Oct 13, 2004, at 8:01 AM, Thomas Heller wrote: > "Fredrik Lundh" writes: > >> Thomas Heller wrote: >> >>> To be honest, I'm against code in the core that depends on whether >>> pywin32 is installed or not. >> >>>> does py2exe support "if 0" blocks? >>> >>> py2exe uses modulefinder to scan the byte code of compiled modules, >>> and >>> Python doesn't generate code for 'if 0' blocks, so, yes. >> >> just one more question: is there a specific problem with win32all? >> if this >> is a general problem, how do you deal with other optional modules in >> the >> standard library (e.g. _xmlrpclib) ? > > If _xmlrpclib is installed, py2exe will find it if the script uses > xmlrpclib. I guess that is what the user expects. If _xmlrpclib is > not > installed, py2exe will warn that it is missing (this warning is > probably > not very useful, and it seems Bob has supressed this warning in > py2app). I have disabled all missing module warnings, because I've found that are more of a nuisance than anything else. However, I have made it possible to generate a GraphViz DOT graph of everything that did get included, which is useful when you are having problems with too many or too few dependencies being included. Unlike modulefinder, py2app.modulegraph keeps track of why modules have been included, so it's easy enough to say that "pydoc should not depend on Tkinter", and Tkinter won't be included unless something else needs it. For cases like xmlrpclib->_xmlrpclib or xml -> _xmlplus, with a semi-standalone build (depends on an existing Python installation), a dotted edge will be drawn between the module that uses xmlrpclib and _xmlrpclib. The graph output only includes modules that ended up in the application bundle, so the edge is dotted to represent that the dependency exists, but was indirect (since xmlrpclib would not have been shown). That said, if you really wanted to display missing modules, you could find them pretty easily from the ModuleGraph instance, just walk it and keep instances of MissingModule. -bob From jasone at canonware.com Wed Oct 13 16:54:04 2004 From: jasone at canonware.com (Jason Evans) Date: Wed Oct 13 16:53:32 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <1f7befae041011203471688554@mail.gmail.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> Message-ID: <20041013145404.GB33145@canonware.com> On Mon, Oct 11, 2004 at 11:34:34PM -0400, Tim Peters wrote: > [Tim Peters] > >> I think you're in for a rude awakening if you expect any language's gc > >> support to help you manage arbitrary blobs of memory obtained from the > >> system malloc(). > > [Jason Evans] > > I don't expect that. What I do expect is for there to be clean and > > efficient mechanisms for integrating custom Python object types into > > Python. Python more or less meets this expectation. > > OK ... but you must want more than just that, else this thread > wouldn't have started . I initially started this thread thinking that I had some useful suggestions about how Python's GC might be modified. Had I realized in advance that my suggestions were based on my own misconceptions about Python's GC, I would not have bothered the people in this forum. I apologize for reducing the signal to noise ratio of python-dev. > > Many of the issues I ran into had to do with the impedence mismatch between > > my previous experience and the way Python works. > > I sure believe that. Python's cyclic gc is unusual in several respects. Indeed. This brings up a question about documentation: as far as I know, I've read all of the available documentation regarding cyclic GC (the "Supporting cyclic garbage collection" section of "Extending and Embedding the Python Interpreter"), but those docs came nowhere near giving me an adequate understanding of the design rationale behind the cyclic GC. Did I overlook some documentation? Of course, it may be that I am actually at a cognitive disadvantage to someone who has only experienced Python's memory management strategy... > > I appreciate you taking the time to respond to me, especially since you've > > helped me gain a deeper understanding of how Python's GC works. However, > > please get it out of your mind that I am incompetent and/or set in my ways. > > Those aren't in my head. I do think you're so steeped in the details > of your code that you're not nearly as successful at communicating the > *salient* points as you think you are, though. I'm so steeped in the > details of Python's gc that a similar claim could easily be made of > me. I don't expect it to be clear the first time, though, so play > 20-questions trying to tease out what you really need to know about > it. If it's any consolation, this process has been painful on both > sides . My critical error in trying to communicate the salient points of the Crux code was that I didn't realize the importance of comminicating the fact that the trees are implemented as a combination of separated low level tree data structures and Python wrappers to the low level implementation. Thanks for your help, Tim. Jason From aparente at adobe.com Wed Oct 13 17:15:50 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Wed Oct 13 17:16:26 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <416CBEE4.5040908@v.loewis.de> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> Message-ID: <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> Thanks all for your responses, I am not interested in Itanium, only in x86_64. Thanks again, alex. At 10:36 PM 10/12/2004, Martin v. L?wis wrote: >Alexandre Parenteau wrote: >>Sorry if I'm asking something that has been answered before, but I could >>not find a clear answer anywhere. >>Is there a project page for Windows 64bits? Is it already on CVS? >>My company tries to assess if we can plan on using python on Windows 64 bits. >>We want to do it natively (i.e. no 32b emulation), because we have some >>python extensions we want to use in the process. > >Itanium or AMD64? For Itanium, we have installers (of Python 2.4) available: > >http://www.python.org/2.4/ > >For AMD64, you need to compile it yourself. > >Regards, >Martin From aahz at pythoncraft.com Wed Oct 13 17:19:56 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Oct 13 17:19:59 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041013145404.GB33145@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> <20041013145404.GB33145@canonware.com> Message-ID: <20041013151956.GA9469@panix.com> On Wed, Oct 13, 2004, Jason Evans wrote: > > Indeed. This brings up a question about documentation: as far as I > know, I've read all of the available documentation regarding cyclic GC > (the "Supporting cyclic garbage collection" section of "Extending and > Embedding the Python Interpreter"), but those docs came nowhere near > giving me an adequate understanding of the design rationale behind the > cyclic GC. Did I overlook some documentation? > > Of course, it may be that I am actually at a cognitive disadvantage > to someone who has only experienced Python's memory management > strategy... As Tim said earlier, though less pungently, "Read the source, Luke." -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From nas at arctrix.com Wed Oct 13 17:25:26 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Wed Oct 13 17:25:31 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041013145404.GB33145@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> <20041013145404.GB33145@canonware.com> Message-ID: <20041013152526.GB595@mems-exchange.org> On Wed, Oct 13, 2004 at 07:54:04AM -0700, Jason Evans wrote: > This brings up a question about documentation: as far as I know, > I've read all of the available documentation regarding cyclic GC > (the "Supporting cyclic garbage collection" section of "Extending > and Embedding the Python Interpreter"), but those docs came > nowhere near giving me an adequate understanding of the design > rationale behind the cyclic GC. Did I overlook some > documentation? It's a pretty old document and perhaps a little out of date but here's something I wrote while working on the Python GC: http://arctrix.com/nas/python/gc/ Looking back, I realize that the constraints were pretty extreme which probably accounts for the unique implementation. A few off the top of my head: * binary compatible with previous version of Python * optional * small overhead for programs that don't create reference cycles * __del__ methods should still work Cheers, Neil From tim.peters at gmail.com Wed Oct 13 17:31:17 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 13 17:31:23 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: References: Message-ID: <1f7befae041013083132866606@mail.gmail.com> [Thomas Heller] ... > But subprocess, the Python 2.4 version, will *never* use it, because > _subprocess is always available (isn't this now even a builtin module, in > python24.dll?). Yes, it is. I haven't had time to read the PEP, but I assumed some "backward compatibility" constraint is at work here too, since, e.g., subprocess.py has try: False except NameError: False = 0 True = 1 and that doesn't make sense for 2.4 either. From barry at python.org Wed Oct 13 17:35:54 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 13 17:36:08 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <1f7befae041013083132866606@mail.gmail.com> References: <1f7befae041013083132866606@mail.gmail.com> Message-ID: <1097681754.8607.26.camel@geddy.wooz.org> On Wed, 2004-10-13 at 11:31, Tim Peters wrote: > Yes, it is. I haven't had time to read the PEP, but I assumed some > "backward compatibility" constraint is at work here too, since, e.g., > subprocess.py has > > try: > False > except NameError: > False = 0 > True = 1 > > and that doesn't make sense for 2.4 either. If so, then it should be documented in PEP 291, otherwise backward compatibility stuff is fair game for ripping out. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041013/f49b5d7e/attachment-0001.pgp From anthony at interlink.com.au Wed Oct 13 17:39:49 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 13 17:40:07 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> Message-ID: <416D4C45.3040802@interlink.com.au> Alexandre Parenteau wrote: > Thanks all for your responses, > > I am not interested in Itanium, only in x86_64. Python's worked on the Alphas for approximately ever, and works on Itanium, so I can't imagine you'll find any 32-bit-isms left in the code. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From jcarlson at uci.edu Wed Oct 13 17:37:38 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 13 17:45:21 2004 Subject: [Python-Dev] Struct module format code addition In-Reply-To: <001501c4b102$96de0480$e841fea9@oemcomputer> References: <20041012232318.091E.JCARLSON@uci.edu> <001501c4b102$96de0480$e841fea9@oemcomputer> Message-ID: <20041013081413.0927.JCARLSON@uci.edu> > Looking at the struct feature request, I think it would be harmless to > add it. OTOH, it is easily handled in python and will be easier still > when the binascii functions get put in. Arguably, the feature is > somewhat trivial and won't change anyone's life for good or ill. It makes processing one's data a 1-pass affair rather than a 2-pass affair. Not a big deal for most, but it gets me about a 20% speedup on a few formats, and saves me from writing custom translations every time I want some nonstandard sized integer. > So, you have to ask yourself whether it is worth jamming down everyone's > throats to get it in after the beta goes out. For my money, the feature > request has already consumed more developer resources than it could ever > save. Good points Raymond. If someone decides they want to add it, great, I'll even write some documentation and field sf.net bug reports and such in regards to the struct module. If not, I'll just use binascii and bite my tongue. My apologies for raising a stink. - Josiah From tim.peters at gmail.com Wed Oct 13 18:15:23 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 13 18:15:27 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) In-Reply-To: References: <416C114B.8000906@interlink.com.au> Message-ID: <1f7befae041013091525b1835@mail.gmail.com> [Fredrik Lundh] > can bug categories be associated with developers? Yes. For example, I see that the "Regular Expressions" category is auto-assigned to effbot. > if so, would it perhaps make sense to add Peter as a developer, That would be fine by me, provided he asks for it (it's a commitment, despite that developers routinely vanish for years at a time ). > and add a category for the subprocess system? That one's harder to call. Once a category is added, it can never be removed. So I'd wait on that to see whether enough reports came in against subprocess to justify it. Since it's unlikely that an original reporter will assign "the right" category to begin with, and the original submission is the only time at which auto-assignment has an effect, the major value in all this would be achieved simply by having a subprocess expert with commit privileges. From aparente at adobe.com Wed Oct 13 18:27:16 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Wed Oct 13 18:27:52 2004 Subject: [Python-Dev] About the Wise installer on Windows Message-ID: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Hi, Again forgive me if the answer is somewhere, but I could not find it. I am looking for a way to build and make an installer out of Python on a Win x86_64 platform which is not supported yet by the Python team. My plan is to compile Python for this platform (several developers on the list told me it should be straightforward), and make an installer out of the Python binaries in order to be able to distribute the Python binaries internally. I downloaded the evaluation copy of Wise Installer System 9, but it complains that the installer file (which I assume is PCBuild\Python20.wse) is corrupted. I need to establish exactly the version of the installer to use, if we need to acquire a license for the Wise installer. - Could someone give me some pointer about the exact version of the Wise Installer to use in order to be able to replicate the same installer that the Python team is distributing on Windows? - The ISS files (Inno) do not seem up to date. Is anybody having a ISS version of the installer which works for 2.3.4? Thanks, alex. From anthony at interlink.com.au Wed Oct 13 19:10:48 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 13 19:10:59 2004 Subject: [Python-Dev] 2.4b1 release, this Friday (AU time) -- imminent TRUNK FREEZING Message-ID: <416D6198.5000606@interlink.com.au> Ok, I'm going to be cutting the Python release Friday (Australian time). This means that I'm declaring the trunk frozen for the release in about 24 hours. Lets say at 18:00 hours UTC on Thursday 14th (that's 14:00 on the US east coast, if my memory is correct, and about 04:00 where here in Australia). No-one except for the usual team (Fred, Martin, myself) should check into the trunk after that. As per usual, I'll probably keep the trunk frozen for a time after the release, in case of brown-paper-bag issues. Note that this is the last chance you have to get those behaviour- changing bugs^Wfixes introduced, so get to it! After b1, it will become _much_ harder to get new fixes introduced. And if there's a pending bug or patch that needs some lovin', add a comment to it or mail me - I think I've responded to, or fixed, all the ones I've been told about, but if I missed one, tell me now. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From astrand at lysator.liu.se Wed Oct 13 19:50:54 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 19:51:07 2004 Subject: [Python-Dev] test_subprocess fails on OSX In-Reply-To: <416D3AAE.6090907@interlink.com.au> References: <416D3AAE.6090907@interlink.com.au> Message-ID: On Thu, 14 Oct 2004, Anthony Baxter wrote: > On OSX 10.3.4, test_subprocess is failing intermittently when run > from regrtest: > > test_subprocess > this bit of output is from a test of stdout in a different process ... > test test_subprocess failed -- Traceback (most recent call last): > File "/Users/anthonybaxter/pyhead/src/Lib/test/test_subprocess.py", > line 434, in test_close_fds > self.assertEqual(p.stdout.read(), "3") > AssertionError: '4' != '3' > > Running just the test by itself works. I'm guessing it could be part of > the test suite leaking a FD or something? That shouldn't matter, since when you are using close_fds=True, the subprocess module will close all filedescriptors. For example, you can insert additional lines with os.pipe() at the beginning of test_close_fds, and the test (should) still succeed(s). /Peter ?strand From astrand at lysator.liu.se Wed Oct 13 19:56:06 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 19:56:12 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <1f7befae041013083132866606@mail.gmail.com> References: <1f7befae041013083132866606@mail.gmail.com> Message-ID: On Wed, 13 Oct 2004, Tim Peters wrote: > > But subprocess, the Python 2.4 version, will *never* use it, because > > _subprocess is always available (isn't this now even a builtin module, in > > python24.dll?). > > Yes, it is. I haven't had time to read the PEP, but I assumed some > "backward compatibility" constraint is at work here too, since, e.g., > subprocess.py has > > try: > False The PEP doesn't say anything about this, but my intention is that the module should work on Python 2.2 and newer. My employer really needs this. If the subprocess module in the Python CVS won't work with 2.2, I'm forced to keep a separate "fork". I would be happy if this could be avoided. /Peter ?strand From astrand at lysator.liu.se Wed Oct 13 20:03:18 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 20:03:31 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <1097681754.8607.26.camel@geddy.wooz.org> References: <1f7befae041013083132866606@mail.gmail.com> <1097681754.8607.26.camel@geddy.wooz.org> Message-ID: >> Yes, it is. I haven't had time to read the PEP, but I assumed some >> "backward compatibility" constraint is at work here too, since, e.g., >> subprocess.py has >If so, then it should be documented in PEP 291, otherwise backward >compatibility stuff is fair game for ripping out. Below is a trivial patch to PEP 291. I'm stepping forward as a maintainer, if noone else wants to do the job. diff -u -r1.11 pep-0291.txt --- pep-0291.txt 20 Aug 2004 07:32:06 -0000 1.11 +++ pep-0291.txt 13 Oct 2004 18:00:14 -0000 @@ -90,6 +90,7 @@ modulefinder Thomas Heller 2.2 Just van Rossum platform Marc-Andre Lemburg 1.5.2 + subprocess Peter Astrand 2.2 Tool Maintainer(s) Python Version /Peter ?strand From barry at python.org Wed Oct 13 20:07:17 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 13 20:07:22 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: References: <1f7befae041013083132866606@mail.gmail.com> Message-ID: <1097690837.8609.39.camel@geddy.wooz.org> On Wed, 2004-10-13 at 13:56, Peter Astrand wrote: > The PEP doesn't say anything about this, but my intention is that the > module should work on Python 2.2 and newer. My employer really needs this. > If the subprocess module in the Python CVS won't work with 2.2, I'm > forced to keep a separate "fork". I would be happy if this could be > avoided. There's no reason to fork it. (Reasonable) backward compatibility in the 2.4 library can be supported as long as it's documented, which is now is. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041013/dc2f70c5/attachment-0001.pgp From theller at python.net Wed Oct 13 20:12:33 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 20:12:19 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> (Alexandre Parenteau's message of "Wed, 13 Oct 2004 09:27:16 -0700") References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Message-ID: Alexandre Parenteau writes: > Hi, > > Again forgive me if the answer is somewhere, but I could not find it. > > I am looking for a way to build and make an installer out of Python on > a Win x86_64 platform which is not supported yet by the Python team. > > My plan is to compile Python for this platform (several developers on > the list told me it should be straightforward), and make an installer > out of the Python binaries in order to be able to distribute the > Python binaries internally. > > I downloaded the evaluation copy of Wise Installer System 9, but it > complains that the installer file (which I assume is > PCBuild\Python20.wse) is corrupted. > > I need to establish exactly the version of the installer to use, if we > need to acquire a license for the Wise installer. Wise Installation System 9.0 - standard edition is what is used to build the Python 2.3 installers. The problem that you see has the usual cause - the PCBuild\Python20.wse file in the Python-2.3.4.tgz file, downloaded from Python.org, has unix line endings instead of dos line endings. You should convert them (loading the file into write, and saving it again) and it should load fine. Thomas From astrand at lysator.liu.se Wed Oct 13 20:15:18 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 20:15:26 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) In-Reply-To: <1f7befae041013091525b1835@mail.gmail.com> References: <416C114B.8000906@interlink.com.au> <1f7befae041013091525b1835@mail.gmail.com> Message-ID: On Wed, 13 Oct 2004, Tim Peters wrote: > > if so, would it perhaps make sense to add Peter as a developer, > > That would be fine by me, provided he asks for it (it's a commitment, > despite that developers routinely vanish for years at a time ). I'm planning to be around for some time :) Developer access would be good, if I should maintain the subprocess module. (My SF account is "astrand".) > > and add a category for the subprocess system? > > That one's harder to call. Once a category is added, it can never be > removed. So I'd wait on that to see whether enough reports came in > against subprocess to justify it. Since it's unlikely that an > original reporter will assign "the right" category to begin with, and > the original submission is the only time at which auto-assignment has > an effect, the major value in all this would be achieved simply by > having a subprocess expert with commit privileges. Sounds good to me. /Peter ?strand From martin at v.loewis.de Wed Oct 13 20:51:46 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 20:51:46 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041012232318.091E.JCARLSON@uci.edu> References: <20041011232519.08C8.JCARLSON@uci.edu> <416CBBB0.9040809@v.loewis.de> <20041012232318.091E.JCARLSON@uci.edu> Message-ID: <416D7942.2000005@v.loewis.de> Josiah Carlson wrote: >>For those features, the >>potential user community is *much* larger than for the feature under >>discussion, and I feel I would waste my time for adding a feature that >>only few users ever need. > > > And yet you continue the discussion? This tends to get off-topic, but I typically respond when I'm asked a question. As you keep asking questions, I keep responding :-) > If every modification to Python required a community of people, who all > joined together to advocate something, then nothing would ever get done. No. Modifications to fix bugs don't need debating. Only new features do. I do wish that people would focus more on fixing bugs than on introducing new features. > People would be spending all their time trying to gather groups of > people to agree with them that functionality K is necessary. And that would be a good thing, atleast in the specific case. Even if it is undebated that the feature (convert numbers into odd-sized byte strings) is desirable, the specific API needs careful consideration, since it cannot be changed easily after it has been added. So all these new features are a serious legacy if they later turn out to be ill-designed. > I'm not even saying that we should add a new module. I'm not even > saying we should add a new function to a module. Heck, I'm not even > asking for a new argument to a function that previously existed. I am, > quite literally, asking for the equivalent of less than one bit in a > flag (there are currently 22 format/endian characters, which are all > orthogonal, which would require 5 bits, if they were flags). Correct. Yet I would feel more comfortable if you had proposed a new module, or a new function to an existing module. The former would allow distribution independent of Python, and the latter would not put a burden on users of the existing functions. > The API already exists. The framework already exists. I'm not asking > for Python to interpret something that was valid before as something > different now. Yes. The extension you propose probably does not cause backward compatibility problems. Strictly speaking, there are programs that break under this extension, but this is the case for any extension, and such programs likely don't occur in real life. > Try to read the above paragraph outside of the context of struct and the > RFE. Does it make sense to include the change now? No. For such a change, we need to study whether there alternative APIs which achieve the same effect (which there are), and whether the new flag puts an additional learning burden on users of the existing API (which it does). > If every request to interpret a new flag required significant community > involvement, goodness, would it take an act of Guido to get a commit > done? No, I have committed new features myself in the past. See the CVS log for details. It is this specific change I'm opposed to, at this point in time. Regards, Martin From martin at v.loewis.de Wed Oct 13 21:06:04 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 21:06:03 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <416D4C45.3040802@interlink.com.au> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> <416D4C45.3040802@interlink.com.au> Message-ID: <416D7C9C.5080600@v.loewis.de> Anthony Baxter wrote: > Python's worked on the Alphas for approximately ever, and > works on Itanium, so I can't imagine you'll find any 32-bit-isms > left in the code. Actually, there are plenty of them, however it is unlikely that anybody will encounter them for a few more years. The biggest problem is that the number of elements in a sequence (string, list, tuple) is represented as "int", whereas it should be a signed variant of size_t. I have a patch that corrects many of these issues, but I plan to publish that only after 2.4. The problem becomes only apparent if you have more than 2G items. For a string, this might be a significant limitation. For a list, it requires 16GiB of memory to represent the list alone, not accounting for any items in the list, so very few people have sufficient hardware to even run a test case that demonstrates the problem. As for Alphas: their Windows port was a 32-bit port; Microsoft did not have Win64 at that time. Regards, Martin From martin at v.loewis.de Wed Oct 13 21:15:08 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 21:15:14 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Message-ID: <416D7EBC.5040906@v.loewis.de> Alexandre Parenteau wrote: > I am looking for a way to build and make an installer out of Python on a > Win x86_64 platform which is not supported yet by the Python team. Out of curiosity: where did you obtain the operating system, and where will you obtain a C compiler for the platform? To my knowledge, Microsoft has released neither, yet. > My plan is to compile Python for this platform (several developers on > the list told me it should be straightforward), and make an installer > out of the Python binaries in order to be able to distribute the Python > binaries internally. > > I downloaded the evaluation copy of Wise Installer System 9, but it > complains that the installer file (which I assume is > PCBuild\Python20.wse) is corrupted. Alternatively to that approach, you can use the Python 2.4 code base, and build an MSI file (using Tools/msi); no need for any external tool except for a platform SDK installation, and PythonWin. Regards, Martin From nbastin at opnet.com Wed Oct 13 21:31:38 2004 From: nbastin at opnet.com (Nick Bastin) Date: Wed Oct 13 21:32:00 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <416D7EBC.5040906@v.loewis.de> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> Message-ID: <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> On Oct 13, 2004, at 3:15 PM, Martin v. L?wis wrote: > Alexandre Parenteau wrote: >> I am looking for a way to build and make an installer out of Python >> on a Win x86_64 platform which is not supported yet by the Python >> team. > > Out of curiosity: where did you obtain the operating system, and where > will you obtain a C compiler for the platform? To my knowledge, > Microsoft has released neither, yet. Microsoft made available an OS to the general public (2003 Pro, I think) for x86_64 at *least* 6 months ago, maybe more, and MSDN members have access to other OS builds as well (at least 2003 server). You have at least 2 compiler options, namely MSVC++ and GCC. ICL does not (yet) support x64_64. -- Nick From jack at performancedrivers.com Wed Oct 13 22:05:37 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Wed Oct 13 22:05:42 2004 Subject: [Python-Dev] 2.4b1 release, this Friday (AU time) -- imminent TRUNK FREEZING In-Reply-To: <416D6198.5000606@interlink.com.au> References: <416D6198.5000606@interlink.com.au> Message-ID: <20041013200537.GJ10507@performancedrivers.com> On Thu, Oct 14, 2004 at 03:10:48AM +1000, Anthony Baxter wrote: > Ok, I'm going to be cutting the Python release Friday (Australian > time). This means that I'm declaring the trunk frozen for the release > in about 24 hours. Lets say at 18:00 hours UTC on Thursday 14th > (that's 14:00 on the US east coast, if my memory is correct, and > about 04:00 where here in Australia). > US East Coast is currently GMT -400, West Coast is -700 I never pretend to know how daylight savings time works but I don't think it will change on a Friday. -Jack ps, flying from Philly to Syndey (layover in LAX) has the fun effect of taking no "time" going one way, and taking two "days" going the other. From martin at v.loewis.de Wed Oct 13 22:11:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 22:11:11 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> Message-ID: <416D8BE0.3000305@v.loewis.de> Nick Bastin wrote: > Microsoft made available an OS to the general public (2003 Pro, I think) > for x86_64 at *least* 6 months ago, maybe more, and MSDN members have > access to other OS builds as well (at least 2003 server). That was no release (in the sense of a product release) of the system, though, was it? My understanding that the system is still considered in beta by Microsoft. > You have at > least 2 compiler options, namely MSVC++ and GCC. ICL does not (yet) > support x64_64. How can you get a copy of MSVC++? VC.NET 2003 does not come with an AMD64 compiler to my knowledge, nor does the February 2003 (?) platform SDK incorporate one (which does incorporate an Itanium compiler). Regards, Martin From tim.peters at gmail.com Wed Oct 13 22:21:04 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 13 22:21:11 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Message-ID: <1f7befae04101313214c5e4904@mail.gmail.com> [Alexandre Parenteau] > ... > - The ISS files (Inno) do not seem up to date. FYI, they were never up to date, and were never used to build a released installer. > Is anybody having a ISS version of the installer which works for 2.3.4? Sorry, I don't know of an Inno installer for any version of Python. Because there was a minor (but endless) stream of complaints that building the Python Windows installer required Wise, I checked in the "most of the way there, at the time" .iss file so that someone else could volunteer to finish it. Since nobody cared enough to do so, I ignored such complaints ever after . From nbastin at opnet.com Wed Oct 13 22:30:31 2004 From: nbastin at opnet.com (Nick Bastin) Date: Wed Oct 13 22:30:50 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <416D8BE0.3000305@v.loewis.de> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> Message-ID: On Oct 13, 2004, at 4:11 PM, Martin v. L?wis wrote: > Nick Bastin wrote: >> Microsoft made available an OS to the general public (2003 Pro, I >> think) for x86_64 at *least* 6 months ago, maybe more, and MSDN >> members have access to other OS builds as well (at least 2003 >> server). > > That was no release (in the sense of a product release) of the system, > though, was it? My understanding that the system is still considered > in beta by Microsoft. Yeah, that's why I didn't say 'released'. It's still a beta, but it certainly works and people are doing software development on it. My point was that people definitely have windows for x86_64, so it doesn't seem that unreasonable that they'd be trying to use Python on it - this isn't an isolated case. >> You have at least 2 compiler options, namely MSVC++ and GCC. ICL >> does not (yet) support x64_64. > > How can you get a copy of MSVC++? VC.NET 2003 does not come with an > AMD64 compiler to my knowledge, nor does the February 2003 (?) platform > SDK incorporate one (which does incorporate an Itanium compiler). There is a version of MSVC on the beta DDK CD. -- Nick From martin at v.loewis.de Wed Oct 13 22:59:45 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 22:59:44 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> Message-ID: <416D9741.4080004@v.loewis.de> Nick Bastin wrote: > Yeah, that's why I didn't say 'released'. It's still a beta, but it > certainly works and people are doing software development on it. My > point was that people definitely have windows for x86_64, so it doesn't > seem that unreasonable that they'd be trying to use Python on it - this > isn't an isolated case. I didn't mean to suggest it was unreasonable. I was just wondering what precisely you need (in addition to the actual hardware, of course). Thanks for your elaboration, Martin From jcarlson at uci.edu Wed Oct 13 23:01:12 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 13 23:08:33 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <416D7942.2000005@v.loewis.de> References: <20041012232318.091E.JCARLSON@uci.edu> <416D7942.2000005@v.loewis.de> Message-ID: <20041013130013.0944.JCARLSON@uci.edu> > > People would be spending all their time trying to gather groups of > > people to agree with them that functionality K is necessary. > > And that would be a good thing, atleast in the specific case. Even > if it is undebated that the feature (convert numbers into odd-sized > byte strings) is desirable, the specific API needs careful > consideration, since it cannot be changed easily after it has been > added. So all these new features are a serious legacy if they later > turn out to be ill-designed. If it turns out that the feature is ill-designed, then there is a standard method of dealing with it: 1. Declare that it will be Deprecated in the future. 2. Start raising a DeprecationWarning for at least one major version of Python. 3. Remove the functionality In the case of struct and this addition, if it is the case that this modification is catastrophic to Python (very unlikely), I would hope that someone would have said, "hey, this breaks applications A, B, C, D, ..., and it inclusion was the biggest mistake Python has ever made, even moreso than anyone ever responding to Josiah* in the first place." * Me trying to be funny. > > I'm not even saying that we should add a new module. I'm not even > > saying we should add a new function to a module. Heck, I'm not even > > asking for a new argument to a function that previously existed. I am, > > quite literally, asking for the equivalent of less than one bit in a > > flag (there are currently 22 format/endian characters, which are all > > orthogonal, which would require 5 bits, if they were flags). > > Correct. Yet I would feel more comfortable if you had proposed a new > module, or a new function to an existing module. The former would allow > distribution independent of Python, and the latter would not put a > burden on users of the existing functions. For those who use struct as it is now, the proposed additional format codes raise a struct.error exception in older versions of Python. If people are relying on pack/unpack with characters 'g' or 'G' raising a struct.error exception, then something is definitely wrong with the way they have been using struct. If people want to use the modifications with a previous version of Python, the changes are easily backported (I think one can even take structmodule.c from CVS and compile it with an older source tree). > Yes. The extension you propose probably does not cause backward > compatibility problems. Strictly speaking, there are programs that > break under this extension, but this is the case for any extension, > and such programs likely don't occur in real life. Agreed. > No. For such a change, we need to study whether there alternative APIs > which achieve the same effect (which there are), and whether the new > flag puts an additional learning burden on users of the existing API > (which it does). Tim Peters' alternative long2bytes and bytes2long looked quite usable, and indeed produces strikingly similar results, for single item packing and unpacking scenarios. The learning curve of struct was claimed to be steep in the case of native byte alignments, or for those not familliar with C types or structs. Since these codes are not native to any platform, it doesn't make the native byte alignment stuff any more difficult. If people are having trouble understanding the concept of C structs, I don't believe that two new format characters are going to be significantly more difficult to swallow than the 17 others that already exist. Especially when the standard C type codes 'cbhilqspBHILQ' are likely sufficient for their needs. In the case of them not being sufficient, well hey, they would have another two options that may do it for them. > > If every request to interpret a new flag required significant community > > involvement, goodness, would it take an act of Guido to get a commit > > done? > > No, I have committed new features myself in the past. See the CVS log > for details. It is this specific change I'm opposed to, at this point > in time. I was trying to be funny, to perhaps lighten the mood. Hence "act of Guido" rather than "act of God" (where the latter is a cliche). - Josiah From aparente at adobe.com Wed Oct 13 23:29:12 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Wed Oct 13 23:29:51 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <416D8BE0.3000305@v.loewis.de> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> Message-ID: <6.1.1.1.2.20041013135327.03c96e60@mailsj.corp.adobe.com> Thanks for all your responses, Unfortunately I don't know the specifics of how to get the compilers and OS. I am only in charge to find out if Python-Dev has made already the work. I suspect you get the OS and compilers from the MSDN Subscriber program. I will certainly let you know of our experience if/when we decide to compile Python on x86_64. It might not be before a while though, so probably someone else will be able to report before us. Thanks again, alex. At 01:11 PM 10/13/2004, Martin v. L?wis wrote: >Nick Bastin wrote: >>Microsoft made available an OS to the general public (2003 Pro, I think) >>for x86_64 at *least* 6 months ago, maybe more, and MSDN members have >>access to other OS builds as well (at least 2003 server). > >That was no release (in the sense of a product release) of the system, >though, was it? My understanding that the system is still considered >in beta by Microsoft. > >>You have at least 2 compiler options, namely MSVC++ and GCC. ICL does >>not (yet) support x64_64. > >How can you get a copy of MSVC++? VC.NET 2003 does not come with an >AMD64 compiler to my knowledge, nor does the February 2003 (?) platform >SDK incorporate one (which does incorporate an Itanium compiler). > >Regards, >Martin From abkhd at hotmail.com Thu Oct 14 00:05:11 2004 From: abkhd at hotmail.com (A.B., Khalid) Date: Thu Oct 14 00:06:06 2004 Subject: [Python-Dev] Inno-Setup for Python 2.3.4 Message-ID: [Alexandre Parenteau] >... >- The ISS files (Inno) do not seem up to date.Is anybody having a ISS >version of the installer which works for 2.3.4? Hello Alex, I have been working on and off on an Inno-Setup installer for the MinGW compiled Python I've been working on. The reason being I have Win98 so the Msi.py approach does not work. The script is still not finished, nor is it identical to setup script found in the Python 2.3.4 sources. Moreover, it does not make any registry entries yet, which I am thinking and hoping of doing either through a Python script (which runs after install is finished and before uninstall is started) or an external DLL so as to be able to revert changes. But even in its current unfinished form, the script works for me in that it gathers all of the needed Python 2.3.4 files into one package and offers to install Python and Python's extensions, and uninstalls the files as needed. All the regrtest tests pass when Python 2.3.4 is installed from that script. One can run Python's scripts from the DOS Shell and if file association is urgently needed then hints from the Python.iss found in the original Python source distribution can be of immediate help. Get it here (3.49 KB): http://jove.prohosting.com/iwave/ipython/Inno-Python234.zip Regards Khalid PS: I have pymingw.py which is enclosed in the zip file where I have python23.dll and friends. It will be picked up by the script from there and placed in site-packages for use when uninstalling Python. _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/ From anthony at interlink.com.au Thu Oct 14 05:04:42 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Oct 14 05:05:02 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <416D7C9C.5080600@v.loewis.de> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> <416D4C45.3040802@interlink.com.au> <416D7C9C.5080600@v.loewis.de> Message-ID: <416DECCA.6030408@interlink.com.au> Martin v. L?wis wrote: > As for Alphas: their Windows port was a 32-bit port; Microsoft > did not have Win64 at that time. However, the Unix port was 64 bits. -- Anthony Baxter It's never too late to have a happy childhood. From tim.peters at gmail.com Thu Oct 14 05:16:42 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Oct 14 05:16:45 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) In-Reply-To: References: <416C114B.8000906@interlink.com.au> <1f7befae041013091525b1835@mail.gmail.com> Message-ID: <1f7befae041013201677f808ed@mail.gmail.com> [/F] >>> if so, would it perhaps make sense to add Peter as a developer, [Uncle Timmy] >> That would be fine by me, provided he asks for it (it's a commitment, >> despite that developers routinely vanish for years at a time ). [Peter Astrand] > I'm planning to be around for some time :) Developer access would be good, > if I should maintain the subprocess module. (My SF account is "astrand".) If there are no objections before I wake up again, and nobody else has done it, I'll add Peter as a Python developer. From python at rcn.com Thu Oct 14 05:35:01 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Oct 14 05:36:32 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess -updated popen5module) In-Reply-To: <1f7befae041013201677f808ed@mail.gmail.com> Message-ID: <000001c4b19e$c9f5e120$e841fea9@oemcomputer> > [/F] > >>> if so, would it perhaps make sense to add Peter as a developer, > > [Uncle Timmy] > >> That would be fine by me, provided he asks for it (it's a commitment, > >> despite that developers routinely vanish for years at a time ). > > [Peter Astrand] > > I'm planning to be around for some time :) Developer access would be > good, > > if I should maintain the subprocess module. (My SF account is > "astrand".) > > If there are no objections before I wake up again, and nobody else has > done it, I'll add Peter as a Python developer. Peter, your permissions are enabled. Welcome. Raymond From nbastin at opnet.com Wed Oct 13 23:40:45 2004 From: nbastin at opnet.com (Nick Bastin) Date: Thu Oct 14 06:26:18 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013135327.03c96e60@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> <6.1.1.1.2.20041013135327.03c96e60@mailsj.corp.adobe.com> Message-ID: <8A2D172C-1D60-11D9-8B3F-000D932927FE@opnet.com> On Oct 13, 2004, at 5:29 PM, Alexandre Parenteau wrote: > Thanks for all your responses, > > Unfortunately I don't know the specifics of how to get the compilers > and OS. I am only in charge to find out if Python-Dev has made already > the work. > > I suspect you get the OS and compilers from the MSDN Subscriber > program. > > I will certainly let you know of our experience if/when we decide to > compile Python on x86_64. It might not be before a while though, so > probably someone else will be able to report before us. Oh, well, if you just wanted to know if it worked... :-) It does work. IIRC, you need to play with pyconfig.h a bit for a few 64-bit things, but it pretty much works out of the box. It might be easier at this point to check out the pyconfig.h that's used for the Itanium build - I don't think that was available when I did the first AMD64 build here. -- Nick From bac at OCF.Berkeley.EDU Thu Oct 14 07:20:28 2004 From: bac at OCF.Berkeley.EDU (Brett C) Date: Thu Oct 14 07:20:36 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] Message-ID: <416E0C9C.1020005@ocf.berkeley.edu> So we were all rather quiet in the last half of September. The whole summary fits on two sheets of 8.5x11 (normally it is over 10 and I have hit over 20 when I was summarizing *everything*). Going to send this out no earlier than Friday night so send in corrections by then. ---------------------------------- ===================== Summary Announcements ===================== Wow. This must have been the easiest summary I have ever done. Why can't they all be like this? I didn't even skip that much! ========= Summaries ========= ------------------------------------------ Assume nothing when mutability is possible ------------------------------------------ Tim Peters discovered a new way to create an infinite list thanks to generator expressions. But what really came out of this whole discussion came about when someone else came up with an example that exposed a bug in list.extend(). The first thing was that "you can't assume anything about a mutable object after potentially calling back into Python." Basically you can't assume the state of any mutable object was not changed if you execute Python code from C. While it might seem handy to store state while in a loop for instance, you can't count on things not change by the time you get control back so you just have to do it the hard way and get state all over again. Second was that you need to be careful when dealing with iterators. If you mutate an iterator while iterating you don't have a guarantee it won't explode in your face. Unless you explicitly support it, document it, and take care to protect against it then just don't assume you can mutate an iterator while using it. Contributing threads: - `A cute new way to get an infinite loop <>`__ - `More data points <>`__ ---------------------------- The less licenses the better ---------------------------- The idea of copying some code from OpenSSH_ for better pty handling was proposed. This was frowned upon since that becomes one more legal issue to keep track of. Minimizing the licenses that Python must keep track of and make sure to comply with, no matter how friendly, is a good thing. .. _OpenSSH: http://www.openssh.com/ Contributing threads: - `using openssh's pty code <>`__ ------------------------------------------------------------------------ Trying to deal with the exception hierarchy and a backwards-friendly way ------------------------------------------------------------------------- Nick Coghlan came up with the idea of having a tuple that contained all of the exceptions you normally would not want to catch in a blanket 'except' statement; KeyboardInterrupt, MemoryError, SystemExit, etc.). This tuple was proposed to live in sys.special_exceptions with the intended usage of:: try: pass # stuff... except sys.special_exceptions: raise # exceptions that you would not want to catch should keep propogating up the call chain except: pass # if you reach here the exception should not be a *huge* deal Obviously the best solution is to just clean up the exception hierarchy, but that breaks backwards-compatibility. But this idea seemed to lose steam. Contributing threads: - `Proposing a sys.special_exceptions tuple <>`__ =============== Skipped Threads =============== - Decimal, copyright and license - Planning to drop gzip compression for future releases. - built on beer? - Noam's open regex requests - Socket/Asyncore bug needs attention - open('/dev/null').read() -> MemoryError - Finding the module from PyTypeObject? - Odd compile errors for bad genexps - Running a module as a script From python-dev-list at cstork.org Thu Oct 14 07:34:09 2004 From: python-dev-list at cstork.org (Christian Stork) Date: Thu Oct 14 07:34:13 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 In-Reply-To: <416C24E2.4020304@alumni.sfu.ca> References: <416C24E2.4020304@alumni.sfu.ca> Message-ID: <20041014053409.GA24001@anthony.ics.uci.edu> On Tue, Oct 12, 2004 at 08:39:30PM +0200, Aaron Bingham wrote: > Fredrik Lundh wrote: ... > >I don't know how to add ISO-8859-1 characters to a Latex document, > >but in HTML notation, Peter's last name should be Åstrand. > > > > > I don't know if it's allowed for for Python docs, but the easiest way I > know of is to put > \usepackage{isolatin1} > in the prelude and just use regular ISO-8859-1 characters in the file. isolatin1.sty is obsolete. Use inputenc.sty instead. See page 11 of http://www.tug.org/tex-archive/info/l2tabu/english/l2tabuen.pdf for further details. -- Chris Stork <> Support eff.org! <> http://www.ics.uci.edu/~cstork/ OpenPGP fingerprint: B08B 602C C806 C492 D069 021E 41F3 8C8D 50F9 CA2F From ncoghlan at iinet.net.au Thu Oct 14 14:01:37 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 14 14:00:35 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <416E0C9C.1020005@ocf.berkeley.edu> References: <416E0C9C.1020005@ocf.berkeley.edu> Message-ID: <416E6AA1.2030107@iinet.net.au> Brett C wrote: > - Running a module as a script That reminds me - the version of this that got checked in is restricted to top-level modules in order to keep things simple. I put a recipe up on the Python cookbook for those that wanted to be able to easily run scripts that live inside packages. The recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 Cheers, Nick. From barry at python.org Thu Oct 14 15:00:14 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 15:00:18 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <416E6AA1.2030107@iinet.net.au> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> Message-ID: <1097758814.7686.31.camel@geddy.wooz.org> On Thu, 2004-10-14 at 08:01, Nick Coghlan wrote: > That reminds me - the version of this that got checked in is restricted > to top-level modules in order to keep things simple. I put a recipe up > on the Python cookbook for those that wanted to be able to easily run > scripts that live inside packages. > > The recipe: > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 Why not add this to the stdlib? I'd call it pyrun.py. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/9795713e/attachment.pgp From theller at python.net Thu Oct 14 15:06:07 2004 From: theller at python.net (Thomas Heller) Date: Thu Oct 14 15:05:50 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <1097758814.7686.31.camel@geddy.wooz.org> (Barry Warsaw's message of "Thu, 14 Oct 2004 09:00:14 -0400") References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> Message-ID: Barry Warsaw writes: > On Thu, 2004-10-14 at 08:01, Nick Coghlan wrote: > >> That reminds me - the version of this that got checked in is restricted >> to top-level modules in order to keep things simple. I put a recipe up >> on the Python cookbook for those that wanted to be able to easily run >> scripts that live inside packages. >> >> The recipe: >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 > > Why not add this to the stdlib? I'd call it pyrun.py. ... and call it automatically when the '-m' implementation detects a dotted name? Thomas From mwh at python.net Thu Oct 14 15:35:23 2004 From: mwh at python.net (Michael Hudson) Date: Thu Oct 14 15:35:25 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041013151956.GA9469@panix.com> (aahz@pythoncraft.com's message of "Wed, 13 Oct 2004 11:19:56 -0400") References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> <20041013145404.GB33145@canonware.com> <20041013151956.GA9469@panix.com> Message-ID: <2mhdoxfmn8.fsf@starship.python.net> Aahz writes: > On Wed, Oct 13, 2004, Jason Evans wrote: >> >> Indeed. This brings up a question about documentation: as far as I >> know, I've read all of the available documentation regarding cyclic GC >> (the "Supporting cyclic garbage collection" section of "Extending and >> Embedding the Python Interpreter"), but those docs came nowhere near >> giving me an adequate understanding of the design rationale behind the >> cyclic GC. Did I overlook some documentation? >> >> Of course, it may be that I am actually at a cognitive disadvantage >> to someone who has only experienced Python's memory management >> strategy... > > As Tim said earlier, though less pungently, "Read the source, Luke." Indeed, but also: don't be scared! Modules/gcmodule.c isn't that long, and much of what it is is comments. Mind you, it was probably easier to understand before it worked properly with weak references... Cheers, mwh -- Unfortunately, nigh the whole world is now duped into thinking that silly fill-in forms on web pages is the way to do user interfaces. -- Erik Naggum, comp.lang.lisp From just at letterror.com Thu Oct 14 15:49:44 2004 From: just at letterror.com (Just van Rossum) Date: Thu Oct 14 15:49:40 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <416E6AA1.2030107@iinet.net.au> Message-ID: Nick Coghlan wrote: > Brett C wrote: > > - Running a module as a script > > That reminds me - the version of this that got checked in is > restricted to top-level modules in order to keep things simple. I put > a recipe up on the Python cookbook for those that wanted to be able > to easily run scripts that live inside packages. > > The recipe: > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 (Hm, I was going to reply to the discussion there, but I think it's more appropriate here.) I understand the difficulty of implementing it, but as a user I find it a really really stupid restriction. I routively run (doc)tests of individual modules, which usually are submodules of a package. Using -m to do this would help me tremendously. As it stands, -m doesn't help me at all. I'd even go so far and -1 the entire feature if it doesn't support submodules. I guess it's too late for that :) Just From barry at python.org Thu Oct 14 15:58:06 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 15:58:13 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> Message-ID: <1097762286.8756.12.camel@geddy.wooz.org> On Thu, 2004-10-14 at 09:06, Thomas Heller wrote: > ... and call it automatically when the '-m' implementation detects a > dotted name? +1. Also, if a non-dotted module name can't be found, pyrun should be called to see if the module is actually a package (with an __main__ that lives in the package's __init__.py). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/cded5503/attachment.pgp From bob at redivi.com Thu Oct 14 16:04:42 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Oct 14 16:05:20 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <1097762286.8756.12.camel@geddy.wooz.org> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <1097762286.8756.12.camel@geddy.wooz.org> Message-ID: On Oct 14, 2004, at 9:58 AM, Barry Warsaw wrote: > On Thu, 2004-10-14 at 09:06, Thomas Heller wrote: > >> ... and call it automatically when the '-m' implementation detects a >> dotted name? > > +1. Also, if a non-dotted module name can't be found, pyrun should be > called to see if the module is actually a package (with an __main__ > that > lives in the package's __init__.py). Wouldn't it make more sense to look for main rather than __main__? It seems that __main__ (as a function) occurs EXACTLY ZERO times in the standard library ;) -bob From barry at python.org Thu Oct 14 16:14:12 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 16:14:16 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <1097762286.8756.12.camel@geddy.wooz.org> Message-ID: <1097763252.8756.29.camel@geddy.wooz.org> On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: > > +1. Also, if a non-dotted module name can't be found, pyrun should be > > called to see if the module is actually a package (with an __main__ > > that > > lives in the package's __init__.py). > > Wouldn't it make more sense to look for main rather than __main__? It > seems that __main__ (as a function) occurs EXACTLY ZERO times in the > standard library ;) I think that's a fine convention, so +1. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/2ad7d4e6/attachment.pgp From hpk at trillke.net Thu Oct 14 17:10:10 2004 From: hpk at trillke.net (holger krekel) Date: Thu Oct 14 17:10:13 2004 Subject: [Python-Dev] PyPy Vilnius Sprint 15-23 nov 2004 Message-ID: <20041014151010.GQ15456@solar.trillke.net> Hi Pythonistas and interested developers, PyPy, the python-in-python implementation, is steadily moving on. The next coding sprint will take place in Vilnius, Lithunia, from 15th to 23rd of November, 2004 and is organized by the nice Programmers of Vilnius (POV) company. See http://codespeak.net/pypy/index.cgi?doc for more in-depth information about PyPy. Again, we will be heading towards a first generated C version of our already pretty compliant Python interpreter and types implementation. Last time, before the EuroPython 2004 conference, we actually had a similar goal (a PyPy/C-version) but discovered we had to largely refactor the basic model for attribute accesses. We are now closely mirroring the marvelous "descriptor"-mechanism of CPython. If you are interested to participate in our fun and somewhat mind-altering python sprint event then please subscribe at http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintAttendants and look around for more information. You'll find that most of the core PyPy developers are already determined to come. There are also many areas that need attention so that we should have tasks suited for different levels of expertise. At http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintTasks you'll find our sprint planning task list which will probably grow in the next weeks. Note that our EU funding efforts are at the final stage now. In the next weeks, quite likely before the Vilnius sprint, we _hope_ to get a "go!" from the european commission. One side effect would be that coders - probably restricted to european citizens - may generally apply for getting travel and accomodation costs refunded for PyPy sprints. This would reduce the barrier of entry to the question if you like to spend your time with a pypy sprint. However, we probably need some time to work out the details after when we get more information from the EU. If you have any questions don't hesitate to contact pypy-sprint@codespeak.net or one of us personally. cheers & a bientot, Holger Krekel, Armin Rigo From abkhd at hotmail.com Wed Oct 13 04:32:35 2004 From: abkhd at hotmail.com (Khalid A. B.) Date: Thu Oct 14 21:23:18 2004 Subject: [Python-Dev] test_cwd of test_subprocess.py fails on Win98 Message-ID: Hello On Win98 test_cwd of test_subprocess.py tries to assert that "C:\WINDOWS\TEMP" is equal to "C:\WINDOWS\Temp" but since "TEMP" != "Temp" it fails, even when they are the same directory. Details follow: $ python -i Python 2.4a3 (#56, Oct 13 2004, 02:29:05) [GCC 3.4.1 (mingw special)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>import os, sys, subprocess >>>tmpdir = os.getenv("TEMP", "/tmp") >>>tmpdir = os.path.realpath(tmpdir) >>>tmpdir 'C:\\WINDOWS\\TEMP' >>>p = subprocess.Popen([sys.executable, "-c", ... 'import sys,os;' \ ... 'sys.stdout.write(os.getcwd())'], ... stdout=subprocess.PIPE, ... cwd=tmpdir) >>>subProcessTmpDir = p.stdout.read() >>>subProcessTmpDir 'C:\\WINDOWS\\Temp' >>>tmpdir == subProcessTmpDir False >>> Here is what fixed it for me: Index: python/dist/src/Lib/test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.4 diff -u -d -r1.4 test_subprocess.py --- python/dist/src/Lib/test/test_subprocess.py 12 Oct 2004 22:29:54 -0000 1.4 +++ python/dist/src/Lib/test/test_subprocess.py 13 Oct 2004 02:27:13 -0000 @@ -192,9 +192,16 @@ def test_cwd(self): tmpdir = os.getenv("TEMP", "/tmp") tmpdir = os.path.realpath(tmpdir) - p = subprocess.Popen([sys.executable, "-c", - 'import sys,os;' \ - 'sys.stdout.write(os.getcwd())'], + commandStr = [sys.executable, "-c"] + if mswindows: + tmpdir = tmpdir.upper() + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd().upper())') + else: + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd())') + + p = subprocess.Popen(commandStr, stdout=subprocess.PIPE, cwd=tmpdir) self.assertEqual(p.stdout.read(), tmpdir) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/ From ncoghlan at iinet.net.au Thu Oct 14 22:38:36 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 14 22:37:33 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: References: Message-ID: <416EE3CC.1030003@iinet.net.au> Just van Rossum wrote: > I understand the difficulty of implementing it, but as a user I find it > a really really stupid restriction. I routively run (doc)tests of > individual modules, which usually are submodules of a package. Using -m > to do this would help me tremendously. As it stands, -m doesn't help me > at all. I'd even go so far and -1 the entire feature if it doesn't > support submodules. I guess it's too late for that :) This may be more a case of my misjudging py-dev's likely collective reaction than anything else. Support for '-m' was lukewarm enough when it last came up, that I didn't expect to get a good reaction if I suggested adding a stdlib module in order to enhance it to support packages. While I wrote a patch to enable it (#1043356 - it uses the simple C-level strategy of 'try to locate at the top level, if that doesn't work, hand it over to the Python version'), we seemed to be too close to the beta to push for inclusion this time around. Add in the fact that I was about to be moving back to Brisbane after being Oregon for three months. . . (I'm back in Brisbane now, though) At the moment, '-m's behaviour is pretty easy to explain: "Look for a top-level module with the specified name. If it is found, run it as if its name had been fully specified on the command line. If it is not found, report an error" The behaviour currently implemented in the enhancement patch is: "Look for a top-level module with the specified name. If it is found, run it as if its name had been fully specified on the command line. If it is not found, attempt to import any containing package, then look for the module within that package. Run the located module as for a top-level module. If it is still not found, report an error. Note: For modules within packages, this differs slightly from running them directly from the command line by filename. Using this switch, the script's containing package is fully imported prior to execution of the script. This does not happen when the script's filename is used directly." As an implementation detail, the top-level scan is done in C, the scan that understands packages is done in Python. The main reasons for that are that the top-level scan gets used to *find* the Python version if it's needed, and even a simple scan looking for dots is a pain in C (although that *would* likely be slightly quicker than the current 'failed lookup' approach for scripts inside modules, it would also be slightly slower for top-level modules, as well as adding more code to main.c). Selling the additional complexity was the main reason I didn't expect to get a good reaction to this idea with the 2.4 beta almost out the door. I'm happy to make whatever changes to that patch are needed for inclusion (e.g. changing the module name, adding it to the docs underwhatever name is chosen) - I guess it's mainly Anthony's call whether he's prepared to accept such a change after the 2.4b1 release. Cheers, Nick. P.S. I'd also like some feedback on a quirk of the current version of that patch - as noted on SF, at the moment it potentially tramples on argv[0] at the C-level, which seems questionable given the existence of Py_GetArgcArgv(). The simplest way around that is to *not* set sys.argv[0] correctly when running pyrun/execmodule implicitly (i.e. sys.argv[0] may contain the name of the interpreter, or a command line switch, rather than the name of pyrun/execmodule - document this possibility with a comment in the __main__ block of pyrun/execmodule). From ncoghlan at iinet.net.au Thu Oct 14 23:06:55 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 14 23:05:52 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <1097763252.8756.29.camel@geddy.wooz.org> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <1097762286.8756.12.camel@geddy.wooz.org> <1097763252.8756.29.camel@geddy.wooz.org> Message-ID: <416EEA6F.60708@iinet.net.au> Barry Warsaw wrote: > On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: > > >>>+1. Also, if a non-dotted module name can't be found, pyrun should be >>>called to see if the module is actually a package (with an __main__ >>>that >>>lives in the package's __init__.py). Actually, if we were going to do something for packages, I'd be more inclined to look for a script called __main__.py in the package directory, rather than looking for a __main__ function inside __init__.py. Or else simply run __init__.py itself as __main__ (i.e. allow the use of the existing 'Python main' idiom inside a package's __init__.py) (Interestingly, that's at least the second time it has been suggested to turn this idea into 'C-like main functions for Python'. '-m' is about another way to invoke the current 'if __name__ == "__main__":' idiom. It is most definitely *not* about creating a new idiom for main functions - an activity which would seem to be PEP-worthy) >>Wouldn't it make more sense to look for main rather than __main__? It >>seems that __main__ (as a function) occurs EXACTLY ZERO times in the >>standard library ;) > > > I think that's a fine convention, so +1. Except we'd be violating Python's tradition that magic methods start and end with double underscores, as well as potentially breaking existing scripts. At the moment, the following will print 'Hello World', but with 'def main' being special, it would do nothing: def main(*args, **kwds): pass if __name__ == "__main__: print "Hello world" main() I know that I often do any sys.argv manipulation in the 'if __name__' block rather than inside my main() function (which is often actually called 'main', and almost always takes a list of arguments rather than looking at sys.argv for itself). So while I'd be quite happy for code that included a "def __main__" to break (since the Python docs advise against using names in that format), allowing "def main" to suddenly acquire a magic meaning may not break the stdlib, but I bet it would break a lot of single-purpose scripts that are out there. Cheers, Nick. From pje at telecommunity.com Thu Oct 14 23:16:39 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Oct 14 23:16:13 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <416EEA6F.60708@iinet.net.au> References: <1097763252.8756.29.camel@geddy.wooz.org> <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <1097762286.8756.12.camel@geddy.wooz.org> <1097763252.8756.29.camel@geddy.wooz.org> Message-ID: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> At 07:06 AM 10/15/04 +1000, Nick Coghlan wrote: >Barry Warsaw wrote: >>On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: >> >>>>+1. Also, if a non-dotted module name can't be found, pyrun should be >>>>called to see if the module is actually a package (with an __main__ that >>>>lives in the package's __init__.py). > >Actually, if we were going to do something for packages, I'd be more >inclined to look for a script called __main__.py in the package directory, >rather than looking for a __main__ function inside __init__.py. Or else >simply run __init__.py itself as __main__ (i.e. allow the use of the >existing 'Python main' idiom inside a package's __init__.py) > >(Interestingly, that's at least the second time it has been suggested to >turn this idea into 'C-like main functions for Python'. '-m' is about >another way to invoke the current 'if __name__ == "__main__":' idiom. It >is most definitely *not* about creating a new idiom for main functions - >an activity which would seem to be PEP-worthy) Perhaps this means that -m is premature? I personally would rather wait for 2.5 if it means we get a nice, future-proof "main" convention out of the deal. While -m would not then be "backward compatible" with existing scripts, people could start changing scripts to match the convention as soon as there was an accepted PEP. From barry at python.org Thu Oct 14 23:16:33 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 23:16:40 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416EE3CC.1030003@iinet.net.au> References: <416EE3CC.1030003@iinet.net.au> Message-ID: <1097788593.8756.85.camel@geddy.wooz.org> On Thu, 2004-10-14 at 16:38, Nick Coghlan wrote: > While I wrote a patch to enable it (#1043356 - it uses the simple > C-level strategy of 'try to locate at the top level, if that doesn't > work, hand it over to the Python version'), we seemed to be too close to > the beta to push for inclusion this time around. Add in the fact that I > was about to be moving back to Brisbane after being Oregon for three > months. . . (I'm back in Brisbane now, though) Well, you're practically next door to Anthony (at least, compared to some of us), so I suggest you hop on down there on Friday, buy Anthony a beer or ten as a friendly convincer for slipping this into beta 1. To seal the deal, you can even offer to help (or threaten to harass) him while he makes the release. A win all around! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/395dbf4c/attachment.pgp From Scott.Daniels at Acm.Org Thu Oct 14 23:56:55 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu Oct 14 23:55:45 2004 Subject: [Python-Dev] Patch wanted Message-ID: Ray Hettinger's fix 1.29 to PC/pyconfig.h: Revision 1.29 - (view) (download) (annotate) - [select for diffs] CVS Tags: r24a2, r24a3 Changes since 1.28: +4 -2 lines Restore compilation on MSVC++ 6.0 =================================== /* Atleast VC 7.1 has them. If some compiler does not provide them, #ifdef appropriately .*/ #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 =================================== /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of 1200. If some compiler does not provide them, modify the #if appropriately. */ #if _MSC_VER != 1200 #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 #endif =================================== Tests the wrong thing. His test presumes every non VC 7.1 PC compiler has those types defined. For non-MSC compilers, _MSC_VER is undefined, and the expression #if _MSC_VER != 1200 becomes: #if 0 != 1200 (Not what was intended, I presume). Even: #if _MSC_VER >= 1200 would be preferable (better failure, will presumably work with VC 7.2), but I believe the correct test should be: #if HAVE_STD_INT as I inexpertly explained in patch 1020042. This affects any C compilers on Windows that don't have the uintptr_t and intptr_t types (those without the newer include file stdint.h) except VC 6.0. Such compilers cannot compile any file that contains #include "Python.h" I think this needs to be fixed before 2.4 goes out. -- -- Scott David Daniels Scott.Daniels@Acm.Org From ncoghlan at iinet.net.au Fri Oct 15 01:27:05 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Oct 15 01:26:24 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> References: <1097763252.8756.29.camel@geddy.wooz.org> <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <1097762286.8756.12.camel@geddy.wooz.org> <1097763252.8756.29.camel@geddy.wooz.org> <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> Message-ID: <416F0B49.1080008@iinet.net.au> Phillip J. Eby wrote: > Perhaps this means that -m is premature? I personally would rather wait > for 2.5 if it means we get a nice, future-proof "main" convention out of > the deal. While -m would not then be "backward compatible" with > existing scripts, people could start changing scripts to match the > convention as soon as there was an accepted PEP. I see the two issues as entirely orthogonal. A new main function idiom should not have to rely on the use of a particular command line switch (particularly since there is no way to 'switch off' the current idiom without significant changes to the interpreter). If we were actually developing a new idiom, I would be thinking more along the following lines: "After initial execution of the __main__ module in non-interactive mode (e.g. running a script supplied by filename or using -m), the interpreter looks for a function also called __main__ in the __main__ module's namespace. If such a function is found, it is executed by the interpreter as if the call __main__() had been appended to the end of the supplied script." Existing scripts would continue to work as is, or else could be converted to use the new idiom by replacing the line "if __name__ == '__main__':" with the line "def __main__():" Scripts that need to be backwards compatible can use the old idiom and include the following to mark themselves as definitely being usable as scripts: def __main__(): pass Or else, they can be made compatible with older versions by adding the following snippet to the end: if __name__ == '__main__': import sys if sys.version_info[0:2] < [2, 5]: __main__() About the only real practical (as opposed to aesthetic) advantage I see to such an idiom is that the question "Is this module useful as a script?" can easily be answered as "Yes" by looking for a __main__ function. (We can't really answer 'No' definitively, since a script may be using the existing idiom without using the 'def __main__(): pass' trick). Although "def __main__():" could be easier to explain to beginners than the current approach. That might also be a slight benefit. *stops to think for a bit* Actually, while proofreading this, one other potential advantage occured to me - this might allow C extensions and the like to define __main__ functions, and so be usable as scripts via -m despite their non-script packaging. Although that seems like an awful lot of work for not much gain - it would presumably be easier to just distribute a package that included both the extension module and a 'launcher' Python script. Anyway, regardless of whether a new main idiom is ever selected or not, I just don't see the link between that and being able to search for scripts using Python's module namespace instead of the OS filesystem's namespace. Cheers, Nick. From python at rcn.com Fri Oct 15 04:55:46 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 15 04:57:31 2004 Subject: [Python-Dev] Patch wanted In-Reply-To: Message-ID: <001701c4b262$78984240$e841fea9@oemcomputer> > Revision 1.29 . . . > Tests the wrong thing . . . > This affects any C compilers on Windows that don't have the uintptr_t > and intptr_t types (those without the newer include file stdint.h) > except VC 6.0. Here's a proposed patch: --- pyconfig.h 23 Sep 2004 19:11:32 -0000 1.30 +++ pyconfig.h 15 Oct 2004 02:44:27 -0000 @@ -273,10 +273,12 @@ /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of 1200. If some compiler does not provide them, modify the #if appropriately. */ -#if _MSC_VER != 1200 +#if defined(_MSC_VER) +#if _MSC_VER > 1200 #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 -#endif +#endif /* _MSC_VER > 1200 */ +#endif /* _MSC_VER */ #endif > Ray Hettinger's fix 1.29 to PC/pyconfig.h . . . > His test presumes . . . Please watch the personalisms. I'm not the one who broke the build in the first place. Ideally, someone who knows all about the various compilers can propose a generic fix so that a lot of the ifdeffery in this file can be taken out. For someone trying to repair a broken build, they can often only test in one environment. Raymond Side note: It looks like I may be the only one testing/maintaining the MSVC++6.0 build. It got broken again this week, so I have to fix it up tonight (looks like a minor repair though). From kbk at shore.net Fri Oct 15 05:48:43 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri Oct 15 05:48:50 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200410150348.i9F3mhO6013761@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 240 open ( -1) / 2655 closed (+15) / 2895 total (+14) Bugs : 766 open ( +0) / 4514 closed (+22) / 5280 total (+22) RFE : 155 open ( +1) / 131 closed ( +0) / 286 total ( +1) New / Reopened Patches ______________________ True/False instead of 1/0 in libstdtypes.tex (2004-10-06) CLOSED http://python.org/sf/1041364 opened by Gerrit Holl bbetter document popenX 'cmd' argument (2004-10-07) CLOSED http://python.org/sf/1042705 opened by Jeff Epler broken TeX markup (2004-10-08) CLOSED http://python.org/sf/1042969 opened by George Yoshida Decimal.py enhancements (2004-10-08) CLOSED http://python.org/sf/1043218 opened by Facundo Batista Enhancing '-m' to support packages (2004-10-09) http://python.org/sf/1043356 opened by Nick Coghlan External storage protocol for large email messages (2004-10-09) http://python.org/sf/1043706 opened by Barry A. Warsaw tarfile: add extractall() method (2004-10-10) http://python.org/sf/1043890 opened by Lars Gust?bel tarfile.py: patch for bug #1017553 (2004-10-10) http://python.org/sf/1043972 opened by Lars Gust?bel Bug #1011380 alternate fix - locking is noop without threads (2004-10-11) CLOSED http://python.org/sf/1044089 opened by Nick Coghlan Add FreeBSD* to configure.in for --enable-shared (2004-10-11) http://python.org/sf/1044395 opened by James William Pye improved configure.in output (2004-10-12) http://python.org/sf/1045620 opened by Wummel Remove unncessary NLST from ftp transfers (2004-10-12) http://python.org/sf/1045783 opened by Chris Cogdon Consistent retrieval of Python version. (2004-10-14) http://python.org/sf/1046831 opened by uiltje Buffer overwrite in PyUnicode_AsWideChar (2004-10-14) http://python.org/sf/1047269 opened by Thomas Heller Patches Closed ______________ -m option to run a module as a script (2004-09-27) http://python.org/sf/1035498 closed by rhettinger True/False instead of 1/0 in libstdtypes.tex (2004-10-06) http://python.org/sf/1041364 closed by rhettinger pdb enhancements and bug fixes (2004-02-12) http://python.org/sf/896011 closed by jlgijsbers bbetter document popenX 'cmd' argument (2004-10-08) http://python.org/sf/1042705 closed by jlgijsbers popen2.popen2() 'cmd' list syntax docstring (2004-02-01) http://python.org/sf/888380 closed by jlgijsbers broken TeX markup (2004-10-08) http://python.org/sf/1042969 closed by jlgijsbers Decimal.py enhancements (2004-10-08) http://python.org/sf/1043218 closed by rhettinger replacing rfc822.formatdate (2004-08-09) http://python.org/sf/1005857 closed by anthonybaxter email/__init__.py: Parse headersonly (2004-03-20) http://python.org/sf/920037 closed by bwarsaw Bad behavior of email.Charset.Charset when locale is tr_TR (2003-12-29) http://python.org/sf/866982 closed by bwarsaw mailbox / fromline matching (2002-02-22) http://python.org/sf/521478 closed by bwarsaw Bug #1011380 alternate fix - locking is noop without threads (2004-10-10) http://python.org/sf/1044089 closed by tim_one Logger file objects (2004-08-02) http://python.org/sf/1001864 closed by vsajip Signal fixes for BSD systems (2004-06-18) http://python.org/sf/975056 closed by anthonybaxter Allow ldshared to be overridden by environment var (2004-07-01) http://python.org/sf/983206 closed by anthonybaxter New / Reopened Bugs ___________________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) CLOSED http://python.org/sf/1041325 opened by Arthur Lutz SimpleXMLRPCServer example is broken (2004-10-06) CLOSED http://python.org/sf/1041501 opened by Kent Johnson Thread management corrupts heap (2004-10-06) CLOSED http://python.org/sf/1041645 opened by benson margulies Patch (2004-10-07) CLOSED http://python.org/sf/1042179 opened by Michiel de Hoon Lib/compiler chokes on certain genexps (2004-10-07) CLOSED http://python.org/sf/1042238 opened by Michael Hudson mimetypes.guess_extension('text/plain') = '.ksh' ??? (2004-10-08) http://python.org/sf/1043134 opened by Robert Kiendl Uninterruptable loop (2004-10-09) http://python.org/sf/1043446 opened by Jp Calderone compile error with stlport (2004-10-10) http://python.org/sf/1044299 opened by Debug Can't raise "C API version mismatch" warning (2004-10-11) http://python.org/sf/1044382 opened by Frank Sonnenburg docs for Py_UNICODE are wrong (2004-10-11) http://python.org/sf/1044479 opened by Thomas Heller Unable to install Docs - Linux (2004-10-11) http://python.org/sf/1044872 opened by Colin J. Williams strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 opened by Sebastien JUST Problems with os.system and ulimit -f (2004-10-12) http://python.org/sf/1045509 opened by Tomasz Kowaltowski test_subprocess vs Windows (2004-10-12) CLOSED http://python.org/sf/1045748 opened by Tim Peters warning '"_POSIX_C_SOURCE" redefined' when include Python.h (2004-10-13) http://python.org/sf/1045893 opened by Eugene Sizikov urllib2: better import ftplib and gopherlib etc late (2004-10-13) http://python.org/sf/1046077 opened by Robert Kiendl HTMLParser fix to accept mailformed tag attributes (2004-10-13) http://python.org/sf/1046092 opened by Vsevolod Novikov improving distutils swig support (2004-10-14) CLOSED http://python.org/sf/1046404 opened by Anthony Baxter improving distutils swig support (2) (2004-10-14) CLOSED http://python.org/sf/1046644 opened by Lars Immisch difflib.HtmlDiff doc errors (EASY) (2004-10-13) CLOSED http://python.org/sf/1046690 opened by Dan Gass Error in the given example (2004-10-14) http://python.org/sf/1046800 opened by Vishnu httplib index out of range (2004-10-14) CLOSED http://python.org/sf/1046855 opened by Grzegorz Makarewicz improving distutils swig support - documentation (2004-10-14) http://python.org/sf/1046945 opened by Lars Immisch cgitb failures (2004-10-14) http://python.org/sf/1047397 opened by Robin Becker Bugs Closed ___________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) http://python.org/sf/1041325 closed by arthur_lutz SimpleXMLRPCServer example is broken (2004-10-06) http://python.org/sf/1041501 closed by akuchling Thread management corrupts heap (2004-10-06) http://python.org/sf/1041645 closed by tim_one PyOS_InputHook broken (2004-09-19) http://python.org/sf/1030629 closed by mwh Patch (2004-10-07) http://python.org/sf/1042179 closed by mwh Lib/compiler chokes on certain genexps (2004-10-07) http://python.org/sf/1042238 closed by mwh Enhancements to pdb.py when invoked as script (2003-06-09) http://python.org/sf/751124 closed by jlgijsbers shutils.rmtree() uses excessive amounts of memory (2004-09-09) http://python.org/sf/1025127 closed by jlgijsbers Conflicting descriptions of application order of decorators (2004-09-21) http://python.org/sf/1031897 closed by akuchling x, y in curses window object documentation (2004-09-04) http://python.org/sf/1022311 closed by akuchling Confusing description of strict option for email.Parser (2004-09-22) http://python.org/sf/1032960 closed by bwarsaw email package uses \n to rebuild content of a message (2004-06-01) http://python.org/sf/964433 closed by bwarsaw smtpd.py docstring says wrong default class (2004-08-16) http://python.org/sf/1010102 closed by bwarsaw email: mishandles Content-Transfer-Enc. for text/* messages (2004-05-09) http://python.org/sf/950747 closed by bwarsaw Email message croaks the new email pkg parser (2004-09-19) http://python.org/sf/1030941 closed by bwarsaw Make GIL acquire/release functions noops if PyEval_Init (2004-08-18) http://python.org/sf/1011380 closed by tim_one test_subprocess vs Windows (2004-10-12) http://python.org/sf/1045748 closed by tim_one BSD restartable signals not correctly disabled (2004-06-09) http://python.org/sf/969574 closed by anthonybaxter Solaris likes sys/loadavg.h (2004-06-22) http://python.org/sf/977343 closed by anthonybaxter improving distutils swig support (2004-10-14) http://python.org/sf/1046404 closed by anthonybaxter improving distutils swig support (2) (2004-10-14) http://python.org/sf/1046644 closed by anthonybaxter difflib.HtmlDiff doc errors (EASY) (2004-10-13) http://python.org/sf/1046690 closed by tim_one httplib index out of range (2004-10-14) http://python.org/sf/1046855 closed by rhettinger From ilya at bluefir.net Fri Oct 15 06:25:14 2004 From: ilya at bluefir.net (Ilya Sandler) Date: Fri Oct 15 06:23:42 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> References: <1097763252.8756.29.camel@geddy.wooz.org> <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <1097762286.8756.12.camel@geddy.wooz.org> <1097763252.8756.29.camel@geddy.wooz.org> <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> Message-ID: On Thu, 14 Oct 2004, Phillip J. Eby wrote: > Perhaps this means that -m is premature? I personally would rather wait > for 2.5 if it means we get a nice, future-proof "main" convention out of > the deal. While -m would not then be "backward compatible" with existing > scripts, people could start changing scripts to match the convention as > soon as there was an accepted PEP. But to me -m option and __main__() conventions seem like orthogonal features... Even if current __name__=="__main__" blocks get replaced by a magic __main__() function, you would still benefit from -m cmd line option Or is there some hidden dependency? Ilya > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ilya%40bluefir.net > From anthony at interlink.com.au Fri Oct 15 06:36:37 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Oct 15 06:37:01 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <1097788593.8756.85.camel@geddy.wooz.org> References: <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> Message-ID: <416F53D5.7090606@interlink.com.au> Barry Warsaw wrote: > Well, you're practically next door to Anthony (at least, compared to > some of us), so I suggest you hop on down there on Friday, buy Anthony a > beer or ten as a friendly convincer for slipping this into beta 1. To > seal the deal, you can even offer to help (or threaten to harass) him > while he makes the release. A win all around! I'm extremely unconvinced that the semantics of "-m package" or "-m package.module" are suitably well thought out to see it in b1. If a single compelling way of making it work can be seen in the next week, _maybe_ we could sneak it into b2, but I'm really not hopeful. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From fdrake at acm.org Fri Oct 15 07:40:13 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 15 07:40:26 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416F53D5.7090606@interlink.com.au> References: <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> Message-ID: <200410150140.13330.fdrake@acm.org> On Friday 15 October 2004 12:36 am, Anthony Baxter wrote: > I'm extremely unconvinced that the semantics of "-m package" or > "-m package.module" are suitably well thought out to see it in b1. I don't see this as a useful feature at all, so I guess I'm biased, but if it's hard to decide just what the right semantics are, this certainly isn't the time for it. We won't be able to change it substantially later due to backward compatibility constraints. > If a single compelling way of making it work can be seen in the > next week, _maybe_ we could sneak it into b2, but I'm really not > hopeful. Let's keep it out of 2.4 and see what proposals show up for 2.5. -Fred -- Fred L. Drake, Jr. From ncoghlan at iinet.net.au Fri Oct 15 09:57:19 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Oct 15 09:56:15 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416F53D5.7090606@interlink.com.au> References: <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> Message-ID: <416F82DF.7010005@iinet.net.au> Anthony Baxter wrote: > I'm extremely unconvinced that the semantics of "-m package" or > "-m package.module" are suitably well thought out to see it in b1. This was one of the reasons I wrote execmodule - to figure out semantics which made sense, after my initial attempt at supporting "-m package.module" failed to handle __path__ correctly. Executing anything other than PY_SOURCE or PY_COMPILED modules simply doesn't make sense to me (both execmodule and the current '-m' implementation report errors if you try to do so) > If a single compelling way of making it work can be seen in the > next week, _maybe_ we could sneak it into b2, but I'm really not > hopeful. The approach of 'import the containing package' seems to work fairly well. I can't see any other way to get at the package's __path__ variable in order to locate the module inside it. I haven't seen any good reasons to lift the PY_SOURCE/PY_COMPILED restriction, though. FWIW, the example which convinced me that running modules inside packages was valuable was "python -m pychecker.checker