From ben+python at benfinney.id.au Thu Oct 1 00:18:57 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 01 Oct 2009 08:18:57 +1000 Subject: [Python-Dev] Python 2.6.3rc1 available References: Message-ID: <87pr98yt2m.fsf@benfinney.id.au> Barry Warsaw writes: > The first (and hopefully last) release candidate for Python 2.6.3 is > now available [?] Thank you for this announcement, and the efforts that go into making this work available. *Especially* thank you for avoiding the oxymoronic ?Released: 2.6.3 release candidate? or similar. I hope this signifies a trend toward using the more accurate term ?available? for announcing availability of anything that's not an actual release. -- \ ?Guaranteed to work throughout its useful life.? ?packaging for | `\ clockwork toy, Hong Kong | _o__) | Ben Finney -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From amcnabb at mcnabbs.org Thu Oct 1 00:58:00 2009 From: amcnabb at mcnabbs.org (Andrew McNabb) Date: Wed, 30 Sep 2009 16:58:00 -0600 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: References: <200909290127.03215.steve@pearwood.info> <212C8A6F-CEC3-44A0-91F4-C5814DF44B04@python.org> <20090930201735.GD27796@mcnabbs.org> Message-ID: <20090930225800.GF27796@mcnabbs.org> On Wed, Sep 30, 2009 at 02:40:20PM -0700, Steven Bethard wrote: > > > Also, is it possible to add these subparsers dynamically? ?For example, > > you would want to be able to load a module immediately after parsing the > > name instead of having to keep a predetermined list of all module names. > > I'm pretty sure that bzr dynamically loads modules this way. ?Can > > argparse help with this? > > You can probably already do this. I'm not 100% sure what you want to > do, but it's certainly possible to define an argparse.Action that > loads a module when it's invoked. It might look something like:: > > class MyAction(argparse.Action): > def __call__(self, parser, namespace, value, option_string=None): > mod = __import__(value) # or whatever This looks much easier than what I was able to do in optparse. Cool. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868 From vinay_sajip at yahoo.co.uk Thu Oct 1 01:03:16 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 30 Sep 2009 23:03:16 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: Message-ID: Steven Bethard gmail.com> writes: > There's a lot of code already out there (in the standard library and > other places) that uses %-style formatting, when in Python 3.0 we > should be encouraging {}-style formatting. We should really provide > some sort of transition plan. Consider an example from the logging > docs: > > logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") > > We'd like to support both this style as well as the following style: > > logging.Formatter("{asctime} - {name} - {levelname} - {message}") > In logging at least, there are two different places where the formatting issue crops up. The first is creating the "message" part of the the logging event, which is made up of a format string and arguments. The second is the one Steven's mentioned: formatting the message along with other event data such as time of occurrence, level, logger name etc. into the final text which is output. Support for both % and {} forms in logging would need to be considered in these two places. I sort of liked Martin's proposal about using different keyword arguments, but apart from the ugliness of "dicttemplate" and the fact that "fmt" is already used in Formatter.__init__ as a keyword argument, it's possible that two different keyword arguments "fmt" and "format" both referring to format strings might be confusing to some users. Benjamin's suggestion of providing a flag to Formatter seems slightly better, as it doesn't change what existing positional or keyword parameters do, and just adds an additional, optional parameter which can start off with a default of False and transition to a default of True. However, AFAICT these approaches only cover the second area where formatting options are chosen - not the creation of the message from the parameters passed to the logging call itself. Of course one can pass arbitrary objects as messages which contain their own formatting logic. This has been possible since the very first release but I'm not sure that it's widely used, as it's usually easier to pass strings. So instead of passing a string and arguments such as logger.debug("The %s is %d", "answer", 42) one can currently pass, for a fictitious class PercentMessage, logger.debug(PercentMessage("The %s is %d", "answer", 42)) and when the time comes to obtain the formatted message, LogRecord.getMessage calls str() on the PercentMessage instance, whose __str__ will use %-formatting to get the actual message. Of course, one can also do for example logger.debug(BraceMessage("The {} is {}", "answer", 42)) where the __str__() method on the BraceMessage will do {} formatting. Of course, I'm not suggesting we actually use the names PercentMessage and BraceMessage, I've just used them there for clarity. Also, although Raymond has pointed out that it seems likely that no one ever needs *both* types of format string, what about the case where application A depends on libraries B and C, and they don't all share the same preferences regarding which format style to use? ISTM no-one's brought this up yet, but it seems to me like a real issue. It would certainly appear to preclude any approach that configured a logging-wide or logger-wide flag to determine how to interpret the format string. Another potential issue is where logging events are pickled and sent over sockets to be finally formatted and output on different machines. What if a sending machine has a recent version of Python, which supports {} formatting, but a receiving machine doesn't? It seems that at the very least, it would require a change to SocketHandler and DatagramHandler to format the "message" part into the LogRecord before pickling and sending. While making this change is simple, it represents a potential backwards-incompatible problem for users who have defined their own handlers for doing something similar. Apart from thinking through the above issues, the actual formatting only happens in two locations - LogRecord.getMessage and Formatter.format - so making the code do either %- or {} formatting would be simple, as long as it knows which of % and {} to pick. Does it seems too onerous to expect people to pass an additional "use_format" keyword argument with every logging call to indicate how to interpret the message format string? Or does the PercentMessage/BraceMessage type approach have any mileage? What do y'all think? Regards, Vinay Sajip From tjreedy at udel.edu Thu Oct 1 01:47:21 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 30 Sep 2009 19:47:21 -0400 Subject: [Python-Dev] Announcing PEP 3136 In-Reply-To: <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> References: <20090929202035.GA5492@gmail.com> <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> Message-ID: Yuvgoog Greenle wrote: > I like how python has a minimalistic and powerful syntax (-1 for the > break ___ PEP). > > Also, I really dislike the for/else ambiguity "butterflies". Properly understood, no ambiguity. while c: x() is equivalent to hypothetical label z: if c: x() goto: z So while c: x() else: y() is equivalent to label 1: if c: x() goto: 1 else" y() The else clause fires (once) if and when the if/while condition evaluates as false. Break and continue are restricted *unconditional* goto statements, and so *cannot* trigger an else clause. In for loops, the implied condition is 'there is another item in the collection represented by the iterable'. For any more, move to another list. Terry Jan Reedy From brett at python.org Thu Oct 1 02:29:04 2009 From: brett at python.org (Brett Cannon) Date: Wed, 30 Sep 2009 17:29:04 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: On Wed, Sep 30, 2009 at 16:03, Vinay Sajip wrote: > Steven Bethard gmail.com> writes: > >> There's a lot of code already out there (in the standard library and >> other places) that uses %-style formatting, when in Python 3.0 we >> should be encouraging {}-style formatting. We should really provide >> some sort of transition plan. Consider an example from the logging >> docs: >> >> logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") >> >> We'd like to support both this style as well as the following style: >> >> logging.Formatter("{asctime} - {name} - {levelname} - {message}") >> > > In logging at least, there are two different places where the formatting issue > crops up. > > The first is creating the "message" part of the the logging event, which is > made up of a format string and arguments. > > The second is the one Steven's mentioned: formatting the message along with > other event data such as time of occurrence, level, logger name etc. into the > final text which is output. > > Support for both % and {} forms in logging would need to be considered in > these two places. I sort of liked Martin's proposal about using different > keyword arguments, but apart from the ugliness of "dicttemplate" and the fact > that "fmt" is already used in Formatter.__init__ as a keyword argument, it's > possible that two different keyword arguments "fmt" and "format" both referring > to format strings might be confusing to some users. > > Benjamin's suggestion of providing a flag to Formatter seems slightly better, > as it doesn't change what existing positional or keyword parameters do, and > just adds an additional, optional parameter which can start off with a default > of False and transition to a default of True. > > However, AFAICT these approaches only cover the second area where formatting > options are chosen - not the creation of the message from the parameters passed > to the logging call itself. > > Of course one can pass arbitrary objects as messages which contain their own > formatting logic. This has been possible since the very first release but I'm > not sure that it's widely used, as it's usually easier to pass strings. So > instead of passing a string and arguments such as > > logger.debug("The %s is %d", "answer", 42) > > one can currently pass, for a fictitious class PercentMessage, > > logger.debug(PercentMessage("The %s is %d", "answer", 42)) > > and when the time comes to obtain the formatted message, LogRecord.getMessage > calls str() on the PercentMessage instance, whose __str__ will use %-formatting > to get the actual message. > > Of course, one can also do for example > > logger.debug(BraceMessage("The {} is {}", "answer", 42)) > > where the __str__() method on the BraceMessage will do {} formatting. > > Of course, I'm not suggesting we actually use the names PercentMessage and > BraceMessage, I've just used them there for clarity. > > Also, although Raymond has pointed out that it seems likely that no one ever > needs *both* types of format string, what about the case where application A > depends on libraries B and C, and they don't all share the same preferences > regarding which format style to use? ISTM no-one's brought this up yet, but it > seems to me like a real issue. It would certainly appear to preclude any > approach that configured a logging-wide or logger-wide flag to determine how to > interpret the format string. > > Another potential issue is where logging events are pickled and sent over > sockets to be finally formatted and output on different machines. What if a > sending machine has a recent version of Python, which supports {} formatting, > but a receiving machine doesn't? It seems that at the very least, it would > require a change to SocketHandler and DatagramHandler to format the "message" > part into the LogRecord before pickling and sending. While making this change > is simple, it represents a potential backwards-incompatible problem for users > who have defined their own handlers for doing something similar. > > Apart from thinking through the above issues, the actual formatting only > happens in two locations - LogRecord.getMessage and Formatter.format - so > making the code do either %- or {} formatting would be simple, as long as it > knows which of % and {} to pick. > > Does it seems too onerous to expect people to pass an additional "use_format" > keyword argument with every logging call to indicate how to interpret the > message format string? Or does the PercentMessage/BraceMessage type approach > have any mileage? What do y'all think? I personally prefer the keyword argument approach to act as a flag, but that's me. As for the PercentMessage/BraceMessage, I would make sure that you just simply take the string format and simply apply the arguments later to cut down on the amount of parentheses butting up against each other: ``logger.debug(BraceMessage("The {} is {}"), "answer", 42)``. It's still an acceptable solution that provides a clear transition: simply provide the two classes, deprecate PercentMessage or bare string usage, require BraceMessage, remove requirement. This wrapper approach also provides a way for libraries that have not shifted over to still work with PEP 3101 strings by letting the user wrap the string to be interpolated themselves and then to pass it in to the libraries. It's just unfortunate that any transition would have this cost of wrapping all strings for a while. I suspect most people will simply import the wrapping class and give it some short name like people do with gettext. -Brett From solipsis at pitrou.net Thu Oct 1 02:47:09 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Oct 2009 00:47:09 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: Message-ID: Vinay Sajip yahoo.co.uk> writes: > > Does it seems too onerous to expect people to pass an additional "use_format" > keyword argument with every logging call to indicate how to interpret the > message format string? Or does the PercentMessage/BraceMessage type approach > have any mileage? What do y'all think? What about the proposal I made earlier? (support for giving a callable, so that you pass the "{foobar}".format method when you want new-style formatting) From stephen at xemacs.org Thu Oct 1 04:53:25 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 01 Oct 2009 11:53:25 +0900 Subject: [Python-Dev] PEP 3144 review. In-Reply-To: References: <8517e9350909140944x104bcc63g62f86b3af7c6ca9@mail.gmail.com> <4AC12AC6.80304@v.loewis.de> <4AC2A9D7.4080603@gmail.com> <5c6f2a5d0909300131g5fcb255eoe9403f54777fc677@mail.gmail.com> <79990c6b0909300252w1dec6e2dlc8efdd104e5116ed@mail.gmail.com> <5c6f2a5d0909300318p4b5404d4wb41e2add4ea8f88a@mail.gmail.com> <4AC33BA1.8050203@gmail.com> <5c6f2a5d0909300433i5b0d0833qe1a9d239bd303dfc@mail.gmail.com> Message-ID: <87fxa3keoq.fsf@uwakimon.sk.tsukuba.ac.jp> Daniel Stutzbach writes: > People have voiced support for the IPNetwork class The obvious use cases are 1. parsing and validating strings that purport to describe networks, 2. providing a received for the result of a function that deduces the network from an arbitrary address and a mask 3. (controversial) providing the function in (2) as a method or constructor 4. holding certain information associated with a network such as a. broadcast address b. (controversial) the arbitrary address from (2) (I count this as obvious because I'm a packrat and hate to throw away any data, but really if generated as in 2 I see no use case) 5. providing an iterator for the addresses in the network. Note that (1) and (2) have analogous use cases for IPv?Address. > and for the occasional utility of an .ip field. I assume they have > good use cases. [Please don't do that, not even ironically. 90% of the heat in this thread has come because people discussed proposed syntax without agreeing on a common use case to compare the semantics to.] The only use case I can imagine for .ip is to represent a single gateway: def make_gateway(network, gateway): gw = IPv4Network(network) # eg network = '192.168.1.0/0' gw.ip = IPv4Address(gateway).ip # eg gateway = '192.168.2.1' return gw Note that this usage *cannot* be served by parsing "denormalized network descriptions" as the gateway address (ie, make_gateway cannot be a one-argument function). I did think about a use case like that for DNS SRV records, but servers are looked up by service, not by network, in most cases. So gateways are really the only service that fits this model. Also, it seems to me that this usage is well-served by the change to the definition of __eq__ to compare .network and .prefixlen, viz. def find_gateway_address(network, gateway_list): "raises ValueError if no gateway known for network" # it's interesting that there's no sequence method for this return gateway_list[gateway_list.index(network)].ip > I worry that this discussion has focused too much on the details of > ipaddr (and the false dichotomy of "ipaddr versus nothing"), It's not a false dichotomy; this PEP proposes adding ipaddr, not any other library. Of course if ipaddr needs additional functionality, it should be added before approval. But you seem to question whether any of the functionality is necessary, and I don't recall any posts of the form, "this is really sweet, but it needs salt". > without properly tackling the question of "What use-cases for IP > addresses are sufficiently universal* that they belong in the > standard library?" I think that the use cases given above are pretty universal. The string parsing cases alone give sufficient rationale for existence of at least one class, and since addresses and networks are conceptually different (for me, anyway), I am happy to have two different classes. From vinay_sajip at yahoo.co.uk Thu Oct 1 08:15:35 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 06:15:35 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> Message-ID: Antoine Pitrou pitrou.net> writes: > Why not allow logging.Formatter to take a callable, which would in turn call > the callable with keyword arguments? > > Therefore, you could write: > logging.Formatter("{asctime} - {name} - {level} - {msg}".format) > > and then: > logging.critical(name="Python", msg="Buildbots are down") This seems perhaps usable for a Formatter instantiation (infrequent) but a problem for the case where you want to convert format_str + args -> message (potentially frequent, and less readable). Another problem is that logging calls already use keyword arguments (extra, exc_info) and so backward compatibility might be compromised. It also feels like passing a callable could encourage patterns of usage which restrict our flexibility for future changes: we want for now to just allow choosing between % and {}, but a callable can do anything. That's more flexible, to be sure, but more specialized formatting requirements are already catered for using e.g. the PercentMessage/BraceMessage approach. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Oct 1 08:51:34 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 06:51:34 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <5353D5BD-1862-42C2-913F-C75AD73002DC@python.org> Message-ID: Barry Warsaw python.org> writes: > > This is a very interesting idea. > > Note that one of the reasons to /at least/ support {}-strings also is > that %-strings are simply too error prone in many situations. For > example, if I decide to support internationalization of log format > strings, and all I can use is %-strings, it's almost guaranteed that I > will have bugs because a translator forgot the trailing 's'. This > exactly the motivation that led to PEP 292 $-strings. > > In fact, while we're at it, it would be kind of cool if I could use $- > strings in log templates. Antoine's idea of accepting a callable > might fit that bill nicely. You're already covered if you use the PercentMessage/BraceMessage approach I mentioned elsewhere in this thread. Suppose: #Just typing this in, it's not tested or anything class DollarMessage: def __init__(self, fmt, *args, **kwargs): self.fmt = fmt self.args = args self.kwargs = kwargs def __str__(self): return string.Template(self.fmt).substitute(*args, **kwargs) From vinay_sajip at yahoo.co.uk Thu Oct 1 08:59:40 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 06:59:40 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <5353D5BD-1862-42C2-913F-C75AD73002DC@python.org> Message-ID: Vinay Sajip yahoo.co.uk> writes: > #Just typing this in, it's not tested or anything > class DollarMessage: > def __init__(self, fmt, *args, **kwargs): > self.fmt = fmt > self.args = args > self.kwargs = kwargs > > def __str__(self): > return string.Template(self.fmt).substitute(*args, **kwargs) > Whoops, sorry, pressed the "post" button by accident on my previous post. The above substitute call should of course say string.Template(self.fmt).substitute(*self.args, **self.kwargs) and you can alias DollarMessage (or whatever name you choose) as _ or __, say. As far as the Formatter formatting goes, it's easy enough to subclass Formatter to format using whatever approach you want. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Oct 1 09:07:52 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 07:07:52 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: Message-ID: Brett Cannon python.org> writes: > As for the PercentMessage/BraceMessage, I would make sure that you > just simply take the string format and simply apply the arguments > later to cut down on the amount of parentheses butting up against each > other: ``logger.debug(BraceMessage("The {} is {}"), "answer", 42)``. The problem with that is that BraceMessage.__str__() wouldn't know what arguments to use to produce the message. > cost of wrapping all strings for a while. I suspect most people will > simply import the wrapping class and give it some short name like > people do with gettext. > Yes, logger.debug(__("The {} is {}", "answer", 42)) isn't ideal but perhaps liveable with. And hopefully with a decent editor, the paren-butting annoyance will be minimized. Regards, Vinay Sajip From scott+python-dev at scottdial.com Thu Oct 1 09:36:26 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Thu, 01 Oct 2009 03:36:26 -0400 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> Message-ID: <4AC45BFA.3020802@scottdial.com> Barry Warsaw wrote: > If not, I'll try to > spend some time over the next few days looking at outstanding bugs, and > marking release blockers, etc. I'd like to draw attention to: http://bugs.python.org/issue5949 Which is a regression of imaplib.py introduced in r57680. Ever since I switched to python 2.6 on my box, I have had issues with getmail[1] getting stuck in an infinite loop swallowing memory (note: only applies to IMAP SSL connections). While this code is present in older versions of python, it seems to have become a problem recently (2009-05-06 is the earliest report on the issue) perhaps due to a version bump of OpenSSL? I never noticed the problem in python2.5 even though the code is unchanged. Nevertheless, the code is clearly wrong and should be corrected. I would appreciate this bug being resolved before the next release as it effects me on a daily basis. I have submitted a patch, which reflects my local solution. -Scott [1] http://pyropus.ca/software/getmail/ -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From scott+python-dev at scottdial.com Thu Oct 1 09:55:28 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Thu, 01 Oct 2009 03:55:28 -0400 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC45BFA.3020802@scottdial.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> Message-ID: <4AC46070.1010706@scottdial.com> Scott Dial wrote: > While this code is present in > older versions of python, it seems to have become a problem recently > (2009-05-06 is the earliest report on the issue) perhaps due to a > version bump of OpenSSL? I never noticed the problem in python2.5 even > though the code is unchanged. To answer my own question, this was introduced in r64578 in ssl.py by the addition of the suppress_ragged_eofs feature (seems to originate with issue1251). While it is a change in the behavior of the ssl module, the change falls in line with other file-like objects and their handling of EOF, so the bug still falls to imaplib. In other words, this bug appeared in 2.6 and 3.0, and is present in all subsequent versions. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From p.f.moore at gmail.com Thu Oct 1 10:58:59 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Oct 2009 09:58:59 +0100 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> Message-ID: <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> 2009/9/30 Robert Kern : > I am blissfully unaware of the problems Paul mentioned about Windows GUI-mode programs. :-) > I'm not sure what would make a program "GUI-mode" or not. Certainly, I have written > Python programs that use wxPython and PyQt on Windows that print to stdout/stderr, > and they appear to work fine. It's the difference between using python.exe and pythonw.exe. There's a flag in the executable header saying whether the executable is "console mode" or "gui mode". GUI mode programs are run (by the OS) without a console (or if run from a console prompt, they automatically detach from that console, much like Unix programs which fork themselves to leave the terminal group (did I get the terminology right?) but done by the OS). As a result, the program has no valid stdin/out/err handles. Any attempt to write to them causes the program to crash. Traceback (most recent call last): File "hello.py", line 13, in main() File "hello.py", line 7, in main sys.stdout.flush() IOError: [Errno 9] Bad file descriptor (Question - is it *ever* possible for a Unix program to have invalid file descriptors 0,1 and 2? At startup - I'm assuming anyone who does os.close(1) knows what they are doing!) Paul. From mal at egenix.com Thu Oct 1 11:37:26 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 01 Oct 2009 11:37:26 +0200 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC33DE1.3040304@trueblade.com> References: <4AC2CD65.30905@v.loewis.de> <4AC33DE1.3040304@trueblade.com> Message-ID: <4AC47856.2090307@egenix.com> Eric Smith wrote: > Martin v. L?wis wrote: >> Steven Bethard wrote: >>> There's a lot of code already out there (in the standard library and >>> other places) that uses %-style formatting, when in Python 3.0 we >>> should be encouraging {}-style formatting. >> >> I don't agree that we should do that. I see nothing wrong with using >> % substitution. I agree with Martin. Both approaches have their ups and downs, but forcing users to move from %-formatting to .format()-formatting will just frustrate them: having to convert several thousand such (working) uses in their code with absolutely no benefit simply doesn't look like a good way to spend your time. In addition to the code changes, such a move would also render existing translations of the %-formatted string templates useless. > It's a maintenance burden. There are several outstanding bugs with it, > admittedly not of any great significance. I've been putting time into > fixing at least one of them. When Mark and I did short-float-repr, at > least half of my time was consumed with %-formatting, mostly because of > how it does memory management. Why not allow both and use .format() for those cases where %-formatting doesn't work too well ? > On the plus side, %-formatting is (and always will be) faster than > str.format(). Its very limitations make it possible for it to be fast. > > I'd note that PEP 3101 calls str.format() a replacement for > %-formatting, not an alternate mechanism to achieve the same end. I think that's a wording we should change. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 01 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From floris.bruynooghe at gmail.com Thu Oct 1 11:38:25 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 1 Oct 2009 10:38:25 +0100 Subject: [Python-Dev] OT: detatching from terminal (was: Re: PEP 389: argparse - new command line parsing module) In-Reply-To: <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> Message-ID: <20091001093825.GA8945@laurie.devork> On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote: > (Question - is it *ever* possible for a Unix program to have invalid > file descriptors 0,1 and 2? At startup - I'm assuming anyone who does > os.close(1) knows what they are doing!) Normally you don't close fd 0, 1 & 2 but rather redirect them to /dev/null (a black hole). That saves you from nastiness when a library or something misbahaves and tries to use one of those. It would have been nice if windows GUI-mode programs as you describe them would do something similar. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From jon+python-dev at unequivocal.co.uk Thu Oct 1 12:39:38 2009 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Thu, 1 Oct 2009 11:39:38 +0100 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> Message-ID: <20091001103938.GI642@snowy.squish.net> On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote: > (Question - is it *ever* possible for a Unix program to have invalid > file descriptors 0,1 and 2? At startup - I'm assuming anyone who does > os.close(1) knows what they are doing!) Yes, at startup you just have the file descriptors your parent process gave you. This may or may not include 0, 1, and 2, and may or may not include other descriptors too. I seem to recall there have been security holes caused in the past by people assuming 0-2 will exist. From solipsis at pitrou.net Thu Oct 1 12:48:44 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Oct 2009 10:48:44 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> Message-ID: Vinay Sajip yahoo.co.uk> writes: > > This seems perhaps usable for a Formatter instantiation (infrequent) but a > problem for the case where you want to convert format_str + args -> message > (potentially frequent, and less readable). Why is it a problem? I don't understand. It certainly is less pleasant to write "{foo}".format or "{0} {1}".format than it is to write "{0} {1}" alone, but it's still prettier and easier to remember than the special wrappers people are proposing here. > Another problem is that logging > calls already use keyword arguments (extra, exc_info) and so backward > compatibility might be compromised. Then logging can just keep recognizing those special keyword arguments, and forward the others to the formatting function. > It also feels like passing a callable could > encourage patterns of usage which restrict our flexibility for future changes: Which future changes are you thinking about? AFAIK, there hasn't been a single change in logging output formatting in years. Rejecting a present change on the basis that it "restricts our flexibility for future changes" sounds like the worst kind of argument to me :-) > That's more flexible, to be sure, but more specialized formatting > requirements are already catered for using e.g. the PercentMessage/BraceMessage > approach. Except that having to wrap format strings with "PercentMessage" or "BraceMessage" is horrible. Python is not Java. Regards Antoine. From ncoghlan at gmail.com Thu Oct 1 13:09:49 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 01 Oct 2009 21:09:49 +1000 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC45BFA.3020802@scottdial.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> Message-ID: <4AC48DFD.7020607@gmail.com> Scott Dial wrote: > I would appreciate this bug being resolved before the next release as it > effects me on a daily basis. I have submitted a patch, which reflects my > local solution. Unfortunately, it's almost certainly too late to get this into 2.6.3. It really needed to be brought up back when Barry called for identification of significant 2.6 bugs and patches rather than after the RC had already been released. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Thu Oct 1 14:21:56 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Oct 2009 08:21:56 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <5353D5BD-1862-42C2-913F-C75AD73002DC@python.org> Message-ID: <034CA2BD-6DA0-43C4-A52C-66DD3885EBF4@python.org> On Oct 1, 2009, at 2:51 AM, Vinay Sajip wrote: > You're already covered if you use the PercentMessage/BraceMessage > approach I > mentioned elsewhere in this thread. Suppose: > > #Just typing this in, it's not tested or anything > class DollarMessage: > def __init__(self, fmt, *args, **kwargs): > self.fmt = fmt > self.args = args > self.kwargs = kwargs > > def __str__(self): > return string.Template(self.fmt).substitute(*args, **kwargs) Right, thanks. This is a very cool feature that I didn't know logging supported! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Thu Oct 1 14:25:56 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Oct 2009 08:25:56 -0400 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC48DFD.7020607@gmail.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> <4AC48DFD.7020607@gmail.com> Message-ID: <7A758FD5-5052-4E9F-B164-397EF19947A3@python.org> On Oct 1, 2009, at 7:09 AM, Nick Coghlan wrote: > Scott Dial wrote: >> I would appreciate this bug being resolved before the next release >> as it >> effects me on a daily basis. I have submitted a patch, which >> reflects my >> local solution. > > Unfortunately, it's almost certainly too late to get this into > 2.6.3. It > really needed to be brought up back when Barry called for > identification > of significant 2.6 bugs and patches rather than after the RC had > already > been released. Right. Scott, you can get reviews and support for the patch now so that the bug can be addressed in 2.6.4. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From vinay_sajip at yahoo.co.uk Thu Oct 1 14:42:38 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 12:42:38 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> Message-ID: Antoine Pitrou pitrou.net> writes: > > Why is it a problem? I don't understand. It certainly is less pleasant to > write "{foo}".format or "{0} {1}".format than it is to write "{0} {1}" alone, > but it's still prettier and easier to remember than the special wrappers > people are proposing here. Well, it's less readable, as I said in parentheses. It would work, of course. And the special wrappers needn't be too intrusive: __ = BraceMessage logger.debug(__("Message with {0} {1}", 1, "argument")) > Then logging can just keep recognizing those special keyword arguments, and > forward the others to the formatting function. It just means that you can't pass those values through, and what if some of them are used somewhere in existing code? > > It also feels like passing a callable could > > encourage patterns of usage which restrict our flexibility for future > > changes: > > Which future changes are you thinking about? AFAIK, there hasn't been a It's the Rumsfeldian "We don't know what we don't know" ;-) > single change in logging output formatting in years. Rejecting a present > change on the basis that it "restricts our flexibility for future changes" > sounds like the worst kind of argument to me Now don't get upset and take it as a rejection, as we're still in the kicking-ideas-around stage ;-) I'm just saying how it feels to me. I agree that logging output formatting hasn't changed in years, and that's because there's been no particular need for it to change (some changes *were* made in the very early days to support a single dict argument). Now that time for change has perhaps come. I'm just trying to think ahead, and can't claim to have got a definitive answer up my sleeve. Passing a callable has upsides and downsides, and ISTM it's always worth focusing on the downsides to make sure they don't come back and bite you later. I don't foresee any specific problem - I'm just uneasy about it. > Except that having to wrap format strings with "PercentMessage" or > "BraceMessage" is horrible. Python is not Java. Amen. I'd say "Yeccchh!" too, if it literally had to be like that. And I also note that there are voices here saying that support for %-formatting shouldn't, or doesn't need to, change, at least until Python 4.0. So consider the following tentative suggestion, which is off the top of my head and offered as a discussion point: Suppose that if you want to use %-formatting, everything stays as is. No backward-compatibility headaches. To support {}-formatting, add an extra class which I've called BraceMessage. Consider this name a "working title", as no doubt a better name will suggest itself, but for now this name makes it clear what we're talking about. If any module wants to use {} formatting for their logging, they can add the line from logging import BraceMessage as __ I've used two underscores, since _ might be being used for gettext, but obviously the importer can use whatever name they want. and then they can use logger.debug(__("The {0} is {1}", "answer", 42)) which I think is more readable than putting in ".format" following the string literal. It's not a *huge* point, perhaps, but "Readability counts". This has the side benefit that if e.g. Barry wanted to use string.Template for formatting, he's just got to replace the above import with something like from logging import DollarMessage as __ Another "working title", please note. And while I've shown these classes being imported from logging, it doesn't make sense to put them there if this idea were to fly in a more general context. Then, perhaps string would be a better home for these classes. Regards, Vinay Sajip From solipsis at pitrou.net Thu Oct 1 15:07:50 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Oct 2009 13:07:50 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> Message-ID: Hello, > Well, it's less readable, as I said in parentheses. It would work, of course. > And the special wrappers needn't be too intrusive: > > __ = BraceMessage > > logger.debug(__("Message with {0} {1}", 1, "argument")) Ah, I hadn't thought about that. It looks a bit less awful indeed. I'm of the opinion, however, that %-formatting should remain the default and shouldn't need a wrapper. There's another possibility, which is to build the wrapping directly around the logger. That is, if I want a %-style logger, I do: logger = logging.getLogger("smtp") logger.debug("incoming email from %s", sender_address) and I want a {}-style logger, I do: logger = logging.getLogger("smtp", style="{}") logger.debug("incoming email from {addr}", addr=sender_address) (of course, different users of the "smtp" logger can request different formatting styles when calling getLogger().) We could combine the various proposals to give users flexible APIs. Of course, it generally smells of "there's more than one way to do it". > It's the Rumsfeldian "We don't know what we don't know" Is this guy in the Python community? :-) > I'm just trying to think ahead, and can't claim to have got a definitive answer > up my sleeve. Sure, we have some time until 2.7/3.2 anyway. Regards Antoine. From p.f.moore at gmail.com Thu Oct 1 15:11:27 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Oct 2009 14:11:27 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> Message-ID: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> 2009/10/1 Vinay Sajip : > If any module wants to use {} formatting for their logging, they can add the > line > > from logging import BraceMessage as __ > > I've used two underscores, since _ might be being used for gettext, but > obviously the importer can use whatever name they want. > > and then they can use > > logger.debug(__("The {0} is {1}", "answer", 42)) > > which I think is more readable than putting in ".format" following the string > literal. It's not a *huge* point, perhaps, but "Readability counts". > > This has the side benefit that if e.g. Barry wanted to use string.Template for > formatting, he's just got to replace the above import with something like > > from logging import DollarMessage as __ > > Another "working title", please note. And while I've shown these classes being > imported from logging, it doesn't make sense to put them there if this idea > were to fly in a more general context. Then, perhaps string would be a better > home for these classes. This seems to me to be almost the same as the previous suggestion of having a string subclass: class BraceFormatter(str): def __mod__(self, other): # Needs more magic here to cope with dict argument return self.format(*other) __ = BraceFormatter logger.debug(__("The {0} is {1}"), "answer", 42) The only real differences are 1. The positioning of the closing parenthesis 2. The internal implementation of logger.debug needs to preserve string subclasses properly But the benefit is that the approach allows anyone to use brace formatting in any API that currently accepts % format (assuming string subclasses don't get mangled). On the one hand, I'd prefer a more general solution. On the other, I'm nervous about that "assuming string subclasses..." proviso. I've no real answer, just offering the point up for consideration. Paul. From vinay_sajip at yahoo.co.uk Thu Oct 1 15:29:29 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 13:29:29 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: Paul Moore gmail.com> writes: > This seems to me to be almost the same as the previous suggestion of > having a string subclass: > > class BraceFormatter(str): > def __mod__(self, other): > # Needs more magic here to cope with dict argument > return self.format(*other) > > __ = BraceFormatter > > logger.debug(__("The {0} is {1}"), "answer", 42) > > The only real differences are > > 1. The positioning of the closing parenthesis > 2. The internal implementation of logger.debug needs to preserve > string subclasses properly > The other difference is that my suggestion supports Barry's desire to use string.Template with no muss, no fuss ;-) Plus, very little additional work is required compared to your items 1 and 2. ISTM BraceMessage would be something like this, clsss BraceMessage: def __init__(self, fmt, *args, **kwargs): self.fmt = fmt self.args = args self.kwargs = kwargs def __str__(self): return self.fmt.format(*self.args, **self.kwargs) Regards, Vinay From vinay_sajip at yahoo.co.uk Thu Oct 1 15:39:04 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 13:39:04 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: Paul Moore gmail.com> writes: > 2. The internal implementation of logger.debug needs to preserve > string subclasses properly > > But the benefit is that the approach allows anyone to use brace > formatting in any API that currently accepts % format (assuming string > subclasses don't get mangled). > > On the one hand, I'd prefer a more general solution. On the other, I'm > nervous about that "assuming string subclasses..." proviso. > You're right to be nervous: it's not hard to mangle subtypes. >>> class mystr(str): pass ... >>> s = mystr("Abc") >>> s 'Abc' >>> type(s) >>> s2 = s.replace("A", "a") >>> s2 'abc' >>> type(s2) >>> From vinay_sajip at yahoo.co.uk Thu Oct 1 16:37:29 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 14:37:29 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> Message-ID: Antoine Pitrou pitrou.net> writes: > There's another possibility, which is to build the wrapping directly around the > logger. That is, if I want a %-style logger, I do: > > logger = logging.getLogger("smtp") > logger.debug("incoming email from %s", sender_address) > > and I want a {}-style logger, I do: > > logger = logging.getLogger("smtp", style="{}") > logger.debug("incoming email from {addr}", addr=sender_address) > > (of course, different users of the "smtp" logger can request different > formatting styles when calling getLogger().) There's a LoggerAdapter class already in the system which is used to wrap loggers so that additional contextual information (e.g. network or database connection information) can be added to logs. The LoggerAdapter could fulfill this "wrapping" function. > We could combine the various proposals to give users flexible APIs. Of course, > it generally smells of "there's more than one way to do it". Yeah, that bothers me too. > > It's the Rumsfeldian "We don't know what we don't know" > > Is this guy in the Python community? Not sure, but I believe he's a piece of work and not a guy to get on the wrong side of ;-) Regards, Vinay Sajip From foom at fuhm.net Thu Oct 1 17:42:38 2009 From: foom at fuhm.net (James Y Knight) Date: Thu, 1 Oct 2009 11:42:38 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: <2650CF0D-E8B8-42C0-9063-9B88E634DBF6@fuhm.net> On Oct 1, 2009, at 9:11 AM, Paul Moore wrote: > This seems to me to be almost the same as the previous suggestion of > having a string subclass: > > class BraceFormatter(str): > def __mod__(self, other): > # Needs more magic here to cope with dict argument > return self.format(*other) > > __ = BraceFormatter > > logger.debug(__("The {0} is {1}"), "answer", 42) I'd rather make that: class BraceFormatter: def __init__(self, s): self.s = s def __mod__(self, other): # Needs more magic here to cope with dict argument return s.format(*other) __ = BraceFormatter That is, *not* a string subclass. Then if someone attempts to mangle it, or use it for anything but %, it fails loudly. James From python at rcn.com Thu Oct 1 17:50:28 2009 From: python at rcn.com (Raymond Hettinger) Date: Thu, 1 Oct 2009 08:50:28 -0700 Subject: [Python-Dev] transitioning from % to {} formatting References: <200910010034.27427.steve@pearwood.info> Message-ID: <68A57C0BA3E84957A968BE16CA3E5ADC@RaymondLaptop1> [Vinay Sajip] > And the special wrappers needn't be too intrusive: > > __ = BraceMessage > > logger.debug(__("Message with {0} {1}", 1, "argument")) It looks like the BraceMessage would have to re-instantiate on every invocation. Raymond From janssen at parc.com Thu Oct 1 18:01:21 2009 From: janssen at parc.com (Bill Janssen) Date: Thu, 1 Oct 2009 09:01:21 PDT Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC46070.1010706@scottdial.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> <4AC46070.1010706@scottdial.com> Message-ID: <59160.1254412881@parc.com> Scott Dial wrote: > Scott Dial wrote: > > While this code is present in > > older versions of python, it seems to have become a problem recently > > (2009-05-06 is the earliest report on the issue) perhaps due to a > > version bump of OpenSSL? I never noticed the problem in python2.5 even > > though the code is unchanged. > > To answer my own question, this was introduced in r64578 in ssl.py by > the addition of the suppress_ragged_eofs feature (seems to originate > with issue1251). While it is a change in the behavior of the ssl module, > the change falls in line with other file-like objects and their handling > of EOF, so the bug still falls to imaplib. > > In other words, this bug appeared in 2.6 and 3.0, and is present in all > subsequent versions. Thanks for bringing this up, Scott. Bill From foom at fuhm.net Thu Oct 1 18:15:15 2009 From: foom at fuhm.net (James Y Knight) Date: Thu, 1 Oct 2009 12:15:15 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> Message-ID: <894C0271-B3CC-49CD-B098-41EFBD27EF49@fuhm.net> On Sep 30, 2009, at 1:01 PM, Antoine Pitrou wrote: > Why not allow logging.Formatter to take a callable, which would in > turn call the > callable with keyword arguments? > > Therefore, you could write: > logging.Formatter("{asctime} - {name} - {level} - {msg}".format) > > and then: > logging.critical(name="Python", msg="Buildbots are down") > > All this without having to learn about a separate "compatibility > wrapper object". It's a nice idea -- but I think it's better for the wrapper (whatever form it takes) to support __mod__ so that logging.Formatter (and everything else) doesn't need to be modified to be able to know about how to use both callables and "%"ables. Is it possible for a C function like str.format to have other methods defined on its function type? James From vinay_sajip at yahoo.co.uk Thu Oct 1 18:21:00 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 16:21:00 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <68A57C0BA3E84957A968BE16CA3E5ADC@RaymondLaptop1> Message-ID: Raymond Hettinger rcn.com> writes: > It looks like the BraceMessage would have to re-instantiate on every > invocation. True, because the arguments to the instantiation are kept around as a BraceMessage instance until the time comes to actually format the message (which might be never). Since typically in performance-sensitive code, the isEnabledFor pattern is used to avoid doing unnecessary work, as in if logger.isEnabledFor(logging.DEBUG): logger.debug(__("The {0} is {1}", "answer", 42)) The BraceMessage invocation overhead is only incurred when needed, as is the cost of computing the additional arguments. As I understand it {}-formatting is slower than %-formatting anyway, and if this pattern is used only for {}-formatting, then there will be no additional overhead for %-formatting and some additional overhead for {}-formatting. I'm not sure what that instantiation cost will be relative to the overall time for an "average" call - whatever that is ;-) - though. Other approaches to avoid instantiation could be considered: for example, making __ a callable which remembers previous calls and caches instances keyed by the call arguments. But this will incur memory overhead and some processing overhead and I'm not sure if it really buys you enough to warrant doing it. Regards, Vinay Sajip From glenn at nevcal.com Thu Oct 1 18:49:24 2009 From: glenn at nevcal.com (Glenn Linderman) Date: Thu, 01 Oct 2009 09:49:24 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: <4AC4DD94.1010205@nevcal.com> On approximately 9/30/2009 4:03 PM, came the following characters from the keyboard of Vinay Sajip: > Steven Bethard gmail.com> writes: > >> There's a lot of code already out there (in the standard library and >> other places) that uses %-style formatting, when in Python 3.0 we >> should be encouraging {}-style formatting. We should really provide >> some sort of transition plan. Consider an example from the logging >> docs: >> >> logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") >> >> We'd like to support both this style as well as the following style: >> >> logging.Formatter("{asctime} - {name} - {levelname} - {message}") >> > > In logging at least, there are two different places where the formatting issue > crops up. > > The first is creating the "message" part of the the logging event, which is > made up of a format string and arguments. > > The second is the one Steven's mentioned: formatting the message along with > other event data such as time of occurrence, level, logger name etc. into the > final text which is output. It seems to me that most of the discussion of in this thread is concerned with the first issue... and yet I see the second as the harder issue, and it has gotten less press. Abstracting this away from logger, I think the problem has three cases: 1) Both the format message and all the parameters are supplied in a single API call. This is really a foolish API, because def API( fmt, p1, p2, p3 ): str = fmt % (p1, p2, p3) could have just as easily been documented originally as def API( str ): where the user is welcome to supply a string such as API( fmt % (p1, p2, p3 )) and if done this way, the conversion to .format is obvious... and all under the users control. 2) The format message and the parameters are supplied to separate APIs, because the format message is common to many invocations of the other APIs that supply parameters, and is cached by the API. This is sufficient to break the foolishness of #1, but is really just a subset of #3, so any solutions to #3 apply here. 3) The format message and the parameters for it may be supplied by the same or separate APIs, but one or both are incomplete, and are augmented by the API. In other words, one or both of the following cases: 3a) The user supplied format message may include references to named parameters that are documented by the API, and supplied by the API, rather than by the user. 3b) The user supplied format string may be embedded into a larger format string by the API, which contains references to other values that the user must also supply. In either case of 3a or 3b, the user has insufficient information to perform the whole format operation and pass the result to the API. In both cases, the API that accepts the format string must be informed whether it is a % or {} string, somehow. This could be supplied to the API that accepts the string, or to some other related API that sets a format mode. Internally, the code would have to be able to manipulate both types of formats. > Support for both % and {} forms in logging would need to be considered in > these two places. I sort of liked Martin's proposal about using different > keyword arguments, but apart from the ugliness of "dicttemplate" and the fact > that "fmt" is already used in Formatter.__init__ as a keyword argument, it's > possible that two different keyword arguments "fmt" and "format" both referring > to format strings might be confusing to some users. > > Benjamin's suggestion of providing a flag to Formatter seems slightly better, > as it doesn't change what existing positional or keyword parameters do, and > just adds an additional, optional parameter which can start off with a default > of False and transition to a default of True. > > However, AFAICT these approaches only cover the second area where formatting > options are chosen - not the creation of the message from the parameters passed > to the logging call itself. The above three paragraphs are unclear to me. I think they might be referring to case 2 or 3, though. > Of course one can pass arbitrary objects as messages which contain their own > formatting logic. This has been possible since the very first release but I'm > not sure that it's widely used, as it's usually easier to pass strings. So > instead of passing a string and arguments such as > > logger.debug("The %s is %d", "answer", 42) > > one can currently pass, for a fictitious class PercentMessage, > > logger.debug(PercentMessage("The %s is %d", "answer", 42)) > > and when the time comes to obtain the formatted message, LogRecord.getMessage > calls str() on the PercentMessage instance, whose __str__ will use %-formatting > to get the actual message. > > Of course, one can also do for example > > logger.debug(BraceMessage("The {} is {}", "answer", 42)) > > where the __str__() method on the BraceMessage will do {} formatting. > > Of course, I'm not suggesting we actually use the names PercentMessage and > BraceMessage, I've just used them there for clarity. It seems that the above is only referring to case 1? And doesn't help with case 2 or 3? > Also, although Raymond has pointed out that it seems likely that no one ever > needs *both* types of format string, what about the case where application A > depends on libraries B and C, and they don't all share the same preferences > regarding which format style to use? ISTM no-one's brought this up yet, but it > seems to me like a real issue. It would certainly appear to preclude any > approach that configured a logging-wide or logger-wide flag to determine how to > interpret the format string. Agreed here... a single global state would not make modular upgrades to a complex program easy... the state would be best included with particular instance objects, especially when such instance objects exist already. The format type parameter could be provided to the instance, instead of globally. > Another potential issue is where logging events are pickled and sent over > sockets to be finally formatted and output on different machines. What if a > sending machine has a recent version of Python, which supports {} formatting, > but a receiving machine doesn't? It seems that at the very least, it would > require a change to SocketHandler and DatagramHandler to format the "message" > part into the LogRecord before pickling and sending. While making this change > is simple, it represents a potential backwards-incompatible problem for users > who have defined their own handlers for doing something similar. > > Apart from thinking through the above issues, the actual formatting only > happens in two locations - LogRecord.getMessage and Formatter.format - so > making the code do either %- or {} formatting would be simple, as long as it > knows which of % and {} to pick. > > Does it seems too onerous to expect people to pass an additional "use_format" > keyword argument with every logging call to indicate how to interpret the > message format string? Or does the PercentMessage/BraceMessage type approach > have any mileage? What do y'all think? These last 3 paragraphs seem to be very related to logger, specifically. The first of the 3 does point out a concern for systems that interoperate across networks: if the format strings and parameters are exposed separately across networks, whatever types are sent must be usable at the receiver, or at least appropriate version control must be required so that incompatible systems can be detected and reported. On approximately 9/30/2009 5:47 PM, came the following characters from the keyboard of Antoine Pitrou: > Vinay Sajip yahoo.co.uk> writes: >> Does it seems too onerous to expect people to pass an additional >> "use_format" keyword argument with every logging call to indicate how >> to interpret the message format string? Or does the >> PercentMessage/BraceMessage type approach have any mileage? What do >> y'all think? > > What about the proposal I made earlier? > (support for giving a callable, so that you pass the "{foobar}".format > method when you want new-style formatting) This "callable" technique seems to only support case 1 and 2, but not 3, unless I misunderstand it. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From scott+python-dev at scottdial.com Thu Oct 1 19:47:08 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Thu, 01 Oct 2009 13:47:08 -0400 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC48DFD.7020607@gmail.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> <4AC48DFD.7020607@gmail.com> Message-ID: <4AC4EB1C.7030302@scottdial.com> Nick Coghlan wrote: > Scott Dial wrote: >> I would appreciate this bug being resolved before the next release as it >> effects me on a daily basis. I have submitted a patch, which reflects my >> local solution. > > Unfortunately, it's almost certainly too late to get this into 2.6.3. It > really needed to be brought up back when Barry called for identification > of significant 2.6 bugs and patches rather than after the RC had already > been released. > I understand. I didn't figure out the bug until after rc1 landed. It was only happening spuriously with my mail server, never when I manually tried to invoke the problem. I was forced to let my cronjob run until the kernel killed getmail for OOM, giving me the trace shown in the issue. It is impossible to break anything with this patch, as no program could proceed. The 3-line patch merely converts it back into the exception that was originally raised prior to 2.6, so it's not a new behavior in that respect either. I wouldn't have anticipated this much resistance to removing an infinite loop from the standard library. I could also for this patch from the angle that it allows a remote host the ability to execute a denial-of-service attack (even by accident!) since the infinite loop appends an empty string to a list on every loop, taking CPU time and memory with it. Allow me to be naive for a moment and say, is this not the point of rc1 but to catch bugs that should not be in the final? Of course, it's Barry's call. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From brett at python.org Thu Oct 1 20:03:39 2009 From: brett at python.org (Brett Cannon) Date: Thu, 1 Oct 2009 11:03:39 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: On Thu, Oct 1, 2009 at 06:29, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> This seems to me to be almost the same as the previous suggestion of >> having a string subclass: >> >> class BraceFormatter(str): >> ? ? def __mod__(self, other): >> ? ? ? ? # Needs more magic here to cope with dict argument >> ? ? ? ? return self.format(*other) >> >> __ = BraceFormatter >> >> logger.debug(__("The {0} is {1}"), "answer", 42) >> So I created this last night: import collections class braces_fmt(str): def __mod__(self, stuff): if isinstance(stuff, tuple): return self.__class__(self.format(*stuff)) elif isinstance(stuff, collections.Mapping): return self.__class__(self.format(**stuff)) else: return self.__class__(self.format(stuff)) The biggest issue is that ``"%s" % {'a': 42}`` substitutes the dict instead of throwing an error that str.format() would do with the code above. But what's nice about this is I think I can use this now w/ any library that expects % interpolation and it should basically work. >> The only real differences are >> >> 1. The positioning of the closing parenthesis >> 2. The internal implementation of logger.debug needs to preserve >> string subclasses properly >> > > The other difference is that my suggestion supports Barry's desire to use > string.Template with no muss, no fuss ;-) I don't think Paul's suggestion requires much more work to support string.Template, simply a subclass that implements __mod__> > Plus, very little additional work is > required compared to your items 1 and 2. ISTM BraceMessage would be something > like this, > > clsss BraceMessage: > ? ?def __init__(self, fmt, *args, **kwargs): > ? ? ? ?self.fmt = fmt > ? ? ? ?self.args = args > ? ? ? ?self.kwargs = kwargs > > ? ?def __str__(self): > ? ? ? ?return self.fmt.format(*self.args, **self.kwargs) I guess my question is what's the point of the class if you are simply converting it before you pass it in to the logger? To be lazy about the formatting call? Otherwise you could simply call str.format() with your arguments before you pass the string into the logger and not have to wrap anything. -Brett From barry at python.org Thu Oct 1 21:28:22 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Oct 2009 15:28:22 -0400 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC4EB1C.7030302@scottdial.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> <4AC48DFD.7020607@gmail.com> <4AC4EB1C.7030302@scottdial.com> Message-ID: On Oct 1, 2009, at 1:47 PM, Scott Dial wrote: > Allow me to be naive for a moment and say, > is this not the point of rc1 but to catch bugs that should not be in > the > final? Actually, no. The point of an rc is to avoid a brown paper bag mistake, i.e. something we really fscked up in the release like breaking "import sys" or the build process, etc. We're always gonna have bugs, so that can't be it. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From ncoghlan at gmail.com Thu Oct 1 23:15:14 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 02 Oct 2009 07:15:14 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC4DD94.1010205@nevcal.com> References: <4AC4DD94.1010205@nevcal.com> Message-ID: <4AC51BE2.8040206@gmail.com> Glenn Linderman wrote: > 1) Both the format message and all the parameters are supplied in a > single API call. This is really a foolish API, because > > def API( fmt, p1, p2, p3 ): > str = fmt % (p1, p2, p3) > > could have just as easily been documented originally as > > def API( str ): > > where the user is welcome to supply a string such as > > API( fmt % (p1, p2, p3 )) > > and if done this way, the conversion to .format is obvious... and all > under the users control. The lazy APIs actually make a lot of sense, particularly when there is a chance that the function being called may be able to avoid the formatting call altogether. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Thu Oct 1 23:33:41 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 21:33:41 +0000 (GMT) Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: <69124.66598.qm@web25807.mail.ukl.yahoo.com> > So I created this last night: > > import collections > > class braces_fmt(str): > > def __mod__(self, stuff): > if isinstance(stuff, tuple): > return self.__class__(self.format(*stuff)) > elif isinstance(stuff, collections.Mapping): > return self.__class__(self.format(**stuff)) > else: > return self.__class__(self.format(stuff)) > > The biggest issue is that ``"%s" % {'a': 42}`` substitutes the dict > instead of throwing an error that str.format() would do with the code > above. But what's nice about this is I think I can use this now w/ any > library that expects % interpolation and it should basically work. So there's no need to change modules like logging to explicitly provide support for {}-formatting? What's not to like? ;-) Something like this perhaps should have been added in at the same time as str.format went in. > I don't think Paul's suggestion requires much more work to support > string.Template, simply a subclass that implements __mod__ True. > I guess my question is what's the point of the class if you are simply > converting it before you pass it in to the logger? To be lazy about > the formatting call? Otherwise you could simply call str.format() with > your arguments before you pass the string into the logger and not have > to wrap anything. That's exactly the reason - to defer the formatting until it's needed. Otherwise you can always format the string yourself,as you say, and pass it as the single argument in the logging call - logging won't know or care if it was passed in as a literal, or was computed by %-, {}-, $- or any other formatting approach. Regards, Vinay Sajip From ncoghlan at gmail.com Thu Oct 1 23:11:57 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 02 Oct 2009 07:11:57 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: <4AC51B1D.7040203@gmail.com> Vinay Sajip wrote: > You're right to be nervous: it's not hard to mangle subtypes. > >>>> class mystr(str): pass > ... >>>> s = mystr("Abc") >>>> s > 'Abc' >>>> type(s) > >>>> s2 = s.replace("A", "a") >>>> s2 > 'abc' >>>> type(s2) > It's also difficult for the subclass to prevent this without creating an infinite loop... (I only spent about 10 minutes looking into it the other day, but that's what happened in all of my naive attempts at doing it in pure Python code). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From sridharr at activestate.com Thu Oct 1 23:48:19 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 01 Oct 2009 14:48:19 -0700 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AA7ADCB.6080101@gmail.com> <11A6545D-7204-4F61-B55B-1CC77CB5645E@python.org> <9D506035-7C2D-4929-A134-E88EEB7B7D9E@python.org> <21F203FA-05CE-433B-AD16-1E7804C2C08B@python.org> Message-ID: On Wed, 30 Sep 2009 13:06:47 -0700, Sridhar Ratnakumar wrote: > On Wed, 30 Sep 2009 12:44:14 -0700, Barry Warsaw > wrote: > >> 2.6.3rc1 builds fine on Linux x86/x86_64, MacOSX 10.4 ppc/x86, Windows >> 32bit/64bit, HP-UX, AIX and Solaris just like 2.6.2 did. >> Thanks for the feedback! Did you run the test suite on any of these? > > I will run the tests sometime tonight and let you know. No new significant failures have been found. (Some trivial issues have been reported in the bug tracker). -srid From ncoghlan at gmail.com Thu Oct 1 23:54:20 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 02 Oct 2009 07:54:20 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <69124.66598.qm@web25807.mail.ukl.yahoo.com> References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> Message-ID: <4AC5250C.8030403@gmail.com> Vinay Sajip wrote: > So there's no need to change modules like logging to explicitly > provide support for {}-formatting? What's not to like? ;-) Something > like this perhaps should have been added in at the same time as > str.format went in. I believe classes like fmt_braces/fmt_dollar/fmt_percent will be part of a solution, but they aren't a complete solution on their own. (Naming the three major string formatting techniques by the key symbols involved is a really good idea though) The two major problems with them: 1. It's easy to inadvertently convert them back to normal strings. If a formatting API even calls "str" on the format string then we end up with a problem (and switching to containment instead of inheritance doesn't really help, since all objects implement __str__). 2. They don't help with APIs that expect a percent-formatted string and do more with it than just pass it to str.__mod__ (e.g. inspecting it for particular values such as '%(asctime)s') Still, it's worth considering adding the three fmt_* classes to the string module to see how far they can get us in adapting the formats for different APIs. Note that I don't think these concepts are fully baked yet, so we shouldn't do anything in a hurry - and anything that does happen should be via a PEP so we can flush out more issues. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Oct 1 23:25:19 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 02 Oct 2009 07:25:19 +1000 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: <4AC4EB1C.7030302@scottdial.com> References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AC45BFA.3020802@scottdial.com> <4AC48DFD.7020607@gmail.com> <4AC4EB1C.7030302@scottdial.com> Message-ID: <4AC51E3F.9070601@gmail.com> Scott Dial wrote: > Allow me to be naive for a moment and say, > is this not the point of rc1 but to catch bugs that should not be in the > final? For an x.y.0 rc I would usually agree with you, but for x.y.z (where z > 0), the intended situation is for there to be zero functional changes between the rc and the release. (This actually holds true for the final rc in x.y.0 release as well). It's unfortunate, but the imminent point release gets people's attention in a way that the original "there is going to be a point release around this date" never seems to. If the release managers didn't draw a line and said "no more bug fixes, even minor ones" then we'd never get point releases out even close to on time. This is particularly so for bugs that aren't a regression from x.y.(z-1) - it's most often regressions which are marked as release blockers rather than newly discovered (or analysed) bugs which have existed in the series since the x.y.0 release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From foom at fuhm.net Fri Oct 2 00:03:47 2009 From: foom at fuhm.net (James Y Knight) Date: Thu, 1 Oct 2009 18:03:47 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC5250C.8030403@gmail.com> References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> Message-ID: <51684EC1-96B7-480E-A01D-871879308E2F@fuhm.net> On Oct 1, 2009, at 5:54 PM, Nick Coghlan wrote: > I believe classes like fmt_braces/fmt_dollar/fmt_percent will be > part of > a solution, but they aren't a complete solution on their own. (Naming > the three major string formatting techniques by the key symbols > involved > is a really good idea though) > > 1. It's easy to inadvertently convert them back to normal strings. > If a > formatting API even calls "str" on the format string then we end up > with > a problem (and switching to containment instead of inheritance doesn't > really help, since all objects implement __str__). Using containment instead of inheritance makes sure none of the *other* operations people do on strings will appear to work, at least (substring, contains, etc). I bet explicitly calling str() on a format string is even more rare than attempting to do those things. > 2. They don't help with APIs that expect a percent-formatted string > and > do more with it than just pass it to str.__mod__ (e.g. inspecting it > for > particular values such as '%(asctime)s') True, but I don't think there's many such cases in the first place, and such places can be fixed to not do that as they're found. Until they are fixed, fmt_braces will loudly fail when used with that API (assuming fmt_braces is not a subclass of str). James From vinay_sajip at yahoo.co.uk Fri Oct 2 00:12:03 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 22:12:03 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> <51684EC1-96B7-480E-A01D-871879308E2F@fuhm.net> Message-ID: James Y Knight fuhm.net> writes: > Using containment instead of inheritance makes sure none of the > *other* operations people do on strings will appear to work, at least > (substring, contains, etc). I bet explicitly calling str() on a format > string is even more rare than attempting to do those things. Actually, logging calls str() on the object passed as the first argument in a logging call such as logger.debug(), which can either be a format string or an arbitrary object whose __str__() returns the format string. Regards, Vinay Sajip From steven.bethard at gmail.com Fri Oct 2 00:19:38 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 1 Oct 2009 15:19:38 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: On Thu, Oct 1, 2009 at 11:03 AM, Brett Cannon wrote: > So I created this last night: > > import collections > > class braces_fmt(str): > > ? ?def __mod__(self, stuff): > ? ? ? ?if isinstance(stuff, tuple): > ? ? ? ? ? ?return self.__class__(self.format(*stuff)) > ? ? ? ?elif isinstance(stuff, collections.Mapping): > ? ? ? ? ? ?return self.__class__(self.format(**stuff)) > ? ? ? ?else: > ? ? ? ? ? ?return self.__class__(self.format(stuff)) > > The biggest issue is that ``"%s" % {'a': 42}`` substitutes the dict > instead of throwing an error that str.format() would do with the code > above. But what's nice about this is I think I can use this now w/ any > library that expects % interpolation and it should basically work. I see how this could allow a user to supply a {}-format string to an API that accepts only %-format strings. But I still don't see the transition strategy for the API itself. That is, how does the %-format API use this to eventually switch to {}-format strings? Could someone please lay it out for me, step by step, showing what happens in each version? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From vinay_sajip at yahoo.co.uk Fri Oct 2 00:39:03 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Oct 2009 22:39:03 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > I believe classes like fmt_braces/fmt_dollar/fmt_percent will be part of > a solution, but they aren't a complete solution on their own. (Naming > the three major string formatting techniques by the key symbols involved > is a really good idea though) > > The two major problems with them: > > 1. It's easy to inadvertently convert them back to normal strings. If a > formatting API even calls "str" on the format string then we end up with > a problem (and switching to containment instead of inheritance doesn't > really help, since all objects implement __str__). > > 2. They don't help with APIs that expect a percent-formatted string and > do more with it than just pass it to str.__mod__ (e.g. inspecting it for > particular values such as '%(asctime)s') Good point as far as the general case is concerned, though it's perhaps not that critical for logging. By which I mean, it's not unreasonable for Formatter.__init__ to grow a "style" keyword parameter which determines whether it uses %-, {}- or $-formatting. Then the formatter can look for '%(asctime)s', '{asctime}' or '$asctime' according to the style. Just to clarify - LogRecord.getMessage *will* call str() on a message object if it's not a string or Unicode object. For 2.x the logic is if type(msg) not in (unicode, str): msg = str(msg) and for 3.x the check is for isinstance(msg, str). > Still, it's worth considering adding the three fmt_* classes to the > string module to see how far they can get us in adapting the formats for > different APIs. > > Note that I don't think these concepts are fully baked yet, so we > shouldn't do anything in a hurry - and anything that does happen should > be via a PEP so we can flush out more issues. Yes, we're just "kicking the tires" on the various ideas. There are things still a bit up in the air such as what happens when pickling and sending to an older version of Python, etc. which still need to be resolved for logging, at least. Regards, Vinay Sajip From barry at python.org Fri Oct 2 00:40:51 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Oct 2009 18:40:51 -0400 Subject: [Python-Dev] Python 2.6.3 In-Reply-To: References: <1BEC6749-D905-456F-BD34-9AC8B3F4CF2F@python.org> <4AA7ADCB.6080101@gmail.com> <11A6545D-7204-4F61-B55B-1CC77CB5645E@python.org> <9D506035-7C2D-4929-A134-E88EEB7B7D9E@python.org> <21F203FA-05CE-433B-AD16-1E7804C2C08B@python.org> Message-ID: On Oct 1, 2009, at 5:48 PM, Sridhar Ratnakumar wrote: > No new significant failures have been found. (Some trivial issues > have been reported in the bug tracker). Fantastic, thanks. I'll be tagging the final release soon. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From eric at trueblade.com Fri Oct 2 00:58:31 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 01 Oct 2009 18:58:31 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> Message-ID: <4AC53417.1040405@trueblade.com> Vinay Sajip wrote: > Good point as far as the general case is concerned, though it's perhaps not that > critical for logging. By which I mean, it's not unreasonable for > Formatter.__init__ to grow a "style" keyword parameter which determines whether > it uses %-, {}- or $-formatting. Then the formatter can look for '%(asctime)s', > '{asctime}' or '$asctime' according to the style. It's tangential, but in the str.format case you don't want to check for just '{asctime}', because you might want '{asctime:%Y-%m-%d}', for example. But there are ways to delay computing the time until you're sure it's actually being used in the format string, without parsing the format string. Now that I think of it, the same technique could be used with %-formatting: import datetime class DelayedStr: def __init__(self, fn): self.fn = fn self.obj = None def __str__(self): if self.obj is None: self.obj = self.fn() return self.obj.__str__() def current_time(): print "calculating time" return datetime.datetime.now() # will not compute current time print '%(msg)s' % {'asctime':DelayedStr(current_time), 'msg':'test'} # will compute current time: same dict used as before print '%(asctime)s %(msg)s' % {'asctime':DelayedStr(current_time), 'msg':'test'} Eric. From foom at fuhm.net Fri Oct 2 01:13:28 2009 From: foom at fuhm.net (James Y Knight) Date: Thu, 1 Oct 2009 19:13:28 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <200910010034.27427.steve@pearwood.info> <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: <5EC96CF6-C48F-41E1-B32C-816045DBE91D@fuhm.net> On Oct 1, 2009, at 6:19 PM, Steven Bethard wrote: > I see how this could allow a user to supply a {}-format string to an > API that accepts only %-format strings. But I still don't see the > transition strategy for the API itself. That is, how does the %-format > API use this to eventually switch to {}-format strings? Could someone > please lay it out for me, step by step, showing what happens in each > version? Here's what I said in my first message, suggesting this change. Copy&pasted below: I wrote: > 1) introduce the above feature, and recommend in docs that people > only ever use new-style format strings, wrapping the string in > newstyle_formatstr() when necessary for passing to an API which uses > % internally. > 2) A long time later...deprecate str.__mod__; don't deprecate > newstyle_formatstr.__mod__. > 3) A while after that (maybe), remove str.__mod__ and replace all > calls in Python to % (used as a formatting operator) with .format() > so that the default is to use newstyle format strings for all APIs > from then on. So do (1) in 3.2. Then do (2) in 3.4, and (3) in 3.6. I skipped two versions each time because of how widely this API is used, and the likely pain that doing the transition quickly would cause. But I guess you *could* do it in one version each step. James From brett at python.org Fri Oct 2 01:39:49 2009 From: brett at python.org (Brett Cannon) Date: Thu, 1 Oct 2009 16:39:49 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC5250C.8030403@gmail.com> References: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> Message-ID: On Thu, Oct 1, 2009 at 14:54, Nick Coghlan wrote: > Vinay Sajip wrote: >> So there's no need to change modules like logging to explicitly >> provide support for {}-formatting? What's not to like? ;-) Something >> like this perhaps should have been added in at the same time as >> str.format went in. > > I believe classes like fmt_braces/fmt_dollar/fmt_percent will be part of > a solution, but they aren't a complete solution on their own. I agree. I view them more as a band-aid over APIs that only accept % formatting but the user of the library wants to use {} formatting. > (Naming > the three major string formatting techniques by the key symbols involved > is a really good idea though) > > The two major problems with them: > > 1. It's easy to inadvertently convert them back to normal strings. If a > formatting API even calls "str" on the format string then we end up with > a problem (and switching to containment instead of inheritance doesn't > really help, since all objects implement __str__). > Well, you can override the methods on str to always return the proper thing, e.g. ``def __str__(self): return self``. Do the same for __add__() and all other methods on strings that return a string themselves. It should be possible to prevent Python code from stripping off the class. > 2. They don't help with APIs that expect a percent-formatted string and > do more with it than just pass it to str.__mod__ (e.g. inspecting it for > particular values such as '%(asctime)s') > Nope, they don't and people would need to be warned against this. > Still, it's worth considering adding the three fmt_* classes to the > string module to see how far they can get us in adapting the formats for > different APIs. > > Note that I don't think these concepts are fully baked yet, so we > shouldn't do anything in a hurry - and anything that does happen should > be via a PEP so we can flush out more issues. Having a PEP that lays out how we think people should consider transitioning their code would be good. -Brett From steven.bethard at gmail.com Fri Oct 2 03:13:12 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 1 Oct 2009 18:13:12 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: On Thu, Oct 1, 2009 at 4:35 PM, Brett Cannon wrote: > On Thu, Oct 1, 2009 at 15:19, Steven Bethard wrote: >> On Thu, Oct 1, 2009 at 11:03 AM, Brett Cannon wrote: >>> class braces_fmt(str): >>> >>> ? ?def __mod__(self, stuff): >>> ? ? ? ?if isinstance(stuff, tuple): >>> ? ? ? ? ? ?return self.__class__(self.format(*stuff)) >>> ? ? ? ?elif isinstance(stuff, collections.Mapping): >>> ? ? ? ? ? ?return self.__class__(self.format(**stuff)) >>> ? ? ? ?else: >>> ? ? ? ? ? ?return self.__class__(self.format(stuff)) >>> >>> The biggest issue is that ``"%s" % {'a': 42}`` substitutes the dict >>> instead of throwing an error that str.format() would do with the code >>> above. But what's nice about this is I think I can use this now w/ any >>> library that expects % interpolation and it should basically work. >> >> Could someone please lay it out for me, step by step, showing what >> happens in each version? > > 1. Nothing changes as hopefully the wrapper works fine (as people are > pointing out, though, my approach needs to override __str__() to > return 'self', else the str type will just return what it has > internally in its buffer). > > 2. Raise a deprecation warning when ``isinstance(ob, brace_fmt)`` is > false. When a class is passed in that is a subclass of brace_fmt, call > ob.format() on it. > > 3. Require the subclass. > > 4. Remove the requirement and always call ob.format(). Thanks Brett, that's clear. So you save one version over the proposal of adding a format= flag to the API. On Thu, Oct 1, 2009 at 4:13 PM, James Y Knight wrote: > Here's what I said in my first message, suggesting this change. Copy&pasted > below: [snip steps that only talk about str.__mod__, not an API that uses it] I didn't understand how you wanted to apply your suggestion to an API (instead of str.__mod__) the first time and I still don't understand it. Is what Brett has proposed the same thing? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From songofacandy at gmail.com Fri Oct 2 04:07:57 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 2 Oct 2009 11:07:57 +0900 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? Message-ID: I found this hg's issue. http://mercurial.selenic.com/bts/msg8375 I think below fix is not enabled on Mac OS X. http://svn.python.org/view/python/trunk/Include/pyport.h?view=diff&pathrev=43219&r1=36792&r2=36793 I can't confirm it because I am not Mac OS X user. Can anyone confirm it? -- Naoki INADA From tjreedy at udel.edu Fri Oct 2 07:49:34 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 02 Oct 2009 01:49:34 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: Antoine Pitrou wrote: > Vinay Sajip yahoo.co.uk> writes: >> Does it seems too onerous to expect people to pass an additional "use_format" >> keyword argument with every logging call to indicate how to interpret the >> message format string? Or does the PercentMessage/BraceMessage type approach >> have any mileage? What do y'all think? > > What about the proposal I made earlier? > (support for giving a callable, so that you pass the "{foobar}".format method > when you want new-style formatting) As someone who likes .format() and who already uses such bound methods to print, such as in emsg = "...".format ... if c: print(emsg(arg, barg)) I find this **MUCH** preferable to the ugly and seemingly unnecessary wrapper class idea being bandied about. This would be scarcely worse than passing the string itself. Terry Jan Reedy From steven.bethard at gmail.com Fri Oct 2 09:12:28 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 2 Oct 2009 00:12:28 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: On Thu, Oct 1, 2009 at 10:49 PM, Terry Reedy wrote: > As someone who likes .format() and who already uses such bound methods to > print, such as in > > emsg = "...".format > ... > ? if c: print(emsg(arg, barg)) > > I find this **MUCH** preferable to the ugly and seemingly unnecessary > wrapper class idea being bandied about. This would be scarcely worse than > passing the string itself. But it's not much of a transition plan. Or are you suggesting: (1) Make API accept callables (2) Issue warnings for regular strings (3) Throw exceptions for regular strings (4) Allow regular strings again, but assume {}-style formatting ? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From ronaldoussoren at mac.com Fri Oct 2 09:39:21 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 02 Oct 2009 09:39:21 +0200 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? In-Reply-To: References: Message-ID: <92839421634652657618189239787554831247-Webmail@me.com> On Friday, 02 October, 2009, at 04:07AM, "INADA Naoki" wrote: >I found this hg's issue. >http://mercurial.selenic.com/bts/msg8375 > >I think below fix is not enabled on Mac OS X. >http://svn.python.org/view/python/trunk/Include/pyport.h?view=diff&pathrev=43219&r1=36792&r2=36793 > >I can't confirm it because I am not Mac OS X user. >Can anyone confirm it? Do you have a testcase that shows what the problem is? Ronald From brett at python.org Fri Oct 2 01:35:44 2009 From: brett at python.org (Brett Cannon) Date: Thu, 1 Oct 2009 16:35:44 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> Message-ID: On Thu, Oct 1, 2009 at 15:19, Steven Bethard wrote: > On Thu, Oct 1, 2009 at 11:03 AM, Brett Cannon wrote: >> So I created this last night: >> >> import collections >> >> class braces_fmt(str): >> >> ? ?def __mod__(self, stuff): >> ? ? ? ?if isinstance(stuff, tuple): >> ? ? ? ? ? ?return self.__class__(self.format(*stuff)) >> ? ? ? ?elif isinstance(stuff, collections.Mapping): >> ? ? ? ? ? ?return self.__class__(self.format(**stuff)) >> ? ? ? ?else: >> ? ? ? ? ? ?return self.__class__(self.format(stuff)) >> >> The biggest issue is that ``"%s" % {'a': 42}`` substitutes the dict >> instead of throwing an error that str.format() would do with the code >> above. But what's nice about this is I think I can use this now w/ any >> library that expects % interpolation and it should basically work. > > I see how this could allow a user to supply a {}-format string to an > API that accepts only %-format strings. But I still don't see the > transition strategy for the API itself. That is, how does the %-format > API use this to eventually switch to {}-format strings? Could someone > please lay it out for me, step by step, showing what happens in each > version? First off, a wrapper like this I think is a temporary solution for libraries that do not have any transition strategy, not a replacement for one that is thought out (e.g. using a flag when appropriate). With that said, you could transition by: 1. Nothing changes as hopefully the wrapper works fine (as people are pointing out, though, my approach needs to override __str__() to return 'self', else the str type will just return what it has internally in its buffer). 2. Raise a deprecation warning when ``isinstance(ob, brace_fmt)`` is false. When a class is passed in that is a subclass of brace_fmt, call ob.format() on it. 3. Require the subclass. 4. Remove the requirement and always call ob.format(). -Brett From songofacandy at gmail.com Fri Oct 2 10:29:39 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 2 Oct 2009 17:29:39 +0900 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? In-Reply-To: <92839421634652657618189239787554831247-Webmail@me.com> References: <92839421634652657618189239787554831247-Webmail@me.com> Message-ID: > Do you have a testcase that shows what the problem is? > > Ronald >>> s = '\xa0' >>> assert s.strip() == s >>> import locale >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8') 'en_US.UTF-8' >>> assert s.strip() == s Second assert failed on Snow Leopard. -- Naoki INADA From solipsis at pitrou.net Fri Oct 2 11:34:30 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Oct 2009 09:34:30 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: Message-ID: Steven Bethard gmail.com> writes: > > But it's not much of a transition plan. Or are you suggesting: The question is why we want a transition plan that will bother everyone with no tangible benefits for the user. Regards Antoine. From barry at python.org Fri Oct 2 12:56:58 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 2 Oct 2009 06:56:58 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: On Oct 2, 2009, at 5:34 AM, Antoine Pitrou wrote: > Steven Bethard gmail.com> writes: >> >> But it's not much of a transition plan. Or are you suggesting: > > The question is why we want a transition plan that will bother > everyone with no > tangible benefits for the user. Forcing a transition to {}-strings might have limited usefulness, but enabling their use has clear benefits. You just then have to decide how important TOOWTDI is. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From status at bugs.python.org Fri Oct 2 18:07:38 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 2 Oct 2009 18:07:38 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20091002160738.226BE78644@psf.upfronthosting.co.za> ACTIVITY SUMMARY (09/25/09 - 10/02/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2410 open (+17) / 16447 closed (+24) / 18857 total (+41) Open issues with patches: 958 Average duration of open issues: 673 days. Median duration of open issues: 430 days. Open Issues Breakdown open 2376 (+17) pending 33 ( +0) Issues Created Or Reopened (44) _______________________________ Mismatching use of memory APIs 09/28/09 CLOSED http://bugs.python.org/issue6836 reopened r.david.murray patch threading.local subclasses don't cleanup their state and it gets 09/29/09 CLOSED http://bugs.python.org/issue6990 reopened pitrou patch, needs review PEP 314 inconsistency (authors/author/maintainer) 09/28/09 http://bugs.python.org/issue6992 reopened techtonik Python 3.1 interpreter Fatal Error on windows with unknown encod 09/25/09 CLOSED http://bugs.python.org/issue6995 created bahaelaila7 Python 2.6.2 & 3.1.1 Installer 09/25/09 CLOSED http://bugs.python.org/issue6996 created deepakp python26 compiled on snow leopard x86_64 09/25/09 CLOSED http://bugs.python.org/issue6997 created monway Bug in Tutorial (introduction.rst) 09/25/09 CLOSED http://bugs.python.org/issue6998 created cofi TypeError in pathfix.py 09/25/09 CLOSED http://bugs.python.org/issue6999 created amcnabb patch optional second argument of string.capwords not documented 09/25/09 CLOSED http://bugs.python.org/issue7000 created MLModel turtle.py bug-fixes, backported from 3.1 [issue5923] 09/25/09 CLOSED http://bugs.python.org/issue7001 created gregorlingl patch help me to set variables - 64 bit 09/26/09 CLOSED http://bugs.python.org/issue7002 created ashish finish documentation of user local paths 09/26/09 http://bugs.python.org/issue7003 created benjamin.peterson Py_RETURN_BOOL() convenience macro 09/27/09 CLOSED http://bugs.python.org/issue7004 created jon patch ConfigParser does not handle options without values 09/27/09 http://bugs.python.org/issue7005 created mkindahl patch The replacement suggested for callable(x) in py3k is not equival 09/27/09 http://bugs.python.org/issue7006 created milko.krachounov Tiny inconsistency in the orthography of "url encoded" in the do 09/27/09 http://bugs.python.org/issue7007 created MLModel str.title() misbehaves with apostrophes 09/27/09 CLOSED http://bugs.python.org/issue7008 created nickd random.randint docs 09/28/09 CLOSED http://bugs.python.org/issue7009 created jgsack Rounding when converting float tuple to str 09/28/09 CLOSED http://bugs.python.org/issue7010 created scientist47 tarinfo has problems with longlinks 09/28/09 CLOSED http://bugs.python.org/issue7011 created do3cc Tabs is better than spaces for indentation 09/28/09 CLOSED http://bugs.python.org/issue7012 created rohdef Httplib read routine is not tolerant to not well-formed chunked 09/28/09 http://bugs.python.org/issue7013 created Andrei Korostelev patch Logging incompatible with IronPython 09/28/09 CLOSED http://bugs.python.org/issue7014 created michael.foord Getting call trace while executing "modules spam" at help prompt 09/29/09 http://bugs.python.org/issue7015 created rishikesh patch .pyc files are set executable if the .py file is too 09/29/09 CLOSED http://bugs.python.org/issue7016 created stevenjd os.listdir behaviour 09/29/09 CLOSED http://bugs.python.org/issue7017 created bigaddo Recommend "*" over "#" in getargs.c typecodes 09/29/09 http://bugs.python.org/issue7018 created pitrou unmarshaling of artificial strings can produce funny longs. 09/29/09 http://bugs.python.org/issue7019 created cfbolz patch regression in pywin32 build due to 2.6.3rc1 09/30/09 CLOSED http://bugs.python.org/issue7020 created srid subprocess.Popen doesn't work without "executable=" 09/30/09 CLOSED http://bugs.python.org/issue7021 created hc Doc update for io module 09/30/09 CLOSED http://bugs.python.org/issue7022 created pakal Marshal doesn't release GIL while dumping 09/30/09 CLOSED http://bugs.python.org/issue7023 created zengke patch webbrowser : Could not open ftp server using webbrowser.open() 09/30/09 http://bugs.python.org/issue7024 created hmo Python 3.1 and 3.2 (dev) opensearch.xml still references 3.0 09/30/09 CLOSED http://bugs.python.org/issue7025 created jon test_urllib: unsetting missing 'env' variable 10/01/09 http://bugs.python.org/issue7026 created srid test_io.py: codecs.IncrementalDecoder is sometimes None 10/01/09 http://bugs.python.org/issue7027 created srid Add int.hex for symmetry with float.hex 10/02/09 http://bugs.python.org/issue7028 reopened mark.dickinson Improve pybench 10/01/09 http://bugs.python.org/issue7029 created krisvale patch Update version{added,changed} entries in py3k unittest docs 10/01/09 http://bugs.python.org/issue7030 created mark.dickinson add assertIsInstance 10/01/09 CLOSED http://bugs.python.org/issue7031 created georg.brandl patch, patch, easy, needs review Make assertMultilineEqual default for unicode string comparison 10/01/09 http://bugs.python.org/issue7032 created michael.foord C/API - Document exceptions 10/02/09 http://bugs.python.org/issue7033 created lekma While Loop Bug on a game of 21, help! 10/02/09 CLOSED http://bugs.python.org/issue7034 created PixelHead777 codecs error handlers lack documentation 10/02/09 http://bugs.python.org/issue7035 created olau Issues Now Closed (42) ______________________ Subprocess error with I/O redirection to Pipes 341 days http://bugs.python.org/issue4192 pakal os.popen2 and os.popen3 in python 2.6 incompatible with os.popen 221 days http://bugs.python.org/issue5329 pjenvey patch Python 3.0 grammar is ambiguous with the addition of star_expr 201 days http://bugs.python.org/issue5460 benjamin.peterson http://docs.python.org/dev/library/unittest.html#load-tests-prot 71 days http://bugs.python.org/issue6515 michael.foord csv.Sniffer.sniff on data with doublequotes doesn't set up the d 60 days http://bugs.python.org/issue6606 jtate patch Integer & Long types: Performance improvement of 1.6x to 2x for 42 days http://bugs.python.org/issue6713 mark.dickinson patch httplib and array do not play together well 33 days http://bugs.python.org/issue6790 pitrou patch, easy Mismatching use of memory APIs 0 days http://bugs.python.org/issue6836 krisvale patch Test creation in unittest.TestProgram should be done in one plac 6 days http://bugs.python.org/issue6956 michael.foord Extension modules fail to build on OS X 10.6 using python.org 2. 8 days http://bugs.python.org/issue6957 ronaldoussoren Add Python command line flags to configure logging 10 days http://bugs.python.org/issue6958 vinay.sajip patch numpy extensions to distutils... are a source of improvements fo 3 days http://bugs.python.org/issue6968 tjreedy Add the SIO_KEEPALIVE_VALS option to socket.ioctl 5 days http://bugs.python.org/issue6971 krisvale patch, patch, easy Compile error for Python-2.6.2 on Solaris 1 days http://bugs.python.org/issue6989 ashish threading.local subclasses don't cleanup their state and it gets 0 days http://bugs.python.org/issue6990 pitrou patch, needs review logging encoding failes some situation 1 days http://bugs.python.org/issue6991 naoki patch importing of "time" module is terribly slow 0 days http://bugs.python.org/issue6993 ndyankov enumerate dosctring has a typo 0 days http://bugs.python.org/issue6994 ezio.melotti Python 3.1 interpreter Fatal Error on windows with unknown encod 0 days http://bugs.python.org/issue6995 r.david.murray Python 2.6.2 & 3.1.1 Installer 0 days http://bugs.python.org/issue6996 deepakp python26 compiled on snow leopard x86_64 0 days http://bugs.python.org/issue6997 monway Bug in Tutorial (introduction.rst) 0 days http://bugs.python.org/issue6998 ezio.melotti TypeError in pathfix.py 0 days http://bugs.python.org/issue6999 benjamin.peterson patch optional second argument of string.capwords not documented 1 days http://bugs.python.org/issue7000 ezio.melotti turtle.py bug-fixes, backported from 3.1 [issue5923] 3 days http://bugs.python.org/issue7001 r.david.murray patch help me to set variables - 64 bit 0 days http://bugs.python.org/issue7002 brett.cannon Py_RETURN_BOOL() convenience macro 1 days http://bugs.python.org/issue7004 jon patch str.title() misbehaves with apostrophes 2 days http://bugs.python.org/issue7008 gvanrossum random.randint docs 0 days http://bugs.python.org/issue7009 orsenthil Rounding when converting float tuple to str 0 days http://bugs.python.org/issue7010 rhettinger tarinfo has problems with longlinks 0 days http://bugs.python.org/issue7011 do3cc Tabs is better than spaces for indentation 1 days http://bugs.python.org/issue7012 gvanrossum Logging incompatible with IronPython 1 days http://bugs.python.org/issue7014 michael.foord .pyc files are set executable if the .py file is too 0 days http://bugs.python.org/issue7016 r.david.murray os.listdir behaviour 1 days http://bugs.python.org/issue7017 bigaddo regression in pywin32 build due to 2.6.3rc1 2 days http://bugs.python.org/issue7020 srid subprocess.Popen doesn't work without "executable=" 0 days http://bugs.python.org/issue7021 hc Doc update for io module 1 days http://bugs.python.org/issue7022 pitrou Marshal doesn't release GIL while dumping 0 days http://bugs.python.org/issue7023 benjamin.peterson patch Python 3.1 and 3.2 (dev) opensearch.xml still references 3.0 0 days http://bugs.python.org/issue7025 benjamin.peterson add assertIsInstance 0 days http://bugs.python.org/issue7031 georg.brandl patch, patch, easy, needs review While Loop Bug on a game of 21, help! 0 days http://bugs.python.org/issue7034 PixelHead777 Top Issues Most Discussed (10) ______________________________ 27 str.title() misbehaves with apostrophes 2 days closed http://bugs.python.org/issue7008 27 PEP 314 inconsistency (authors/author/maintainer) 4 days open http://bugs.python.org/issue6992 23 zipfile.ZipFile overwrites files outside destination path 10 days open http://bugs.python.org/issue6972 14 no longer possible to hash arrays 135 days open http://bugs.python.org/issue6071 11 ssl.SSLSocket.recv() implementation may not work with non-block 380 days open http://bugs.python.org/issue3890 10 regression in pywin32 build due to 2.6.3rc1 2 days closed http://bugs.python.org/issue7020 9 unmarshaling of artificial strings can produce funny longs. 3 days open http://bugs.python.org/issue7019 9 httplib and array do not play together well 33 days closed http://bugs.python.org/issue6790 7 Add int.hex for symmetry with float.hex 0 days open http://bugs.python.org/issue7028 6 threading.local subclasses don't cleanup their state and it get 0 days closed http://bugs.python.org/issue6990 From steven.bethard at gmail.com Fri Oct 2 19:23:47 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 2 Oct 2009 10:23:47 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: On Fri, Oct 2, 2009 at 2:34 AM, Antoine Pitrou wrote: > Steven Bethard gmail.com> writes: >> >> But it's not much of a transition plan. Or are you suggesting: > > The question is why we want a transition plan that will bother everyone with no > tangible benefits for the user. I think Guido expressed my feelings pretty well: On Wed, Sep 30, 2009 at 10:37 AM, Guido van Rossum wrote: > In the long run, say Python 4, I think we don't need both, and we > should get rid of one. My preference is still getting rid of > %-formatting, due to the problems with it that prompted the design of > {}-formatting (no need to reiterate the list here). I agree with this 100% but I can't see it working unless we have some sort of transition plan. Just saying "ok, switch your format strings from % to {}" didn't work in Python 3.0 for various good reasons, and I can't imagine it will work in Python 4.0 unless we have a transition plan. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From kmtracey at gmail.com Fri Oct 2 20:46:29 2009 From: kmtracey at gmail.com (Karen Tracey) Date: Fri, 02 Oct 2009 14:46:29 -0400 Subject: [Python-Dev] Announcing PEP 3136 In-Reply-To: <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.co m> References: <20090929202035.GA5492@gmail.com> <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> Message-ID: <4ac64a8d.1a135e0a.7580.11d0@mx.google.com> At 05:15 PM 9/30/2009, Yuvgoog Greenle wrote: >I like how python has a minimalistic and powerful syntax (-1 for the >break ___ PEP). > >Also, I really dislike the for/else ambiguity "butterflies". > >When is the else after a loop executed? >1. When the loop isn't entered at all. >2. When the loop terminates through exhaustion of the list (does >this include when the list was empty?) >3. When the loop didn't exit because of a break statement. > >HINTS: The way django does it is opposite the way python does it and >there may be more than one correct answer. Django's template language does not have for/else, it has for/empty: http://docs.djangoproject.com/en/dev/ref/templates/builtins/#for-empty This construct did come from an external snippet that used 'else' instead of 'empty'. However when it was integrated into Django the 'else' name was specifically rejected because it did the opposite of what for/else does in Python. Karen From python at rcn.com Fri Oct 2 20:56:54 2009 From: python at rcn.com (Raymond Hettinger) Date: Fri, 2 Oct 2009 11:56:54 -0700 Subject: [Python-Dev] transitioning from % to {} formatting References: Message-ID: [Steven Bethard] >. Just saying "ok, switch your format strings > from % to {}" didn't work in Python 3.0 for various good reasons, and > I can't imagine it will work in Python 4.0 unless we have a transition > plan. Do the users get any say in this? I imagine that some people are heavily invested in %-formatting. Because there has been limited uptake on {}-formatting (afaict), we still have limited experience with knowing that it is actually better, less error-prone, easier to learn/rember, etc. Outside a handful of people on this list, I have yet to see anyone adopt it as the preferred syntax. Raymond From solipsis at pitrou.net Fri Oct 2 21:08:21 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Oct 2009 19:08:21 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: Message-ID: Raymond Hettinger rcn.com> writes: > > Because there has been limited uptake on {}-formatting (afaict), > we still have limited experience with knowing that it is actually > better, less error-prone, easier to learn/rember, etc. It is known to be quite slower. The following timings are on the py3k branch: - with positional arguments: $ ./python -m timeit -s "s='%s %s'; t = ('hello', 'world')" "s % t" 1000000 loops, best of 3: 0.313 usec per loop $ ./python -m timeit -s "f='{} {}'.format; t = ('hello', 'world')" "f(*t)" 1000000 loops, best of 3: 0.572 usec per loop - with named arguments: $ ./python -m timeit -s "s='%(a)s %(b)s'; d = dict(a='hello', b='world')" "s % d" 1000000 loops, best of 3: 0.387 usec per loop $ ./python -m timeit -s "f='{a} {b}'.format; d = dict(a='hello', b='world')" "f(**d)" 1000000 loops, best of 3: 0.581 usec per loop Regards Antoine. From mg at lazybytes.net Fri Oct 2 21:43:48 2009 From: mg at lazybytes.net (Martin Geisler) Date: Fri, 02 Oct 2009 21:43:48 +0200 Subject: [Python-Dev] transitioning from % to {} formatting References: Message-ID: <87y6nt37kb.fsf@hbox.dyndns.org> "Raymond Hettinger" writes: > [Steven Bethard] >>. Just saying "ok, switch your format strings >> from % to {}" didn't work in Python 3.0 for various good reasons, and >> I can't imagine it will work in Python 4.0 unless we have a transition >> plan. > > Do the users get any say in this? I'm a user! :-) I hate calling methods on string literals, I think it looks very odd to have code like this: "Displaying {0} of {1} revisions".format(x, y) Will we be able to write this as "Displaying {0} of {1} revisions" % (x, y) too? > I imagine that some people are heavily invested in %-formatting. > > Because there has been limited uptake on {}-formatting (afaict), we > still have limited experience with knowing that it is actually better, > less error-prone, easier to learn/rember, etc. Outside a handful of > people on this list, I have yet to see anyone adopt it as the > preferred syntax. I've skimmed over the PEP, and the new {}-syntax seems to have some nice features. But I've not seen it used anywhere yet. -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From barry at python.org Fri Oct 2 22:21:43 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 2 Oct 2009 16:21:43 -0400 Subject: [Python-Dev] RELEASED Python 2.6.3 Message-ID: <5B2596C4-1128-4B5B-8471-FE0409369D2D@python.org> On behalf of the Python community, I'm happy to announce the availability of Python 2.6.3. This is the latest production-ready version in the Python 2.6 series. Somewhere near 100 bugs have been fixed since Python 2.6.2 was released in April 2009. Please see the NEWS file for all the gory details. http://www.python.org/download/releases/2.6.3/NEWS.txt Source tarballs, Windows installers and the Mac OS X disk image can be downloaded from the Python 2.6.3 page. http://www.python.org/download/releases/2.6.3/ For more information on Python 2.6 in general, please see http://docs.python.org/whatsnew/2.6.html Please report bugs for any Python version in the Python tracker. http://bugs.python.org Enjoy, -Barry Barry Warsaw barry at python.org Python 2.6 Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From steven.bethard at gmail.com Fri Oct 2 22:50:36 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 2 Oct 2009 13:50:36 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <87y6nt37kb.fsf@hbox.dyndns.org> References: <87y6nt37kb.fsf@hbox.dyndns.org> Message-ID: On Fri, Oct 2, 2009 at 12:43 PM, Martin Geisler wrote: > I hate calling methods on string literals, I think it looks very odd to > have code like this: > > ?"Displaying {0} of {1} revisions".format(x, y) > > Will we be able to write this as > > ?"Displaying {0} of {1} revisions" % (x, y) > > too? I doubt it. One of the major complaints about the %-style formatting was that the use of % produced (somewhat) unexpected errors because of how operator precedence works:: >>> '{0}'.format(4 + 1) '5' >>> '%s' % 4 + 1 Traceback (most recent call last): File "", line 1, in TypeError: cannot concatenate 'str' and 'int' objects Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From jan.hosang at gmail.com Fri Oct 2 23:32:38 2009 From: jan.hosang at gmail.com (Jan Hosang) Date: Fri, 2 Oct 2009 23:32:38 +0200 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? In-Reply-To: References: <92839421634652657618189239787554831247-Webmail@me.com> Message-ID: <2ECCFBA0-1A70-4D12-925C-CE620D430AAD@gmail.com> Confirmed on 10.6 for 2.6.3 and 2.7a0. For 3.2a0 both asserts fail. On 02.10.2009, at 10:29, INADA Naoki wrote: >> Do you have a testcase that shows what the problem is? >> >> Ronald > >>>> s = '\xa0' >>>> assert s.strip() == s >>>> import locale >>>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8') > 'en_US.UTF-8' >>>> assert s.strip() == s > > Second assert failed on Snow Leopard. > > -- > Naoki INADA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jan.hosang%40gmail.com From steven.bethard at gmail.com Fri Oct 2 23:59:25 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 2 Oct 2009 14:59:25 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: On Fri, Oct 2, 2009 at 11:56 AM, Raymond Hettinger wrote: > [Steven Bethard] >> Just saying "ok, switch your format strings >> from % to {}" didn't work in Python 3.0 for various good reasons, and >> I can't imagine it will work in Python 4.0 unless we have a transition >> plan. > > Do the users get any say in this? > I imagine that some people are heavily invested in %-formatting. > > Because there has been limited uptake on {}-formatting (afaict), > we still have limited experience with knowing that it is actually > better, less error-prone, easier to learn/rember, etc. ? Outside > a handful of people on this list, I have yet to see anyone adopt > it as the preferred syntax. Sure, I guess this is a possibility too, and it could make the transition process I have to work through for argparse much easier. ;-) To be clear, are you suggesting that APIs that currently support only %-formatting shouldn't bother supporting {}-formatting at all? Or are you suggesting that they should support both, but support for %-formatting should never go away? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From songofacandy at gmail.com Sat Oct 3 01:40:52 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 3 Oct 2009 08:40:52 +0900 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? In-Reply-To: <2ECCFBA0-1A70-4D12-925C-CE620D430AAD@gmail.com> References: <92839421634652657618189239787554831247-Webmail@me.com> <2ECCFBA0-1A70-4D12-925C-CE620D430AAD@gmail.com> Message-ID: > Confirmed on 10.6 for 2.6.3 and 2.7a0. For 3.2a0 both asserts fail. OK. `s = '\xa0'` should be `s = b'\xa0'`. Should I file a bug? -- Naoki INADA From andrew.svetlov at gmail.com Sat Oct 3 03:06:24 2009 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 2 Oct 2009 21:06:24 -0400 Subject: [Python-Dev] PEP 3121 Message-ID: <270ea8310910021806v667776f6x419470ec25d5ba56@mail.gmail.com> What real status of this document? As I figured out py3k trunk uses this ones. Most part of 'battery included' modules don't use this feature, leaving m_traverse, m_clear and m_free NULL pointers. There are only exception: _io. But looks like all underground python machinery is already ported to support PEP. Maybe it's good point to update PEP? "accepted; may not be implemented yet" sounds a bit confusing. Just insert somethere close to top of content: "Implemented by Python itself, but builtin modules should be converted". Next question: are where all 'python standard library' modules looks like "not converted yet" really need to be refactored? Or we can split this huge set to 'looks already good' and 'requires some work'? And just add comment for every PyModuleDef definition like: "Completely satisfy PEP 3121, signed and stamped by Martin von Loewis and GvR".? Last question: PEP refers to "multiple interpreters" feature of python. This feature presented at least in 2.xx branch (don't remember about 1.xx - up to 2003 I used python as scripting language only and not care about C interface). I never saw usage of this one in real use cases. Is there projects (preferable with code sources, of course) what runs multiple interpreter in same process? From tjreedy at udel.edu Sat Oct 3 03:39:11 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 02 Oct 2009 21:39:11 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: Steven Bethard wrote: > On Thu, Oct 1, 2009 at 10:49 PM, Terry Reedy wrote: >> As someone who likes .format() and who already uses such bound methods to >> print, such as in >> >> emsg = "...".format >> ... >> if c: print(emsg(arg, barg)) >> >> I find this **MUCH** preferable to the ugly and seemingly unnecessary >> wrapper class idea being bandied about. This would be scarcely worse than >> passing the string itself. > > But it's not much of a transition plan. It is a 'plan' to transition from not being able to use the new formatting, which I prefer, throughout the stdlib, to being able to do so. I believe most, even if not all, find that acceptable. Certainly, I think you should be able to implement the above for argparse before submitting it. And I would hope that 3.2, in a year, is generally .format usable. This is the first step in a possible long-term replacement, but there is currently no consensus to do any more than this. So I think it premature to do any more. I would agree, for instance, that an auto-translation tool is needed. Terry Jan Reedy From python at rcn.com Sat Oct 3 05:03:58 2009 From: python at rcn.com (Raymond Hettinger) Date: Fri, 2 Oct 2009 20:03:58 -0700 Subject: [Python-Dev] transitioning from % to {} formatting References: Message-ID: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> [Terry Reedy] > I would agree, for instance, that an auto-translation > tool is needed. We should get one written. ISTM, every %-formatting string is directly translatable to an equivalent {}-formatting string. Raymond From benjamin at python.org Sat Oct 3 05:11:35 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 2 Oct 2009 22:11:35 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <1afaf6160910022011w46ff842ax4ee0b56a9b86cc45@mail.gmail.com> 2009/10/2 Raymond Hettinger : > > [Terry Reedy] >> >> I would agree, for instance, that an auto-translation tool is needed. > > We should get one written. ?ISTM, every %-formatting > string is directly translatable to an equivalent {}-formatting string. Yes, but not all are possible to guess with mere static analysis. -- Regards, Benjamin From ncoghlan at gmail.com Sat Oct 3 07:15:00 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Oct 2009 15:15:00 +1000 Subject: [Python-Dev] PEP 3121 In-Reply-To: <270ea8310910021806v667776f6x419470ec25d5ba56@mail.gmail.com> References: <270ea8310910021806v667776f6x419470ec25d5ba56@mail.gmail.com> Message-ID: <4AC6DDD4.1040608@gmail.com> Andrew Svetlov wrote: > Maybe it's good point to update PEP? > "accepted; may not be implemented yet" sounds a bit confusing. That's the status though - the PEP is accepted, implementation is ongoing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Oct 3 07:15:53 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Oct 2009 15:15:53 +1000 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <4AC29F71.3000307@gmail.com> References: <200909290127.03215.steve@pearwood.info> <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> Message-ID: <4AC6DE09.3030003@gmail.com> Toshio Kuratomi wrote: > About exit(), I agree with others about wanting to catch the exception > myself and then choosing to exit from the code. I'm not sure that it's > actually useful in practice, though...it might just feel cleaner but not > actually be that helpful. As others have pointed out, if you want to do that then you can just catch SystemExit. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Oct 3 07:19:55 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Oct 2009 15:19:55 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <87y6nt37kb.fsf@hbox.dyndns.org> Message-ID: <4AC6DEFB.4040402@gmail.com> Steven Bethard wrote: > On Fri, Oct 2, 2009 at 12:43 PM, Martin Geisler wrote: >> I hate calling methods on string literals, I think it looks very odd to >> have code like this: >> >> "Displaying {0} of {1} revisions".format(x, y) >> >> Will we be able to write this as >> >> "Displaying {0} of {1} revisions" % (x, y) >> >> too? > > I doubt it. One of the major complaints about the %-style formatting > was that the use of % produced (somewhat) unexpected errors because of > how operator precedence works:: > >>>> '{0}'.format(4 + 1) > '5' >>>> '%s' % 4 + 1 > Traceback (most recent call last): > File "", line 1, in > TypeError: cannot concatenate 'str' and 'int' objects > > Steve The other major problem with the use of the mod operator is the bugs encountered with "fmt % obj" when obj happened to be a tuple or a dict. So no, the switch to a method rather than an operator was deliberate. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tjreedy at udel.edu Sat Oct 3 07:23:33 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 03 Oct 2009 01:23:33 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <1afaf6160910022011w46ff842ax4ee0b56a9b86cc45@mail.gmail.com> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <1afaf6160910022011w46ff842ax4ee0b56a9b86cc45@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > 2009/10/2 Raymond Hettinger : >> [Terry Reedy] >>> I would agree, for instance, that an auto-translation tool is needed. >> We should get one written. ISTM, every %-formatting >> string is directly translatable to an equivalent {}-formatting string. > > Yes, but not all are possible to guess with mere static analysis. Even if perfect is not possible, decently good would probably cover a lot of use cases. From ncoghlan at gmail.com Sat Oct 3 07:24:41 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Oct 2009 15:24:41 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: <4AC6E019.4000701@gmail.com> Raymond Hettinger wrote: > Because there has been limited uptake on {}-formatting (afaict), > we still have limited experience with knowing that it is actually > better, less error-prone, easier to learn/rember, etc. Outside > a handful of people on this list, I have yet to see anyone adopt > it as the preferred syntax. A self-fulfilling prophecy if ever I heard one... uptake is limited because there's a large legacy code base that doesn't use it and many APIs don't support it, so we shouldn't bother trying to increase the number of APIs that *do* support it? I'm starting to think that a converter between the two format mini-languages may be the way to go though. fmt_braces is meant to provide a superset of the capabilites of fmt_percent, so a forward converter shouldn't be too hard. A reverse converter may have to punt with ValueError when it finds things that cannot be expressed in the fmt_percent mini language though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Sat Oct 3 07:37:55 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 03 Oct 2009 14:37:55 +0900 Subject: [Python-Dev] PEP 3144 review. In-Reply-To: <4AC33BA1.8050203@gmail.com> References: <8517e9350909140944x104bcc63g62f86b3af7c6ca9@mail.gmail.com> <4AC03410.4010104@v.loewis.de> <4AC11855.7080406@v.loewis.de> <4AC12AC6.80304@v.loewis.de> <4AC2A9D7.4080603@gmail.com> <5c6f2a5d0909300131g5fcb255eoe9403f54777fc677@mail.gmail.com> <79990c6b0909300252w1dec6e2dlc8efdd104e5116ed@mail.gmail.com> <5c6f2a5d0909300318p4b5404d4wb41e2add4ea8f88a@mail.gmail.com> <4AC33BA1.8050203@gmail.com> Message-ID: <878wfthwb0.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > However, while I'd still be a little happier if the .ip attribute > went away all together and another means was found to conveniently > associate an IPAddress and an IPNetwork, keeping it doesn't bother > me anywhere near as much as having network equivalence defined in > terms of something other than the network ID and the netmask. s/equivalence/equality/ There are several notions of equivalence in play, and all can be implemented as methods. The question is which one is going to be "==". From larry.bugbee at boeing.com Sat Oct 3 08:08:10 2009 From: larry.bugbee at boeing.com (Bugbee, Larry) Date: Fri, 2 Oct 2009 23:08:10 -0700 Subject: [Python-Dev] transitioning from % to {} formatting Message-ID: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> > > Do the users get any say in this? > > I'm a user! :-) > > I hate calling methods on string literals, I think it looks very odd > to have code like this: > > "Displaying {0} of {1} revisions".format(x, y) Ugh! Good point. Is Python to be an easy-to-learn-and-remember language? I submit we are losing that one. To a user, this will be confusing. To a C programmer coming over to Python, especially so. Some of what makes %-formatting easy to remember is its parallel in C. I'm conflicted. Philosophically I like the idea of mnemonic names over positional variables and allowing variable values determined elsewhere to be inserted in print strings. It is appealing. Unless the benefit is at least 2x, a change should not be made, and I don't think this benefit rises to where it is worth the confusion and problems. ...and converting the legacy base. And forget pretty, not that %-formatting is pretty either. Besides, according to the bench, it is slower too. And it will take editors a while before the new syntax is supported and colorized, thus some errors for a while. ...and if one wants a "{" or a "}" in the printed output, one has to escape it? That is -2x over wanting a "%" in the output. So until I see a *significant* benefit, my vote is *not* remove %-formatting. Make both available and if {} is to win, it will. From foom at fuhm.net Sat Oct 3 08:36:38 2009 From: foom at fuhm.net (James Y Knight) Date: Sat, 3 Oct 2009 02:36:38 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: On Oct 2, 2009, at 2:56 PM, Raymond Hettinger wrote: > Do the users get any say in this? > I imagine that some people are heavily invested in %-formatting. > > Because there has been limited uptake on {}-formatting (afaict), > we still have limited experience with knowing that it is actually > better, less error-prone, easier to learn/rember, etc. Outside > a handful of people on this list, I have yet to see anyone adopt > it as the preferred syntax. Well, I actually think it was a pretty bad idea to introduce {} formatting, because %-formatting is well-known in many other languages, and $-formatting is used by basically all the rest. So the introduction of {}-formatting has always seemed silly to me, and I wish it had not happened. HOWEVER, much worse than having a new, different, and strange formatting convention is having *multiple* formatting conventions arbitrarily used in different places within the language, with no rhyme or reason. So, given that brace-formatting was added, and that it's been declared the way forward, I'd *greatly* prefer it taking over everywhere in python, instead of having to use a mixture. James From g.brandl at gmx.net Sat Oct 3 11:22:04 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 03 Oct 2009 11:22:04 +0200 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> Message-ID: Bugbee, Larry schrieb: >>> Do the users get any say in this? >> >> I'm a user! :-) >> >> I hate calling methods on string literals, I think it looks very odd to >> have code like this: >> >> "Displaying {0} of {1} revisions".format(x, y) > > Ugh! Good point. > > Is Python to be an easy-to-learn-and-remember language? I submit we are > losing that one. To a user, this will be confusing. To a C programmer > coming over to Python, especially so. Some of what makes %-formatting easy > to remember is its parallel in C. To a .NET programmer coming to Python (think IronPython), %-formatting will be strange, while {}-style formatting will be familiar. I don't know how many programmers come to Python from C nowadays. I also think that we are merely so accustomed to ``"..." % bar`` that we never think about new users who go "Huh? String modulo string?" -- why should that initial discomfort be less than that of calling a method on a literal? Being able to call a method on literals is simply consistent; literals are objects as well as an object referred to by a name. There are languages where you can't even call ``foo[bar].baz()`` because method calls only work on single names. I call that ugly. > I'm conflicted. Philosophically I like the idea of mnemonic names over > positional variables and allowing variable values determined elsewhere to be > inserted in print strings. It is appealing. > > Unless the benefit is at least 2x, a change should not be made, I don't see how numbers like "2x" can be applied when measuring the benefit of a language feature. > and I don't > think this benefit rises to where it is worth the confusion and problems. > ...and converting the legacy base. And forget pretty, not that %-formatting > is pretty either. Besides, according to the bench, it is slower too. And it > will take editors a while before the new syntax is supported and colorized, > thus some errors for a while. This is no different from all other language features. Some editors I encountered do not yet color the with statement. Should I therefore refrain from using it? Highlighting formatting placeholders in string literals is questionable anyway, since a highlighter has no way of knowing whether a given string literal will be used for formatting. > ....and if one wants a "{" or a "}" in the printed output, one has to escape > it? That is -2x over wanting a "%" in the output. Again a number. Where does it come from? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ncoghlan at gmail.com Sat Oct 3 11:40:19 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Oct 2009 19:40:19 +1000 Subject: [Python-Dev] PEP 3144 review. In-Reply-To: <878wfthwb0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8517e9350909140944x104bcc63g62f86b3af7c6ca9@mail.gmail.com> <4AC03410.4010104@v.loewis.de> <4AC11855.7080406@v.loewis.de> <4AC12AC6.80304@v.loewis.de> <4AC2A9D7.4080603@gmail.com> <5c6f2a5d0909300131g5fcb255eoe9403f54777fc677@mail.gmail.com> <79990c6b0909300252w1dec6e2dlc8efdd104e5116ed@mail.gmail.com> <5c6f2a5d0909300318p4b5404d4wb41e2add4ea8f88a@mail.gmail.com> <4AC33BA1.8050203@gmail.com> <878wfthwb0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4AC71C03.5020904@gmail.com> Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > However, while I'd still be a little happier if the .ip attribute > > went away all together and another means was found to conveniently > > associate an IPAddress and an IPNetwork, keeping it doesn't bother > > me anywhere near as much as having network equivalence defined in > > terms of something other than the network ID and the netmask. > > s/equivalence/equality/ > > There are several notions of equivalence in play, and all can be > implemented as methods. The question is which one is going to be "==". I have just enough pure math in my educational background to think of 'equality' as the 'default equivalence class', but yes, the meaning of "==" is exactly what I was referring to. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ubershmekel at gmail.com Sat Oct 3 12:45:09 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Sat, 3 Oct 2009 13:45:09 +0300 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <4AC6DE09.3030003@gmail.com> References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> Message-ID: <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> I haven't checked if it's possible, but I suggest Argparse have it's own exception class that inherits from SystemExit and that exception would be thrown. ParseError, or something similar. I suggest this just because it would be more readable I guess and would exactly explain why this code exits. On Sat, Oct 3, 2009 at 8:15 AM, Nick Coghlan wrote: > Toshio Kuratomi wrote: >> About exit(), I agree with others about wanting to catch the exception >> myself and then choosing to exit from the code. ?I'm not sure that it's >> actually useful in practice, though...it might just feel cleaner but not >> actually be that helpful. > > As others have pointed out, if you want to do that then you can just > catch SystemExit. > > Cheers, > Nick. > > -- > Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ubershmekel%40gmail.com > From steven.bethard at gmail.com Sat Oct 3 17:12:49 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 3 Oct 2009 08:12:49 -0700 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> Message-ID: On Sat, Oct 3, 2009 at 3:45 AM, Yuvgoog Greenle wrote: > I haven't checked if it's possible, but I suggest Argparse have it's > own exception class that inherits from SystemExit and that exception > would be thrown. > > ParseError, or something similar. > > I suggest this just because it would be more readable I guess and > would exactly explain why this code exits. I've never seen such an idiom before (subclassing SystemExit) but it would certainly be possible create an ArgumentParserExit exception like that. Then you would have your choice of overriding .exit() or catching the exception. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From fuzzyman at voidspace.org.uk Sat Oct 3 17:17:36 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Oct 2009 16:17:36 +0100 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> Message-ID: <4AC76B10.1000209@voidspace.org.uk> Steven Bethard wrote: > On Sat, Oct 3, 2009 at 3:45 AM, Yuvgoog Greenle wrote: > >> I haven't checked if it's possible, but I suggest Argparse have it's >> own exception class that inherits from SystemExit and that exception >> would be thrown. >> >> ParseError, or something similar. >> >> I suggest this just because it would be more readable I guess and >> would exactly explain why this code exits. >> > > I've never seen such an idiom before (subclassing SystemExit) but it > would certainly be possible create an ArgumentParserExit exception > like that. Then you would have your choice of overriding .exit() or > catching the exception. > > Steve > Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From steven.bethard at gmail.com Sat Oct 3 17:41:36 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 3 Oct 2009 08:41:36 -0700 Subject: [Python-Dev] summary of transitioning from % to {} formatting Message-ID: I thought it might be useful for those who don't have time to read a million posts to have a summary of what's happened in the formatting discussion. The basic problem is that many APIs in the standard library and elsewhere support only %-formatting and not {}-formatting, e.g. logging.Formatter accepts:: logging.Formatter(fmt="%(asctime)s - %(name)s") but not:: logging.Formatter(fmt="{asctime} - {name}") There seems to be mostly agreement that these APIs should at least support both formatting styles, and a sizable group (including Guido) believe that %-formatting should eventually be phased out (e.g. by Python 4). There are a number of competing proposals on how to allow such APIs to start accepting {}-format strings: * Add a parameter which declares the type of format string:: logging.Formatter(fmt="{asctime} - {name}", format=BRACES) The API code would then switch between %-format and {}-format based on the value of that parameter. If %-formatting is to be deprecated, this could be done by first deprecating format=PERCENTS and requiring format=BRACES, and then changing the default to format=BRACES. * Create string subclasses which convert % use to .format calls:: __ = brace_fmt logging.Formatter(fmt=__("{asctime} - {name}")) The API code wouldn't have to change at all at first, as applying % to brace_fmt objects would call .format() instead. If %-formatting is to be deprecated, this could be done by first deprecating plain strings and requiring brace_fmt strings, and then allowing plain strings again but assuming they are {}-format strings. * Teach the API to accept callables as well as strings:: logging.Formatter(fmt="{asctime} - {name}".format) The API code would just call the object with .format() style arguments if a callable was given instead of a string. If %-formatting is to be deprecated, this could be done by first deprecating plain strings and requiring callables, and then allowing plain strings again but assuming they are {}-format strings * Create translators between %-format and {}-format:: assert to_braces("%(asctime)s") == "{asctime}" assert to_percents("{asctime}") == "%(asctime)s" these could then either be used outside of the API:: logging.Formatter(fmt=to_percents("{asctime} - {name}")) or they could be used within the API combined with some sort of heuristic for guessing whether a {}-format string or a %-format string was passed in:: logging.Formatter(fmt="{asctime} - {name}") If %-formatting is to be deprecated, the transition strategy here is trivial. However, no one has yet written translators, and it is not clear what heuristics should be used, e.g. should the method just try %-formatting first and then {}-formatting if it fails? I don't think there is consensus yet on which of these proposals should be the "blessed" one. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From steven.bethard at gmail.com Sat Oct 3 17:49:18 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 3 Oct 2009 08:49:18 -0700 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <4AC76B10.1000209@voidspace.org.uk> References: <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> <4AC76B10.1000209@voidspace.org.uk> Message-ID: On Sat, Oct 3, 2009 at 8:17 AM, Michael Foord wrote: > Steven Bethard wrote: >> On Sat, Oct 3, 2009 at 3:45 AM, Yuvgoog Greenle >> wrote: >>> I haven't checked if it's possible, but I suggest Argparse have it's >>> own exception class that inherits from SystemExit and that exception >>> would be thrown. >> >> I've never seen such an idiom before (subclassing SystemExit) but it >> would certainly be possible create an ArgumentParserExit exception >> like that. Then you would have your choice of overriding .exit() or >> catching the exception. > > Why not just catch SystemExit? If you want a custom exception the overriding > .exit() should be sufficient. I'm certainly fine with that -- I'm just trying to make sure I've addressed whatever needs addressed to get argparse in. > I'd be much more interested in Guido's suggestion of auto-generated custom > help messages for sub-commands. Maybe I misunderstood, but I think this is already the default argparse behavior, no? import argparse parser = argparse.ArgumentParser() parser.add_argument('--foo') subparsers = parser.add_subparsers() parser1 = subparsers.add_parser('1') parser1.add_argument('--bar') parser2 = subparsers.add_parser('2') parser2.add_argument('baz') parser.parse_args(['--help']) >>> import argparse >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo') >>> subparsers = parser.add_subparsers() >>> parser1 = subparsers.add_parser('1') >>> parser1.add_argument('--bar') >>> parser2 = subparsers.add_parser('2') >>> parser2.add_argument('baz') >>> # top level argument help >>> parser.parse_args(['--help']) usage: [-h] [--foo FOO] {1,2} ... positional arguments: {1,2} optional arguments: -h, --help show this help message and exit --foo FOO >>> # help for subparser 1 >>> parser.parse_args(['1', '--help']) usage: 1 [-h] [--bar BAR] optional arguments: -h, --help show this help message and exit --bar BAR >>> # help for subparser 2 >>> parser.parse_args(['2', '--help']) usage: 2 [-h] baz positional arguments: baz optional arguments: -h, --help show this help message and exit Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From solipsis at pitrou.net Sat Oct 3 18:00:48 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Oct 2009 16:00:48 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?summary_of_transitioning_from_=25_to_=7B?= =?utf-8?q?=7D_formatting?= References: Message-ID: Steven Bethard gmail.com> writes: > > If %-formatting is to be deprecated, the transition strategy here > is trivial. However, no one has yet written translators, and it is > not clear what heuristics should be used, e.g. should the method > just try %-formatting first and then {}-formatting if it fails? This would be a reasonable heuristic. It should be done only on the first call, though, and then the result remembered (for both performance reasons and consistency). The cases where a format string would function in both formatting styles and expect the same parameters must be very rare in the real world. Regards Antoine. From p.f.moore at gmail.com Sat Oct 3 18:08:13 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Oct 2009 17:08:13 +0100 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: <79990c6b0910030908ia71714cpd6f7b06afa85191@mail.gmail.com> 2009/10/3 Antoine Pitrou : > Steven Bethard gmail.com> writes: >> >> ? If %-formatting is to be deprecated, the transition strategy here >> ? is trivial. However, no one has yet written translators, and it is >> ? not clear what heuristics should be used, e.g. should the method >> ? just try %-formatting first and then {}-formatting if it fails? > > This would be a reasonable heuristic. It should be done only on the first call, > though, and then the result remembered (for both performance reasons and > consistency). > > The cases where a format string would function in both formatting styles and > expect the same parameters must be very rare in the real world. Define "fails": >>> "{a} {b} c" % {'a':12} '{a} {b} c' That didn't fail... Paul. From rdmurray at bitdance.com Sat Oct 3 18:19:32 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 3 Oct 2009 12:19:32 -0400 (EDT) Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <79990c6b0910030908ia71714cpd6f7b06afa85191@mail.gmail.com> References: <79990c6b0910030908ia71714cpd6f7b06afa85191@mail.gmail.com> Message-ID: On Sat, 3 Oct 2009 at 17:08, Paul Moore wrote: > 2009/10/3 Antoine Pitrou : >> Steven Bethard gmail.com> writes: >>> >>> ? If %-formatting is to be deprecated, the transition strategy here >>> ? is trivial. However, no one has yet written translators, and it is >>> ? not clear what heuristics should be used, e.g. should the method >>> ? just try %-formatting first and then {}-formatting if it fails? >> >> This would be a reasonable heuristic. It should be done only on the first call, >> though, and then the result remembered (for both performance reasons and >> consistency). >> >> The cases where a format string would function in both formatting styles and >> expect the same parameters must be very rare in the real world. > > Define "fails": > >>>> "{a} {b} c" % {'a':12} > '{a} {b} c' > > That didn't fail... Also, what if both fail? Which failure's error message gets printed? --David (RDM) From solipsis at pitrou.net Sat Oct 3 18:23:24 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 03 Oct 2009 18:23:24 +0200 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <79990c6b0910030908ia71714cpd6f7b06afa85191@mail.gmail.com> References: <79990c6b0910030908ia71714cpd6f7b06afa85191@mail.gmail.com> Message-ID: <1254587004.5470.8.camel@localhost> > Define "fails": > > >>> "{a} {b} c" % {'a':12} > '{a} {b} c' > > That didn't fail... Ah, my bad. I had completely overlooked that formatting was laxist when faced with unused named parameters. Then we need something smarter, like counting the number of unescaped "%" characters, the number of pairs of braces, and try first the formatting style expecting the greatest number of parameters. It can either be done cheaply in Python, or more rigourously by providing an extra API to the existing C parsing routines. This might look fragile, but we should not forget that we are talking about uses - logging, etc. - which will most of time involve very simple format strings, such that making an automatic decision is not too difficult. In the more undecideable cases, the heuristic might decide to raise an exception and require the developer to specify the formatting style explicitly (by adding e.g. style='%' or style='{}' to the method call). Regards Antoine. From fuzzyman at voidspace.org.uk Sat Oct 3 18:21:43 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Oct 2009 17:21:43 +0100 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: References: <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> <4AC76B10.1000209@voidspace.org.uk> Message-ID: <4AC77A17.7050101@voidspace.org.uk> Steven Bethard wrote: > [snip...] >> I'd be much more interested in Guido's suggestion of auto-generated custom >> help messages for sub-commands. >> > > Maybe I misunderstood, but I think this is already the default > argparse behavior, no? > > Cool. I didn't realise that help for subcommands was already implemented. :-) Michael > import argparse > parser = argparse.ArgumentParser() > parser.add_argument('--foo') > subparsers = parser.add_subparsers() > parser1 = subparsers.add_parser('1') > parser1.add_argument('--bar') > parser2 = subparsers.add_parser('2') > parser2.add_argument('baz') > parser.parse_args(['--help']) > > >>>> import argparse >>>> parser = argparse.ArgumentParser() >>>> parser.add_argument('--foo') >>>> subparsers = parser.add_subparsers() >>>> parser1 = subparsers.add_parser('1') >>>> parser1.add_argument('--bar') >>>> parser2 = subparsers.add_parser('2') >>>> parser2.add_argument('baz') >>>> > > >>>> # top level argument help >>>> parser.parse_args(['--help']) >>>> > usage: [-h] [--foo FOO] {1,2} ... > > positional arguments: > {1,2} > > optional arguments: > -h, --help show this help message and exit > --foo FOO > > >>>> # help for subparser 1 >>>> parser.parse_args(['1', '--help']) >>>> > usage: 1 [-h] [--bar BAR] > > optional arguments: > -h, --help show this help message and exit > --bar BAR > > >>>> # help for subparser 2 >>>> parser.parse_args(['2', '--help']) >>>> > usage: 2 [-h] baz > > positional arguments: > baz > > optional arguments: > -h, --help show this help message and exit > > Steve > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From barry at python.org Sat Oct 3 18:30:59 2009 From: barry at python.org (Barry Warsaw) Date: Sat, 3 Oct 2009 12:30:59 -0400 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> On Oct 3, 2009, at 11:41 AM, Steven Bethard wrote: > I thought it might be useful for those who don't have time to read a > million posts to have a summary of what's happened in the formatting > discussion. > > The basic problem is that many APIs in the standard library and > elsewhere support only %-formatting and not {}-formatting, e.g. > logging.Formatter accepts:: > logging.Formatter(fmt="%(asctime)s - %(name)s") > but not:: > logging.Formatter(fmt="{asctime} - {name}") I'd like to at least keep in mind $-strings. I don't have any experience with {}-strings w.r.t. i18n translators, but I'm very confident that $-strings are better for translators than %-strings. OT1H {}-strings don't suffer from the #1 problem of %-strings: leaving off the trailing 's' or whatever. OTOH, I suspect that $-strings are still easier for the simple substitution case; for example, it's conceivable that translators may forget the trailing close brace. Since we're likely to have multiple formatting styles at least for Python 3's lifetime, I'd like any solution we come up with to at least not preclude the use of $-strings. I also don't think this is a case of anti-TOOWTDI. For most situations {}-strings are great (IMO), but in the specific translation domain, I suspect $-strings are still better. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From fuzzyman at voidspace.org.uk Sat Oct 3 18:40:45 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Oct 2009 17:40:45 +0100 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> References: <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> <4AC76B10.1000209@voidspace.org.uk> <4AC77A17.7050101@voidspace.org.uk> <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> Message-ID: <4AC77E8D.9070205@voidspace.org.uk> Yuvgoog Greenle wrote: > On Sat, Oct 3, 2009 at 7:21 PM, Michael Foord wrote: > >> [snip...] >> Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. >> I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands. >> > > Check it out: > > def ParseAndRun(): > crazy_external_function_that_might_exit() > > # Argparse blah blah > parser.parse_args() > > if __name__ == "__main__": > try: > ParseAndRun() > except SystemExit: > # was it crazy_external_function_that_might_exit or an argparse error? > > > I know this might come through as bike shedding but it's just > customary python that every module have it's own exception types as to > not mix them up with others. > > Then subclass and override .exit() as discussed - or put proper exception handling around the call to parse_args() (optionally rethrowing with whatever custom exception type you wish). Michael > --yuv > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From dickinsm at gmail.com Sat Oct 3 18:34:41 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 3 Oct 2009 17:34:41 +0100 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: <5c6f2a5d0910030934u7399d871yfb71f3319ce6183e@mail.gmail.com> On Sat, Oct 3, 2009 at 4:41 PM, Steven Bethard wrote: > I thought it might be useful for those who don't have time to read a > million posts to have a summary of what's happened in the formatting > discussion. Definitely useful. Thanks for the summary! [...] > * Add a parameter which declares the type of format string:: > ? ?logging.Formatter(fmt="{asctime} - {name}", format=BRACES) > ?The API code would then switch between %-format and {}-format > ?based on the value of that parameter. If %-formatting is to be > ?deprecated, this could be done by first deprecating > ?format=PERCENTS and requiring format=BRACES, and then changing the > ?default to format=BRACES. +1. > * Create string subclasses which convert % use to .format calls:: > ? ?__ = brace_fmt > ? ?logging.Formatter(fmt=__("{asctime} - {name}")) > ?The API code wouldn't have to change at all at first, as applying > ?% to brace_fmt objects would call .format() instead. If > ?%-formatting is to be deprecated, this could be done by first > ?deprecating plain strings and requiring brace_fmt strings, and > ?then allowing plain strings again but assuming they are {}-format > ?strings. Uurgh. This just feels... icky. A badly-rationalized -1 from me. > * Teach the API to accept callables as well as strings:: > ? ?logging.Formatter(fmt="{asctime} - {name}".format) > ?The API code would just call the object with .format() style > ?arguments if a callable was given instead of a string. If > ?%-formatting is to be deprecated, this could be done by first > ?deprecating plain strings and requiring callables, and then > ?allowing plain strings again but assuming they are {}-format > ?strings +0.5. Seems like it could work, but the first solution feels cleaner. > * Create translators between %-format and {}-format:: > ? ?assert to_braces("%(asctime)s") == "{asctime}" > ? ?assert to_percents("{asctime}") == "%(asctime)s" > ?these could then either be used outside of the API:: > ? ?logging.Formatter(fmt=to_percents("{asctime} - {name}")) > ?or they could be used within the API combined with some sort of > ?heuristic for guessing whether a {}-format string or a %-format > ?string was passed in:: > ? ?logging.Formatter(fmt="{asctime} - {name}") > ?If %-formatting is to be deprecated, the transition strategy here > ?is trivial. However, no one has yet written translators, and it is > ?not clear what heuristics should be used, e.g. should the method > ?just try %-formatting first and then {}-formatting if it fails? I'm reserving judgement on this one until it becomes clear how feasible it is. Without having thought about it too hard, this sounds potentially tricky and bug-prone. Mark From ubershmekel at gmail.com Sat Oct 3 18:38:31 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Sat, 3 Oct 2009 19:38:31 +0300 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <4AC77A17.7050101@voidspace.org.uk> References: <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> <4AC76B10.1000209@voidspace.org.uk> <4AC77A17.7050101@voidspace.org.uk> Message-ID: <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> On Sat, Oct 3, 2009 at 7:21 PM, Michael Foord wrote: > [snip...] > Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. > I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands. Check it out: def ParseAndRun(): crazy_external_function_that_might_exit() # Argparse blah blah parser.parse_args() if __name__ == "__main__": try: ParseAndRun() except SystemExit: # was it crazy_external_function_that_might_exit or an argparse error? I know this might come through as bike shedding but it's just customary python that every module have it's own exception types as to not mix them up with others. --yuv From python at mrabarnett.plus.com Sat Oct 3 19:02:52 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 03 Oct 2009 18:02:52 +0100 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: <4AC783BC.70404@mrabarnett.plus.com> Steven Bethard wrote: > I thought it might be useful for those who don't have time to read a > million posts to have a summary of what's happened in the formatting > discussion. > > The basic problem is that many APIs in the standard library and > elsewhere support only %-formatting and not {}-formatting, e.g. > logging.Formatter accepts:: > logging.Formatter(fmt="%(asctime)s - %(name)s") > but not:: > logging.Formatter(fmt="{asctime} - {name}") > > There seems to be mostly agreement that these APIs should at least > support both formatting styles, and a sizable group (including Guido) > believe that %-formatting should eventually be phased out (e.g. by > Python 4). There are a number of competing proposals on how to allow > such APIs to start accepting {}-format strings: > > * Add a parameter which declares the type of format string:: > logging.Formatter(fmt="{asctime} - {name}", format=BRACES) > The API code would then switch between %-format and {}-format > based on the value of that parameter. If %-formatting is to be > deprecated, this could be done by first deprecating > format=PERCENTS and requiring format=BRACES, and then changing the > default to format=BRACES. > > * Create string subclasses which convert % use to .format calls:: > __ = brace_fmt > logging.Formatter(fmt=__("{asctime} - {name}")) > The API code wouldn't have to change at all at first, as applying > % to brace_fmt objects would call .format() instead. If > %-formatting is to be deprecated, this could be done by first > deprecating plain strings and requiring brace_fmt strings, and > then allowing plain strings again but assuming they are {}-format > strings. > > * Teach the API to accept callables as well as strings:: > logging.Formatter(fmt="{asctime} - {name}".format) > The API code would just call the object with .format() style > arguments if a callable was given instead of a string. If > %-formatting is to be deprecated, this could be done by first > deprecating plain strings and requiring callables, and then > allowing plain strings again but assuming they are {}-format > strings > I'm not keen on deprecating strings in favour of callables and then callables in favour of strings. > * Create translators between %-format and {}-format:: > assert to_braces("%(asctime)s") == "{asctime}" > assert to_percents("{asctime}") == "%(asctime)s" > these could then either be used outside of the API:: > logging.Formatter(fmt=to_percents("{asctime} - {name}")) > or they could be used within the API combined with some sort of > heuristic for guessing whether a {}-format string or a %-format > string was passed in:: > logging.Formatter(fmt="{asctime} - {name}") > If %-formatting is to be deprecated, the transition strategy here > is trivial. However, no one has yet written translators, and it is > not clear what heuristics should be used, e.g. should the method > just try %-formatting first and then {}-formatting if it fails? > Another possibility: A StringFormat class with subclasses PercentStringFormat, BraceStringFormat, and perhaps DollarStringFormat. Or: A StringFormat class with methods parse_percent_format, parse_brace_format, and parse_dollar_format. There could also be a format-guesser method. > I don't think there is consensus yet on which of these proposals > should be the "blessed" one. > From python at mrabarnett.plus.com Sat Oct 3 19:14:03 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 03 Oct 2009 18:14:03 +0100 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <5c6f2a5d0910030934u7399d871yfb71f3319ce6183e@mail.gmail.com> References: <5c6f2a5d0910030934u7399d871yfb71f3319ce6183e@mail.gmail.com> Message-ID: <4AC7865B.9020607@mrabarnett.plus.com> Mark Dickinson wrote: > On Sat, Oct 3, 2009 at 4:41 PM, Steven Bethard wrote: >> I thought it might be useful for those who don't have time to read a >> million posts to have a summary of what's happened in the formatting >> discussion. > > Definitely useful. Thanks for the summary! > > [...] > >> * Add a parameter which declares the type of format string:: >> logging.Formatter(fmt="{asctime} - {name}", format=BRACES) >> The API code would then switch between %-format and {}-format >> based on the value of that parameter. If %-formatting is to be >> deprecated, this could be done by first deprecating >> format=PERCENTS and requiring format=BRACES, and then changing the >> default to format=BRACES. > > +1. [snip] 'fmt' (which is an abbreviation for 'format') and 'format'? Maybe 'fmt' and 'style' instead? From p.f.moore at gmail.com Fri Oct 2 15:20:28 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2009 14:20:28 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC53417.1040405@trueblade.com> References: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> <4AC53417.1040405@trueblade.com> Message-ID: <79990c6b0910020620l5ff4e774wce36f6d48d1043ef@mail.gmail.com> 2009/10/1 Eric Smith : > It's tangential, but in the str.format case you don't want to check for just > '{asctime}', because you might want '{asctime:%Y-%m-%d}', for example. > > But there are ways to delay computing the time until you're sure it's > actually being used in the format string, without parsing the format string. > Now that I think of it, the same technique could be used with %-formatting: Still tangential, but it seems to me that this discussion has exposed a couple of areas where the logging interface is less than ideal: - The introspection of the format string to delay computing certain items (Eric's suggestion may be an improvement here). - The "call str() on any non-string object to get a format string" API (which precludes string subclasses). I suspect other APIs will exist with similar issues once the whole question of supporting multiple format syntaxes gets wider publicity... Paul. From steve at pearwood.info Sat Oct 3 19:29:52 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Oct 2009 03:29:52 +1000 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> References: <4AC77A17.7050101@voidspace.org.uk> <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> Message-ID: <200910040429.53011.steve@pearwood.info> On Sun, 4 Oct 2009 03:38:31 am Yuvgoog Greenle wrote: > Check it out: > > def ParseAndRun(): > ? ? crazy_external_function_that_might_exit() > > ? ? # Argparse blah blah > ? ? parser.parse_args() > > if __name__ == "__main__": > ? ? try: > ? ? ? ? ParseAndRun() > ? ? except SystemExit: > ? ? ? ? # was it crazy_external_function_that_might_exit or an > argparse error? Does it matter? What difference does it make? > I know this might come through as bike shedding but it's just > customary python that every module have it's own exception types as > to not mix them up with others. You are mistaken. >>> import os >>> os.path.split(45) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.6/posixpath.py", line 82, in split i = p.rfind('/') + 1 AttributeError: 'int' object has no attribute 'rfind' Plain, ordinary AttributeError, not OSModuleSubclassOfAttributeError. I could show a thousand other examples. It simply isn't true that all, or even most, modules have their own exception types. There's no reason to treat SystemExit as special in this regard. optparse, for example, calls sys.exit(), which operates by raising SystemExit. Ordinary SystemExit, not a special module-specific subclass. -- Steven D'Aprano From fuzzyman at voidspace.org.uk Sat Oct 3 19:35:29 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Oct 2009 18:35:29 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <79990c6b0910020620l5ff4e774wce36f6d48d1043ef@mail.gmail.com> References: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> <4AC53417.1040405@trueblade.com> <79990c6b0910020620l5ff4e774wce36f6d48d1043ef@mail.gmail.com> Message-ID: <4AC78B61.2080501@voidspace.org.uk> Paul Moore wrote: > 2009/10/1 Eric Smith : > >> It's tangential, but in the str.format case you don't want to check for just >> '{asctime}', because you might want '{asctime:%Y-%m-%d}', for example. >> >> But there are ways to delay computing the time until you're sure it's >> actually being used in the format string, without parsing the format string. >> Now that I think of it, the same technique could be used with %-formatting: >> > > Still tangential, but it seems to me that this discussion has exposed > a couple of areas where the logging interface is less than ideal: > > - The introspection of the format string to delay computing certain > items (Eric's suggestion may be an improvement here). > - The "call str() on any non-string object to get a format string" API > (which precludes string subclasses). > Calling str on non-string objects to get a format string does not (prima-facie) preclude string subclasses: >>> class X(str): pass ... >>> class Y(object): ... def __str__(self): ... return X('foo') ... >>> a = Y() >>> type(str(a)) Michael > I suspect other APIs will exist with similar issues once the whole > question of supporting multiple format syntaxes gets wider > publicity... > > Paul. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From vinay_sajip at yahoo.co.uk Sat Oct 3 19:38:36 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 3 Oct 2009 17:38:36 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <79990c6b0910010611u6c772f11w7029c3154fa7be49@mail.gmail.com> <69124.66598.qm@web25807.mail.ukl.yahoo.com> <4AC5250C.8030403@gmail.com> <4AC53417.1040405@trueblade.com> <79990c6b0910020620l5ff4e774wce36f6d48d1043ef@mail.gmail.com> Message-ID: Paul Moore gmail.com> writes: > Still tangential, but it seems to me that this discussion has exposed > a couple of areas where the logging interface is less than ideal: > > - The introspection of the format string to delay computing certain > items (Eric's suggestion may be an improvement here). Yes, but that's an implementation detail and not part of the logging interface. It can be changed without any particular additional impact on user code - when I say "additional" I mean apart from the need to change the format strings to {} format, which they would have to do anyway at some point. > - The "call str() on any non-string object to get a format string" API > (which precludes string subclasses). It doesn't preclude string subclasses: it just calls str() on an arbitrary message object to get the string representation for that object. The return value is used to interpolate into the formatted output, and that's all. So I don't understand what's being precluded and how - please elaborate. Thanks & regards, Vinay Sajip From solipsis at pitrou.net Sat Oct 3 19:50:20 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Oct 2009 17:50:20 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?summary_of_transitioning_from_=25_to_=7B?= =?utf-8?q?=7D_formatting?= References: <4AC783BC.70404@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > > Another possibility: > > A StringFormat class with subclasses PercentStringFormat, > BraceStringFormat, and perhaps DollarStringFormat. > > Or: > > A StringFormat class with methods parse_percent_format, > parse_brace_format, and parse_dollar_format. There could also be a > format-guesser method. I'm sorry to say this, but I think these suggestions are getting foolish. We core developers might have an interest in transitioning users from one formatting style to another, but the users mostly don't care and don't want to bother. Imposing on users the explicit use of such wrapper classes, moreover with such awfully long-winded names, is not helpful to them at all, and it will earn Python the reputation of a language which imposes silly constructs in the name of purity. If we can't find a way to make things almost transparent, we should IMO abandon the whole idea of a transition. Regards Antoine. From steve at pearwood.info Sat Oct 3 20:01:56 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Oct 2009 04:01:56 +1000 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: <200910040501.57077.steve@pearwood.info> On Sun, 4 Oct 2009 01:41:36 am Steven Bethard wrote: > I thought it might be useful for those who don't have time to read a > million posts to have a summary of what's happened in the formatting > discussion. > > The basic problem is that many APIs in the standard library and > elsewhere support only %-formatting and not {}-formatting, e.g. > logging.Formatter accepts:: > logging.Formatter(fmt="%(asctime)s - %(name)s") > but not:: > logging.Formatter(fmt="{asctime} - {name}") Why is this a problem? Is it because the APIs require extra functionality that only {} formatting can provide? Possibly, but I doubt it -- I expect that the reason is: (1) Some people would like to deprecate % formatting, and they can't while the std lib uses % internally. (2) Some APIs in the std lib are tightly coupled to their internal implementation, and so you can't change their implementation without changing the API as well. Remove either of these issues, and the problem becomes a non-problem, and no action is needed. Personally, I'd like to see no further talk about deprecating % until at least Python 3.2, that is, the *earliest* we would need to solve this issue would be 3.3. As the Zen says, "Although never is often better than *right* now." > There seems to be mostly agreement that these APIs should at least > support both formatting styles, and a sizable group (including Guido) > believe that %-formatting should eventually be phased out (e.g. by > Python 4). -1 on that. Time will tell if I change my mind in a couple of years, but I suspect not -- for simple formatting, I far prefer %. Judging by the reaction on comp.lang.python when this has been discussed in the past, I think a large (or at least loud) proportion of Python programmers agree with me. > There are a number of competing proposals on how to allow > such APIs to start accepting {}-format strings: > > * Add a parameter which declares the type of format string:: > logging.Formatter(fmt="{asctime} - {name}", format=BRACES) > The API code would then switch between %-format and {}-format > based on the value of that parameter. If %-formatting is to be > deprecated, this could be done by first deprecating > format=PERCENTS and requiring format=BRACES, and then changing the > default to format=BRACES. +0.5 > * Create string subclasses which convert % use to .format calls:: > __ = brace_fmt > logging.Formatter(fmt=__("{asctime} - {name}")) There are a few problems with this approach: - Nobody has yet demonstrated that this brace_fmt class is even possible. It should be easy to handle a restricted set of simple templates (e.g. of the form "%(name)s" only), but what of the full range of behaviour supported by % formatting? - Even if it is doable, it is a wrapper class, which means the module will suffer a performance hit on every call. At this time, we have no idea what the magnitude of that hit will be, but .format() is already slower than % so it will likely be significant. - It strikes me as hideously ugly. I for one would delay using it as long as possible, no matter how many DepreciationWarnings I got. I'd drag my feet and avoid changing and complain loudly and then become sullen and resentful when I couldn't avoid making the change. I'd much rather go straight from %-based templates to {} in a single step than have this Frankenstein monster intermediate. [...] > * Teach the API to accept callables as well as strings:: > logging.Formatter(fmt="{asctime} - {name}".format) > The API code would just call the object with .format() style > arguments if a callable was given instead of a string. +0.5 > * Create translators between %-format and {}-format:: > assert to_braces("%(asctime)s") == "{asctime}" > assert to_percents("{asctime}") == "%(asctime)s" +1, assuming such translators are even possible. Being optimistic, such translators would have one additional benefit: they would enable modules to completely decouple the API they offer from their internal implementation, without paying a runtime cost on every call, just a single once-off translation at initialisation time. In theory, this could mean that modules could, if they choose, continue to offer an API based on % long after str.__mod__ is removed from the language. -- Steven D'Aprano From steve at pearwood.info Sat Oct 3 20:22:28 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Oct 2009 04:22:28 +1000 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <9d153b7c0910031046y6f25fc2fg9b80d96ca0b3e61b@mail.gmail.com> References: <200910040429.53011.steve@pearwood.info> <9d153b7c0910031046y6f25fc2fg9b80d96ca0b3e61b@mail.gmail.com> Message-ID: <200910040522.28374.steve@pearwood.info> On Sun, 4 Oct 2009 04:46:19 am Yuvgoog Greenle wrote: > I just think that if a parser error is causing the SystemExit, I > would rather catch a parser error than catching a SystemExit for the > sake of readability. It saves me the comments: > > # Catching SystemExit because parse_args() throws SystemExit on > parser errors. But why are you catching the error? As a general rule, you *want* your command line app to exit if it can't understand the arguments you pass to it. What else can it do? Guess what you wanted? Assuming you have a reason for catching the exception, I don't see that there's any difference in readability between these: parser = argparse.ArgumentParser() setup_args(parser) try: ns = parser.parse_args() except argparse.ParserError: process_error() else: main(ns) and: parser = argparse.ArgumentParser() setup_args(parser) try: ns = parser.parse_args() except SystemExit: process_error() else: main(ns) You don't need a comment warning that you are catching SystemExit because parse_args raises SystemExit, any more than you need a comment saying that you are catching ValueError because some function raises ValueError. The fact that you are catching an exception implies that the function might raise that exception. A comment like: "Catching SystemExit because parse_args() throws SystemExit on parser errors." is up them with comments like this: x += 1 # Add 1 to x. -- Steven D'Aprano From brett at python.org Sat Oct 3 20:27:20 2009 From: brett at python.org (Brett Cannon) Date: Sat, 3 Oct 2009 11:27:20 -0700 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <200910040501.57077.steve@pearwood.info> References: <200910040501.57077.steve@pearwood.info> Message-ID: On Sat, Oct 3, 2009 at 11:01, Steven D'Aprano wrote: > On Sun, 4 Oct 2009 01:41:36 am Steven Bethard wrote: >> I thought it might be useful for those who don't have time to read a >> million posts to have a summary of what's happened in the formatting >> discussion. >> >> The basic problem is that many APIs in the standard library and >> elsewhere support only %-formatting and not {}-formatting, e.g. >> logging.Formatter accepts:: >> ? logging.Formatter(fmt="%(asctime)s - %(name)s") >> but not:: >> ? logging.Formatter(fmt="{asctime} - {name}") > > Why is this a problem? Is it because the APIs require extra > functionality that only {} formatting can provide? Possibly, but I > doubt it -- I expect that the reason is: > > (1) Some people would like to deprecate % formatting, and they can't > while the std lib uses % internally. > > (2) Some APIs in the std lib are tightly coupled to their internal > implementation, and so you can't change their implementation without > changing the API as well. > > Remove either of these issues, and the problem becomes a non-problem, > and no action is needed. Personally, I'd like to see no further talk > about deprecating % until at least Python 3.2, that is, the *earliest* > we would need to solve this issue would be 3.3. As the Zen > says, "Although never is often better than *right* now." No one is saying we should deprecate % any time soon on strings themselves or anywhere. This discussion is purely in regards to argparse and logging to transition *their* APIs over to {} formatting which would most likely involve some deprecation for *using* % formatting in those APIs. But % formatting on strings themselves is not directly being discussed here. -Brett From brett at python.org Sat Oct 3 20:05:50 2009 From: brett at python.org (Brett Cannon) Date: Sat, 3 Oct 2009 11:05:50 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: On Fri, Oct 2, 2009 at 20:03, Raymond Hettinger wrote: > > [Terry Reedy] >> >> I would agree, for instance, that an auto-translation tool is needed. > > We should get one written. ?ISTM, every %-formatting > string is directly translatable to an equivalent {}-formatting string. Why don't we start something in the sandbox and see how far we can get. If no one beats me to it I will add the directory some time today and we can start hashing out the solution there. -Brett From ubershmekel at gmail.com Sat Oct 3 19:46:19 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Sat, 3 Oct 2009 20:46:19 +0300 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <200910040429.53011.steve@pearwood.info> References: <4AC77A17.7050101@voidspace.org.uk> <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> <200910040429.53011.steve@pearwood.info> Message-ID: <9d153b7c0910031046y6f25fc2fg9b80d96ca0b3e61b@mail.gmail.com> On Sat, Oct 3, 2009 at 8:29 PM, Steven D'Aprano wrote: > I could show a thousand other examples. It simply isn't true that all, > or even most, modules have their own exception types. I might be wrong on this. Your point is extra true for modules in the standard library (which is what we're talking about for argparse). I just think that if a parser error is causing the SystemExit, I would rather catch a parser error than catching a SystemExit for the sake of readability. It saves me the comments: # Catching SystemExit because parse_args() throws SystemExit on parser errors. # Subclassing ArgumentParser and overriding exit because I don't want to exit() upon parser errors. So I'm sorry if what I said was irrelevant. I've never written or taken part of writing a std-lib module. --yuv From nd at perlig.de Sat Oct 3 20:35:04 2009 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Sat, 3 Oct 2009 20:35:04 +0200 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <200910040522.28374.steve@pearwood.info> References: <9d153b7c0910031046y6f25fc2fg9b80d96ca0b3e61b@mail.gmail.com> <200910040522.28374.steve@pearwood.info> Message-ID: <200910032035.05117@news.perlig.de> * Steven D'Aprano wrote: > You don't need a comment warning that you are catching SystemExit > because parse_args raises SystemExit, any more than you need a comment > saying that you are catching ValueError because some function raises > ValueError. The fact that you are catching an exception implies that > the function might raise that exception. A comment like: > > "Catching SystemExit because parse_args() throws SystemExit on parser > errors." > > is up them with comments like this: > > x += 1 # Add 1 to x. It's semantically different. You usually don't catch SystemExit directly, because you want your programs to be stopped. Additionally, a library exiting your program is badly designed, as it's unexpected. Thatswhy such a comment is useful. Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse. That way all parties here should be satisifed. (I do the same all the time in my signal handlers - that's another reason I'd rather not catch SystemExit directly as well :-) nd -- "Umfassendes Werk (auch fuer Umsteiger vom Apache 1.3)" -- aus einer Rezension From python at mrabarnett.plus.com Sat Oct 3 21:09:38 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 03 Oct 2009 20:09:38 +0100 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: <4AC783BC.70404@mrabarnett.plus.com> Message-ID: <4AC7A172.2050808@mrabarnett.plus.com> Antoine Pitrou wrote: > MRAB mrabarnett.plus.com> writes: >> Another possibility: >> >> A StringFormat class with subclasses PercentStringFormat, >> BraceStringFormat, and perhaps DollarStringFormat. >> >> Or: >> >> A StringFormat class with methods parse_percent_format, >> parse_brace_format, and parse_dollar_format. There could also be a >> format-guesser method. > > I'm sorry to say this, but I think these suggestions are getting > foolish. We core developers might have an interest in transitioning > users from one formatting style to another, but the users mostly > don't care and don't want to bother. Imposing on users the explicit > use of such wrapper classes, moreover with such awfully long-winded > names, is not helpful to them at all, and it will earn Python the > reputation of a language which imposes silly constructs in the name > of purity. > Fair enough. The purpose of the post was really just so that we cover as many possibilities as we can so that if at some time in the future someone asks why we didn't consider such-and-such then we won't be saying "Oh, never thought of that!". :-) > If we can't find a way to make things almost transparent, we should > IMO abandon the whole idea of a transition. > From robert.kern at gmail.com Sat Oct 3 23:21:46 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 03 Oct 2009 16:21:46 -0500 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <200910040522.28374.steve@pearwood.info> References: <200910040429.53011.steve@pearwood.info> <9d153b7c0910031046y6f25fc2fg9b80d96ca0b3e61b@mail.gmail.com> <200910040522.28374.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Sun, 4 Oct 2009 04:46:19 am Yuvgoog Greenle wrote: > >> I just think that if a parser error is causing the SystemExit, I >> would rather catch a parser error than catching a SystemExit for the >> sake of readability. It saves me the comments: >> >> # Catching SystemExit because parse_args() throws SystemExit on >> parser errors. > > But why are you catching the error? As a general rule, you *want* your > command line app to exit if it can't understand the arguments you pass > to it. What else can it do? Guess what you wanted? There are uses of argparse outside of command line apps. For example, I use it to parse --options for IPython %magic commands. Of course, I just subclass ArgumentParser and replace the .error() method. I require a particular IPython exception type in order for the error to be recognized correctly in the %magic subsystem. The other use case, as mentioned earlier, was for debugging your parser on the interactive prompt. A custom subclass may also be able to hold more machine-readable information about the error than the formatted error message, but I don't have any particular use cases about what may be the most useful. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From vinay_sajip at yahoo.co.uk Sun Oct 4 00:27:27 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 3 Oct 2009 22:27:27 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: Brett Cannon python.org> writes: > Why don't we start something in the sandbox and see how far we can > get. If no one beats me to it I will add the directory some time today > and we can start hashing out the solution there. > I've done a first cut of a converter from %-format to {}-format strings. I'm not sure where you want to put it in the sandbox, I've created a gist on GitHub: http://gist.github.com/200936 Not thoroughly tested, but runs in interactive mode so you can try things out. All feedback appreciated! Regards, Vinay From vinay_sajip at yahoo.co.uk Sun Oct 4 00:29:34 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 3 Oct 2009 22:29:34 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <4AC6E019.4000701@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > I'm starting to think that a converter between the two format > mini-languages may be the way to go though. > > fmt_braces is meant to provide a superset of the capabilites of > fmt_percent, so a forward converter shouldn't be too hard. A reverse > converter may have to punt with ValueError when it finds things that > cannot be expressed in the fmt_percent mini language though. I've done a first cut of a forward (% -> {}) converter: http://gist.github.com/200936 but I'm not sure there's a case for a converter in the reverse direction, if we're encouraging movement in one particular direction. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sun Oct 4 00:33:33 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 3 Oct 2009 22:33:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: Raymond Hettinger rcn.com> writes: > We should get one written. ISTM, every %-formatting > string is directly translatable to an equivalent {}-formatting string. > I've made a start, but I'm not sure how best to handle the '#' and ' ' conversion flags. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sun Oct 4 00:43:18 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 3 Oct 2009 22:43:18 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?summary_of_transitioning_from_=25_to_=7B?= =?utf-8?q?=7D_formatting?= References: <200910040501.57077.steve@pearwood.info> Message-ID: Brett Cannon python.org> writes: > No one is saying we should deprecate % any time soon on strings > themselves or anywhere. This discussion is purely in regards to > argparse and logging to transition *their* APIs over to {} formatting > which would most likely involve some deprecation for *using* % > formatting in those APIs. But % formatting on strings themselves is > not directly being discussed here. While I have no problem with supporting {}-formatting in logging if we find a good way of doing it, one thing that bothers me about transitioning the logging API to deprecate or otherwise de-emphasise %-formatting is the question of performance. Now, str.format is more flexible than str.__mod__ and so some performance loss may be tolerable in many scenarios. However, in the feedback I regularly get about logging, people are concerned about performance. No-one ever comes up with any hard numbers, but logging is often bashed as slow (see e.g. Andrii Mishkovskyi's LoggingPackage page on the Python wiki). I don't especially want to add fuel to the fire, as any performance degradation caused by supporting {}-formatting will likely just result in more finger-pointing at logging in general. Regards, Vinay Sajip From brett at python.org Sun Oct 4 00:32:48 2009 From: brett at python.org (Brett Cannon) Date: Sat, 3 Oct 2009 15:32:48 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: On Sat, Oct 3, 2009 at 15:27, Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> Why don't we start something in the sandbox and see how far we can >> get. If no one beats me to it I will add the directory some time today >> and we can start hashing out the solution there. >> > > I've done a first cut of a converter from %-format to {}-format strings. I'm not > sure where you want to put it in the sandbox, I've created a gist on GitHub: > > http://gist.github.com/200936 > > Not thoroughly tested, but runs in interactive mode so you can try things out. > All feedback appreciated! I mentioned it in the README. We now have two separate attempts at converting from % to {}. If people are serious about this we should start to work on a single implementation. -Brett From benjamin at python.org Sun Oct 4 00:38:55 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 3 Oct 2009 17:38:55 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <1afaf6160910031538g1d196ca4w61f3328a9fa8eb90@mail.gmail.com> 2009/10/3 Vinay Sajip : > Raymond Hettinger rcn.com> writes: > >> We should get one written. ?ISTM, every %-formatting >> string is directly translatable to an equivalent {}-formatting string. >> > > I've made a start, but I'm not sure how best to handle the '#' and ' ' > conversion flags. Mine handles it differently for each type specifier. -- Regards, Benjamin From tjreedy at udel.edu Sun Oct 4 01:31:01 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 03 Oct 2009 19:31:01 -0400 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: Steven Bethard wrote: > I thought it might be useful for those who don't have time to read a > million posts to have a summary of what's happened in the formatting > discussion. definitely > > The basic problem is that many APIs in the standard library and > elsewhere support only %-formatting and not {}-formatting, e.g. > logging.Formatter accepts:: > logging.Formatter(fmt="%(asctime)s - %(name)s") > but not:: > logging.Formatter(fmt="{asctime} - {name}") > > There seems to be mostly agreement that these APIs should at least > support both formatting styles, and a sizable group (including Guido) > believe that %-formatting should eventually be phased out (e.g. by > Python 4). There are a number of competing proposals on how to allow > such APIs to start accepting {}-format strings: > > * Add a parameter which declares the type of format string:: > logging.Formatter(fmt="{asctime} - {name}", format=BRACES) > The API code would then switch between %-format and {}-format > based on the value of that parameter. If %-formatting is to be > deprecated, this could be done by first deprecating > format=PERCENTS and requiring format=BRACES, and then changing the > default to format=BRACES. ... > * Create translators between %-format and {}-format:: > assert to_braces("%(asctime)s") == "{asctime}" > assert to_percents("{asctime}") == "%(asctime)s" > these could then either be used outside of the API:: > logging.Formatter(fmt=to_percents("{asctime} - {name}")) > or they could be used within the API combined with some sort of > heuristic for guessing whether a {}-format string or a %-format > string was passed in:: > logging.Formatter(fmt="{asctime} - {name}") How about combining these two. Add an optional form or style=xxx parameter -- which could also allow DOLLARS for $-formats -- for resolving ambiguities. If not present, make a reasonable guess. IE, if string has no '%' and multiple {} pairs, or no {} pairs and multiple %s, the guess is at least .999 sure. which is to not, not hardly a guess. The extra arg can be supplied if and when needed. The machinery for this should be not be logging specific, so it can be used through throughout the library, and exposed so that others can use it. Terry Jan Reedy From python at mrabarnett.plus.com Sun Oct 4 01:38:50 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Oct 2009 00:38:50 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <4AC7E08A.9050601@mrabarnett.plus.com> Vinay Sajip wrote: > Brett Cannon python.org> writes: > >> Why don't we start something in the sandbox and see how far we can >> get. If no one beats me to it I will add the directory some time today >> and we can start hashing out the solution there. >> > > I've done a first cut of a converter from %-format to {}-format strings. I'm not > sure where you want to put it in the sandbox, I've created a gist on GitHub: > > http://gist.github.com/200936 > > Not thoroughly tested, but runs in interactive mode so you can try things out. > All feedback appreciated! > Interesting that you're using the %-format to translate %-formats to {}-formats! :-) From vinay_sajip at yahoo.co.uk Sun Oct 4 01:49:06 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 3 Oct 2009 23:49:06 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4AC7E08A.9050601@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > Interesting that you're using the %-format to translate %-formats to > {}-formats! Yes, ironic, isn't it? ;-) Regards, Vinay Sajip From python at mrabarnett.plus.com Sun Oct 4 02:11:39 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Oct 2009 01:11:39 +0100 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: Message-ID: <4AC7E83B.2010102@mrabarnett.plus.com> Terry Reedy wrote: > Steven Bethard wrote: >> I thought it might be useful for those who don't have time to read a >> million posts to have a summary of what's happened in the formatting >> discussion. > > definitely > >> >> The basic problem is that many APIs in the standard library and >> elsewhere support only %-formatting and not {}-formatting, e.g. >> logging.Formatter accepts:: >> logging.Formatter(fmt="%(asctime)s - %(name)s") >> but not:: >> logging.Formatter(fmt="{asctime} - {name}") >> >> There seems to be mostly agreement that these APIs should at least >> support both formatting styles, and a sizable group (including Guido) >> believe that %-formatting should eventually be phased out (e.g. by >> Python 4). There are a number of competing proposals on how to allow >> such APIs to start accepting {}-format strings: >> >> * Add a parameter which declares the type of format string:: >> logging.Formatter(fmt="{asctime} - {name}", format=BRACES) >> The API code would then switch between %-format and {}-format >> based on the value of that parameter. If %-formatting is to be >> deprecated, this could be done by first deprecating >> format=PERCENTS and requiring format=BRACES, and then changing the >> default to format=BRACES. > > ... > >> * Create translators between %-format and {}-format:: >> assert to_braces("%(asctime)s") == "{asctime}" >> assert to_percents("{asctime}") == "%(asctime)s" >> these could then either be used outside of the API:: >> logging.Formatter(fmt=to_percents("{asctime} - {name}")) >> or they could be used within the API combined with some sort of >> heuristic for guessing whether a {}-format string or a %-format >> string was passed in:: >> logging.Formatter(fmt="{asctime} - {name}") > > How about combining these two. Add an optional form or style=xxx > parameter -- which could also allow DOLLARS for $-formats -- for > resolving ambiguities. If not present, make a reasonable guess. IE, if > string has no '%' and multiple {} pairs, or no {} pairs and multiple %s, > the guess is at least .999 sure. which is to not, not hardly a guess. > The extra arg can be supplied if and when needed. > Maybe if the style is a callable then that's the formatting function? (Or is that asking for trouble? :-)) In that case the style could default to a function which guesses which style is being used and then the appropriate function. > The machinery for this should be not be logging specific, so it can be > used through throughout the library, and exposed so that others can use it. > From steve at pearwood.info Sun Oct 4 03:26:51 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Oct 2009 11:26:51 +1000 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <200910032035.05117@news.perlig.de> References: <200910040522.28374.steve@pearwood.info> <200910032035.05117@news.perlig.de> Message-ID: <200910041226.51573.steve@pearwood.info> On Sun, 4 Oct 2009 05:35:04 am Andr? Malo wrote: > * Steven D'Aprano wrote: > > You don't need a comment warning that you are catching SystemExit > > because parse_args raises SystemExit, any more than you need a > > comment saying that you are catching ValueError because some > > function raises ValueError. The fact that you are catching an > > exception implies that the function might raise that exception. A > > comment like: > > > > "Catching SystemExit because parse_args() throws SystemExit on > > parser errors." > > > > is up them with comments like this: > > > > x += 1 # Add 1 to x. > > It's semantically different. You usually don't catch SystemExit > directly, because you want your programs to be stopped. Exactly -- so why catch it at all? But even if there is a good reason to catch it, there's still no reason to subclass SystemExit unless argparse wants to distinguish different types of fatal error, and allow code to catch some but not all. But frankly, if I'm having a hard time thinking of a reason to catch SystemExit, I'm having an even harder time thinking why you'd want to (say) catch SystemExitTooManyArguments but not SystemExitMissingArgument. > Additionally, > a library exiting your program is badly designed, as it's unexpected. It's not unexpected for an argument parser. Do you know any applications that run when given invalid arguments? As a general rule, what can the application do? Guess what you wanted? > Thatswhy such a comment is useful. The comment doesn't tell you anything that wasn't obvious from the code. It is pointless. > Here's what I'd do: I'd subclass SystemExit in this case and raise > the subclass from argparse. In the following code snippet: try: ns = argparse.parse_args() except SystemExit: ... is there any confusion between SystemExit raised by parse_args and SystemExit raised by other components? *What* other components? If SystemExit was raised in that try block, where could it have come from other than parse_args? Do you write comments like these? try: value = mydict[key] except KeyError: # Catch KeyError raised by dict lookup ... try: n = mylist.index(x) except ValueError: # Catch ValueError raised by mylist.index ... Useless comments are worse than no comments, because useless comments waste the readers' time and they risk becoming out of sync with the code and turning into misleading comments. > That way all parties here should be satisifed. No. It wastes the time of the argparse developer, it wastes the time of people learning argparse, it wastes the time of people who read the code and wonder what's the difference between SystemExit and ArgparseSystemExit. (Answer: there is no difference.) -- Steven D'Aprano From steve at pearwood.info Sun Oct 4 03:29:23 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Oct 2009 11:29:23 +1000 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: References: <200910040522.28374.steve@pearwood.info> Message-ID: <200910041229.23279.steve@pearwood.info> On Sun, 4 Oct 2009 08:21:46 am Robert Kern wrote: > There are uses of argparse outside of command line apps. For example, > I use it to parse --options for IPython %magic commands. Of course, I > just subclass ArgumentParser and replace the .error() method. Exactly. There are uses for catching SystemExit, which is why it's an exception and not an unconditional exit. But they're rare, and not difficult to deal with: in your case, having argparse raise a subclass of SystemExit won't help you, you would still need to subclass, and in other cases, you can just catch SystemExit. > The other use case, as mentioned earlier, was for debugging your > parser on the interactive prompt. A custom subclass may also be able > to hold more machine-readable information about the error than the > formatted error message, but I don't have any particular use cases > about what may be the most useful. Nobody has requested that the exception expose more information. They've requested that argparse paint the SystemExit a slightly different shade of yellow to the colour it already is -- this is pure bike-shedding. Subclassing SystemExit just in case someday in the indefinite future there comes a need to add extra information to the exception falls foul of You Ain't Gonna Need It. Keep it simple -- if, someday, such a need becomes apparent, then subclass. -- Steven D'Aprano From kmtracey at gmail.com Thu Oct 1 03:50:25 2009 From: kmtracey at gmail.com (Karen Tracey) Date: Wed, 30 Sep 2009 21:50:25 -0400 Subject: [Python-Dev] Announcing PEP 3136 In-Reply-To: <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> References: <20090929202035.GA5492@gmail.com> <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> Message-ID: On Wed, Sep 30, 2009 at 5:15 PM, Yuvgoog Greenle wrote: > I like how python has a minimalistic and powerful syntax (-1 for the break > ___ PEP). > Also, I really dislike the for/else ambiguity "butterflies". > When is the else after a loop executed? 1. When the loop isn't entered at > all. > 2. When the loop terminates through exhaustion of the list (does this > include when the list was empty?) > 3. When the loop didn't exit because of a break statement. > > HINTS: The way django does it is opposite the way python does it and there > may be more than one correct answer. > > Django's template language does not have for/else, it has for/empty: http://docs.djangoproject.com/en/dev/ref/templates/builtins/#for-empty This construct did come from an external snippet that used 'else' instead of 'empty'. However when it was moved into Django the 'else' name was specifically rejected because it did the opposite of what for/else does in Python. Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Oct 3 20:37:42 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 3 Oct 2009 13:37:42 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <1afaf6160910031137h2d298f75y99cde9666616bd22@mail.gmail.com> 2009/10/3 Brett Cannon : > On Fri, Oct 2, 2009 at 20:03, Raymond Hettinger wrote: >> >> [Terry Reedy] >>> >>> I would agree, for instance, that an auto-translation tool is needed. >> >> We should get one written. ?ISTM, every %-formatting >> string is directly translatable to an equivalent {}-formatting string. > > Why don't we start something in the sandbox and see how far we can > get. If no one beats me to it I will add the directory some time today > and we can start hashing out the solution there. I've already started a converter. It's here: https://code.launchpad.net/~gutworth/+junk/mod2format -- Regards, Benjamin From ncoghlan at gmail.com Sat Oct 3 11:37:00 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 03 Oct 2009 19:37:00 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> Message-ID: <4AC71B3C.10908@gmail.com> Bugbee, Larry wrote: > So until I see a *significant* benefit, my vote is *not* remove > %-formatting. Make both available and if {} is to win, it will. Percent formatting isn't going anywhere (certainly not for the next decade or so). However, its well documented limitations (especially the lack of extensibility) and error-prone syntax (handling of dicts and tuples, forgetting trailing type codes in namespace formatting, operator precedence issues) are what led directly to PEP 3101 and the introduction of brace formatting. For the basic cases that percent formatting actually covers, it does a reasonable job and is often more concise and currently executes faster than the brace formatted equivalent (this was especially the case before support for implicit positional argument numbering was added to brace formatting). The advantages of brace formatting don't really start to show up until you actually start using the *new* features that the approach provides, such as formatting for non-builtin types (e.g. Decimal and datetime objects), referring to argument subscripts and attributes in field definitions and the optional explicit numbering of references to positional arguments (allowing the order of interpolation to be changed in the format string without having to change the argument order in all uses of that format string). Also, don't forget that the percent formatting code has been around for more than a decade and a half and has been optimised over that time. The brace formatting code, on the other hand, is relatively new and probably still offers plenty of opportunities for optimisation (although the additionally flexibility in the brace formatting approach means that it is unlikely to ever catch up completely to the raw speed of percent formatting). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From benjamin at python.org Thu Oct 1 00:21:12 2009 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 30 Sep 2009 17:21:12 -0500 Subject: [Python-Dev] Python 2.6.3rc1 available In-Reply-To: <87pr98yt2m.fsf@benfinney.id.au> References: <87pr98yt2m.fsf@benfinney.id.au> Message-ID: <1afaf6160909301521pfcc347fsa0f297c72945dabe@mail.gmail.com> 2009/9/30 Ben Finney : > Barry Warsaw writes: > >> The first (and hopefully last) release candidate for Python 2.6.3 is >> now available > [?] > > Thank you for this announcement, and the efforts that go into making > this work available. > > *Especially* thank you for avoiding the oxymoronic ?Released: 2.6.3 > release candidate? or similar. I hope this signifies a trend toward > using the more accurate term ?available? for announcing availability of > anything that's not an actual release. Alphas, betas, and release candidates are all releases, too. -- Regards, Benjamin From ncoghlan at gmail.com Sun Oct 4 10:13:59 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Oct 2009 18:13:59 +1000 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: <4AC783BC.70404@mrabarnett.plus.com> Message-ID: <4AC85947.3030702@gmail.com> Antoine Pitrou wrote: > If we can't find a way to make things almost transparent, we should IMO abandon > the whole idea of a transition. Yep - this is the reason some attempts at actual format translation implementations started up as a result of the previous discussion. Without that, I suspect any API transitions are going to be too much hassle to be worthwhile. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Oct 4 10:23:06 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Oct 2009 18:23:06 +1000 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: <200910040501.57077.steve@pearwood.info> Message-ID: <4AC85B6A.2040600@gmail.com> Brett Cannon wrote: > No one is saying we should deprecate % any time soon on strings > themselves or anywhere. This discussion is purely in regards to > argparse and logging to transition *their* APIs over to {} formatting > which would most likely involve some deprecation for *using* % > formatting in those APIs. But % formatting on strings themselves is > not directly being discussed here. Actually, I think percent formatting and brace formatting should remain fully supported peers for the life of 3.x (including in APIs such as argparse and logging). No matter how much we might like the new formatting system (and I certainly prefer it), there's a long period of evangelisation and performance improvements that needs to happen before we can realistically start to deprecate percent formatting based APIs in any way shape or form. Until that happens, then users do have the right to dig in their heels and say "look, the limitations on percent formatting really don't bother us, certainly not enough to switch to a new formatting system". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at barrys-emacs.org Sun Oct 4 14:10:57 2009 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 4 Oct 2009 13:10:57 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC47856.2090307@egenix.com> References: <4AC2CD65.30905@v.loewis.de> <4AC33DE1.3040304@trueblade.com> <4AC47856.2090307@egenix.com> Message-ID: <941A2586-F79E-4F5E-A77F-01FA0C183BBB@barrys-emacs.org> On 1 Oct 2009, at 10:37, M.-A. Lemburg wrote: > Eric Smith wrote: >> Martin v. L?wis wrote: >>> Steven Bethard wrote: >>>> There's a lot of code already out there (in the standard library >>>> and >>>> other places) that uses %-style formatting, when in Python 3.0 we >>>> should be encouraging {}-style formatting. >>> >>> I don't agree that we should do that. I see nothing wrong with using >>> % substitution. > > I agree with Martin. > > Both approaches have their ups and downs, but forcing users to move > from %-formatting to .format()-formatting will just frustrate them: > having to convert several thousand such (working) uses in their code > with absolutely no benefit simply doesn't look like a good way to > spend your time. I agree you cannot force the move to {} format. There are programs that expose the %(name)s in user interfaces for customisation. > > In addition to the code changes, such a move would also render > existing translations of the %-formatted string templates useless. Speaking of translation support has xgettext been updated to support {}? It is a life saver to have xgettext report that "This %s and %s" is not translatable. Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Oct 4 10:25:50 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Oct 2009 18:25:50 +1000 Subject: [Python-Dev] Python 2.6.3rc1 available In-Reply-To: <1afaf6160909301521pfcc347fsa0f297c72945dabe@mail.gmail.com> References: <87pr98yt2m.fsf@benfinney.id.au> <1afaf6160909301521pfcc347fsa0f297c72945dabe@mail.gmail.com> Message-ID: <4AC85C0E.9000800@gmail.com> Benjamin Peterson wrote: > 2009/9/30 Ben Finney : >> Barry Warsaw writes: >> >>> The first (and hopefully last) release candidate for Python 2.6.3 is >>> now available >> [?] >> >> Thank you for this announcement, and the efforts that go into making >> this work available. >> >> *Especially* thank you for avoiding the oxymoronic ?Released: 2.6.3 >> release candidate? or similar. I hope this signifies a trend toward >> using the more accurate term ?available? for announcing availability of >> anything that's not an actual release. > > Alphas, betas, and release candidates are all releases, too. Not really - they're just key builds that are made available in the lead up to the actual release. Using " is available" for the pre-release builds and then "RELEASED: " for the actual release as Barry did this time around makes a great deal of sense to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sat Oct 3 22:09:36 2009 From: brett at python.org (Brett Cannon) Date: Sat, 3 Oct 2009 13:09:36 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: On Sat, Oct 3, 2009 at 11:05, Brett Cannon wrote: > On Fri, Oct 2, 2009 at 20:03, Raymond Hettinger wrote: >> >> [Terry Reedy] >>> >>> I would agree, for instance, that an auto-translation tool is needed. >> >> We should get one written. ?ISTM, every %-formatting >> string is directly translatable to an equivalent {}-formatting string. > > Why don't we start something in the sandbox and see how far we can > get. If no one beats me to it I will add the directory some time today > and we can start hashing out the solution there. Created str_fmt_conversion in the sandbox to hold any work people want to do on converting % formats to {} automatically. -Brett From ctrachte at gmail.com Fri Oct 2 22:23:11 2009 From: ctrachte at gmail.com (Carl Trachte) Date: Fri, 2 Oct 2009 14:23:11 -0600 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <87y6nt37kb.fsf@hbox.dyndns.org> References: <87y6nt37kb.fsf@hbox.dyndns.org> Message-ID: <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> >> Do the users get any say in this? > > I'm a user! :-) > > I hate calling methods on string literals, I think it looks very odd to > have code like this: > > "Displaying {0} of {1} revisions".format(x, y) > > Will we be able to write this as > > "Displaying {0} of {1} revisions" % (x, y) > > too? > >> I imagine that some people are heavily invested in %-formatting. >> >> Because there has been limited uptake on {}-formatting (afaict), we >> still have limited experience with knowing that it is actually better, >> less error-prone, easier to learn/rember, etc. Outside a handful of >> people on this list, I have yet to see anyone adopt it as the >> preferred syntax. > > I've skimmed over the PEP, and the new {}-syntax seems to have some nice > features. But I've not seen it used anywhere yet. > Rami Chowdhury posted this to a mailing list; I've been using it (perhaps unintentionally promoting it) as part of non-English, non-ASCII font outreach: > def ?????_??(???): > ??? = "????? {0}. ???? ?????".format(???) > print(???) > > def say_greeting(name): > to_say = "Greetings, {0}. How are you?".format(name) > print(to_say) As a user, my assumption was {} was going forward, rain or shine, and everyone should be on board by Python 3.2. (I thought once the Talin PEP got approved, that was it). I wrote Steven Bethard privately about this. Sorry for the intrusion. From songofacandy at gmail.com Mon Oct 5 03:52:31 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 5 Oct 2009 10:52:31 +0900 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> References: <87y6nt37kb.fsf@hbox.dyndns.org> <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> Message-ID: What about using string prefix 'f'? f"{foo} and {bar}" % something == "{foo} and {bar}.format(something) s = f"{foo}" t = "%(bar)s" s + t # raises Exception Transition plan: n: Just add F prefix. And adding "format_string" in future. n+1: deprecate __mod__() without 'F'. n+2: libraries use .format() and deprecate __mod__() with 'F' n+3: remove __mod__() -- Naoki INADA From songofacandy at gmail.com Mon Oct 5 04:07:07 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 5 Oct 2009 11:07:07 +0900 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <1afaf6160910041854v55edcc96we4fb2d6a7f2da948@mail.gmail.com> References: <87y6nt37kb.fsf@hbox.dyndns.org> <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> <1afaf6160910041854v55edcc96we4fb2d6a7f2da948@mail.gmail.com> Message-ID: > -1 That requires keeping formatting information around in every string instance. Adding new "format_string" class avoids it. unicode("foo") <=> u"foo" format_string("foo") <=> f"foo" This way's pros: * Many libraries can use one transition way. * Transition stage syncs to Python version. "library A uses {} and library B uses %" problem not happen in transition. * We have experience same way on unicode. -- Naoki INADA From tjreedy at udel.edu Mon Oct 5 06:43:05 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 05 Oct 2009 00:43:05 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> References: <87y6nt37kb.fsf@hbox.dyndns.org> <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> Message-ID: Carl Trachte wrote: >> I've skimmed over the PEP, and the new {}-syntax seems to have some nice >> features. But I've not seen it used anywhere yet. I am using it with 3.1 in an unreleased book I am still writing, and will in any code I publish. > > > Rami Chowdhury posted this to a mailing list; I've been using it > (perhaps unintentionally promoting it) as part of non-English, > non-ASCII font outreach: > >> def ?????_??(???): >> ??? = "????? {0}. ???? ?????".format(???) >> print(???) >> >> def say_greeting(name): >> to_say = "Greetings, {0}. How are you?".format(name) >> print(to_say) > > As a user, my assumption was {} was going forward, rain or shine, and > everyone should be on board by Python 3.2. Autonumbering, added in 3.1, makes '{}' as easy to write for simple cases as '%s'. That was one complaint about the original 3.0 version. Another was and still is the lack of conversion, which is being worked on. (I thought once the Talin > PEP got approved, that was it). I wrote Steven Bethard privately > about this. > > Sorry for the intrusion. > > tjr From martin at v.loewis.de Mon Oct 5 07:30:45 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 05 Oct 2009 07:30:45 +0200 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: <4AC98485.5050608@v.loewis.de> > I agree with this 100% but I can't see it working unless we have some > sort of transition plan. Just saying "ok, switch your format strings > from % to {}" didn't work in Python 3.0 for various good reasons, and > I can't imagine it will work in Python 4.0 unless we have a transition > plan. That doesn't mean we have to have a transition plan *now*. Creating one after Python 3.5 is released (i.e. in 2015 or so) might be sufficient. To create a transition plan, you first need *consensus* that you actually do want to transition. I don't think such consensus is available, and might not be available for a few more years. Pushing the issue probably delays obtaining consensus. Regards, Martin From ncoghlan at gmail.com Sun Oct 4 10:03:24 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Oct 2009 18:03:24 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <4AC6E019.4000701@gmail.com> Message-ID: <4AC856CC.4010806@gmail.com> Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> I'm starting to think that a converter between the two format >> mini-languages may be the way to go though. >> >> fmt_braces is meant to provide a superset of the capabilites of >> fmt_percent, so a forward converter shouldn't be too hard. A reverse >> converter may have to punt with ValueError when it finds things that >> cannot be expressed in the fmt_percent mini language though. > > I've done a first cut of a forward (% -> {}) converter: > > http://gist.github.com/200936 > > but I'm not sure there's a case for a converter in the reverse direction, if > we're encouraging movement in one particular direction. It would allow an application to still use brace formatting throughout even if one particularly library only accepted percent formatting. Probably not worth the effort at this point though, as if we can get a reliable forward converter happening then it may become possible for APIs to correctly guess which kind of format string they have been passed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From vinay_sajip at yahoo.co.uk Mon Oct 5 10:27:49 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 5 Oct 2009 08:27:49 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: Raymond Hettinger rcn.com> writes: > We should get one written. ISTM, every %-formatting > string is directly translatable to an equivalent {}-formatting string. I'm not sure you can always get equivalent output from the formatting, though. For example: >>> "%0#8x" % 0x1234 '0x001234' >>> "{0:0>#8x}".format(0x1234) '000x1234' >>> Someone please tell me if there's a better {}-format string which I've missed! Regards, Vinay Sajip From fredrik at pythonware.com Mon Oct 5 10:43:22 2009 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 5 Oct 2009 10:43:22 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? Message-ID: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> it's revews like this that makes me wonder if releasing open source is a good idea: no egg - worst seen ever, remove it from pypi or provide an egg (jensens, 2009-10-05, 0 points) From p.f.moore at gmail.com Mon Oct 5 10:41:08 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2009 09:41:08 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <79990c6b0910050141r46338bdcjb7219a0658b01a5@mail.gmail.com> 2009/10/5 Vinay Sajip : > Raymond Hettinger rcn.com> writes: >> We should get one written. ?ISTM, every %-formatting >> string is directly translatable to an equivalent {}-formatting string. > > I'm not sure you can always get equivalent output from the formatting, though. > For example: > >>>> "%0#8x" % 0x1234 > '0x001234' >>>> "{0:0>#8x}".format(0x1234) > '000x1234' >>>> > > Someone please tell me if there's a better {}-format string which I've missed! Ironically, >>> "0x{0:0>6x}".format(0x1234) '0x001234' ignoring the # modifier altogether... Paul. From python at mrabarnett.plus.com Mon Oct 5 10:47:30 2009 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 05 Oct 2009 09:47:30 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <4AC9B2A2.3010808@mrabarnett.plus.com> Vinay Sajip wrote: > Raymond Hettinger rcn.com> writes: >> We should get one written. ISTM, every %-formatting >> string is directly translatable to an equivalent {}-formatting string. > > I'm not sure you can always get equivalent output from the formatting, though. > For example: > >>>> "%0#8x" % 0x1234 > '0x001234' >>>> "{0:0>#8x}".format(0x1234) > '000x1234' > > Someone please tell me if there's a better {}-format string which I've missed! > >>> "{0:#08x}".format(0x1234) '0x001234' From ben+python at benfinney.id.au Mon Oct 5 10:59:41 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Oct 2009 19:59:41 +1100 Subject: [Python-Dev] eggs now mandatory for pypi? References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> Message-ID: <87hbuew70i.fsf@benfinney.id.au> Fredrik Lundh writes: > it's revews like this that makes me wonder if releasing open source is > a good idea: > > no egg - worst seen ever, remove it from pypi or provide an egg > (jensens, 2009-10-05, 0 points) Heh. If harsh, uninformed responses make you wonder whether it's worth doing free software, then you're in the wrong social philanthropy movement. Make the software good quality, release it in the standard format (in this case, a Distutils ?sdist? on PyPI), respond in a timely manner to genuine requests for help. Eggs are certainly not a requirement, nor is responding to hostility like the above comment. If you *want* to respond, you can politely direct them to where they can learn how to install Python distributions ? no mention of eggs at all. -- \ ?What is it that makes a complete stranger dive into an icy | `\ river to save a solid gold baby? Maybe we'll never know.? ?Jack | _o__) Handey | Ben Finney From vinay_sajip at yahoo.co.uk Mon Oct 5 12:16:53 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 5 Oct 2009 10:16:53 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4AC9B2A2.3010808@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > >>> "{0:#08x}".format(0x1234) > '0x001234' Good call, but here's another case: >>> "%#o" % 0x1234 '011064' I don't see how to automatically convert the "%#o" spec, though of course there are ways of fudging it. The obvious conversion doesn't give the same value: >>> "{0:#o}".format(0x1234) '0o11064' I couldn't see a backward-compatibility mode for str.format generating just a leading "0" for octal alternative format (the C style) as opposed to "0o". Regards, Vinay Sajip From orsenthil at gmail.com Mon Oct 5 12:25:47 2009 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 5 Oct 2009 15:55:47 +0530 Subject: [Python-Dev] eggs now mandatory for pypi? Message-ID: <20091005102547.GA23457@ubuntu.ubuntu-domain> On Mon, Oct 05, 2009 at 10:43:22AM +0200, Fredrik Lundh wrote: > it's revews like this that makes me wonder if releasing open source is > a good idea: > no egg - worst seen ever, remove it from pypi or provide an egg > (jensens, 2009-10-05, 0 points) > Greetings effbot. :) As you might already know, "eggs" are not mandatory for pypi. No where in the documentation it mentions so. Also many good packages in pypi are not eggs (MaL's egenix packages for e.g) and we have known his dislike for that particular format too. But still, sometimes people have found eggs to be convenient. That was a particularly harsh comment from jensens, should be ignored or he should be corrected that eggs are not mandatory. But, i have a feeling that this is going to open up the well known, can-of-worms on what is the best distribution format for python packages debate. -- Senthil No one may kill a man. Not for any purpose. It cannot be condoned. -- Kirk, "Spock's Brain", stardate 5431.6 From fredrik at pythonware.com Mon Oct 5 10:58:06 2009 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 5 Oct 2009 10:58:06 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> Message-ID: <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> Oh, it was just yet another Zope developer behaving like an ass. Why am I not surprised? On Mon, Oct 5, 2009 at 10:43 AM, Fredrik Lundh wrote: > it's reviews like this that makes me wonder if releasing open source is > a good idea: > > ? no egg - worst seen ever, remove it from pypi or provide an egg > (jensens, 2009-10-05, 0 points) > > > From ncoghlan at gmail.com Mon Oct 5 12:49:22 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 05 Oct 2009 20:49:22 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC98485.5050608@v.loewis.de> References: <4AC98485.5050608@v.loewis.de> Message-ID: <4AC9CF32.4010404@gmail.com> Martin v. L?wis wrote: > That doesn't mean we have to have a transition plan *now*. Creating one > after Python 3.5 is released (i.e. in 2015 or so) might be sufficient. > > To create a transition plan, you first need *consensus* that you > actually do want to transition. I don't think such consensus is > available, and might not be available for a few more years. Pushing > the issue probably delays obtaining consensus. Agreed, but that doesn't rule out discussions of what can be done to make such a transition easier. And just as 2to3 makes the overall Python transition practical, a percent to brace format translator should make an eventual formatting transition feasible. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fdrake at gmail.com Mon Oct 5 12:47:29 2009 From: fdrake at gmail.com (Fred Drake) Date: Mon, 5 Oct 2009 06:47:29 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> Message-ID: <9cee7ab80910050347k6c5985b3m7290f64099526a2b@mail.gmail.com> On Mon, Oct 5, 2009 at 4:58 AM, Fredrik Lundh wrote: > Oh, it was just yet another Zope developer behaving like an ass. ?Why > am I not surprised? Actually, most of us Zope developers prefer sdist packages, not eggs. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From ncoghlan at gmail.com Mon Oct 5 13:02:16 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 05 Oct 2009 21:02:16 +1000 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> Message-ID: <4AC9D238.5000501@gmail.com> Fredrik Lundh wrote: > Oh, it was just yet another Zope developer behaving like an ass. Why > am I not surprised? Tarring an entire community for the actions of one twit is more than a little unfair. It's fine that you don't like eggs and it's fine that you don't want to provide them. There's a reason egg-based packaging tools make it fairly easy to create an egg from a normal sdist package. As for people being twits... it's the internet. There isn't a lot to be done other than to ignore them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From lists at cheimes.de Mon Oct 5 13:07:02 2009 From: lists at cheimes.de (Christian Heimes) Date: Mon, 05 Oct 2009 13:07:02 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> Message-ID: <4AC9D356.5050202@cheimes.de> Fredrik Lundh wrote: > Oh, it was just yet another Zope developer behaving like an ass. Why > am I not surprised? > > > > On Mon, Oct 5, 2009 at 10:43 AM, Fredrik Lundh wrote: >> it's reviews like this that makes me wonder if releasing open source is >> a good idea: >> >> no egg - worst seen ever, remove it from pypi or provide an egg >> (jensens, 2009-10-05, 0 points) >> >> Actually Jens is a Plone developer ... I'm including Jens in this discussion so he may shed some light on his comment. Christian From fredrik at pythonware.com Mon Oct 5 13:28:43 2009 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 5 Oct 2009 13:28:43 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4AC9D238.5000501@gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> <4AC9D238.5000501@gmail.com> Message-ID: <368a5cd50910050428i30bded11v3d408b72b75e1e62@mail.gmail.com> He's not the first one from the Zope community (whatever that is) that's behaved this way on this specific topic. The problem here is that a certain (marginal) user community decides to standardize on a certain distribution model, and then goes off attacking people who've released stuff *before* they did that. That *is* a community problem. (Luckily, there are people helping out, and the "nice people driven-development" rule overrides that other rule I mentioned, so things will get tweaked sooner or later.) On Mon, Oct 5, 2009 at 1:02 PM, Nick Coghlan wrote: > Fredrik Lundh wrote: >> Oh, it was just yet another Zope developer behaving like an ass. ?Why >> am I not surprised? > > Tarring an entire community for the actions of one twit is more than a > little unfair. > > It's fine that you don't like eggs and it's fine that you don't want to > provide them. There's a reason egg-based packaging tools make it fairly > easy to create an egg from a normal sdist package. > > As for people being twits... it's the internet. There isn't a lot to be > done other than to ignore them. > > Cheers, > Nick. > > -- > Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia > --------------------------------------------------------------- > From jens at bluedynamics.com Mon Oct 5 13:26:29 2009 From: jens at bluedynamics.com (Jens W. Klein) Date: Mon, 05 Oct 2009 13:26:29 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4AC9D356.5050202@cheimes.de> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> <4AC9D356.5050202@cheimes.de> Message-ID: <1254741989.7368.22.camel@minime> Am Montag, den 05.10.2009, 13:07 +0200 schrieb Christian Heimes: > Fredrik Lundh wrote: > > Oh, it was just yet another Zope developer behaving like an ass. Why > > am I not surprised? > > > > > > > > On Mon, Oct 5, 2009 at 10:43 AM, Fredrik Lundh wrote: > >> it's reviews like this that makes me wonder if releasing open source is > >> a good idea: > >> > >> no egg - worst seen ever, remove it from pypi or provide an egg > >> (jensens, 2009-10-05, 0 points) > >> > >> > > Actually Jens is a Plone developer ... > > I'm including Jens in this discussion so he may shed some light on his > comment. > > Christian I wrote a private message to Frederik (gmail account) and apologised for my wrong comment. It was targeted wrong. Note-to-self: Never post when angry about some $whatever. And as far as I understand PIL is not the problem, but the packaging/ setuptools. For the records: PIL is a great piece of software and I dont want to miss it. I hope in future we have a packaging-tool solving those problems. regards Jens From nad at acm.org Mon Oct 5 14:06:56 2009 From: nad at acm.org (Ned Deily) Date: Mon, 05 Oct 2009 05:06:56 -0700 Subject: [Python-Dev] Package install failures in 2.6.3 Message-ID: There has been some discussion on the distutils-sig list over the past few days about a change to distutils released with 2.6.3 which inadvertently causes a number (all?) packages with C extensions that use setuptools/easy_install to fail during builds/installs with a rather cryptic error message. A number of popular packages are affected directly or indirectly through dependencies. The problem is summarized in the issue tracker here; http://bugs.python.org/issue7064 which includes links to the distutils list archive and the setuptools and Distribute project bug trackers. Assuming that distutils is not changed in a forthcoming 2.6.4, users and, possibly package maintainers, will need to take some action: either migrate from setuptools to Distribute (an easy migration), or wait for a fix to setuptools or the package or to the distribution (if there is one). I'm not proposing to debate the merits of all of the options here. However, if a 2.6.4 release is to be pushed out quickly for other reasons, a one-time window of opportunity would be available and it would be prudent to at least consider the possibility of a distutils fix. In any case, it seems like there should be some effort to make package maintainers aware of the problem so they can take whatever steps necessary to update their packages or help files or web sites etc. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Mon Oct 5 14:22:06 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 14:22:06 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: Message-ID: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> On Mon, Oct 5, 2009 at 2:06 PM, Ned Deily wrote: > Assuming that distutils is not changed in a forthcoming 2.6.4 Since the changes made in Distutils were bug fixes that kept all public API backward compatible, I don't see why this should be changed. Setuptools development has been discontinued for a year, and does patches on Distutils code. Some of these patches are sensitive to any change made on Distutils, wether those changes are internal or not. It's up to the Setuptools project to provide a release that fixes this problem. And this fix is one or two lines long. Distutils will *not* check in its code if it was patched by Setuptools, to make sure Setuptools 0.6c9 still works. > I'm not proposing to debate the merits of all of the options here. > However, if a 2.6.4 release is to be pushed out quickly for other > reasons, ?a one-time window of opportunity would be available and it > would be prudent to at least consider the possibility of a distutils fix. What about a Setuptools release, with the proper fix ? Tarek From mal at egenix.com Mon Oct 5 14:27:46 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Oct 2009 14:27:46 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> Message-ID: <4AC9E642.3070508@egenix.com> Tarek Ziad? wrote: >> I'm not proposing to debate the merits of all of the options here. >> However, if a 2.6.4 release is to be pushed out quickly for other >> reasons, a one-time window of opportunity would be available and it >> would be prudent to at least consider the possibility of a distutils fix. > > What about a Setuptools release, with the proper fix ? +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From skip at pobox.com Mon Oct 5 14:50:13 2009 From: skip at pobox.com (skip at pobox.com) Date: Mon, 5 Oct 2009 07:50:13 -0500 Subject: [Python-Dev] [New-bugs-announce] [issue7064] Python 2.6.3 / setuptools 0.6c9: extension module builds fail with KeyError In-Reply-To: <1254742115.57.0.346118025313.issue7064@psf.upfronthosting.co.za> References: <1254742115.57.0.346118025313.issue7064@psf.upfronthosting.co.za> Message-ID: <19145.60293.661555.312221@montanaro.dyndns.org> Ned> Due to a change in distutils released with Python 2.6.3, packages Ned> that use setuptools (version 0.6c9, as of this writing), or the Ned> easy_install command, to build C extension modules fail ... ... Ned> Among the packages known to be affected include lxml, Ned> zope-interface, jinja2, and, hence, packages dependent on these Ned> packages (e.g. sphinx, twisted, etc.). Maybe the Python test suite should include tests with a small number of widely used non-core packages like setuptools. I realize the pybots project exists to tackle this sort of stuff in greater detail. I'm thinking more of a smoke test than a comprehensive test suite covering all external packages. Setuptools is particularly important because so many extension authors use it. If it breaks it implicitly breaks a lot of PyPI packages. Skip From ziade.tarek at gmail.com Mon Oct 5 15:04:11 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 15:04:11 +0200 Subject: [Python-Dev] [New-bugs-announce] [issue7064] Python 2.6.3 / setuptools 0.6c9: extension module builds fail with KeyError In-Reply-To: <19145.60293.661555.312221@montanaro.dyndns.org> References: <1254742115.57.0.346118025313.issue7064@psf.upfronthosting.co.za> <19145.60293.661555.312221@montanaro.dyndns.org> Message-ID: <94bdd2610910050604k5cc20cdblf24a65c307a1cfdf@mail.gmail.com> On Mon, Oct 5, 2009 at 2:50 PM, wrote: > > ? ?Ned> Due to a change in distutils released with Python 2.6.3, packages > ? ?Ned> that use setuptools (version 0.6c9, as of this writing), or the > ? ?Ned> easy_install command, to build C extension modules fail ... > ? ?... > ? ?Ned> Among the packages known to be affected include lxml, > ? ?Ned> zope-interface, jinja2, and, hence, packages dependent on these > ? ?Ned> packages (e.g. sphinx, twisted, etc.). > > Maybe the Python test suite should include tests with a small number of > widely used non-core packages like setuptools. ?I realize the pybots project > exists to tackle this sort of stuff in greater detail. ?I'm thinking more of > a smoke test than a comprehensive test suite covering all external packages. > Setuptools is particularly important because so many extension authors use > it. ?If it breaks it implicitly breaks a lot of PyPI packages. > I have created 6 months ago such a buildbot that downloads tarballs from the community, and run a few distutils commands on them, and make sure the result is similar in 2.6/2.7. and for "sdist" that the resulting tarball is similar. It was running over Twisted and Numpy, but has been discontinued because it was on my own server, where it was hard to keep it up (cpu/bandwidth) If the Snakebite project could host my buildbot (or at least slaves) or if the PSF could pay for a dedicated server for this, we would be able to trigger such warnings, and provide an e-mail service to package maintainers for example. The build could occur everytime Distutils *or* the project changes. > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From jnoller at gmail.com Mon Oct 5 15:13:44 2009 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 5 Oct 2009 09:13:44 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> Message-ID: <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> On Mon, Oct 5, 2009 at 4:43 AM, Fredrik Lundh wrote: > it's revews like this that makes me wonder if releasing open source is > a good idea: > > ? no egg - worst seen ever, remove it from pypi or provide an egg > (jensens, 2009-10-05, 0 points) > > Unfortunately; we're now staring down the barrel of having youtube-style comments on Python packages on the index. From barry at python.org Mon Oct 5 15:32:28 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Oct 2009 09:32:28 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> Message-ID: On Oct 5, 2009, at 8:22 AM, Tarek Ziad? wrote: >> I'm not proposing to debate the merits of all of the options here. >> However, if a 2.6.4 release is to be pushed out quickly for other >> reasons, a one-time window of opportunity would be available and it >> would be prudent to at least consider the possibility of a >> distutils fix. > > What about a Setuptools release, with the proper fix ? If setuptools can be made to work with Python 2.6.3 /and/ earlier versions of Python 2.6.x, then it should be patched and an update released. If not, then perhaps we should revert the change in a quick Python 2.6.4. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From olemis at gmail.com Mon Oct 5 16:07:52 2009 From: olemis at gmail.com (Olemis Lang) Date: Mon, 5 Oct 2009 09:07:52 -0500 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <1254741989.7368.22.camel@minime> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> <4AC9D356.5050202@cheimes.de> <1254741989.7368.22.camel@minime> Message-ID: <24ea26600910050707t3d5cbfccu9a3e9113503ef0d5@mail.gmail.com> On Mon, Oct 5, 2009 at 6:26 AM, Jens W. Klein wrote: > Am Montag, den 05.10.2009, 13:07 +0200 schrieb Christian Heimes: >> Fredrik Lundh wrote: >> > >> > Oh, it was just yet another Zope developer behaving like an ass. ?Why >> > am I not surprised? >> > >> > On Mon, Oct 5, 2009 at 10:43 AM, Fredrik Lundh wrote: >> >> it's reviews like this that makes me wonder if releasing open source is >> >> a good idea: >> >> Peace for the world brothers . Lots of yoga, meditation, tai-chi ... and tons of FOSS >> >> ? no egg - worst seen ever, remove it from pypi or provide an egg >> >> (jensens, 2009-10-05, 0 points) >> >> [...] > > Note-to-self: Never post when > angry about some $whatever. e.g. I don't do it either when I'm drunk or thinking about Britney ;o) > And as far as I understand PIL is not the > problem, but the packaging/ setuptools. well I just mentioned few times before (and J. P. Eby said something too, many times actually ;o) that setuptools solved a crucial problem that was unsolved when it was released > For the records: PIL is a great > piece of software and I dont want to miss it. > Yes, and the fact is that without setuptools many other wonderful (commands | libs | frameworks | apps | software) would be in the darkness. I mention some of them : - setuptools `test` command - Trac - PasteDeploy - ... and here fuel is over ... there are a lot believe me ;o) the "small" price to pay is that there are a few annoying implementation details in there ... > I hope in future we have a packaging-tool solving those problems. > ... or OTOH that some time is needed to improve it ;o) but considering the benefits ... Besides you could use {{{ $ easy_install -Z eggs_are_awesome.egg }}} and the package will be installed directly in the file system ( modify .pth + PYTHONPATH if desired ;o). -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html From ziade.tarek at gmail.com Mon Oct 5 16:17:07 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 16:17:07 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> Message-ID: <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> On Mon, Oct 5, 2009 at 3:32 PM, Barry Warsaw wrote: > On Oct 5, 2009, at 8:22 AM, Tarek Ziad? wrote: > >>> I'm not proposing to debate the merits of all of the options here. >>> However, if a 2.6.4 release is to be pushed out quickly for other >>> reasons, ?a one-time window of opportunity would be available and it >>> would be prudent to at least consider the possibility of a distutils fix. >> >> What about a Setuptools release, with the proper fix ? > > If setuptools can be made to work with Python 2.6.3 /and/ earlier versions > of Python 2.6.x, then it should be patched and an update released. ?If not, > then perhaps we should revert the change in a quick Python 2.6.4. It's technically possible to fix Setuptools. We did this fix on Distribute, and the patch is 2 lines long. But it's just a matter of having the maintainer doing it. A few months ago we couldn't make him fix and release the bug that made setuptools fail with svn 1.6, and the year before it took several months to get it fixed for svn 1.5 (a one line, not risky change !!!) That's why we have forked and created Distribute, to provide bug fixes. If PJE is not concerned anymore by the maintenance, imho he should let someone that is willing to do it take over the maintenance of his package to fix this (and the other problems). That is not a new problem. Beware that I don't want to run in any new vicious thread here: I had my share of those. So about taking over Setuptools maintenance : 1/ I am not saying it should be me, and I am not saying that I am offended that PJE didn't open the maintenance of setuptools to me. I think he should trust the community and let the maintenance of setuptools be done by all the people that are actively working on the topic. 2/ No, as someone told me in IRC, that's not an evil plan of mine to make people switch to Distribute. This is not in our interest, it's a loss-loss situation. Now I am strongly opposed to revert any bug fix change in Distutils just because it breaks Setuptools, which is unmaintained since a year. We have been struggling over a year with this issue. And we are still struggling because we have to work in a fork to try to provide solutions for the community, with a lot of bootstrapping issues. Now I am astonished that we are talking about reverting changes in Distutils that were done for bugfixes, for a third party package that does monkey patches on Distutils. If this choice wins here, it means that setuptools and the stdlib are tied together, and that the setuptools package should be integrated to the stdlib *immediatly*. Tarek From mal at egenix.com Mon Oct 5 16:27:00 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Oct 2009 16:27:00 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: <4ACA0234.1070700@egenix.com> Tarek Ziad? wrote: > Now I am astonished that we are talking about reverting changes in > Distutils that were done for bugfixes, for a third party package that > does monkey > patches on Distutils. > > If this choice wins here, it means that setuptools and the stdlib are > tied together, and that the setuptools package should be integrated to > the stdlib *immediatly*. We've discussed ways of doing that some years ago and found that it was not possible to make ends join. I'd much rather like to see some of the features in setuptools get implemented in distutils. eGenix could contribute a bdist_egg implementation that doesn't rely on setuptools and its monkey patches - along with some other new commands that people might find useful such as the ability to build Unix libraries, optional and self-configuring extensions, autoconf-style setups, etc. (see mxSetup.py in egenix-mx-base for all the details). We'd just need some help with integrating the things into distutils, since we currently don't have the bandwidth for such things. Aside: After importing setuptools a lot of things in distutils no longer work as advertised. This is due to the heavy monkey patching being applied in setuptools. In some cases there's on other way to adapt distutils to ones needs, but most of the times, a little more thought put into the OO design of a tool makes it possible to play nice with distutils without any monkey patching. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Mon Oct 5 16:32:13 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Oct 2009 10:32:13 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: On Oct 5, 2009, at 10:17 AM, Tarek Ziad? wrote: >> If setuptools can be made to work with Python 2.6.3 /and/ earlier >> versions >> of Python 2.6.x, then it should be patched and an update released. >> If not, >> then perhaps we should revert the change in a quick Python 2.6.4. > > It's technically possible to fix Setuptools. We did this fix on > Distribute, and > the patch is 2 lines long. My question was less about the political aspects than the technical aspects. I gather you're saying that the fix to setuptools will make it work in 2.6.3 without inadvertently breaking it for 2.6.2, 2.6.1, and 2.6.0, right? Have you tried the fix in those older versions to be sure? If, as I hope, the answer to that is "yes", then I strongly support releasing a fixed setuptools instead of reverting the change to Python. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mal at egenix.com Mon Oct 5 16:32:23 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Oct 2009 16:32:23 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <4ACA0234.1070700@egenix.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <4ACA0234.1070700@egenix.com> Message-ID: <4ACA0377.3070903@egenix.com> M.-A. Lemburg wrote: > Tarek Ziad? wrote: >> Now I am astonished that we are talking about reverting changes in >> Distutils that were done for bugfixes, for a third party package that >> does monkey >> patches on Distutils. >> >> If this choice wins here, it means that setuptools and the stdlib are >> tied together, and that the setuptools package should be integrated to >> the stdlib *immediatly*. > > We've discussed ways of doing that some years ago and found that > it was not possible to make ends join. [just realized that the above could be mistaken for "make ends meet" which has a completely different meaning in English than the German idiom; what I meant was that no consensus could be reached] -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From skip at pobox.com Mon Oct 5 16:40:02 2009 From: skip at pobox.com (skip at pobox.com) Date: Mon, 5 Oct 2009 09:40:02 -0500 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: <19146.1346.257750.667842@montanaro.dyndns.org> Tarek> That's why we have forked and created Distribute, to provide bug Tarek> fixes. I suspect you might need to publicize this a bit better. The first I heard of Distribute or non-responsiveness of PJE regarding setuptools was this thread. (I don't read comp.lang.python anymore. I do read python-dev and comp.lang.python.announce. Maybe I just missed it.) Tarek> Now I am astonished that we are talking about reverting changes Tarek> in Distutils that were done for bugfixes, for a third party Tarek> package that does monkey patches on Distutils. As I said, I was completely unaware of the problems you're addressing with Distribute. My guess is that many extension writers and almost certainly those people who install extensions will be similarly unaware of the issues. Skip From daniel at stutzbachenterprises.com Mon Oct 5 16:50:43 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Mon, 5 Oct 2009 09:50:43 -0500 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: On Mon, Oct 5, 2009 at 9:32 AM, Barry Warsaw wrote: > If, as I hope, the answer to that is "yes", then I strongly support > releasing a fixed setuptools instead of reverting the change to Python. > How do your propose to get the author of setuptools to release a new version? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Mon Oct 5 16:50:46 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 16:50:46 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> On Mon, Oct 5, 2009 at 4:32 PM, Barry Warsaw wrote: > On Oct 5, 2009, at 10:17 AM, Tarek Ziad? wrote: > >>> If setuptools can be made to work with Python 2.6.3 /and/ earlier >>> versions >>> of Python 2.6.x, then it should be patched and an update released. ?If >>> not, >>> then perhaps we should revert the change in a quick Python 2.6.4. >> >> It's technically possible to fix Setuptools. We did this fix on >> Distribute, and >> the patch is 2 lines long. > > My question was less about the political aspects than the technical aspects. > ?I gather you're saying that the fix to setuptools will make it work in > 2.6.3 without inadvertently breaking it for 2.6.2, 2.6.1, and 2.6.0, right? > ?Have you tried the fix in those older versions to be sure? > > If, as I hope, the answer to that is "yes", then I strongly support > releasing a fixed setuptools instead of reverting the change to Python. Yes it does. The fix makes sure build_ext.get_ext_filename still works as it is supposed (just returning the extension of a filename depending on the environment) under all versions, *and* continues to do extra work for the Setuptools internal needs when it is called by Setuptools itself on various places. It's two lines in Setuptools. But beware that a new Setuptools release might take a few months and maybe will never happen. That is why a suggested solution was to install Distribute, because it addresses among other bug fixes, this one. Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From ziade.tarek at gmail.com Mon Oct 5 16:53:41 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 16:53:41 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <4ACA0234.1070700@egenix.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <4ACA0234.1070700@egenix.com> Message-ID: <94bdd2610910050753x5f1eab38w3d24d3d8eacd09b4@mail.gmail.com> On Mon, Oct 5, 2009 at 4:27 PM, M.-A. Lemburg wrote: > Tarek Ziad? wrote: >> Now I am astonished that we are talking about reverting changes in >> Distutils that were done for bugfixes, for a third party package that >> does monkey >> patches on Distutils. >> >> If this choice wins here, it means that setuptools and the stdlib are >> tied together, and that the setuptools package should be integrated to >> the stdlib *immediatly*. > > We've discussed ways of doing that some years ago and found that > it was not possible to make ends join. > > I'd much rather like to see some of the features in setuptools > get implemented in distutils. eGenix could contribute a > bdist_egg implementation that doesn't rely on setuptools and its > monkey patches - along with some other new commands that people > might find useful such as the ability to build Unix libraries, > optional and self-configuring extensions, autoconf-style setups, > etc. (see mxSetup.py in egenix-mx-base for all the details). > > We'd just need some help with integrating the things into > distutils, since we currently don't have the bandwidth for such > things. What about making the Distribute project the laboratory for this work ? It's open to contributions. We are also planning to implement early versions of the latest PEP proposals concerning packaging in this project, once they are finished and accepted, before it goes into Distutils itself. > > Aside: After importing setuptools a lot of things in distutils no > longer work as advertised. This is due to the heavy monkey patching > being applied in setuptools. Yes the code there is very sensitive to any change. Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From ziade.tarek at gmail.com Mon Oct 5 17:02:02 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 17:02:02 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <19146.1346.257750.667842@montanaro.dyndns.org> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <19146.1346.257750.667842@montanaro.dyndns.org> Message-ID: <94bdd2610910050802v72a3c8aeq9698d78d3f5777fa@mail.gmail.com> On Mon, Oct 5, 2009 at 4:40 PM, wrote: > > ? ?Tarek> That's why we have forked and created Distribute, to provide bug > ? ?Tarek> fixes. > > I suspect you might need to publicize this a bit better. ?The first I heard > of Distribute or non-responsiveness of PJE regarding setuptools was this > thread. ?(I don't read comp.lang.python anymore. ?I do read python-dev and > comp.lang.python.announce. ?Maybe I just missed it.) No you didn't miss it. That's probably my fault because the only places I publicize about it are my blog (indexed in planet python) and the distutils-SIG. Maybe I should blog a summary of the situation and post it to python annoucement as well. > > ? ?Tarek> Now I am astonished that we are talking about reverting changes > ? ?Tarek> in Distutils that were done for bugfixes, for a third party > ? ?Tarek> package that does monkey patches on Distutils. > > As I said, I was completely unaware of the problems you're addressing with > Distribute. ?My guess is that many extension writers and almost certainly > those people who install extensions will be similarly unaware of the issues. Right, and an end-user can't be aware of those. But until they depend on a project that is not maintained and that we, active people, can't maintain; we will face such problems. Now I am dedicated to help all extension writers out there, and as a matter of fact I've been doing it since a few weeks on #distutils on freenode, and all the results of this work are contributed in Distribute. I have written down two important things so far from this thread: 1/ the need to provide a builbdot service for packagers that wish to try their package against the latest Distutils 2/ the need to publicize what is going on for a wider audience. Tarek From mal at egenix.com Mon Oct 5 17:11:54 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Oct 2009 17:11:54 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <20091005102547.GA23457@ubuntu.ubuntu-domain> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> Message-ID: <4ACA0CBA.2030200@egenix.com> Senthil Kumaran wrote: > On Mon, Oct 05, 2009 at 10:43:22AM +0200, Fredrik Lundh wrote: >> it's revews like this that makes me wonder if releasing open source is >> a good idea: >> no egg - worst seen ever, remove it from pypi or provide an egg >> (jensens, 2009-10-05, 0 points) >> > > Greetings effbot. :) > As you might already know, "eggs" are not mandatory for pypi. No where > in the documentation it mentions so. Also many good packages in pypi > are not eggs (MaL's egenix packages for e.g) and we have known his > dislike for that particular format too. I don't dislike the format itself, just the tools that are available for using it. We've implemented our own bdist_egg now which doesn't use setuptools and will start to ship eggs in addition to our prebuilt format with the next releases. As time permits, we'll also write an egg installer/uninstaller (one that does only that ;-). > But still, sometimes people have found eggs to be convenient. > > That was a particularly harsh comment from jensens, should be ignored > or he should be corrected that eggs are not mandatory. > > But, i have a feeling that this is going to open up the well known, > can-of-worms on what is the best distribution format for python > packages debate. Let's hope not :-) Besides, the discussions should be about the tools used for package distribution - the egg format itself is fine. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Oct 5 17:26:02 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Oct 2009 17:26:02 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050753x5f1eab38w3d24d3d8eacd09b4@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <4ACA0234.1070700@egenix.com> <94bdd2610910050753x5f1eab38w3d24d3d8eacd09b4@mail.gmail.com> Message-ID: <4ACA100A.8020602@egenix.com> Tarek Ziad? wrote: > On Mon, Oct 5, 2009 at 4:27 PM, M.-A. Lemburg wrote: >> Tarek Ziad? wrote: >>> Now I am astonished that we are talking about reverting changes in >>> Distutils that were done for bugfixes, for a third party package that >>> does monkey >>> patches on Distutils. >>> >>> If this choice wins here, it means that setuptools and the stdlib are >>> tied together, and that the setuptools package should be integrated to >>> the stdlib *immediatly*. >> >> We've discussed ways of doing that some years ago and found that >> it was not possible to make ends join. >> >> I'd much rather like to see some of the features in setuptools >> get implemented in distutils. eGenix could contribute a >> bdist_egg implementation that doesn't rely on setuptools and its >> monkey patches - along with some other new commands that people >> might find useful such as the ability to build Unix libraries, >> optional and self-configuring extensions, autoconf-style setups, >> etc. (see mxSetup.py in egenix-mx-base for all the details). >> >> We'd just need some help with integrating the things into >> distutils, since we currently don't have the bandwidth for such >> things. > > What about making the Distribute project the laboratory for this work ? > It's open to contributions. The way the distutils extensions are implemented (by sub-classing distutils classes) make it easier to add them back to core distutils, rather than to integrate them into another 3rd party distutils extension. I'm not sure how adding them to Distribute would help, since then you'd only be able to access them using Distribute. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Mon Oct 5 17:28:27 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Oct 2009 08:28:27 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> Message-ID: \>> I hate calling methods on string literals, I think it looks very odd >> to have code like this: >> >> ? "Displaying {0} of {1} revisions".format(x, y) > > Ugh! ?Good point. This objection was made years ago when we introduced "separator".join(list_of_things), and I don't think ignoring it has caused any casualties. In fact, it serves as an early reminder to the user that string literals are, in fact, objects like all others. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ziade.tarek at gmail.com Mon Oct 5 17:42:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 17:42:50 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <4ACA100A.8020602@egenix.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <4ACA0234.1070700@egenix.com> <94bdd2610910050753x5f1eab38w3d24d3d8eacd09b4@mail.gmail.com> <4ACA100A.8020602@egenix.com> Message-ID: <94bdd2610910050842p615a92a4je57560c04b2d076d@mail.gmail.com> On Mon, Oct 5, 2009 at 5:26 PM, M.-A. Lemburg wrote: > Tarek Ziad? wrote: >> On Mon, Oct 5, 2009 at 4:27 PM, M.-A. Lemburg wrote: >>> Tarek Ziad? wrote: >>>> Now I am astonished that we are talking about reverting changes in >>>> Distutils that were done for bugfixes, for a third party package that >>>> does monkey >>>> patches on Distutils. >>>> >>>> If this choice wins here, it means that setuptools and the stdlib are >>>> tied together, and that the setuptools package should be integrated to >>>> the stdlib *immediatly*. >>> >>> We've discussed ways of doing that some years ago and found that >>> it was not possible to make ends join. >>> >>> I'd much rather like to see some of the features in setuptools >>> get implemented in distutils. eGenix could contribute a >>> bdist_egg implementation that doesn't rely on setuptools and its >>> monkey patches - along with some other new commands that people >>> might find useful such as the ability to build Unix libraries, >>> optional and self-configuring extensions, autoconf-style setups, >>> etc. (see mxSetup.py in egenix-mx-base for all the details). >>> >>> We'd just need some help with integrating the things into >>> distutils, since we currently don't have the bandwidth for such >>> things. >> >> What about making the Distribute project the laboratory for this work ? >> It's open to contributions. > > The way the distutils extensions are implemented (by sub-classing > distutils classes) make it easier to add them back to core > distutils, rather than to integrate them into another 3rd > party distutils extension. > > I'm not sure how adding them to Distribute would help, since then > you'd only be able to access them using Distribute. I was thinking about the release cycles but this is maybe because I am unaware of the way eGenix packages are developed, so I might be wrong. Distribute has the bandwidth to work on this with many contributors with a smaller release cycle than stdlib, allowing more feedback. Notice that I am also doing nightly builds of Distutils that can be installed and tried in released version of Python, and that can be used instead of the Python's embed Distutils (see the installable tarballs at nightly.ziade.org). so maybe it's just a matter of continuous integration Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From barry at python.org Mon Oct 5 17:44:40 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Oct 2009 11:44:40 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: On Oct 5, 2009, at 10:50 AM, Tarek Ziad? wrote: >> If, as I hope, the answer to that is "yes", then I strongly support >> releasing a fixed setuptools instead of reverting the change to >> Python. > > Yes it does. Excellent, thanks. > The fix makes sure build_ext.get_ext_filename still works as it is > supposed > (just returning the extension of a filename depending on the > environment) under all versions, > > *and* continues to do extra work for the Setuptools internal needs > when > it is called by Setuptools itself on various places. > > It's two lines in Setuptools. > > But beware that a new Setuptools release might take a few months and > maybe will never happen. > That is why a suggested solution was to install Distribute, because it > addresses among other > bug fixes, this one. I think we have two other choices. 2) PJE releases a new version of setuptools that fixes this problem. 3) We (a.k.a. Tarek with our blessing) hijacks the setuptools name (e.g. on cheeseshop) and releases a new version I still hope #2 happens, but let's have a deadline for when the more drastic measures will be taken. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Oct 5 17:45:57 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Oct 2009 11:45:57 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: <2B4FDAB5-7D76-42BD-9352-8A98CB82EC7F@python.org> On Oct 5, 2009, at 10:50 AM, Daniel Stutzbach wrote: > On Mon, Oct 5, 2009 at 9:32 AM, Barry Warsaw wrote: > If, as I hope, the answer to that is "yes", then I strongly support > releasing a fixed setuptools instead of reverting the change to > Python. > > How do your propose to get the author of setuptools to release a new > version? He either will or he won't. Let's give a deadline for when it must happen or "we" will do what we need to do. I think a week to 10 days is reasonable. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From g.brandl at gmx.net Mon Oct 5 18:00:22 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 05 Oct 2009 18:00:22 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> Message-ID: Jesse Noller schrieb: > On Mon, Oct 5, 2009 at 4:43 AM, Fredrik Lundh wrote: >> it's revews like this that makes me wonder if releasing open source is >> a good idea: >> >> no egg - worst seen ever, remove it from pypi or provide an egg >> (jensens, 2009-10-05, 0 points) >> >> > > Unfortunately; we're now staring down the barrel of having > youtube-style comments on Python packages on the index. Yes, unfortunately. I originally thought that restricting the commenters to those with a PyPI account would make them useful, but seeing this one (even if it was not intended) and the comment on hgsvn that belongs into a bug tracker instead, I'm not so sure anymore. Georg From asmodai at in-nomine.org Mon Oct 5 18:29:06 2009 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Mon, 5 Oct 2009 18:29:06 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: <20091005162905.GF7162@nexus.in-nomine.org> -On [20091005 17:45], Barry Warsaw (barry at python.org) wrote: >2) PJE releases a new version of setuptools that fixes this problem. Will all due respect to PJE, but I sincerely doubt this will happen. There's still a bunch of outstanding patches over at the setuptools tracker, including SVN 1.x support, which have been open for more than a year. Personally I consider setuptools unmaintained and have started moving my projects over to Distribute. So, if you decide to wait, make sure to put a deadline on it. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B The earth laughs in flowers... From skip at pobox.com Mon Oct 5 18:36:49 2009 From: skip at pobox.com (skip at pobox.com) Date: Mon, 5 Oct 2009 11:36:49 -0500 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050802v72a3c8aeq9698d78d3f5777fa@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <19146.1346.257750.667842@montanaro.dyndns.org> <94bdd2610910050802v72a3c8aeq9698d78d3f5777fa@mail.gmail.com> Message-ID: <19146.8353.817970.409627@montanaro.dyndns.org> Tarek> No you didn't miss it. That's probably my fault because the only Tarek> places I publicize about it are my blog (indexed in planet Tarek> python) and the distutils-SIG. Bloggers beware!!! Not everyone reads blogs. (I don't unless someone calls my attention to something of particular interest.) Even if everyone did read blogs, the risk of missing a particular post is extremely high considering the number of planet.python.org subscriptions. I don't know how many blogs are aggregated on planet.python.org but a quick scan suggests it's well over 100 at this point. Moral of the story: If you have something to announce, announce it in the proper channel: python-announce-list at python.org. Skip From solipsis at pitrou.net Mon Oct 5 18:44:02 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2009 16:44:02 +0000 (UTC) Subject: [Python-Dev] Package install failures in 2.6.3 References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: Hello, > Now I am astonished that we are talking about reverting changes in > Distutils that were done for bugfixes, for a third party package that > does monkey > patches on Distutils. I think we should avoid any questions of responsability here (besides, it is quite clear that you, Tarek, are not responsible for the pre-existing mess, and the lack of maintenance on certain popular software). Knowing who is responsible doesn't make our users' life any better if nothing's fixed in time. The only question is, given that 2.6.x is supposed to be a bug-fix branch, do we want to fix that incompatibility with a widely deployed existing piece of software? Whether or not the incompatibility is legitimate (i.e., whether setuptools is badly written and relies on implementation details) is beyond the point, IMO. [ The issue, of course, is quite different if we come to talk about trunk. ] > If this choice wins here, it means that setuptools and the stdlib are > tied together, and that the setuptools package should be integrated to > the stdlib *immediatly*. Oh, certainly not. I don't think anybody would want that at this point, even the most hardcore supporters of setuptools :-) Regards Antoine. From ziade.tarek at gmail.com Mon Oct 5 18:53:31 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 18:53:31 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: <94bdd2610910050953p7158aabcx429f535ce7c674b8@mail.gmail.com> On Mon, Oct 5, 2009 at 6:44 PM, Antoine Pitrou wrote: > > The only question is, given that 2.6.x is supposed to be a bug-fix branch, do we > want to fix that incompatibility with a widely deployed existing piece of > software? Whether or not the incompatibility is legitimate (i.e., whether > setuptools is badly written and relies on implementation details) is beyond the > point, IMO. Right. As discussed with Barry, I will work on an internal special case in Distutils in the 2.6 maint. branch so it fixes the incompatibility with the Setuptools 0.6c9 release. Tarek. From stephen at xemacs.org Mon Oct 5 19:13:04 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Oct 2009 02:13:04 +0900 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050802v72a3c8aeq9698d78d3f5777fa@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <19146.1346.257750.667842@montanaro.dyndns.org> <94bdd2610910050802v72a3c8aeq9698d78d3f5777fa@mail.gmail.com> Message-ID: <871vlhvk67.fsf@uwakimon.sk.tsukuba.ac.jp> Tarek Ziad? writes: > Maybe I should blog a summary of the situation and post it to > python annoucement as well. Please don't. It's unlikely to do anything except incite a lot of flames. From mal at egenix.com Mon Oct 5 19:06:22 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Oct 2009 19:06:22 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050953p7158aabcx429f535ce7c674b8@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050953p7158aabcx429f535ce7c674b8@mail.gmail.com> Message-ID: <4ACA278E.9090505@egenix.com> Tarek Ziad? wrote: > On Mon, Oct 5, 2009 at 6:44 PM, Antoine Pitrou wrote: >> >> The only question is, given that 2.6.x is supposed to be a bug-fix branch, do we >> want to fix that incompatibility with a widely deployed existing piece of >> software? Whether or not the incompatibility is legitimate (i.e., whether >> setuptools is badly written and relies on implementation details) is beyond the >> point, IMO. > > Right. As discussed with Barry, I will work on an internal special > case in Distutils in the 2.6 maint. > branch so it fixes the incompatibility with the Setuptools 0.6c9 release. You could add a selective work-around that just triggers if setuptools has been loaded on the 2.6 branch. if sys.modules.has_key('setuptools'): # monkey-patch fix-up setuptools to work with the new code ... or the other way around: if sys.modules.has_key('setuptools'): # undo r1234 to make setuptools 0.6c9 happy again ... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Mon Oct 5 19:08:07 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 5 Oct 2009 19:08:07 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <4ACA278E.9090505@egenix.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050953p7158aabcx429f535ce7c674b8@mail.gmail.com> <4ACA278E.9090505@egenix.com> Message-ID: <94bdd2610910051008r2a4d5733ud3f5be4959c715a7@mail.gmail.com> On Mon, Oct 5, 2009 at 7:06 PM, M.-A. Lemburg wrote: > Tarek Ziad? wrote: >> On Mon, Oct 5, 2009 at 6:44 PM, Antoine Pitrou wrote: >>> >>> The only question is, given that 2.6.x is supposed to be a bug-fix branch, do we >>> want to fix that incompatibility with a widely deployed existing piece of >>> software? Whether or not the incompatibility is legitimate (i.e., whether >>> setuptools is badly written and relies on implementation details) is beyond the >>> point, IMO. >> >> Right. As discussed with Barry, I will work on an internal special >> case in Distutils in the 2.6 maint. >> branch so it fixes the incompatibility with the Setuptools 0.6c9 release. > > You could add a selective work-around that just triggers if setuptools > has been loaded on the 2.6 branch. > > if sys.modules.has_key('setuptools'): > ? # monkey-patch fix-up setuptools to work with the new code > ? ... > > or the other way around: > > if sys.modules.has_key('setuptools'): > ? # undo r1234 to make setuptools 0.6c9 happy again Yes I am working on this. I am putting setuptools specific build_ext command into the distutils tests to reproduce the issue, then will do a specific setuptools fix, > ? ... > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source ?(#1, Oct 05 2009) >>>> Python/Zope Consulting and Support ... ? ? ? ?http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... ? ? ? ? ? ? http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... ? ? ? ?http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > ? eGenix.com Software, Skills and Services GmbH ?Pastor-Loeh-Str.48 > ? ?D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > ? ? ? ? ? Registered at Amtsgericht Duesseldorf: HRB 46611 > ? ? ? ? ? ? ? http://www.egenix.com/company/contact/ > -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From jnoller at gmail.com Mon Oct 5 19:38:06 2009 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 5 Oct 2009 13:38:06 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> Message-ID: <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> On Mon, Oct 5, 2009 at 12:00 PM, Georg Brandl wrote: > Jesse Noller schrieb: >> On Mon, Oct 5, 2009 at 4:43 AM, Fredrik Lundh wrote: >>> it's revews like this that makes me wonder if releasing open source is >>> a good idea: >>> >>> ? no egg - worst seen ever, remove it from pypi or provide an egg >>> (jensens, 2009-10-05, 0 points) >>> >>> >> >> Unfortunately; we're now staring down the barrel of having >> youtube-style comments on Python packages on the index. > > Yes, unfortunately. I originally thought that restricting the commenters > to those with a PyPI account would make them useful, but seeing this one > (even if it was not intended) and the comment on hgsvn that belongs into > a bug tracker instead, I'm not so sure anymore. > > Georg There would need to be a fair amount of work to make the system useful and almost self-policing. Not to mention people can make plenty of fake pypi accounts for pure astroturfing reasons. jesse From jan.matejek at novell.com Mon Oct 5 19:18:14 2009 From: jan.matejek at novell.com (Jan Matejek) Date: Mon, 05 Oct 2009 19:18:14 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: References: Message-ID: <4ACA2A56.7080501@novell.com> Dne 20.9.2009 18:42, Antoine Pitrou napsal(a): > Le Sun, 20 Sep 2009 10:33:23 -0600, Zooko O'Whielacronx a ?crit : >> >> By the way, I was investigating this, and discovered an issue on the >> Mandriva tracker which suggests that they intend to switch to UCS4 in >> the next release in order to avoid compatibility problems like these. > > Trying to use a Fedora or Suse RPM under Mandriva (or the other way > round) isn't reasonable and is certainly not supported. > I don't understand why this so-called "compatibility problem" should be > taken seriously by anyone. You're not making sense. No distro is an island - plus, upstream distributors have a nasty habit of providing RPMs only for Fedora. I don't see what is bad about improving compatibility in a place where the setting doesn't hurt one way or the other. Besides, the more compatibility we achieve now, the easier time we'll have once python makes it into LSB regards m. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jmatejek%40suse.cz From guido at python.org Mon Oct 5 19:54:22 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Oct 2009 10:54:22 -0700 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> Message-ID: On Mon, Oct 5, 2009 at 10:38 AM, Jesse Noller wrote: > On Mon, Oct 5, 2009 at 12:00 PM, Georg Brandl wrote: >> Jesse Noller schrieb: >>> On Mon, Oct 5, 2009 at 4:43 AM, Fredrik Lundh wrote: >>>> it's revews like this that makes me wonder if releasing open source is >>>> a good idea: >>>> >>>> ? no egg - worst seen ever, remove it from pypi or provide an egg >>>> (jensens, 2009-10-05, 0 points) >>>> >>>> >>> >>> Unfortunately; we're now staring down the barrel of having >>> youtube-style comments on Python packages on the index. >> >> Yes, unfortunately. I originally thought that restricting the commenters >> to those with a PyPI account would make them useful, but seeing this one >> (even if it was not intended) and the comment on hgsvn that belongs into >> a bug tracker instead, I'm not so sure anymore. >> >> Georg > > There would need to be a fair amount of work to make the system useful > and almost self-policing. Not to mention people can make plenty of > fake pypi accounts for pure astroturfing reasons. It seems like a worthy cause though. User ratings and comments are the future for "app store" style sites such as PyPI, and spam unfortunately comes with the terrain. There are plenty of things we can learn about fighting spam and other forms of vandalism from other areas of the social web, including our very own wiki, and other wikis (WikiPedia survives despite spam). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at gmail.com Mon Oct 5 19:58:17 2009 From: fdrake at gmail.com (Fred Drake) Date: Mon, 5 Oct 2009 13:58:17 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> Message-ID: <9cee7ab80910051058t36b83bbck2ad366536d83063c@mail.gmail.com> On Mon, Oct 5, 2009 at 1:54 PM, Guido van Rossum wrote: > User ratings and comments are the > future for "app store" style sites such as PyPI Interestingly, I consider sites like PyPI as developer resources rather than the more end-user-centric "App Store" sites. While I don't consider it bad to provide support for commenting and rating, I find it much less valuable than for end-user-centric sites. But then, some may find me a bit retro. :-/ -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From benjamin at python.org Sat Oct 3 23:39:20 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 3 Oct 2009 16:39:20 -0500 Subject: [Python-Dev] % to .format() converter Message-ID: <1afaf6160910031439q544ffc3bw82e3300ce0d03e66@mail.gmail.com> Interested in the recent threads about % and str.format(), I wrote a little Python 3 script that converts % format strings to format strings compatible with str.format(). I invite you to try it out. The code is at https://code.launchpad.net/~gutworth/+junk/mod2format (That means you can do "bzr branch lp:~gutworth/+junk/mod2format".) It should basically work except for some obscure cases like "%#.0f". -- Regards, Benjamin From doug.hellmann at gmail.com Mon Oct 5 13:58:27 2009 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 5 Oct 2009 07:58:27 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <87hbuew70i.fsf@benfinney.id.au> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <87hbuew70i.fsf@benfinney.id.au> Message-ID: <5EB6ECE5-8C3F-4C7E-8922-15EE2B9D4D56@gmail.com> On Oct 5, 2009, at 4:59 AM, Ben Finney wrote: > Fredrik Lundh writes: > >> it's revews like this that makes me wonder if releasing open source >> is >> a good idea: >> >> no egg - worst seen ever, remove it from pypi or provide an egg >> (jensens, 2009-10-05, 0 points) > > Heh. If harsh, uninformed responses make you wonder whether it's worth > doing free software, then you're in the wrong social philanthropy > movement. > > Make the software good quality, release it in the standard format (in > this case, a Distutils ?sdist? on PyPI), respond in a timely manner to > genuine requests for help. Eggs are certainly not a requirement, nor > is > responding to hostility like the above comment. > > If you *want* to respond, you can politely direct them to > where they can learn how to > install Python distributions ? no mention of eggs at all. Package owners are not allowed to comment on their own PyPI packages, so responding to comments requires tracking down the person who left them and contacting them in some other way. Doug From solipsis at pitrou.net Mon Oct 5 20:09:47 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 05 Oct 2009 20:09:47 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: <4ACA2A56.7080501@novell.com> References: <4ACA2A56.7080501@novell.com> Message-ID: <1254766187.6226.41.camel@localhost> Le lundi 05 octobre 2009 ? 19:18 +0200, Jan Matejek a ?crit : > > I > don't see what is bad about improving compatibility in a place where the > setting doesn't hurt one way or the other. I can't speak for Mandriva, but I'm sure they care more about not breaking user installs when they upgrade to Mandriva X + 1, than about making it possible to use Fedora RMPs on Mandriva. In any case, this is quite off-topic for python-dev. If you are motivated about this, you should try to convinve the Mandriva developers instead. Regards Antoine. From jnoller at gmail.com Mon Oct 5 20:21:41 2009 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 5 Oct 2009 14:21:41 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> Message-ID: <4222a8490910051121g95538b8n4f24591dd819d9c0@mail.gmail.com> On Mon, Oct 5, 2009 at 1:54 PM, Guido van Rossum wrote: > On Mon, Oct 5, 2009 at 10:38 AM, Jesse Noller wrote: >> On Mon, Oct 5, 2009 at 12:00 PM, Georg Brandl wrote: >>> Jesse Noller schrieb: >>>> On Mon, Oct 5, 2009 at 4:43 AM, Fredrik Lundh wrote: >>>>> it's revews like this that makes me wonder if releasing open source is >>>>> a good idea: >>>>> >>>>> ? no egg - worst seen ever, remove it from pypi or provide an egg >>>>> (jensens, 2009-10-05, 0 points) >>>>> >>>>> >>>> >>>> Unfortunately; we're now staring down the barrel of having >>>> youtube-style comments on Python packages on the index. >>> >>> Yes, unfortunately. I originally thought that restricting the commenters >>> to those with a PyPI account would make them useful, but seeing this one >>> (even if it was not intended) and the comment on hgsvn that belongs into >>> a bug tracker instead, I'm not so sure anymore. >>> >>> Georg >> >> There would need to be a fair amount of work to make the system useful >> and almost self-policing. Not to mention people can make plenty of >> fake pypi accounts for pure astroturfing reasons. > > It seems like a worthy cause though. User ratings and comments are the > future for "app store" style sites such as PyPI, and spam > unfortunately comes with the terrain. There are plenty of things we > can learn about fighting spam and other forms of vandalism from other > areas of the social web, including our very own wiki, and other wikis > (WikiPedia survives despite spam). > I agree that feedback, commentary/etc is a Good Thing; but doing it right is not an easy thing, and typically implementing it poorly leads to spam, people filing bugs in comments, astroturfing, etc. Just on first glance, I could see immediate improvements around: * Moderation * Allowing authors to respond * Flagging as spam * Upvoting/downvoting * Nested conversations And so on. Sites like stackoverflow/reddit/hackernews/etc have spent a lot of time "doing it right". I know, I know - patches welcome. The problem here is that I would make an argument that in the case of PyPI nothing is better than what we have currently. jesse From solipsis at pitrou.net Mon Oct 5 20:20:58 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2009 18:20:58 +0000 (UTC) Subject: [Python-Dev] eggs now mandatory for pypi? References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> Message-ID: Guido van Rossum python.org> writes: > > There are plenty of things we > can learn about fighting spam and other forms of vandalism from other > areas of the social web, including our very own wiki, and other wikis > (WikiPedia survives despite spam). Doesn't Wikipedia have a lot of human eyes watching, however? From olemis at gmail.com Mon Oct 5 20:38:14 2009 From: olemis at gmail.com (Olemis Lang) Date: Mon, 5 Oct 2009 13:38:14 -0500 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4222a8490910051121g95538b8n4f24591dd819d9c0@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <4222a8490910051121g95538b8n4f24591dd819d9c0@mail.gmail.com> Message-ID: <24ea26600910051138m5c014523gf48b61be7c1f8fc3@mail.gmail.com> On Mon, Oct 5, 2009 at 1:21 PM, Jesse Noller wrote: > On Mon, Oct 5, 2009 at 1:54 PM, Guido van Rossum wrote: >> [...] >> >> User ratings and comments are the >> future for "app store" style sites such as PyPI, and spam >> unfortunately comes with the terrain. There are plenty of things we >> can learn about fighting spam and other forms of vandalism from other >> areas of the social web, including our very own wiki, and other wikis >> (WikiPedia survives despite spam). >> > > I agree that feedback, commentary/etc is a Good Thing; but doing it > right is not an easy thing, and typically implementing it poorly leads > to spam, people filing bugs in comments, astroturfing, etc. Just on > first glance, I could see immediate improvements around: > > * Moderation [...] > * Flagging as spam * Captcha ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html From pje at telecommunity.com Mon Oct 5 20:48:37 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 05 Oct 2009 14:48:37 -0400 Subject: [Python-Dev] "PIL" vs. "Imaging" (was Re: eggs now mandatory for pypi?) In-Reply-To: <1254741989.7368.22.camel@minime> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> <4AC9D356.5050202@cheimes.de> <1254741989.7368.22.camel@minime> Message-ID: <20091005184844.647633A4045@sparrow.telecommunity.com> At 01:26 PM 10/5/2009 +0200, Jens W. Klein wrote: >And as far as I understand PIL is not the >problem, but the packaging/ setuptools. For the records: PIL is a great >piece of software and I dont want to miss it. Actually, the problem is that the package is called "PIL" in Pypi, but the actual installed package is called "Imaging". If you want to depend on PIL, you need to add "Imaging" to your dependencies, and add a dependency link to http://effbot.org/downloads/ in order to make it work... at least on platforms where you have a compiler. If you're working on Windows, you need to depend on PIL in order to get a download to work, but once installed, it's still called Imaging. So, pretty much you're hosed either way. ;-) If the package had just *one* name, used consistently for source, binaries, and PyPI registration, it would work. Unfortunately, setuptools is not a human being and can't figure out that "PIL" and "Imaging" are two different names for the same thing. (I'm guessing that PIL was registered on PyPI manually, before the "setup.py register" command existed. Heck, it was probably being distributed before the distutils even existed, and indeed before there were such things as "packages" in Python.) When I was first writing setuptools, I tried to get it to work with effbot.org packages as best I could, but there were some things I just couldn't do without hardcoding special cases, and working around this problem was one of them. From pje at telecommunity.com Mon Oct 5 20:52:49 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 05 Oct 2009 14:52:49 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.co m> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> Message-ID: <20091005185255.D4EA53A408F@sparrow.telecommunity.com> At 02:22 PM 10/5/2009 +0200, Tarek Ziad? wrote: >Setuptools development has been discontinued for a year, and does >patches on Distutils code. Some of these patches are sensitive to any change >made on Distutils, wether those changes are internal or not. Setuptools is also not the only thing out there that subclasses distutils commands in general, or the build_ext command in particular. Numpy, Twisted, the mx extensions... there are plenty of things out there that subclass distutils commands, quite in adherence to the rules. (Note too that subclassing != patching, and the ability to subclass and substitute distutils commands is documented.) It's therefore not appropriate to treat the issue as if it were setuptools-specific; it could have broken any other major (or minor) package's subclassing of the build_ext command. From pje at telecommunity.com Mon Oct 5 20:55:21 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 05 Oct 2009 14:55:21 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.co m> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> Message-ID: <20091005185528.530E53A408F@sparrow.telecommunity.com> At 04:17 PM 10/5/2009 +0200, Tarek Ziad? wrote: >Now I am astonished that we are talking about reverting changes in >Distutils that were done for bugfixes, for a third party package that >does monkey patches on Distutils. Subclassing is not monkeypatching, so if you consider the above to be a general policy, then the developers of at least Numpy and Twisted should consider themselves warned that bugfix releases may break their build processes. From fredrik at pythonware.com Mon Oct 5 21:03:38 2009 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 5 Oct 2009 21:03:38 +0200 Subject: [Python-Dev] "PIL" vs. "Imaging" (was Re: eggs now mandatory for pypi?) In-Reply-To: <20091005184844.647633A4045@sparrow.telecommunity.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> <4AC9D356.5050202@cheimes.de> <1254741989.7368.22.camel@minime> <20091005184844.647633A4045@sparrow.telecommunity.com> Message-ID: <368a5cd50910051203g7b71140ep6b26e3e66e13a2d@mail.gmail.com> On Mon, Oct 5, 2009 at 8:48 PM, P.J. Eby wrote: > names for the same thing. ?(I'm guessing that PIL was registered on PyPI > manually, before the "setup.py register" command existed. ?Heck, it was > probably being distributed before the distutils even existed, and indeed > before there were such things as "packages" in Python.) $ more PIL/Image.py ... # partial release history: # 1995-09-09 fl Created # 1996-03-11 fl PIL release 0.0 # 1996-04-30 fl PIL release 0.1b1 ... Looks like the quickest fix is to distribute two source packages, one built the traditional way to eliminate breakage for people using the current tarball layout, and one built with "sdist". From brett at python.org Mon Oct 5 20:21:01 2009 From: brett at python.org (Brett Cannon) Date: Mon, 5 Oct 2009 11:21:01 -0700 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: On Mon, Oct 5, 2009 at 08:44, Barry Warsaw wrote: > On Oct 5, 2009, at 10:50 AM, Tarek Ziad? wrote: > > If, as I hope, the answer to that is "yes", then I strongly support >>> releasing a fixed setuptools instead of reverting the change to Python. >>> >> >> Yes it does. >> > > Excellent, thanks. > > The fix makes sure build_ext.get_ext_filename still works as it is >> supposed >> (just returning the extension of a filename depending on the >> environment) under all versions, >> >> *and* continues to do extra work for the Setuptools internal needs when >> it is called by Setuptools itself on various places. >> >> It's two lines in Setuptools. >> >> But beware that a new Setuptools release might take a few months and >> maybe will never happen. >> That is why a suggested solution was to install Distribute, because it >> addresses among other >> bug fixes, this one. >> > > I think we have two other choices. > > 2) PJE releases a new version of setuptools that fixes this problem. > > 3) We (a.k.a. Tarek with our blessing) hijacks the setuptools name (e.g. on > cheeseshop) and releases a new version > > I still hope #2 happens, but let's have a deadline for when the more > drastic measures will be taken. Or we VERY clearly and loudly state in the release notes and anywhere else we can that Distribute has replaced setuptools and everyone should upgrade immediately because of critical bugs like the one under discussion. I should also mention this bug was not unknown. I discovered it after Distribute 0.6 was released as I always run cutting edge interpreters. Never bothered to report it until Distribute 0.6.1 was released which Tarek fixed in less than a week. I never bothered to report it for setuptools as I know it isn't maintained. It's probably in our best interest to just get people over to Distribute, let it continue to hijack setuptools, and slowly let that name fade out if it is going to continue to be unmaintained. I have to admit I find it really disheartening that we are letting an unmaintained project dictate how we fix a bug. I really hope this is a one-time deal and from this point forward we all move the community towards Distribute so we never feel pressured like this again. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Mon Oct 5 21:22:40 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 05 Oct 2009 20:22:40 +0100 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: <4ACA4780.2050404@voidspace.org.uk> Brett Cannon wrote: > > > On Mon, Oct 5, 2009 at 08:44, Barry Warsaw > wrote: > > On Oct 5, 2009, at 10:50 AM, Tarek Ziad? wrote: > > If, as I hope, the answer to that is "yes", then I > strongly support > releasing a fixed setuptools instead of reverting the > change to Python. > > > Yes it does. > > > Excellent, thanks. > > > The fix makes sure build_ext.get_ext_filename still works as > it is supposed > (just returning the extension of a filename depending on the > environment) under all versions, > > *and* continues to do extra work for the Setuptools internal > needs when > it is called by Setuptools itself on various places. > > It's two lines in Setuptools. > > But beware that a new Setuptools release might take a few > months and > maybe will never happen. > That is why a suggested solution was to install Distribute, > because it > addresses among other > bug fixes, this one. > > > I think we have two other choices. > > 2) PJE releases a new version of setuptools that fixes this problem. > > 3) We (a.k.a. Tarek with our blessing) hijacks the setuptools name > (e.g. on cheeseshop) and releases a new version > > I still hope #2 happens, but let's have a deadline for when the > more drastic measures will be taken. > > > Or we VERY clearly and loudly state in the release notes and anywhere > else we can that Distribute has replaced setuptools and everyone > should upgrade immediately because of critical bugs like the one under > discussion. +1 - this sounds like a good solution. I'm intrigued as to whether setuptools does monkey patch distutils or subclass though, they are fundamentally different concepts. Michael > > I should also mention this bug was not unknown. I discovered it after > Distribute 0.6 was released as I always run cutting edge interpreters. > Never bothered to report it until Distribute 0.6.1 was released which > Tarek fixed in less than a week. I never bothered to report it for > setuptools as I know it isn't maintained. > > It's probably in our best interest to just get people over to > Distribute, let it continue to hijack setuptools, and slowly let that > name fade out if it is going to continue to be unmaintained. I have to > admit I find it really disheartening that we are letting an > unmaintained project dictate how we fix a bug. I really hope this is a > one-time deal and from this point forward we all move the community > towards Distribute so we never feel pressured like this again. > > -Brett > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From barry at python.org Mon Oct 5 22:19:50 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Oct 2009 16:19:50 -0400 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <4AC858B9.605@gmail.com> References: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> <4AC858B9.605@gmail.com> Message-ID: <2F47507B-B9FB-4556-8755-8B561ECB293E@python.org> On Oct 4, 2009, at 4:11 AM, Nick Coghlan wrote: > Barry Warsaw wrote: >> I also don't think this is a case of anti-TOOWTDI. For most >> situations >> {}-strings are great (IMO), but in the specific translation domain, I >> suspect $-strings are still better. > > I agree that keeping string.Template around is valid due to its > focus on > being very simple to use (especially for non-coders producing > translation templates) > > However, string.Template is already set up to work for the relevant > APIs > in that domain so I don't think it will really be affected by any > changes to the underlying language and library support for brace > formatting. What if you wanted to translate log messages? Printing to a StringIO first really isn't a very good solution. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From pje at telecommunity.com Mon Oct 5 22:29:52 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 05 Oct 2009 16:29:52 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: <20091005202959.757973A4045@sparrow.telecommunity.com> At 11:21 AM 10/5/2009 -0700, Brett Cannon wrote: >I have to admit I find it really disheartening that we are letting >an unmaintained project dictate how we fix a bug. Setuptools was not the only project broken by this change. See: http://bugs.python.org/issue7020#msg93355 Apparently, I'm not the only person who read the get_ext_filename() docstring that way, and subclassed build_ext in a way that broke it. (In this case, it was pywin32.) IMO, the fact that it broke Pywin32 in 2.6.3rc1 should have been considered evidence that the change was not suitable for a bugfix release. From pje at telecommunity.com Mon Oct 5 22:31:52 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 05 Oct 2009 16:31:52 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <4ACA4780.2050404@voidspace.org.uk> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> <4ACA4780.2050404@voidspace.org.uk> Message-ID: <20091005203159.AB8C43A4045@sparrow.telecommunity.com> At 08:22 PM 10/5/2009 +0100, Michael Foord wrote: >I'm intrigued as to whether setuptools does monkey patch distutils >or subclass though, they are fundamentally different concepts. It does both; however, the specific issue under discussion is purely a subclassing problem, and one which pywin32 ran afoul of as well. From ncoghlan at gmail.com Mon Oct 5 22:26:20 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 06 Oct 2009 06:26:20 +1000 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <5EB6ECE5-8C3F-4C7E-8922-15EE2B9D4D56@gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <87hbuew70i.fsf@benfinney.id.au> <5EB6ECE5-8C3F-4C7E-8922-15EE2B9D4D56@gmail.com> Message-ID: <4ACA566C.4070902@gmail.com> Doug Hellmann wrote: > On Oct 5, 2009, at 4:59 AM, Ben Finney wrote: >> If you *want* to respond, you can politely direct them to >> where they can learn how to >> install Python distributions ? no mention of eggs at all. > > Package owners are not allowed to comment on their own PyPI packages, so > responding to comments requires tracking down the person who left them > and contacting them in some other way. Really? I can understand package owners not being able to add recommendations for their own packages, but if they can't add comments how are they meant to correct misunderstandings or redirect inappropriate messages to the correct forums? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Oct 5 22:24:39 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 06 Oct 2009 06:24:39 +1000 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <9cee7ab80910051058t36b83bbck2ad366536d83063c@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <9cee7ab80910051058t36b83bbck2ad366536d83063c@mail.gmail.com> Message-ID: <4ACA5607.8060100@gmail.com> Fred Drake wrote: > On Mon, Oct 5, 2009 at 1:54 PM, Guido van Rossum wrote: >> User ratings and comments are the >> future for "app store" style sites such as PyPI > > Interestingly, I consider sites like PyPI as developer resources > rather than the more end-user-centric "App Store" sites. > > While I don't consider it bad to provide support for commenting and > rating, I find it much less valuable than for end-user-centric sites. > But then, some may find me a bit retro. :-/ When it comes to comments and recommendations for selecting software packages, developers *are* the end users :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Oct 5 22:41:25 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 06 Oct 2009 06:41:25 +1000 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <2F47507B-B9FB-4556-8755-8B561ECB293E@python.org> References: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> <4AC858B9.605@gmail.com> <2F47507B-B9FB-4556-8755-8B561ECB293E@python.org> Message-ID: <4ACA59F5.1080103@gmail.com> Barry Warsaw wrote: > On Oct 4, 2009, at 4:11 AM, Nick Coghlan wrote: > >> Barry Warsaw wrote: >>> I also don't think this is a case of anti-TOOWTDI. For most situations >>> {}-strings are great (IMO), but in the specific translation domain, I >>> suspect $-strings are still better. >> >> I agree that keeping string.Template around is valid due to its focus on >> being very simple to use (especially for non-coders producing >> translation templates) >> >> However, string.Template is already set up to work for the relevant APIs >> in that domain so I don't think it will really be affected by any >> changes to the underlying language and library support for brace >> formatting. > > What if you wanted to translate log messages? Printing to a StringIO > first really isn't a very good solution. Oh, I see what you meant now - you were pointing out that lazy formatting APIs (such as logging) already don't work properly for alternative formatting mechanisms (such as string.Template). (Although printing to a String IO doesn't seem necessary - simply pre-formatting the message seems like it should work, just as preformatting brace formatted messages already works today). The idea of adding an optional "fmt_style" callable to these APIs (defaulting to None, which would imply the use of the mod operator) is becoming more appealing... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Oct 5 22:20:47 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 06 Oct 2009 06:20:47 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> Message-ID: <4ACA551F.9090309@gmail.com> Guido van Rossum wrote: > \>> I hate calling methods on string literals, I think it looks very odd >>> to have code like this: >>> >>> "Displaying {0} of {1} revisions".format(x, y) >> Ugh! Good point. > > This objection was made years ago when we introduced > "separator".join(list_of_things), and I don't think ignoring it has > caused any casualties. In fact, it serves as an early reminder to the > user that string literals are, in fact, objects like all others. The other string literal method I personally use reasonably often is to create lists of strings by using "a list of strings".split() instead of ['a', 'list', 'of', 'strings'] (while the character count isn't all that different, I find the former is easier to both write and read without all those quotes and commas). It's a trick I've seen plenty of other people use as well. So I would agree that method invocation on literals (particularly string literals) is an already established language idiom. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From foom at fuhm.net Mon Oct 5 22:56:16 2009 From: foom at fuhm.net (James Y Knight) Date: Mon, 5 Oct 2009 16:56:16 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: <8EFA8DDA-BC90-4248-9CF3-42F7CF33301B@fuhm.net> On Oct 5, 2009, at 2:21 PM, Brett Cannon wrote: > I should also mention this bug was not unknown. I discovered it > after Distribute 0.6 was released as I always run cutting edge > interpreters. Never bothered to report it until Distribute 0.6.1 was > released which Tarek fixed in less than a week. I never bothered to > report it for setuptools as I know it isn't maintained. > > It's probably in our best interest to just get people over to > Distribute, let it continue to hijack setuptools, and slowly let > that name fade out if it is going to continue to be unmaintained. I > have to admit I find it really disheartening that we are letting an > unmaintained project dictate how we fix a bug. I really hope this is > a one-time deal and from this point forward we all move the > community towards Distribute so we never feel pressured like this > again. Even though the bug was noticed, nobody thought that, just perhaps, breaking other software in a minor point release might be a bad idea, no matter whether it was updated in less-than-a-week, or mostly- unmaintained? Once you have an API that you encourage people to subclass, *of course* it dictates how you can fix a bug. James From barry at python.org Mon Oct 5 23:01:15 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 5 Oct 2009 17:01:15 -0400 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <4ACA59F5.1080103@gmail.com> References: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> <4AC858B9.605@gmail.com> <2F47507B-B9FB-4556-8755-8B561ECB293E@python.org> <4ACA59F5.1080103@gmail.com> Message-ID: <0B84287E-F693-4399-9C6D-BDFE7F1AA4C5@python.org> On Oct 5, 2009, at 4:41 PM, Nick Coghlan wrote: > Oh, I see what you meant now - you were pointing out that lazy > formatting APIs (such as logging) already don't work properly for > alternative formatting mechanisms (such as string.Template). Yep. > (Although printing to a String IO doesn't seem necessary - simply > pre-formatting the message seems like it should work, just as > preformatting brace formatted messages already works today). Yep. :) > The idea of adding an optional "fmt_style" callable to these APIs > (defaulting to None, which would imply the use of the mod operator) is > becoming more appealing... -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From fdrake at gmail.com Mon Oct 5 23:06:50 2009 From: fdrake at gmail.com (Fred Drake) Date: Mon, 5 Oct 2009 17:06:50 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACA5607.8060100@gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <9cee7ab80910051058t36b83bbck2ad366536d83063c@mail.gmail.com> <4ACA5607.8060100@gmail.com> Message-ID: <9cee7ab80910051406j7397138du834084704e52fdfb@mail.gmail.com> On Mon, Oct 5, 2009 at 4:24 PM, Nick Coghlan wrote: > When it comes to comments and recommendations for selecting software > packages, developers *are* the end users :) Yes, most certainly. But developers as consumers are very different from application users as consumers, which is what I was getting at. The convenience interfaces for commenting on a library are far less valuable for developers, IMO, since developers are expected to better understand how their context impacts their perception. Useful feedback from a developer just doesn't fit will into the giant-pile-of-comments UIs conventional for non-developers. If I'm wrong about that, then I'm saddened by the state of the profession. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From martin at v.loewis.de Mon Oct 5 23:22:22 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 05 Oct 2009 23:22:22 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: <4ACA638E.30609@v.loewis.de> >> My question was less about the political aspects than the technical aspects. >> I gather you're saying that the fix to setuptools will make it work in >> 2.6.3 without inadvertently breaking it for 2.6.2, 2.6.1, and 2.6.0, right? >> Have you tried the fix in those older versions to be sure? >> >> If, as I hope, the answer to that is "yes", then I strongly support >> releasing a fixed setuptools instead of reverting the change to Python. > > Yes it does. > > The fix makes sure build_ext.get_ext_filename still works as it is supposed > (just returning the extension of a filename depending on the > environment) under all versions, I don't think this is factually correct. I assume you talk about distribute change e07e5309cd2a, right? While that change arranges to always return a result when called with an unqualified module name, it doesn't do the additional computation that setuptools does in that case, which may give something different in case the extension is a Library or in case _links_to_dynamic is true. So I would rather expect that distribute is now broken in these cases (which are seemingly rare, though). > It's two lines in Setuptools. But that change may be incorrect. Regards, Martin From olemis at gmail.com Mon Oct 5 23:23:50 2009 From: olemis at gmail.com (Olemis Lang) Date: Mon, 5 Oct 2009 16:23:50 -0500 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <9cee7ab80910051406j7397138du834084704e52fdfb@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <9cee7ab80910051058t36b83bbck2ad366536d83063c@mail.gmail.com> <4ACA5607.8060100@gmail.com> <9cee7ab80910051406j7397138du834084704e52fdfb@mail.gmail.com> Message-ID: <24ea26600910051423y570d91drabcbe7b58364f141@mail.gmail.com> On Mon, Oct 5, 2009 at 4:06 PM, Fred Drake wrote: > On Mon, Oct 5, 2009 at 4:24 PM, Nick Coghlan wrote: >> When it comes to comments and recommendations for selecting software >> packages, developers *are* the end users :) > > Yes, most certainly. ?But developers as consumers are very different > from application users as consumers, which is what I was getting at. > > The convenience interfaces for commenting on a library are far less > valuable for developers, IMO, since developers are expected to better > understand how their context impacts their perception. ?Useful > feedback from a developer just doesn't fit will into the > giant-pile-of-comments UIs conventional for non-developers. > +1 IMO : - decision matrix are useful to decide which lib to use (i.e. which one supports the features I need ;o). BTW that's something cool about wikipedia ;o) - project metrics and build results are useful to have a idea of project dev status (e.g. coverage, test results, ...). - the rest goes to issue tracker + project (sites | wikis). that's what they are for ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html From olemis at gmail.com Mon Oct 5 23:30:53 2009 From: olemis at gmail.com (Olemis Lang) Date: Mon, 5 Oct 2009 16:30:53 -0500 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <24ea26600910051423y570d91drabcbe7b58364f141@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <9cee7ab80910051058t36b83bbck2ad366536d83063c@mail.gmail.com> <4ACA5607.8060100@gmail.com> <9cee7ab80910051406j7397138du834084704e52fdfb@mail.gmail.com> <24ea26600910051423y570d91drabcbe7b58364f141@mail.gmail.com> Message-ID: <24ea26600910051430h3b2d3185lb93401b0a702276f@mail.gmail.com> On Mon, Oct 5, 2009 at 4:23 PM, Olemis Lang wrote: > On Mon, Oct 5, 2009 at 4:06 PM, Fred Drake wrote: >> On Mon, Oct 5, 2009 at 4:24 PM, Nick Coghlan wrote: >>> When it comes to comments and recommendations for selecting software >>> packages, developers *are* the end users :) >> >> Yes, most certainly. ?But developers as consumers are very different >> from application users as consumers, which is what I was getting at. >> >> The convenience interfaces for commenting on a library are far less >> valuable for developers, IMO, since developers are expected to better >> understand how their context impacts their perception. ?Useful >> feedback from a developer just doesn't fit will into the >> giant-pile-of-comments UIs conventional for non-developers. >> > > +1 > > IMO : > > ?- decision matrix are useful to decide which lib to use (i.e. which > ? ?one supports the features I need ;o). BTW that's something cool about > ? ?wikipedia ;o) I mean feature matrix > ?- project metrics and build results are useful to have a idea > ? ?of project dev status (e.g. coverage, test results, ...). > ?- the rest goes to issue tracker + project (sites | wikis). that's what > ? ?they are for ;o) > > -- > Regards, > > Olemis. > > Blog ES: http://simelo-es.blogspot.com/ > Blog EN: http://simelo-en.blogspot.com/ > > Featured article: > Nabble - Trac Users - Coupling trac and symfony framework ?- > http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html > -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html From vinay_sajip at yahoo.co.uk Mon Oct 5 23:46:52 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 5 Oct 2009 21:46:52 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?summary_of_transitioning_from_=25_to_=7B?= =?utf-8?q?=7D_formatting?= References: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> <4AC858B9.605@gmail.com> <2F47507B-B9FB-4556-8755-8B561ECB293E@python.org> <4ACA59F5.1080103@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > Oh, I see what you meant now - you were pointing out that lazy > formatting APIs (such as logging) already don't work properly for > alternative formatting mechanisms (such as string.Template). > Logging doesn't work automatically with string.Template as it pre-dates string.Template, but it can be made to work without too much trouble: (a) Subclass Formatter to use $-formatting instead of %-formatting for e.g. the lines written to log files, and (b) Use a class like DollarMessage which I mentioned elsewhere in this thread, to format logging call format strings and arguments. Both of these approaches will also work for {}-formatting. The present thread really started out with a view to suggesting that the stdlib start adopting {}-format as "native", rather than %-format. Would it be helpful if I added a section to the Python docs about how to use alternative formatting systems with logging? Regards, Vinay Sajip From benjamin at python.org Mon Oct 5 04:07:52 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 4 Oct 2009 21:07:52 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <87y6nt37kb.fsf@hbox.dyndns.org> <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> <1afaf6160910041854v55edcc96we4fb2d6a7f2da948@mail.gmail.com> Message-ID: <1afaf6160910041907m7dc6b82ftaa08506cad9c67c2@mail.gmail.com> 2009/10/4 INADA Naoki : >> -1 That requires keeping formatting information around in every string instance. > Adding new "format_string" class avoids it. > ?unicode("foo") <=> u"foo" > ?format_string("foo") <=> f"foo" > > This way's pros: > * Many libraries can use one transition way. > * Transition stage syncs to Python version. "library A uses {} and > library B uses %" problem not happen in transition. > * We have experience same way on unicode. And all are outweighted by the introduction of a new string class. -- Regards, Benjamin From martin at v.loewis.de Tue Oct 6 00:10:25 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 06 Oct 2009 00:10:25 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <24ea26600910051138m5c014523gf48b61be7c1f8fc3@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <4222a8490910051121g95538b8n4f24591dd819d9c0@mail.gmail.com> <24ea26600910051138m5c014523gf48b61be7c1f8fc3@mail.gmail.com> Message-ID: <4ACA6ED1.5020302@v.loewis.de> >> * Moderation > [...] >> * Flagging as spam > > * Captcha ? In the specific case, neither would have helped. a) the user making the comment that the package author felt to be impolite was a real user, with an established, well-known identity, and long-time member of the community (IIUC). That didn't stop him from making comments that the package author considered as impolite. b) the comment itself was not spam, by any convention that I'm familiar with. It was negative, but it was on-topic (in that it commented on the very package that it was attached to). I have said this already in various other places: I feel that the freedom of speech is of high value, and that PyPI is not only (and perhaps not primarily) made for the package authors - but for the package users. If we get spam, we certainly need (and have) moderation. But in the specific case, I probably would not have moderated the comment away since it was not spam. Regards, Martin From benjamin at python.org Mon Oct 5 22:57:15 2009 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 5 Oct 2009 15:57:15 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACA551F.9090309@gmail.com> References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> <4ACA551F.9090309@gmail.com> Message-ID: <1afaf6160910051357t32301c3aq593ac1745c807290@mail.gmail.com> 2009/10/5 Nick Coghlan : > So I would agree that method invocation on literals (particularly string > literals) is an already established language idiom. And who hasn't ever used 4.56.as_integer_ratio()? :) -- Regards, Benjamin From fuzzyman at voidspace.org.uk Tue Oct 6 00:23:39 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 05 Oct 2009 23:23:39 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <1afaf6160910051357t32301c3aq593ac1745c807290@mail.gmail.com> References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> <4ACA551F.9090309@gmail.com> <1afaf6160910051357t32301c3aq593ac1745c807290@mail.gmail.com> Message-ID: <4ACA71EB.4050003@voidspace.org.uk> Benjamin Peterson wrote: > 2009/10/5 Nick Coghlan : > >> So I would agree that method invocation on literals (particularly string >> literals) is an already established language idiom. >> > > And who hasn't ever used 4.56.as_integer_ratio()? :) > > > > I've tried 4.__add__ a few times (not for a while now though). Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From ncoghlan at gmail.com Sun Oct 4 10:11:37 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Oct 2009 18:11:37 +1000 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> References: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> Message-ID: <4AC858B9.605@gmail.com> Barry Warsaw wrote: > I also don't think this is a case of anti-TOOWTDI. For most situations > {}-strings are great (IMO), but in the specific translation domain, I > suspect $-strings are still better. I agree that keeping string.Template around is valid due to its focus on being very simple to use (especially for non-coders producing translation templates) However, string.Template is already set up to work for the relevant APIs in that domain so I don't think it will really be affected by any changes to the underlying language and library support for brace formatting. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Oct 4 09:56:20 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Oct 2009 17:56:20 +1000 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> References: <4AC2846E.8060304@g.nevcal.com> <4AC29F71.3000307@gmail.com> <4AC6DE09.3030003@gmail.com> <9d153b7c0910030345v5a062b08h5092fe0f2ca3e8b7@mail.gmail.com> <4AC76B10.1000209@voidspace.org.uk> <4AC77A17.7050101@voidspace.org.uk> <9d153b7c0910030938k66310859h510f67237e319ee5@mail.gmail.com> Message-ID: <4AC85524.1030401@gmail.com> Yuvgoog Greenle wrote: > I know this might come through as bike shedding but it's just > customary python that every module have it's own exception types as to > not mix them up with others. Not in my Python world it isn't. While that is sometimes the right answer, more often the right answer is to properly scope your try/except statement in order to avoid catching exceptions from code you aren't interested in. If you end up stuck with a library call that lacks both granularity and specificity of exceptions then you live with it (and maybe file a bug report/feature request with the library author). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Tue Oct 6 06:43:55 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Oct 2009 13:43:55 +0900 Subject: [Python-Dev] comments vs spam in PyPI [was: eggs now mandatory for pypi?] In-Reply-To: References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> Message-ID: <87ws39t9mc.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > Guido van Rossum python.org> writes: > > > > There are plenty of things we > > can learn about fighting spam and other forms of vandalism from other > > areas of the social web, including our very own wiki, and other wikis > > (WikiPedia survives despite spam). > > Doesn't Wikipedia have a lot of human eyes watching, however? Yes. In fact Wikipedia's real issue is not spam, but edit wars. It just happens that the same people who are willing to watch to make sure that nobody "corrects" the facts consider spam damage, too, and they get rid of it as part of their mission. What this means is that the most active parts of the Wikipedia are also quickly policed. What's left is much less attractive for spammers, and there are a number of volunteers willing to respond fairly promptly to reports of spam in articles that nobody currently considers "theirs". I think it could probably be adapted to Python community scale, but it would probably require new mechanisms (spam reporting and possibly cleaning -- in Wikipedia you hit the revert button, choose a known good version, and you're done) in PyPI, and recruitment of volunteers to take care of spam to products currently not "owned". IMO it would be better to design developer-specific mechanisms rather than a generic commenting vehicle, cf. Fred Drake's thinking. From pje at telecommunity.com Tue Oct 6 06:47:41 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 00:47:41 -0400 Subject: [Python-Dev] comments vs spam in PyPI [was: eggs now mandatory for pypi?] In-Reply-To: <87ws39t9mc.fsf@uwakimon.sk.tsukuba.ac.jp> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <87ws39t9mc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20091006044752.A4C4C3A4045@sparrow.telecommunity.com> At 01:43 PM 10/6/2009 +0900, Stephen J. Turnbull wrote: >IMO it would be better to design developer-specific mechanisms rather >than a generic commenting vehicle, cf. Fred Drake's thinking. For example, having a packages reddit (nb: open source, written in Python), where people can upvote or downvote packages and leave comments. That's probably the minimum amount of checks and balances required to avoid problems of the sort the PyPI commenting feature is already having, since others will be able to reply to the comments, and downvote nonsense into oblivion. (Alternatively, shutting off the comment system would also work. Nothing stops people from using Google to search for "foo sucks" or "problems using foo" if they want to research what's been said about a package.) From vinay_sajip at yahoo.co.uk Tue Oct 6 08:09:55 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 6 Oct 2009 06:09:55 +0000 (UTC) Subject: [Python-Dev] comments vs spam in PyPI [was: eggs now mandatory for pypi?] References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <4222a8490910050613w6f631a54m2cecd2a6c7d9d889@mail.gmail.com> <4222a8490910051038u779e8f71s177c7f17ba92abf1@mail.gmail.com> <87ws39t9mc.fsf@uwakimon.sk.tsukuba.ac.jp> <20091006044752.A4C4C3A4045@sparrow.telecommunity.com> Message-ID: P.J. Eby telecommunity.com> writes: > For example, having a packages reddit (nb: open source, written in > Python), where people can upvote or downvote packages and leave > comments. That's probably the minimum amount of checks and balances > required to avoid problems of the sort the PyPI commenting feature is > already having, since others will be able to reply to the comments, > and downvote nonsense into oblivion. Seems like a reasonable idea, given that there's already a Python reddit which seems popular. There was also www.cheeserater.com (which appears down at the moment) which was a Django demo site (source available at http://github.com/jacobian/cheeserater) for rating PyPI packages. I just created a "cheeseshop" sub-reddit to play around with. > (Alternatively, shutting off the comment system would also > work. Nothing stops people from using Google to search for "foo > sucks" or "problems using foo" if they want to research what's been > said about a package.) Yes, but it's good to have all the feedback in one place, if possible. Regards, Vinay Sajip From ncoghlan at gmail.com Tue Oct 6 12:32:10 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 06 Oct 2009 20:32:10 +1000 Subject: [Python-Dev] summary of transitioning from % to {} formatting In-Reply-To: References: <05DB5F3D-A69B-4F4A-A24C-5216666A50B3@python.org> <4AC858B9.605@gmail.com> <2F47507B-B9FB-4556-8755-8B561ECB293E@python.org> <4ACA59F5.1080103@gmail.com> Message-ID: <4ACB1CAA.2000009@gmail.com> Vinay Sajip wrote: > Both of these approaches will also work for {}-formatting. The present thread > really started out with a view to suggesting that the stdlib start adopting > {}-format as "native", rather than %-format. > > Would it be helpful if I added a section to the Python docs about how to use > alternative formatting systems with logging? That would probably be a good idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From bheemesh at gmail.com Tue Oct 6 12:19:32 2009 From: bheemesh at gmail.com (bheemesh v) Date: Tue, 6 Oct 2009 15:49:32 +0530 Subject: [Python-Dev] Help needed for tar archive creation with changed UID/GID Message-ID: <670534840910060319t4af51e38r5fca7e11f4f4d8f8@mail.gmail.com> Hello, I am a newbie to this mailing list, i have seen in some mail discussions about a tar archive creation by forcibly setting the UID/GID to a specific user (say root). Te code mentioned goes like this: tar = tarfile.open("foo.tar.gz", "w:gz") for filename in filenames: tarinfo = tar.gettarinfo(filename) if tarinfo.isreg(): fobj = open(filename) else: fobj = None tarinfo.uid = 0 tarinfo.gid = 0 tar.addfile(tarinfo, fobj) tar.close() But this is not really working for me as when i am trying to un-tar or un-compress contents to a new directory and contents still have diff GID & UID in the long listing of directory. Can somebody suggest an alternative or a working case if already changes are done here. Thanks in advance. Best regards, Bheemesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Oct 6 12:41:30 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 06 Oct 2009 12:41:30 +0200 Subject: [Python-Dev] Help needed for tar archive creation with changed UID/GID In-Reply-To: <670534840910060319t4af51e38r5fca7e11f4f4d8f8@mail.gmail.com> References: <670534840910060319t4af51e38r5fca7e11f4f4d8f8@mail.gmail.com> Message-ID: bheemesh v wrote: > I am a newbie to this mailing list, i have seen in some mail discussions > about a tar archive creation by forcibly setting the UID/GID to a specific > user (say root). Note that this is the mailing list about core development of the CPython interpreter, not about general Python development. Please refer your request to python-list at python.org or tutor at python.org instead. Stefan From fuzzyman at voidspace.org.uk Tue Oct 6 12:43:34 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 06 Oct 2009 11:43:34 +0100 Subject: [Python-Dev] Help needed for tar archive creation with changed UID/GID In-Reply-To: <670534840910060319t4af51e38r5fca7e11f4f4d8f8@mail.gmail.com> References: <670534840910060319t4af51e38r5fca7e11f4f4d8f8@mail.gmail.com> Message-ID: <4ACB1F56.8020008@voidspace.org.uk> bheemesh v wrote: > Hello, > > I am a newbie to this mailing list, i have seen in some mail > discussions about a tar archive creation by forcibly setting the > UID/GID to a specific user (say root). Hello Bheemesh, This mailing list is for the development of Python, not for developing *with* Python. There are other more appropriate mailing lists where you will hopefully get helpful answers: The Python tutor list http://mail.python.org/mailman/listinfo/tutor The comp.lang.python newsgroup The email gateway: http://mail.python.org/mailman/listinfo/python-list A google groups interface: http://groups.google.com/group/comp.lang.python/topics?lnk=li&hl=en&pli=1 All the best, Michael Foord > > Te code mentioned goes like this: > tar = tarfile.open("foo.tar.gz", "w:gz") > for filename in filenames: > tarinfo = tar.gettarinfo(filename) > if tarinfo.isreg(): > fobj = open(filename) > else: > fobj = None > > > tarinfo.uid = 0 > tarinfo.gid = 0 > tar.addfile(tarinfo, fobj) > tar.close() > > But this is not really working for me as when i am trying to un-tar or > un-compress contents to a new directory and contents still have diff > GID & UID in the long listing of directory. > > Can somebody suggest an alternative or a working case if already > changes are done here. > > Thanks in advance. > > Best regards, > Bheemesh > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From cournape at gmail.com Tue Oct 6 12:55:56 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2009 19:55:56 +0900 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <20091005185255.D4EA53A408F@sparrow.telecommunity.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <20091005185255.D4EA53A408F@sparrow.telecommunity.com> Message-ID: <5b8d13220910060355u1c3081ag1c80874b1aba715@mail.gmail.com> 2009/10/6 P.J. Eby : > At 02:22 PM 10/5/2009 +0200, Tarek Ziad? wrote: >> >> Setuptools development has been discontinued for a year, and does >> patches on Distutils code. Some of these patches are sensitive to any >> change >> made on Distutils, wether those changes are internal or not. > > Setuptools is ?also not the only thing out there that subclasses distutils > commands in general, or the build_ext command in particular. ?Numpy, > Twisted, the mx extensions... ?there are plenty of things out there that > subclass distutils commands, quite in adherence to the rules. ?(Note too > that subclassing != patching, and the ability to subclass and substitute > distutils commands is documented.) > > It's therefore not appropriate to treat the issue as if it were > setuptools-specific; it could have broken any other major (or minor) > package's subclassing of the build_ext command. The internal vs published API difference does not make much sense in distutils case anyway, since a lot of implementation details are necessary to make non trivial extension work. When working on numpy.distutils, I almost always have to look at distutils sources since the docs are vastly insufficient, and even then, the code is so bad that quite often the only way to interact with distutils is to "reverse engineer" its behavior by trial and error. cheers, David From chris at simplistix.co.uk Tue Oct 6 15:00:24 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:00:24 +0100 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACA566C.4070902@gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <87hbuew70i.fsf@benfinney.id.au> <5EB6ECE5-8C3F-4C7E-8922-15EE2B9D4D56@gmail.com> <4ACA566C.4070902@gmail.com> Message-ID: <4ACB3F68.4030900@simplistix.co.uk> Nick Coghlan wrote: > Really? I can understand package owners not being able to add > recommendations for their own packages, but if they can't add comments > how are they meant to correct misunderstandings or redirect > inappropriate messages to the correct forums? Indeed, yet another reason I would like to have the option to disable comments for my packages. I make sure there are advertised mailing lists available for all my packages, and they are all much for featureful than PyPI should ever hope to be... https://sourceforge.net/tracker/?func=detail&aid=2872293&group_id=66150&atid=513503 cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 6 15:01:52 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:01:52 +0100 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACA0CBA.2030200@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> Message-ID: <4ACB3FC0.1010906@simplistix.co.uk> M.-A. Lemburg wrote: > We've implemented our own bdist_egg now which doesn't use setuptools > and will start to ship eggs in addition to our prebuilt format with > the next releases. Egg-cellent ;-) Any chance this tool is open source? Better yet, could it make its way into distutils asap? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 6 15:02:38 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:02:38 +0100 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> Message-ID: <4ACB3FEE.9060708@simplistix.co.uk> Fredrik Lundh wrote: > Oh, it was just yet another Zope developer behaving like an ass. Why > am I not surprised? Actually Plohn, there aren't that many Zope developers left ;-) Chris - looking mournfully at his sig... -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 6 15:06:24 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:06:24 +0100 Subject: [Python-Dev] "PIL" vs. "Imaging" (was Re: eggs now mandatory for pypi?) In-Reply-To: <368a5cd50910051203g7b71140ep6b26e3e66e13a2d@mail.gmail.com> References: <368a5cd50910050143n7735652ftcc202363d9878e2c@mail.gmail.com> <368a5cd50910050158gca67163i5ba7e993269c7948@mail.gmail.com> <4AC9D356.5050202@cheimes.de> <1254741989.7368.22.camel@minime> <20091005184844.647633A4045@sparrow.telecommunity.com> <368a5cd50910051203g7b71140ep6b26e3e66e13a2d@mail.gmail.com> Message-ID: <4ACB40D0.3000006@simplistix.co.uk> Fredrik Lundh wrote: > Looks like the quickest fix is to distribute two source packages, one > built the traditional way to eliminate breakage for people using the > current tarball layout, and one built with "sdist". Why the need for two? Or, asked differently, who would be hurt if the distribution name became PIL? (distribution name != package names in distribution) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 6 15:17:51 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:17:51 +0100 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <94bdd2610910050842p615a92a4je57560c04b2d076d@mail.gmail.com> References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <4ACA0234.1070700@egenix.com> <94bdd2610910050753x5f1eab38w3d24d3d8eacd09b4@mail.gmail.com> <4ACA100A.8020602@egenix.com> <94bdd2610910050842p615a92a4je57560c04b2d076d@mail.gmail.com> Message-ID: <4ACB437F.9060602@simplistix.co.uk> Tarek Ziad? wrote: > Notice that I am also doing nightly builds of Distutils that can be installed > and tried in released version of Python, and that can be used instead of the > Python's embed Distutils (see the installable tarballs at nightly.ziade.org). > so maybe it's just a matter of continuous integration Could these be released as normal sdist packages on PyPI? That would seem the best way to get them used... (bonus points for working with buildout,virtualenv,pip,etc) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From mal at egenix.com Tue Oct 6 15:18:22 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Oct 2009 15:18:22 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB3FC0.1010906@simplistix.co.uk> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> Message-ID: <4ACB439E.6050102@egenix.com> Chris Withers wrote: > M.-A. Lemburg wrote: >> We've implemented our own bdist_egg now which doesn't use setuptools >> and will start to ship eggs in addition to our prebuilt format with >> the next releases. > > Egg-cellent ;-) > > Any chance this tool is open source? Better yet, could it make its way > into distutils asap? mxSetup.py, the module implementing all our distutils extensions, is available in egenix-mx-base which is open source: http://www.egenix.com/products/python/mxBase/ Here's current snapshot: http://downloads.egenix.com/python/egenix-mx-base-3.2.0_20091006.zip or as egg http://downloads.egenix.com/python/egenix_mx_base-3.2.0_20091006-py2.6-linux-x86_64.egg Note how the package name is mangled... easy_install requires this. The file name also doesn't tell you that the above is for a UCS2 Python build. Again, easy_install fails with that information added to the "py2.6" version marker. We're still tweaking the implementation a bit, but it's stabilizing rather quickly. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From chris at simplistix.co.uk Tue Oct 6 15:19:33 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:19:33 +0100 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: <94bdd2610910050522u5e7e7dfawcf44f23b4b786b69@mail.gmail.com> <94bdd2610910050717m388d8e86y868f2126781046fc@mail.gmail.com> <94bdd2610910050750g38613d83jb3978f7b23e2aec5@mail.gmail.com> Message-ID: <4ACB43E5.6030003@simplistix.co.uk> Barry Warsaw wrote: > 2) PJE releases a new version of setuptools that fixes this problem. > > 3) We (a.k.a. Tarek with our blessing) hijacks the setuptools name (e.g. > on cheeseshop) and releases a new version It's a shame you didn't suggest this sooner. It would have avoided the need for the seperate Distribute project and all the horrible hacks it entails for every package that uses setuptools... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 6 15:28:26 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:28:26 +0100 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB439E.6050102@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> Message-ID: <4ACB45FA.3080707@simplistix.co.uk> M.-A. Lemburg wrote: > mxSetup.py, the module implementing all our distutils extensions, > is available in egenix-mx-base which is open source: > > http://www.egenix.com/products/python/mxBase/ I have memories of mxBase having a load of other stuff in it too? Would it be possible to split just the distutils extensions into their own package and get that package as an sdist on PyPI? That said, I still think getting the bdist_egg code, at least, into Distutils in the core as soon as possible would be a very good thing. It's kinda odd having the python equivalent of jars or gems only being created by third party software that's no longer maintained... > Note how the package name is mangled... easy_install requires this. > The file name also doesn't tell you that the above is for > a UCS2 Python build. Again, easy_install fails with that information > added to the "py2.6" version marker. Surely this stuff all belongs in the static metadata file that keeps on overcomplicating itself in discussions? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 6 15:45:31 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 14:45:31 +0100 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4AC371D7.9000904@simplistix.co.uk> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4AB8D910.8050601@simplistix.co.uk> <94bdd2610909221106h868a158yb29bdd77f96eac9@mail.gmail.com> <4AB92ADE.9060700@simplistix.co.uk> <87vdjauoq6.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> Message-ID: <4ACB49FB.3000407@simplistix.co.uk> Hi Phil, It's almost a week since I made this offer. I haven't heard anything from you. If I've missed anything please let me know and I'll track it down, otherwise I hope you can have a look at this some time soon. cheers, Chris Chris Withers wrote: > P.J. Eby wrote: >> Here's what actually happened, if anyone cares. Tarek and friends >> announced a fork of setuptools. I reviewed the work and saw that -- >> for the most part -- I was happy with it, and opined as how I might be >> willing to bless the the "package inquisition" team as official >> maintainers of the 0.6 branch of setuptools, so that I could work on >> the fun bits I've long planned for 0.7, but never felt free to start >> on while there was so much still needing to be done on 0.6. > > If this offer is still available, I'd lake to take you up on it. > I'd be more than willing to merge changes on the 0.6 distribute branch > back into the setuptools codebase and do whatever is needed to get a new > setuptools release out. > > Why? Because there are a *lot* of copies of ez_setup.py and buildout's > bootstrap.py that will need replacing if it doesn't happen. I think it'd > be better for the python community all round if setuptools just > continued in maintenance mode until whatever-ends-up-in-the-core exists > and people want to use... > > I'm happy to submit to whatever supervision is needed for you to trust > me to do this, and I promise to be as careful as I can with this. I > *know* how important this is and want to make it work... > >> All I want is for good stuff to happen for setuptools users and Python >> users in general, so I don't think all the suspicion and backbiting is >> merited. > > Fine, if that's true, I apologize (even spelled correctly!) for any > previous involvement in this, but please help me help you achieve your > aims... > > To put this into a way that makes sense to me: I'm volunteering to keep > distribute 0.6 and setuptools 0.6 in sync, no more, no less, and try and > keep that as uncontroversial as possible, and get setuptools 0.6 > releases out to match distribute 0.6 releases as soon as I can. > > Again, I feel I need to stress that the *only* reason I want to do this > is to bring the benefits of the distribute work to the existing > setuptools codebase, with appropriate attribution if that makes a > difference. > > Apologies if any of this is offensive to anyone. For once (really!) I > really mean that :-) > > cheers, > > Chris > -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ronaldoussoren at mac.com Tue Oct 6 15:49:23 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 06 Oct 2009 15:49:23 +0200 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? In-Reply-To: References: <92839421634652657618189239787554831247-Webmail@me.com> <2ECCFBA0-1A70-4D12-925C-CE620D430AAD@gmail.com> Message-ID: On 3 Oct, 2009, at 1:40, INADA Naoki wrote: >> Confirmed on 10.6 for 2.6.3 and 2.7a0. For 3.2a0 both asserts fail. > > OK. > `s = '\xa0'` should be `s = b'\xa0'`. > > Should I file a bug? Please do, that helps us remember that the issue exists. Ronald > > -- > Naoki INADA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From arcriley at gmail.com Tue Oct 6 15:54:07 2009 From: arcriley at gmail.com (Arc Riley) Date: Tue, 6 Oct 2009 09:54:07 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB45FA.3080707@simplistix.co.uk> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> Message-ID: I'll make the argument that feedback is useful, comments are much less so and a lot more work. It would be more useful to allow package users post feedback, visible only to the package maintainer, and also add support for bugtracker links/etc. Is the intention of Pypi really to turn it into a social networking site? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Tue Oct 6 16:00:29 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 6 Oct 2009 16:00:29 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACA0CBA.2030200@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> Message-ID: <94bdd2610910060700n4d948c3boc716a7979ac22bd5@mail.gmail.com> On Mon, Oct 5, 2009 at 5:11 PM, M.-A. Lemburg wrote: > Senthil Kumaran wrote: >> On Mon, Oct 05, 2009 at 10:43:22AM +0200, Fredrik Lundh wrote: >>> it's revews like this that makes me wonder if releasing open source is >>> a good idea: >>> ? ?no egg - worst seen ever, remove it from pypi or provide an egg >>> (jensens, 2009-10-05, 0 points) >>> >> >> Greetings effbot. :) >> As you might already know, "eggs" are not mandatory for pypi. No where >> in the documentation it mentions so. Also many good packages in pypi >> are not eggs (MaL's egenix packages for e.g) and we have known his >> dislike for that particular format too. > > I don't dislike the format itself, just the tools that are > available for using it. > > We've implemented our own bdist_egg now which doesn't use setuptools > and will start to ship eggs in addition to our prebuilt format with > the next releases. > > As time permits, we'll also write an egg installer/uninstaller > (one that does only that ;-). > >> But still, sometimes people have found eggs to be convenient. The idea of PEP 376 we are working on is to have a unique unified standard for distributions installed in site-packages, instead of the current situation where we have many installation formats. (Distutils pre-Python 2.5, Distutils post-Python2.5, Setuptools self-contained Eggs, ..) because it makes it simple to provide APIs to query what's installed, and ways to uninstall things. That's what I would call the "Missing APIs" to make Distutils a package manager, or at least a basis for package managers. Having an egg installer/uninstaller that works with Setuptools-like self-contained eggs sounds like a package manager system on its own and yet another format. But as far as we went in the discussions for PEP 376, it seemed like a first step would be to work with a single installation format and provides all APIs for it, then to think about having other formats and ways to query them in a second move. That's why I don't think its the right time to add a bdist_egg command in Distutils, until PEP 376 is fully defined (and maybe extended to support several format). Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From songofacandy at gmail.com Tue Oct 6 16:32:32 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 6 Oct 2009 23:32:32 +0900 Subject: [Python-Dev] Old libc's isspace() bug on FreeBSD brings back on Mac OS X? In-Reply-To: References: <92839421634652657618189239787554831247-Webmail@me.com> <2ECCFBA0-1A70-4D12-925C-CE620D430AAD@gmail.com> Message-ID: I filed as issue7072. On Tue, Oct 6, 2009 at 10:49 PM, Ronald Oussoren wrote: > > On 3 Oct, 2009, at 1:40, INADA Naoki wrote: > >>> Confirmed on 10.6 for 2.6.3 and 2.7a0. For 3.2a0 both asserts fail. >> >> OK. >> `s = '\xa0'` should be `s = b'\xa0'`. >> >> Should I file a bug? > > Please do, that helps us remember that the issue exists. > > Ronald > >> >> -- >> Naoki INADA ? >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > -- Naoki INADA From mal at egenix.com Tue Oct 6 16:39:12 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Oct 2009 16:39:12 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB45FA.3080707@simplistix.co.uk> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> Message-ID: <4ACB5690.1080208@egenix.com> Chris Withers wrote: > M.-A. Lemburg wrote: >> mxSetup.py, the module implementing all our distutils extensions, >> is available in egenix-mx-base which is open source: >> >> http://www.egenix.com/products/python/mxBase/ > > I have memories of mxBase having a load of other stuff in it too? Yep, lots of it. > Would it be possible to split just the distutils extensions into their > own package and get that package as an sdist on PyPI? Nope. The main reason for having switched to distributions is to make things easier to manage - both for us and our users. > That said, I still think getting the bdist_egg code, at least, into > Distutils in the core as soon as possible would be a very good thing. > It's kinda odd having the python equivalent of jars or gems only being > created by third party software that's no longer maintained... > >> Note how the package name is mangled... easy_install requires this. >> The file name also doesn't tell you that the above is for >> a UCS2 Python build. Again, easy_install fails with that information >> added to the "py2.6" version marker. > > Surely this stuff all belongs in the static metadata file that keeps on > overcomplicating itself in discussions? The complicated stuff does belong somewhere else, but the basic things need to go into the filename - simply because you have to build separate files for each Python build dimension and want to be able to put everything into a single directory or index: * Python version * Architecture/Platform * UCS2/UCS4 E.g. to solve the UCS2/4 part, we had to setup separate PyPI-style index sites. We'd really like to avoid that if possible. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From chris at simplistix.co.uk Tue Oct 6 17:45:29 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Oct 2009 16:45:29 +0100 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB5690.1080208@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> Message-ID: <4ACB6619.4080807@simplistix.co.uk> M.-A. Lemburg wrote: >> Would it be possible to split just the distutils extensions into their >> own package and get that package as an sdist on PyPI? > > Nope. shame :-( > The complicated stuff does belong somewhere else, but the > basic things need to go into the filename But everyone's basic things are different. > - simply because > you have to build separate files for each Python build > dimension and want to be able to put everything into > a single directory or index: > > * Python version > * Architecture/Platform > * UCS2/UCS4 > > E.g. to solve the UCS2/4 part, we had to setup separate > PyPI-style index sites. We'd really like to avoid that if > possible. Well yeah, and the only sane way I can think to handle this is to have a metadata file that gets uploaded with each distribution that covers all these things (and the other things that other people need) and then have the index (which would hopefully be PyPI on the whole) be queryable along the lines of "give me a download url for a distribution named X that is for Python 2.6, Win32, UCS4, PostGreSQL 8.5, BlagBlah"... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From mal at egenix.com Tue Oct 6 18:03:08 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Oct 2009 18:03:08 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB6619.4080807@simplistix.co.uk> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> Message-ID: <4ACB6A3C.6000909@egenix.com> Chris Withers wrote: > M.-A. Lemburg wrote: >> The complicated stuff does belong somewhere else, but the >> basic things need to go into the filename > > But everyone's basic things are different. The really basic stuff is the same for everyone since it's dictated by the way Python works on computers :-) >> - simply because >> you have to build separate files for each Python build >> dimension and want to be able to put everything into >> a single directory or index: >> >> * Python version >> * Architecture/Platform >> * UCS2/UCS4 >> >> E.g. to solve the UCS2/4 part, we had to setup separate >> PyPI-style index sites. We'd really like to avoid that if >> possible. > > Well yeah, and the only sane way I can think to handle this is to have a > metadata file that gets uploaded with each distribution that covers all > these things (and the other things that other people need) and then have > the index (which would hopefully be PyPI on the whole) be queryable > along the lines of "give me a download url for a distribution named X > that is for Python 2.6, Win32, UCS4, PostGreSQL 8.5, BlagBlah"... So you want the indirection to happen even before fetching the download package ? That's a good idea - and also a lot better than the web page link scanning that setuptools is doing. It goes a bit in the direction of what we had in mind with writing for our clients: a tool that looks at the Python installation and automatically finds/downloads/installs the right package from our website. Only that we wanted to put up a single index file for that tool to use, not one per package. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Tue Oct 6 19:52:34 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 13:52:34 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB439E.6050102@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> Message-ID: <20091006175244.D1FF03A4116@sparrow.telecommunity.com> At 03:18 PM 10/6/2009 +0200, M.-A. Lemburg wrote: >Note how the package name is mangled... easy_install requires this. >The file name also doesn't tell you that the above is for >a UCS2 Python build. Again, easy_install fails with that information >added to the "py2.6" version marker. Btw, every couple years or so I've sent out a call on the distutils-SIG to try to get consensus on a format for the platform tag information used by setuptools. (The first time was before easy_install even existed.) Unfortunately, it's rare that anybody contributes more than criticism of the existing scheme. The last updates to the scheme were by Mac folks, who at least had a clear vision of what was needed. There's been some win32/win64 discussion in the past, but no proposals or patches of comparable specificity. Linux and other unices haven't had any concrete suggestions at all, AFAICR. Anyway... just saying that if you *have* a concretely implementable proposal about what to do with platform tags, please do share on the distutils-sig. Platform tags are one area that's reasonably easy to upgrade in a backward-compatible-ish way; newer versions of easy_install can still recognize older tags, although new eggs won't be recognized by older installers (forcing people to upgrade setuptools in that case). From pje at telecommunity.com Tue Oct 6 20:03:51 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 14:03:51 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4ACB49FB.3000407@simplistix.co.uk> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4AB8D910.8050601@simplistix.co.uk> <94bdd2610909221106h868a158yb29bdd77f96eac9@mail.gmail.com> <4AB92ADE.9060700@simplistix.co.uk> <87vdjauoq6.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> Message-ID: <20091006180400.BA69D3A4116@sparrow.telecommunity.com> At 02:45 PM 10/6/2009 +0100, Chris Withers wrote: >>To put this into a way that makes sense to me: I'm volunteering to >>keep distribute 0.6 and setuptools 0.6 in sync, no more, no less, >>and try and keep that as uncontroversial as possible, and get >>setuptools 0.6 releases out to match distribute 0.6 releases as soon as I can. That may not be as easy as it sounds; Distribute deleted various things from the setuptools tree (e.g. the release.sh script, used for issuing releases) and of course it adds other stuff (e.g. stuff to overwrite setuptools). So you'd need to screen the diffs. Second, I still intend to move setuptools 0.7 forward at some point, which means the patches also need to go to the trunk. If I had the time to co-ordinate and supervise all this, I'd have the time to just do it myself. If you want to help with that, the most helpful thing would be for you to consolidate all the changes into a pair of patches: one for the 0.6 branch and one for the 0.7 trunk. These patches would also need to exclude the SVN 1.6 changes (as I already have a change for that in my working copy, not yet checked in). They would also need to include appropriate changelog and documentation updates. If you can get those to me by Friday, I'll have them reviewed, applied, and released by Monday. I was already planning to spend a little time on bug closing and patch application this coming weekend, so if you can do this for me first, it will maximize the number I can get done. From pje at telecommunity.com Tue Oct 6 20:11:40 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 14:11:40 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACB6A3C.6000909@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACB6A3C.6000909@egenix.com> Message-ID: <20091006181149.133E83A4116@sparrow.telecommunity.com> At 06:03 PM 10/6/2009 +0200, M.-A. Lemburg wrote: >Chris Withers wrote: > > Well yeah, and the only sane way I can think to handle this is to have a > > metadata file that gets uploaded with each distribution that covers all > > these things (and the other things that other people need) and then have > > the index (which would hopefully be PyPI on the whole) be queryable > > along the lines of "give me a download url for a distribution named X > > that is for Python 2.6, Win32, UCS4, PostGreSQL 8.5, BlagBlah"... > >So you want the indirection to happen even before fetching >the download package ? > >That's a good idea - and also a lot better than the web page >link scanning that setuptools is doing. They're not mutually exclusive, btw. More information could be added to the PyPI "Simple" API to include this sort of thing, visible to newer installers and invisible to older ones. The main reason why easy_install has a relatively simple, REST outgoing API is that it allows people to make their own compatible indexes just by throwing some files in a directory and turning on Apache directory indexing. By using a... what do they call it, "microformat"? to mark up other information in the file, it'd still be possible for someone to slap up a web page and use it as a package index. >It goes a bit in the direction of what we had in mind with writing >for our clients: a tool that looks at the Python installation and >automatically finds/downloads/installs the right package from >our website. Only that we wanted to put up a single index file >for that tool to use, not one per package. That'd get a bit big for PyPI, though, with thousands of listed packages. It's already pretty slow whenever somebody miscapitalizes or mis-punctuates a package name and easy_install has to read the "file" that just lists all the package names. That, of course, is more a function of the fact that PyPI's "simple" API doesn't support canonicalization, although I believe certain other PyPI-mirroring and package indexing apps do. From mal at egenix.com Tue Oct 6 20:17:44 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Oct 2009 20:17:44 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <20091006175244.D1FF03A4116@sparrow.telecommunity.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <20091006175244.D1FF03A4116@sparrow.telecommunity.com> Message-ID: <4ACB89C8.3030405@egenix.com> P.J. Eby wrote: > At 03:18 PM 10/6/2009 +0200, M.-A. Lemburg wrote: >> Note how the package name is mangled... easy_install requires this. >> The file name also doesn't tell you that the above is for >> a UCS2 Python build. Again, easy_install fails with that information >> added to the "py2.6" version marker. > > Btw, every couple years or so I've sent out a call on the distutils-SIG > to try to get consensus on a format for the platform tag information > used by setuptools. (The first time was before easy_install even > existed.) Unfortunately, it's rare that anybody contributes more than > criticism of the existing scheme. The last updates to the scheme were > by Mac folks, who at least had a clear vision of what was needed. > There's been some win32/win64 discussion in the past, but no proposals > or patches of comparable specificity. Linux and other unices haven't > had any concrete suggestions at all, AFAICR. > > Anyway... just saying that if you *have* a concretely implementable > proposal about what to do with platform tags, please do share on the > distutils-sig. Platform tags are one area that's reasonably easy to > upgrade in a backward-compatible-ish way; newer versions of easy_install > can still recognize older tags, although new eggs won't be recognized by > older installers (forcing people to upgrade setuptools in that case). Here's a very simple Python version scheme we've been using for years: pyX.X_ucsY with X.X being the major.minor Python version and Y being either 2 or 4 depending on the UCS-build that was used to build the egg. The "_ucsY" part is not needed for pure-Python archives (but it doesn't hurt using it there as well) and also not needed on Windows, since Python is always UCS2 on Windows. easy_install should first look for the full version string and then fall back to the non-UCS one. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Tue Oct 6 21:16:07 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Oct 2009 12:16:07 -0700 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091006180400.BA69D3A4116@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <87vdjauoq6.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> Message-ID: On Tue, Oct 6, 2009 at 11:03 AM, P.J. Eby wrote: > At 02:45 PM 10/6/2009 +0100, Chris Withers wrote: >>> >>> To put this into a way that makes sense to me: I'm volunteering to keep >>> distribute 0.6 and setuptools 0.6 in sync, no more, no less, and try and >>> keep that as uncontroversial as possible, and get setuptools 0.6 releases >>> out to match distribute 0.6 releases as soon as I can. > > That may not be as easy as it sounds; Distribute deleted various things from > the setuptools tree (e.g. the release.sh script, used for issuing releases) > and of course it adds other stuff (e.g. stuff to overwrite setuptools). ?So > you'd need to screen the diffs. > > Second, I still intend to move setuptools 0.7 forward at some point, which > means the patches also need to go to the trunk. Dream on. > If I had the time to co-ordinate and supervise all this, I'd have the time > to just do it myself. I think at this point the community should not be forced wait for you to get a new supply of round tuits. The wait has been too long already. You can stay on in an advisory role, but I don't think it's reasonable to block development or decisions until you have time. > If you want to help with that, the most helpful thing would be for you to > consolidate all the changes into a pair of patches: one for the 0.6 branch > and one for the 0.7 trunk. > > These patches would also need to exclude the SVN 1.6 changes (as I already > have a change for that in my working copy, not yet checked in). ?They would > also need to include appropriate changelog and documentation updates. > > If you can get those to me by Friday, I'll have them reviewed, applied, and > released by Monday. ?I was already planning to spend a little time on bug > closing and patch application this coming weekend, so if you can do this for > me first, it will maximize the number I can get done. That's great, but I don't think it solves the structural problem, which is that you don't have enough time to devote to the project. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From olemis at gmail.com Tue Oct 6 21:56:04 2009 From: olemis at gmail.com (Olemis Lang) Date: Tue, 6 Oct 2009 14:56:04 -0500 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <87vdjauoq6.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> Message-ID: <24ea26600910061256n748ecd05o94ea921a3b2327e8@mail.gmail.com> On Tue, Oct 6, 2009 at 2:16 PM, Guido van Rossum wrote: > On Tue, Oct 6, 2009 at 11:03 AM, P.J. Eby wrote: >> At 02:45 PM 10/6/2009 +0100, Chris Withers wrote: >>>> >>>> To put this into a way that makes sense to me: I'm volunteering to keep >>>> distribute 0.6 and setuptools 0.6 in sync, no more, no less, and try and >>>> keep that as uncontroversial as possible, and get setuptools 0.6 releases >>>> out to match distribute 0.6 releases as soon as I can. >> >> That may not be as easy as it sounds; Distribute deleted various things from >> the setuptools tree (e.g. the release.sh script, used for issuing releases) >> and of course it adds other stuff (e.g. stuff to overwrite setuptools). ?So >> you'd need to screen the diffs. >> >> Second, I still intend to move setuptools 0.7 forward at some point, which >> means the patches also need to go to the trunk. > > Dream on. > >> If I had the time to co-ordinate and supervise all this, I'd have the time >> to just do it myself. > > I think at this point the community should not be forced wait for you > to get a new supply of round tuits. The wait has been too long > already. You can stay on in an advisory role, but I don't think it's > reasonable to block development or decisions until you have time. > Is it possible to fork `setuptools` somehow ? This way : - patches may be built for setuptools 0.7 if it evolves - community can continue development in the ?new? branch. all this easy if a DVCS is used ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html From ronaldoussoren at mac.com Tue Oct 6 22:06:41 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 06 Oct 2009 22:06:41 +0200 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <24ea26600910061256n748ecd05o94ea921a3b2327e8@mail.gmail.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <87vdjauoq6.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <24ea26600910061256n748ecd05o94ea921a3b2327e8@mail.gmail.com> Message-ID: <94A5D218-CC9D-46DD-8DB8-FCECEF8D3F93@mac.com> On 6 Oct, 2009, at 21:56, Olemis Lang wrote: > > Is it possible to fork `setuptools` somehow ? This way : Yes, see Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From pje at telecommunity.com Tue Oct 6 22:08:42 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 16:08:42 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <87vdjauoq6.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> Message-ID: <20091006200852.551BC3A4116@sparrow.telecommunity.com> At 12:16 PM 10/6/2009 -0700, Guido van Rossum wrote: >I think at this point the community should not be forced wait for you >to get a new supply of round tuits. The wait has been too long >already. You can stay on in an advisory role, but I don't think it's >reasonable to block development or decisions until you have time. Tarek didn't think so either, which is why he created a fork, called Distribute. So I'm not really clear on what you're trying to say here (or in the rest of the email, for that matter). From guido at python.org Tue Oct 6 22:14:18 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Oct 2009 13:14:18 -0700 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091006200852.551BC3A4116@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> Message-ID: I'm saying that I don't expect setuptools 0.7 to appear before Tarek's Distribute is mature and in widespread use. IOW I support Tarek's fork and suggest nobody hold their breath waiting for setuptools 0.7. On Tue, Oct 6, 2009 at 1:08 PM, P.J. Eby wrote: > At 12:16 PM 10/6/2009 -0700, Guido van Rossum wrote: >> >> I think at this point the community should not be forced wait for you >> to get a new supply of round tuits. The wait has been too long >> already. You can stay on in an advisory role, but I don't think it's >> reasonable to block development or decisions until you have time. > > Tarek didn't think so either, which is why he created a fork, called > Distribute. ?So I'm not really clear on what you're trying to say here (or > in the rest of the email, for that matter). > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at egenix.com Tue Oct 6 22:17:14 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Oct 2009 22:17:14 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <20091006181149.133E83A4116@sparrow.telecommunity.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACB6A3C.6000909@egenix.com> <20091006181149.133E83A4116@sparrow.telecommunity.com> Message-ID: <4ACBA5CA.2020902@egenix.com> P.J. Eby wrote: > At 06:03 PM 10/6/2009 +0200, M.-A. Lemburg wrote: >> It goes a bit in the direction of what we had in mind with writing >> for our clients: a tool that looks at the Python installation and >> automatically finds/downloads/installs the right package from >> our website. Only that we wanted to put up a single index file >> for that tool to use, not one per package. > > That'd get a bit big for PyPI, though, with thousands of listed > packages. It's already pretty slow whenever somebody miscapitalizes or > mis-punctuates a package name and easy_install has to read the "file" > that just lists all the package names. That, of course, is more a > function of the fact that PyPI's "simple" API doesn't support > canonicalization, although I believe certain other PyPI-mirroring and > package indexing apps do. The above was only meant for our eGenix packages, not PyPI. For PyPI the natural answer to this is to use the already existing PyPI XML-RPC interface: http://wiki.python.org/moin/PyPiXmlRpc This would only have to grow a few more fields for the release downloads in order to be able to match the user's Python configuration to the available release downloads. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From zookog at gmail.com Tue Oct 6 22:51:32 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Tue, 6 Oct 2009 14:51:32 -0600 Subject: [Python-Dev] [New-bugs-announce] [issue7064] Python 2.6.3 / setuptools 0.6c9: extension module builds fail with KeyError In-Reply-To: <94bdd2610910050604k5cc20cdblf24a65c307a1cfdf@mail.gmail.com> References: <1254742115.57.0.346118025313.issue7064@psf.upfronthosting.co.za> <19145.60293.661555.312221@montanaro.dyndns.org> <94bdd2610910050604k5cc20cdblf24a65c307a1cfdf@mail.gmail.com> Message-ID: Here are three buildbot farms for three different projects that exercise various features of setuptools: build, install, sdist_dsc, bdist_egg, sdist, and various specific requirements that our projects have, such as the "Desert Island Build" in which setuptools is not allowed to download anything from the Internet at build time or else it flunks the test. http://allmydata.org/buildbot/waterfall http://allmydata.org/buildbot-pycryptopp/waterfall http://allmydata.org/buildbot-zfec/waterfall I would love it if new versions of setuptools/Distribute would make more of these tests pass on more platforms or at least avoid causing any regressions in these tests on these platforms. Unfortunately, we can't really deploy new versions of setuptools/Distribute to this buildbot farm in order to re-run all the tests, because the only way that I know of to trigger all the tests is to make a commit to our central darcs repository for Tahoe-LAFS, pycryptopp, or zfec, and I don't want to do that to experiment with new versions of setuptools/Distribute. Does anyone know how to use a buildbot farm like this one to run tests without committing patches to the central repository? Regards, Zooko From pje at telecommunity.com Tue Oct 6 23:17:33 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 17:17:33 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> Message-ID: <20091006211742.9E43E3A4116@sparrow.telecommunity.com> At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote: >suggest nobody hold their breath waiting for setuptools 0.7. I've never suggested or implied otherwise. But, if you like Distribute so much, why not just add it directly to the stdlib? ;-) AFAIK, the only reason they've had multiple releases of it is to address the issues with its hijacking of setuptools; in a stdlib version all that stuff could be dropped. Otherwise, it'd already be "mature". There are many wins to be had from such a move, not the least of which is that it allows something to go into the stdlib that isn't (branding-wise) associated with me or setuptools, and lets it become owned/supported by an even wider community. Less political bickering, and the some of the technical results I hoped for all along are achieved. Yay, open source. From pje at telecommunity.com Tue Oct 6 23:19:51 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 06 Oct 2009 17:19:51 -0400 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACBA5CA.2020902@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACB6A3C.6000909@egenix.com> <20091006181149.133E83A4116@sparrow.telecommunity.com> <4ACBA5CA.2020902@egenix.com> Message-ID: <20091006212000.34B423A4116@sparrow.telecommunity.com> At 10:17 PM 10/6/2009 +0200, M.-A. Lemburg wrote: >P.J. Eby wrote: > > At 06:03 PM 10/6/2009 +0200, M.-A. Lemburg wrote: > >> It goes a bit in the direction of what we had in mind with writing > >> for our clients: a tool that looks at the Python installation and > >> automatically finds/downloads/installs the right package from > >> our website. Only that we wanted to put up a single index file > >> for that tool to use, not one per package. > > > > That'd get a bit big for PyPI, though, with thousands of listed > > packages. It's already pretty slow whenever somebody miscapitalizes or > > mis-punctuates a package name and easy_install has to read the "file" > > that just lists all the package names. That, of course, is more a > > function of the fact that PyPI's "simple" API doesn't support > > canonicalization, although I believe certain other PyPI-mirroring and > > package indexing apps do. > >The above was only meant for our eGenix packages, not PyPI. > >For PyPI the natural answer to this is to use the already existing >PyPI XML-RPC interface: > > http://wiki.python.org/moin/PyPiXmlRpc > >This would only have to grow a few more fields for the release >downloads in order to be able to match the user's Python configuration >to the available release downloads. Yes, but it's not easily duplicated by somebody who's just got a plain ol' web page, nor is it amenable to static page mirroring. A GET-only REST API, on the other hand, is compatible with both. From greg.ewing at canterbury.ac.nz Sat Oct 3 03:29:06 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Oct 2009 13:29:06 +1200 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: Message-ID: <4AC6A8E2.4030304@canterbury.ac.nz> Has anyone considered the idea of having the string % operator behave intelligently according to the contents of the format string? If it contains one or more valid %-formats, use old-style formatting; if it contains one or more valid {}-formats, use new-style formatting. Ambiguous cases could arise, of course, but hopefully they will be fairly rare, and raising an exception would point out the problem and allow it to be fixed. -- Greg From greg.ewing at canterbury.ac.nz Thu Oct 1 02:16:06 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Oct 2009 13:16:06 +1300 Subject: [Python-Dev] PEP 3144 review. In-Reply-To: <8517e9350909301448v4a7f891q3e12cec01ebde44a@mail.gmail.com> References: <8517e9350909140944x104bcc63g62f86b3af7c6ca9@mail.gmail.com> <4AC12AC6.80304@v.loewis.de> <4AC2A9D7.4080603@gmail.com> <5c6f2a5d0909300131g5fcb255eoe9403f54777fc677@mail.gmail.com> <79990c6b0909300252w1dec6e2dlc8efdd104e5116ed@mail.gmail.com> <5c6f2a5d0909300318p4b5404d4wb41e2add4ea8f88a@mail.gmail.com> <4AC33BA1.8050203@gmail.com> <5c6f2a5d0909300433i5b0d0833qe1a9d239bd303dfc@mail.gmail.com> <4AC3CED3.80104@v.loewis.de> <8517e9350909301448v4a7f891q3e12cec01ebde44a@mail.gmail.com> Message-ID: <4AC3F4C6.9080107@canterbury.ac.nz> Peter Moody wrote: > it's useful to take an > address like 192.168.1.100/24 and derive a bunch of information from > it (like the network address, broadcast address, containing supernets, > etc), but still remember that the original address was 192.168.1.100. > having a separate class or two for this is overly burdensome in my > mind. Seems to me what you want isn't so much an IPNetwork that can optionally have an address, as an IPAddress that can optionally have a mask, and methods for deriving those other things from it. -- Greg From greg.ewing at canterbury.ac.nz Thu Oct 1 01:35:01 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Oct 2009 12:35:01 +1300 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC38BD0.9070608@v.loewis.de> References: <4AC2CD65.30905@v.loewis.de> <4AC33DE1.3040304@trueblade.com> <4AC38BD0.9070608@v.loewis.de> Message-ID: <4AC3EB25.1000307@canterbury.ac.nz> Martin v. L?wis wrote: > So if a decision was made to eventually remove % formatting, it would > be reasonable to start migrating code to PEP 3101. However, no such > decision has been made (and hopefully won't be throughout 3.x) If it's not done during 3.x, then by the time 4.x comes around, there will still be heaps of code around that uses %-formatting, so it won't be possible to remove it in 4.x either. -- Greg From greg.ewing at canterbury.ac.nz Thu Oct 1 02:05:49 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Oct 2009 13:05:49 +1300 Subject: [Python-Dev] Announcing PEP 3136 In-Reply-To: <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> References: <20090929202035.GA5492@gmail.com> <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> Message-ID: <4AC3F25D.8000907@canterbury.ac.nz> Yuvgoog Greenle wrote: > When is the else after a loop executed? > 1. When the loop isn't entered at all. > 2. When the loop terminates through exhaustion of the list (does this > include when the list was empty?) > 3. When the loop didn't exit because of a break statement. 1 and 3 are just special cases of 2, when you include the empty-list case. So you're making it more complicated in your mind than it really is. > Any chances of cleaning this one up for python 4? I doubt that many other people will see anything here that needs cleaning up. -- Greg From greg.ewing at canterbury.ac.nz Thu Oct 1 00:41:42 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Oct 2009 10:41:42 +1200 Subject: [Python-Dev] PEP 3144 review. In-Reply-To: <79990c6b0909300252w1dec6e2dlc8efdd104e5116ed@mail.gmail.com> References: <8517e9350909140944x104bcc63g62f86b3af7c6ca9@mail.gmail.com> <200909280330.30643.steve@pearwood.info> <4AC03410.4010104@v.loewis.de> <4AC11855.7080406@v.loewis.de> <4AC12AC6.80304@v.loewis.de> <4AC2A9D7.4080603@gmail.com> <5c6f2a5d0909300131g5fcb255eoe9403f54777fc677@mail.gmail.com> <79990c6b0909300252w1dec6e2dlc8efdd104e5116ed@mail.gmail.com> Message-ID: <4AC3DEA6.2050804@canterbury.ac.nz> Paul Moore wrote: > linus and snoopy are hosts not networks, so > making them IPv4Network classes seems wrong. I'd instinctively make > them IPv4Address objects (which, I believe, would work). However, by allowing IPNetwork objects to also contain a host address, we seem to be officially sanctioning such abuse, or at least turning a blind eye to it. It's possible that the user is thinking of linus and snoopy as being *both* networks and host addresses, in which case this kind of mistake will be easy to make. Not to mention the potential confusion in the mind of someone trying to read or maintain the code. My feeling is that we shouldn't accept this kind of compromise as being "good enough". Once a module is in the stdlib, we're going to be stuck with it for quite a while, so I think it's worth taking some time to make sure that what we adopt is based on a conceptually sound design. Peter is of course within his rights not to change his module in this way. But this just means that ipaddr is not the right module to adopt, and we should design another one and give it a different name. -- Greg From guido at python.org Wed Oct 7 05:44:36 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Oct 2009 20:44:36 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AC6A8E2.4030304@canterbury.ac.nz> References: <4AC6A8E2.4030304@canterbury.ac.nz> Message-ID: On Fri, Oct 2, 2009 at 6:29 PM, Greg Ewing wrote: > Has anyone considered the idea of having the string % operator > behave intelligently according to the contents of the format > string? > > If it contains one or more valid %-formats, use old-style > formatting; if it contains one or more valid {}-formats, > use new-style formatting. > > Ambiguous cases could arise, of course, but hopefully they > will be fairly rare, and raising an exception would point > out the problem and allow it to be fixed. Hm... The % operator already does too much guessing: if the string contains exactly one %-format, the argument may be either a size-1 tuple or a non-tuple, otherwise it has to be a size-N tuple, except if the %-formats use the %(name)X form, then the argument must always be a dict. It doesn't sound to me as if adding more guesswork is going to improve its reliability. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From david.lyon at preisshare.net Wed Oct 7 07:04:17 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 07 Oct 2009 01:04:17 -0400 Subject: [Python-Dev] =?utf-8?q?eggs_now_mandatory_for_pypi=3F?= In-Reply-To: <20091006175244.D1FF03A4116@sparrow.telecommunity.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <20091006175244.D1FF03A4116@sparrow.telecommunity.com> Message-ID: <75c06ee14c3ef81164d5b4a9f3177799@preisshare.net> On Tue, 06 Oct 2009 13:52:34 -0400, "P.J. Eby" wrote: > Btw, every couple years or so I've sent out a call on the > distutils-SIG to try to get consensus on a format for the platform > tag information used by setuptools. (The first time was before > easy_install even existed.) Unfortunately, it's rare that anybody > contributes more than criticism of the existing scheme. The last > updates to the scheme were by Mac folks, who at least had a clear > vision of what was needed. There's been some win32/win64 discussion > in the past, but no proposals or patches of comparable > specificity. Consider the fact that Mac users won't answer or discuss posts by some windows users.. Why do this ? Why stir up platform bias? Fact is Phil, windows users get *turned-away* at the door in distutils... or ignored. Issues can hang around for years. And your post is now asking "why no windows users?" Go figure.. nobody answers them or takes issues seriously. Distutils for windows is very, very dead.. grave-ware in-fact. It should be quite obvious that windows users are forked off.. Why would windows people make proposals when they know it won't go anywhere... David From ubershmekel at gmail.com Wed Oct 7 09:08:37 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Wed, 7 Oct 2009 09:08:37 +0200 Subject: [Python-Dev] Announcing PEP 3136 In-Reply-To: <4AC3F25D.8000907@canterbury.ac.nz> References: <20090929202035.GA5492@gmail.com> <9d153b7c0909301415u39c8f198qea4777036e441289@mail.gmail.com> <4AC3F25D.8000907@canterbury.ac.nz> Message-ID: <9d153b7c0910070008l70b115cfma791e892ea25e54e@mail.gmail.com> This thread moved to python-ideas so please post only there. http://mail.python.org/pipermail/python-ideas/2009-October/005924.html --yuv From mal at egenix.com Wed Oct 7 09:45:13 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 09:45:13 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <75c06ee14c3ef81164d5b4a9f3177799@preisshare.net> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <20091006175244.D1FF03A4116@sparrow.telecommunity.com> <75c06ee14c3ef81164d5b4a9f3177799@preisshare.net> Message-ID: <4ACC4709.4090405@egenix.com> David Lyon wrote: > Distutils for windows is very, very dead.. grave-ware in-fact. Now that is not true at all. We have a native Windows installer (bdist_wininst) and an MSI builder (bdist_msi) that both work great on Windows. Plus there are add-ons for other installers such as NSIS and InnoSetup. Please don't forget that setuptools is not distutils, it's just a package extending distutils. If you find bugs related to Windows, please file a bug report. If you have specific proposals for distutils on Windows, you can use the mailing list or send patches to the tracker. The ham/egg ratio on the mailing list tends to be very much on the egg side, so the tracker can help get more attention from folks doing distutils without setuptools. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From benjamin at python.org Mon Oct 5 03:54:34 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 4 Oct 2009 20:54:34 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <87y6nt37kb.fsf@hbox.dyndns.org> <426ada670910021323g3ab0d206t46a00d6210260dd0@mail.gmail.com> Message-ID: <1afaf6160910041854v55edcc96we4fb2d6a7f2da948@mail.gmail.com> 2009/10/4 INADA Naoki : > What about using string prefix 'f'? > > ?f"{foo} and {bar}" % something == "{foo} and {bar}.format(something) > > ?s = f"{foo}" > ?t = "%(bar)s" > ?s + t ?# raises Exception > > Transition plan: > n: Just add F prefix. And adding "format_string" in future. > n+1: deprecate __mod__() without 'F'. > n+2: libraries use .format() and deprecate __mod__() with 'F' > n+3: remove __mod__() -1 That requires keeping formatting information around in every string instance. -- Regards, Benjamin From hrvoje.niksic at avl.com Wed Oct 7 13:26:59 2009 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 07 Oct 2009 13:26:59 +0200 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> Message-ID: <4ACC7B03.3040700@avl.com> Paul Moore wrote: > Traceback (most recent call last): > File "hello.py", line 13, in > main() > File "hello.py", line 7, in main > sys.stdout.flush() > IOError: [Errno 9] Bad file descriptor > > (Question - is it *ever* possible for a Unix program to have invalid > file descriptors 0,1 and 2? At startup - I'm assuming anyone who does > os.close(1) knows what they are doing!) Of course; simply use the >&- pseudo-redirection, which has been a standard sh feature (later inherited by ksh and bash, but not csh) for ~30 years. The error message is amusing, too: $ python -c 'print "foo"' >&- close failed in file object destructor: Error in sys.excepthook: Original exception was: Adding an explicit flush results in a more understandable error message: Traceback (most recent call last): File "", line 1, in IOError: [Errno 9] Bad file descriptor From swapnil.st at gmail.com Wed Oct 7 16:34:48 2009 From: swapnil.st at gmail.com (Swapnil Talekar) Date: Wed, 7 Oct 2009 20:04:48 +0530 Subject: [Python-Dev] Python byte-compiled and optimized code Message-ID: I am working on deploying Python on VxWorks platform as a part of project for my company. Accordingly, I would like to know couple of things from python's core-developers. Although I think I already know the answers for most of the questions, we need a confirmation from the community 1) Is the byte-compiled .pyc file and optimized .pyo file platform-independent?(including python versions 3.x) *If yes, is it guaranteed to stay that way in future?* 2) If the the generation of .pyc file fails (say, due to write protected area), does that stop the script execution? 3) Is it possible to redirect the location of the generation of .pyc files to other than that of the corresponding .py files? Regards, Swapnil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Oct 7 16:49:44 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Oct 2009 14:49:44 +0000 (UTC) Subject: [Python-Dev] A new way to configure logging Message-ID: At present, configuration of Python's logging package can be done in one of two ways: 1. Create a ConfigParser-readable configuration file and use logging.config.fileConfig() to read and implement the configuration therein. 2. Use the logging API to programmatically configure logging using getLogger(), addHandler() etc. The first of these works for simple cases but does not cover all of the logging API (e.g. Filters). The second of these provides maximal control, but besides requiring users to write the configuration code, it fixes the configuration in Python code and does not facilitate changing it easily at runtime. In addition, the ConfigParser format appears to engender dislike (sometimes strong dislike) in some quarters. Though it was chosen because it was the only configuration format supported in the stdlib at that time, many people regard it (or perhaps just the particular schema chosen for logging's configuration) as 'crufty' or 'ugly', in some cases apparently on purely aesthetic grounds. Recent versions of Python of course support an additional battery-included format which can be used for configuration - namely, JSON. Other options, such as YAML, are also possible ways of configuring systems, Google App Engine-style, and PyYAML has matured nicely. There has also been talk on the django-dev mailing list about providing better support for using Python logging in Django. When it happens (as of course I hope it does) this has the consequence that many new users who use Django but are relatively inexperienced in Python (e.g. in PHP shops which are moving to Django) will become exposed to Python logging. As Django is configured using a Python module and use of ConfigParser-style files is not a common approach in that ecosystem, users will find either of the two approaches outlined above a particular pain point when configuring logging for their Django applications and websites, unless something is done to avoid it. All three of the contenders for the title of "commonly found configuration mechanism" - JSON, YAML and Python code - will be expressible, in Python, as Python dicts. So it seems to make sense to add, to logging.config, a new callable bound to "dictConfig" which will take a single dictionary argument and configure logging from that dictionary. An important facet of implementing such a scheme will be the format or schema which the dictionary has to adhere to. I have started working on what such a schema would look like, and if people here think it's a good idea to go ahead with this, I'll provide the details of the schema on a separate post which I'll also cross-post on comp.lang.python so that I can get feedback from there, too. In outline, the scheme I have in mind will look like this, in terms of the new public API: class DictConfigurator: def __init__(self, config): #config is a dict-like object (duck-typed) import copy self.config = copy.deepcopy(config) def configure(self): # actually do the configuration here using self.config dictConfigClass = DictConfigurator def dictConfig(config): dictConfigClass(config).configure() This allows easy replacement of DictConfigurator with a suitable subclass where needed. What's the general feeling here about this proposal? All comments and suggestions will be gratefully received. Regards, Vinay Sajip From fdrake at gmail.com Wed Oct 7 17:12:09 2009 From: fdrake at gmail.com (Fred Drake) Date: Wed, 7 Oct 2009 11:12:09 -0400 Subject: [Python-Dev] Python byte-compiled and optimized code In-Reply-To: References: Message-ID: <9cee7ab80910070812y7d9e2504qca040f981adc0f0d@mail.gmail.com> On Wed, Oct 7, 2009 at 10:34 AM, Swapnil Talekar wrote: > 1) Is the byte-compiled .pyc file and optimized .pyo file > platform-independent?(including python versions 3.x) Yes. > If yes, is it > guaranteed to stay that way in future? Yes. > 2)?If the the generation of .pyc file fails (say, due to write protected > area), does that stop the script execution? No. It only means the bytecode won't be cached. > 3) Is it possible to redirect the location of the generation of .pyc files > to other than that of the corresponding .py files? I think some support for this has been developed, at least experimentally, but I'm not sure if it's part of a stable release or not. It's more likely in Python 3.x, which I'm significantly less familiar with. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From olemis at gmail.com Wed Oct 7 17:19:27 2009 From: olemis at gmail.com (Olemis Lang) Date: Wed, 7 Oct 2009 10:19:27 -0500 Subject: [Python-Dev] A new way to configure logging In-Reply-To: References: Message-ID: <24ea26600910070819r364f6715l9164a754bc20b8f4@mail.gmail.com> On Wed, Oct 7, 2009 at 9:49 AM, Vinay Sajip wrote: > At present, configuration of Python's logging package can be done in one of two > ways: > > 1. Create a ConfigParser-readable configuration file and use > logging.config.fileConfig() to read and implement the configuration therein. > 2. Use the logging API to programmatically configure logging using getLogger(), > addHandler() etc. > [...] Wow ! Great explanation ! ;o) > > All three of the contenders for the title of "commonly found configuration > mechanism" - JSON, YAML and Python code - will be expressible, in Python, as > Python dicts. .INI files too { 'section1' : {'opt11' : 'val1', 'opt12' : 32323}, 'section1' : {'opt21' : 'val2', 'opt22' : 32323}, 'section1' : {'opt31' : 'val3', 'opt32' : 32323}, ... } In fact it seems that this is a typical characteristic of config formats (CMIIW) > So it seems to make sense to add, to logging.config, a new > callable bound to "dictConfig" which will take a single dictionary argument and > configure logging from that dictionary. > This kind of problems is similar to the one mentioned in another thread about modifying config options after executing commands. In that case I mentioned that the same dict-like interface also holds for WinReg and so on ... So thinking big (yes ! I have a big head ! ) I think that the best approach in this case is to build an adaptation (simple) layer on top of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific extensions for these formats . Perhaps the proper interfaces are already there (e.g. `dict`, `shelve` ) and I'm just blind and looking somewhere else ;o) If `dict` (possibly nested ;o) is enough to represent config files then the new method should be ok ... IMHO [...] > > In outline, the scheme I have in mind will look like this, in terms of the new > public API: > > class DictConfigurator: > ? ?def __init__(self, config): #config is a dict-like object (duck-typed) ======== > ? ? ? ?import copy > ? ? ? ?self.config = copy.deepcopy(config) Why ? > > ? ?def configure(self): > ? ? ? ?# ?actually do the configuration here using self.config > > dictConfigClass = DictConfigurator > > def dictConfig(config): > ? ?dictConfigClass(config).configure() > > This allows easy replacement of DictConfigurator with a suitable subclass where > needed. > extension is cool ... what's the point about adding the new method instead of using `DictConfigurator` directly ? > What's the general feeling here about this proposal? > I'm feeling things ... :) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html From solipsis at pitrou.net Wed Oct 7 17:24:27 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2009 15:24:27 +0000 (UTC) Subject: [Python-Dev] Python byte-compiled and optimized code References: <9cee7ab80910070812y7d9e2504qca040f981adc0f0d@mail.gmail.com> Message-ID: Fred Drake gmail.com> writes: > > 3) Is it possible to redirect the location of the generation of .pyc files > > to other than that of the corresponding .py files? > > I think some support for this has been developed, at least > experimentally, but I'm not sure if it's part of a stable release or > not. It's more likely in Python 3.x, which I'm significantly less > familiar with. I haven't seen any such thing. If you need pyc files to reside in read-only directories, the best solution would be to generate them at install time (when you have write access to the directories). Regards Antoine. From vinay_sajip at yahoo.co.uk Wed Oct 7 17:49:56 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Oct 2009 15:49:56 +0000 (UTC) Subject: [Python-Dev] A new way to configure logging References: <24ea26600910070819r364f6715l9164a754bc20b8f4@mail.gmail.com> Message-ID: Olemis Lang gmail.com> writes: > This kind of problems is similar to the one mentioned in another > thread about modifying config options after executing commands. In > that case I mentioned that the same dict-like interface also holds for > WinReg and so on ... > > So thinking big (yes ! I have a big head ! ) I think that the best > approach in this case is to build an adaptation (simple) layer on top > of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific > extensions for these formats . Perhaps the proper interfaces are > already there (e.g. `dict`, `shelve` ) and I'm just blind and looking > somewhere else ;o) Sorry, you've lost me :-) > > ? ? ? ?import copy > > ? ? ? ?self.config = copy.deepcopy(config) > > Why ? So I'm free to mutate self.config as I see fit. > extension is cool ... what's the point about adding the new method > instead of using `DictConfigurator` directly ? When you say "the new method", if you mean "configure" - the pattern is so that a subclass can override __init__ and do additional setup before configure() is called. Regards, Vinay Sajip From ksande at uva.nl Wed Oct 7 16:54:58 2009 From: ksande at uva.nl (Koen van de Sande) Date: Wed, 07 Oct 2009 16:54:58 +0200 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: References: Message-ID: <4ACCABC2.6030402@uva.nl> > If setuptools can be made to work with Python 2.6.3 /and/ earlier > versions of Python 2.6.x, then it should be patched and an update > released. If not, then perhaps we should revert the change in a quick > Python 2.6.4. If there is going to be any quick 2.6.4 release, can you consider a fix to the building of extensions under Windows ( http://bugs.python.org/issue4120 )? Extensions built without this patch applied will not work if the MSVC9 runtimes are installed locally (in the Python folder) instead of system-wide (WinSxS). PIL and Matplotlib now use this patch when building extensions. But many others (e.g. PyGame) still create problematic extensions. -Koen From paul at rudin.co.uk Wed Oct 7 18:06:56 2009 From: paul at rudin.co.uk (Paul Rudin) Date: Wed, 07 Oct 2009 17:06:56 +0100 Subject: [Python-Dev] A new way to configure logging References: Message-ID: <87ws37dw7z.fsf@rudin.co.uk> Vinay Sajip writes: > What's the general feeling here about this proposal? All comments and > suggestions will be gratefully received. > How about the global logging configuration being available as an object supporting the usual dictionary interface? So you could, for example, do something like: "logging.dictconfig.update(partialconfig)" From olemis at gmail.com Wed Oct 7 18:26:42 2009 From: olemis at gmail.com (Olemis Lang) Date: Wed, 7 Oct 2009 11:26:42 -0500 Subject: [Python-Dev] A new way to configure logging In-Reply-To: References: <24ea26600910070819r364f6715l9164a754bc20b8f4@mail.gmail.com> Message-ID: <24ea26600910070926g6b742492xd6be4fa07a932a0c@mail.gmail.com> On Wed, Oct 7, 2009 at 10:49 AM, Vinay Sajip wrote: > Olemis Lang gmail.com> writes: > >> This kind of problems is similar to the one mentioned in another >> thread about modifying config options after executing commands. In >> that case I mentioned that the same dict-like interface also holds for >> WinReg and so on ... >> >> So thinking big (yes ! I have a big head ! ) I think that the best >> approach in this case is to build an adaptation (simple) layer on top >> of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific >> extensions for these formats . Perhaps the proper interfaces are >> already there (e.g. `dict`, `shelve` ) and I'm just blind and looking >> somewhere else ;o) > > Sorry, you've lost me :-) > Never mind . I was just trying to say that using `dict`, an adapter could be implemented (if not already there ;o) for multiple formats like the ones I mentioned above and the solution would cover many config formats ... and also I was saying that I was not sure about whether this is correct , I mean for *ANY* config formats, but definitely will work for many ;o) >> > ? ? ? ?import copy >> > ? ? ? ?self.config = copy.deepcopy(config) >> >> Why ? > > So I'm free to mutate self.config as I see fit. > why not to let the user do it if he | she thinks so. I mean if somebody supplies in a temporary mapping that can be written then why to spend that time cloning the dict ? If the function has side-effects, well just document it ;o) >> extension is cool ... what's the point about adding the new method >> instead of using `DictConfigurator` directly ? > > When you say "the new method", if you mean "configure" - the pattern is so that > a subclass can override __init__ and do additional setup before configure() is > called. > Sorry I was confused. I was talking about `dictConfig` *FUNCTION* (to be added to logging.config ... isn't it ? ;o). I mean if dictConfig is sematically equivalent to a simple `dc.configure` why to add such function ? PS: Not a big problem anyway ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Looking for a technique to create flexible, graphical dashboards ... - http://feedproxy.google.com/~r/TracGViz-full/~3/71kRhT34BgU/43789 From olemis at gmail.com Wed Oct 7 18:29:26 2009 From: olemis at gmail.com (Olemis Lang) Date: Wed, 7 Oct 2009 11:29:26 -0500 Subject: [Python-Dev] A new way to configure logging In-Reply-To: <87ws37dw7z.fsf@rudin.co.uk> References: <87ws37dw7z.fsf@rudin.co.uk> Message-ID: <24ea26600910070929u383b3718yc5b7d573149677a@mail.gmail.com> On Wed, Oct 7, 2009 at 11:06 AM, Paul Rudin wrote: > Vinay Sajip writes: > > >> What's the general feeling here about this proposal? All comments and >> suggestions will be gratefully received. >> > > How about the global logging configuration being available as an object > supporting the usual dictionary interface? So you could, for example, do > something like: "logging.dictconfig.update(partialconfig)" > Seems interesting ... yes ! ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Search for milestone change in change history - Trac Users ... - http://feedproxy.google.com/~r/TracGViz-full/~3/Gsj4xLjuCtk/578d40b734e5e753 From zookog at gmail.com Wed Oct 7 18:56:49 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 7 Oct 2009 10:56:49 -0600 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091006211742.9E43E3A4116@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> Message-ID: +1 For a large number of people [1, 2, 3], setuptools is already a critical part of Python. Make it official. Let everyone know that future releases of Python will not break setuptools/Distribute, and that they can rely on backwards-compatibility with the myriad existing packages. Make the next release of the distutils standard lib module be Distribute. (Perhaps some people will complain, but they can channel their energy into improving the new distutils.) Regards, Zooko [1] The majority of professional developers using Python rely on setuptools to distribute and to use packages: http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/ [2] setuptools is one of the most downloaded packages on PyPI: http://pypi.python.org/pypi/setuptools http://blog.delaguardia.com.mx/tags/pypi [3] about one fifth of Debian users who install python2.5 also install python-pkg-resources: http://qa.debian.org/popcon.php?package=python-setuptools http://qa.debian.org/popcon.php?package=python2.5 From zookog at gmail.com Wed Oct 7 19:07:37 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 7 Oct 2009 11:07:37 -0600 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: References: <4AB67430.2060306@egenix.com> Message-ID: Folks: I accidentally sent this letter just to MAL when I intended it to python-dev. Please read it, as it explains why the issue I'm raising is not just the "we should switch to ucs4 because it is better" issue that was previously settled by GvR. This is a current, practical problem that is preventing people from distributing and using Python packages with binary extension modules on Linux. Regards, Zooko ---------- Forwarded message ---------- From: Zooko O'Whielacronx Date: Sun, Sep 27, 2009 at 11:43 AM Subject: Re: [Python-Dev] please consider changing --enable-unicode default to ucs4 To: "M.-A. Lemburg" Folks: I'm sorry, I think I didn't make my concern clear. ?My users, and lots of other users, are having a problem with incompatibility between Python binary extension modules. ?One way to improve the situation would be if the Python devs would use their "bully pulpit" -- their unique position as a source respected by all Linux distributions -- and say "We recommend that Linux distributions use UCS4 for compatibility with one another". ?This would not abrogate anyone's ability to choose their preferred setting nor, as far as I can tell, would it interfere with the ongoing development of Python. Here are the details: I'm the maintainer of several Python packages. ?I work hard to make it easy for users, even users who don't know anything about Python, to use my software. ?There have been many pain points in this process and I've spent a lot of time on it for about three years now working on packaging, including the tools such as setuptools and distutils and the new "distribute" tool. ?Python packaging has been improving during these years -- things are looking up. One of the remaining pain points is that I can distribute binaries of my Python extension modules for Windows or Mac, but if I distribute a binary Python extension module on Linux, then if the user has a different UCS2/UCS4 setting then they won't be able to use the extension module. ?The current de facto standard for Linux is UCS4 -- it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. ?The vast majority of Linux users in practice have UCS4, and most binary Python modules are compiled for UCS4. That means that a few folks will get left out. ?Those folks, from my experience, are people who built their python executable themselves without specifying an override for the default, and the smaller Linux distributions who insist on doing whatever upstream Python devs recommend instead of doing whatever the other Linux distros are doing. ?One of the data points that I reported was a Python interpreter that was built locally on an Ubuntu server. ?Since the person building it didn't know to override the default setting of --enable-unicode, he ended up with a Python interpreter built for UCS2, even though all the Python extension modules shipped by Ubuntu were built with UCS4. These are not isolated incidents. ?The following google searches suggest that a number of people spend time trying to figure out why Python extension modules fail on their linux systems: http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefined+symbol Another data point is the Mandriva Linux distribution. ?It is probably much smaller than Debian, Ubuntu, or RedHat, but it is still one of the major, well-known distributions. ?I requested of the Python maintainer for Mandriva, Michael Scherer, that they switch from UCS2 to UCS4 in order to reduce compatibility problems like these. ?His answer as I understood it was that it is best to follow the recommendations of the upstream Python devs by using the default setting instead of choosing a setting for himself. (Now we could implement a protocol which would show whether a given Python package was compiled for UCS2 or UCS4. ?That would be good. Hopefully it would make incompatibility more explicit and understandable to users. ?Here is a ticket for that -- which project I am contributing to: http://bugs.python.org/setuptools/issue78 . However, even if we implement that feature in the distribute tool (the successor to setuptools), users who build their own python or who use a Linux distribution that follows upstream configuration defaults will still be unable to use most Python packages with compiled extension modules.) In a message on this thread, MvL wrote: > "For that reason I think it's also better that the configure script > continues to default to UTF-16 -- this will give the UTF-16 support > code the necessary exercise." > > This is effectively a BDFL pronouncement. Nothing has changed the > validity of the premise of the statement, so the conclusion remains > valid, as well. My understand of the earlier thread was that someone suggested that UCS4 would be technically better and GvR decided that there were technical reasons to continue actively maintaining the UCS2 code. This thread is different: I'm saying that users are suffering packaging problems and asking for help with that. ?The way that python-dev can help is to make it so that people who choose "Whatever the upstream devs prefer" (--enable-unicde) on Linux get the de facto standard setting. In the earlier thread, GvR wrote: "I think we should continue to leave this up to the distribution. AFAIK many Linux distros already use UCS4 for everything anyway.". ?But at least some distributions are asking the upstream Python devs to choose for them, by leaving the setting at the default. Hm, pondering GvR's words: "I think we should continue to leave this up to the distribution", I have a new proposal: make it so that *on Linux only* "--enable-unicode" errors out with an error message saying "please choose either --enable-unicode=ucs2 or --enable-unicode=ucs4. ucs4 is the most widely used setting on Linux. See http://python.org/wiki/UCS2_vs_UCS4 for details.". ?This would force those Linux distributions who are *not* currently deciding to decide. Thank you for your attention. Regards, Zooko From tjreedy at udel.edu Wed Oct 7 18:54:32 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 Oct 2009 12:54:32 -0400 Subject: [Python-Dev] Python byte-compiled and optimized code In-Reply-To: <9cee7ab80910070812y7d9e2504qca040f981adc0f0d@mail.gmail.com> References: <9cee7ab80910070812y7d9e2504qca040f981adc0f0d@mail.gmail.com> Message-ID: Fred Drake wrote: > On Wed, Oct 7, 2009 at 10:34 AM, Swapnil Talekar wrote: >> 1) Is the byte-compiled .pyc file and optimized .pyo file >> platform-independent?(including python versions 3.x) > > Yes. To be clear, CPython's .pyc is platform independent for a particular x.y version, and should, I believe, stay the same for all x.y.z releases, but the format of .pyc may change slightly from x.y to x.(y+1) From vinay_sajip at yahoo.co.uk Wed Oct 7 18:55:48 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Oct 2009 16:55:48 +0000 (UTC) Subject: [Python-Dev] A new way to configure logging References: <87ws37dw7z.fsf@rudin.co.uk> Message-ID: Paul Rudin writes: > How about the global logging configuration being available as an object > supporting the usual dictionary interface? So you could, for example, do > something like: "logging.dictconfig.update(partialconfig)" A "partial configuration" only makes sense under certain limited conditions. For example, a dict used for configuration will map to a graph of objects - filters, formatters, handlers and loggers - which by necessity need to be referred to via a string key in the dict - because you can't easily encode a graph in a dict which comes from, say, a YAML or JSON file. These keys are temporary and just used to indicate e.g. which handlers to attach to which loggers for that particular configuration call: { "handlers": { "h1": { configuration for a handler }, "h2": { configuration for another handler }, "h3": { configuration for yet another handler }, }, "loggers": { "foo": { "level": "DEBUG", "handlers": ["h1", "h3"], }, "bar.baz": { "level": "INFO", "handlers": ["h2", "h3"], }, "spam.eggs": { "level": "ERROR", "handlers": ["h1", "h3"], }, } } Here, the keys "h1", "h2" etc. are just used to encode connections in the graph, and are useless outside the context of the specific configuration call. If you pass a partial configuration, to be implemented incrementally, there's no way of knowing if (in general) it's going to be consistent with the existing configuration. Nor can you refer to e.g. handlers, formatters or filters you passed in earlier configuration calls. Since logger names such as "foo", "bar.baz" in the above example *are* meaningful across multiple configuration calls, it *would be* possible to make certain changes in the configuration - such as changing levels of loggers - by passing in a partial configuration. However, if this "partial" functionality is provided without restriction, it's hard to ensure that a user doesn't inadvertently misconfigure logging. And while that is equally true with a one-shot configuration, it's at least easier to debug. Things can get pretty murky from a diagnostic point of view if code can make arbitrary changes to the logging configuration during application execution, so that's not a good practice to encourage :-) Also, providing a readable dict-like interface to the internal configuration will almost certainly lead to backward-compatibility headaches in the future. Regards, Vinay Sajip From phd at phd.pp.ru Wed Oct 7 19:12:48 2009 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 7 Oct 2009 21:12:48 +0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> Message-ID: <20091007171248.GB1901@phd.pp.ru> On Wed, Oct 07, 2009 at 10:56:49AM -0600, Zooko O'Whielacronx wrote: > For a large number of people [1, 2, 3], setuptools is already a > critical part of Python. Make it official. Let everyone know that > future releases of Python will not break setuptools/Distribute, and > that they can rely on backwards-compatibility Who will pay the price of maintaining that part of stdlib and the price of compatibility? Because there is always a price; should the core developers be burdened? Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From barry at python.org Wed Oct 7 19:18:04 2009 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2009 13:18:04 -0400 Subject: [Python-Dev] Python 2.6.4rc1 Message-ID: Hello everyone. The source tarballs and Windows installers for Python 2.6.4rc1 are now available: http://www.python.org/download/releases/2.6.4/ Please download them, install them, and try to use them with your projects and environments. Let us know if you encounter any problems with them. Hopefully we can avoid the situation with 2.6.3 having such critical bugs. 2.6.4 final is planned for 18-October. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From scott+python-dev at scottdial.com Wed Oct 7 19:22:12 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 07 Oct 2009 13:22:12 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091006211742.9E43E3A4116@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> Message-ID: <4ACCCE44.8050905@scottdial.com> P.J. Eby wrote: > At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote: >> suggest nobody hold their breath waiting for setuptools 0.7. > > I've never suggested or implied otherwise. > > But, if you like Distribute so much, why not just add it directly to the > stdlib? ;-) > > There are many wins to be had from such a move, not the least of which > is that it allows something to go into the stdlib that isn't > (branding-wise) associated with me or setuptools, and lets it become > owned/supported by an even wider community. I think the biggest problem here is that the brand ("setuptools") is so ingrained in the world that someone releasing something setuptools-but-not-setuptools ("Distribute") is at a serious disadvantage. Your insistence on retaining the name "setuptools" for yourself and your vapor-ware only harms the community at-large. Even experienced developers are unaware of Distribute's existence.. I was entirely unaware until yourself and PJE got in a bickering exchange on Python-Dev this past month. The extremely high visibility of your stale package compared to their non-stale package is quite unfortunate. You are free to say, "that is their problem," but you are not free to say, "it is not my fault." > AFAIK, the only reason they've had multiple releases of it is to > address the issues with its hijacking of setuptools; in a stdlib > version all that stuff could be dropped. Otherwise, it'd already be > "mature". IOW, you acknowledge that your own position and requiring them to tolerate the parallel existence (in the world) of setuptools harms Distribute. I fail to see how this relates to being in the stdlib. As long as it is not called "setuptools" and packages in the world say "import setuptools", then there are conflicts they will be forced to managed. And moreover, as long as a stale, perhaps buggy version of setuptools is the compatibility model they must emulate, they will have a hard time coexisting. > Less political bickering, and the some of the technical results I > hoped for all along are achieved. Yay, open source. And yet, political bickering seems to be all you are good for in this case. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From solipsis at pitrou.net Wed Oct 7 19:21:53 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2009 17:21:53 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: Vinay Sajip yahoo.co.uk> writes: > > >>> "%0#8x" % 0x1234 > '0x001234' > >>> "{0:0>#8x}".format(0x1234) > '000x1234' Apart from the sheer unreadability of the {}-style format string, the result looks rather unexpected from a human being's point of view. (in those situations, I would output the 0x manually anyway, such as: >>> "0x%06x" % 0x1234 '0x001234' >>> "0x{:06x}".format(0x1234) '0x001234' ) Regards Antoine. From scott+python-dev at scottdial.com Wed Oct 7 19:26:01 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 07 Oct 2009 13:26:01 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: Message-ID: <4ACCCF29.7030003@scottdial.com> Barry Warsaw wrote: > 2.6.4 final is planned for 18-October. Barry, I suspect this release is primarily to quench the problems with distutils, but.. http://bugs.python.org/issue5949 doesn't seem to have been addressed by you. And this seems like it would be another unfortunate loss of an opportunity. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From mal at egenix.com Wed Oct 7 19:27:55 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 19:27:55 +0200 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> Message-ID: <4ACCCF9B.5060303@egenix.com> Zooko O'Whielacronx wrote: > +1 > > For a large number of people [1, 2, 3], setuptools is already a > critical part of Python. Make it official. Let everyone know that > future releases of Python will not break setuptools/Distribute, and > that they can rely on backwards-compatibility with the myriad existing > packages. Make the next release of the distutils standard lib module > be Distribute. > > (Perhaps some people will complain, but they can channel their energy > into improving the new distutils.) We've had that discussion in past and a few times. setuptools/Distribute is not distutils, just an add-on (and there are others as well). There are good concepts in setuptools, but the implementation is just not suitable for stdlib integration. Hopefully, Distribute can fix the hacks, trim down the number of features to what people really use, add a proper uninstall command and then we can have the add-to-the-stdlib discussion again. Having more competition will also help, e.g. ActiveState's PyPM looks promising (provided they choose to open-source it) and then there's pip. Things look promising, indeed. > Regards, > > Zooko > > [1] The majority of professional developers using Python rely on > setuptools to distribute and to use packages: > http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/ > [2] setuptools is one of the most downloaded packages on PyPI: > http://pypi.python.org/pypi/setuptools > http://blog.delaguardia.com.mx/tags/pypi > [3] about one fifth of Debian users who install python2.5 also install > python-pkg-resources: > http://qa.debian.org/popcon.php?package=python-setuptools > http://qa.debian.org/popcon.php?package=python2.5 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vinay_sajip at yahoo.co.uk Wed Oct 7 19:34:02 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Oct 2009 17:34:02 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: Antoine Pitrou pitrou.net> writes: > > Vinay Sajip yahoo.co.uk> writes: > > > > >>> "%0#8x" % 0x1234 > > '0x001234' > > >>> "{0:0>#8x}".format(0x1234) > > '000x1234' > > Apart from the sheer unreadability of the {}-style format string, the result > looks rather unexpected from a human being's point of view. > > (in those situations, I would output the 0x manually anyway, such as: > >>> "0x%06x" % 0x1234 > '0x001234' > >>> "0x{:06x}".format(0x1234) > '0x001234' > ) > Well of course, but I asked the question in the context of providing an *automatic* converter from %-format strings to {}-format. At the moment, it doesn't seem like a 100%-faithful automatic conversion is feasible. Regards, Vinay Sajip From ziade.tarek at gmail.com Wed Oct 7 19:37:07 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 7 Oct 2009 19:37:07 +0200 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4ACCCF9B.5060303@egenix.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> Message-ID: <94bdd2610910071037h19b014c3i67bf402169eada9f@mail.gmail.com> On Wed, Oct 7, 2009 at 7:27 PM, M.-A. Lemburg wrote: > Zooko O'Whielacronx wrote: >> +1 >> >> For a large number of people [1, 2, 3], setuptools is already a >> critical part of Python. ?Make it official. ?Let everyone know that >> future releases of Python will not break setuptools/Distribute, and >> that they can rely on backwards-compatibility with the myriad existing >> packages. ?Make the next release of the distutils standard lib module >> be Distribute. >> >> (Perhaps some people will complain, but they can channel their energy >> into improving the new distutils.) > > We've had that discussion in past and a few times. setuptools/Distribute > is not distutils, just an add-on (and there are others as well). > > There are good concepts in setuptools, but the implementation is just > not suitable for stdlib integration. > > Hopefully, Distribute can fix the hacks, trim down the number of > features to what people really use, add a proper uninstall command and > then we can have the add-to-the-stdlib discussion again. > > Having more competition will also help, e.g. ActiveState's PyPM looks > promising (provided they choose to open-source it) and then there's > pip. > > Things look promising, indeed. > 100% Agreed ! And before considering adding "bits" of Distribute (or Setuptools) in Distutils, we need to finish our work on PEP 376 and PEP 386, and the changes we've planned for PEP 341. The work we've done there so far in these PEPs is basically trying to have a consensus on the missing standard that is lacking in Distutils, and that was partially solved by Setuptools. I have good hopes that these PEP will be finished before 2.7 is out. Notice that PEP 382 (namespaced packages) is already a great step forward. (by the way, I thought this one was accepted, shouldn't its status updated ?) Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From zookog at gmail.com Wed Oct 7 19:37:14 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 7 Oct 2009 11:37:14 -0600 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4ACCCF9B.5060303@egenix.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> Message-ID: Thanks for the reply, MAL. How would we judge whether Distribute is ready for inclusion in the Python standard lib? Maybe if it has a few more releases, leaving a trail of "closed: fixed" issue tickets behind it? Regards, Zooko From zookog at gmail.com Wed Oct 7 19:41:06 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 7 Oct 2009 11:41:06 -0600 Subject: [Python-Dev] Python byte-compiled and optimized code In-Reply-To: References: Message-ID: You might be interested in the new PYTHONDONTWRITEBYTECODE environment variable supported as of Python 2.6. I personally think it is a great improvement. :-) Regards, Zooko From barry at python.org Wed Oct 7 19:46:04 2009 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2009 13:46:04 -0400 Subject: [Python-Dev] Package install failures in 2.6.3 In-Reply-To: <4ACCABC2.6030402@uva.nl> References: <4ACCABC2.6030402@uva.nl> Message-ID: <15834B11-EA17-4D9B-B7BB-EBFF25BD5965@python.org> On Oct 7, 2009, at 10:54 AM, Koen van de Sande wrote: > If there is going to be any quick 2.6.4 release, can you consider a > fix to the building of extensions under Windows ( http://bugs.python.org/issue4120 > )? > Extensions built without this patch applied will not work if the > MSVC9 runtimes are installed locally (in the Python folder) instead > of system-wide (WinSxS). > PIL and Matplotlib now use this patch when building extensions. But > many others (e.g. PyGame) still create problematic extensions. I'm not in favor of including this in 2.6.4 since this wasn't a regression in 2.6.3. However if Martin von Loewis approves of the patch and feels strongly that it should go in 2.6.4, I'll allow it. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mal at egenix.com Wed Oct 7 20:05:27 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 20:05:27 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: References: <4AB67430.2060306@egenix.com> <4AC07309.1010608@egenix.com> Message-ID: <4ACCD867.1080501@egenix.com> Zooko O'Whielacronx wrote: > Dear MAL and python-dev: > > I failed to explain the problem that users are having. I will try > again, and this time I will omit my ideas about how to improve things > and just focus on describing the problem. > > Some users are having trouble using Python packages containing binary > extensions on Linux. I want to provide such binary Python packages > for Linux for the pycryptopp project > (http://allmydata.org/trac/pycryptopp ) and the zfec project > (http://allmydata.org/trac/zfec ). I also want to make it possible > for users to install the Tahoe-LAFS project (http://allmydata.org ) > without having a compiler or Python header files. (You'd be surprised > at how often Tahoe-LAFS users try to do this on Linux. Linux is no > longer only for people who have the knowledge and patience to compile > software themselves.) Tahoe-LAFS also depends on many packages that > are maintained by other people and are not packaged or distributed by > me -- pyOpenSSL, simplejson, etc.. > > There have been several hurdles in the way that we've overcome, and no > doubt there will be more, but the current hurdle is that there are two > "formats" for Python extension modules that are used on Linux -- UCS2 > and UCS4. If a user gets a Python package containing a compiled > extension module which was built for the wrong UCS2/4 setting, he will > get mysterious (to him) "undefined symbol" errors at import time. Zooko, I really fail to see the reasoning here: Why would people who know how to build their own Python interpreter on Linux and expect it to work like the distribution-provided one, have a problem looking up the distribution-used configuration settings ? This is like compiling your own Linux kernel without using the same configuration as the distribution kernel and still expecting the distribution kernel modules to load without problems. Note that this has nothing to do with compiling your own Python extensions. Python's distutils will automatically use the right settings for compiling those, based on the configuration of the Python interpreter used for running the compilation - which will usually be the distribution interpreter. Your argument doesn't really live up to the consequences of switching to UCS4. Just as data-point: eGenix has been shipping binaries for Python packages for several years and while we do occasionally get reports about UCS2/UCS4 mismatches, those are really in the minority. I'd also question using the UCS4 default only on Linux. If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. However, before we can make such a decision, we need more data about the consequences. That is: * memory footprint changes * performance changes For both Python 2.x and 3.x. After all, UCS4 uses twice as much memory for all Unicode objects as UCS2. Since Python 3.x uses Unicode for all strings, I'd expect such a change to have more impact there. We'd also need to look into possible problems with different compilers using different wchar_t sizes on the same platform (I doubt that there are any). On Windows, the default is fixed since Windows uses UTF-16 for everything Unicode, so UCS2 will for a long time be the only option on that platform. That said, it'll take a while for distributions to upgrade, so you're always better off getting the tools you're using to deal with the problem for you and your users, since those are easier to upgrade. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Wed Oct 7 20:09:39 2009 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2009 14:09:39 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: <4ACCCF29.7030003@scottdial.com> References: <4ACCCF29.7030003@scottdial.com> Message-ID: <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> On Oct 7, 2009, at 1:26 PM, Scott Dial wrote: > I suspect this release is primarily to quench the problems with > distutils, but.. > > http://bugs.python.org/issue5949 > > doesn't seem to have been addressed by you. And this seems like it > would > be another unfortunate loss of an opportunity. I want us to be very careful about 2.6.4. This isn't a normal bug fix release, it's specifically there to remove the brown paper bag of 2.6.3 from our heads. So let's be conservative and fix this one in 2.6.5. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mal at egenix.com Wed Oct 7 20:13:27 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 20:13:27 +0200 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> Message-ID: <4ACCDA47.6010605@egenix.com> Zooko O'Whielacronx wrote: > Thanks for the reply, MAL. > > How would we judge whether Distribute is ready for inclusion in the > Python standard lib? Maybe if it has a few more releases, leaving a > trail of "closed: fixed" issue tickets behind it? I guess it'll just have to take the usual path of new modules/packages for the stdlib... http://www.python.org/dev/peps/pep-0002/ (the PEP is a bit outdated, but it still provides a good basis for the process) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From eric at trueblade.com Wed Oct 7 20:21:01 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 07 Oct 2009 14:21:01 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <4ACCDC0D.2080800@trueblade.com> Antoine Pitrou wrote: > Vinay Sajip yahoo.co.uk> writes: >>>>> "%0#8x" % 0x1234 >> '0x001234' >>>>> "{0:0>#8x}".format(0x1234) >> '000x1234' > > Apart from the sheer unreadability of the {}-style format string, the result > looks rather unexpected from a human being's point of view. > > (in those situations, I would output the 0x manually anyway, such as: >>>> "0x%06x" % 0x1234 > '0x001234' >>>> "0x{:06x}".format(0x1234) > '0x001234' > ) "#" formatting was added to int.__format__ in order to support this case: >>> format(10, '#6x') ' 0xa' Without '#', there's no way to specify a field width but still have the '0x' up against the digits (at least not without generating an intermediate result and figuring out the width manually). The fact that it works in combination with '0' or '>' (not sure which one makes it unreadable to you) wasn't really the point of the feature. Eric. From solipsis at pitrou.net Wed Oct 7 20:25:19 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2009 18:25:19 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?please_consider_changing_--enable-unicode_?= =?utf-8?q?default_to=09ucs4?= References: <4AB67430.2060306@egenix.com> Message-ID: Zooko O'Whielacronx gmail.com> writes: > > I accidentally sent this letter just to MAL when I intended it to > python-dev. Please read it, as it explains why the issue I'm raising > is not just the "we should switch to ucs4 because it is better" issue > that was previously settled by GvR. For what it's worth, with stringbench under py3k, an UCS2 build is roughly 8% faster than an UCS4 build (190 s. total against 206 s.). From pje at telecommunity.com Wed Oct 7 20:57:10 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Oct 2009 14:57:10 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4ACCCE44.8050905@scottdial.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCE44.8050905@scottdial.com> Message-ID: <20091007185722.D35B23A407A@sparrow.telecommunity.com> I'm not really sure how to reply to your email, since it seems to be based on several major misunderstandings. So, just a few key points: * Distribute 0.6.x is a stable maintenance branch, much like setuptools 0.6 * Distribute 0.7 is vaporware, very much like setuptools 0.7 * Packages using distribute 0.6.x still say "import setuptools" * Suggesting that the stdlib include the *other guys' code* is not "political bickering" - it's a positive proposal for moving forward that eliminates both technical and political obstacles. If you thought I was being sarcastic or ironic, you are mistaken. At 01:22 PM 10/7/2009 -0400, Scott Dial wrote: >P.J. Eby wrote: > > At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote: > >> suggest nobody hold their breath waiting for setuptools 0.7. > > > > I've never suggested or implied otherwise. > > > > But, if you like Distribute so much, why not just add it directly to the > > stdlib? ;-) > > > > There are many wins to be had from such a move, not the least of which > > is that it allows something to go into the stdlib that isn't > > (branding-wise) associated with me or setuptools, and lets it become > > owned/supported by an even wider community. > >I think the biggest problem here is that the brand ("setuptools") is so >ingrained in the world that someone releasing something >setuptools-but-not-setuptools ("Distribute") is at a serious >disadvantage. Your insistence on retaining the name "setuptools" for >yourself and your vapor-ware only harms the community at-large. > >Even experienced developers are unaware of Distribute's existence.. I >was entirely unaware until yourself and PJE got in a bickering exchange >on Python-Dev this past month. The extremely high visibility of your >stale package compared to their non-stale package is quite unfortunate. >You are free to say, "that is their problem," but you are not free to >say, "it is not my fault." > > > AFAIK, the only reason they've had multiple releases of it is to > > address the issues with its hijacking of setuptools; in a stdlib > > version all that stuff could be dropped. Otherwise, it'd already be > > "mature". > >IOW, you acknowledge that your own position and requiring them to >tolerate the parallel existence (in the world) of setuptools harms >Distribute. I fail to see how this relates to being in the stdlib. As >long as it is not called "setuptools" and packages in the world say >"import setuptools", then there are conflicts they will be forced to >managed. And moreover, as long as a stale, perhaps buggy version of >setuptools is the compatibility model they must emulate, they will have >a hard time coexisting. > > > Less political bickering, and the some of the technical results I > > hoped for all along are achieved. Yay, open source. > >And yet, political bickering seems to be all you are good for in this case. > > > >-- >Scott Dial >scott at scottdial.com >scodial at cs.indiana.edu >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com From ronaldoussoren at mac.com Wed Oct 7 21:13:58 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 07 Oct 2009 21:13:58 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: <4ACCD867.1080501@egenix.com> References: <4AB67430.2060306@egenix.com> <4AC07309.1010608@egenix.com> <4ACCD867.1080501@egenix.com> Message-ID: On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: > > > If we do go for a change, we should use sizeof(wchar_t) > as basis for the new default - on all platforms that > provide a wchar_t type. I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's that deal with Unicode text use ucs16. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From pje at telecommunity.com Wed Oct 7 21:35:18 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Oct 2009 15:35:18 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4ACCCF9B.5060303@egenix.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> Message-ID: <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> At 07:27 PM 10/7/2009 +0200, M.-A. Lemburg wrote: >Having more competition will also help, e.g. ActiveState's PyPM looks >promising (provided they choose to open-source it) and then there's >pip. Note that both PyPM and pip use setuptools as an important piece of their implementation (as does buildout), so they are technically the competition of easy_install, rather than setuptools per se. IOW, putting setuptools in the stdlib wouldn't be declaring a victor in the installation tools competition, it'd simply be providing infrastructure for (present and future) tools to build on. From fuzzyman at voidspace.org.uk Wed Oct 7 21:38:20 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 07 Oct 2009 20:38:20 +0100 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> Message-ID: <4ACCEE2C.4050308@voidspace.org.uk> P.J. Eby wrote: > At 07:27 PM 10/7/2009 +0200, M.-A. Lemburg wrote: >> Having more competition will also help, e.g. ActiveState's PyPM looks >> promising (provided they choose to open-source it) and then there's >> pip. > > Note that both PyPM and pip use setuptools as an important piece of > their implementation (as does buildout), so they are technically the > competition of easy_install, rather than setuptools per se. > > IOW, putting setuptools in the stdlib wouldn't be declaring a victor > in the installation tools competition, it'd simply be providing > infrastructure for (present and future) tools to build on. > I'd like to see PEP 370 (user site-packages folders) compatibility as a pre-condition of moving Distribute (or components of it) into the standard library. There are some issues around PEP 370 for alternative implementations that I'd like to address as a compatibility fix for Python 2.6 as well. :-) Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From ronaldoussoren at mac.com Wed Oct 7 21:42:08 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 07 Oct 2009 21:42:08 +0200 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: Message-ID: On 7 Oct, 2009, at 20:53, Brett Cannon wrote: > I just tried building out of svn and a ton of tests that rely on > urllib failed because the _scproxy module wasn't built and it > unconditionally imports it under darwin. Turns out that it requires > the Mac toolbox glue to be built which I always skip since I don't > care about it. > > I am fairly certain this toolbox glue dependency for urllib is a > regression as I used to always be able to get networking working w/o > the Mac-specific modules. Bummer. _scproxy was introduced in 2.6.3 to fix a crash in the ctypes based code it replaces, as well as very vague failures in the same code on OSX 10.5 Server on PPC machines. Luckily the fix is easy enough: move some code in setup.py from the block that checks for "not --disable-toolbox-glue", the _scproxy module has no dependencies on the toolbox glue. The attached patch should fix the issue, Ronald > > -Brett > > On Wed, Oct 7, 2009 at 10:18, Barry Warsaw wrote: > Hello everyone. > > The source tarballs and Windows installers for Python 2.6.4rc1 are > now available: > > http://www.python.org/download/releases/2.6.4/ > > Please download them, install them, and try to use them with your > projects and environments. Let us know if you encounter any > problems with them. Hopefully we can avoid the situation with 2.6.3 > having such critical bugs. > > 2.6.4 final is planned for 18-October. > > Cheers, > -Barry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: compile-scproxy-unconditional.txt URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From brett at python.org Wed Oct 7 20:12:15 2009 From: brett at python.org (Brett Cannon) Date: Wed, 7 Oct 2009 11:12:15 -0700 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> Message-ID: On Wed, Oct 7, 2009 at 10:37, Zooko O'Whielacronx wrote: > Thanks for the reply, MAL. > > How would we judge whether Distribute is ready for inclusion in the > Python standard lib? Maybe if it has a few more releases, leaving a > trail of "closed: fixed" issue tickets behind it? > When Tarek says it's ready. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Oct 7 22:13:45 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 22:13:45 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: References: <4AB67430.2060306@egenix.com> <4AC07309.1010608@egenix.com> <4ACCD867.1080501@egenix.com> Message-ID: <4ACCF679.3020109@egenix.com> Ronald Oussoren wrote: > > On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: >> >> >> If we do go for a change, we should use sizeof(wchar_t) >> as basis for the new default - on all platforms that >> provide a wchar_t type. > > I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's > that deal with Unicode text use ucs16. Is that true for non-Carbon APIs as well ? This is what I found on the web (in summary): Apple chose to go with UTF-16 at about the same time as Microsoft did and used sizeof(wchar_t) == 2 for Mac OS. When they moved to Mac OS X, they switched wchar_t to sizeof(wchar_t) == 4. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Wed Oct 7 22:21:59 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2009 20:21:59 +0000 (UTC) Subject: [Python-Dev] eggs now mandatory for pypi? References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> Message-ID: Arc Riley gmail.com> writes: > > Is the intention of Pypi really to turn it into a social networking site?? Sure, why not? It's not like there are enough social networking sites nowadays, are there? :) Regards Antoine. From ronaldoussoren at mac.com Wed Oct 7 22:24:10 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 07 Oct 2009 22:24:10 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: <4ACCF679.3020109@egenix.com> References: <4AB67430.2060306@egenix.com> <4AC07309.1010608@egenix.com> <4ACCD867.1080501@egenix.com> <4ACCF679.3020109@egenix.com> Message-ID: <8D09A004-31C0-445F-BCEF-6FCFFC5774AA@mac.com> On 7 Oct, 2009, at 22:13, M.-A. Lemburg wrote: > Ronald Oussoren wrote: >> >> On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: >>> >>> >>> If we do go for a change, we should use sizeof(wchar_t) >>> as basis for the new default - on all platforms that >>> provide a wchar_t type. >> >> I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix >> API's >> that deal with Unicode text use ucs16. > > Is that true for non-Carbon APIs as well ? > > This is what I found on the web (in summary): > > Apple chose to go with UTF-16 at about the same time as Microsoft did > and used sizeof(wchar_t) == 2 for Mac OS. When they moved to Mac OS X, > they switched wchar_t to sizeof(wchar_t) == 4. > Both Carbon and the modern APIs use UTF-16. What I don't quite get in the UTF-16 vs. UTF-32 discussion is why UTF-32 would be useful, because if you want to do generic Unicode processing you have to look at sequences of composed characters (base characters + composing marks) anyway instead of separate code points. Not that I'm a unicode expert in any way... Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Wed Oct 7 22:23:57 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Oct 2009 20:23:57 +0000 (UTC) Subject: [Python-Dev] a new setuptools release? References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCE44.8050905@scottdial.com> <20091007185722.D35B23A407A@sparrow.telecommunity.com> Message-ID: P.J. Eby telecommunity.com> writes: > * Distribute 0.6.x is a stable maintenance branch, much like setuptools 0.6 I'm new to this particular discussion so feel free to correct me if I'm wrong, but ISTM that Distribute 0.6.x differs markedly from setuptools 0.6 in that the former has an active maintainer and the latter does not, or am I wrong about that? Regards, Vinay Sajip From sridharr at activestate.com Wed Oct 7 22:43:20 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Wed, 07 Oct 2009 13:43:20 -0700 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> Message-ID: On Wed, 07 Oct 2009 12:35:18 -0700, P.J. Eby wrote: > At 07:27 PM 10/7/2009 +0200, M.-A. Lemburg wrote: > Having more competition will also help, e.g. ActiveState's PyPM looks > promising (provided they choose to open-source it) and then there's > pip. > Note that both PyPM and pip use setuptools as an important piece of > their implementation (as does buildout), so they are technically the > competition of easy_install, rather than setuptools per se. > IOW, putting setuptools in the stdlib wouldn't be declaring a victor in > the installation tools competition, it'd simply be providing > infrastructure for (present and future) tools to build on. PyPM client relies on pkg_resources *only*[1]. Specifically for 1) the version comparison algorithm: $ grep pkg_resources pypm/client/version.py import pkg_resources return cmp(pkg_resources.parse_version(pkg1.version), pkg_resources.parse_version(pkg2.version)) 2) parsing the "install_requires" string: $ grep parse pypm/client/dependency.py return [pkg_resources.Requirement.parse(reqstring) Both these features are definitely worthy of addition to stdlib but only after *standardizing* them (like PEP 376 does with .egg-info structure and files list). Now that Distribute is getting some visibility, it will be extremely good for the community to add distribute-0.7 (which would include the above two features apart from others) to the stdlib. -srid *** [1] The backend code (our mirroring component) also uses setuptools.package_index From mal at egenix.com Wed Oct 7 22:46:19 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 22:46:19 +0200 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> Message-ID: <4ACCFE1B.5020408@egenix.com> P.J. Eby wrote: > At 07:27 PM 10/7/2009 +0200, M.-A. Lemburg wrote: >> Having more competition will also help, e.g. ActiveState's PyPM looks >> promising (provided they choose to open-source it) and then there's >> pip. > > Note that both PyPM and pip use setuptools as an important piece of > their implementation (as does buildout), so they are technically the > competition of easy_install, rather than setuptools per se. > > IOW, putting setuptools in the stdlib wouldn't be declaring a victor in > the installation tools competition, it'd simply be providing > infrastructure for (present and future) tools to build on. I'm sure that some implementation of some of the concepts of setuptools will end up in the stdlib - in a well-integrated and distutils compatible form. Perhaps we can even find a way to remove the need for .pth files and long sys.path lists :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From brett at python.org Wed Oct 7 21:46:32 2009 From: brett at python.org (Brett Cannon) Date: Wed, 7 Oct 2009 12:46:32 -0700 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: Message-ID: On Wed, Oct 7, 2009 at 12:42, Ronald Oussoren wrote: > > On 7 Oct, 2009, at 20:53, Brett Cannon wrote: > > I just tried building out of svn and a ton of tests that rely on urllib > failed because the _scproxy module wasn't built and it unconditionally > imports it under darwin. Turns out that it requires the Mac toolbox glue to > be built which I always skip since I don't care about it. > I am fairly certain this toolbox glue dependency for urllib is a regression > as I used to always be able to get networking working w/o the Mac-specific > modules. > > > Bummer. _scproxy was introduced in 2.6.3 to fix a crash in the ctypes based > code it replaces, as well as very vague failures in the same code on OSX > 10.5 Server on PPC machines. > > Luckily the fix is easy enough: move some code in setup.py from the block > that checks for "not --disable-toolbox-glue", the _scproxy module has no > dependencies on the toolbox glue. > > The attached patch should fix the issue, > > Patch fixed it. Barry, can Ronald apply the patch? -Brett > Ronald > > > > > -Brett > > On Wed, Oct 7, 2009 at 10:18, Barry Warsaw wrote: > >> Hello everyone. >> >> The source tarballs and Windows installers for Python 2.6.4rc1 are now >> available: >> >> http://www.python.org/download/releases/2.6.4/ >> >> Please download them, install them, and try to use them with your projects >> and environments. Let us know if you encounter any problems with them. >> Hopefully we can avoid the situation with 2.6.3 having such critical bugs. >> >> 2.6.4 final is planned for 18-October. >> >> Cheers, >> -Barry >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Oct 7 23:01:12 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Oct 2009 17:01:12 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCE44.8050905@scottdial.com> <20091007185722.D35B23A407A@sparrow.telecommunity.com> Message-ID: <20091007210124.544F33A407A@sparrow.telecommunity.com> At 08:23 PM 10/7/2009 +0000, Vinay Sajip wrote: >P.J. Eby writes: > > > * Distribute 0.6.x is a stable maintenance branch, much like setuptools 0.6 > >I'm new to this particular discussion so feel free to correct me if I'm wrong, >but ISTM that Distribute 0.6.x differs markedly from setuptools 0.6 >in that the >former has an active maintainer and the latter does not, or am I wrong about >that? That depends on how you define "active". I haven't resigned, nor do I plan to. There's simply a long mean time between releases at the moment, and some people feel that it's too long. That's certainly their prerogative, but it doesn't mean I have to agree with such characterizations. From ziade.tarek at gmail.com Wed Oct 7 23:02:22 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 7 Oct 2009 23:02:22 +0200 Subject: [Python-Dev] a new setuptools release? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> Message-ID: <94bdd2610910071402p4399ec36y20e966bbe28d70bf@mail.gmail.com> On Wed, Oct 7, 2009 at 10:43 PM, Sridhar Ratnakumar wrote: > PyPM client relies on pkg_resources *only*[1]. Specifically for > > 1) the version comparison algorithm: [...] > > 2) parsing the "install_requires" string: > [...] > > Both these features are definitely worthy of addition to stdlib but only > after *standardizing* them (like PEP 376 does with .egg-info structure and > files list). Now that Distribute is getting some visibility, it will be > extremely good for the community to add distribute-0.7 (which would include > the above two features apart from others) to the stdlib. Three remarks : - Distutils contains already a version comparison algorithm, and the version comparison algorithm is being reworked in PEP 386. The goal is to provide a version scheme that allows adding a field like install_requires in PEP 341, and would allow package manager to compare versions. - The roadmap of Distribute includes the splitting you are mentioning - I don't think that adding Distribute-0.7 to the stdlib it the best plan : I'd rather see bits of them included in Distutils, with Distribute being an incubator because its release cycle is shorter. I will push on python-dev a detailed roadmap of Distutils we had in mind and what it means for Python 2.7. Give me a day or so to prepare it. Regards Tarek From pje at telecommunity.com Wed Oct 7 23:06:48 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 07 Oct 2009 17:06:48 -0400 Subject: [Python-Dev] a new setuptools release? In-Reply-To: <4ACCFE1B.5020408@egenix.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <20091006211742.9E43E3A4116@sparrow.telecommunity.com> <4ACCCF9B.5060303@egenix.com> <20091007193533.8A4AE3A407A@sparrow.telecommunity.com> <4ACCFE1B.5020408@egenix.com> Message-ID: <20091007210700.BB3403A408F@sparrow.telecommunity.com> At 10:46 PM 10/7/2009 +0200, M.-A. Lemburg wrote: >P.J. Eby wrote: > > At 07:27 PM 10/7/2009 +0200, M.-A. Lemburg wrote: > >> Having more competition will also help, e.g. ActiveState's PyPM looks > >> promising (provided they choose to open-source it) and then there's > >> pip. > > > > Note that both PyPM and pip use setuptools as an important piece of > > their implementation (as does buildout), so they are technically the > > competition of easy_install, rather than setuptools per se. > > > > IOW, putting setuptools in the stdlib wouldn't be declaring a victor in > > the installation tools competition, it'd simply be providing > > infrastructure for (present and future) tools to build on. > >I'm sure that some implementation of some of the concepts of >setuptools will end up in the stdlib - in a well-integrated and >distutils compatible form. > >Perhaps we can even find a way to remove the need for .pth files >and long sys.path lists :-) Setuptools doesn't *require* either of those now. Installing in flat, distutils-compatible format has been supported for years, and .pth files are only needed for good namespace package support. But PEP 382 makes namespace packages possible without .pth files, so even that could go away. From barry at python.org Wed Oct 7 23:09:21 2009 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2009 17:09:21 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: Message-ID: <65922EB3-DD7A-49E1-B482-462E35D1F890@python.org> On Oct 7, 2009, at 3:46 PM, Brett Cannon wrote: > > > On Wed, Oct 7, 2009 at 12:42, Ronald Oussoren > wrote: > > On 7 Oct, 2009, at 20:53, Brett Cannon wrote: > >> I just tried building out of svn and a ton of tests that rely on >> urllib failed because the _scproxy module wasn't built and it >> unconditionally imports it under darwin. Turns out that it requires >> the Mac toolbox glue to be built which I always skip since I don't >> care about it. >> >> I am fairly certain this toolbox glue dependency for urllib is a >> regression as I used to always be able to get networking working w/ >> o the Mac-specific modules. > > Bummer. _scproxy was introduced in 2.6.3 to fix a crash in the > ctypes based code it replaces, as well as very vague failures in the > same code on OSX 10.5 Server on PPC machines. > > Luckily the fix is easy enough: move some code in setup.py from the > block that checks for "not --disable-toolbox-glue", the _scproxy > module has no dependencies on the toolbox glue. > > The attached patch should fix the issue, > > > Patch fixed it. Barry, can Ronald apply the patch? Since this is a 2.6.3 regression, yes. Thanks, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mal at egenix.com Wed Oct 7 23:21:09 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Oct 2009 23:21:09 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: <8D09A004-31C0-445F-BCEF-6FCFFC5774AA@mac.com> References: <4AB67430.2060306@egenix.com> <4AC07309.1010608@egenix.com> <4ACCD867.1080501@egenix.com> <4ACCF679.3020109@egenix.com> <8D09A004-31C0-445F-BCEF-6FCFFC5774AA@mac.com> Message-ID: <4ACD0645.8060207@egenix.com> Ronald Oussoren wrote: > > On 7 Oct, 2009, at 22:13, M.-A. Lemburg wrote: > >> Ronald Oussoren wrote: >>> >>> On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: >>>> >>>> >>>> If we do go for a change, we should use sizeof(wchar_t) >>>> as basis for the new default - on all platforms that >>>> provide a wchar_t type. >>> >>> I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's >>> that deal with Unicode text use ucs16. >> >> Is that true for non-Carbon APIs as well ? >> >> This is what I found on the web (in summary): >> >> Apple chose to go with UTF-16 at about the same time as Microsoft did >> and used sizeof(wchar_t) == 2 for Mac OS. When they moved to Mac OS X, >> they switched wchar_t to sizeof(wchar_t) == 4. >> > > Both Carbon and the modern APIs use UTF-16. Thanks for that data point. So UTF-16 would be the more natural choice on Mac OS X, despite the choice of sizeof(wchar_t). > What I don't quite get in the UTF-16 vs. UTF-32 discussion is why UTF-32 > would be useful, because if you want to do generic Unicode processing > you have to look at sequences of composed characters (base characters + > composing marks) anyway instead of separate code points. Not that I'm a > unicode expert in any way... Very true. It's one of the reasons why I'm not much of a UCS4-fan - it only helps with surrogates and that's about it. Combining characters, various types of control code points (e.g. joiners, bidirectional marks, breaks, non-breaks, annotations) context sensitive casing, bidirectional marks and other such features found in scripts cause very similar problems - often much harder to solve, since they are not as easily identifiable as surrogate high and low code points. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Thu Oct 8 00:24:08 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2009 22:24:08 +0000 (UTC) Subject: [Python-Dev] PEP 389: argparse - new command line parsing module References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> <4ACC7B03.3040700@avl.com> Message-ID: Hrvoje Niksic avl.com> writes: > > Of course; simply use the >&- pseudo-redirection, which has been a > standard sh feature (later inherited by ksh and bash, but not csh) for > ~30 years. The error message is amusing, too: > > $ python -c 'print "foo"' >&- > close failed in file object destructor: > Error in sys.excepthook: > > Original exception was: Python 3 complains at startup and is a bit more explicit: $ ./python -c '1' >&- Fatal Python error: Py_Initialize: can't initialize sys standard streams OSError: [Errno 9] Bad file descriptor Abandon Of course, if it is stderr that you explicitly close, you lose all reporting: $ ./python -c '1' 2>&- Abandon Regards Antoine. From nyamatongwe at gmail.com Thu Oct 8 00:55:35 2009 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Thu, 8 Oct 2009 09:55:35 +1100 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: <8D09A004-31C0-445F-BCEF-6FCFFC5774AA@mac.com> References: <4AB67430.2060306@egenix.com> <4AC07309.1010608@egenix.com> <4ACCD867.1080501@egenix.com> <4ACCF679.3020109@egenix.com> <8D09A004-31C0-445F-BCEF-6FCFFC5774AA@mac.com> Message-ID: <50862ebd0910071555m10cf794xd1949fe45dc48312@mail.gmail.com> Ronald Oussoren: > Both Carbon and the modern APIs use UTF-16. If Unicode size standardization is seen as sufficiently beneficial then UTF-16 would be more widely applicable than UTF-32. Unix mostly uses 8-bit APIs which are either explicitly UTF-8 (such as GTK+) or can accept UTF-8 when the locale is set to UTF-8. They don't accept UTF-32. It is possible that Unix could move towards UTF-32 but that hasn't been the case up to now and with both OS X and Windows being UTF-16, it is more likely that UTF-16 APIs will become more popular on Unix. Neil From p.f.moore at gmail.com Wed Oct 7 17:05:43 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2009 16:05:43 +0100 Subject: [Python-Dev] A new way to configure logging In-Reply-To: References: Message-ID: <79990c6b0910070805t5feacb34ice669ca5d0bc0546@mail.gmail.com> 2009/10/7 Vinay Sajip : > What's the general feeling here about this proposal? All comments and > suggestions will be gratefully received. +1 One option I would have found useful in some code I wrote would be to extend the configuration - class DictConfigurator: ... def extend(self, moreconfig): import copy more = copy.deepcopy(moreconfig) # Not sure if this is needed? self.config.update(more) Paul. From rhamph at gmail.com Thu Oct 8 02:10:25 2009 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 7 Oct 2009 18:10:25 -0600 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: References: Message-ID: On Sun, Sep 20, 2009 at 10:17, Zooko O'Whielacronx wrote: > On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: >> AFAIK, C extensions should fail loading when they have the wrong UCS2/4 setting. > > That would be an improvement! ?Unfortunately we instead get mysterious > misbehavior of the module, e.g.: > > http://bugs.python.org/setuptools/msg309 > http://allmydata.org/trac/tahoe/ticket/704#comment:5 The real issue here is getting confused because python's option is misnamed. We support UTF-16 and UTF-32, not UCS-2 and UCS-4. This means that when decoding UTF-8, any scalar value outside the BMP will be split into a pair of surrogates on UTF-16 builds; if we were using UCS-2 that'd be an error instead (and *nothing* would understand surrogates.) Yet we are getting an error here. However, if you look at the details you'll notice it's on a 6-byte UTF-8 code unit sequence, corresponding in the second link to U+6E657770. Although the originally UTF-8 left open the possibility of including up to 31 bits (or U+7FFFFFFF), this was removed in RFC 3629 and is now strictly prohibited. The modern unicode character set itself also imposes that restriction. There is nothing beyond U+10FFFF. Nothing should create a such a high code point, and even if it happened internally a RFC 3629-conformant UTF-8 encoder must refuse to pass it through. Something more subtle must be going on. Possibly several bugs (such as a non-conformant encoder or garbage being misinterpreted as UTF-8). -- Adam Olsen, aka Rhamphoryncus From david.lyon at preisshare.net Thu Oct 8 02:17:34 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 07 Oct 2009 20:17:34 -0400 Subject: [Python-Dev] =?utf-8?q?eggs_now_mandatory_for_pypi=3F?= In-Reply-To: <4ACB6619.4080807@simplistix.co.uk> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> Message-ID: On Tue, 06 Oct 2009 16:45:29 +0100, Chris Withers wrote: > Well yeah, and the only sane way I can think to handle this is to have a > metadata file that gets uploaded with each distribution that covers all > these things (and the other things that other people need) and then have > the index (which would hopefully be PyPI on the whole) be queryable > along the lines of "give me a download url for a distribution named X > that is for Python 2.6, Win32, UCS4, PostGreSQL 8.5, BlagBlah"... Exactly. I'd like to see that and it sounds like a very simple and reasonable proposal. One could say that much of the setuptools cloud came about because of the lack of the queryable download url. Setuptools does a lot of work here to 'work-around' the ommission on pypi of a package download url. I'm just with you 100% on this... David From v+python at g.nevcal.com Thu Oct 8 03:46:44 2009 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 07 Oct 2009 18:46:44 -0700 Subject: [Python-Dev] A new way to configure logging In-Reply-To: References: Message-ID: <4ACD4484.2080409@g.nevcal.com> On approximately 10/7/2009 7:49 AM, came the following characters from the keyboard of Vinay Sajip: > In outline, the scheme I have in mind will look like this, in terms of the new > public API: > > class DictConfigurator: > def __init__(self, config): #config is a dict-like object (duck-typed) > import copy > self.config = copy.deepcopy(config) > > def configure(self): > # actually do the configuration here using self.config > > dictConfigClass = DictConfigurator > > def dictConfig(config): > dictConfigClass(config).configure() > > This allows easy replacement of DictConfigurator with a suitable subclass where > needed. Concept sounds good, and the idea of separating the syntax of the configuration file from the action of configuring is a clever way of avoiding the "syntax of the (day|platform|environment)" since everyone seems to invent new formats. So people that want to expose a text file to their users have the choice of syntaxes to expose, and then they do logCfg = readMyFavoriteSyntax( logCfgFile ) # or extract a subset of a larger config file for their project DictConfigurator( logCfg ) But DictConfigurator the name seems misleading... like you are configuring how dicts work, rather than how logs work. Maybe with more context this is not a problem, but if it is a standalone class, it is confusing what it does, by name alone. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From vinay_sajip at yahoo.co.uk Thu Oct 8 07:43:20 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 05:43:20 +0000 (UTC) Subject: [Python-Dev] A new way to configure logging References: <79990c6b0910070805t5feacb34ice669ca5d0bc0546@mail.gmail.com> Message-ID: Paul Moore gmail.com> writes: > One option I would have found useful in some code I wrote would be to > extend the configuration - > > class DictConfigurator: > ... > def extend(self, moreconfig): > import copy > more = copy.deepcopy(moreconfig) # Not sure if this is needed? > self.config.update(more) See my response to Paul Rudin's post about incremental configuration. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Oct 8 07:45:02 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 05:45:02 +0000 (UTC) Subject: [Python-Dev] A new way to configure logging References: <4ACD4484.2080409@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > But DictConfigurator the name seems misleading... like you are > configuring how dicts work, rather than how logs work. Maybe with more > context this is not a problem, but if it is a standalone class, it is > confusing what it does, by name alone. Of course the fully qualified name would be logging.config.DictConfigurator which does provide the requisite context, but if I think of/someone suggests a better name I'll certainly use it. From brett at python.org Wed Oct 7 20:53:04 2009 From: brett at python.org (Brett Cannon) Date: Wed, 7 Oct 2009 11:53:04 -0700 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: Message-ID: I just tried building out of svn and a ton of tests that rely on urllib failed because the _scproxy module wasn't built and it unconditionally imports it under darwin. Turns out that it requires the Mac toolbox glue to be built which I always skip since I don't care about it. I am fairly certain this toolbox glue dependency for urllib is a regression as I used to always be able to get networking working w/o the Mac-specific modules. -Brett On Wed, Oct 7, 2009 at 10:18, Barry Warsaw wrote: > Hello everyone. > > The source tarballs and Windows installers for Python 2.6.4rc1 are now > available: > > http://www.python.org/download/releases/2.6.4/ > > Please download them, install them, and try to use them with your projects > and environments. Let us know if you encounter any problems with them. > Hopefully we can avoid the situation with 2.6.3 having such critical bugs. > > 2.6.4 final is planned for 18-October. > > Cheers, > -Barry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Oct 8 08:11:23 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 06:11:23 +0000 (UTC) Subject: [Python-Dev] Logging, Unicode and sockets Message-ID: > Thanks to http://bugs.python.org/issue7077 I've noticed that the socket-based logging handlers - SocketHandler, DatagramHandler and SysLogHandler - aren't Unicode-aware and can break in the presence of Unicode messages. I'd like to fix this by giving these handlers an optional (encoding=None) parameter in their __init__, and then using this to encode on output. If no encoding is specified, is it best to use locale.getpreferredencoding(), sys.getdefaultencoding(), sys.getfilesystemencoding(), 'utf-8' or something else? On my system: >>> sys.getdefaultencoding() 'ascii' >>> sys.getfilesystemencoding() 'mbcs' >>> locale.getpreferredencoding() 'cp1252' which suggests to me that the locale.getpreferredencoding() should be the default. However, as I'm not a Unicode maven, any suggestions would be welcome. Regards, Vinay Sajip From mal at egenix.com Thu Oct 8 09:35:57 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 09:35:57 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> Message-ID: <4ACD965D.5080701@egenix.com> David Lyon wrote: > On Tue, 06 Oct 2009 16:45:29 +0100, Chris Withers > wrote: > >> Well yeah, and the only sane way I can think to handle this is to have a >> metadata file that gets uploaded with each distribution that covers all >> these things (and the other things that other people need) and then have >> the index (which would hopefully be PyPI on the whole) be queryable >> along the lines of "give me a download url for a distribution named X >> that is for Python 2.6, Win32, UCS4, PostGreSQL 8.5, BlagBlah"... > > Exactly. > > I'd like to see that and it sounds like a very simple and reasonable > proposal. > > One could say that much of the setuptools cloud came about because of > the lack of the queryable download url. Setuptools does a lot of work > here to 'work-around' the ommission on pypi of a package download url. I think you two have missed my email to PJE. In summary: PyPI already does have mechanism for querying download URLs. http://wiki.python.org/moin/PyPiXmlRpc It only lacks some more meta-data that would be required to match up the user's configuration with the one used in the uploaded URLs. And it would be handy to have a mechanism to just upload the URL data rather than the whole package. Looking at distutils' upload command, PyPI currently appears to only support uploads of the package file itself. Basically, most of the mechnisms are already there, they just need some extra care. BTW: Who would I need to contact for the PyPI side to work out an upload_url distutils command ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Oct 8 09:47:03 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 09:47:03 +0200 Subject: [Python-Dev] please consider changing --enable-unicode default to ucs4 In-Reply-To: References: Message-ID: <4ACD98F7.6060700@egenix.com> Adam Olsen wrote: > On Sun, Sep 20, 2009 at 10:17, Zooko O'Whielacronx wrote: >> On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: >>> AFAIK, C extensions should fail loading when they have the wrong UCS2/4 setting. >> >> That would be an improvement! Unfortunately we instead get mysterious >> misbehavior of the module, e.g.: >> >> http://bugs.python.org/setuptools/msg309 >> http://allmydata.org/trac/tahoe/ticket/704#comment:5 I agree that a better error message would help. I'm just not sure how to achieve that. The error message you currently see gets generated by the dynamic linker trying to resolve a Python Unicode API symbol: the API names are mangled to assure that you cannot mix UCS2 interpreters and UCS4 extensions (and vice-versa). We could try to scan the linker error message for 'Py.*UCS.' and then replace the message with a more helpful one (in importdl.c), but I'm not sure how portable that is. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Thu Oct 8 10:31:40 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 10:31:40 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) Message-ID: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Here's a quick summary of the main things that are going to happen in Distutils, and Distribute, and a few words on virtualenv and pip. (there is much much more work going on, but I don't want to drown people with details) = Distutils = Distutils is a package manager and competes with OS package managers. This is a good thing because, unless you are developing a library or an application that will only run one specific system that has its own packaging system like Debian, you will be able to reach much more people. Of course the goal is to avoid making the work of a Debian packager (or any other OS that has a package manager) too hard. In other words, re-packaging a Distutils-based project should be easy and Distutils should not get in their way (or as less as possible). But right now Distutils is incomplete in many ways and we are trying to fix'em. == What's installed ? what's the installation format ? how to uninstall ? == First, it's an incomplete package manager : you can install a distribution using it, but there's no way to list installed distributions. Worst, you can't uninstall a distribution. PEP 376 resolves this, and once it's finished, the goal is to include the APIs described there into Distutils itself and into the pkgutil module in stdlib. Notice that there's an implementation at http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376 so people can see what we are talking about. Another problem that popped during the last years is the fact that, in the same site-packages, depending on the tool that was used to install a Distribution, and depending if this distribution uses Distutils or Setuptools, you can have different installation formats. End-users end up with zipped eggs (one file), unzipped eggs (one self-contained format in a directory) and regular Distutils (packages and modules in site-packages). And the Metadata are also located in many different places depending on the installation format used. That can't be. there's no point to keep various installation format in the *same* site-packages directory. PEP 376 also resolves this by describing a *unique* format that works for all. Once this is finished, Distutils will implement it by changing the install command accordingly. - Work left to do in PEP 376 : restrict its scope to a disk-based, file-based site-packages. - Goal: 2.7 / 3.2 == Dependencies == The other feature that makes a packaging system nice is dependencies. e.g. a way to list in a distribution, the distributions it requires to run. As a matter of fact, PEP 314 has introduced in the Metadata new fields for this purpose ("Requires", "Provides and "Obsoletes"). So, you can write things like "Requires: lxml >= 2.2.1", meaning that your distribution requires lxml 2.2.1 or a newer version to run. But this was just description fields and Distutils was not providing any feature based on these new fields. In fact, no third-party tool either provided a feature based on those fields. Setuptools provided "easy_install" a script that looks for the dependencies and install them, by querying the Python Package Index (PyPI). But this feature was implemented with its own metadata: you can add an "install_requires" option in the setup() call in setup.py, and it will end up in a "requires.txt" file at installation time that is located alongside the Metadata for you distribution. So the goal is to review PEP 314 and update the Metadata w.r.t. the setuptools feedback and community usage. Once it's done, Distutils will implement this new metadata version and promote its usage. Promoting its usage means that Distutils will provide some APIs to work with these APIs, like a version comparison algorithm. And while we're at it, we need to work out some inconsistency with the "Author" and "Maintainer" fields. (The latter doesn't exists in the Metadata but exists on setup.py side). - Work left to do in PEP 314 : finish PEP 386, finish the discussion on the "maintainer" field. - Goal: 2.7 / 3.2 == Version comparison == Once you provide dependency fields in the metadata, you need to provide a version scheme: a way to compare two versions. Distutils has two version comparison algorithms that are not used in its code and in only one place in the stdlib where it could be removed with no pain. One version scheme is "strict" and one is "loose". And Setuptools has another one, which is more heuristic (it will deal with any version string and compare it, wether it's wrong or not). PEP 386 goal is to describe a version scheme that can be used by all and if we can meet a consensus there, we can move on and add it as a reference in the update done in PEP 314, besides the dependencies fields. Then, in Distutils we can deprecate the existing version comparison algorithms and provide a new one based on PEP 386 and promote its usage. One very important point: we will not force the community to use the scheme described in PEP 386, but *there is* already a de-facto convention on version schemes at PyPI if you use Pip or easy_install, so let's have a documented standard for this, and a reference implementation in Distutils. There's an implementation at http://bitbucket.org/tarek/distutilsversion that is kept up-to-date with PEP 386. - Work left to do in PEP 386 : another round with the community - Goal: 2.7 / 3.2 == The fate of setup.py, and static metadata == Setup.py is a CLI to create distribution, install them etc. You can also use it to retrieve the metadata of a distribution. For example you can call "python setup.py --name" and the name will be displayed. That's fine. That's great for developers. But there's a major flaw: it's Python code. It's a problem because, depending on the complexity of this file, an OS packager that just wants to get the metadata for the platform he's working on, will run arbitrary code that mught do unwanted things (or even that light not work) So we are going to separate the metadata description from setup.py, in a static configuration file, that can be open and read by anyone without running any code. The only problem with this is the fact that some metadata fields might depend on the execution environment. For instance, once "Requires" is re-defined and re-introduced via PEP 314, we will have cases where "pywin32" will be a dependency to have only on win32 systems. So we've worked on that lately in Distutils-SIG and came up with a micro-language, based on a ConfigParser file, that allows writing metadata fields that depends on sys.platform etc. I won't detail the syntax here but the idea is that the interpretation of this file can be done with a vanilla Python without running arbitrary code. In other words : we will be able to get the metadata for a distribution without having to install it or to run any setup.py command. One use case is the ability to list all dependencies a distribution requires for a given platform, just by querying PyPI. So I am adding this in Distutils for 2.7. Of course setup.py stays, and this is backward compatible. - Work left to do : publish the final syntax, and do the implementation - Goal: 2.7 / 3.2 == The fate of bdist_* commands == During last Pycon summit we said that we would remove commands like bdist_rpm because Python is unable, due to its release cycle, to do a good work there. Here's an example: I have from time to time cryptic issues in the issue tracker from people from Fedora (or any rpm-based system), and I have all the pain in the world for these very specific problems to do the proper fix unless some RPM expert helps around. And by the time it's detected then fixed, it can be year(s) before it's available on their side. That's why, depending on the communities, commands like bdist_rpm are just totally ignored, and OS packager have their own tools. So the best way to handle this is to ask these communities to build their own tool and to encourage them to use Distutils as a basis for that. This does not concern bdist_* commands for win32 because those are very stable and don't change too much: Windows doesn't have a package manager that would require these commands to evolve with it. Anyways, when we said that we would remove bdist_rpm, this was very controversial because some people use it and love it. So what is going to happen is a status-quo: no bdist_* command will be removed but no new bdist_* command wil be added. That's why I've encouraged Andrew and Garry, that are working on a bdist_deb command, to keep it in the "stdeb" project, and eventually we will refer to it in the Distutils documentation if this bdist_deb comply with Distutils standard. It doesn't right now because it uses a custom version of the Distribution class (through Setuptools) that doesn't behave like Distutils' one anymore. For Distutils, I'll add some documentation explaining this, and a section that will list community-driven commands. - Work left to do : write the documentation - Goal: 2.7 / 3.2 = Distribute = I won't explain here again why we have forked, I think it's obvious to anyone here now. I'll rather explain what we are planning in Distribute and how it will interact with Distutils. Distribute has two branches: - 0.6.x : provides a Setuptools-0.6c9 compatible version - 0.7.x : will provide a refactoring == 0.6.x == Not "much" is going to happen here, we want this branch to be helpful to the community *today* by addressing the 40-or-so bugs that were found in Setuptools and never fixed. This is eventually happen soon because its development is fast : there are up to 5 commiters that are working on it very often (and the number grows weekly.) The biggest issue with this branch is that it is providing the same packages and modules setuptools does, and this requires some bootstrapping work where we make sure once Distribute is installed, all Distribution that requires Setuptools will continue to work. This is done by faking the metadata of Setuptools 0.6c9. That's the only way we found to do this. There's one major thing though: thanks to the work of Lennart, Alex, Martin, this branch supports Python 3, which is great to have to speed up Py3 adoption. The goal of the 0.6.x is to remove as much bugs as we can, and try if possible to remove the patches done on Distutils. We will support 0.6.x maintenance for years and we will promote its usage everywhere instead of Setuptools. Some new commands are added there, when they are helpful and don't interact with the rest. I am thinking about "upload_docs" that let you upload documentation to PyPI. The goal is to move it to Distutils at some point, if the documentation feature of PyPI stays and starts to be used. == 0.7.x == We've started to refactor Distribute with this roadmap in mind (and no, as someone said, it's not vaporware, we've done a lot already) - 0.7.x can be installed and used with 0.6.x - easy_install is going to be deprecated ! use Pip ! - the version system will be deprecated, in favor of the one in Distutils - no more Distutils monkey-patch that happens once you use the code (things like 'from distutils import cmd; cmd.Command = CustomCommand') - no more custom site.py (that is: if something misses in Python's site.py we'll add it there instead of patching it) - no more namespaced packages system, if PEP 381 (namespaces package support) makes it to 2.7 - The code is splitted in many packages and might be distributed under several distributions. - distribute.resources: that's the old pkg_resources, but reorganized in clean, pep-8 modules. This package will only contain the query APIs and will focus on being PEP 376 compatible. We will promote its usage and see if Pip wants to use it as a basis. And maybe PyPM once it's open source ? ( ). It will probably shrink a lot though, once the stdlib provides PEP 376 support. - distribute.entrypoints: that's the old pkg_resources entry points system, but on its own. it uses distribute.resources - distribute.index: that's package_index and a few other things. everything required to interact with PyPI. We will promote its usage and see if Pip wants to use it as a basis. - distribute.core (might be renamed to main): that's everything else, and uses the other packages. Goal: A first release before (or when) Python 2.7 / 3.2 is out. = Virtualenv and the multiple version support in Distribute = (I am not saying "We" here because this part was not discussed yet with everyone) Virtualenv allows you to create an isolated environment to install some distribution without polluting the main site-packages, a bit like a user site-packages. My opinion is that this tool exists only because Python doesn't support the installation of multiple versions for the same distributions. But if PEP 376 and PEP 386 support are added in Python, we're not far from being able to provide multiple version support with the help of importlib. Setuptools provided a multiple version support but I don't like its implementation and the way its works. I would like to create a new site-packages format that can contains several versions of the same distribution, and : - a special import system using importlib that would automatically pick the latest version, thanks to PEP 376. - an API to force at runtime a specific version (that would be located at the beginning of all imports, like __future__) - a layout that is compatible with the way OS packagers works with python packages Goal: a prototype asap (one was started under the "VSP" name (virtual site-packages) but not finished yet) Regards Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From mal at egenix.com Thu Oct 8 10:38:06 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 10:38:06 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <94bdd2610910080133y788cb0d6xd67aaf3b58ac48a0@mail.gmail.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <94bdd2610910080133y788cb0d6xd67aaf3b58ac48a0@mail.gmail.com> Message-ID: <4ACDA4EE.4050005@egenix.com> Tarek Ziad? wrote: > On Thu, Oct 8, 2009 at 9:35 AM, M.-A. Lemburg wrote: >> BTW: Who would I need to contact for the PyPI side to work out >> an upload_url distutils command ? > > I am not sure to understand. How upload_url will differ from the > register command > that let you upload metadata at PyPI ? The register command only uploads the package's meta-data. The upload command uploads a file and registers the file's details with PyPI as well. I'd like to add an upload_url command which only registers a file's details (including the URL, but without the actual file) with PyPI and perhaps extend the number of attributes sent to PyPI with more file configuration information (upload currently only sends the Python version). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Thu Oct 8 10:33:01 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 10:33:01 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACD965D.5080701@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> Message-ID: <94bdd2610910080133y788cb0d6xd67aaf3b58ac48a0@mail.gmail.com> On Thu, Oct 8, 2009 at 9:35 AM, M.-A. Lemburg wrote: > BTW: Who would I need to contact for the PyPI side to work out > an upload_url distutils command ? I am not sure to understand. How upload_url will differ from the register command that let you upload metadata at PyPI ? Tarek From ziade.tarek at gmail.com Thu Oct 8 10:59:48 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 10:59:48 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACDA4EE.4050005@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <94bdd2610910080133y788cb0d6xd67aaf3b58ac48a0@mail.gmail.com> <4ACDA4EE.4050005@egenix.com> Message-ID: <94bdd2610910080159r319b0fdtd02b779bba893e18@mail.gmail.com> On Thu, Oct 8, 2009 at 10:38 AM, M.-A. Lemburg wrote: > Tarek Ziad? wrote: >> On Thu, Oct 8, 2009 at 9:35 AM, M.-A. Lemburg wrote: >>> BTW: Who would I need to contact for the PyPI side to work out >>> an upload_url distutils command ? >> >> I am not sure to understand. How upload_url will differ from the >> register command >> that let you upload metadata at PyPI ? > > The register command only uploads the package's meta-data. > > The upload command uploads a file and registers the file's > details with PyPI as well. > > I'd like to add an upload_url command which only registers a > file's details (including the URL, but without the actual file) > with PyPI and perhaps extend the number of attributes sent to > PyPI with more file configuration information (upload currently > only sends the Python version). Sounds like an option to be added in upload. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source ?(#1, Oct 08 2009) >>>> Python/Zope Consulting and Support ... ? ? ? ?http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... ? ? ? ? ? ? http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... ? ? ? ?http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > ? eGenix.com Software, Skills and Services GmbH ?Pastor-Loeh-Str.48 > ? ?D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > ? ? ? ? ? Registered at Amtsgericht Duesseldorf: HRB 46611 > ? ? ? ? ? ? ? http://www.egenix.com/company/contact/ > -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From p.f.moore at gmail.com Thu Oct 8 10:56:58 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Oct 2009 09:56:58 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <79990c6b0910080156s2e540184r255624d2166cbae2@mail.gmail.com> 2009/10/8 Tarek Ziad? : > Here's a quick summary of the main things that are going to happen in > Distutils, and Distribute, and a few words on virtualenv and pip. > (there is much much more work going on, but I don't want to drown > people with details) Thanks for this summary. The overview was getting lost in all the details. > == What's installed ? what's the installation format ? how to uninstall ? == > > First, it's an incomplete package manager : you can install a > distribution using it, but there's no way to list installed > distributions. Worst, you can't uninstall a distribution. > > PEP 376 resolves this, and once it's finished, the goal is to include > the APIs described there into Distutils itself and into the pkgutil > module in stdlib. Notice that there's an implementation at > http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376 > so people can see what we are talking about. > > Another problem that popped during the last years is the fact that, in > the same site-packages, depending on the tool that was used to install > a Distribution, and depending if this distribution uses Distutils or > Setuptools, you can have different installation formats. > > End-users end up with zipped eggs (one file), unzipped eggs (one > self-contained format in a directory) and regular Distutils (packages > and modules in site-packages). And the Metadata are also located in > many different places depending on the installation format used. > > That can't be. there's no point to keep various installation format in > the *same* site-packages directory. > > PEP 376 also resolves this by describing a *unique* format that works > for all. Once this is finished, Distutils will implement it by > changing the install command accordingly. > > - Work left to do in PEP 376 : restrict its scope to a disk-based, > file-based site-packages. > - Goal: 2.7 / 3.2 +1 This addresses my biggest concern with the way PEP 376 was going (largely prompted by my suggestions, so I'll take the blame for this!) which was that PEP 376 was trying to define list/uninstall methods for package formats (i.e. zip files/eggs) that it couldn't install itself. I presume that when you refer to the restriction in scope, you do mean that PEP 375 will lose all support for zip files, as well as the generalised PEP 302 support I proposed? I still believe that special-casing zip files is wrong - PEP 302 deliberately abstracted the import mechanisms and built zipfile support on top of that, and I still believe that's a good thing. But expanding PEP 302 to cover the requirements of PEP 376 looked more and more inappropriate the more I wrote code to do it - what would be needed would be a *new* protocol on top of which zip-based formats could be layered. I'm quite happy for that to be outside the scope of PEP 376, though, as distutils itself doesn't support any such formats. The egg format One thing missing from your roadmap (unless I missed it) is the fate of the egg (zipfile) format. If it's to remain a valid option (bdist_egg and the like) then I'd assume that Distribute needs to be the place to develop it. One thing it will need is an equivalent to PEP 376 introspection and uninstall features. And *that* is where I believe a standard protocol belongs - developed based on the real use requirements for egg files, with a suitable file-based layer which can be added into distutils (somewhat like importlib does for PEP 302) once the protocol is agreed and standardised. Paul. From ziade.tarek at gmail.com Thu Oct 8 11:11:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 11:11:10 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910080156s2e540184r255624d2166cbae2@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <79990c6b0910080156s2e540184r255624d2166cbae2@mail.gmail.com> Message-ID: <94bdd2610910080211q5ff4fd09wf62c44dd4a680a8f@mail.gmail.com> On Thu, Oct 8, 2009 at 10:56 AM, Paul Moore wrote: > > +1 > > This addresses my biggest concern with the way PEP 376 was going > (largely prompted by my suggestions, so I'll take the blame for this!) > which was that PEP 376 was trying to define list/uninstall methods for > package formats (i.e. zip files/eggs) that it couldn't install itself. > > I presume that when you refer to the restriction in scope, you do mean > that PEP 375 will lose all support for zip files, as well as the > generalised PEP 302 support I proposed? I still believe that > special-casing zip files is wrong - PEP 302 deliberately abstracted > the import mechanisms and built zipfile support on top of that, and I > still believe that's a good thing. But expanding PEP 302 to cover the > requirements of PEP 376 looked more and more inappropriate the more I > wrote code to do it - what would be needed would be a *new* protocol > on top of which zip-based formats could be layered. I'm quite happy > for that to be outside the scope of PEP 376, though, as distutils > itself doesn't support any such formats. Yes exactly so. > > The egg format > > One thing missing from your roadmap (unless I missed it) is the fate > of the egg (zipfile) format. If it's to remain a valid option > (bdist_egg and the like) then I'd assume that Distribute needs to be > the place to develop it. > One thing it will need is an equivalent to > PEP 376 introspection and uninstall features. And *that* is where I > believe a standard protocol belongs - developed based on the real use > requirements for egg files, with a suitable file-based layer which can > be added into distutils (somewhat like importlib does for PEP 302) > once the protocol is agreed and standardised. Distribute is planning to continue supporting the egg format, but will probably drop the zipped version. This is still under discussions though so don't take this as the final word. In any case, distribute.resources will probably add the missing bits on the top of a PEP 376 -compliant Distutils to provide support for any file querying Distutils will not provide (for installation and uninstallation) Tarek From cournape at gmail.com Thu Oct 8 11:42:28 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 8 Oct 2009 18:42:28 +0900 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <5b8d13220910080242q19efb5dbre3fae15462253bfb@mail.gmail.com> On Thu, Oct 8, 2009 at 5:31 PM, Tarek Ziad? wrote: > = Virtualenv and the multiple version support in Distribute = > > (I am not saying "We" here because this part was not discussed yet > with everyone) > > Virtualenv allows you to create an isolated environment to install > some distribution without polluting the > main site-packages, a bit like a user site-packages. > > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. I am really worried about this, because it may encourage people to use multiple versions as a bandaid to maintaining backward compatibility. At least with virtual-env, the problem is restricted to the user. Generalized multiple, side by side installation has been tried in many different contexts, and I have never seen a single one working and not bringing more problems that it solved. One core problem being the exponential number of combination (package A depends on B and C, B depends on one version of D, C on another version of D). Being able to install *some* libraries in multiple versions is OK, but generalizing is very dangerous IMHO. David From ronaldoussoren at mac.com Thu Oct 8 11:49:38 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 08 Oct 2009 11:49:38 +0200 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: <65922EB3-DD7A-49E1-B482-462E35D1F890@python.org> References: <65922EB3-DD7A-49E1-B482-462E35D1F890@python.org> Message-ID: <8374F9DB-487C-454E-AB31-453E388329AA@mac.com> On 7 Oct, 2009, at 23:09, Barry Warsaw wrote: > On Oct 7, 2009, at 3:46 PM, Brett Cannon wrote: > >> >> >> On Wed, Oct 7, 2009 at 12:42, Ronald Oussoren >> wrote: >> >> On 7 Oct, 2009, at 20:53, Brett Cannon wrote: >> >>> I just tried building out of svn and a ton of tests that rely on >>> urllib failed because the _scproxy module wasn't built and it >>> unconditionally imports it under darwin. Turns out that it >>> requires the Mac toolbox glue to be built which I always skip >>> since I don't care about it. >>> >>> I am fairly certain this toolbox glue dependency for urllib is a >>> regression as I used to always be able to get networking working w/ >>> o the Mac-specific modules. >> >> Bummer. _scproxy was introduced in 2.6.3 to fix a crash in the >> ctypes based code it replaces, as well as very vague failures in >> the same code on OSX 10.5 Server on PPC machines. >> >> Luckily the fix is easy enough: move some code in setup.py from the >> block that checks for "not --disable-toolbox-glue", the _scproxy >> module has no dependencies on the toolbox glue. >> >> The attached patch should fix the issue, >> >> >> Patch fixed it. Barry, can Ronald apply the patch? > > Since this is a 2.6.3 regression, yes. I've commited the patch (r75282) Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From ziade.tarek at gmail.com Thu Oct 8 11:59:12 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 11:59:12 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <5b8d13220910080242q19efb5dbre3fae15462253bfb@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <5b8d13220910080242q19efb5dbre3fae15462253bfb@mail.gmail.com> Message-ID: <94bdd2610910080259q31cf830fsea147f612a05ece9@mail.gmail.com> On Thu, Oct 8, 2009 at 11:42 AM, David Cournapeau wrote: > On Thu, Oct 8, 2009 at 5:31 PM, Tarek Ziad? wrote: > >> = Virtualenv and the multiple version support in Distribute = >> >> (I am not saying "We" here because this part was not discussed yet >> with everyone) >> >> Virtualenv allows you to create an isolated environment to install >> some distribution without polluting the >> main site-packages, a bit like a user site-packages. >> >> My opinion is that this tool exists only because Python doesn't >> support the installation of multiple versions for the same >> distributions. > > I am really worried about this, because it may encourage people to use > multiple versions as a bandaid to maintaining backward compatibility. > At least with virtual-env, the problem is restricted to the user. > > Generalized multiple, side by side installation ?has been tried in > many different contexts, and I have never seen a single one working > and not bringing more problems that it solved. One core problem being > the exponential number of combination (package A depends on B and C, B > depends on one version of D, C on another version of D). Being able to > install *some* libraries in multiple versions is OK, but generalizing > is very dangerous IMHO. The goal here is to be able to have a *single* instance of "Foo 1.2" on your system, managed by the admin sys. that can be shared and used for several python apps. And yes, I don't expect to have so many different versions of many distributions on a system, but just to provide a mechanism to allow it. But that's just an idea we had with some people, and it still requires a lot of work and brainstroming (even if the prototype kinda work) I should've mentioned that I've added it at the end of the roadmap because I had the feeling that something has to be done to reunite zc.buildout, virtualenv and Python one day. But consider this as the topic for the "fun sprint" we would have at Pycon, after we will have issued other points. If you are interested in this particular topic, I propose that we continue this discussion on distutils-SIG. Regards Tarek From kiorky at cryptelium.net Thu Oct 8 12:00:15 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 08 Oct 2009 12:00:15 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4ACDB82F.7010006@cryptelium.net> Tarek Ziad? a ?crit : > Here's a quick summary of the main things that are going to happen in > Distutils, and Distribute, and a few words on virtualenv and pip. > (there is much much more work going on, but I don't want to drown > people with details) > > = Distutils = > > Distutils is a package manager and competes with OS package managers. > This is a good thing because, unless you are developing a library or > an application that will only run one specific system that has its own > packaging system like Debian, you will be able to reach much more > people. Of course the goal is to avoid making the work of a Debian > packager (or any other OS that has a package manager) too hard. In > other words, re-packaging a Distutils-based project should be easy and > Distutils should not get in their way (or as less as possible). > > But right now Distutils is incomplete in many ways and we are trying to fix'em. > > == What's installed ? what's the installation format ? how to uninstall ? == > > First, it's an incomplete package manager : you can install a > distribution using it, but there's no way to list installed > distributions. Worst, you can't uninstall a distribution. > > PEP 376 resolves this, and once it's finished, the goal is to include > the APIs described there into Distutils itself and into the pkgutil > module in stdlib. Notice that there's an implementation at > http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376 > so people can see what we are talking about. > > Another problem that popped during the last years is the fact that, in > the same site-packages, depending on the tool that was used to install > a Distribution, and depending if this distribution uses Distutils or > Setuptools, you can have different installation formats. > > End-users end up with zipped eggs (one file), unzipped eggs (one > self-contained format in a directory) and regular Distutils (packages > and modules in site-packages). And the Metadata are also located in > many different places depending on the installation format used. > > That can't be. there's no point to keep various installation format in > the *same* site-packages directory. > > PEP 376 also resolves this by describing a *unique* format that works > for all. Once this is finished, Distutils will implement it by > changing the install command accordingly. > > - Work left to do in PEP 376 : restrict its scope to a disk-based, > file-based site-packages. > - Goal: 2.7 / 3.2 > > == Dependencies == > > The other feature that makes a packaging system nice is dependencies. > e.g. a way to list in a distribution, the distributions it requires to > run. As a matter of fact, PEP 314 has introduced in the Metadata new > fields for this purpose ("Requires", "Provides and "Obsoletes"). So, > you can write things like "Requires: lxml >= 2.2.1", meaning that your > distribution requires lxml 2.2.1 or a newer version to run. But this > was just description fields and Distutils was not providing any > feature based on these new fields. > > In fact, no third-party tool either provided a feature based on those > fields. Setuptools provided "easy_install" a script that looks for the > dependencies and install them, by querying the Python Package Index > (PyPI). But this feature was implemented with its own metadata: you > can add an "install_requires" option in the setup() call in setup.py, > and it will end up in a "requires.txt" file at installation time that > is located alongside the Metadata for you distribution. > > So the goal is to review PEP 314 and update the Metadata w.r.t. the > setuptools feedback and community usage. Once it's done, Distutils > will implement this new metadata version and promote its usage. > Promoting its usage means that Distutils will provide some APIs to > work with these APIs, like a version comparison algorithm. > > And while we're at it, we need to work out some inconsistency with the > "Author" and "Maintainer" fields. (The latter doesn't exists in the > Metadata but exists on setup.py side). > > - Work left to do in PEP 314 : finish PEP 386, finish the discussion > on the "maintainer" field. > - Goal: 2.7 / 3.2 > > == Version comparison == > > Once you provide dependency fields in the metadata, you need to > provide a version scheme: a way to compare two versions. Distutils has > two version comparison algorithms that are not used in its code and in > only one place in the stdlib where it could be removed with no pain. > One version scheme is "strict" and one is "loose". And Setuptools has > another one, which is more heuristic (it will deal with any version > string and compare it, wether it's wrong or not). > > PEP 386 goal is to describe a version scheme that can be used by all > and if we can meet a consensus there, we can move on and add > it as a reference in the update done in PEP 314, besides the > dependencies fields. Then, in Distutils we can deprecate the existing > version > comparison algorithms and provide a new one based on PEP 386 and > promote its usage. > > One very important point: we will not force the community to use the > scheme described in PEP 386, but *there is* already a > de-facto convention on version schemes at PyPI if you use Pip or > easy_install, so let's have a documented standard for this, > and a reference implementation in Distutils. > > There's an implementation at > http://bitbucket.org/tarek/distutilsversion that is kept up-to-date > with PEP 386. > > - Work left to do in PEP 386 : another round with the community > - Goal: 2.7 / 3.2 > > == The fate of setup.py, and static metadata == > > Setup.py is a CLI to create distribution, install them etc. You can > also use it to retrieve the metadata of a distribution. For > example you can call "python setup.py --name" and the name will be > displayed. That's fine. That's great for developers. > > But there's a major flaw: it's Python code. It's a problem because, > depending on the complexity of this file, an OS packager that just > wants to get the metadata for the platform he's working on, will run > arbitrary code that mught do unwanted things (or even that light not > work) > > So we are going to separate the metadata description from setup.py, in > a static configuration file, that can be open and read by anyone > without > running any code. The only problem with this is the fact that some > metadata fields might depend on the execution environment. For > instance, once "Requires" is re-defined and re-introduced via PEP 314, > we will have cases where "pywin32" will be a dependency to have only > on win32 systems. > > So we've worked on that lately in Distutils-SIG and came up with a > micro-language, based on a ConfigParser file, that allows > writing metadata fields that depends on sys.platform etc. I won't > detail the syntax here but the idea is that the interpretation > of this file can be done with a vanilla Python without running arbitrary code. > > In other words : we will be able to get the metadata for a > distribution without having to install it or to run any setup.py > command. > One use case is the ability to list all dependencies a distribution > requires for a given platform, just by querying PyPI. > > So I am adding this in Distutils for 2.7. > > Of course setup.py stays, and this is backward compatible. > > - Work left to do : publish the final syntax, and do the implementation > - Goal: 2.7 / 3.2 > > == The fate of bdist_* commands == > > During last Pycon summit we said that we would remove commands like > bdist_rpm because Python is unable, due to its release cycle, > to do a good work there. Here's an example: I have from time to time > cryptic issues in the issue tracker from people from Fedora (or any > rpm-based system), and I have all the pain in the world for these very > specific problems to do the proper fix unless some RPM expert helps > around. And by the time it's detected then fixed, it can be year(s) > before it's available on their side. That's why, depending on the > communities, commands like bdist_rpm are just totally ignored, and OS > packager have their own tools. > > So the best way to handle this is to ask these communities to build > their own tool and to encourage them to use Distutils as a basis for > that. > > This does not concern bdist_* commands for win32 because those are > very stable and don't change too much: Windows doesn't have a package > manager that would require these commands to evolve with it. > > Anyways, when we said that we would remove bdist_rpm, this was very > controversial because some people use it and love it. > > So what is going to happen is a status-quo: no bdist_* command will be > removed but no new bdist_* command wil be added. That's why I've > encouraged Andrew and Garry, that are working on a bdist_deb command, > to keep it in the "stdeb" project, and eventually we will > refer to it in the Distutils documentation if this bdist_deb comply > with Distutils standard. It doesn't right now because it uses a > custom version of the Distribution class (through Setuptools) that > doesn't behave like Distutils' one anymore. > > For Distutils, I'll add some documentation explaining this, and a > section that will list community-driven commands. > > - Work left to do : write the documentation > - Goal: 2.7 / 3.2 > > = Distribute = > > I won't explain here again why we have forked, I think it's obvious to > anyone here now. I'll rather explain what > we are planning in Distribute and how it will interact with Distutils. > > Distribute has two branches: > > - 0.6.x : provides a Setuptools-0.6c9 compatible version > - 0.7.x : will provide a refactoring > > == 0.6.x == > > Not "much" is going to happen here, we want this branch to be helpful > to the community *today* by addressing the 40-or-so bugs > that were found in Setuptools and never fixed. This is eventually > happen soon because its development is > fast : there are up to 5 commiters that are working on it very often > (and the number grows weekly.) > > The biggest issue with this branch is that it is providing the same > packages and modules setuptools does, and this > requires some bootstrapping work where we make sure once Distribute is > installed, all Distribution that requires Setuptools > will continue to work. This is done by faking the metadata of > Setuptools 0.6c9. That's the only way we found to do this. > > There's one major thing though: thanks to the work of Lennart, Alex, > Martin, this branch supports Python 3, > which is great to have to speed up Py3 adoption. > > The goal of the 0.6.x is to remove as much bugs as we can, and try if > possible to remove the patches done > on Distutils. We will support 0.6.x maintenance for years and we will > promote its usage everywhere instead of > Setuptools. > > Some new commands are added there, when they are helpful and don't > interact with the rest. I am thinking > about "upload_docs" that let you upload documentation to PyPI. The > goal is to move it to Distutils > at some point, if the documentation feature of PyPI stays and starts to be used. > > == 0.7.x == > > We've started to refactor Distribute with this roadmap in mind (and > no, as someone said, it's not vaporware, > we've done a lot already) > > - 0.7.x can be installed and used with 0.6.x > > - easy_install is going to be deprecated ! use Pip ! > > - the version system will be deprecated, in favor of the one in Distutils > > - no more Distutils monkey-patch that happens once you use the code > (things like 'from distutils import cmd; cmd.Command = CustomCommand') > > - no more custom site.py (that is: if something misses in Python's > site.py we'll add it there instead of patching it) > > - no more namespaced packages system, if PEP 381 (namespaces package > support) makes it to 2.7 > > - The code is splitted in many packages and might be distributed under > several distributions. > > - distribute.resources: that's the old pkg_resources, but > reorganized in clean, pep-8 modules. This package will > only contain the query APIs and will focus on being PEP 376 > compatible. We will promote its usage and see if Pip wants > to use it as a basis. And maybe PyPM once it's open source ? > ( ). > It will probably shrink a lot though, once the stdlib provides PEP 376 support. > > - distribute.entrypoints: that's the old pkg_resources entry points > system, but on its own. it uses distribute.resources > > - distribute.index: that's package_index and a few other things. > everything required to interact with PyPI. We will promote > its usage and see if Pip wants to use it as a basis. > > - distribute.core (might be renamed to main): that's everything > else, and uses the other packages. > > > Goal: A first release before (or when) Python 2.7 / 3.2 is out. > > > = Virtualenv and the multiple version support in Distribute = > > (I am not saying "We" here because this part was not discussed yet > with everyone) > > Virtualenv allows you to create an isolated environment to install > some distribution without polluting the > main site-packages, a bit like a user site-packages. > > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. Not only, for me it exists also to provide a way to isolate/chroot an application from the host system. > But if PEP 376 and PEP 386 support are added in Python, we're not far > from being able to provide multiple version support with > the help of importlib. > > Setuptools provided a multiple version support but I don't like its > implementation and the way its works. > I would like to create a new site-packages format that can contains > several versions of the same distribution, and : > > - a special import system using importlib that would automatically > pick the latest version, thanks to PEP 376. > - an API to force at runtime a specific version (that would be located > at the beginning of all imports, like __future__) > - a layout that is compatible with the way OS packagers works with > python packages > > Goal: a prototype asap (one was started under the "VSP" name (virtual > site-packages) but not finished yet) > > Regards > Tarek > -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From lists at cheimes.de Thu Oct 8 13:17:46 2009 From: lists at cheimes.de (Christian Heimes) Date: Thu, 08 Oct 2009 13:17:46 +0200 Subject: [Python-Dev] PEP 370 and IronPython Message-ID: <4ACDCA5A.4090609@cheimes.de> Dear Pythonistas! Michael Foord has written a blog posting [1] regarding IronPython, site packages and my PEP 370. I'm referring to the section that talks about an issue with PEP 370. --- If you install CPython 2.6 on Windows it creates a folder in the location: %APPDATA%\Python\Python26\site-packages. On my machine the full path is: C:\Users\michael\AppData\Roaming\Python\Python26\site-packages. This is put on the path by site.py when you run Python and as it is a user directory it doesn't require elevation to install there. You install into the user directory rather than the 'usual' location with: python setup.py --user install Herein lies another problem though. IronPython uses the standard Python site.py, meaning that currently it shares the user site-packages folder with CPython. This is not ideal as typically you will want separate libraries install for IronPython. When Jython ports to Python 2.6 it will have the same issue. --- I didn't think about IronPython during the design phase of the PEP. I didn't come into my mind that another implementation may require a different site packages directory. After spinning my head around the issue a simple solution came to me. The solution requires a new attribute in the sys module that contains the name of the implementation. As an alternative we could use the first field of sys.subversion but I prefer an explicit attribute. I'm proposing "sys.name" with a value of "CPython", "IronPython", "Jython" or "PyPy". The site module uses the information of the attribute to modify the path to the user site directory. The altered user site directories are: CPython: ~/.local/lib/python2.6/site-packages %APPDATA%/Python/Python26 IronPython: ~/.local/lib/ironpython2.6/site-packages %APPDATA%/Python/IronPython26 Jython: ~/.local/lib/jython2.6/site-packages %APPDATA%/Python/Jython26 Proposed patch: Index: Lib/site.py =================================================================== --- Lib/site.py (revision 75282) +++ Lib/site.py (working copy) @@ -74,7 +74,12 @@ USER_SITE = None USER_BASE = None +interpreter_name = getattr(sys, "name", "CPython") +if interpreter_name == "CPython": + # b/w compatible name + interpreter_name = "python" + def makepath(*paths): dir = os.path.abspath(os.path.join(*paths)) return dir, os.path.normcase(dir) @@ -253,10 +258,12 @@ return USER_SITE if os.name == "nt": - USER_SITE = os.path.join(user_base, "Python" + sys.version[0] + - sys.version[2], "site-packages") + USER_SITE = os.path.join(user_base, interpreter_name.capitalize() + + sys.version[0] + sys.version[2], + "site-packages") else: - USER_SITE = os.path.join(user_base, "lib", "python" + sys.version[:3], + USER_SITE = os.path.join(user_base, "lib", + interpreter_name.lower() + sys.version[:3], "site-packages") return USER_SITE The patch doesn't modify the search path for global directories. These can be changed by altering sys.prefix and sys.exec_prefix. Christian [1] http://ironpython-urls.blogspot.com/2009/10/distributing-ironpython-packages.html From mal at egenix.com Thu Oct 8 13:27:57 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 13:27:57 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACDB82F.7010006@cryptelium.net> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> Message-ID: <4ACDCCBD.4040908@egenix.com> I have just a few comments to make, so I'm trimming the long quote a bit... > Tarek Ziad? a ?crit : >> == Version comparison == >> >> One very important point: we will not force the community to use the >> scheme described in PEP 386, but *there is* already a >> de-facto convention on version schemes at PyPI if you use Pip or >> easy_install, so let's have a documented standard for this, >> and a reference implementation in Distutils. >> >> There's an implementation at >> http://bitbucket.org/tarek/distutilsversion that is kept up-to-date >> with PEP 386. >> >> - Work left to do in PEP 386 : another round with the community >> - Goal: 2.7 / 3.2 Please make sure that the version scheme is compatible with other tools we use in the distutils chain: RPM and the MSI compiler. Some data points: RPM doesn't like hyphens in version strings. MSI requires a strict N.N.N format. >> == The fate of setup.py, and static metadata == >> >> So we are going to separate the metadata description from setup.py, in >> a static configuration file, that can be open and read by anyone >> without >> running any code. The only problem with this is the fact that some >> metadata fields might depend on the execution environment. For >> instance, once "Requires" is re-defined and re-introduced via PEP 314, >> we will have cases where "pywin32" will be a dependency to have only >> on win32 systems. How are you planning to deal with dynamic version strings, e.g. say you want to build a snapshot release the current date and revision number included in the version string ? I suppose this file is going to be uploaded to PyPI, so the file would have to be formatted at build time and include markers to be replaced with values taken from the distribution meta-data. >> - Work left to do : publish the final syntax, and do the implementation Is there a PEP for this ? >> == The fate of bdist_* commands == >> >> So the best way to handle this is to ask these communities to build >> their own tool and to encourage them to use Distutils as a basis for >> that. Rather than having external communities build their own tools and then basically fork distutils in order to get these implemented, I think it'd be better to make the set of distutils commands extensible via namespace packages. >> So what is going to happen is a status-quo: no bdist_* command will be >> removed but no new bdist_* command will be added. I hope that comment means "no new bdist_* command will be added *right now*". It's well possible to create new bdist_* commands that don't rely on external tool chains or only on ones that don't change often. I'm thinking of bdist_inno (Windows installer using InnoSetup) and bdist_nsis (Windows installer using NSIS) here. >> = Virtualenv and the multiple version support in Distribute = >> >> (I am not saying "We" here because this part was not discussed yet >> with everyone) >> >> Virtualenv allows you to create an isolated environment to install >> some distribution without polluting the >> main site-packages, a bit like a user site-packages. >> >> My opinion is that this tool exists only because Python doesn't >> support the installation of multiple versions for the same >> distributions. > > Not only, for me it exists also to provide a way to isolate/chroot an > application from the host system. I agree: IMHO, that's the main use case for things like virtualenv. Note that Python has had a PYTHONHOME environment variable for ages. This pretty much serves the same purpose and together with setting PYTHONPATH can easily be used to create your own private Python universe. >> But if PEP 376 and PEP 386 support are added in Python, we're not far >> from being able to provide multiple version support with >> the help of importlib. Before putting much work into this: do you really think that having multiple versions of the same package in the same Python installation is a good idea ? Examples: What if you have an application that uses two modules which each require two different versions of the same package ? Would that load the same package twice, causing globals used by these package to no work (e.g. isinstance(x, class_name) or global caches) ? This sounds a lot like DLL- or RPM-hell to me. I think it's much better to keep things simple and under user control, e.g. by encouraging use of virtualenv-like setups and perhaps adding better native support for these to Python. If the user finds that there's a version conflict this can then be resolved during environment setup instead of hoping for the best and waiting for application failures at run-time. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Oct 8 13:28:48 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 13:28:48 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <94bdd2610910080159r319b0fdtd02b779bba893e18@mail.gmail.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <94bdd2610910080133y788cb0d6xd67aaf3b58ac48a0@mail.gmail.com> <4ACDA4EE.4050005@egenix.com> <94bdd2610910080159r319b0fdtd02b779bba893e18@mail.gmail.com> Message-ID: <4ACDCCF0.8070107@egenix.com> Tarek Ziad? wrote: > On Thu, Oct 8, 2009 at 10:38 AM, M.-A. Lemburg wrote: >> Tarek Ziad? wrote: >>> On Thu, Oct 8, 2009 at 9:35 AM, M.-A. Lemburg wrote: >>>> BTW: Who would I need to contact for the PyPI side to work out >>>> an upload_url distutils command ? >>> >>> I am not sure to understand. How upload_url will differ from the >>> register command >>> that let you upload metadata at PyPI ? >> >> The register command only uploads the package's meta-data. >> >> The upload command uploads a file and registers the file's >> details with PyPI as well. >> >> I'd like to add an upload_url command which only registers a >> file's details (including the URL, but without the actual file) >> with PyPI and perhaps extend the number of attributes sent to >> PyPI with more file configuration information (upload currently >> only sends the Python version). > > Sounds like an option to be added in upload. With a little code refactoring that would be possible as well, yes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Thu Oct 8 13:43:24 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2009 11:43:24 +0000 (UTC) Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: Hello, Thanks for the summary :) [Distribute 0.7] > - easy_install is going to be deprecated ! use Pip ! Apparently there are a couple of things Pip doesn't support: ? It cannot install from eggs. It only installs from source. (Maybe this will be changed sometime, but it?s low priority.) ? ? It is incompatible with some packages that customize distutils or setuptools in their setup.py files. ? ? Maybe it doesn?t work on Windows. At least, the author doesn?t test on Windows often. ? (from http://pip.openplans.org/#differences-from-easy-install) > Virtualenv allows you to create an isolated environment to install > some distribution without polluting the > main site-packages, a bit like a user site-packages. > > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. virtualenv has the virtue of keeping your main installation clean and manageable. Besides, as someone said, it also allows you to create completely isolated environments if you want full control over your deployment (especially when you're not the sysadmin). Regards Antoine. From solipsis at pitrou.net Thu Oct 8 13:47:25 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2009 11:47:25 +0000 (UTC) Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> Message-ID: Le Thu, 08 Oct 2009 13:27:57 +0200, M.-A. Lemburg a ?crit?: > > This sounds a lot like DLL- or RPM-hell to me. > > I think it's much better to keep things simple and under user control, > e.g. by encouraging use of virtualenv-like setups and perhaps adding > better native support for these to Python. Agreed. virtualenv enables a feature while keeping things simple and clear. The proposed scheme (multi-versioned stuff) is supposed to enable the same feature, but by making things more complicated. ;) From ncoghlan at gmail.com Thu Oct 8 13:53:39 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 08 Oct 2009 21:53:39 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> Message-ID: <4ACDD2C3.4090608@gmail.com> Antoine Pitrou wrote: > Vinay Sajip yahoo.co.uk> writes: >>>>> "%0#8x" % 0x1234 >> '0x001234' >>>>> "{0:0>#8x}".format(0x1234) >> '000x1234' > > Apart from the sheer unreadability of the {}-style format string, the result > looks rather unexpected from a human being's point of view. The percent format string is pretty unreadable too - you're just more used to it, so it doesn't look as weird :) Vinay's problem above is due to using the wrong alignment flag: ">", which says to right align everything, instead of "=", which says to left align the sign character and the numeric prefix with the fill character inserted in the middle. In this particular case he could also use the zero-padding shortcut which leaves out the alignment flag altogether (and implies a "0=" alignment format). That is (using 2.6/3.1): >>> "%#08x" % 0x1234 '0x001234' >>> "%0#8x" % 0x1234 '0x001234' >>> "{0:#08x}".format(0x1234) '0x001234' >>> "{0:0>#8x}".format(0x1234) '000x1234' >>> "{0:0=#8x}".format(0x1234) '0x001234' Adding in the sign bit gives the following translations: >>> "%+#08x" % 0x1234 '+0x01234' >>> "%+0#8x" % 0x1234 '+0x01234' >>> "{0:+#08x}".format(0x1234) '+0x01234' >>> "{0:0>+#8x}".format(0x1234) '0+0x1234' >>> "{0:0=+#8x}".format(0x1234) '+0x01234' Note that ">" alignment is actually now *broken* on trunk and py3k, since ">" and "=" are now behaving exactly the same instead of the former right aligning the entire number including the sign bit and prefix: >>> "{:0>+8x}".format(0x1234) '+0001234' >>> "{:0=+8x}".format(0x1234) '+0001234' (bug assigned to Eric: http://bugs.python.org/issue7081) Note that, since percent formatting doesn't allow specification of the fill characters or the field alignment, translations should probably rely on the simple field width specifier, optionally selecting zero padding by preceding it with a zero. It should never be necessary to use the full alignment spec for translated formats. The other thing to keep in mind is that brace formatting is fussier about the order of things - items *must* appear in the order they are listed in PEP 3101 (i.e. if wanting a zero padded field with leading sign and numeric prefix, you must write "+#0"). Percent format, on the other hand, allows the "#", "+" and "0" to be placed in any order you like (although they must appear before the field width definition, precision specifier and type code). As far as I can see, that leaves the prefixing of octal numbers ("0o" vs "0") as the only true incompatibility between percent formatting and brace formatting, and even for those the incompatibility is limited to cases where a field width is specified without leading zeroes or a sign character is specified. In other cases, the translation can just stick a leading literal "0" in front of the field in the brace formatting string. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Oct 8 14:22:00 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 08 Oct 2009 22:22:00 +1000 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACDCA5A.4090609@cheimes.de> References: <4ACDCA5A.4090609@cheimes.de> Message-ID: <4ACDD968.1070205@gmail.com> Christian Heimes wrote: > The solution requires a new attribute in the sys module that contains > the name of the implementation. As an alternative we could use the first > field of sys.subversion but I prefer an explicit attribute. I'm > proposing "sys.name" with a value of "CPython", "IronPython", "Jython" > or "PyPy". My Google skills almost failed me, but searching for "sys.vm" found me what I was after: http://bugs.python.org/issue4242 (a discussion relating to a similar need in the context of flagging implementation specific tests). As mentioned in that discussion, as of Python 2.6, you can do the following: >>> import platform >>> platform.python_implementation() 'CPython' (Although according to the function docstring, PyPy is currently missing from the list of known implementations) Importing yet-another-module for use in site.py doesn't sound like a great idea, so it may make sense to migrate that information into the sys module is this approach is taken. "sys.name" is a little generic though - something more explicit like "sys.vm" would be better. > The site module uses the information of the attribute to > modify the path to the user site directory. Implementation specific user directories sound like a good idea indeed. An alternative to a lookup table approach might be to be a little more direct and just retrieve the final part of the user specific directory name directly from a new function in the sys module. Then different VM authors can choose any directory name they want without CPython's site.py needing to know anything else about them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From zvezdan at zope.com Thu Oct 8 14:45:54 2009 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Thu, 8 Oct 2009 08:45:54 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> Message-ID: On Oct 7, 2009, at 2:09 PM, Barry Warsaw wrote: > I want us to be very careful about 2.6.4. This isn't a normal bug > fix release, it's specifically there to remove the brown paper bag > of 2.6.3 from our heads. So let's be conservative and fix this one > in 2.6.5. Can we get the readline patch reviewed as well. It was applied to trunk already: http://svn.python.org/view?view=rev&revision=74970 It's marked for backport to 2.6 as "needs review": http://bugs.python.org/issue6877 It fixes a BusError crash that was just swiped under a rug by disabling the build in setup.py. :-) FWIW, the part of the patch that changes setup.py will be useful for another comparison of Mac OS X release numbers. The line 1362 on the trunk (1364 on the 2.6 branch) compares strings and, unfortunately, '10.' < '8.' for Darwin 10 (Mac OS X 10.6 Snow Leopard): if os.uname()[2] > '8.': So, it should be changed to: if os_release > 8: where ``os_release`` is introduced by the patch mentioned above as: os_release = int(os.uname()[2].split('.')[0]) on the line 550 on the trunk. Regards, Zvezdan From hodgestar+pythondev at gmail.com Thu Oct 8 14:52:24 2009 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Thu, 8 Oct 2009 14:52:24 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad? wrote: > = Virtualenv and the multiple version support in Distribute = ... > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. This is not at all how I use virtualenv. For me virtualenv is a sandbox so that I don't have to become root whenever I need to install a Python package for testing purposes and to allow me to hop between sets of installed Python packages while developing on multiple Python projects. I also use it as a sandbox for build bots so that multiple bots on the same machine can each build their own projects with just the known dependencies installed. Schiavo Simon From ziade.tarek at gmail.com Thu Oct 8 14:54:44 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 14:54:44 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACDCCBD.4040908@egenix.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> Message-ID: <94bdd2610910080554m530bfdbeg2bf010c813c71873@mail.gmail.com> On Thu, Oct 8, 2009 at 1:27 PM, M.-A. Lemburg wrote: > >>> == The fate of setup.py, and static metadata == >>> >>> So we are going to separate the metadata description from setup.py, in >>> a static configuration file, that can be open and read by anyone >>> without >>> running any code. The only problem with this is the fact that some >>> metadata fields might depend on the execution environment. For >>> instance, once "Requires" is re-defined and re-introduced via PEP 314, >>> we will have cases where "pywin32" will be a dependency to have only >>> on win32 systems. > > How are you planning to deal with dynamic version strings, e.g. > say you want to build a snapshot release the current date and > revision number included in the version string ? > > I suppose this file is going to be uploaded to PyPI, so the > file would have to be formatted at build time and include > markers to be replaced with values taken from the distribution > meta-data. The metadata will always be customizable from setup.py somehow. We won't treat those cases : if the metadata is provided in the static file, you can get it without any further processing, if not you will get an "UNKOWN" value for it and you will have to grab the whole archive and run setup.py to get it. Now the example you've shown is in my opinion not a problem : you can create the static value for "version" that you put in your static file with some pre-processing that occurs when you build your release. (ala autoconf) > >>> - Work left to do : publish the final syntax, and do the implementation > > Is there a PEP for this ? No, but if you ask for it I can write it. > >>> == The fate of bdist_* commands == >>> >>> So the best way to handle this is to ask these communities to build >>> their own tool and to encourage them to use Distutils as a basis for >>> that. > > Rather than having external communities build their own > tools and then basically fork distutils in order to get > these implemented, I think it'd be better to make the > set of distutils commands extensible via namespace packages. I am not sure, why they would fork distutils. Distutils is already extensible, you can configure in distutils.cfg a list of namespace packages, that contains commands and they'll get loaded by dist.py. > >>> So what is going to happen is a status-quo: no bdist_* command will be >>> removed but no new bdist_* command will be added. > > I hope that comment means "no new bdist_* command will be added > *right now*". It's well possible to create new bdist_* commands > that don't rely on external tool chains or only on ones that > don't change often. > > I'm thinking of bdist_inno (Windows installer using InnoSetup) > and bdist_nsis (Windows installer using NSIS) here. As I said, windows don't have any packaging system so I don't think it's a problem to have more installers for this platform. Regards Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From ziade.tarek at gmail.com Thu Oct 8 15:16:27 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 15:16:27 +0200 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACDCA5A.4090609@cheimes.de> References: <4ACDCA5A.4090609@cheimes.de> Message-ID: <94bdd2610910080616w27b43020sa52670269e01b7ff@mail.gmail.com> On Thu, Oct 8, 2009 at 1:17 PM, Christian Heimes wrote: [...] > The altered user site directories are: > > CPython: > ?~/.local/lib/python2.6/site-packages > ?%APPDATA%/Python/Python26 > > IronPython: > ?~/.local/lib/ironpython2.6/site-packages > ?%APPDATA%/Python/IronPython26 > > Jython: > ?~/.local/lib/jython2.6/site-packages > ?%APPDATA%/Python/Jython26 > Hi Christian, +1 I have a suggestion though, that completes Nick's answer. distutils/command/install.py already contains install schemes and imports USER_SITE and USER_BASE to builds user-specific install schemes. I think it wouldn't hurt for clarity to reunite all these schemes in a single place in the stdlib (so in sys maybe?) Tarek From fuzzyman at voidspace.org.uk Thu Oct 8 15:52:57 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 08 Oct 2009 14:52:57 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4ACDEEB9.2000308@voidspace.org.uk> Simon Cross wrote: > On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad? wrote: > >> = Virtualenv and the multiple version support in Distribute = >> > ... > >> My opinion is that this tool exists only because Python doesn't >> support the installation of multiple versions for the same >> distributions. >> > > This is not at all how I use virtualenv. For me virtualenv is a > sandbox so that I don't have to become root whenever I need to install > a Python package for testing purposes and to allow me to hop between > sets of installed Python packages while developing on multiple Python > projects. I also use it as a sandbox for build bots so that multiple > bots on the same machine can each build their own projects with just > the known dependencies installed. > > This is exactly why I use virtualenv as well. I don't recall ever having wanted / needed to install multiple versions of the same library - whilst I can appreciate that it *can* be a real issue it has never been a problem for me. Michael > Schiavo > Simon > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From anders at 0x63.nu Thu Oct 8 15:43:44 2009 From: anders at 0x63.nu (Anders Waldenborg) Date: Thu, 08 Oct 2009 15:43:44 +0200 Subject: [Python-Dev] locals() different behaviour when tracing? Message-ID: Is it intentional that locals() returns a copy/snapshot of the local variables sometimes, and sometimes a reference? Like when enabling tracing, such as in the code pasted below. The documentation ("Update and return a dictionary containing the current scope's local variables.") is pretty unclear. import sys def X(): l = locals() i = "foo" print "Is 'i' in stored locals()? ", ('i' in l) print "Running normally" X() print "Enabling tracing" def t(*a): return t sys.settrace(t) X() From v+python at g.nevcal.com Thu Oct 8 09:44:22 2009 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 08 Oct 2009 00:44:22 -0700 Subject: [Python-Dev] A new way to configure logging In-Reply-To: References: <4ACD4484.2080409@g.nevcal.com> Message-ID: <4ACD9856.1080206@g.nevcal.com> On approximately 10/7/2009 10:45 PM, came the following characters from the keyboard of Vinay Sajip: > Glenn Linderman g.nevcal.com> writes: > >> But DictConfigurator the name seems misleading... like you are >> configuring how dicts work, rather than how logs work. Maybe with more >> context this is not a problem, but if it is a standalone class, it is >> confusing what it does, by name alone. > > Of course the fully qualified name would be logging.config.DictConfigurator > which does provide the requisite context, but if I think of/someone suggests a > better name I'll certainly use it. Yes, I suspected it might be something like that, and that does provide the context, so I'm content. Thanks for verifying it, and the rest is bikeshedding... In thinking more about this, I guess I would have the same comment about the existing fileConfig() API also... it isn't file that is being configured, but the logs. readConfigFromFile would have been more descriptive (and lots longer :( ). Had fileConfig been called "bulkConfig" or "configAll" (these names attempt to contrast the function of this API versus the individual APIs that configure one thing at a time) taking a file parameter, then it could just simply/suddenly also allow a dict parameter, and no new API would have to be created. Too late for this, however, and with the new API, it would be possible to deprecate the fileConfig API, and tell the user to use the ConfigParser (or JSON or YAML or whatever) to create the dict, and then pass that to DictConfigurator (or bulkConfig or whatever). -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From vinay_sajip at yahoo.co.uk Thu Oct 8 16:16:18 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 14:16:18 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4ACDD2C3.4090608@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > Vinay's problem above is due to using the wrong alignment flag: ">", > which says to right align everything, instead of "=", which says to left > align the sign character and the numeric prefix with the fill character > inserted in the middle. In this particular case he could also use the > zero-padding shortcut which leaves out the alignment flag altogether > (and implies a "0=" alignment format). > [snip] > Note that, since percent formatting doesn't allow specification of the > fill characters or the field alignment, translations should probably > rely on the simple field width specifier, optionally selecting zero > padding by preceding it with a zero. It should never be necessary to use > the full alignment spec for translated formats. > > The other thing to keep in mind is that brace formatting is fussier > about the order of things - items *must* appear in the order they are > listed in PEP 3101 (i.e. if wanting a zero padded field with leading > sign and numeric prefix, you must write "+#0"). Percent format, on the > other hand, allows the "#", "+" and "0" to be placed in any order you > like (although they must appear before the field width definition, > precision specifier and type code). > > As far as I can see, that leaves the prefixing of octal numbers ("0o" vs > "0") as the only true incompatibility between percent formatting and > brace formatting, and even for those the incompatibility is limited to > cases where a field width is specified without leading zeroes or a sign > character is specified. In other cases, the translation can just stick a > leading literal "0" in front of the field in the brace formatting string. Helpful analysis there, Nick, thanks. Bonzer ;-) There's also the corner case of things like %#.0f which, when asked to format 3e100, will print 3.e+100 whereas the translated format {0:.0f}, will print 3e+100 for the same value. BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see if it was worth raising an enhancement request on the bug tracker using "O" to generate compatible rendering for octals. Regards, Vinay Sajip From eric at trueblade.com Thu Oct 8 16:24:33 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 08 Oct 2009 10:24:33 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4ACDD2C3.4090608@gmail.com> Message-ID: <4ACDF621.4070603@trueblade.com> Vinay Sajip wrote: > BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see if it was > worth raising an enhancement request on the bug tracker using "O" to generate > compatible rendering for octals. I didn't get your message, could you resend?. I was thinking the same thing, but it seems like a transition step. I'd rather not keep such backward compatibility hacks (if you will) around for the long haul. How about a flag (maybe '*') at the start of the format specification which says "operate in backward compatibility mode"? We could document it as being only useful for the % to {} translator, and promise to remove it at some point in the future. Either actually deprecate it or just promise to deprecate it in the future. Eric. From olemis at gmail.com Thu Oct 8 16:26:17 2009 From: olemis at gmail.com (Olemis Lang) Date: Thu, 8 Oct 2009 09:26:17 -0500 Subject: [Python-Dev] A new way to configure logging In-Reply-To: <4ACD9856.1080206@g.nevcal.com> References: <4ACD4484.2080409@g.nevcal.com> <4ACD9856.1080206@g.nevcal.com> Message-ID: <24ea26600910080726h715dbc4ex23bce17740a3c390@mail.gmail.com> On Thu, Oct 8, 2009 at 2:44 AM, Glenn Linderman wrote: > On approximately 10/7/2009 10:45 PM, came the following characters from the > keyboard of Vinay Sajip: >> >> Glenn Linderman g.nevcal.com> writes: >> >>> But DictConfigurator the name seems misleading... like you are >>> configuring how dicts work, rather than how logs work. ?Maybe with more >>> context this is not a problem, but if it is a standalone class, it is >>> confusing what it does, by name alone. >> >> Of course the fully qualified name would be >> logging.config.DictConfigurator >> which does provide the requisite context, but if I think of/someone >> suggests a >> better name I'll certainly use it. > [...] > > In thinking more about this, I guess I would have the same comment about the > existing fileConfig() API also... it isn't file that is being configured, > but the logs. ?readConfigFromFile would have been more descriptive (and lots > longer :( ). > cfgFromIni | cfgFromFile ? I prefer the former because YAML, shelves, JSON & Co can also be stored in a file , so at least I get a better understanding of the format in use. Otherwise I'd had to read the docs or figure out which one > Had fileConfig been called "bulkConfig" or "configAll" (these names attempt > to contrast the function of this API versus the individual APIs that > configure one thing at a time) taking a file parameter, then it could just > simply/suddenly also allow a dict parameter, and no new API would have to be > created. ?Too late for this, however, and with the new API, it would be > possible to deprecate the fileConfig API, and tell the user to use the > ConfigParser (or JSON or YAML or whatever) to create the dict, and then pass > that to DictConfigurator (or bulkConfig or whatever). > similar and consistent with what I mentioned before but IMO fileConfig should remain for backwards compatibility Objections ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Looking for a technique to create flexible, graphical dashboards ... - http://feedproxy.google.com/~r/TracGViz-full/~3/71kRhT34BgU/43789 From barry at python.org Thu Oct 8 16:30:11 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Oct 2009 10:30:11 -0400 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: On Oct 8, 2009, at 4:31 AM, Tarek Ziad? wrote: > - no more namespaced packages system, if PEP 381 (namespaces package > support) makes it to 2.7 Make that PEP 382: http://www.python.org/dev/peps/pep-0382/ -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From vinay_sajip at yahoo.co.uk Thu Oct 8 16:47:59 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 14:47:59 +0000 (GMT) Subject: [Python-Dev] A new way to configure logging In-Reply-To: <09B60FD3-D227-43E3-B901-5418C7FA950C@gmail.com> References: <09B60FD3-D227-43E3-B901-5418C7FA950C@gmail.com> Message-ID: <137556.63116.qm@web25801.mail.ukl.yahoo.com> > I've had bad experiences in the past with dictionary-based APIs. They seem > "simpler" in the short run, because the user "only needs to create some > dictionaries". Once the complexity of that nested dictionary grows to a certain > point, though, one has to refer back to documentation constantly to make sure Fair point, and agreed that the schema needs some care, here's roughly what I'm thinking of as an example configuration (YAML): formatters: brief: format: '%(levelname)-8s: %(name)-15s: %(message)s' precise: format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s' filters: allow_foo: name: foo handlers: console: class : logging.StreamHandler formatter: brief level : INFO stream : sys.stdout filters: [allow_foo] file: class : logging.handlers.RotatingFileHandler formatter: precise filename: logconfig.log maxBytes: 1024 backupCount: 3 debugfile: class : logging.FileHandler formatter: precise filename: logconfig-detail.log mode: a loggers: foo: level : ERROR handlers: [debugfile] spam: level : CRITICAL handlers: [debugfile] propagate: no bar.baz: level: WARNING root: level : DEBUG handlers : [console, file] It's not too deeply nested, and I can't see any need for it being more deeply nested. It more or less mirrors the way you'd have to write the corresponding ConfigParser-compatible configuration, but if you use a dict as a common format, you have the benefits which I mentioned of supporting JSON, YAML or Python code in a Django settings.py. > the structure conforms to the "schema". Building a simple config tree using > light-weight classes with documented APIs tends to be more sustainable in the > long run. When you say 'config tree using light-weight classes', I presume you mean a parallel set of classes to the actual logging classes which will be instantiated? ISTM logging classes are already reasonably usable for programmatic configuration, but this doesn't address e.g. updating configurations easily without program code changes, or the ability to configure from say a YAML node, where YAML may be being used for application configuration as a whole. (Unless of course I've misunderstood what you're getting at.) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Oct 8 16:54:53 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 14:54:53 +0000 (GMT) Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACDF621.4070603@trueblade.com> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4ACDD2C3.4090608@gmail.com> <4ACDF621.4070603@trueblade.com> Message-ID: <67182.50646.qm@web25806.mail.ukl.yahoo.com> > I didn't get your message, could you resend?. Resent, it may have been stopped by your spam filters since it came from my vinay-underscore-sajip-at-red-hyphen-dove-dot-com address. The subject was "Python str.format() and octal formatting compatibility". > I was thinking the same thing, but it seems like a transition step. I'd rather > not keep such backward compatibility hacks (if you will) around for the long > haul. How about a flag (maybe '*') at the start of the format specification > which says "operate in backward compatibility mode"? We could document it as > being only useful for the % to {} translator, and promise to remove it at some > point in the future. Either actually deprecate it or just promise to deprecate > it in the future. I don't much mind matter exactly which mechanism we use to distinguish between 0o and 0 prefixes, as long as it's one most people are happy with :-) Regards, Vinay Sajip From mal at egenix.com Thu Oct 8 16:55:16 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 16:55:16 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080554m530bfdbeg2bf010c813c71873@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> <94bdd2610910080554m530bfdbeg2bf010c813c71873@mail.gmail.com> Message-ID: <4ACDFD54.3080202@egenix.com> Tarek Ziad? wrote: > On Thu, Oct 8, 2009 at 1:27 PM, M.-A. Lemburg wrote: >> >>>> == The fate of setup.py, and static metadata == >>>> >>>> So we are going to separate the metadata description from setup.py, in >>>> a static configuration file, that can be open and read by anyone >>>> without >>>> running any code. The only problem with this is the fact that some >>>> metadata fields might depend on the execution environment. For >>>> instance, once "Requires" is re-defined and re-introduced via PEP 314, >>>> we will have cases where "pywin32" will be a dependency to have only >>>> on win32 systems. >> >> How are you planning to deal with dynamic version strings, e.g. >> say you want to build a snapshot release the current date and >> revision number included in the version string ? >> >> I suppose this file is going to be uploaded to PyPI, so the >> file would have to be formatted at build time and include >> markers to be replaced with values taken from the distribution >> meta-data. > > The metadata will always be customizable from setup.py somehow. > > We won't treat those cases : if the metadata is provided in the static file, > you can get it without any further processing, if not you will get an > "UNKOWN" value > for it and you will have to grab the whole archive and run setup.py to get it. > > Now the example you've shown is in my opinion not a problem : you can > create the static value for "version" that you put in your static file > with some pre-processing that > occurs when you build your release. (ala autoconf) Hmm, then I'm not really sure I understand the proposal for a new static meta-data file... distutils already creates the static meta-data file PKG-INFO when building sdist distributions and that's done based on the data read from the setup() call. It would be easy to use that file as static meta-data file in other distribution formats as well (e.g. like you proposed in PEP 376). OTOH, the register command sends the meta-data directly to the PyPI database, so it doesn't even need another file for storing away the meta data. Unless I'm missing something important (which I probably am), the only added value of the new format over PKG-INFO is the micro-language. Couldn't we just stick that into the values of the various meta-data fields, e.g. to have the Requires field be different depending on platform ? We'd then only have one meta-data format in distutils - the PKG-INFO one. >>>> - Work left to do : publish the final syntax, and do the implementation >> >> Is there a PEP for this ? > > No, but if you ask for it I can write it. Please do. The format will need documentation anyway and the motivation should be made clear as well. >>>> == The fate of bdist_* commands == >>>> >>>> So the best way to handle this is to ask these communities to build >>>> their own tool and to encourage them to use Distutils as a basis for >>>> that. >> >> Rather than having external communities build their own >> tools and then basically fork distutils in order to get >> these implemented, I think it'd be better to make the >> set of distutils commands extensible via namespace packages. > > I am not sure, why they would fork distutils. Not really a fork, but a copy of distutils in order to give the user the complete tool set in one download, rather than having to configure a system installed distutils to load the new command. > Distutils is already extensible, you can configure in distutils.cfg a > list of namespace packages, > that contains commands and they'll get loaded by dist.py. Right, but dist.py could load all distutils.command modules it can find to make the installation of extensions more user friendly (like in a plugin system). Just a minor nit, though. >>>> So what is going to happen is a status-quo: no bdist_* command will be >>>> removed but no new bdist_* command will be added. >> >> I hope that comment means "no new bdist_* command will be added >> *right now*". It's well possible to create new bdist_* commands >> that don't rely on external tool chains or only on ones that >> don't change often. >> >> I'm thinking of bdist_inno (Windows installer using InnoSetup) >> and bdist_nsis (Windows installer using NSIS) here. > > As I said, windows don't have any packaging system so I don't think > it's a problem to have more installers for this platform. Ok. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From eric at trueblade.com Thu Oct 8 16:55:21 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 08 Oct 2009 10:55:21 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <4ACDF621.4070603@trueblade.com> Message-ID: <4ACDFD59.4060200@trueblade.com> Christian Tanzer wrote: > Eric Smith wrote at Thu, 08 Oct 2009 10:24:33 -0400: > >> Vinay Sajip wrote: >>> BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see if it was >>> worth raising an enhancement request on the bug tracker using "O" to generate >>> compatible rendering for octals. >> I didn't get your message, could you resend?. >> >> I was thinking the same thing, but it seems like a transition step. I'd >> rather not keep such backward compatibility hacks (if you will) around >> for the long haul. How about a flag (maybe '*') at the start of the >> format specification which says "operate in backward compatibility >> mode"? We could document it as being only useful for the % to {} >> translator, and promise to remove it at some point in the future. Either >> actually deprecate it or just promise to deprecate it in the future. > > That doesn't seem very useful to me. IIUC, the point of the translator > is to allow porting of the millions of existing %-formating operations > to the new-style .format. > > If the result of that is deprecated or removed a few years from now, > all maintainers of long existing code have exactly the same problem. I was thinking of it as a transition step until all application code switched to {} formatting. In which case the application has to deal with it. > IMHO, either the translation is done once and gives identical output or > it isn't worth doing at all. I disagree. I doubt even 0.001% of all format strings involve octal formatting. Is it really worth not providing a transition path if it can't cover this case? Eric. From benjamin at python.org Thu Oct 8 16:25:10 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 8 Oct 2009 09:25:10 -0500 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACDD968.1070205@gmail.com> References: <4ACDCA5A.4090609@cheimes.de> <4ACDD968.1070205@gmail.com> Message-ID: <1afaf6160910080725q888d72r5da38ad9baa0696e@mail.gmail.com> 2009/10/8 Nick Coghlan : > As mentioned in that discussion, as of Python 2.6, you can do the following: >>>> import platform >>>> platform.python_implementation() > 'CPython' > > (Although according to the function docstring, PyPy is currently missing > from the list of known implementations) That docstring should be updated then because I added support for PyPy in 2.7 and 3.1. -- Regards, Benjamin From tanzer at swing.co.at Thu Oct 8 17:08:29 2009 From: tanzer at swing.co.at (Christian Tanzer) Date: Thu, 08 Oct 2009 15:08:29 -0000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: Your message of "Thu, 08 Oct 2009 10:55:21 -0400" <4ACDFD59.4060200@trueblade.com> References: <4ACDFD59.4060200@trueblade.com> Message-ID: Eric Smith wrote at Thu, 08 Oct 2009 10:55:21 -0400: > >>> BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see if it was > >>> worth raising an enhancement request on the bug tracker using "O" to generate > >>> compatible rendering for octals. > >> I didn't get your message, could you resend?. > >> > >> I was thinking the same thing, but it seems like a transition step. I'd > >> rather not keep such backward compatibility hacks (if you will) around > >> for the long haul. How about a flag (maybe '*') at the start of the > >> format specification which says "operate in backward compatibility > >> mode"? We could document it as being only useful for the % to {} > >> translator, and promise to remove it at some point in the future. Either > >> actually deprecate it or just promise to deprecate it in the future. > > > > That doesn't seem very useful to me. IIUC, the point of the translator > > is to allow porting of the millions of existing %-formating operations > > to the new-style .format. > > > > If the result of that is deprecated or removed a few years from now, > > all maintainers of long existing code have exactly the same problem. > > I was thinking of it as a transition step until all application code > switched to {} formatting. In which case the application has to deal > with it. You lost me here. All that talk of deprecating %-formatting makes me really nervous. %-formatting is pervasive in all existing Python code. Without an automatic translator that is 100% accurate, porting all that code to {}-formatting is not possible. Heck, it's not even possible to grep for all instances of %-formatting. How do you suppose that maintainers could ever do the transition from %- to {}-formatting manually? > > IMHO, either the translation is done once and gives identical output or > > it isn't worth doing at all. > > I disagree. I doubt even 0.001% of all format strings involve octal > formatting. Is it really worth not providing a transition path if it > can't cover this case? If %-formatting is first deprecated then removed from Python and there is no automatic transition path that effectively means that existing code using %-formatting is forced to stay at whatever Python version was the last one supporting %-formatting. I surely hope nobody is seriously considering such a scenario. Perl 6 seems harmless in comparison. -- Christian Tanzer http://www.c-tanzer.at/ From tanzer at swing.co.at Thu Oct 8 16:51:00 2009 From: tanzer at swing.co.at (Christian Tanzer) Date: Thu, 08 Oct 2009 14:51:00 -0000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: Your message of "Thu, 08 Oct 2009 10:24:33 -0400" <4ACDF621.4070603@trueblade.com> References: <4ACDF621.4070603@trueblade.com> Message-ID: Eric Smith wrote at Thu, 08 Oct 2009 10:24:33 -0400: > Vinay Sajip wrote: > > BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see if it was > > worth raising an enhancement request on the bug tracker using "O" to generate > > compatible rendering for octals. > > I didn't get your message, could you resend?. > > I was thinking the same thing, but it seems like a transition step. I'd > rather not keep such backward compatibility hacks (if you will) around > for the long haul. How about a flag (maybe '*') at the start of the > format specification which says "operate in backward compatibility > mode"? We could document it as being only useful for the % to {} > translator, and promise to remove it at some point in the future. Either > actually deprecate it or just promise to deprecate it in the future. That doesn't seem very useful to me. IIUC, the point of the translator is to allow porting of the millions of existing %-formating operations to the new-style .format. If the result of that is deprecated or removed a few years from now, all maintainers of long existing code have exactly the same problem. IMHO, either the translation is done once and gives identical output or it isn't worth doing at all. -- Christian Tanzer http://www.c-tanzer.at/ From eric at trueblade.com Thu Oct 8 17:27:14 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 08 Oct 2009 11:27:14 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> References: <4ACDF621.4070603@trueblade.com> <4ACDFD59.4060200@trueblade.com> <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> Message-ID: <4ACE04D2.1010100@trueblade.com> Benjamin Peterson wrote: > 2009/10/8 Eric Smith : >> Christian Tanzer wrote: >>> IMHO, either the translation is done once and gives identical output or >>> it isn't worth doing at all. >> I disagree. I doubt even 0.001% of all format strings involve octal >> formatting. Is it really worth not providing a transition path if it can't >> cover this case? > > It's also really easy to just write 0{:o} like my translator does. > > I apologize for not having looked at anyone's implementation yet. But I suspect you'll have a problem with this case unless int.__format__ has special logic for backward compatible octal formatting: >>> '%# 5o' % 8 ' 010' Eric. From solipsis at pitrou.net Thu Oct 8 17:38:48 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2009 15:38:48 +0000 (UTC) Subject: [Python-Dev] locals() different behaviour when tracing? References: Message-ID: Anders Waldenborg 0x63.nu> writes: > > Is it intentional that locals() returns a copy/snapshot of the local > variables sometimes, and sometimes a reference? Like when enabling > tracing, such as in the code pasted below. Since someone else opened a bug, I answered there. Anyone, feel free to correct me if my answer is wrong. http://bugs.python.org/issue7083 Regards Antoine. From python at mrabarnett.plus.com Thu Oct 8 17:52:20 2009 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 08 Oct 2009 16:52:20 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACDF621.4070603@trueblade.com> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4ACDD2C3.4090608@gmail.com> <4ACDF621.4070603@trueblade.com> Message-ID: <4ACE0AB4.8030707@mrabarnett.plus.com> Eric Smith wrote: > Vinay Sajip wrote: >> BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see >> if it was >> worth raising an enhancement request on the bug tracker using "O" to >> generate >> compatible rendering for octals. > > I didn't get your message, could you resend?. > > I was thinking the same thing, but it seems like a transition step. I'd > rather not keep such backward compatibility hacks (if you will) around > for the long haul. How about a flag (maybe '*') at the start of the > format specification which says "operate in backward compatibility > mode"? We could document it as being only useful for the % to {} > translator, and promise to remove it at some point in the future. Either > actually deprecate it or just promise to deprecate it in the future. > I think the flag should be '%'. From fuzzyman at voidspace.org.uk Thu Oct 8 17:56:35 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 08 Oct 2009 16:56:35 +0100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <4ACDFD59.4060200@trueblade.com> Message-ID: <4ACE0BB3.90206@voidspace.org.uk> Christian Tanzer wrote: > [snip...] >>> IMHO, either the translation is done once and gives identical output or >>> it isn't worth doing at all. >>> >> I disagree. I doubt even 0.001% of all format strings involve octal >> formatting. Is it really worth not providing a transition path if it >> can't cover this case? >> > > If %-formatting is first deprecated then removed from Python and there > is no automatic transition path that effectively means that existing > code using %-formatting is forced to stay at whatever Python version > was the last one supporting %-formatting. > > I surely hope nobody is seriously considering such a scenario. Perl 6 > seems harmless in comparison. > That is vastly overstating it. Making 'with' and 'as' keywords and removing string exceptions (which have already happened) will affect far more programs than a minor incompatibility in transitioning string formatting. Michael > -- > Christian Tanzer http://www.c-tanzer.at/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From python at mrabarnett.plus.com Thu Oct 8 17:57:35 2009 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 08 Oct 2009 16:57:35 +0100 Subject: [Python-Dev] Logging, Unicode and sockets In-Reply-To: References: Message-ID: <4ACE0BEF.6070609@mrabarnett.plus.com> Vinay Sajip wrote: > Thanks to > > http://bugs.python.org/issue7077 > > I've noticed that the socket-based logging handlers - SocketHandler, > DatagramHandler and SysLogHandler - aren't Unicode-aware and can break in the > presence of Unicode messages. I'd like to fix this by giving these handlers an > optional (encoding=None) parameter in their __init__, and then using this to > encode on output. If no encoding is specified, is it best to use > locale.getpreferredencoding(), sys.getdefaultencoding(), > sys.getfilesystemencoding(), 'utf-8' or something else? On my system: > >>>> sys.getdefaultencoding() > 'ascii' >>>> sys.getfilesystemencoding() > 'mbcs' >>>> locale.getpreferredencoding() > 'cp1252' > > which suggests to me that the locale.getpreferredencoding() should be the > default. However, as I'm not a Unicode maven, any suggestions would be welcome. > Well, encodings can vary from machine to machine, and if the encoding doesn't cover all the Unicode codepoints then you could get an encoding exception. For these reasons I'd vote for UTF-8. From doug.hellmann at gmail.com Thu Oct 8 18:00:35 2009 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Thu, 8 Oct 2009 12:00:35 -0400 Subject: [Python-Dev] A new way to configure logging In-Reply-To: <137556.63116.qm@web25801.mail.ukl.yahoo.com> References: <09B60FD3-D227-43E3-B901-5418C7FA950C@gmail.com> <137556.63116.qm@web25801.mail.ukl.yahoo.com> Message-ID: <98209F12-9EA9-462C-8330-025B8254B6DA@gmail.com> On Oct 8, 2009, at 10:47 AM, Vinay Sajip wrote: >> I've had bad experiences in the past with dictionary-based APIs. >> They seem > >> "simpler" in the short run, because the user "only needs to create >> some >> dictionaries". Once the complexity of that nested dictionary grows >> to a certain >> point, though, one has to refer back to documentation constantly to >> make sure > > Fair point, and agreed that the schema needs some care, here's > roughly what I'm thinking of as an example configuration (YAML): > > formatters: > brief: > format: '%(levelname)-8s: %(name)-15s: %(message)s' > precise: > format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s' > filters: > allow_foo: > name: foo > handlers: > console: > class : logging.StreamHandler > formatter: brief > level : INFO > stream : sys.stdout > filters: [allow_foo] > file: > class : logging.handlers.RotatingFileHandler > formatter: precise > filename: logconfig.log > maxBytes: 1024 > backupCount: 3 > debugfile: > class : logging.FileHandler > formatter: precise > filename: logconfig-detail.log > mode: a > loggers: > foo: > level : ERROR > handlers: [debugfile] > spam: > level : CRITICAL > handlers: [debugfile] > propagate: no > bar.baz: > level: WARNING > > root: > level : DEBUG > handlers : [console, file] > > It's not too deeply nested, and I can't see any need for it being > more deeply nested. It more or less mirrors the way you'd have to > write the corresponding ConfigParser-compatible configuration, but > if you use a dict as a common format, you have the benefits which I > mentioned of supporting JSON, YAML or Python code in a Django > settings.py. Yes, that makes sense. >> the structure conforms to the "schema". Building a simple config >> tree using >> light-weight classes with documented APIs tends to be more >> sustainable in the >> long run. > > When you say 'config tree using light-weight classes', I presume you > mean a parallel set of classes to the actual logging classes which > will be instantiated? ISTM logging classes are already reasonably > usable for programmatic configuration, but this doesn't address e.g. > updating configurations easily without program code changes, or the > ability to configure from say a YAML node, where YAML may be being > used for application configuration as a whole. (Unless of course > I've misunderstood what you're getting at.) The probably are light-weight enough, but I didn't want to assume those classes would be used directly. I suspect they could be, though. I guess there aren't *so* many levels or keys that the data structure will become unwieldy. Doug From jnoller at gmail.com Thu Oct 8 18:04:38 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 8 Oct 2009 12:04:38 -0400 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACDEEB9.2000308@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDEEB9.2000308@voidspace.org.uk> Message-ID: <4222a8490910080904m62947983w6900b816623565a5@mail.gmail.com> On Thu, Oct 8, 2009 at 9:52 AM, Michael Foord wrote: > Simon Cross wrote: >> >> ?On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad? >> wrote: >> >>> >>> = Virtualenv and the multiple version support in Distribute = >>> >> >> ... >> >>> >>> My opinion is that this tool exists only because Python doesn't >>> support the installation of multiple versions for the same >>> distributions. >>> >> >> This is not at all how I use virtualenv. For me virtualenv is a >> sandbox so that I don't have to become root whenever I need to install >> a Python package for testing purposes and to allow me to hop between >> sets of installed Python packages while developing on multiple Python >> projects. I also use it as a sandbox for build bots so that multiple >> bots on the same machine can each build their own projects with just >> the known dependencies installed. >> >> > > This is exactly why I use virtualenv as well. I don't recall ever having > wanted / needed to install multiple versions of the same library - whilst I > can appreciate that it *can* be a real issue it has never been a problem for > me. > > Michael +1 - virtualenv, AFAIK is used for sandboxing/isolation of various environments, not dealing with multiple versions within the *same* environment. Of course, it does solve the "being dependent on a specific version" of a dependency because it *is* sandboxed from everything else. Adding multiple version support doesn't remove the need for sandboxing. From tanzer at swing.co.at Thu Oct 8 18:08:41 2009 From: tanzer at swing.co.at (Christian Tanzer) Date: Thu, 08 Oct 2009 16:08:41 -0000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: Your message of "Thu, 08 Oct 2009 16:56:35 +0100" <4ACE0BB3.90206@voidspace.org.uk> References: <4ACE0BB3.90206@voidspace.org.uk> Message-ID: Michael Foord wrote at Thu, 08 Oct 2009 16:56:35 +0100: > > If %-formatting is first deprecated then removed from Python and there > > is no automatic transition path that effectively means that existing > > code using %-formatting is forced to stay at whatever Python version > > was the last one supporting %-formatting. > > > > I surely hope nobody is seriously considering such a scenario. Perl 6 > > seems harmless in comparison. > > > That is vastly overstating it. Making 'with' and 'as' keywords and > removing string exceptions (which have already happened) will affect far > more programs than a minor incompatibility in transitioning string > formatting. `with` and `as` are trivial to fix and certainly not pervasive in existing code. String exceptions have been deprecated for years. -- Christian Tanzer http://www.c-tanzer.at/ From fuzzyman at voidspace.org.uk Thu Oct 8 18:08:57 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 08 Oct 2009 17:08:57 +0100 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACDCA5A.4090609@cheimes.de> References: <4ACDCA5A.4090609@cheimes.de> Message-ID: <4ACE0E99.7050102@voidspace.org.uk> Christian Heimes wrote: > [snip...] > The altered user site directories are: > > CPython: > ~/.local/lib/python2.6/site-packages > %APPDATA%/Python/Python26 > > IronPython: > ~/.local/lib/ironpython2.6/site-packages > %APPDATA%/Python/IronPython26 > > Jython: > ~/.local/lib/jython2.6/site-packages > %APPDATA%/Python/Jython26 > > +1 I really like this scheme. The important thing for IronPython is that we can get it into Python 2.6 (along with other fixes to make distutils compatible with IronPython - like not attempting to bytecode-compile when sys.dont_write_bytecode is True). All the best, Michael Foord -- http://www.ironpythoninaction.com/ From benjamin at python.org Thu Oct 8 17:07:53 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 8 Oct 2009 10:07:53 -0500 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACDFD59.4060200@trueblade.com> References: <4ACDF621.4070603@trueblade.com> <4ACDFD59.4060200@trueblade.com> Message-ID: <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> 2009/10/8 Eric Smith : > Christian Tanzer wrote: >> IMHO, either the translation is done once and gives identical output or >> it isn't worth doing at all. > > I disagree. I doubt even 0.001% of all format strings involve octal > formatting. Is it really worth not providing a transition path if it can't > cover this case? It's also really easy to just write 0{:o} like my translator does. -- Regards, Benjamin From a.badger at gmail.com Thu Oct 8 18:11:49 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 8 Oct 2009 09:11:49 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACDCCBD.4040908@egenix.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> Message-ID: <20091008161148.GG28231@clingman.lan> On Thu, Oct 08, 2009 at 01:27:57PM +0200, M.-A. Lemburg wrote: > > Tarek Ziad? a ?crit : > >> But if PEP 376 and PEP 386 support are added in Python, we're not far > >> from being able to provide multiple version support with > >> the help of importlib. > > Before putting much work into this: do you really think that having > multiple versions of the same package in the same Python installation > is a good idea ? > I think it is a good idea. > Examples: > What if you have an application that uses two modules which each > require two different versions of the same package ? Would that > load the same package twice, causing globals used by these package > to no work (e.g. isinstance(x, class_name) or global caches) ? > That's not how it should work. Look at other systems that allow for installing multiple versions of a library -- for instance, loading dynamic shared objects in C * You can install multiple versions of a library in parallel * The dynamic loader will pick the version of the library that is appropriate from the list of available options (the program specifies the SONAME it needs -- library name plus API version. The loader then chooses the most recent revision that matches that API version.) * When one binary needs multiple API versions of the library, the application cannot start. The last point addresses your concern -- depending on multiple, incompatible versions of a library is prohibited. The programmer of the application needs to make the code run with a single version of the code. > This sounds a lot like DLL- or RPM-hell to me. > RPM-hell (I'm not sure if DLL hell is the same, I have the vague impression that it is the lack of enough version specification rather than too much but I don't know for sure). is similar but it takes place on the end-user's system. This should take place on the programmer's system instead. End-users are not in a position to fix things like RPM-hell. Programmers are. Example RPM-hell: Application Foo requires libbar-1.x Application Baz requires libbar-2.x The user may either have Foo or Baz installed on their system with the appropriate libbar but not both. They depend on the packagers and developers of Foo and Bar to do one of the following to resolve the situation: * Port Foo and Baz to use the same version of libbar. * Package libbar in such a way that libbar-1 and libbar-2 are parallel installable on the system. Then they can install two separate packages, libbar1-1.0 and libbar2-2.0. Example of similar Distutils multiple version problem: The programmer creates an application Foo that depends on python-bar-1.x. He has recently started work on a file that imports python-baz-1.0. python-baz depends on python-bar-2.x. The first time he tries to run his new code, python gives him an error message that it is impossible to satisfy the version requirements for python-bar. Depending on how the versions are specified, the error message could be very specific and helpful: Impossible version requirements: bar Requires: python-baz>=2.0, < 3.0 foo.py Requires: python-baz >=1.0, < 2.0 The programmer can then discard their new code, port foo.py to python-baz-2.x, or port python-bar to python-baz-1.x and submit a patch to the upstream of that module. Note two things about this scenario: 1) The programmer is the person who is responsible for creating the conflict and for resolving it. They are the proper authority for making the decision to port to python-baz-2.x or not using python-bar. The end-user who is not responsible is not impacted by this at all. 2) The programmer would have had to deal with this issue whether we allow multiple versions to be installed or not. With multiple version support we may be able to get them better error messages (depending on how the dependency information is formatted and how completely it was specified in the app and modules). > I think it's much better to keep things simple and under user > control, e.g. by encouraging use of virtualenv-like setups > and perhaps adding better native support for these to Python. > > If the user finds that there's a version conflict this can > then be resolved during environment setup instead of hoping > for the best and waiting for application failures at run-time. > For the class of user that is actually a developer, it might be somewhat true that version conflicts should be resolved by them. But for the class of user that is an end-user, version conflicts are a totally foreign concept. They should be dealt with by the person who is coding the application for them. Also note, the ability to have multiple versions makes things easier for system packagers and provides an easy alternative to a virtualenv for end-users. * System packagers: virtualenv does not provide a method suitable for system packagers. The nearest adaptation would be for the system packager to install python packages into their own hierarchy not in the PYTHONPATH. Then they would need to create a virtualenv-like directory that symlinks to the packages in that directory. Then they would need to write a wrapper script for the application that put that virtualenv-like directory into the PYTHONPATH before any other site package directories and have that wrapper call the real binary. This is needless complication for the typical virtualenv install so the work is not likely to be done there and it's incredibly hacky for the system packager so the work is not likely to be done there. * End users: virtualenv creates a whole environment for the application to live in. If python understood how to use multiple versions then we'd only need to install the versions of packages that didn't match up with what's already on the system into the user's personal site-package directory. Then the module loader would take care of loading the module from the personal site-packages since it met the version requirements instead of the system copy. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From a.badger at gmail.com Thu Oct 8 18:17:32 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 8 Oct 2009 09:17:32 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <20091008161732.GH28231@clingman.lan> On Thu, Oct 08, 2009 at 02:52:24PM +0200, Simon Cross wrote: > On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad? wrote: > > = Virtualenv and the multiple version support in Distribute = > ... > > My opinion is that this tool exists only because Python doesn't > > support the installation of multiple versions for the same > > distributions. > Let's actually look at these reasons: > This is not at all how I use virtualenv. For me virtualenv is a > sandbox so that I don't have to become root whenever I need to install > a Python package for testing purposes This is needing to install multiple versions and use the newly installed version for testing. > and to allow me to hop between > sets of installed Python packages while developing on multiple Python > projects. This is the ability to install multiple versions and specify different versions for different projects you're working on. > I also use it as a sandbox for build bots so that multiple > bots on the same machine can each build their own projects with just > the known dependencies installed. > This is the only use in the list that is virtualenv specific. The rest are cases of needing to install multiple versions on the system. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From doug.hellmann at gmail.com Thu Oct 8 14:21:15 2009 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Thu, 8 Oct 2009 08:21:15 -0400 Subject: [Python-Dev] A new way to configure logging In-Reply-To: References: Message-ID: <09B60FD3-D227-43E3-B901-5418C7FA950C@gmail.com> On Oct 7, 2009, at 10:49 AM, Vinay Sajip wrote: > All three of the contenders for the title of "commonly found > configuration > mechanism" - JSON, YAML and Python code - will be expressible, in > Python, as > Python dicts. So it seems to make sense to add, to logging.config, a > new > callable bound to "dictConfig" which will take a single dictionary > argument and > configure logging from that dictionary. I've had bad experiences in the past with dictionary-based APIs. They seem "simpler" in the short run, because the user "only needs to create some dictionaries". Once the complexity of that nested dictionary grows to a certain point, though, one has to refer back to documentation constantly to make sure the structure conforms to the "schema". Building a simple config tree using light-weight classes with documented APIs tends to be more sustainable in the long run. Doug From benjamin at python.org Thu Oct 8 16:21:56 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 8 Oct 2009 09:21:56 -0500 Subject: [Python-Dev] locals() different behaviour when tracing? In-Reply-To: References: Message-ID: <1afaf6160910080721u459320c3va63baffe2b7137d0@mail.gmail.com> 2009/10/8 Anders Waldenborg : > > Is it intentional that locals() returns a copy/snapshot of the local > variables sometimes, and sometimes a reference? Like when enabling tracing, > such as in the code pasted below. The documentation ("Update and return a > dictionary containing the current scope's local variables.") is pretty > unclear. Yes, it does, and that's why the documentation says changing it is undefined. :) -- Regards, Benjamin From solipsis at pitrou.net Thu Oct 8 18:28:52 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2009 16:28:52 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Distutils_and_Distribute_roadmap_=28and_so?= =?utf-8?q?me=09words=09on_Virtualenv=2C_Pip=29?= References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> Message-ID: Toshio Kuratomi gmail.com> writes: > > This is needing to install multiple versions and use the newly installed > version for testing. [...] What you're missing is that having separate environments has a virtue of cleanliness, understandability and robustness that a multiple-versioned solution doesn't have. While the technical merits are debatable I'm sure some people definitely prefer to manage a virtualenv-based version. Regards Antoine. From fuzzyman at voidspace.org.uk Thu Oct 8 18:30:00 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 08 Oct 2009 17:30:00 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <20091008161732.GH28231@clingman.lan> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> Message-ID: <4ACE1388.5020602@voidspace.org.uk> Toshio Kuratomi wrote: > On Thu, Oct 08, 2009 at 02:52:24PM +0200, Simon Cross wrote: > >> On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad? wrote: >> >>> = Virtualenv and the multiple version support in Distribute = >>> >> ... >> >>> My opinion is that this tool exists only because Python doesn't >>> support the installation of multiple versions for the same >>> distributions. >>> > Let's actually look at these reasons: > > >> This is not at all how I use virtualenv. For me virtualenv is a >> sandbox so that I don't have to become root whenever I need to install >> a Python package for testing purposes >> > > This is needing to install multiple versions and use the newly installed > version for testing. > > Not really - it is wanting to install a single version of a library that you don't want to install into your 'main' (whether that be user or system) Python install. It is sandboxing and orthogonal to multiple versions. >> and to allow me to hop between >> sets of installed Python packages while developing on multiple Python >> projects. >> > > This is the ability to install multiple versions and specify different > versions for different projects you're working on. > No, it is working on multiple projects that have *different* dependencies and being able to work in an environment that *only* has direct dependencies installed - again sandboxed from your main Python environment. The fact that virtualenv allows you to have multiple different versions of the same library installed in the different environments you create is completely separate from the motivation that causes many people to use it. What virtualenv *doesn't* do (I believe) is allow you to have multiple versions of libraries installed within a single environment and switch between them (at least it doesn't offer anything beyond what setuptools or pip provides). Michael > >> I also use it as a sandbox for build bots so that multiple >> bots on the same machine can each build their own projects with just >> the known dependencies installed. >> >> > This is the only use in the list that is virtualenv specific. The rest are > cases of needing to install multiple versions on the system. > > -Toshio > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From masklinn at masklinn.net Thu Oct 8 18:35:45 2009 From: masklinn at masklinn.net (Masklinn) Date: Thu, 8 Oct 2009 18:35:45 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <20091008161732.GH28231@clingman.lan> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> Message-ID: <7DD3FAEA-88FE-48EE-9B92-972F5C8EFB3E@masklinn.net> On 8 Oct 2009, at 18:17 , Toshio Kuratomi wrote: > >> This is not at all how I use virtualenv. For me virtualenv is a >> sandbox so that I don't have to become root whenever I need to >> install >> a Python package for testing purposes > > This is needing to install multiple versions and use the newly > installed > version for testing. > No it's not. It's keeping the python package *being tested* out of the system's or user's site-package because it's potentially unstable or unneeded. It provides a trivial way of *removing* the package to get rid of it: delete the virtualenv. No trace anywhere that the package was ever installed, no impact on the system (apart from the potential side-effects of executing the system). The issue here isn't "multiple installed packages", it will more than likely be the only version of itself: note that it's a package being tested, not an *upgrade* being tested. The issues solved are: * not having to become root (solved by PEP 370 if it ever lands) * minimizing as much as possible the impact of testing the package on the system (not solved by any other solution) >> and to allow me to hop between >> sets of installed Python packages while developing on multiple Python >> projects. > > This is the ability to install multiple versions and specify different > versions for different projects you're working on. > No, this is the ability to not needlessly clutter site-packages, keep them clean, tight, focused; and the ability to keep a project's environment (such as its dependencies) clear and repeatable. Nowhere was it indicated that multiple versions were involved. Both of those *might* imply issues of multiple versions concurrently installed on the system, and virtualenv would incidentally solve these issues, but these issues are *not* the core of either use case. They're at best peripheral and potential. From masklinn at masklinn.net Thu Oct 8 18:37:40 2009 From: masklinn at masklinn.net (Masklinn) Date: Thu, 8 Oct 2009 18:37:40 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <20091008161732.GH28231@clingman.lan> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> Message-ID: <7FC2E2D2-16AF-490B-9E9B-FB465C02D357@masklinn.net> On 8 Oct 2009, at 18:17 , Toshio Kuratomi wrote: >> This is not at all how I use virtualenv. For me virtualenv is a >> sandbox so that I don't have to become root whenever I need to >> install >> a Python package for testing purposes > > This is needing to install multiple versions and use the newly > installed > version for testing. > No it's not. It's keeping the python package *being tested* out of the system's or user's site-package because it's potentially unstable or unneeded. It provides a trivial way of *removing* the package to get rid of it: delete the virtualenv. No trace anywhere that the package was ever installed, no impact on the system (apart from the potential side-effects of executing the system). The issue here isn't "multiple installed packages", it will more than likely be the only version of itself: note that it's a package being tested, not an *upgrade* being tested. The issues solved are: * not having to become root (solved by PEP 370 if it ever lands) * minimizing as much as possible the impact of testing the package on the system (not solved by any other solution) >> and to allow me to hop between >> sets of installed Python packages while developing on multiple Python >> projects. > > This is the ability to install multiple versions and specify different > versions for different projects you're working on. > No, this is the ability to not needlessly clutter site-packages, keep them clean, tight, focused; and the ability to keep a project's environment (such as its dependencies) clear and repeatable. Nowhere was it indicated that multiple versions were involved. Both of those *might* imply issues of multiple versions concurrently installed on the system, and virtualenv would incidentally solve these issues, but these issues are *not* the core of either use case. They're at best peripheral and potential From ziade.tarek at gmail.com Thu Oct 8 18:44:08 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 8 Oct 2009 18:44:08 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACDFD54.3080202@egenix.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> <94bdd2610910080554m530bfdbeg2bf010c813c71873@mail.gmail.com> <4ACDFD54.3080202@egenix.com> Message-ID: <94bdd2610910080944q1785aad0nc15937610701190f@mail.gmail.com> On Thu, Oct 8, 2009 at 4:55 PM, M.-A. Lemburg wrote: > OTOH, the register command sends the meta-data directly to > the PyPI database, so it doesn't even need another file > for storing away the meta data. > > Unless I'm missing something important (which I probably am), > the only added value of the new format over PKG-INFO is the > micro-language. That's one. The other one is to avoid to deal with yet another configuration file alongside setup.py, and just deal with a [metadata]?section in setup.cfg. And PKG-INFO stays a static, platform-specific file that exists only when the distribution is installed on the target platform. >> No, but if you ask for it I can write it. > > Please do. The format will need documentation anyway and the > motivation should be made clear as well. Ok I will Tarek From p.f.moore at gmail.com Thu Oct 8 00:30:26 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2009 23:30:26 +0100 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: References: <9d153b7c0909281414w55515a9blec86d55a3fdeb362@mail.gmail.com> <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> <4ACC7B03.3040700@avl.com> Message-ID: <79990c6b0910071530x2bd08dc7vc90c66d6f48c3a01@mail.gmail.com> 2009/10/7 Antoine Pitrou : > Python 3 complains at startup and is a bit more explicit: > > $ ./python -c '1' >&- > Fatal Python error: Py_Initialize: can't initialize sys standard streams > OSError: [Errno 9] Bad file descriptor > Abandon > > Of course, if it is stderr that you explicitly close, you lose all reporting: > > $ ./python -c '1' 2>&- > Abandon Hmm, how does Python 3's pythonw work on Windows? (I don't have a Windows installation of Python 3 to hand at the moment) Paul From vinay_sajip at yahoo.co.uk Thu Oct 8 18:51:53 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 16:51:53 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?transitioning_from_=25_to_=7B=7D_formattin?= =?utf-8?q?g?= References: <4ACDF621.4070603@trueblade.com> <4ACDFD59.4060200@trueblade.com> <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> Message-ID: Benjamin Peterson python.org> writes: > It's also really easy to just write 0{:o} like my translator does. How does that cope when handed a negative number to format? >>> "%#0o" % -1234 '-02322' >>> "0{0:o}".format(-1234) '0-2322' Regards, Vinay Sajip From guido at python.org Thu Oct 8 18:55:11 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Oct 2009 09:55:11 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: References: <4ACDFD59.4060200@trueblade.com> Message-ID: On Thu, Oct 8, 2009 at 8:08 AM, Christian Tanzer wrote: > All that talk of deprecating %-formatting makes me really nervous. > %-formatting is pervasive in all existing Python code. > > Without an automatic translator that is 100% accurate, porting all > that code to {}-formatting is not possible. Heck, it's not even > possible to grep for all instances of %-formatting. > > How do you suppose that maintainers could ever do the transition from > %- to {}-formatting manually? This is pretty much the situation with integer division (you can only recognize it by running the code), and yet we figured a way to change that in 3.x. Or take classic classes vs. new-style classes. They cannot be translated 100% automatically either. The solution is to support the old and new style in parallel for a really long time -- we did this with int division (read PEP 238), we did it with classes, and we can do it again with formatting. Unless I missed something, we're not planning to remove %-formatting until Python 4.0 comes along, which we won't even start until a long time after everyone has switched to some version of 3.x. So the same approach will apply: support both forms, nudge people to start using the new form, wait, nudge some more, etc. So, yes, we will continue to make noise about this. And yes you should opportunistically migrate your code to {}-formatting, like when you're rewriting some code anyway. One of the nice things about {}-formatting is that in most cases (things like the logging API excluded) you can change it one format string at a time. And no, the sky isn't falling. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kiorky at cryptelium.net Thu Oct 8 18:56:19 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 08 Oct 2009 18:56:19 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <20091008161148.GG28231@clingman.lan> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> <20091008161148.GG28231@clingman.lan> Message-ID: <4ACE19B3.4060106@cryptelium.net> Toshio Kuratomi a ?crit : > On Thu, Oct 08, 2009 at 01:27:57PM +0200, M.-A. Lemburg wrote: >>> Tarek Ziad? a ?crit : >>>> But if PEP 376 and PEP 386 support are added in Python, we're not far >>>> from being able to provide multiple version support with >>>> the help of importlib. >> Before putting much work into this: do you really think that having >> multiple versions of the same package in the same Python installation >> is a good idea ? >> > I think it is a good idea. > >> Examples: >> What if you have an application that uses two modules which each >> require two different versions of the same package ? Would that >> load the same package twice, causing globals used by these package >> to no work (e.g. isinstance(x, class_name) or global caches) ? >> > That's not how it should work. Look at other systems that allow for > installing multiple versions of a library -- for instance, loading dynamic > shared objects in C > * You can install multiple versions of a library in parallel > * The dynamic loader will pick the version of the library that is > appropriate from the list of available options (the program specifies the > SONAME it needs -- library name plus API version. The loader then > chooses the most recent revision that matches that API version.) > * When one binary needs multiple API versions of the library, the > application cannot start. > > The last point addresses your concern -- depending on multiple, incompatible > versions of a library is prohibited. The programmer of the application > needs to make the code run with a single version of the code. > >> This sounds a lot like DLL- or RPM-hell to me. >> > RPM-hell (I'm not sure if DLL hell is the same, I have the vague impression > that it is the lack of enough version specification rather than too much but > I don't know for sure). is similar but it takes place on the end-user's > system. This should take place on the programmer's system instead. > End-users are not in a position to fix things like RPM-hell. Programmers > are. > > Example RPM-hell: > Application Foo requires libbar-1.x > Application Baz requires libbar-2.x > > The user may either have Foo or Baz installed on their system with the > appropriate libbar but not both. They depend on the packagers and > developers of Foo and Bar to do one of the following to resolve the > situation: > > * Port Foo and Baz to use the same version of libbar. > * Package libbar in such a way that libbar-1 and libbar-2 are parallel > installable on the system. Then they can install two separate packages, > libbar1-1.0 and libbar2-2.0. > > Example of similar Distutils multiple version problem: > The programmer creates an application Foo that depends on python-bar-1.x. He > has recently started work on a file that imports python-baz-1.0. python-baz > depends on python-bar-2.x. The first time he tries to run his new code, > python gives him an error message that it is impossible to satisfy the > version requirements for python-bar. Depending on how the versions are > specified, the error message could be very specific and helpful: > > Impossible version requirements: > bar Requires: python-baz>=2.0, < 3.0 > foo.py Requires: python-baz >=1.0, < 2.0 > > The programmer can then discard their new code, port foo.py to > python-baz-2.x, or port python-bar to python-baz-1.x and submit a patch to > the upstream of that module. Note two things about this scenario: > > 1) The programmer is the person who is responsible for creating the conflict > and for resolving it. They are the proper authority for making the decision > to port to python-baz-2.x or not using python-bar. The end-user who is not > responsible is not impacted by this at all. > 2) The programmer would have had to deal with this issue whether we allow > multiple versions to be installed or not. With multiple version support we > may be able to get them better error messages (depending on how the > dependency information is formatted and how completely it was specified in > the app and modules). > >> I think it's much better to keep things simple and under user >> control, e.g. by encouraging use of virtualenv-like setups >> and perhaps adding better native support for these to Python. >> >> If the user finds that there's a version conflict this can >> then be resolved during environment setup instead of hoping >> for the best and waiting for application failures at run-time. >> > For the class of user that is actually a developer, it might be somewhat > true that version conflicts should be resolved by them. But for the class > of user that is an end-user, version conflicts are a totally foreign > concept. They should be dealt with by the person who is coding the > application for them. > > Also note, the ability to have multiple versions makes things easier for > system packagers and provides an easy alternative to a virtualenv for > end-users. > > * System packagers: virtualenv does not provide a method suitable for system Yes, there is no doubt virtualenv is useless for system packagers but: * System and applications deployment have not to be tied. It s up to the user to install things system wide or to use locally isolation technics. Virtualenv is one of those. As a conclusion, there are not very much problem for system packagers as if their users have specific needs, they will do something Outside the system. If not, they use their global python with packages installed in that global one. > packagers. The nearest adaptation would be for the system packager to > install python packages into their own hierarchy not in the PYTHONPATH. > Then they would need to create a virtualenv-like directory that symlinks > to the packages in that directory. Then they would need to write a > wrapper script for the application that put that virtualenv-like directory > into the PYTHONPATH before any other site package directories and have > that wrapper call the real binary. This is needless complication for the > typical virtualenv install so the work is not likely to be done there and > it's incredibly hacky for the system packager so the work is not likely to > be done there. > * End users: virtualenv creates a whole environment for the application to > live in. If python understood how to use multiple versions then we'd only > need to install the versions of packages that didn't match up with what's > already on the system into the user's personal site-package directory. > Then the module loader would take care of loading the module from the > personal site-packages since it met the version requirements instead of > the system copy. > I dont want to have interaction between my different hosted projects, isolation is the only thing i want. As many other people there already. I do not want at all to have to deal with multiple version picking system and debug an application at 2:00am because of that. > -Toshio > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/kiorky%40cryptelium.net -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From kiorky at cryptelium.net Thu Oct 8 19:05:09 2009 From: kiorky at cryptelium.net (kiorky) Date: Thu, 08 Oct 2009 19:05:09 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <20091008161148.GG28231@clingman.lan> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> <20091008161148.GG28231@clingman.lan> Message-ID: <4ACE1BC5.2070706@cryptelium.net> Toshio Kuratomi a ?crit : > On Thu, Oct 08, 2009 at 01:27:57PM +0200, M.-A. Lemburg wrote: >>> Tarek Ziad? a ?crit : >>>> But if PEP 376 and PEP 386 support are added in Python, we're not far >>>> from being able to provide multiple version support with >>>> the help of importlib. >> Before putting much work into this: do you really think that having >> multiple versions of the same package in the same Python installation >> is a good idea ? >> > I think it is a good idea. > >> Examples: >> What if you have an application that uses two modules which each >> require two different versions of the same package ? Would that >> load the same package twice, causing globals used by these package >> to no work (e.g. isinstance(x, class_name) or global caches) ? >> > That's not how it should work. Look at other systems that allow for > installing multiple versions of a library -- for instance, loading dynamic > shared objects in C > * You can install multiple versions of a library in parallel > * The dynamic loader will pick the version of the library that is > appropriate from the list of available options (the program specifies the > SONAME it needs -- library name plus API version. The loader then > chooses the most recent revision that matches that API version.) > * When one binary needs multiple API versions of the library, the > application cannot start. > > The last point addresses your concern -- depending on multiple, incompatible > versions of a library is prohibited. The programmer of the application > needs to make the code run with a single version of the code. > >> This sounds a lot like DLL- or RPM-hell to me. >> > RPM-hell (I'm not sure if DLL hell is the same, I have the vague impression > that it is the lack of enough version specification rather than too much but > I don't know for sure). is similar but it takes place on the end-user's > system. This should take place on the programmer's system instead. > End-users are not in a position to fix things like RPM-hell. Programmers > are. > > Example RPM-hell: > Application Foo requires libbar-1.x > Application Baz requires libbar-2.x > > The user may either have Foo or Baz installed on their system with the > appropriate libbar but not both. They depend on the packagers and > developers of Foo and Bar to do one of the following to resolve the > situation: > > * Port Foo and Baz to use the same version of libbar. > * Package libbar in such a way that libbar-1 and libbar-2 are parallel > installable on the system. Then they can install two separate packages, > libbar1-1.0 and libbar2-2.0. > > Example of similar Distutils multiple version problem: > The programmer creates an application Foo that depends on python-bar-1.x. He > has recently started work on a file that imports python-baz-1.0. python-baz > depends on python-bar-2.x. The first time he tries to run his new code, > python gives him an error message that it is impossible to satisfy the > version requirements for python-bar. Depending on how the versions are > specified, the error message could be very specific and helpful: > > Impossible version requirements: > bar Requires: python-baz>=2.0, < 3.0 > foo.py Requires: python-baz >=1.0, < 2.0 > > The programmer can then discard their new code, port foo.py to > python-baz-2.x, or port python-bar to python-baz-1.x and submit a patch to > the upstream of that module. Note two things about this scenario: > > 1) The programmer is the person who is responsible for creating the conflict > and for resolving it. They are the proper authority for making the decision > to port to python-baz-2.x or not using python-bar. The end-user who is not > responsible is not impacted by this at all. > 2) The programmer would have had to deal with this issue whether we allow > multiple versions to be installed or not. With multiple version support we > may be able to get them better error messages (depending on how the > dependency information is formatted and how completely it was specified in > the app and modules). > >> I think it's much better to keep things simple and under user >> control, e.g. by encouraging use of virtualenv-like setups >> and perhaps adding better native support for these to Python. >> >> If the user finds that there's a version conflict this can >> then be resolved during environment setup instead of hoping >> for the best and waiting for application failures at run-time. >> > For the class of user that is actually a developer, it might be somewhat > true that version conflicts should be resolved by them. But for the class > of user that is an end-user, version conflicts are a totally foreign > concept. They should be dealt with by the person who is coding the > application for them. > > Also note, the ability to have multiple versions makes things easier for > system packagers and provides an easy alternative to a virtualenv for > end-users. > > * System packagers: virtualenv does not provide a method suitable for system Yes, there is no doubt virtualenv is useless for system packagers but: * System and applications deployment have not to be tied. It s up to the user to install things system wide or to use locally isolation technics. Virtualenv is one of those. As a conclusion, there are not very much problem for system packagers as if their users have specific needs, they will do something Outside the system. If not, they use their global python with packages installed in that global one. > packagers. The nearest adaptation would be for the system packager to > install python packages into their own hierarchy not in the PYTHONPATH. > Then they would need to create a virtualenv-like directory that symlinks > to the packages in that directory. Then they would need to write a > wrapper script for the application that put that virtualenv-like directory > into the PYTHONPATH before any other site package directories and have > that wrapper call the real binary. This is needless complication for the > typical virtualenv install so the work is not likely to be done there and > it's incredibly hacky for the system packager so the work is not likely to > be done there. > * End users: virtualenv creates a whole environment for the application to > live in. If python understood how to use multiple versions then we'd only > need to install the versions of packages that didn't match up with what's > already on the system into the user's personal site-package directory. > Then the module loader would take care of loading the module from the > personal site-packages since it met the version requirements instead of > the system copy. > I dont want to have interaction between my different hosted projects, isolation is the only thing i want. As many other people there already. I do not want at all to have to deal with multiple version picking system and debug an application at 2:00am because of that. > -Toshio > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/kiorky%40cryptelium.net -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From python at rcn.com Thu Oct 8 19:14:19 2009 From: python at rcn.com (Raymond Hettinger) Date: Thu, 8 Oct 2009 10:14:19 -0700 Subject: [Python-Dev] transitioning from % to {} formatting References: <4ACDFD59.4060200@trueblade.com> Message-ID: <4B9C0675D45347A887DBC074153641EA@RaymondLaptop1> [Christian Tanzer] >> How do you suppose that maintainers could ever do the transition from >> %- to {}-formatting manually? [Guido van Rossum] > This is pretty much the situation with integer division (you can only > recognize it by running the code), Do you think there may be some possible parallel to the -3 option to flag cases of %-formatting? If so, that could be helpful. > So, yes, we will continue to make noise about this. And yes you should > opportunistically migrate your code to {}-formatting, like when you're > rewriting some code anyway. One of the nice things about {}-formatting > is that in most cases (things like the logging API excluded) you can > change it one format string at a time. I've already have some code that mixes the styles (using {} for new stuff). Raymond From amauryfa at gmail.com Thu Oct 8 19:19:59 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 8 Oct 2009 19:19:59 +0200 Subject: [Python-Dev] PEP 389: argparse - new command line parsing module In-Reply-To: <79990c6b0910071530x2bd08dc7vc90c66d6f48c3a01@mail.gmail.com> References: <79990c6b0909291331u36bf0693r52ffd1a07e9ef500@mail.gmail.com> <4AC2846E.8060304@g.nevcal.com> <79990c6b0910010158k2edf20ceh9d6b4f5c47a37995@mail.gmail.com> <4ACC7B03.3040700@avl.com> <79990c6b0910071530x2bd08dc7vc90c66d6f48c3a01@mail.gmail.com> Message-ID: 2009/10/8 Paul Moore : > 2009/10/7 Antoine Pitrou : >> Python 3 complains at startup and is a bit more explicit: >> >> $ ./python -c '1' >&- >> Fatal Python error: Py_Initialize: can't initialize sys standard streams >> OSError: [Errno 9] Bad file descriptor >> Abandon >> >> Of course, if it is stderr that you explicitly close, you lose all reporting: >> >> $ ./python -c '1' 2>&- >> Abandon > > Hmm, how does Python 3's pythonw work on Windows? (I don't have a > Windows installation of Python 3 to hand at the moment) When running pythonw, fileno(stdout) is negative, so sys.stdout is set to None, and print() silently returns. But the situation is not perfect, see http://bugs.python.org/issue6501 where Apache provides a stdout, but python cannot determine the encoding because there is no console. -- Amaury Forgeot d'Arc From cournape at gmail.com Thu Oct 8 19:22:32 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Oct 2009 02:22:32 +0900 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <7DD3FAEA-88FE-48EE-9B92-972F5C8EFB3E@masklinn.net> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> <7DD3FAEA-88FE-48EE-9B92-972F5C8EFB3E@masklinn.net> Message-ID: <5b8d13220910081022h19835524sea287e3fb256b0c6@mail.gmail.com> On Fri, Oct 9, 2009 at 1:35 AM, Masklinn wrote: > On 8 Oct 2009, at 18:17 , Toshio Kuratomi wrote: >> >>> This is not at all how I use virtualenv. For me virtualenv is a >>> sandbox so that I don't have to become root whenever I need to install >>> a Python package for testing purposes >> >> This is needing to install multiple versions and use the newly installed >> version for testing. >> > No it's not. It's keeping the python package *being tested* out of the > system's or user's site-package because it's potentially unstable or > unneeded. It provides a trivial way of *removing* the package to get rid of > it: delete the virtualenv. No trace anywhere that the package was ever > installed, no impact on the system ?(apart from the potential side-effects > of executing the system). > > The issue here isn't "multiple installed packages", it will more than likely > be the only version of itself: note that it's a package being tested, not an > *upgrade* being tested. > > The issues solved are: > * not having to become root (solved by PEP 370 if it ever lands) > * minimizing as much as possible the impact of testing the package on the > system (not solved by any other solution) This is not true - stow solves the problem in a more general way (in the sense that it is not restricted to python), at least on platforms which support softlink. The only inconvenience of stow compared to virtual env is namespace packages, but that's because of a design flaw in namespace package (as implemented in setuptools, and hopefully fixed in the upcoming namespace package PEP). Virtualenv provides a possible solution to some deployment problems, and is useful in those cases, but it is too specialized to be included in python itself IMO. cheers, David From guido at python.org Thu Oct 8 19:33:39 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Oct 2009 10:33:39 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4B9C0675D45347A887DBC074153641EA@RaymondLaptop1> References: <4ACDFD59.4060200@trueblade.com> <4B9C0675D45347A887DBC074153641EA@RaymondLaptop1> Message-ID: On Thu, Oct 8, 2009 at 10:14 AM, Raymond Hettinger wrote: > Do you think there may be some possible parallel to the -3 option > to flag cases of %-formatting? ?If so, that could be helpful. Absolutely. This should be simple, since there's just one or two places where to place the warning. We might also automatically turn it on when Python 2.7 is run with -3. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From masklinn at masklinn.net Thu Oct 8 19:41:57 2009 From: masklinn at masklinn.net (Masklinn) Date: Thu, 8 Oct 2009 19:41:57 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <5b8d13220910081022h19835524sea287e3fb256b0c6@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> <7DD3FAEA-88FE-48EE-9B92-972F5C8EFB3E@masklinn.net> <5b8d13220910081022h19835524sea287e3fb256b0c6@mail.gmail.com> Message-ID: On 8 Oct 2009, at 19:22 , David Cournapeau wrote: > > This is not true - stow solves the problem in a more general way (in > the sense that it is not restricted to python), at least on platforms > which support softlink. I was, of course, talking about "pure" Python solutions (but I should indeed have qualified my statement thus), in the general cases there are several solutions to handle that including but not limited to stow you already mentioned, BSD jails or full-blown virtual machines. From mal at egenix.com Thu Oct 8 20:02:13 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Oct 2009 20:02:13 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080944q1785aad0nc15937610701190f@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> <94bdd2610910080554m530bfdbeg2bf010c813c71873@mail.gmail.com> <4ACDFD54.3080202@egenix.com> <94bdd2610910080944q1785aad0nc15937610701190f@mail.gmail.com> Message-ID: <4ACE2925.6090900@egenix.com> Tarek Ziad? wrote: > On Thu, Oct 8, 2009 at 4:55 PM, M.-A. Lemburg wrote: >> OTOH, the register command sends the meta-data directly to >> the PyPI database, so it doesn't even need another file >> for storing away the meta data. >> >> Unless I'm missing something important (which I probably am), >> the only added value of the new format over PKG-INFO is the >> micro-language. > > That's one. The other one is to avoid to deal with yet another > configuration file > alongside setup.py, and just deal with a [metadata] section in setup.cfg. > > And PKG-INFO stays a static, platform-specific file that exists only when the > distribution is installed on the target platform. Hmm, PKG-INFO is generated from the parameters you pass to setup() in setup.py, so the package author will not have to deal with another config file and only has to write down the meta-data in setup.py. Being able to add some meta-data via setup.cfg would probably be an extra feature which some may find handy. However, your summary made also made it look like the meta-data would be required in setup.cfg and that the whole file would get uploaded (including all the options and settings for other distutils commands). Was that intended ? >>> No, but if you ask for it I can write it. >> >> Please do. The format will need documentation anyway and the >> motivation should be made clear as well. > > Ok I will Thanks, I'll wait for the PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From a.badger at gmail.com Thu Oct 8 20:35:08 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 8 Oct 2009 11:35:08 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACE19B3.4060106@cryptelium.net> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDB82F.7010006@cryptelium.net> <4ACDCCBD.4040908@egenix.com> <20091008161148.GG28231@clingman.lan> <4ACE19B3.4060106@cryptelium.net> Message-ID: <20091008183508.GA513@clingman.lan> On Thu, Oct 08, 2009 at 06:56:19PM +0200, kiorky wrote: > > > Toshio Kuratomi a ?crit : > > > > Also note, the ability to have multiple versions makes things easier for > > system packagers and provides an easy alternative to a virtualenv for > > end-users. > > > > * System packagers: virtualenv does not provide a method suitable for system > > Yes, there is no doubt virtualenv is useless for system packagers but: > > * System and applications deployment have not to be tied. > It s up to the user to install things system wide or to use locally isolation > technics. Virtualenv is one of those. > As a conclusion, there are not very much problem for system packagers as if > their users have specific needs, they will do something Outside the system. > If not, they use their global python with packages installed in that global one. > This misses the point. If there's two pieces of software to be deployed via system packages and they use two different versions of a module, it's currently not very easy to do this. I do it with setuptools curently even with all its warts. Having a way to do this from within the stdlib that tied directly into the import mechanism would make for a much cleaner situation. In other words, the suggestion that there is no need for a method to install multiple versions of a module because virtualenv is a better solution is bogus. virtualenv is a better solution for creating isolated environments. It is not a solution for all of the cases that being able to install multiple versions of a library would be. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From a.badger at gmail.com Thu Oct 8 20:40:03 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 8 Oct 2009 11:40:03 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some?words?on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> Message-ID: <20091008184003.GB513@clingman.lan> On Thu, Oct 08, 2009 at 04:28:52PM +0000, Antoine Pitrou wrote: > Toshio Kuratomi gmail.com> writes: > > > > This is needing to install multiple versions and use the newly installed > > version for testing. > [...] > > What you're missing is that having separate environments has a virtue of > cleanliness, understandability and robustness that a multiple-versioned solution > doesn't have. While the technical merits are debatable I'm sure some people > definitely prefer to manage a virtualenv-based version. > I'm not missing it. I'm only saying that the precise requirement that is being stated is not sandboxing (that was listed later). It's being able to use a newly installed library for testing. The essential element of that is being able to install a new version of the library and use that instead of the sytem installed version. sandboxing may be how someone wants to do this but it isn't essential to be able to do this. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From a.badger at gmail.com Thu Oct 8 20:52:53 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 8 Oct 2009 11:52:53 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACE1388.5020602@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <20091008161732.GH28231@clingman.lan> <4ACE1388.5020602@voidspace.org.uk> Message-ID: <20091008185253.GC513@clingman.lan> On Thu, Oct 08, 2009 at 05:30:00PM +0100, Michael Foord wrote: > Toshio Kuratomi wrote: >> On Thu, Oct 08, 2009 at 02:52:24PM +0200, Simon Cross wrote: >> >>> On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad? wrote: >>> >>>> = Virtualenv and the multiple version support in Distribute = >>>> >>> ... >>> >>>> My opinion is that this tool exists only because Python doesn't >>>> support the installation of multiple versions for the same >>>> distributions. >>>> >> Let's actually look at these reasons: >> >> >>> This is not at all how I use virtualenv. For me virtualenv is a >>> sandbox so that I don't have to become root whenever I need to install >>> a Python package for testing purposes >>> >> >> This is needing to install multiple versions and use the newly installed >> version for testing. >> >> > > Not really - it is wanting to install a single version of a library that > you don't want to install into your 'main' (whether that be user or > system) Python install. It is sandboxing and orthogonal to multiple > versions. > What I'm replying to is specifically the need to: "become root whenever I need to install a Python package for testing purposes" That doesn't require sandboxing at all. Can you use sandboxing to do this? Yes. But that is separate to being able to install a non-system version of a library and be able to test it. My quoting of that phrase could have been better -- I missed the reference to sandboxing since it is in a separate clause of the sentence from what I was responding to. >>> >>> and to allow me to hop between >>> sets of installed Python packages while developing on multiple Python >>> projects. >>> >> >> This is the ability to install multiple versions and specify different >> versions for different projects you're working on. >> > > No, it is working on multiple projects that have *different* > dependencies and being able to work in an environment that *only* has > direct dependencies installed - again sandboxed from your main Python > environment. > No. Here what is written is: "allow me to hop between sets of installed Python packages while developing on multiple Python projects." There's nothing about *only* having direct dependencies installed. That's a separate requirement than what was written. > The fact that virtualenv allows you to have multiple different versions > of the same library installed in the different environments you create > is completely separate from the motivation that causes many people to > use it. > Precisely! We see 100% eye-to-eye here. My reply is just trying to say that the ideas of * testing a locally installed, conflicting version of a library * running multiple projects with different, conflicting version requirements are completely satisfiable without sandboxing. Virtualenv is a sandboxing tool. It can be used to perform these tasks. But it isn't necessary. Having sandboxing is an additional feature on top of the base requirements to perform the task. > What virtualenv *doesn't* do (I believe) is allow you to have multiple > versions of libraries installed within a single environment and switch > between them (at least it doesn't offer anything beyond what setuptools > or pip provides). > Yep. Which makes virtualenv unsuitable for certain other problem spaces where sandboxing is inappropriate. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From martin at v.loewis.de Thu Oct 8 21:00:34 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Oct 2009 21:00:34 +0200 Subject: [Python-Dev] Logging, Unicode and sockets In-Reply-To: References: Message-ID: <4ACE36D2.3060308@v.loewis.de> > I've noticed that the socket-based logging handlers - SocketHandler, > DatagramHandler and SysLogHandler - aren't Unicode-aware and can break in the > presence of Unicode messages. I can't understand what the problem with SocketHandler/DatagramHandler is. As they use pickle, they should surely be able to send records with Unicode strings in them, no? OTOH, why is SMTPHandler not in your list? > I'd like to fix this by giving these handlers an > optional (encoding=None) parameter in their __init__, and then using this to > encode on output. For syslog, I don't think that's appropriate. I presume this is meant to follow RFC 5424? If so, it SHOULD send the data in UTF-8, in which case it MUST include a BOM also. A.8 then says that if you are not certain that it is UTF-8 (which you wouldn't be if the application passes a byte string), you MAY omit the BOM. Regards, Martin From sridharr at activestate.com Thu Oct 8 21:35:22 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 08 Oct 2009 12:35:22 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACDEEB9.2000308@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACDEEB9.2000308@voidspace.org.uk> Message-ID: On Thu, 08 Oct 2009 06:52:57 -0700, Michael Foord wrote: > I don't recall ever having wanted / needed to install multiple versions > of the same library - whilst I can appreciate that it *can* be a real > issue it has never been a problem for me. Multiple versions is going to be a mess. It is a pathetic workaround for packages that do not care about backward compatibility (eg: CherryPy-2.x vs CherryPy-3.x). Drop support for multiple versions to force package authors to deal with it. I applaud the Jinja team for doing this: ARMIN: The final version of the Jinja2 Django-inspired template engine was just released. It comes with tons of improvements over the older Jinja1 engine and breaks backwards compatibility to some point over the old Jinja1 engine. It's packaged as a separate package so that you can use both right next to each other for an easier transition. http://lucumr.pocoo.org/2008/7/17/jinja2-final-aka-jinjavitus-released This is something the Python community can also learn from the Perl world. -srid From p.f.moore at gmail.com Thu Oct 8 14:56:05 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Oct 2009 13:56:05 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080211q5ff4fd09wf62c44dd4a680a8f@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <79990c6b0910080156s2e540184r255624d2166cbae2@mail.gmail.com> <94bdd2610910080211q5ff4fd09wf62c44dd4a680a8f@mail.gmail.com> Message-ID: <79990c6b0910080556m1f2d4156v791193da56db40fb@mail.gmail.com> 2009/10/8 Tarek Ziad? : >> The egg format >> >> One thing missing from your roadmap (unless I missed it) is the fate >> of the egg (zipfile) format. If it's to remain a valid option >> (bdist_egg and the like) then I'd assume that Distribute needs to be >> the place to develop it. >> One thing it will need is an equivalent to >> PEP 376 introspection and uninstall features. And *that* is where I >> believe a standard protocol belongs - developed based on the real use >> requirements for egg files, with a suitable file-based layer which can >> be added into distutils (somewhat like importlib does for PEP 302) >> once the protocol is agreed and standardised. > > Distribute is planning to continue supporting the egg format, but will > probably drop the zipped version. > This is still under discussions though so don't take this as the final word. > > In any case, distribute.resources will probably add the missing bits > on the top of a PEP 376 -compliant Distutils > to provide support for any file querying Distutils will not provide > (for installation and uninstallation) [Note: For the purposes of below I use "egg format" to only refer to zipped eggs. I see no value AT ALL in having an unzipped "egg format" which is almost the same as, but subtly different from, the distutils-supported standard filesystem format. If the distutils standard doesn't provide for all the requirements, fix that, don't add another variation!] I'm not sure I follow this. I think it's important to provide for non-filesystem based packages (that's the whole point of things like PEP 302 and importlib!). The fact that core distutils doesn't support installing, querying or removing such packages is fine. They can be supplied by 3rd party systems like Distribute. However, it is *not* fine (in my view) for each independent format to have to implement its own set of interfaces. PEP 302 is precisely the solution for this in terms of import - every importer has a standardised set of protocols to follow, and if it does so, then it will (should :-)) work seamlessly with any Python code. PEP 302 is not perfect, in part because it aims to be minimal rather than comprehensive, hence the fact that not all packages are "zip safe". But that's a failing in PEP 302 rather than in the concept. Multiple package formats should be supported in the same way - with a set of well-defined (via a PEP) interfaces that all package formats must provide, which enable core and user code to be written in a format-independent way. Once such a PEP is written, distutils' PEP 376 support should be rewritten to use those interfaces, and to implement the necessary filesystem-format support layer. (In other words, whereas I support reducing PEP 376's scope to filesystem only for now, I still strongly advocate that the longer-term solution should be based on a user extensible format-independent approach). In this context, eggs are "merely" the first (and most important) example of a format extension, and so should drive the development of a standard. To summarise: I believe that we need a statement of direction on the (zipped) egg format. I see a number of options: 1. Retain it as-is in Distribute, but deprecated and no further development. 2. Deprecate egg support in Distribute, and ultimately drop it. 3. Develop into a reference example of an extensible architecture for adding to PEP 376 support. 4. Extend PEP 376 support to eggs by means of egg-specific approaches hooking into distutils. For the record, I dislike (1), I see (2) as a lost opportunity, I see (3) as the right way to go, but appreciate that it's the most work, and I strongly oppose (4) (not least because it effectively makes a latrge part of PEP 302 and importlib wasted effort). It looks like you're proposing (2), but aren't entirely clear because you're still allowing for non-zipped "egg" formats (which I see no value in, as I noted above). I'm willing to help work on option (3). Once it's decoupled from questions around OS package managers (which are more the province of PEP 376) then I'd hope that my lack of knowledge about package managers is less of a problem. Paul. From vinay_sajip at yahoo.co.uk Thu Oct 8 21:55:15 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 8 Oct 2009 19:55:15 +0000 (UTC) Subject: [Python-Dev] Logging, Unicode and sockets References: <4ACE36D2.3060308@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > I can't understand what the problem with SocketHandler/DatagramHandler > is. As they use pickle, they should surely be able to send records with > Unicode strings in them, no? Of course you are right. When I posted that it was a knee-jerk reaction to the issue that was raised for SysLogHandler configured to use UDP. I did realise a bit later that the issue didn't apply to the other two handlers but I was hoping nobody would notice ;-) > OTOH, why is SMTPHandler not in your list? I assumed smtp.sendmail() would deal with it, as it deals with the wire protocol, but perhaps I was wrong to do so. I noticed that Issue 521270 (SMTP does not handle Unicode) was closed, but I didn't look at it closely. I now see it was perhaps only a partial solution. I did a bit of searching and found this post by Marius Gedminas: http://mg.pov.lt/blog/unicode-emails-in-python.html Now if that's the right approach, shouldn't it be catered for in a more general part of the stdlib than logging - perhaps in smtplib itself? Or, seeing that Marius' post is five years old, is there a better way of doing it using the stdlib as it is now? > For syslog, I don't think that's appropriate. I presume this is meant to > follow RFC 5424? If so, it SHOULD send the data in UTF-8, in which case > it MUST include a BOM also. A.8 then says that if you are not certain > that it is UTF-8 (which you wouldn't be if the application passes a byte > string), you MAY omit the BOM. So ISTM that the right thing to do on 2.x would be: if str to be sent, send as is; if unicode to be sent, encode using utf-8 and send with a BOM. For 3.x, just encode using utf-8 and send with a BOM. Does that seem right? Thanks and regards, Vinay Sajip From nad at acm.org Thu Oct 8 23:28:47 2009 From: nad at acm.org (Ned Deily) Date: Thu, 08 Oct 2009 14:28:47 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: In article <94bdd2610910080131j323b98d9i871bce43465f237a at mail.gmail.com>, Tarek Ziad? wrote: > Here's a quick summary of the main things that are going to happen in > Distutils, and Distribute, and a few words on virtualenv and pip. > (there is much much more work going on, but I don't want to drown > people with details) Thanks for the summary! Unfortunately, as the saying goes, the devil is in the details. Here are a few comments. > = Distribute = > > I won't explain here again why we have forked, I think it's obvious to > anyone here now. I'll rather explain what > we are planning in Distribute and how it will interact with Distutils. > > Distribute has two branches: > > - 0.6.x : provides a Setuptools-0.6c9 compatible version > - 0.7.x : will provide a refactoring > > == 0.6.x == > > Not "much" is going to happen here, we want this branch to be helpful > to the community *today* by addressing the 40-or-so bugs > that were found in Setuptools and never fixed. This is eventually > happen soon because its development is > fast : there are up to 5 commiters that are working on it very often > (and the number grows weekly.) > > The biggest issue with this branch is that it is providing the same > packages and modules setuptools does, and this > requires some bootstrapping work where we make sure once Distribute is > installed, all Distribution that requires Setuptools > will continue to work. This is done by faking the metadata of > Setuptools 0.6c9. That's the only way we found to do this. > > There's one major thing though: thanks to the work of Lennart, Alex, > Martin, this branch supports Python 3, > which is great to have to speed up Py3 adoption. > > The goal of the 0.6.x is to remove as much bugs as we can, and try if > possible to remove the patches done > on Distutils. We will support 0.6.x maintenance for years and we will > promote its usage everywhere instead of > Setuptools. This is a key and, I think, difficult issue that's going to need a lot of thought and work to be successful. setuptools / easy_install filled several unmet needs of both package developers and end users and became a de-facto standard because there was essentially no competition. So, in a sense, there was never a migration to setuptools from something. But now, especially given GvR's blessing of Distribute, there needs to be a migration from setuptools to Distribute, although neither are part of the standard library. There are no references to setuptools or easy_install in any of the standard Python documentation. So how do end users know about easy_install? Their first encounter is/was probably through some recipe on a web site, most likely the installation instructions for some package they want to use. And I suspect pretty much all of those installation instructions direct the user to one of two places, either the PyPi page for setuptools or the peakcommunity wiki page: http://pypi.python.org/pypi/setuptools http://peak.telecommunity.com/DevCenter/EasyInstall Right now those are essentially the ultimate contact points for users. A general Google seach for easy_install gets 357,000 hits; one restricted to pypi.python.org gets 2,500. I don't know how many of the latter are active packages on PyPi but certainly there must be hundreds that currently use setuptools and document the use of easy_install for their users. Those packages' documentation often directs users to download the ez_install script to bootstrap setuptools; I believe there are some packages that download it automatically, if necessary. So, without some other intervention, users will continue to follow the instructions with those hundreds of packages (or on those thousands of websites) and manually or automatically try to install setuptools. Which may or may not work - as witnessed with the release of 2.6.3. Brett and others suggested including some sort of warnings in the python release documentation. Is there consensus that should be done? Starting with 2.6.4? Seems like that would be up to Barry and Guido to pronounce. Ultimately, to successfully migrate to Distribute is going to require a good understanding of the use cases for easy_install (primarily those of end users) and for the setuptools API (those of package developers). Those use cases should be documented somehow, either in the roadmap or a reference elsewhere. This is especially true if your intentions are to deprecate the easy_install command in Distribute 0.7 and if APIs are going to change for package developers. As Antoine points out, pip is not equivalent to easy_install. BTW, keep in mind that today people also use easy_install to manage the installation of packages that do not use setuptools. It will also be important to understand and address the major distribution channels for Python installations, many of which already include setuptools in their distributions. There are the multi-platform distributors, like python.org (which does not currently distribute setuptools), ActiveState, and Enthought. You've already mentioned some of the Unix-y O/S ones: Debian, Fedora, et al. On the OS X side, there's Apple and the major third-party providers of open-source packages, MacPorts and Fink - they all distribute setuptools. I don't have personal experience with the Windows world to know what all the channels are there. Are there other operating systems who package Python and possibly setuptools? They, one way or another, need to know and have some sort of plan going forward for their distributions regarding setuptools and Distribute. And, of course, the stickiest issues in all this are not technical, but rather social. For most of this, the most important part will likely be giving a clear direction to all of the major groups involved: end users, package developers, distributors. How to decide on and then effectively communicate that direction is not at all trivial, I think. Thanks again for your hard work on this, Tarek! -- Ned Deily, nad at acm.org From solipsis at pitrou.net Thu Oct 8 23:41:21 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2009 21:41:21 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Distutils_and_Distribute_roadmap_=28and_so?= =?utf-8?q?me_words=09on_Virtualenv=2C_Pip=29?= References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: Ned Deily acm.org> writes: > > How to decide on and then effectively > communicate that direction is not at all trivial, I think. I think it's quite trivial actually. Since everybody agrees (except perhaps PJE) that Distribute should replace setuptools, the word will spread and trickle quite quickly in the various layers of the community. If you look at the XFree86 -> XOrg transition, I don't think there have been a lot of problems to bring people to understand the change. Regards Antoine. From nad at acm.org Fri Oct 9 00:01:22 2009 From: nad at acm.org (Ned Deily) Date: Thu, 08 Oct 2009 15:01:22 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: In article , Antoine Pitrou wrote: > Ned Deily acm.org> writes: > > How to decide on and then effectively > > communicate that direction is not at all trivial, I think. > > I think it's quite trivial actually. Since everybody agrees (except perhaps > PJE) > that Distribute should replace setuptools, the word will spread and trickle > quite quickly in the various layers of the community. > > If you look at the XFree86 -> XOrg transition, I don't think there have been > a > lot of problems to bring people to understand the change. I'm not sure that's a such a good example: some of the impacts of that transition are still affecting users. The unspoken assumption here is that it is a good idea to make this transition as painless as possible within some unspecified limits of practicality. Sure, at one end of the spectrum, you can just throw things out in the world and hope for the best. I think the Python community has a well-deserved reputation for doing a lot better than that. -- Ned Deily, nad at acm.org From ziade.tarek at gmail.com Fri Oct 9 00:55:56 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 00:55:56 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> On Thu, Oct 8, 2009 at 11:28 PM, Ned Deily wrote: [..] > So, without some other intervention, users will continue to follow the > instructions with those hundreds of packages (or on those thousands of > websites) and manually or automatically try to install setuptools. I think this will be in the hands of the projects developers : if they start to use Distribute, they will start to document the way to install and use it, and the word will spread eventually. [..] > Ultimately, to successfully migrate to Distribute is going to require a > good understanding of the use cases for easy_install (primarily those of > end users) and for the setuptools API (those of package developers). > Those use cases should be documented somehow, either in the roadmap or a > reference elsewhere. This is especially true if your intentions are to > deprecate the easy_install command in Distribute 0.7 and if APIs are > going to change for package developers. As Antoine points out, pip is > not equivalent to easy_install. BTW, keep in mind that today people > also use easy_install to manage the installation of packages that do not > use setuptools. The choice to deprecate easy_install in 0.7 is done because the Pip project is not far to meet all uses cases easy_install users have, and we're betting on the fact that Pip is active and will be much more advanced that what we could do with a 'new' refactored easy_install by the time 0.7 is ready. meaning : why doing this work twice ? > Thanks again for your hard work on this, Tarek! Thanks for the advices Tarek From kiorky at cryptelium.net Fri Oct 9 01:11:09 2009 From: kiorky at cryptelium.net (kiorky) Date: Fri, 09 Oct 2009 01:11:09 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> Message-ID: <4ACE718D.8090400@cryptelium.net> Tarek Ziad? a ?crit : > The choice to deprecate easy_install in 0.7 is done because the Pip project > is not far to meet all uses cases easy_install users have, and we're betting > on the fact that Pip is active and will be much more advanced that what > we could do with a 'new' refactored easy_install by the time 0.7 is ready. But how about retro-compatibility? Like with all those buildout recipes which relies on setuptool APIs (Requirement, PackageIndex and so on), it's a risk to break same at some point. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/kiorky%40cryptelium.net -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From v+python at g.nevcal.com Fri Oct 9 01:13:13 2009 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 08 Oct 2009 16:13:13 -0700 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACDF621.4070603@trueblade.com> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4ACDD2C3.4090608@gmail.com> <4ACDF621.4070603@trueblade.com> Message-ID: <4ACE7209.9060208@g.nevcal.com> On approximately 10/8/2009 7:24 AM, came the following characters from the keyboard of Eric Smith: > Vinay Sajip wrote: >> BTW I sent Eric a private mail re. the "0o" versus "0" issue, to see >> if it was >> worth raising an enhancement request on the bug tracker using "O" to >> generate >> compatible rendering for octals. > > I didn't get your message, could you resend?. > > I was thinking the same thing, but it seems like a transition step. I'd > rather not keep such backward compatibility hacks (if you will) around > for the long haul. How about a flag (maybe '*') at the start of the > format specification which says "operate in backward compatibility > mode"? We could document it as being only useful for the % to {} > translator, and promise to remove it at some point in the future. Either > actually deprecate it or just promise to deprecate it in the future. Seems like the ability for Python {} formatting to be able to match not only old Python % formatting output, but also output created by C's sprintf, and other numeric formatting systems, make this particular feature useful in more scenarios than a "backward compatibility hack". If you want to replace a C program that produces parsed output in a given format, and that given format includes leading-0-octal numbers, then it would be good to have the capability in Python .format, even though Python itself uses 0o prefix. Similar arguments may apply anywhere else that sprintf produces something that .format cannot currently produce. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From ziade.tarek at gmail.com Fri Oct 9 01:21:20 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 01:21:20 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACE718D.8090400@cryptelium.net> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> Message-ID: <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> On Fri, Oct 9, 2009 at 1:11 AM, kiorky wrote: > > > Tarek Ziad? a ?crit : > >> The choice to deprecate easy_install in 0.7 is done because the Pip project >> is not far to meet all uses cases easy_install users have, and we're betting >> on the fact that Pip is active and will be much more advanced that what >> we could do with a 'new' refactored easy_install by the time 0.7 is ready. > > But how about retro-compatibility? > Like with all those buildout recipes which relies on setuptool APIs > (Requirement, PackageIndex and so on), it's a risk to break same at some point. Keep in mind that Distribute 0.6.x will still be maintained and will still provide easy_install and al, and that you will be able to rely on it for your projects. Distribute 0.7.x is to be considered as a new project in some ways : all namespaces will be different and the code will not interfer with a 0.6 installation. So a buildout recipe will have the choice to use the old packages you are mentioning and at some point evolve and use the new tools 0.7 will provide. Tarek From david.lyon at preisshare.net Fri Oct 9 01:50:13 2009 From: david.lyon at preisshare.net (David Lyon) Date: Thu, 08 Oct 2009 19:50:13 -0400 Subject: [Python-Dev] Distutils and Distribute roadmap In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <599f3554ce7cf10c18de71c7c5702c33@preisshare.net> On Thu, 8 Oct 2009 21:41:21 +0000 (UTC), Antoine Pitrou wrote: > I think it's quite trivial actually. Since everybody agrees (except perhaps > PJE) > that Distribute should replace setuptools, the word will spread and trickle > quite quickly in the various layers of the community. I don't think that it's that black and white. PJE put a lot of work (and creativity) into setuptools. There's some degree of tragedy if his project gets forked off simply because of failures in the testing and quality assurance department. Personally, I find the setuptools code somewhat hard to follow. The style goes back to early programming by 'masters' where the only person that could follow it was the master himself. Sadly, that is not what we need in open source. The newer trend is in highly readily readable and easy to understand code. Tarek is excellent at producing high quality code of the type that is required. But his programming world (on the mac) isn't full of the challenges that we poor other unfortunates face on other platforms face. Coming back to the point, setuptools is a pretty poor package to have to fork. It's very difficult to follow and the structure and implementation is cryptic. Perphaps it's a thankless job that Tarek has taken on in that regard - but he seems to be handling the distribute fork quite well. Being new to python, it's taken me a little while to figure out the process. But I've decided to write a static metadata pep which is underway now as an alternate proposal. Being primarily a hacker, writing nice design documents doesn't come easy. If there is any distutils roadmap, I'd like to see improvements that work on Windows. Yes.. python does work on windows and we can't all get our bosses to change to os-x just yet. There's many things in distutils that I could best describe as being unfinished or partially done. Let me make a few rambling direction suggestions... - command line package installation for distutils We need a simple script, not based on setuptools to allow installation, listing and deinstallation of packages. Currently *nothing* exists in distutils to do this. It's a **major** failing. - Improvements to operation on platform Windows Package/Application installation on windows sucks... For example, there's no documented way to get your programs installed to "\Program Files\myApplication" For non windows users, that's where programs are supposed to go. Any other place is "evil" - like where they are going now. If it is only a documentation issue, well can we fix that. All I'm asking for is a matching "elif sys.platform = 'win32'" for every "if sys.platform = 'darwin'".... or less. It's not unreasonable... It's probably a sensible idea for Tarek to keep going with Distribute. My user experience with setuptools was that it wasted a lot of my time unnecessarily. I'm planning to put Distribute support in the Python Package Manager project (pythonpkgmgr) for it's next release. David From david.lyon at preisshare.net Fri Oct 9 02:50:16 2009 From: david.lyon at preisshare.net (David Lyon) Date: Thu, 08 Oct 2009 20:50:16 -0400 Subject: [Python-Dev] =?utf-8?q?eggs_now_mandatory_for_pypi=3F?= In-Reply-To: <4ACD965D.5080701@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> Message-ID: <30910449ef0aad44ac8d419c4e36896f@preisshare.net> On Thu, 08 Oct 2009 09:35:57 +0200, "M.-A. Lemburg" wrote: >> One could say that much of the setuptools cloud came about because of >> the lack of the queryable download url. Setuptools does a lot of work >> here to 'work-around' the ommission on pypi of a package download url. > > I think you two have missed my email to PJE. > > In summary: > > PyPI already does have mechanism for querying download URLs. > > http://wiki.python.org/moin/PyPiXmlRpc > > It only lacks some more meta-data that would be required to > match up the user's configuration with the one used in the > uploaded URLs. > > And it would be handy to have a mechanism to just upload the > URL data rather than the whole package. Looking at distutils' > upload command, PyPI currently appears to only support uploads > of the package file itself. > > Basically, most of the mechnisms are already there, they just need > some extra care. Maybe you're right. I'll look into it. > BTW: Who would I need to contact for the PyPI side to work out > an upload_url distutils command ? pypi is on sourceforge... ask to become a developer I guess.. David From ianb at colorstudy.com Fri Oct 9 03:22:13 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 8 Oct 2009 20:22:13 -0500 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> Message-ID: I'm coming in late and breaking threading, but wanted to reply to Tarek's original email: > - easy_install is going to be deprecated ! use Pip ! Cool! I wouldn't have written pip if I didn't think it would improve substantially on easy_install. Incidentally (because I know people get really enthused about this) Carl Meyer just contributed a feature to pip to do atomic uninstallation. Someone mentioned that easy_install provided some things pip didn't; outside of multi-versioned installs (which I'm not very enthusiastic about) I'm not sure what this is? > - distribute.resources: that's the old pkg_resources, but reorganized in clean, pep-8 modules. This package will > only contain the query APIs and will focus on being PEP 376 compatible. We will promote its usage and see if Pip wants > to use it as a basis. And maybe PyPM once it's open source ? ( ). > It will probably shrink a lot though, once the stdlib provides PEP 376 support. This seems easy enough to use in pip. > - distribute.index: that's package_index and a few other things. everything required to interact with PyPI. We will promote > its usage and see if Pip wants to use it as a basis. This is a little tricky. Primarily because there's a fair amount of logic involved in the indexing (going around to different URLs, parsing links, finding stuff). So long as there is logic, something can go wrong -- often not in the package itself, but simple user error (e.g., it doesn't look where the user thinks it should, or a link is malformed, etc). Because of this, and as a general design goal of pip, I want to show as much as I can about what it is doing and why. This is primarily tied into pip's logging system (which is oriented towards command-line output, and isn't the standard logging system). Also it tracks *why* it got to a certain links. These are the two things I can think of where the index code in pip is tied to pip, and why it would be hard to use an external system. > = Virtualenv and the multiple version support in Distribute = > > (I am not saying "We" here because this part was not discussed yet > with everyone) > > Virtualenv allows you to create an isolated environment to install some distribution without polluting the > main site-packages, a bit like a user site-packages. > > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. > But if PEP 376 and PEP 386 support are added in Python, we're not far > from being able to provide multiple version support with > the help of importlib. Before making workingenv (virtualenv's predecessor) I actively tried to use Setuptools' multi-version support. I found it very unsuccessful. I don't think it was due to any problems with Setuptools -- maybe a slight problem was the conflict between "active" eggs and "multiversion" eggs (where active eggs would be more likely to cause conflicts, while multiversion eggs aren't available until you specifically require them). But that was just awkward, I don't think it was the real problem. The real problem is that a set of packages that makes up a working application is something separate from any library. And you can only know that an application works with an exact set of libraries. Every update can break a working application (and with fairly high probability). You can't know what updates are safe. And it's really a bit tricky to even be sure you know what libraries a package really requires -- lots of libraries might be incidentally available but no longer formally required. (Someone mentioned a coworker that only installed packages with easy_install -m, because he said it kept him honest -- only packages that are explicitly required would be available. But most people don't do this, and it really only solves the one problem of undeclared dependencies) The way both virtualenv and buildout handle this is that libraries will have a single, static version until you explicitly do something to update that version. Both are somewhat focused on a functional unit -- like one virtualenv environment for one task, or one buildout config for one application. Buildout allows for a globally shared set of versioned eggs, but that's really just a little optimization (for disk space or installation speed) -- each egg is brought in only explicitly, at build time, and not as an option during the program's runtime. This is verifiable, stable, and to varying degrees concrete (virtualenv being more concrete than buildout, which tends more towards the declarative). What virtualenv does could certainly be in the Python interpreter (and much more compact as a result, I am sure). PYTHONHOME does it to a degree, though binding a script to a environment through the interpreter listed in #! is more stable than the implicit environment of PYTHONHOME. workingenv used an environmental variable (PYTHONPATH, before PYTHONHOME existed) and it caused problems. Also virtualenv offers more system isolation. If I had my way, buildout would use virtualenv and throw away its funny script generation. If virtualenv had existed before buildout began development, probably things would have gone this way. I think it would make the environment more pleasant for buildout users. Also I wish it used pip instead of its own installation procedure (based on easy_install). I don't think the philosophical differences are that great, and that it's more a matter of history -- because the code is written, there's not much incentive for buildout to remove that code and rely on other libraries (virtualenv and pip). -- Ian Bicking ?| ?http://blog.ianbicking.org ?| ?http://topplabs.org/civichacker From mal at egenix.com Fri Oct 9 10:20:54 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 Oct 2009 10:20:54 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <30910449ef0aad44ac8d419c4e36896f@preisshare.net> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <30910449ef0aad44ac8d419c4e36896f@preisshare.net> Message-ID: <4ACEF266.3020309@egenix.com> David Lyon wrote: > On Thu, 08 Oct 2009 09:35:57 +0200, "M.-A. Lemburg" wrote: >>> One could say that much of the setuptools cloud came about because of >>> the lack of the queryable download url. Setuptools does a lot of work >>> here to 'work-around' the ommission on pypi of a package download url. >> >> I think you two have missed my email to PJE. >> >> In summary: >> >> PyPI already does have mechanism for querying download URLs. >> >> http://wiki.python.org/moin/PyPiXmlRpc >> >> It only lacks some more meta-data that would be required to >> match up the user's configuration with the one used in the >> uploaded URLs. >> >> And it would be handy to have a mechanism to just upload the >> URL data rather than the whole package. Looking at distutils' >> upload command, PyPI currently appears to only support uploads >> of the package file itself. >> >> Basically, most of the mechnisms are already there, they just need >> some extra care. > > Maybe you're right. I'll look into it. > >> BTW: Who would I need to contact for the PyPI side to work out >> an upload_url distutils command ? > > pypi is on sourceforge... ask to become a developer I guess.. I know that the issue tracker for PyPI is (still) on SF, but development appear to happen elsewhere and I can't find any contact information on the PyPI web pages. Is Richard Jones still working on it ? They I could contact him directly to hash this out. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 09 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From kiorky at cryptelium.net Fri Oct 9 10:54:23 2009 From: kiorky at cryptelium.net (kiorky) Date: Fri, 09 Oct 2009 10:54:23 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> Message-ID: <4ACEFA3F.2020706@cryptelium.net> Ian Bicking a ?crit : > I'm coming in late and breaking threading, but wanted to reply to > Tarek's original email: > > This is verifiable, stable, and to varying degrees concrete > (virtualenv being more concrete than buildout, which tends more > towards the declarative). Is that a friday troll ? declarative over control ? > > What virtualenv does could certainly be in the Python interpreter (and > much more compact as a result, I am sure). PYTHONHOME does it to a > degree, though binding a script to a environment through the > interpreter listed in #! is more stable than the implicit environment > of PYTHONHOME. workingenv used an environmental variable (PYTHONPATH, > before PYTHONHOME existed) and it caused problems. Also virtualenv > offers more system isolation. > > If I had my way, buildout would use virtualenv and throw away its > funny script generation. If virtualenv had existed before buildout Which one, the one provided to generate scripts from entry points with the *.egg recipes or the bin/buildout auto regeneration? > began development, probably things would have gone this way. I think > it would make the environment more pleasant for buildout users. Also * I don't think so, buildout is the only tool atm that permit to have really reproducible and isolated environments. Even, if you use the pip freezing machinery, it is not equivalent to buildout, Control! * Buildout can have single part to construct required eggs, at a specific version and let you control that. Pip will just search for this version, see that it's not available and fail. You have even recipes (like minitage.recipe.egg that permit to construct eggs with special version when you apply patches onto, thus, you can have the same egg in different flavors in the same eggs cache available for different projects. Those projects will just have to pin the right version to use, Control!. * Another thing is the "funny script generation", you have not one global site-packages for your project, but one global cache. But from this global cache, your scripts will only have available the eggs you declared, see Control! * Moreover buildout is not only a python packages manager, it's some of its recipes that permit to use it as. Buildout is just a great deployment tool that allow to script and manage your project in a "funny" and "flexible" way, Control! > I wish it used pip instead of its own installation procedure (based on > easy_install). I don't think the philosophical differences are that > great, and that it's more a matter of history -- because the code is And retro-compatibility. > written, there's not much incentive for buildout to remove that code > and rely on other libraries (virtualenv and pip). * Virtualenv is not really needed as buildout uses "eggs caches" and do the isolation from there. Scripts generate the adequate sys.path relying on the application requirements (which, once more, you are in total control). I prefer throught to use a virtualenv for each project (or a bunch of projects), and to bootstrap my buildouts from there. For me, they are complementary to achieve your deployment and isolation needs. > > -- > Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/kiorky%40cryptelium.net -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From ziade.tarek at gmail.com Fri Oct 9 12:03:34 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 12:03:34 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> Message-ID: <94bdd2610910090303v4b3b9b46q3f77266e34c33197@mail.gmail.com> On Fri, Oct 9, 2009 at 3:22 AM, Ian Bicking wrote: > I'm coming in late and breaking threading, but wanted to reply to > Tarek's original email: > >> - easy_install is going to be deprecated ! use Pip ! > > Cool! ?I wouldn't have written pip if I didn't think it would improve > substantially on easy_install. > > Incidentally (because I know people get really enthused about this) > Carl Meyer just contributed a feature to pip to do atomic > uninstallation. Yes I saw that, it's great. And he is now involved in PEP 376 work so Pip could eventually become the first PEP 376 compliant installer/uninstaller. > > Someone mentioned that easy_install provided some things pip didn't; > outside of multi-versioned installs (which I'm not very enthusiastic > about) I'm not sure what this is? Basically what you've listed on Pip front page I think, like 'not tested under windows' But I don't see any blocking point besides some testing, to move from easy_install to pip, and the deprecation of multi-versioned feature seem to go in the direction of the community. >> ? ?- distribute.index: that's package_index and a few other things. everything required to interact with PyPI. We will promote >> ? ? ?its usage and see if Pip wants to use it as a basis. > > This is a little tricky. ?Primarily because there's a fair amount of > logic involved in the indexing (going around to different URLs, > parsing links, finding stuff). ?So long as there is logic, something > can go wrong -- often not in the package itself, but simple user error > (e.g., it doesn't look where the user thinks it should, or a link is > malformed, etc). ?Because of this, and as a general design goal of > pip, I want to show as much as I can about what it is doing and why. > This is primarily tied into pip's logging system (which is oriented > towards command-line output, and isn't the standard logging system). > Also it tracks *why* it got to a certain links. ?These are the two > things I can think of where the index code in pip is tied to pip, and > why it would be hard to use an external system. OK. Maybe this particular package could be used by another tool that needs to work with PyPI. It will also include a set of APIs that corresponds to PyPI XMLPRC services. Regards Tarek -- Tarek Ziad? | http://ziade.org |????????????! |???????????? From ziade.tarek at gmail.com Fri Oct 9 12:06:44 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 12:06:44 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACEF266.3020309@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <30910449ef0aad44ac8d419c4e36896f@preisshare.net> <4ACEF266.3020309@egenix.com> Message-ID: <94bdd2610910090306o63daf0e9w970687a508bc5d55@mail.gmail.com> On Fri, Oct 9, 2009 at 10:20 AM, M.-A. Lemburg wrote: > > I know that the issue tracker for PyPI is (still) on SF, but > development appear to happen elsewhere and I can't find any > contact information on the PyPI web pages. Everything is provided here: http://wiki.python.org/moin/CheeseShopDev > > Is Richard Jones still working on it ? They I could contact him > directly to hash this out. I think Martin vL is the most active developer at this time. Regards Tarek From mal at egenix.com Fri Oct 9 12:12:44 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 Oct 2009 12:12:44 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <94bdd2610910090306o63daf0e9w970687a508bc5d55@mail.gmail.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <30910449ef0aad44ac8d419c4e36896f@preisshare.net> <4ACEF266.3020309@egenix.com> <94bdd2610910090306o63daf0e9w970687a508bc5d55@mail.gmail.com> Message-ID: <4ACF0C9C.8030702@egenix.com> Tarek Ziad? wrote: > On Fri, Oct 9, 2009 at 10:20 AM, M.-A. Lemburg wrote: >> >> I know that the issue tracker for PyPI is (still) on SF, but >> development appear to happen elsewhere and I can't find any >> contact information on the PyPI web pages. > > Everything is provided here: > > http://wiki.python.org/moin/CheeseShopDev Thanks for the URL. >> >> Is Richard Jones still working on it ? They I could contact him >> directly to hash this out. > > I think Martin vL is the most active developer at this time. Ok, then I'll coordinate with Martin. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 09 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Fri Oct 9 12:43:50 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 Oct 2009 10:43:50 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Distutils_and_Distribute_roadmap_=28and_so?= =?utf-8?q?me_words=09on_Virtualenv=2C_Pip=29?= References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> Message-ID: Ian Bicking colorstudy.com> writes: > > Someone mentioned that easy_install provided some things pip didn't; > outside of multi-versioned installs (which I'm not very enthusiastic > about) I'm not sure what this is? http://pip.openplans.org/#differences-from-easy-install If it's obsolete the website should be updated... From lists at cheimes.de Fri Oct 9 13:16:37 2009 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Oct 2009 13:16:37 +0200 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACDD968.1070205@gmail.com> References: <4ACDCA5A.4090609@cheimes.de> <4ACDD968.1070205@gmail.com> Message-ID: <4ACF1B95.50600@cheimes.de> Nick Coghlan wrote: > Christian Heimes wrote: >> The solution requires a new attribute in the sys module that contains >> the name of the implementation. As an alternative we could use the first >> field of sys.subversion but I prefer an explicit attribute. I'm >> proposing "sys.name" with a value of "CPython", "IronPython", "Jython" >> or "PyPy". > > My Google skills almost failed me, but searching for "sys.vm" found me > what I was after: http://bugs.python.org/issue4242 (a discussion > relating to a similar need in the context of flagging implementation > specific tests). Thanks, I knew about the discussion but I didn't know that somebody has already suggested a sys.vm attribute. > As mentioned in that discussion, as of Python 2.6, you can do the following: >>>> import platform >>>> platform.python_implementation() > 'CPython' > > (Although according to the function docstring, PyPy is currently missing > from the list of known implementations) > > Importing yet-another-module for use in site.py doesn't sound like a > great idea, so it may make sense to migrate that information into the > sys module is this approach is taken. "sys.name" is a little generic > though - something more explicit like "sys.vm" would be better. The platform modules uses additional modules like the re module. On my system "import platform" loads 7 additional module. The platform modul also uses multiple function calls and regular expression to detect the vm. I'm not going to blame the author of the function for the way the feature is implemented. It's the best way and I'd probably done it the same way, too. But it's an unnecessary slow down of every interpreter startup. 'sys.vm' it is. > Implementation specific user directories sound like a good idea indeed. > > An alternative to a lookup table approach might be to be a little more > direct and just retrieve the final part of the user specific directory > name directly from a new function in the sys module. Then different VM > authors can choose any directory name they want without CPython's > site.py needing to know anything else about them. I like your idea. It might be a better idea to turn sys.vm into a struct like sys.float_info. The struct can contains a set of required attributes like an id, a capitalized name, the platform (C, .NET, Java) and the suffix for the user site directory. >>> sys.vm sys.vm(id='cpython', name='CPython', platform='C', usersitesuffix='python26') # on win32: Python2.6 >>> sys.vm sys.vm(id='ironpython', name='IronPython', platform='.NET', usersitesuffix='ironpython26) # on win32: IronPython2.6 Christian From solipsis at pitrou.net Fri Oct 9 13:23:26 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 Oct 2009 11:23:26 +0000 (UTC) Subject: [Python-Dev] PEP 370 and IronPython References: <4ACDCA5A.4090609@cheimes.de> <4ACDD968.1070205@gmail.com> <4ACF1B95.50600@cheimes.de> Message-ID: Christian Heimes cheimes.de> writes: > > >>> sys.vm > sys.vm(id='cpython', name='CPython', platform='C', > usersitesuffix='python26') # on win32: Python2.6 > > >>> sys.vm > sys.vm(id='ironpython', name='IronPython', platform='.NET', > usersitesuffix='ironpython26) # on win32: IronPython2.6 `usersitesuffix` should probably be a separate sys attribute, since it doesn't depend on the VM only, but also on the platform and version. Regards Antoine. From lists at cheimes.de Fri Oct 9 13:27:28 2009 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Oct 2009 13:27:28 +0200 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <94bdd2610910080616w27b43020sa52670269e01b7ff@mail.gmail.com> References: <4ACDCA5A.4090609@cheimes.de> <94bdd2610910080616w27b43020sa52670269e01b7ff@mail.gmail.com> Message-ID: <4ACF1E20.8060809@cheimes.de> Tarek Ziad? wrote: > I have a suggestion though, that completes Nick's answer. > > distutils/command/install.py already contains install schemes and > imports USER_SITE and USER_BASE to builds user-specific install > schemes. I think it wouldn't hurt for clarity to reunite all these > schemes in a single place in the stdlib (so in sys maybe?) Hello Tarek! I'm feeling uncomfortable with sticking so much information into the sys module. Another module with installation schemata wouldn't hurt, though. In my opinion it's usually wise to separate configuration from logic. Christian PS: Thanks for your job with distribute! You are a life safer. From lists at cheimes.de Fri Oct 9 13:32:13 2009 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Oct 2009 13:32:13 +0200 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACDD968.1070205@gmail.com> <4ACF1B95.50600@cheimes.de> Message-ID: <4ACF1F3D.7010707@cheimes.de> Antoine Pitrou wrote: > `usersitesuffix` should probably be a separate sys attribute, since it doesn't > depend on the VM only, but also on the platform and version. I don't really care and I'm happy to follow the herd in this matter. :) Christian From lists at cheimes.de Fri Oct 9 13:34:46 2009 From: lists at cheimes.de (Christian Heimes) Date: Fri, 09 Oct 2009 13:34:46 +0200 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACE0E99.7050102@voidspace.org.uk> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> Message-ID: Michael Foord wrote: > I really like this scheme. The important thing for IronPython is that we > can get it into Python 2.6 (along with other fixes to make distutils > compatible with IronPython - like not attempting to bytecode-compile > when sys.dont_write_bytecode is True). I don't think my proposal will land into 2.6. The changes are too severe for a bug fix release. Christian From mal at egenix.com Fri Oct 9 13:47:53 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 Oct 2009 13:47:53 +0200 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACF1B95.50600@cheimes.de> References: <4ACDCA5A.4090609@cheimes.de> <4ACDD968.1070205@gmail.com> <4ACF1B95.50600@cheimes.de> Message-ID: <4ACF22E9.4020000@egenix.com> Christian Heimes wrote: > Nick Coghlan wrote: >> Importing yet-another-module for use in site.py doesn't sound like a >> great idea, so it may make sense to migrate that information into the >> sys module is this approach is taken. "sys.name" is a little generic >> though - something more explicit like "sys.vm" would be better. > > The platform modules uses additional modules like the re module. On my > system "import platform" loads 7 additional module. The platform modul > also uses multiple function calls and regular expression to detect the > vm. I'm not going to blame the author of the function for the way the > feature is implemented. It's the best way and I'd probably done it the > same way, too. But it's an unnecessary slow down of every interpreter > startup. 'sys.vm' it is. > >> Implementation specific user directories sound like a good idea indeed. >> >> An alternative to a lookup table approach might be to be a little more >> direct and just retrieve the final part of the user specific directory >> name directly from a new function in the sys module. Then different VM >> authors can choose any directory name they want without CPython's >> site.py needing to know anything else about them. > > > I like your idea. It might be a better idea to turn sys.vm into a struct > like sys.float_info. The struct can contains a set of required > attributes like an id, a capitalized name, the platform (C, .NET, Java) > and the suffix for the user site directory. > >>>> sys.vm > sys.vm(id='cpython', name='CPython', platform='C', > usersitesuffix='python26') # on win32: Python2.6 > >>>> sys.vm > sys.vm(id='ironpython', name='IronPython', platform='.NET', > usersitesuffix='ironpython26) # on win32: IronPython2.6 +1 on adding something like this, though, I'd use sys.python_implementation or just sys.implementation since not all Python implementations are virtual machines - many compile to machine code or some intermediate format. -1 on importing platform in site.py. Python's startup time is not great to begin with and this would cause the load time to increase even more. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 09 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fuzzyman at voidspace.org.uk Fri Oct 9 13:53:35 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 09 Oct 2009 12:53:35 +0100 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> Message-ID: <4ACF243F.4080408@voidspace.org.uk> Christian Heimes wrote: > Michael Foord wrote: > >> I really like this scheme. The important thing for IronPython is that we >> can get it into Python 2.6 (along with other fixes to make distutils >> compatible with IronPython - like not attempting to bytecode-compile >> when sys.dont_write_bytecode is True). >> > > I don't think my proposal will land into 2.6. The changes are too severe > for a bug fix release. > Right, certainly not adding umpteen new sys attributes. :-) The problem is that the alternative implementations run well behind Python-trunk, indeed it doesn't really make sense for them to put a lot of effort into implementing a version that is still in development. The result is that they discover incompatibilites after a version has gone into 'bugfix only' mode. Whilst the fix you have described (add information to sys that is used by site.py and distutils) is ideal it can only go into 2.7. I would *still* like to see a fix in 2.6 - even if it is simple logic in site.py using sys.platform (if sys.platform == 'cli'; elif sys.platform == 'java' etc). That way IronPython 2.6 is able to be compatible with Python 2.6. This logic might need duplicating in distutils (I haven't looked at how distutils works out where the user site-packages folder is), but it is a 'maintenance only' fix. All the best, Michael > Christian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From fuzzyman at voidspace.org.uk Fri Oct 9 13:53:57 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 09 Oct 2009 12:53:57 +0100 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACF22E9.4020000@egenix.com> References: <4ACDCA5A.4090609@cheimes.de> <4ACDD968.1070205@gmail.com> <4ACF1B95.50600@cheimes.de> <4ACF22E9.4020000@egenix.com> Message-ID: <4ACF2455.8030507@voidspace.org.uk> M.-A. Lemburg wrote: > Christian Heimes wrote: > >> Nick Coghlan wrote: >> >>> Importing yet-another-module for use in site.py doesn't sound like a >>> great idea, so it may make sense to migrate that information into the >>> sys module is this approach is taken. "sys.name" is a little generic >>> though - something more explicit like "sys.vm" would be better. >>> >> The platform modules uses additional modules like the re module. On my >> system "import platform" loads 7 additional module. The platform modul >> also uses multiple function calls and regular expression to detect the >> vm. I'm not going to blame the author of the function for the way the >> feature is implemented. It's the best way and I'd probably done it the >> same way, too. But it's an unnecessary slow down of every interpreter >> startup. 'sys.vm' it is. >> >> >>> Implementation specific user directories sound like a good idea indeed. >>> >>> An alternative to a lookup table approach might be to be a little more >>> direct and just retrieve the final part of the user specific directory >>> name directly from a new function in the sys module. Then different VM >>> authors can choose any directory name they want without CPython's >>> site.py needing to know anything else about them. >>> >> I like your idea. It might be a better idea to turn sys.vm into a struct >> like sys.float_info. The struct can contains a set of required >> attributes like an id, a capitalized name, the platform (C, .NET, Java) >> and the suffix for the user site directory. >> >> >>>>> sys.vm >>>>> >> sys.vm(id='cpython', name='CPython', platform='C', >> usersitesuffix='python26') # on win32: Python2.6 >> >> >>>>> sys.vm >>>>> >> sys.vm(id='ironpython', name='IronPython', platform='.NET', >> usersitesuffix='ironpython26) # on win32: IronPython2.6 >> > > +1 on adding something like this, though, I'd use > sys.python_implementation or just sys.implementation > since not all Python implementations are virtual machines - > many compile to machine code or some intermediate format. > Agrred. > -1 on importing platform in site.py. Python's startup time is not > great to begin with and this would cause the load time to increase > even more. > > Agrred. (And import time is even worse on IronPython.) Michael -- http://www.ironpythoninaction.com/ From p.f.moore at gmail.com Fri Oct 9 14:32:43 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2009 13:32:43 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> Message-ID: <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> 2009/10/9 Antoine Pitrou : > Ian Bicking colorstudy.com> writes: >> >> Someone mentioned that easy_install provided some things pip didn't; >> outside of multi-versioned installs (which I'm not very enthusiastic >> about) I'm not sure what this is? > > http://pip.openplans.org/#differences-from-easy-install > > If it's obsolete the website should be updated... Specifically, combine "only installs from source" with "might not work on Windows" and the result is pretty certainly unusable for C extensions on Windows. You can pretty much guarantee that the average user on Windows won't have a C compiler[1], and even if they do, they won't be able to carefully line up all the 3rd party C libraries needed to build some extensions. Binary packages are essential on Windows. Paul. [1] Heck, some extensions only build with mingw, others only build with MSVC. You need *two* compilers :-( From fuzzyman at voidspace.org.uk Fri Oct 9 14:38:42 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 09 Oct 2009 13:38:42 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> Message-ID: <4ACF2ED2.8010209@voidspace.org.uk> Paul Moore wrote: > 2009/10/9 Antoine Pitrou : > >> Ian Bicking colorstudy.com> writes: >> >>> Someone mentioned that easy_install provided some things pip didn't; >>> outside of multi-versioned installs (which I'm not very enthusiastic >>> about) I'm not sure what this is? >>> >> http://pip.openplans.org/#differences-from-easy-install >> >> If it's obsolete the website should be updated... >> > > Specifically, combine "only installs from source" with "might not work > on Windows" and the result is pretty certainly unusable for C > extensions on Windows. You can pretty much guarantee that the average > user on Windows won't have a C compiler[1], and even if they do, they > won't be able to carefully line up all the 3rd party C libraries > needed to build some extensions. > > Binary packages are essential on Windows. > Definitely. Most Windows users won't have any compilers let alone 2. Michael > Paul. > > [1] Heck, some extensions only build with mingw, others only build > with MSVC. You need *two* compilers :-( > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From exarkun at twistedmatrix.com Fri Oct 9 14:52:05 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 09 Oct 2009 12:52:05 -0000 Subject: [Python-Dev] [New-bugs-announce] [issue7064] Python 2.6.3 / setuptools 0.6c9: extension module builds fail with KeyError In-Reply-To: <94bdd2610910050604k5cc20cdblf24a65c307a1cfdf@mail.gmail.com> References: <1254742115.57.0.346118025313.issue7064@psf.upfronthosting.co.za> <19145.60293.661555.312221@montanaro.dyndns.org> <94bdd2610910050604k5cc20cdblf24a65c307a1cfdf@mail.gmail.com> Message-ID: <20091009125205.24460.527857507.divmod.xquotient.228@boson> On 5 Oct, 01:04 pm, ziade.tarek at gmail.com wrote: >On Mon, Oct 5, 2009 at 2:50 PM, wrote: >> >>? ?Ned> Due to a change in distutils released with Python 2.6.3, >>packages >>? ?Ned> that use setuptools (version 0.6c9, as of this writing), or >>the >>? ?Ned> easy_install command, to build C extension modules fail ... >>? ?... >>? ?Ned> Among the packages known to be affected include lxml, >>? ?Ned> zope-interface, jinja2, and, hence, packages dependent on >>these >>? ?Ned> packages (e.g. sphinx, twisted, etc.). >> >>Maybe the Python test suite should include tests with a small number >>of >>widely used non-core packages like setuptools. ?I realize the pybots >>project >>exists to tackle this sort of stuff in greater detail. ?I'm thinking >>more of >>a smoke test than a comprehensive test suite covering all external >>packages. >>Setuptools is particularly important because so many extension authors >>use >>it. ?If it breaks it implicitly breaks a lot of PyPI packages. > >I have created 6 months ago such a buildbot that downloads tarballs >from the community, >and run a few distutils commands on them, and make sure the result is >similar in 2.6/2.7. >and for "sdist" that the resulting tarball is similar. > >It was running over Twisted and Numpy, but has been discontinued >because >it was on my own server, where it was hard to keep it up >(cpu/bandwidth) > >If the Snakebite project could host my buildbot (or at least slaves) >or if the PSF could pay for a dedicated >server for this, we would be able to trigger such warnings, and >provide an e-mail service to package maintainers for example. > >The build could occur everytime Distutils *or* the project changes. If you want, until Snakebite is up, I can probably provide a slave which can at least do this testing for Twisted and perhaps some other projects. Jean-Paul From chris at simplistix.co.uk Fri Oct 9 17:01:36 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 16:01:36 +0100 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> Message-ID: <4ACF5050.1060701@simplistix.co.uk> Guido van Rossum wrote: > I'm saying that I don't expect setuptools 0.7 to appear before Tarek's > Distribute is mature and in widespread use. IOW I support Tarek's fork > and suggest nobody hold their breath waiting for setuptools 0.7. Well, if this was the BDFL pronouncement that a lot of people have taken it to be, does that allow Martin von Lewis to give the keys to http://pypi.python.org/pypi/setuptools to the "distribute" developers, so we can get on and use the original "setuptools" name without all the confusion and hackery needed to make "distribute" work? cheers, Chris PS: I had thought that MvL couldn't help because of ez_setup.py, but it's only buildout that appears foolish enough to have a hardcoded download of: exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py' ).read() in ez ..and even that will keep working provided PJE doesn't take it down... -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Fri Oct 9 17:23:31 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 17:23:31 +0200 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: <4ACF5050.1060701@simplistix.co.uk> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> Message-ID: <94bdd2610910090823x77df06cdm5ed484c30feccced@mail.gmail.com> On Fri, Oct 9, 2009 at 5:01 PM, Chris Withers wrote: > Guido van Rossum wrote: >> >> I'm saying that I don't expect setuptools 0.7 to appear before Tarek's >> Distribute is mature and in widespread use. IOW I support Tarek's fork >> and suggest nobody hold their breath waiting for setuptools 0.7. > > Well, if this was the BDFL pronouncement that a lot of people have taken it > to be, does that allow Martin von Lewis to give the keys to > http://pypi.python.org/pypi/setuptools to the "distribute" developers, so we > can get on and use the original "setuptools" name without all the confusion > and hackery needed to make "distribute" work? > I am -1 on this. At Python and PyPI level, 'Setuptools' is PJE's project and he's the one that manages the list of users that have extended rights on his project. In Distribute, our setup.py script makes sure setuptools is not in the way. It is explicit and documented, and there's a documentation explaining how to switch back to Setuptools. This exists only because there's no way yet in Python to say that a distribution can't run when another distribution is installed. But it's up to every project and developer to choose to switch to Distribute. And it's up to OS packagers to choose the best strategy at their levels when they package tools for their packaging system (so they pick one or the other and they are able to make a distinction between PJE python distribution and ours) Distribute developers will of course promote Distutils usage everywhere, but hijacking setuptools in PyPI would be a very bad thing imho. Tarek From chris at simplistix.co.uk Fri Oct 9 17:32:05 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 16:32:05 +0100 Subject: [Python-Dev] no consensus on static metadata In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4ACF5775.7080507@simplistix.co.uk> Tarek Ziad? wrote: > == The fate of setup.py, and static metadata == > > So we are going to separate the metadata description from setup.py, in > a static configuration file, that can be open and read by anyone > without > running any code. > So we've worked on that lately in Distutils-SIG and came up with a > micro-language, based on a ConfigParser file, that allows > writing metadata fields that depends on sys.platform etc. I won't > detail the syntax here but the idea is that the interpretation > of this file can be done with a vanilla Python without running arbitrary code. I don't understand how the above two paragraphs are compatible. > So I am adding this in Distutils for 2.7. NB: There was no consensus on this "micro-language" on distutils-sig. While I suspect I don't care as none of my packages rely on anything other than other python packages, others did care, and I found the syntax Tarek was proposing pretty clumsy. Does the PEP386 or the changes you're proposing for PEP314 cater for the extra_requires stuff in setuptools? I know I use that quite heavily and I'm not the only one... -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From solipsis at pitrou.net Fri Oct 9 17:33:35 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 Oct 2009 15:33:35 +0000 (UTC) Subject: [Python-Dev] BDFL pronouncement? References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> Message-ID: Chris Withers simplistix.co.uk> writes: > > Well, if this was the BDFL pronouncement that a lot of people have taken > it to be, does that allow Martin von Lewis to give the keys to > http://pypi.python.org/pypi/setuptools to the "distribute" developers, > so we can get on and use the original "setuptools" name without all the > confusion and hackery needed to make "distribute" work? Er, perhaps this is a bit brutal? I don't think the BDFL has a right of life and death over third-party packages. I mean, we're an anarcho-syndicalist commune, aren't we? WOMAN: Who are the Britons? ARTHUR: Well, we all are. we're all Britons and I am your king. WOMAN: I didn't know we had a king. I thought we were an autonomous collective. DENNIS: You're fooling yourself. We're living in a dictatorship. A self-perpetuating autocracy in which the working classes-- [snip] Regards Antoine. PS : > PS: I had thought that MvL couldn't help because of ez_setup.py, but > it's only buildout that appears foolish enough to have a hardcoded > download of: Some projects also have an SVN "externals" pointing to the setuptools SVN repository, as described here: http://peak.telecommunity.com/doc/ez_setup/index.html From chris at simplistix.co.uk Fri Oct 9 17:37:56 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 16:37:56 +0100 Subject: [Python-Dev] Distutils and Distribute "upload_docs" command In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4ACF58D4.70205@simplistix.co.uk> Tarek Ziad? wrote: > Some new commands are added there, when they are helpful and don't > interact with the rest. I am thinking > about "upload_docs" that let you upload documentation to PyPI. The > goal is to move it to Distutils > at some point, if the documentation feature of PyPI stays and starts to be used. This is already available: http://pypi.python.org/pypi/Sphinx-PyPI-upload How about helping the author of that extension make it work with Distribute or just get that implementation into Distutils core instead? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Oct 9 17:42:04 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 16:42:04 +0100 Subject: [Python-Dev] Bits-of-Distribute naming In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4ACF59CC.5050504@simplistix.co.uk> Tarek Ziad? wrote: > - The code is splitted in many packages and might be distributed under > several distributions. > > - distribute.resources: that's the old pkg_resources, but > reorganized in clean, pep-8 modules. This package will > only contain the query APIs and will focus on being PEP 376 > compatible. We will promote its usage and see if Pip wants > to use it as a basis. And maybe PyPM once it's open source ? > ( ). > It will probably shrink a lot though, once the stdlib provides PEP 376 support. Why not just call it pkg_resources and/or merge it with pkgutil to get it into the python stdlib? > - distribute.entrypoints: that's the old pkg_resources entry points > system, but on its own. it uses distribute.resources Why not get it into the core as distutils.entrypoints? That's where it belongs... > - distribute.index: that's package_index and a few other things. > everything required to interact with PyPI. We will promote > its usage and see if Pip wants to use it as a basis. Why not call in "pypi.client" and "pypi.server" or, better yet, get it into the stdlib as disutils.index.client distutils.index.server? > - distribute.core (might be renamed to main): that's everything > else, and uses the other packages. ...which would be left, and could just be called "distribute". cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Fri Oct 9 17:45:31 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 17:45:31 +0200 Subject: [Python-Dev] no consensus on static metadata In-Reply-To: <4ACF5775.7080507@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF5775.7080507@simplistix.co.uk> Message-ID: <94bdd2610910090845k56f89f7fk2fe0a834f3172a99@mail.gmail.com> On Fri, Oct 9, 2009 at 5:32 PM, Chris Withers wrote: > >> So I am adding this in Distutils for 2.7. > > NB: There was no consensus on this "micro-language" on distutils-sig. > While I suspect I don't care as none of my packages rely on anything other > than other python packages, others did care, and I found the syntax Tarek > was proposing pretty clumsy. Please no bikeshedding again on the syntax. "consensus" doesn't mean that you agreed on the syntax. As I said to MAL, a PEP will be written with the precise syntax, and you will be able to make your voice heard on **Distutils-SIG**, even participate to it. But at the end, a syntax will be added. > Does the PEP386 or the changes you're proposing for PEP314 cater for the > extra_requires stuff in setuptools? I know I use that quite heavily and I'm > not the only one... Let's open a thread on Distutils-SIG about PEP 314 planned changes. Tarek. From chris at simplistix.co.uk Fri Oct 9 17:51:00 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 16:51:00 +0100 Subject: [Python-Dev] supporting multiple versions of one package in one environment is insane In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4ACF5BE4.70907@simplistix.co.uk> Tarek Ziad? wrote: > = Virtualenv and the multiple version support in Distribute = > > (I am not saying "We" here because this part was not discussed yet > with everyone) Good, so maybe take this discussion to distutils-sig first? > Virtualenv allows you to create an isolated environment to install > some distribution without polluting the > main site-packages, a bit like a user site-packages. ...as does buildout, and these are the right type of solution to this problem. > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. No, it exists because per-project specification of requirements is a really good thing. Having multiple versions of the same package in the same python environment it totally batshit crazy... > Setuptools provided a multiple version support but I don't like its > implementation and the way its works. See the "crazy" comment above... > - a special import system using importlib that would automatically > pick the latest version, thanks to PEP 376. As long as I never have to use it or see it, I don't really care, but this seems like creating more work for yourself and another gun pointed at the face, nevermind the foot, of the unwary newbie... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Fri Oct 9 17:56:31 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 17:56:31 +0200 Subject: [Python-Dev] Distutils and Distribute "upload_docs" command In-Reply-To: <4ACF58D4.70205@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF58D4.70205@simplistix.co.uk> Message-ID: <94bdd2610910090856x17808758g2896b80cc0024b9f@mail.gmail.com> On Fri, Oct 9, 2009 at 5:37 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> Some new commands are added there, when they are helpful and don't >> interact with the rest. I am thinking >> about "upload_docs" that let you upload documentation to PyPI. The >> goal is to move it to Distutils >> at some point, if the documentation feature of PyPI stays and starts to be >> used. > > This is already available: > > http://pypi.python.org/pypi/Sphinx-PyPI-upload > > How about helping the author of that extension make it work with Distribute > or just get that implementation into Distutils core instead? It's the case. The author is a Distribute commiter and *he* has added it in Distribute. PS: its' weird, it looks like your answers to the original thread are not belonging to the thread anymore. Is that my gmail or something on your mail client. Tarek From chris at simplistix.co.uk Fri Oct 9 18:03:41 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 17:03:41 +0100 Subject: [Python-Dev] Distutils and Distribute "upload_docs" command In-Reply-To: <94bdd2610910090856x17808758g2896b80cc0024b9f@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF58D4.70205@simplistix.co.uk> <94bdd2610910090856x17808758g2896b80cc0024b9f@mail.gmail.com> Message-ID: <4ACF5EDD.1030707@simplistix.co.uk> Tarek Ziad? wrote: >> How about helping the author of that extension make it work with Distribute >> or just get that implementation into Distutils core instead? > > It's the case. The author is a Distribute commiter and *he* has added > it in Distribute. Great, but about just getting it into Distutils? I can't imagine it's that contentious... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Fri Oct 9 18:05:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 18:05:50 +0200 Subject: [Python-Dev] Bits-of-Distribute naming In-Reply-To: <4ACF59CC.5050504@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF59CC.5050504@simplistix.co.uk> Message-ID: <94bdd2610910090905t6fcdc6d9rfe946bbfbd36a7b9@mail.gmail.com> On Fri, Oct 9, 2009 at 5:42 PM, Chris Withers wrote: > Tarek Ziad? wrote: >> >> - The code is splitted in many packages and might be distributed under >> several distributions. >> >> ? - distribute.resources: that's the old pkg_resources... > Why not just call it pkg_resources and/or merge it with pkgutil to get it > into the python stdlib? Because "pkg_resources" belongs to 0.6 and we want 0.7 to be installable alongside 0.6 if needed. Now as I said earlier, this package might shrink once PEP 376 is finished, and have bits included in pkgutil. > >> ? - distribute.entrypoints: that's the old pkg_resources entry points >> system, but on its own. it uses distribute.resources > > Why not get it into the core as distutils.entrypoints? That's where it > belongs... What do you call 'core' ? distutils.core ? If yes, why does it belong to core ? entrypoint is a standalon plugin system . > >> ? - distribute.index: that's package_index and a few other things. >> everything required to interact with PyPI. We will promote >> ? ? its usage and see if Pip wants to use it as a basis. > > Why not call in "pypi.client" and "pypi.server" or, better yet, get it into > the stdlib as disutils.index.client distutils.index.server? We are keeping a "distribute." namespace for all bits. Now maybe one day it'll make it into Distutils, yes. But not now. Also, have a look at PEP 381 (mirroring infratructure for PyPI) In any case, the splitting is occuring *now* and is not written in the stone, so we can talk about this in Distutils-SIG, and improve it with your feedback. Regards, Tarek From status at bugs.python.org Fri Oct 9 18:07:11 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 9 Oct 2009 18:07:11 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20091009160711.6370E7865B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (10/02/09 - 10/09/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2444 open (+44) / 16471 closed (+14) / 18915 total (+58) Open issues with patches: 976 Average duration of open issues: 670 days. Median duration of open issues: 420 days. Open Issues Breakdown open 2410 (+44) pending 33 ( +0) Issues Created Or Reopened (59) _______________________________ traceback.format_exception_only does not return SyntaxError caro 10/07/09 http://bugs.python.org/issue6962 reopened thewtex patch Doc/reference/datamodel: Slots description needs update 10/02/09 http://bugs.python.org/issue7036 created Byron patch test_asynchat fails on os x 10.6 10/02/09 http://bugs.python.org/issue7037 created chuck test_curses fails on os x 10.6 10/02/09 http://bugs.python.org/issue7038 created chuck test_distutils fails on os x 10.6 10/02/09 CLOSED http://bugs.python.org/issue7039 created chuck test_smtplib fails on os x 10.6 10/02/09 http://bugs.python.org/issue7040 created chuck test_macostools fails on os x 10.6 10/02/09 http://bugs.python.org/issue7041 created chuck test_signal fails on os x 10.6 10/02/09 http://bugs.python.org/issue7042 created chuck patch test_urllib: constructLocalFileUrl returns invalid URLs on Windo 10/03/09 http://bugs.python.org/issue7043 created srid urllib.urlopen crashes when used on Mac OS 10.6.1 through a prox 10/03/09 http://bugs.python.org/issue7044 created jweber utf-8 encoding error 10/03/09 http://bugs.python.org/issue7045 created ArcRiley decimal.py: use DivisionImpossible and DivisionUndefined 10/03/09 http://bugs.python.org/issue7046 created skrah decimal.py: NaN payload conversion 10/03/09 CLOSED http://bugs.python.org/issue7047 created skrah decimal.py: logb: round the result if it is greater than prec 10/03/09 http://bugs.python.org/issue7048 created skrah patch decimal.py: NaN result in pow(x, y, z) with prec 1 10/03/09 http://bugs.python.org/issue7049 created skrah patch Augmented assignment of tuple crashes IDLE 10/03/09 CLOSED http://bugs.python.org/issue7050 created mfitz 'g'/'G' format docs need a little more explanation 10/03/09 http://bugs.python.org/issue7051 created benjamin.peterson "from logging import *" causes an error under Ubuntu Karmic 10/04/09 CLOSED http://bugs.python.org/issue7052 created vak wrong overload of slot wrapper 10/04/09 CLOSED http://bugs.python.org/issue7053 created omatt Python25.chm seems to be broken 10/04/09 CLOSED http://bugs.python.org/issue7054 created Mark77 Automatic test___all__ 10/04/09 http://bugs.python.org/issue7055 created pitrou patch regrtest runtest_inner calls findtestdir unnecessarily 10/04/09 http://bugs.python.org/issue7056 created r.david.murray patch tkinter doc: more 3.x updates 10/04/09 http://bugs.python.org/issue7057 created tjreedy Add some test execution environment protection to regrtest 10/04/09 http://bugs.python.org/issue7058 created r.david.murray patch 'checking getaddrinfo bug' doesn't output the result during ./co 10/05/09 http://bugs.python.org/issue7059 created ezio.melotti test_multiprocessing dictionary changed size errors and hang 10/05/09 http://bugs.python.org/issue7060 created r.david.murray Improve 24.5. turtle doc 10/05/09 http://bugs.python.org/issue7061 created tjreedy No docs for module 'IN' 10/05/09 http://bugs.python.org/issue7062 created RSokolov Memory errors in array.array 10/05/09 http://bugs.python.org/issue7063 created chuck Python 2.6.3 / setuptools 0.6c9: extension module builds fail wi 10/05/09 CLOSED http://bugs.python.org/issue7064 reopened tarek bytes.maketrans segfaults 10/05/09 http://bugs.python.org/issue7065 created egreen patch archive_util.make_archive doesn't restore the cwd if an error is 10/05/09 http://bugs.python.org/issue7066 created ezio.melotti Suggest better documentation of 2 versus 3 on 3.1.1 site. 10/05/09 http://bugs.python.org/issue7067 created srl 2.6.3 does not use specified compiler 10/06/09 CLOSED http://bugs.python.org/issue7068 created gotoh inspect.isabstract to return boolean values only 10/06/09 http://bugs.python.org/issue7069 created gagenellina patch round(x) gives wrong result for large odd integers 10/06/09 http://bugs.python.org/issue7070 created mark.dickinson easy distutils and IronPython compatibility 10/06/09 http://bugs.python.org/issue7071 created michael.foord 26backport isspace(0xa0) is true on Mac OS X 10/06/09 http://bugs.python.org/issue7072 created naoki Python 2.6.3 final windows installer installs a release candida 10/06/09 CLOSED http://bugs.python.org/issue7073 created jamartinh Turtle module crashes python 10/06/09 http://bugs.python.org/issue7074 created tjreedy Bug while put a set on a dict 10/07/09 CLOSED http://bugs.python.org/issue7075 created chucheng Documentation add note about SystemRandom 10/07/09 http://bugs.python.org/issue7076 created sligocki patch SysLogHandler can't handle Unicode 10/07/09 http://bugs.python.org/issue7077 created rszefler struct help in the interpreter does not explain about the fmt op 10/07/09 http://bugs.python.org/issue7078 created orsenthil patch, easy file_close() ignores return value of close_the_file 10/07/09 http://bugs.python.org/issue7079 created stutzbach patch locale.strxfrm raises MemoryError 10/07/09 http://bugs.python.org/issue7080 created egreen patch ">" formatting behaving like "=" formatting 10/08/09 CLOSED http://bugs.python.org/issue7081 created ncoghlan Patch for get_filename in email.message when content-disposition 10/08/09 http://bugs.python.org/issue7082 created DazWorrall patch locals() behaviour differs when tracing is in effect 10/08/09 CLOSED http://bugs.python.org/issue7083 created andbj printing a list releases the GIL carelessly 10/08/09 http://bugs.python.org/issue7084 created pitrou strptime problem 10/08/09 http://bugs.python.org/issue7085 created kakacek logging.handlers.SysLogHandler with TCP support 10/08/09 http://bugs.python.org/issue7086 created enigma patch use keyword 'as' in _xmlplus 10/09/09 CLOSED http://bugs.python.org/issue7087 created s7v7nislands New (in 2.6) functions in signal module are not documented as un 10/09/09 http://bugs.python.org/issue7088 created stutzbach shlex behaves unexpected if newlines are not whitespace 10/09/09 http://bugs.python.org/issue7089 created jjdmol2 patch encoding uncode objects greater than FFFF 10/09/09 CLOSED http://bugs.python.org/issue7090 created msaghaei Distutils build ignores the --compiler command line option 10/09/09 http://bugs.python.org/issue7091 created jmb Test suite emits many DeprecationWarnings when -3 is enabled 10/09/09 http://bugs.python.org/issue7092 created exarkun xmlrpclib.ServerProxy() doesn't support unicode uri 10/09/09 http://bugs.python.org/issue7093 created haypo patch Issues Now Closed (30) ______________________ fromfd() and dup() for _socket on WIndows 701 days http://bugs.python.org/issue1378 planders patch unittest.py sys.exit error 510 days http://bugs.python.org/issue2821 mfitz fix_idioms.py generates bad code 418 days http://bugs.python.org/issue3563 benjamin.peterson patch distutils.util.get_platform() is wrong for universal builds on m 366 days http://bugs.python.org/issue4064 ronaldoussoren patch, needs review make io.BufferedWriter observe max_buffer_size limits 318 days http://bugs.python.org/issue4428 pitrou patch, needs review PEP 383 implementation 157 days http://bugs.python.org/issue5915 exarkun patch reset owner/group to root for distutils tarballs 80 days http://bugs.python.org/issue6516 bheemesh subprocess issue on Win 7 x64 44 days http://bugs.python.org/issue6773 tesla unmarshaling of artificial strings can produce funny longs. 4 days http://bugs.python.org/issue7019 mark.dickinson patch Doc update for io module 5 days http://bugs.python.org/issue7022 pakal webbrowser : Could not open ftp server using webbrowser.open() 5 days http://bugs.python.org/issue7024 orsenthil test_urllib: unsetting missing 'env' variable 4 days http://bugs.python.org/issue7026 orsenthil Add int.hex for symmetry with float.hex 1 days http://bugs.python.org/issue7028 mark.dickinson Improve pybench 8 days http://bugs.python.org/issue7029 krisvale patch test_distutils fails on os x 10.6 0 days http://bugs.python.org/issue7039 ned.deily decimal.py: NaN payload conversion 0 days http://bugs.python.org/issue7047 mark.dickinson Augmented assignment of tuple crashes IDLE 0 days http://bugs.python.org/issue7050 benjamin.peterson "from logging import *" causes an error under Ubuntu Karmic 1 days http://bugs.python.org/issue7052 srid wrong overload of slot wrapper 0 days http://bugs.python.org/issue7053 georg.brandl Python25.chm seems to be broken 0 days http://bugs.python.org/issue7054 Mark77 Python 2.6.3 / setuptools 0.6c9: extension module builds fail wi 1 days http://bugs.python.org/issue7064 schmir 2.6.3 does not use specified compiler 0 days http://bugs.python.org/issue7068 tarek Python 2.6.3 final windows installer installs a release candida 0 days http://bugs.python.org/issue7073 ezio.melotti Bug while put a set on a dict 0 days http://bugs.python.org/issue7075 benjamin.peterson ">" formatting behaving like "=" formatting 0 days http://bugs.python.org/issue7081 ncoghlan locals() behaviour differs when tracing is in effect 0 days http://bugs.python.org/issue7083 andbj use keyword 'as' in _xmlplus 0 days http://bugs.python.org/issue7087 benjamin.peterson encoding uncode objects greater than FFFF 0 days http://bugs.python.org/issue7090 ezio.melotti urllib doesn't correct server returned urls 2030 days http://bugs.python.org/issue918368 AdamN patch Generate numeric/space/linebreak from Unicode database. 1098 days http://bugs.python.org/issue1571184 amaury.forgeotdarc patch Top Issues Most Discussed (10) ______________________________ 16 UnicodeEncodeError - I can't even see license 250 days open http://bugs.python.org/issue5127 13 'g'/'G' format docs need a little more explanation 6 days open http://bugs.python.org/issue7051 13 test_signal fails on os x 10.6 7 days open http://bugs.python.org/issue7042 11 "from logging import *" causes an error under Ubuntu Karmic 1 days closed http://bugs.python.org/issue7052 11 decimal.py: NaN result in pow(x, y, z) with prec 1 6 days open http://bugs.python.org/issue7049 11 decimal.py: logb: round the result if it is greater than prec 6 days open http://bugs.python.org/issue7048 9 Python 2.6.3 / setuptools 0.6c9: extension module builds fail w 1 days closed http://bugs.python.org/issue7064 9 expat parser throws Memory Error when parsing multiple files 60 days open http://bugs.python.org/issue6676 9 array.fromfile not checking I/O errors 223 days open http://bugs.python.org/issue5395 8 utf-8 encoding error 7 days open http://bugs.python.org/issue7045 From chris at simplistix.co.uk Fri Oct 9 18:08:40 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 17:08:40 +0100 Subject: [Python-Dev] Bits-of-Distribute naming In-Reply-To: <94bdd2610910090905t6fcdc6d9rfe946bbfbd36a7b9@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF59CC.5050504@simplistix.co.uk> <94bdd2610910090905t6fcdc6d9rfe946bbfbd36a7b9@mail.gmail.com> Message-ID: <4ACF6008.8000201@simplistix.co.uk> Tarek Ziad? wrote: >>> - distribute.entrypoints: that's the old pkg_resources entry points >>> system, but on its own. it uses distribute.resources >> Why not get it into the core as distutils.entrypoints? That's where it >> belongs... > > What do you call 'core' ? distutils.core ? Sorry, mean stdlib. distutils.entrypoints would seem to be the sensible place. >>> - distribute.index: that's package_index and a few other things. >>> everything required to interact with PyPI. We will promote >>> its usage and see if Pip wants to use it as a basis. >> Why not call in "pypi.client" and "pypi.server" or, better yet, get it into >> the stdlib as disutils.index.client distutils.index.server? > > We are keeping a "distribute." namespace for all bits. Why? > In any case, the splitting is occuring *now* and is not written in the stone, > so we can talk about this in Distutils-SIG, and improve it with your feedback. Shame there hasn't been any discussion of this recently on distutils-sig... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Fri Oct 9 18:12:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 18:12:50 +0200 Subject: [Python-Dev] Distutils and Distribute "upload_docs" command In-Reply-To: <4ACF5EDD.1030707@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF58D4.70205@simplistix.co.uk> <94bdd2610910090856x17808758g2896b80cc0024b9f@mail.gmail.com> <4ACF5EDD.1030707@simplistix.co.uk> Message-ID: <94bdd2610910090912w4018e244jcaa08830b0555669@mail.gmail.com> On Fri, Oct 9, 2009 at 6:03 PM, Chris Withers wrote: > Tarek Ziad? wrote: >>> >>> How about helping the author of that extension make it work with >>> Distribute >>> or just get that implementation into Distutils core instead? >> >> It's the case. The author is a Distribute commiter and *he* has added >> it in Distribute. > > Great, but about just getting it into Distutils? I can't imagine it's that > contentious... This PyPI feature is pretty new. It's great to have the client command on Distribute. If its usage spreads it'll make it to Distutils. If you want to see it included Distutils (I do too) help us promoting its usage Tarek. From chris at simplistix.co.uk Fri Oct 9 18:15:08 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 17:15:08 +0100 Subject: [Python-Dev] Distutils and Distribute "upload_docs" command In-Reply-To: <94bdd2610910090912w4018e244jcaa08830b0555669@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF58D4.70205@simplistix.co.uk> <94bdd2610910090856x17808758g2896b80cc0024b9f@mail.gmail.com> <4ACF5EDD.1030707@simplistix.co.uk> <94bdd2610910090912w4018e244jcaa08830b0555669@mail.gmail.com> Message-ID: <4ACF618C.7000202@simplistix.co.uk> Tarek Ziad? wrote: > This PyPI feature is pretty new. It's great to have the client command > on Distribute. > If its usage spreads it'll make it to Distutils. > > If you want to see it included Distutils (I do too) help us promoting its usage The only thing that discourages me from using the PyPI docs feature is that it only allows documentation per-distribution, not per-version-per-distribution. While you can work around this by building docs for all versions and putting them in one .zip file, that seems like a hack. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Fri Oct 9 18:14:54 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Oct 2009 18:14:54 +0200 Subject: [Python-Dev] Bits-of-Distribute naming In-Reply-To: <4ACF6008.8000201@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF59CC.5050504@simplistix.co.uk> <94bdd2610910090905t6fcdc6d9rfe946bbfbd36a7b9@mail.gmail.com> <4ACF6008.8000201@simplistix.co.uk> Message-ID: <94bdd2610910090914v3eb66ea8p8c954841260edc80@mail.gmail.com> On Fri, Oct 9, 2009 at 6:08 PM, Chris Withers wrote: > > Shame there hasn't been any discussion of this recently on distutils-sig... We had many discussion on this already. I don't mind to have another round, on the contrary, but let's do it in Distutils-SIG We are making a lot of noise now in Python-dev, which is not required Tarek From chris at simplistix.co.uk Fri Oct 9 18:17:12 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 17:17:12 +0100 Subject: [Python-Dev] no consensus on static metadata In-Reply-To: <94bdd2610910090845k56f89f7fk2fe0a834f3172a99@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF5775.7080507@simplistix.co.uk> <94bdd2610910090845k56f89f7fk2fe0a834f3172a99@mail.gmail.com> Message-ID: <4ACF6208.4030506@simplistix.co.uk> Tarek Ziad? wrote: >> NB: There was no consensus on this "micro-language" on distutils-sig. >> While I suspect I don't care as none of my packages rely on anything other >> than other python packages, others did care, and I found the syntax Tarek >> was proposing pretty clumsy. > > Please no bikeshedding again on the syntax. Tarek, throwing "bikeshedding" at an important syntactical discussion like this is verging on rude. Please stop it and accept that the best solution has not been found yet. > As I said to MAL, a PEP will be written with the precise syntax, and you will > be able to make your voice heard on **Distutils-SIG**, even participate to it. Great. >> Does the PEP386 or the changes you're proposing for PEP314 cater for the >> extra_requires stuff in setuptools? I know I use that quite heavily and I'm >> not the only one... > > Let's open a thread on Distutils-SIG about PEP 314 planned changes. OK, look forward to reading it and commenting. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ianb at colorstudy.com Fri Oct 9 18:23:29 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 9 Oct 2009 11:23:29 -0500 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACEFA3F.2020706@cryptelium.net> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <4ACEFA3F.2020706@cryptelium.net> Message-ID: On Fri, Oct 9, 2009 at 3:54 AM, kiorky wrote: >> If I had my way, buildout would use virtualenv and throw away its >> funny script generation. ?If virtualenv had existed before buildout > > Which one, the one provided to generate scripts from entry points with the *.egg > recipes or the bin/buildout auto regeneration? Well, if multi-versioned installs were deprecated, it would not be necessary to use Setuptools' style of script generation. Instead you could simply dereference the entry point, calling the underlying function directly in the script. This detail is probably more of a distutils-sig question, and I don't have a strong opinion. But I was thinking specifically of the egg activation buildout puts at the top of scripts. >> began development, probably things would have gone this way. ?I think >> it would make the environment more pleasant for buildout users. ?Also > > * I don't think so, buildout is the only tool atm that permit to have really > reproducible and isolated environments. Even, if you use the pip freezing > machinery, it is not equivalent to buildout, Control! I believe that to fully insulate buildout you need still virtualenv --no-site-packages. But I'm not arguing that virtualenv/pip makes buildout obsolete, only that they have overlapping functionality, and I think buildout would benefit from making use of that overlap. > * Buildout can have single part to construct required eggs, at a specific > version and let you control that. Pip will just search for this version, see > that it's not available and fail. You have even recipes (like > minitage.recipe.egg that permit to construct eggs with special version when you > apply patches onto, thus, you can have the same egg in different flavors in the > same eggs cache available for different projects. Those projects will just have > to pin the right version to use, Control!. In my own work I use multiple virtualenv environments for this use case, to similar effect. pip of course is not a generalized build tool, but then minitage.recipe.egg is not the main egg installer either. > * Another thing is the "funny script generation", you have not one global > site-packages for your project, but one global cache. But from this global > cache, your scripts will only have available the eggs you declared, see Control! > * Moreover buildout is not only a python packages manager, it's some of its > recipes that permit to use it as. Buildout is just a great deployment tool that > allow to script and manage your project in a "funny" and "flexible" way, Control! Sure; I'm just advocating that buildout more explicitly use some of the functionality of virtualenv/pip (which may require some more features in those tools, but I'm open to that). But specific discussion of this would probably be more appropriate on distutils-sig. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From chris at simplistix.co.uk Fri Oct 9 18:25:18 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 17:25:18 +0100 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: <94bdd2610910090823x77df06cdm5ed484c30feccced@mail.gmail.com> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> <94bdd2610910090823x77df06cdm5ed484c30feccced@mail.gmail.com> Message-ID: <4ACF63EE.9040001@simplistix.co.uk> Tarek Ziad? wrote: > At Python and PyPI level, 'Setuptools' is PJE's project and he's the > one that manages the list of users that have > extended rights on his project. Indeed, if only he would see sense :-( > In Distribute, our setup.py script makes sure setuptools is not in the > way. It is explicit and documented, > and there's a documentation explaining how to switch back to Setuptools. It's a hack though... > This exists only because there's no way yet in Python to say that a > distribution can't run when another distribution is installed. I don't this should ever be needed, except in this edge case caused by one man's stubbornness... > Distribute developers will of course promote Distutils usage > everywhere, but hijacking setuptools in PyPI > would be a very bad thing imho. *sigh* I don't see it as hijacking, provided Guido is making a BDFL pronouncement that you're maintaining this software and PJE finally sees sense and accepts that. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From a.badger at gmail.com Fri Oct 9 18:22:26 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 9 Oct 2009 09:22:26 -0700 Subject: [Python-Dev] supporting multiple versions of one package in one environment is insane In-Reply-To: <4ACF5BE4.70907@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF5BE4.70907@simplistix.co.uk> Message-ID: <20091009162226.GA5033@clingman.lan> On Fri, Oct 09, 2009 at 04:51:00PM +0100, Chris Withers wrote: > Tarek Ziad? wrote: >> Virtualenv allows you to create an isolated environment to install >> some distribution without polluting the >> main site-packages, a bit like a user site-packages. > > ...as does buildout, and these are the right type of solution to this > problem. > where type of problem == sandboxed environment, sure. How do you solve the problem for system packagers? -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From chris at simplistix.co.uk Fri Oct 9 18:29:28 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Oct 2009 17:29:28 +0100 Subject: [Python-Dev] supporting multiple versions of one package in one environment is insane In-Reply-To: <20091009162226.GA5033@clingman.lan> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF5BE4.70907@simplistix.co.uk> <20091009162226.GA5033@clingman.lan> Message-ID: <4ACF64E8.8010501@simplistix.co.uk> Toshio Kuratomi wrote: > On Fri, Oct 09, 2009 at 04:51:00PM +0100, Chris Withers wrote: >> Tarek Ziad? wrote: >>> Virtualenv allows you to create an isolated environment to install >>> some distribution without polluting the >>> main site-packages, a bit like a user site-packages. >> ...as does buildout, and these are the right type of solution to this >> problem. >> > where type of problem == sandboxed environment, sure. How do you solve the > problem for system packagers? What's to stop a system packager either just running the buildout on install, or running the buildout at package build time and then just dropping the resulting environment wherever they want to install applications? Such a package would only be dependent on the right python version at runtime... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ianb at colorstudy.com Fri Oct 9 18:34:10 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 9 Oct 2009 11:34:10 -0500 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> Message-ID: On Fri, Oct 9, 2009 at 7:32 AM, Paul Moore wrote: > 2009/10/9 Antoine Pitrou : >> Ian Bicking colorstudy.com> writes: >>> >>> Someone mentioned that easy_install provided some things pip didn't; >>> outside of multi-versioned installs (which I'm not very enthusiastic >>> about) I'm not sure what this is? >> >> http://pip.openplans.org/#differences-from-easy-install >> >> If it's obsolete the website should be updated... > > Specifically, combine "only installs from source" with "might not work > on Windows" and the result is pretty certainly unusable for C > extensions on Windows. You can pretty much guarantee that the average > user on Windows won't have a C compiler[1], and even if they do, they > won't be able to carefully line up all the 3rd party C libraries > needed to build some extensions. > > Binary packages are essential on Windows. I'll admit I have some blindness when it comes to Windows. I agree binary installation on Windows is important. (I don't think it's very important on other platforms, or at least not very effective in easy_install so it wouldn't be a regression.) I note some other differences in that document: > It cannot install from eggs. It only installs from source. (Maybe this will be changed sometime, but it?s low priority.) Outside of binaries on Windows, I'm still unsure if installing eggs serves a useful purpose. I'm not sure if eggs are any better than wininst binaries either...? > It doesn?t understand Setuptools extras (like package[test]). This should be added eventually. I haven't really seen Setuptools' extras used effectively, so I'm unsure if it's a useful feature. I understand the motivation for extras, but motivated features aren't necessarily useful features. > It is incompatible with some packages that customize distutils or setuptools in their setup.py files. I don't have a solution for this, and generally easy_install does not perform much better than pip in these cases. Work in Distribute hopefully will apply to this issue. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From kiorky at cryptelium.net Fri Oct 9 18:39:11 2009 From: kiorky at cryptelium.net (kiorky) Date: Fri, 09 Oct 2009 18:39:11 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <4ACEFA3F.2020706@cryptelium.net> Message-ID: <4ACF672F.9070701@cryptelium.net> I'm crossposting to continue on distutils. Ian Bicking a ?crit : > On Fri, Oct 9, 2009 at 3:54 AM, kiorky wrote: > Well, if multi-versioned installs were deprecated, it would not be > necessary to use Setuptools' style of script generation. Instead you > could simply dereference the entry point, calling the underlying > function directly in the script. This detail is probably more of a > distutils-sig question, and I don't have a strong opinion. > > But I was thinking specifically of the egg activation buildout puts at > the top of scripts. The egg activation is just the result of the constructed working set, which aggregates all the python packages that fulfill the options inputed by the user. The only needed packages and nothing else or more. >>> began development, probably things would have gone this way. I think >>> it would make the environment more pleasant for buildout users. Also >> * I don't think so, buildout is the only tool atm that permit to have really >> reproducible and isolated environments. Even, if you use the pip freezing >> machinery, it is not equivalent to buildout, Control! > > I believe that to fully insulate buildout you need still virtualenv > --no-site-packages. But I'm not arguing that virtualenv/pip makes Right, I do so, but i also drop nothing in that virtualenv site-packages. > buildout obsolete, only that they have overlapping functionality, and > I think buildout would benefit from making use of that overlap. recipes have more features and are another way to do, complementary or not. (patching, compiling options, pinning version). You have also more control on what is installed/used/shared with buildout than with the others. >> * Buildout can have single part to construct required eggs, at a specific >> version and let you control that. Pip will just search for this version, see >> that it's not available and fail. You have even recipes (like >> minitage.recipe.egg that permit to construct eggs with special version when you >> apply patches onto, thus, you can have the same egg in different flavors in the >> same eggs cache available for different projects. Those projects will just have >> to pin the right version to use, Control!. > > In my own work I use multiple virtualenv environments for this use > case, to similar effect. pip of course is not a generalized build > tool, but then minitage.recipe.egg is not the main egg installer > either. *.recipe.egg can use caches to avoid duplications among many projects environments. You setup seems to duplicate stuff around. >> * Another thing is the "funny script generation", you have not one global >> site-packages for your project, but one global cache. But from this global >> cache, your scripts will only have available the eggs you declared, see Control! >> * Moreover buildout is not only a python packages manager, it's some of its >> recipes that permit to use it as. Buildout is just a great deployment tool that >> allow to script and manage your project in a "funny" and "flexible" way, Control! > > Sure; I'm just advocating that buildout more explicitly use some of > the functionality of virtualenv/pip (which may require some more > features in those tools, but I'm open to that). But specific > discussion of this would probably be more appropriate on > distutils-sig. > -- -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From fuzzyman at voidspace.org.uk Fri Oct 9 18:49:23 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 09 Oct 2009 17:49:23 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> Message-ID: <4ACF6993.5040704@voidspace.org.uk> Ian Bicking wrote: > On Fri, Oct 9, 2009 at 7:32 AM, Paul Moore wrote: > >> 2009/10/9 Antoine Pitrou : >> >>> Ian Bicking colorstudy.com> writes: >>> >>>> Someone mentioned that easy_install provided some things pip didn't; >>>> outside of multi-versioned installs (which I'm not very enthusiastic >>>> about) I'm not sure what this is? >>>> >>> http://pip.openplans.org/#differences-from-easy-install >>> >>> If it's obsolete the website should be updated... >>> >> Specifically, combine "only installs from source" with "might not work >> on Windows" and the result is pretty certainly unusable for C >> extensions on Windows. You can pretty much guarantee that the average >> user on Windows won't have a C compiler[1], and even if they do, they >> won't be able to carefully line up all the 3rd party C libraries >> needed to build some extensions. >> >> Binary packages are essential on Windows. >> > > I'll admit I have some blindness when it comes to Windows. I agree > binary installation on Windows is important. (I don't think it's very > important on other platforms, or at least not very effective in > easy_install so it wouldn't be a regression.) > > I note some other differences in that document: > > >> It cannot install from eggs. It only installs from source. (Maybe this will be changed sometime, but it?s low priority.) >> > > Outside of binaries on Windows, I'm still unsure if installing eggs > serves a useful purpose. I'm not sure if eggs are any better than > wininst binaries either...? > > Many Windows users would be quite happy if the standard mechanism for installing non-source distributions on Windows was via the wininst binaries. I wonder if it is going to be possible to make this compatible with the upcoming distutils package management 'stuff' (querying for installed packages, uninstallation etc) since installation/uninstallation goes through the Windows system package management feature. I guess it would be eminently possible but require some reasonably high level Windows-fu to do. Michael >> It doesn?t understand Setuptools extras (like package[test]). This should be added eventually. >> > > I haven't really seen Setuptools' extras used effectively, so I'm > unsure if it's a useful feature. I understand the motivation for > extras, but motivated features aren't necessarily useful features. > > >> It is incompatible with some packages that customize distutils or setuptools in their setup.py files. >> > > I don't have a solution for this, and generally easy_install does not > perform much better than pip in these cases. Work in Distribute > hopefully will apply to this issue. > > -- http://www.ironpythoninaction.com/ From ianb at colorstudy.com Fri Oct 9 19:02:07 2009 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 9 Oct 2009 12:02:07 -0500 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACF6993.5040704@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> Message-ID: Probably all these discussions are better on distutils-sig (just copying python-dev to note the movement of the discussion) On Fri, Oct 9, 2009 at 11:49 AM, Michael Foord wrote: >> Outside of binaries on Windows, I'm still unsure if installing eggs >> serves a useful purpose. ?I'm not sure if eggs are any better than >> wininst binaries either...? > > Many Windows users would be quite happy if the standard mechanism for > installing non-source distributions on Windows was via the wininst binaries. > > I wonder if it is going to be possible to make this compatible with the > upcoming distutils package management 'stuff' (querying for installed > packages, uninstallation etc) since installation/uninstallation goes through > the Windows system package management feature. ?I guess it would be > eminently possible but require some reasonably high level Windows-fu to do. As far as pip works, it unpacks a package and runs "python setup.py install (and some options that aren't that interesting, but are provided specifically by setuptools)". Well, it's slightly more complicated, but more to the point it doesn't install in-process or dictate how setup.py works, except that it takes some specific options. Running a Windows installer in the same way would be fine, in that sense. Alternately pip could unpack the wininst zip file and install it directly; I'm not sure if that would be better or worse? If wininst uses the central package manager of the OS then certain features (like virtualenv, PYTHONHOME, --prefix, etc) would not be possible. For Distribute (or Setuptools or by association pip) to see that a package is installed, it must have the appropriate metadata. For Setuptools (and Distribute 0.6) this is a directory or file, on sys.path, "Package.egg-info" (or in Package-X.Y.egg/EGG-INFO). If a file, it should be a PKG-INFO file, if a directory it should contain a PKG-INFO file. So however the package gets installed, if that metadata is installed then it can be queried. I don't think querying the Windows system package management would be necessary or desirable. Nobody is trying that with deb/rpm either. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker From solipsis at pitrou.net Fri Oct 9 19:57:57 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 Oct 2009 17:57:57 +0000 (UTC) Subject: [Python-Dev] BDFL pronouncement? References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> <94bdd2610910090823x77df06cdm5ed484c30feccced@mail.gmail.com> <4ACF63EE.9040001@simplistix.co.uk> Message-ID: Chris Withers simplistix.co.uk> writes: > > *sigh* I don't see it as hijacking, provided Guido is making a BDFL > pronouncement that you're maintaining this software Well, what you are proposing *is* hijacking. PJE is not paid or employed by Guido, he is the full author of setuptools. Forking is of course fine (this is free software), but saying "I overthrow you as the maintainer of this software" is not. While you (and I) may be unsatisfied with PJE's maintenance style, personal (dis)affection should not be what motivates our decisions. A piece of software has an author who has some moral rights on it, and should be treated with respect. After all, when an author decides to release code under a free license, it shows a fair amount of trust and respect towards the user; it is only fair that the user shows the same amount of trust and respect (*). (*) it has happened that some users don't, and in that case the author's reaction can be violent. For example, Nessus became proprietary: http://news.cnet.com/Nessus-security-tool-closes-its-source/2100-7344_3-5890093.html Regards Antoine. From brett at python.org Fri Oct 9 20:00:53 2009 From: brett at python.org (Brett Cannon) Date: Fri, 9 Oct 2009 11:00:53 -0700 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACF243F.4080408@voidspace.org.uk> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> Message-ID: On Fri, Oct 9, 2009 at 04:53, Michael Foord wrote: > Christian Heimes wrote: > >> Michael Foord wrote: >> >> >>> I really like this scheme. The important thing for IronPython is that we >>> can get it into Python 2.6 (along with other fixes to make distutils >>> compatible with IronPython - like not attempting to bytecode-compile when >>> sys.dont_write_bytecode is True). >>> >>> >> >> I don't think my proposal will land into 2.6. The changes are too severe >> for a bug fix release. >> >> > > Right, certainly not adding umpteen new sys attributes. :-) > > The problem is that the alternative implementations run well behind > Python-trunk, indeed it doesn't really make sense for them to put a lot of > effort into implementing a version that is still in development. The result > is that they discover incompatibilites after a version has gone into 'bugfix > only' mode. > > Whilst the fix you have described (add information to sys that is used by > site.py and distutils) is ideal it can only go into 2.7. I would *still* > like to see a fix in 2.6 - even if it is simple logic in site.py using > sys.platform (if sys.platform == 'cli'; elif sys.platform == 'java' etc). > That way IronPython 2.6 is able to be compatible with Python 2.6. This logic > might need duplicating in distutils (I haven't looked at how distutils works > out where the user site-packages folder is), but it is a 'maintenance only' > fix. > But it's still a change in semantics. Tossing this into 2.6 would mean that anyone who has worked around the current behaviour is going to have a busted install. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Oct 9 20:13:16 2009 From: brett at python.org (Brett Cannon) Date: Fri, 9 Oct 2009 11:13:16 -0700 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> <94bdd2610910090823x77df06cdm5ed484c30feccced@mail.gmail.com> <4ACF63EE.9040001@simplistix.co.uk> Message-ID: On Fri, Oct 9, 2009 at 10:57, Antoine Pitrou wrote: > Chris Withers simplistix.co.uk> writes: > > > > *sigh* I don't see it as hijacking, provided Guido is making a BDFL > > pronouncement that you're maintaining this software > > Well, what you are proposing *is* hijacking. PJE is not paid or employed by > Guido, he is the full author of setuptools. Forking is of course fine (this > is > free software), but saying "I overthrow you as the maintainer of this > software" > is not. > > While you (and I) may be unsatisfied with PJE's maintenance style, personal > (dis)affection should not be what motivates our decisions. A piece of > software > has an author who has some moral rights on it, and should be treated with > respect. After all, when an author decides to release code under a free > license, > it shows a fair amount of trust and respect towards the user; it is only > fair > that the user shows the same amount of trust and respect (*). > FWIW I agree with Antoine. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Fri Oct 9 20:23:22 2009 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 9 Oct 2009 20:23:22 +0200 Subject: [Python-Dev] Bits-of-Distribute naming In-Reply-To: <4ACF59CC.5050504@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF59CC.5050504@simplistix.co.uk> Message-ID: <319e029f0910091123n95002cama014f9720510100a@mail.gmail.com> 2009/10/9 Chris Withers : > Why not get it into the core as distutils.entrypoints? That's where it > belongs... Well, one reason is that it can't be distributed separately under that name, and hence not be tested separately or bugfixed separately. Rule #1 for inclusion in the standard library is that the module pretty much is dead, and that you don't need new releases or new features. This isn't true for any part of a refactored setuptools. We will need to distribute most of this separately for a while first. That means that distutils.anything doesn't work. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From fuzzyman at voidspace.org.uk Fri Oct 9 20:32:05 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 9 Oct 2009 19:32:05 +0100 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> Message-ID: <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> The *only* change in semantics I'm proposing is for users of IronPython 2.6 which is not even at final release yet. CPython users would be unaffected. Sorry for top-posting, mobile device. Michael -- http://www.ironpythoninaction.com On 9 Oct 2009, at 19:00, Brett Cannon wrote: > > > On Fri, Oct 9, 2009 at 04:53, Michael Foord > wrote: > Christian Heimes wrote: > Michael Foord wrote: > > I really like this scheme. The important thing for IronPython is > that we can get it into Python 2.6 (along with other fixes to make > distutils compatible with IronPython - like not attempting to > bytecode-compile when sys.dont_write_bytecode is True). > > > I don't think my proposal will land into 2.6. The changes are too > severe > for a bug fix release. > > > Right, certainly not adding umpteen new sys attributes. :-) > > The problem is that the alternative implementations run well behind > Python-trunk, indeed it doesn't really make sense for them to put a > lot of effort into implementing a version that is still in > development. The result is that they discover incompatibilites after > a version has gone into 'bugfix only' mode. > > Whilst the fix you have described (add information to sys that is > used by site.py and distutils) is ideal it can only go into 2.7. I > would *still* like to see a fix in 2.6 - even if it is simple logic > in site.py using sys.platform (if sys.platform == 'cli'; elif > sys.platform == 'java' etc). That way IronPython 2.6 is able to be > compatible with Python 2.6. This logic might need duplicating in > distutils (I haven't looked at how distutils works out where the > user site-packages folder is), but it is a 'maintenance only' fix. > > But it's still a change in semantics. Tossing this into 2.6 would > mean that anyone who has worked around the current behaviour is > going to have a busted install. > > -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Oct 9 20:43:06 2009 From: brett at python.org (Brett Cannon) Date: Fri, 9 Oct 2009 11:43:06 -0700 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> Message-ID: On Fri, Oct 9, 2009 at 11:32, Michael Foord wrote: > The *only* change in semantics I'm proposing is for users of IronPython 2.6 > which is not even at final release yet. CPython users would be unaffected. > > Then why can't IronPython patch site.py to do what they want? I still feel uncomfortable changing site.py for this in a micro release. > Sorry for top-posting, mobile device. > > Aahz was the most adamant hater of top-posting and he isn't subscribed to python-dev anymore, so whatever. -Brett > Michael > > > -- > http://www.ironpythoninaction.com > > On 9 Oct 2009, at 19:00, Brett Cannon wrote: > > > > On Fri, Oct 9, 2009 at 04:53, Michael Foord < > fuzzyman at voidspace.org.uk> wrote: > >> Christian Heimes wrote: >> >>> Michael Foord wrote: >>> >>> >>>> I really like this scheme. The important thing for IronPython is that we >>>> can get it into Python 2.6 (along with other fixes to make distutils >>>> compatible with IronPython - like not attempting to bytecode-compile when >>>> sys.dont_write_bytecode is True). >>>> >>>> >>> >>> I don't think my proposal will land into 2.6. The changes are too severe >>> for a bug fix release. >>> >>> >> >> Right, certainly not adding umpteen new sys attributes. :-) >> >> The problem is that the alternative implementations run well behind >> Python-trunk, indeed it doesn't really make sense for them to put a lot of >> effort into implementing a version that is still in development. The result >> is that they discover incompatibilites after a version has gone into 'bugfix >> only' mode. >> >> Whilst the fix you have described (add information to sys that is used by >> site.py and distutils) is ideal it can only go into 2.7. I would *still* >> like to see a fix in 2.6 - even if it is simple logic in site.py using >> sys.platform (if sys.platform == 'cli'; elif sys.platform == 'java' etc). >> That way IronPython 2.6 is able to be compatible with Python 2.6. This logic >> might need duplicating in distutils (I haven't looked at how distutils works >> out where the user site-packages folder is), but it is a 'maintenance only' >> fix. >> > > But it's still a change in semantics. Tossing this into 2.6 would mean that > anyone who has worked around the current behaviour is going to have a busted > install. > > -Brett > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fwierzbicki at gmail.com Fri Oct 9 20:48:55 2009 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Fri, 9 Oct 2009 14:48:55 -0400 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4ACDCA5A.4090609@cheimes.de> References: <4ACDCA5A.4090609@cheimes.de> Message-ID: <4dab5f760910091148t33d732ccr2c529996bc3d07c7@mail.gmail.com> On Thu, Oct 8, 2009 at 7:17 AM, Christian Heimes wrote: > CPython: > ?~/.local/lib/python2.6/site-packages > ?%APPDATA%/Python/Python26 > > IronPython: > ?~/.local/lib/ironpython2.6/site-packages > ?%APPDATA%/Python/IronPython26 > > Jython: > ?~/.local/lib/jython2.6/site-packages > ?%APPDATA%/Python/Jython26 I like this too! Jython has yet to seriously start on 2.6, but when we do, would it help if Jython's sys.version grew a "Jython" in a similar place as IronPython? It would look something like: >>>sys.version '2.6.0 (Jython trunk:6849M, Oct 9 2009, 14:19:04) \n[Java HotSpot(TM) Client VM (Apple Inc.)]' -Frank From a.badger at gmail.com Fri Oct 9 21:20:01 2009 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 9 Oct 2009 12:20:01 -0700 Subject: [Python-Dev] supporting multiple versions of one package in one environment is insane In-Reply-To: <4ACF64E8.8010501@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF5BE4.70907@simplistix.co.uk> <20091009162226.GA5033@clingman.lan> <4ACF64E8.8010501@simplistix.co.uk> Message-ID: <20091009192001.GC5033@clingman.lan> On Fri, Oct 09, 2009 at 05:29:28PM +0100, Chris Withers wrote: > Toshio Kuratomi wrote: >> On Fri, Oct 09, 2009 at 04:51:00PM +0100, Chris Withers wrote: >>> Tarek Ziad? wrote: >>>> Virtualenv allows you to create an isolated environment to install >>>> some distribution without polluting the >>>> main site-packages, a bit like a user site-packages. >>> ...as does buildout, and these are the right type of solution to >>> this problem. >>> >> where type of problem == sandboxed environment, sure. How do you solve the >> problem for system packagers? > > What's to stop a system packager either just running the buildout on > install, or running the buildout at package build time and then just > dropping the resulting environment wherever they want to install > applications? Such a package would only be dependent on the right python > version at runtime... > If buildout creates sandboxed environments like virtualenv then everything here applies: https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries You can also listen/watch the talk I gave at PyCon which is linked from that page: http://pycon.blip.tv/file/2072580/ If it doesn't create sandboxed environments, then you'll need to give me about a paragraph explaining what it does do. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From g.brandl at gmx.net Fri Oct 9 22:36:48 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 09 Oct 2009 22:36:48 +0200 Subject: [Python-Dev] no consensus on static metadata In-Reply-To: <4ACF6208.4030506@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACF5775.7080507@simplistix.co.uk> <94bdd2610910090845k56f89f7fk2fe0a834f3172a99@mail.gmail.com> <4ACF6208.4030506@simplistix.co.uk> Message-ID: Chris Withers schrieb: > Tarek Ziad? wrote: >>> NB: There was no consensus on this "micro-language" on distutils-sig. >>> While I suspect I don't care as none of my packages rely on anything other >>> than other python packages, others did care, and I found the syntax Tarek >>> was proposing pretty clumsy. >> >> Please no bikeshedding again on the syntax. > > Tarek, throwing "bikeshedding" at an important syntactical discussion > like this is verging on rude. Please stop it and accept that the best > solution has not been found yet. At some point a decision must be made, and it will be made by Tarek as the "main packaging guy" (don't want to throw "dictator" around again). For syntax, there often is no such thing as "the best solution". Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Fri Oct 9 22:50:06 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 09 Oct 2009 22:50:06 +0200 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: <4ACF5050.1060701@simplistix.co.uk> References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <94bdd2610909230520y5e4d59by7f192137980b7613@mail.gmail.com> <87tyyt4v1m.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610909231000t4bbeb5b4x3ed715dad883414a@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> Message-ID: Chris Withers schrieb: > Guido van Rossum wrote: >> I'm saying that I don't expect setuptools 0.7 to appear before Tarek's >> Distribute is mature and in widespread use. IOW I support Tarek's fork >> and suggest nobody hold their breath waiting for setuptools 0.7. > > Well, if this was the BDFL pronouncement that a lot of people have taken > it to be, does that allow Martin von Lewis to give the keys to > http://pypi.python.org/pypi/setuptools to the "distribute" developers, > so we can get on and use the original "setuptools" name without all the > confusion and hackery needed to make "distribute" work? That's absurd. There's a certain area where Guido can make pronouncements, but third-party packages is not it. Even if they're hosted on python.org infrastructure. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From guido at python.org Fri Oct 9 22:56:42 2009 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Oct 2009 13:56:42 -0700 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> Message-ID: On Fri, Oct 9, 2009 at 1:50 PM, Georg Brandl wrote: > Chris Withers schrieb: >> Guido van Rossum wrote: >>> I'm saying that I don't expect setuptools 0.7 to appear before Tarek's >>> Distribute is mature and in widespread use. IOW I support Tarek's fork >>> and suggest nobody hold their breath waiting for setuptools 0.7. >> >> Well, if this was the BDFL pronouncement that a lot of people have taken >> it to be, does that allow Martin von Lewis to give the keys to >> http://pypi.python.org/pypi/setuptools to the "distribute" developers, >> so we can get on and use the original "setuptools" name without all the >> confusion and hackery needed to make "distribute" work? > > That's absurd. ?There's a certain area where Guido can make pronouncements, > but third-party packages is not it. ?Even if they're hosted on python.org > infrastructure. Right. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ctb at msu.edu Fri Oct 9 23:00:09 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 9 Oct 2009 14:00:09 -0700 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: References: <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> Message-ID: <20091009210009.GA15674@idyll.org> On Fri, Oct 09, 2009 at 01:56:42PM -0700, Guido van Rossum wrote: > On Fri, Oct 9, 2009 at 1:50 PM, Georg Brandl wrote: > > Chris Withers schrieb: > >> Guido van Rossum wrote: > >>> I'm saying that I don't expect setuptools 0.7 to appear before Tarek's > >>> Distribute is mature and in widespread use. IOW I support Tarek's fork > >>> and suggest nobody hold their breath waiting for setuptools 0.7. > >> > >> Well, if this was the BDFL pronouncement that a lot of people have taken > >> it to be, does that allow Martin von Lewis to give the keys to > >> http://pypi.python.org/pypi/setuptools to the "distribute" developers, > >> so we can get on and use the original "setuptools" name without all the > >> confusion and hackery needed to make "distribute" work? > > > > That's absurd. ?There's a certain area where Guido can make pronouncements, > > but third-party packages is not it. ?Even if they're hosted on python.org > > infrastructure. > > Right. Is that a pronouncement? :) "GvR, the self-limiting BDFL." --titus -- C. Titus Brown, ctb at msu.edu From ndbecker2 at gmail.com Sat Oct 10 00:45:34 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 09 Oct 2009 18:45:34 -0400 Subject: [Python-Dev] Howto handle multilib conflict? Message-ID: Just received: https://bugzilla.redhat.com/show_bug.cgi?id=528237 yum install libotf-devel.i586 libotf-devel.x86_64 yields: Transaction Check Error: file /usr/bin/libotf-config from install of libotf-devel-0.9.8-2.fc11.i586 conflicts with file from package libotf-devel-0.9.8-2.fc11.x86_64 file /usr/share/doc/libotf-devel-0.9.8/example/Makefile from install of libotf-devel-0.9.8-2.fc11.i586 conflicts with file from package libotf-devel-0.9.8-2.fc11.x86_64 What is the recommended way to resolve this? From ndbecker2 at gmail.com Sat Oct 10 00:56:08 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 09 Oct 2009 18:56:08 -0400 Subject: [Python-Dev] Howto handle multilib conflict? References: Message-ID: Sorry, sent to wrong list! Please ignore. From skip at pobox.com Sat Oct 10 01:07:12 2009 From: skip at pobox.com (skip at pobox.com) Date: Fri, 9 Oct 2009 18:07:12 -0500 Subject: [Python-Dev] BDFL pronouncement? In-Reply-To: References: <94bdd2610909220621u54984c94ucf522dbe1fe68671@mail.gmail.com> <20090923204737.A97DC3A4079@sparrow.telecommunity.com> <4AC371D7.9000904@simplistix.co.uk> <4ACB49FB.3000407@simplistix.co.uk> <20091006180400.BA69D3A4116@sparrow.telecommunity.com> <20091006200852.551BC3A4116@sparrow.telecommunity.com> <4ACF5050.1060701@simplistix.co.uk> Message-ID: <19151.49696.479121.865212@montanaro.dyndns.org> >> That's absurd. There's a certain area where Guido can make >> pronouncements, but third-party packages is not it. Even if they're >> hosted on python.org infrastructure. Guido> Right. Now if you were the un-BDFL you could step in here. ;-) This whole topic seems to have a lot of people fairly agitated, so clearly it's important to a significant subset of the Python development community. Might I suggest taking this off python-dev for awhile though? I seem to recall a similar suggestion a couple days ago to take it to distutils-sig or python-ideas, but that seems to have been ignored.) I mostly tuned the entire thread out until I saw that Guido had joined in the fray (sort of). Maybe since I don't distribute a lot of Python packages it's not as important to me. Let me know when you've solved the problem. I trust you. Skip From lists at cheimes.de Sat Oct 10 01:29:55 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Oct 2009 01:29:55 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory Message-ID: Fellow Python developers! In the light of our recent discussion about implementation specific information and user site directory I like to start a new PEP. Much to my regret I wasn't able to contribute to Python core development over the past months. I hope I can find some free time over the next weeks. Before I start with the PEP I like to reach consensus over the goal. Alternative implementations of our beloved programming languages are becoming more important every day. Although IronPython, Jython and PyPy are trying hard to get as compatible to CPython as possible there are numerous reasons for conditional code. General Python unit tests vs. CPython specific unit tests are one reasons, implementation specific module search paths are another. At the moment Python has no easy way to inspect the implementation besides some code in the platform module. I'm proposing two new attributes in the sys module: sys.implementation and sys.userdirsuffix. sys.implementation ------------------ (Proposed by Nick, MAL and others.) sys.implementation is a PyStructSequence that contains various information about the current implementation. Some fields are required to be present on every implementation. Implementations may choose to add additional fields as they see fit. Some fields like compiler are useful for introspection and are already part of sys.version. I like to include them for the sake of completeness. id (required): lower case identifier, for example "cpython", "ironpython", "jython", "pypy" name (required): mixed case name of the implementation, for example "CPython", "IronPython", "Jython", "PyPy" platform (required): platform or language of the implementation, for example "C", ".NET", "Java" runtime (required): lower case runtime library of the implementation, for example "libc", "msvcrt90", "java6", ".net" compiler (required): verbose name of the compiler, for example "GCC 4.3.3", "Java HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" ... sys.userdirsuffix ----------------- sys.userdirsuffix is an implementation and platform specific string that is used to construct the path for the user site directory as explained in PEP 370. The string contains the implementation name as well as the version number of Python. Examples: python2.6 (CPython, Unix) Python26 (CPython, Windows) ironpython2.6 (IronPython, Unix) IronPython26 (IronPython, Windows) ... Comments? Christian From dinov at microsoft.com Sat Oct 10 01:48:04 2009 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 9 Oct 2009 23:48:04 +0000 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> Christian wrote: > sys.implementation is a PyStructSequence that contains various > information about the current implementation. Some fields are required > to be present on every implementation. Implementations may choose to > add > additional fields as they see fit. Some fields like compiler are useful > for introspection and are already part of sys.version. I like to > include > them for the sake of completeness. Given that this is all about establishing some common ground between implementations I would propose not using a CPython specific term here such as PyStructSequence :) The Python docs seem to use structseq for sys.float_info. > compiler (required): > verbose name of the compiler, for example "GCC 4.3.3", "Java > HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" What's the value of this attribute? The main reason I ask is there's no way that I know of to determine the JIT being used in .NET. We could of course fill in some dummy value here for IronPython but I'm also not sure why anyone would use this. Otherwise it looks good to me. From benjamin at python.org Sat Oct 10 01:51:33 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 9 Oct 2009 18:51:33 -0500 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> 2009/10/9 Christian Heimes : > Fellow Python developers! > > In the light of our recent discussion about implementation specific > information and user site directory I like to start a new PEP. Much to > my regret I wasn't able to contribute to Python core development over > the past months. I hope I can find some free time over the next weeks. Glad to have you back! > > Before I start with the PEP I like to reach consensus over the goal. > Alternative implementations of our beloved programming languages are > becoming more important every day. Although IronPython, Jython and PyPy > are trying hard to get as compatible to CPython as possible there are > numerous reasons for conditional code. General Python unit tests vs. > CPython specific unit tests are one reasons, implementation specific > module search paths are another. At the moment Python has no easy way to > inspect the implementation besides some code in the platform module. > > I'm proposing two new attributes in the sys module: sys.implementation > and sys.userdirsuffix. Overall, +1. > > sys.implementation > ------------------ > > (Proposed by Nick, MAL and others.) > > sys.implementation is a PyStructSequence that contains various > information about the current implementation. Some fields are required > to be present on every implementation. Implementations may choose to add > additional fields as they see fit. Some fields like compiler are useful > for introspection and are already part of sys.version. I like to include > them for the sake of completeness. Can we generate sys.version from sys.implementation then? > > id (required): > ?lower case identifier, for example "cpython", "ironpython", "jython", > "pypy" > > name (required): > ?mixed case name of the implementation, for example "CPython", > "IronPython", "Jython", "PyPy" > > platform (required): > ?platform or language of the implementation, for example "C", ".NET", > "Java" > > runtime (required): > ?lower case runtime library of the implementation, for example "libc", > "msvcrt90", "java6", ".net" > > compiler (required): > ?verbose name of the compiler, for example "GCC 4.3.3", "Java > HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" > > ... > > > sys.userdirsuffix > ----------------- Why not site.userdirsuffix? -- Regards, Benjamin From lists at cheimes.de Sat Oct 10 02:24:44 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Oct 2009 02:24:44 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> Message-ID: <4ACFD44C.7040009@cheimes.de> Benjamin Peterson wrote: >> sys.implementation is a PyStructSequence that contains various >> information about the current implementation. Some fields are required >> to be present on every implementation. Implementations may choose to add >> additional fields as they see fit. Some fields like compiler are useful >> for introspection and are already part of sys.version. I like to include >> them for the sake of completeness. > > Can we generate sys.version from sys.implementation then? sys.version is created by Py_GetVersion() which uses Py_GetCompiler() for the compiler information: PyOS_snprintf(version, sizeof(version), "%.80s (%.80s) %.80s", PY_VERSION, Py_GetBuildInfo(), Py_GetCompiler()); For CPython the compiler attribute should be set to the return value of Py_GetCompiler(). >> sys.userdirsuffix >> ----------------- > > Why not site.userdirsuffix? Because all implementations of Python like to use the same, unpatched site module. The sys module is different for every implementation. It's more convenient to have an attribute on the sys module that can be filled by each implementation. I could also add a lookup table for all known implementations to the site module. But what about unknown or new implementations? They would have to wait until we add a new entry for them. The sys.userdirsuffix is more flexible and future prove. Christian From martin at v.loewis.de Sat Oct 10 02:29:18 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Oct 2009 02:29:18 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACEF266.3020309@egenix.com> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <30910449ef0aad44ac8d419c4e36896f@preisshare.net> <4ACEF266.3020309@egenix.com> Message-ID: <4ACFD55E.1090608@v.loewis.de> > I know that the issue tracker for PyPI is (still) on SF, but > development appear to happen elsewhere and I can't find any > contact information on the PyPI web pages. PyPI discussion takes place mostly on catalog-sig. Regards, Martin From benjamin at python.org Sat Oct 10 02:29:48 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 9 Oct 2009 19:29:48 -0500 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD44C.7040009@cheimes.de> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> Message-ID: <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> 2009/10/9 Christian Heimes : > Benjamin Peterson wrote: >>> sys.userdirsuffix >>> ----------------- >> >> Why not site.userdirsuffix? > > Because all implementations of Python like to use the same, unpatched > site module. The sys module is different for every implementation. It's > more convenient to have an attribute on the sys module that can be > filled by each implementation. I could also add a lookup table for all > known implementations to the site module. But what about unknown or new > implementations? They would have to wait until we add a new entry for > them. The sys.userdirsuffix is more flexible and future prove. I think we should make a semi-private (public to the stdlib) module like _sys or _implementation part of the Python VM API. Then, instead of stuffing everything into sys, we can provide this information in modules where it belongs. -- Regards, Benjamin From martin at v.loewis.de Sat Oct 10 02:34:08 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Oct 2009 02:34:08 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> References: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> Message-ID: <4ACFD680.1080001@v.loewis.de> > Given that this is all about establishing some common ground between > implementations I would propose not using a CPython specific term > here such as PyStructSequence :) The Python docs seem to use structseq > for sys.float_info. Also, why does it have to be a sequence in the first place? Wouldn't a plain object with named attributes be good enough? >> compiler (required): >> verbose name of the compiler, for example "GCC 4.3.3", "Java >> HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" > > What's the value of this attribute? The main reason I ask is there's > no way that I know of to determine the JIT being used in .NET. We could > of course fill in some dummy value here for IronPython but I'm also not > sure why anyone would use this. Also, why is it the name of the JIT compiler, and not the name of the source language compiler? Regards, Martin From martin at v.loewis.de Sat Oct 10 02:42:23 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Oct 2009 02:42:23 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <4ACFD86F.3010305@v.loewis.de> > sys.implementation is a PyStructSequence that contains various > information about the current implementation. Some fields are required > to be present on every implementation. I think it is important to confirm in advance that all the implementations listed below agree to implement the PEP "soonish" after it's adopted. "Required" sounds like a strong term - however, if an implementation chooses not to implement the PEP, it can do whatever it wants, including omission of required fields. > id (required): > lower case identifier, for example "cpython", "ironpython", "jython", > "pypy" Doing some bike-shedding: I'd like to not use "cpython" as the name of the python.org implementation. This term, I believe, was coined around JPython, somehow making the implementation language the primary means of distinction. However, there may be alternative implementations written in C (e.g. unladen swallow), or otherwise be "C based" (e.g. pypy). So I propose that the python.org version is identified as "python". > name (required): > mixed case name of the implementation, for example "CPython", > "IronPython", "Jython", "PyPy" Likewise; alternatively "python.org". Regards, Martin From lists at cheimes.de Sat Oct 10 02:44:01 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Oct 2009 02:44:01 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD680.1080001@v.loewis.de> References: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> <4ACFD680.1080001@v.loewis.de> Message-ID: <4ACFD8D1.6050302@cheimes.de> Martin v. L?wis wrote: >> Given that this is all about establishing some common ground between >> implementations I would propose not using a CPython specific term >> here such as PyStructSequence :) The Python docs seem to use structseq >> for sys.float_info. > > Also, why does it have to be a sequence in the first place? Wouldn't > a plain object with named attributes be good enough? Any object with attributes is good enough. For CPython a structseq provides the necessary feature of a read only object with attributes. I figured out it's the easiest way to archive the goal without creating a new class. > >>> compiler (required): >>> verbose name of the compiler, for example "GCC 4.3.3", "Java >>> HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" >> What's the value of this attribute? The main reason I ask is there's >> no way that I know of to determine the JIT being used in .NET. We could >> of course fill in some dummy value here for IronPython but I'm also not >> sure why anyone would use this. > > Also, why is it the name of the JIT compiler, and not the name of the > source language compiler? The term "source language compiler" describes the intent of the field perfectly. Thanks Martin! I was merely guessing what the compiler name may look like on Jython. Christian From greg.ewing at canterbury.ac.nz Sat Oct 10 02:57:06 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 10 Oct 2009 13:57:06 +1300 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD86F.3010305@v.loewis.de> References: <4ACFD86F.3010305@v.loewis.de> Message-ID: <4ACFDBE2.207@canterbury.ac.nz> Martin v. L?wis wrote: > So I propose that the python.org version is identified as "python". But then we won't have a generic term for the language itself, independent of any implementation. The "c" in cpython doesn't necessarily have to refer to the implementation language. Maybe it stands for "classic python". :-) -- Greg From lists at cheimes.de Sat Oct 10 02:58:42 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Oct 2009 02:58:42 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD86F.3010305@v.loewis.de> References: <4ACFD86F.3010305@v.loewis.de> Message-ID: <4ACFDC42.7010306@cheimes.de> Martin v. L?wis wrote: >> sys.implementation is a PyStructSequence that contains various >> information about the current implementation. Some fields are required >> to be present on every implementation. > > I think it is important to confirm in advance that all the > implementations listed below agree to implement the PEP "soonish" after > it's adopted. "Required" sounds like a strong term - however, if an > implementation chooses not to implement the PEP, it can do whatever it > wants, including omission of required fields. Please let me rephrase my suggestion. If an implementation wants implemented the PEP it is required to provide a well defined set of attributes. Of course I'm open for any suggestions regarding names, specification and content of the attribues. > Doing some bike-shedding: I'd like to not use "cpython" as the name of > the python.org implementation. This term, I believe, was coined around > JPython, somehow making the implementation language the primary means > of distinction. However, there may be alternative implementations > written in C (e.g. unladen swallow), or otherwise be "C based" (e.g. > pypy). > > So I propose that the python.org version is identified as "python". CPython is a well established term that is widely used to distinguish between multiple implementations of Python as a language. On the one hand the python.org implementation was the first one. It founded the language and made it popular. On the other hand it may look arrogant from the perspective of the other implementors if we choose to use "python" as the identifier for the python.org implementation. In my opinion we should let Guido decide about the identifier before we have another bike-shedding war. ;-) >> name (required): >> mixed case name of the implementation, for example "CPython", >> "IronPython", "Jython", "PyPy" > > Likewise; alternatively "python.org". Interesting suggestion. Guido? Christian From lists at cheimes.de Sat Oct 10 03:01:12 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Oct 2009 03:01:12 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> References: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> Message-ID: <4ACFDCD8.9050703@cheimes.de> Dino Viehland wrote: > Given that this is all about establishing some common ground between > implementations I would propose not using a CPython specific term > here such as PyStructSequence :) The Python docs seem to use structseq > for sys.float_info. You got me! :) >> compiler (required): >> verbose name of the compiler, for example "GCC 4.3.3", "Java >> HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" > > What's the value of this attribute? The main reason I ask is there's > no way that I know of to determine the JIT being used in .NET. We could > of course fill in some dummy value here for IronPython but I'm also not > sure why anyone would use this. Martin has already answered both points to my satisfaction. Do you agree with him? Christian From dinov at microsoft.com Sat Oct 10 03:05:16 2009 From: dinov at microsoft.com (Dino Viehland) Date: Sat, 10 Oct 2009 01:05:16 +0000 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFDCD8.9050703@cheimes.de> References: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> <4ACFDCD8.9050703@cheimes.de> Message-ID: <1A472770E042064698CB5ADC83A12ACD04A9E9AF@TK5EX14MBXC116.redmond.corp.microsoft.com> Christian wrote: > > Martin has already answered both points to my satisfaction. Do you > agree with him? Sure, source level makes more sense - so we'd have "csc" or "gmcs" if compiled with Mono (assuming there's some way to do that...). From lists at cheimes.de Sat Oct 10 03:07:40 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 10 Oct 2009 03:07:40 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> Message-ID: <4ACFDE5C.7080008@cheimes.de> Benjamin Peterson wrote: > I think we should make a semi-private (public to the stdlib) module > like _sys or _implementation part of the Python VM API. Then, instead > of stuffing everything into sys, we can provide this information in > modules where it belongs. That's an interesting counter proposal. Your idea requires an additional import that I try to avoid. Looking at memory and performance, an additional module that is imported anyway isn't better. In my humble opinion the implementation information belongs into the sys module anyway. A new module just for the user site suffix seems unnecessary. Christian From benjamin at python.org Sat Oct 10 03:14:48 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 9 Oct 2009 20:14:48 -0500 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFDE5C.7080008@cheimes.de> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> <4ACFDE5C.7080008@cheimes.de> Message-ID: <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> 2009/10/9 Christian Heimes : > Benjamin Peterson wrote: >> I think we should make a semi-private (public to the stdlib) module >> like _sys or _implementation part of the Python VM API. Then, instead >> of stuffing everything into sys, we can provide this information in >> modules where it belongs. > > > That's an interesting counter proposal. Your idea requires an additional > import that I try to avoid. Looking at memory and performance, an > additional module that is imported anyway isn't better. In my humble > opinion the implementation information belongs into the sys module > anyway. A new module just for the user site suffix seems unnecessary. But we want to hide that this is an implementation detail from the user. Having a new module just for this attribute might seem like overkill, but I hope that we could use it for more things in the future. Besides, if _sys is a builtin module, importing it will not add much overhead. I forgot to ask before: Does this deprecate platform.python_implementation()? -- Regards, Benjamin From brett at python.org Sat Oct 10 04:35:48 2009 From: brett at python.org (Brett Cannon) Date: Fri, 9 Oct 2009 19:35:48 -0700 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> <4ACFDE5C.7080008@cheimes.de> <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> Message-ID: On Fri, Oct 9, 2009 at 18:14, Benjamin Peterson wrote: > 2009/10/9 Christian Heimes : > > Benjamin Peterson wrote: > >> I think we should make a semi-private (public to the stdlib) module > >> like _sys or _implementation part of the Python VM API. Then, instead > >> of stuffing everything into sys, we can provide this information in > >> modules where it belongs. > > > > > > That's an interesting counter proposal. Your idea requires an additional > > import that I try to avoid. Looking at memory and performance, an > > additional module that is imported anyway isn't better. In my humble > > opinion the implementation information belongs into the sys module > > anyway. A new module just for the user site suffix seems unnecessary. > > But we want to hide that this is an implementation detail from the > user. Having a new module just for this attribute might seem like > overkill, but I hope that we could use it for more things in the > future. To also address Christian's import worry, this new module could contain everything needed to start the interpreter, much like _warnings does compared to warnings. But I honestly don't see why this doesn't belong in sys; it has to do with the system environment which is the interpreter. Yes, some things currently in sys could possibly be put elsewhere (e.g. the exception stuff could be in 'exceptions'), but I don't think sys is that much of an odd collection of code. Without having a desire to eventually clean up the sys module (which I am not proposing as I know that will never fly) I don't see this as being worth it. > Besides, if _sys is a builtin module, importing it will not > add much overhead. > Well, that's assuming the other interpreters makes it built-in. Otherwise it's only a CPython/python.org perk. > > I forgot to ask before: Does this deprecate > platform.python_implementation()? Probably should. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Oct 10 04:49:02 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 9 Oct 2009 21:49:02 -0500 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> <4ACFDE5C.7080008@cheimes.de> <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> Message-ID: <1afaf6160910091949y48da1289wcbdd186404b66853@mail.gmail.com> 2009/10/9 Brett Cannon : > But I honestly don't see why this doesn't belong in sys; it has to do with > the system environment which is the interpreter. Yes, some things currently > in sys could possibly be put elsewhere (e.g. the exception stuff could be in > 'exceptions'), but I don't think sys is that much of an odd collection of > code. Without having a desire to eventually clean up the sys module (which I > am not proposing as I know that will never fly) I don't see this as being > worth it. That doesn't mean we have to make it more dirty. usersiteprefix is used exclusively by site.py to load user modules; it has no function in the interpreter core. >> Besides, if _sys is a builtin module, importing it will not >> add much overhead. > > Well, that's assuming the other interpreters makes it built-in. Otherwise > it's only a CPython/python.org perk. As far as I am aware, all current Python implementations can build in modules. -- Regards, Benjamin From metawilm at gmail.com Sat Oct 10 09:32:42 2009 From: metawilm at gmail.com (Willem Broekema) Date: Sat, 10 Oct 2009 09:32:42 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: On Sat, Oct 10, 2009 at 1:29 AM, Christian Heimes wrote: > I'm proposing two new attributes in the sys module: sys.implementation > and sys.userdirsuffix. This seems like a good idea. I'm not sure this idea will easily be accepted, but I'd like to see the sys module eventually split up in two parts, so it is very obvious to both implementers and users which system-specific features are portable and which are not: a) implementation details of the C implementation (subversion, _*, dllhandle, dont_write_bytecode, settscdump, ..) in one module, b) portable functionality (interpreter name and version etc, builtin_module_names, copyright, excepthook, settrace, ..) in the other - Willem (CLPython) From turnbull at sk.tsukuba.ac.jp Sat Oct 10 10:25:31 2009 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 10 Oct 2009 17:25:31 +0900 Subject: [Python-Dev] Top-posting on this list In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> Message-ID: <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> Brett Cannon writes: > On Fri, Oct 9, 2009 at 11:32, Michael Foord wrote: > > Sorry for top-posting, mobile device. Michael, I for one *very much* appreciate your courtesy. > Aahz was the most adamant hater of top-posting and he isn't subscribed to > python-dev anymore, so whatever. Please, let's encourage people to indicate if they top-post. In the modern environment bandwidth is not really a consideration, but I *do* read to the bottom on lists like this one, where the *great* majority of posters do a good job of trimming. From mal at egenix.com Sat Oct 10 12:27:26 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 10 Oct 2009 12:27:26 +0200 Subject: [Python-Dev] eggs now mandatory for pypi? In-Reply-To: <4ACFD55E.1090608@v.loewis.de> References: <20091005102547.GA23457@ubuntu.ubuntu-domain> <4ACA0CBA.2030200@egenix.com> <4ACB3FC0.1010906@simplistix.co.uk> <4ACB439E.6050102@egenix.com> <4ACB45FA.3080707@simplistix.co.uk> <4ACB5690.1080208@egenix.com> <4ACB6619.4080807@simplistix.co.uk> <4ACD965D.5080701@egenix.com> <30910449ef0aad44ac8d419c4e36896f@preisshare.net> <4ACEF266.3020309@egenix.com> <4ACFD55E.1090608@v.loewis.de> Message-ID: <4AD0618E.6050406@egenix.com> "Martin v. L?wis" wrote: >> I know that the issue tracker for PyPI is (still) on SF, but >> development appear to happen elsewhere and I can't find any >> contact information on the PyPI web pages. > > PyPI discussion takes place mostly on catalog-sig. Ok, then I'll sign up there. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 10 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Sat Oct 10 12:42:19 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 10 Oct 2009 12:42:19 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> <4ACFDE5C.7080008@cheimes.de> <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> Message-ID: <4AD0650B.7080600@egenix.com> Benjamin Peterson wrote: > I forgot to ask before: Does this deprecate platform.python_implementation()? No, platform.py is meant to be portable across multiple Python versions and as such not really suitable for such deprecations. It'll also take a long while before all Python implementations expose a new sys module attribute along the proposed lines. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 10 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Sat Oct 10 13:36:29 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Oct 2009 11:36:29 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_about_sys=2Eimplementation_and_impleme?= =?utf-8?q?ntation=09specific_user_site_directory?= References: Message-ID: Willem Broekema gmail.com> writes: > > a) implementation details of the C implementation (subversion, _*, > dllhandle, dont_write_bytecode, settscdump, ..) in one module, > > b) portable functionality (interpreter name and version etc, > builtin_module_names, copyright, excepthook, settrace, ..) in the > other Where does _getframe() end? :) Regards Antoine. From benjamin at python.org Sat Oct 10 17:11:52 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 10 Oct 2009 10:11:52 -0500 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <1afaf6160910100811p3a79e8a6gf4d6d3a077eb3895@mail.gmail.com> 2009/10/10 Willem Broekema : > On Sat, Oct 10, 2009 at 1:29 AM, Christian Heimes wrote: >> I'm proposing two new attributes in the sys module: sys.implementation >> and sys.userdirsuffix. > > This seems like a good idea. > > I'm not sure this idea will easily be accepted, but I'd like to see > the sys module eventually split up in two parts, so it is very obvious > to both implementers and users which system-specific features are > portable and which are not: I'm afraid this has already been proposed and rejected. See http://www.python.org/dev/peps/pep-3139/. -- Regards, Benjamin From ncoghlan at gmail.com Sat Oct 10 17:41:27 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2009 01:41:27 +1000 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4AD0650B.7080600@egenix.com> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> <4ACFDE5C.7080008@cheimes.de> <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> <4AD0650B.7080600@egenix.com> Message-ID: <4AD0AB27.1040109@gmail.com> M.-A. Lemburg wrote: > Benjamin Peterson wrote: >> I forgot to ask before: Does this deprecate platform.python_implementation()? > > No, platform.py is meant to be portable across multiple Python > versions and as such not really suitable for such deprecations. > > It'll also take a long while before all Python implementations > expose a new sys module attribute along the proposed lines. However, the function could be updated to take advantage of the new sys.implementation attribute when it was available. If it didn't find it, it would fallback to figuring out the way it does now. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Oct 10 17:46:55 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2009 01:46:55 +1000 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD86F.3010305@v.loewis.de> References: <4ACFD86F.3010305@v.loewis.de> Message-ID: <4AD0AC6F.4000908@gmail.com> Martin v. L?wis wrote: > Doing some bike-shedding: I'd like to not use "cpython" as the name of > the python.org implementation. This term, I believe, was coined around > JPython, somehow making the implementation language the primary means > of distinction. However, there may be alternative implementations > written in C (e.g. unladen swallow), or otherwise be "C based" (e.g. > pypy). However, CPython has been used for years to distinguish CPython-the-reference-implementation from Python-the-language-definition. Starting to call it something else now would just confuse people for no good reason. As Greg said, it may be better to think of it as standing for "Classic Python" rather than merely "A Python interpreter written in C". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Sat Oct 10 17:48:58 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 10 Oct 2009 16:48:58 +0100 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <4AD0ACEA.2050506@voidspace.org.uk> Christian Heimes wrote: > Fellow Python developers! > > In the light of our recent discussion about implementation specific > information and user site directory I like to start a new PEP. Much to > my regret I wasn't able to contribute to Python core development over > the past months. I hope I can find some free time over the next weeks. > > Before I start with the PEP I like to reach consensus over the goal. > Alternative implementations of our beloved programming languages are > becoming more important every day. Although IronPython, Jython and PyPy > are trying hard to get as compatible to CPython as possible there are > numerous reasons for conditional code. General Python unit tests vs. > CPython specific unit tests are one reasons, implementation specific > module search paths are another. At the moment Python has no easy way to > inspect the implementation besides some code in the platform module. > > I'm proposing two new attributes in the sys module: sys.implementation > and sys.userdirsuffix. > Why not just sys.implementation as a string? Everything else can trivially be deduced from that anyway. What is the use-case for the extra information? Michael > sys.implementation > ------------------ > > (Proposed by Nick, MAL and others.) > > sys.implementation is a PyStructSequence that contains various > information about the current implementation. Some fields are required > to be present on every implementation. Implementations may choose to add > additional fields as they see fit. Some fields like compiler are useful > for introspection and are already part of sys.version. I like to include > them for the sake of completeness. > > id (required): > lower case identifier, for example "cpython", "ironpython", "jython", > "pypy" > > name (required): > mixed case name of the implementation, for example "CPython", > "IronPython", "Jython", "PyPy" > > platform (required): > platform or language of the implementation, for example "C", ".NET", > "Java" > > runtime (required): > lower case runtime library of the implementation, for example "libc", > "msvcrt90", "java6", ".net" > > compiler (required): > verbose name of the compiler, for example "GCC 4.3.3", "Java > HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" > > ... > > > sys.userdirsuffix > ----------------- > > sys.userdirsuffix is an implementation and platform specific string that > is used to construct the path for the user site directory as explained > in PEP 370. The string contains the implementation name as well as the > version number of Python. > > Examples: > python2.6 (CPython, Unix) > Python26 (CPython, Windows) > ironpython2.6 (IronPython, Unix) > IronPython26 (IronPython, Windows) > ... > > > Comments? > > Christian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From ncoghlan at gmail.com Sat Oct 10 18:05:27 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2009 02:05:27 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> References: <4ACDF621.4070603@trueblade.com> <4ACDFD59.4060200@trueblade.com> <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> Message-ID: <4AD0B0C7.3040700@gmail.com> Benjamin Peterson wrote: > 2009/10/8 Eric Smith : >> Christian Tanzer wrote: >>> IMHO, either the translation is done once and gives identical output or >>> it isn't worth doing at all. >> I disagree. I doubt even 0.001% of all format strings involve octal >> formatting. Is it really worth not providing a transition path if it can't >> cover this case? > > It's also really easy to just write 0{:o} like my translator does. That works so long as the original format string doesn't specify either a space padded field width or else a sign character. For those the extra zero needs to be inserted after the leading characters but before the number, so the formatting engine really has to handle it. I'm actually thinking that having the ability to specify a single 0 as the leading character for octal output is a legitimate feature. There are plenty of other tools out there that use a single leading zero to denote octal numbers (e.g. think of a Python script that generated C code), so having Python be able to produce such numbers makes a lot of sense. Vinay's suggestion of using 'O' instead of 'o' to denote C-style octal formatting instead of Python-style sounds reasonable to me (similar in degree to the upper vs lower case distinction for 'x' and 'X' hex formatting). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Oct 10 18:06:59 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2009 02:06:59 +1000 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACE7209.9060208@g.nevcal.com> References: <837DB8E29A4E4FAEABBA764EC89F3692@RaymondLaptop1> <4ACDD2C3.4090608@gmail.com> <4ACDF621.4070603@trueblade.com> <4ACE7209.9060208@g.nevcal.com> Message-ID: <4AD0B123.40802@gmail.com> Glenn Linderman wrote: > If you want to replace a C program that produces parsed output in a > given format, and that given format includes leading-0-octal numbers, > then it would be good to have the capability in Python .format, even > though Python itself uses 0o prefix. > > Similar arguments may apply anywhere else that sprintf produces > something that .format cannot currently produce. Heh, I should have read this before making my last comment - this is exactly what I was trying to get at :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric at trueblade.com Sat Oct 10 18:09:33 2009 From: eric at trueblade.com (Eric Smith) Date: Sat, 10 Oct 2009 12:09:33 -0400 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4AD0B0C7.3040700@gmail.com> References: <4ACDF621.4070603@trueblade.com> <4ACDFD59.4060200@trueblade.com> <1afaf6160910080807w354506e0ifa2e0df2dec89568@mail.gmail.com> <4AD0B0C7.3040700@gmail.com> Message-ID: <4AD0B1BD.10002@trueblade.com> Nick Coghlan wrote: > Benjamin Peterson wrote: >> 2009/10/8 Eric Smith : >>> Christian Tanzer wrote: >>>> IMHO, either the translation is done once and gives identical output or >>>> it isn't worth doing at all. >>> I disagree. I doubt even 0.001% of all format strings involve octal >>> formatting. Is it really worth not providing a transition path if it can't >>> cover this case? >> It's also really easy to just write 0{:o} like my translator does. > > That works so long as the original format string doesn't specify either > a space padded field width or else a sign character. For those the extra > zero needs to be inserted after the leading characters but before the > number, so the formatting engine really has to handle it. > > I'm actually thinking that having the ability to specify a single 0 as > the leading character for octal output is a legitimate feature. There > are plenty of other tools out there that use a single leading zero to > denote octal numbers (e.g. think of a Python script that generated C > code), so having Python be able to produce such numbers makes a lot of > sense. > > Vinay's suggestion of using 'O' instead of 'o' to denote C-style octal > formatting instead of Python-style sounds reasonable to me (similar in > degree to the upper vs lower case distinction for 'x' and 'X' hex > formatting). Mark points out in http://bugs.python.org/issue7094 that we'd also need to add alternate float formatting for any automated translation facility to work flawlessly. There might be other float issues involving trailing decimals with no zeros that work differently, too. Eric. From ncoghlan at gmail.com Sat Oct 10 18:23:43 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2009 02:23:43 +1000 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4AD0ACEA.2050506@voidspace.org.uk> References: <4AD0ACEA.2050506@voidspace.org.uk> Message-ID: <4AD0B50F.1080008@gmail.com> Michael Foord wrote: > Why not just sys.implementation as a string? Everything else can > trivially be deduced from that anyway. What is the use-case for the > extra information? I think Christian's set of required attributes is much too large (I would only have "name" as a required field), but I can understand the desire to use a structure rather than a simple string. As with sys.float_info, it gives a new namespace to store "info about the implementation" without throwing more and more stuff into the main sys namespace. So it might start with just "name" being compulsory (since people can use lower() to iron out any capitalisation issues), but allow implementation specific attributes to migrate from the sys module to the implementation namespace. So "url" might become a common attribute, with each implementation providing a pointer to their homepage. CPython might decide to put our eventual Hg based replacement for sys.subversion under sys.implementation instead of at the module level, etc. A designated location for "put your implementation specific attributes here" inside the sys module is a lot less hassle than creating a whole new module for the purpose, but gives most of the same benefits. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Sat Oct 10 19:31:19 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 10 Oct 2009 18:31:19 +0100 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> Message-ID: <4AD0C4E7.4030307@voidspace.org.uk> Brett Cannon wrote: > > > On Fri, Oct 9, 2009 at 11:32, Michael Foord > wrote: > > The *only* change in semantics I'm proposing is for users of > IronPython 2.6 which is not even at final release yet. CPython > users would be unaffected. > > > Then why can't IronPython patch site.py to do what they want? I still > feel uncomfortable changing site.py for this in a micro release. The IronPython team currently have legal issues distributing modified versions of the standard library (Dino can correct me if I'm wrong here). That aside, I think that were a Python feature is implemented in such a way as to preclude compatibility with other implementations then we should see it as a bugfix to correct it. I can't see why you should feel uncomfortable (beyond general principle), it seems clear that this change can't affect users of ClassicPython. It is not only IronPython that will benefit but also Jython when they port to Python 2.6. As a general rule the alternative implementations will only start a serious port once a version is released and into maintenance mode. If we aren't able to fix compatibility issues then the implementations will always have to maintain their own patch sets that may or may not get merged back in (or in the case of implementations with awkward legal departments they'll have to ship bugs). All the best, Michael > > On 9 Oct 2009, at 19:00, Brett Cannon > wrote: > >> >> >> On Fri, Oct 9, 2009 at 04:53, Michael Foord >> > wrote: >> >> Christian Heimes wrote: >> >> Michael Foord wrote: >> >> >> I really like this scheme. The important thing for >> IronPython is that we can get it into Python 2.6 >> (along with other fixes to make distutils compatible >> with IronPython - like not attempting to >> bytecode-compile when sys.dont_write_bytecode is True). >> >> >> >> I don't think my proposal will land into 2.6. The changes >> are too severe >> for a bug fix release. >> >> >> >> Right, certainly not adding umpteen new sys attributes. :-) >> >> The problem is that the alternative implementations run well >> behind Python-trunk, indeed it doesn't really make sense for >> them to put a lot of effort into implementing a version that >> is still in development. The result is that they discover >> incompatibilites after a version has gone into 'bugfix only' >> mode. >> >> Whilst the fix you have described (add information to sys >> that is used by site.py and distutils) is ideal it can only >> go into 2.7. I would *still* like to see a fix in 2.6 - even >> if it is simple logic in site.py using sys.platform (if >> sys.platform == 'cli'; elif sys.platform == 'java' etc). That >> way IronPython 2.6 is able to be compatible with Python 2.6. >> This logic might need duplicating in distutils (I haven't >> looked at how distutils works out where the user >> site-packages folder is), but it is a 'maintenance only' fix. >> >> >> But it's still a change in semantics. Tossing this into 2.6 would >> mean that anyone who has worked around the current behaviour is >> going to have a busted install. >> >> -Brett > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From vinay_sajip at yahoo.co.uk Sat Oct 10 21:31:07 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 10 Oct 2009 19:31:07 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Initialization_of_=5F=5Fbuiltins=5F=5F?= Message-ID: >I'm not top-posting, but gmane is giving me a hard time :-( In the py3k branch, logging has the line _unicode = 'unicode' in dir(__builtins__) to determine the existence of Unicode support. The code in trunk, being 1.5.2 compatible, used hasattr(types, 'UnicodeType') I wanted to eliminate the types import and modernise the trunk code a bit, so I copied the py3k line to the trunk version of logging. It didn't work as expected! So I added the line print dir(__builtins__) in logging fairly early on (though not the very first line - just after the __date__ = ... line. Here's what I got with 2.7a0 and 2.6.1: Python 2.7a0 (trunk:75292M, Oct 9 2009, 09:21:05) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import logging ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values'] >>> ActivePython 2.6.1.1 (ActiveState Software Inc.) based on Python 2.6.1 (r261:67515, Dec 5 2008, 13:58:38) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import logging ['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values'] >>> However, if I just do dir(__builtins__) in the interactive prompt, I get the whole shebang. Excuse my ignorance, but how come? Regards, Vinay Sajip From benjamin at python.org Sat Oct 10 21:38:32 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 10 Oct 2009 14:38:32 -0500 Subject: [Python-Dev] Initialization of __builtins__ In-Reply-To: References: Message-ID: <1afaf6160910101238m7189342fib18e036aa63ca6b@mail.gmail.com> 2009/10/10 Vinay Sajip : > Excuse my ignorance, but how come? Because __builtins__ is a module in __main__ and a dict in other places. You should always check __builtin__; __builtins__ is an implementation detail. -- Regards, Benjamin From solipsis at pitrou.net Sat Oct 10 21:40:34 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 Oct 2009 19:40:34 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Initialization_of_=5F=5Fbuiltins=5F=5F?= References: Message-ID: Vinay Sajip yahoo.co.uk> writes: > > In the py3k branch, logging has the line > > _unicode = 'unicode' in dir(__builtins__) Why do you do this? In py3k, unicode is always enabled but it's called "str" and the name "unicode" doesn't exist. > to determine the existence of Unicode support. The code in trunk, being 1.5.2 > compatible, used > > hasattr(types, 'UnicodeType') Why don't you simply write: unicode_support = True try: unicode except NameError: unicode_support = False ? From vinay_sajip at yahoo.co.uk Sat Oct 10 22:03:52 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 10 Oct 2009 20:03:52 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Initialization_of_=5F=5Fbuiltins=5F=5F?= References: Message-ID: Antoine Pitrou pitrou.net> writes: > Why do you do this? > In py3k, unicode is always enabled but it's called "str" and the name "unicode" > doesn't exist. > That wasn't done by me but by GvR (according to svn annotate) in r55818, presumably during the stdlib porting to py3k. I just copied the line over, not thinking it wouldn't work for 2.7. > Why don't you simply write: > > unicode_support = True > try: > unicode > except NameError: > unicode_support = False > That's just about what I've actually done: I was just curious about the difference between py3k and trunk :-) If __builtins__ is an implementation detail which can't be relied on, should the py3k code be changed to the try: form? Or shall I just remove the checks altogether, since Unicode should always be there in 3.x? Regards, Vinay Sajip From eric at trueblade.com Sat Oct 10 22:29:26 2009 From: eric at trueblade.com (Eric Smith) Date: Sat, 10 Oct 2009 16:29:26 -0400 Subject: [Python-Dev] Initialization of __builtins__ In-Reply-To: References: Message-ID: <4AD0EEA6.9090600@trueblade.com> Vinay Sajip wrote: > If __builtins__ is an implementation detail which can't be relied on, should the > py3k code be changed to the try: form? Or shall I just remove the checks > altogether, since Unicode should always be there in 3.x? Remember that the identifier "unicode" isn't present in py3k. There it's "str" and it holds Unicode strings. Unless you're trying to keep the code identical in both branches, I'd just remove the check in py3k and assume str is what you always want to use. Eric. From dinov at microsoft.com Sat Oct 10 23:06:34 2009 From: dinov at microsoft.com (Dino Viehland) Date: Sat, 10 Oct 2009 21:06:34 +0000 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4AD0C4E7.4030307@voidspace.org.uk> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <4AD0C4E7.4030307@voidspace.org.uk> Message-ID: <1A472770E042064698CB5ADC83A12ACD04AA4809@TK5EX14MBXC116.redmond.corp.microsoft.com> Michael wrote: > The IronPython team currently have legal issues distributing modified > versions of the standard library (Dino can correct me if I'm wrong > here). It's actually not due to legal issues although we'd have to check w/ the lawyers if we wanted to do it. It's mainly that we don't want to ship a modified version of the standard library because it seems like it should be standard. It's also a little more convenient for us to not have to maintain a separate copy (although we do already do that for the tests). From brett at python.org Sat Oct 10 23:17:21 2009 From: brett at python.org (Brett Cannon) Date: Sat, 10 Oct 2009 14:17:21 -0700 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: <4AD0C4E7.4030307@voidspace.org.uk> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <4AD0C4E7.4030307@voidspace.org.uk> Message-ID: On Sat, Oct 10, 2009 at 10:31, Michael Foord wrote: > Brett Cannon wrote: > >> >> >> On Fri, Oct 9, 2009 at 11:32, Michael Foord > fuzzyman at voidspace.org.uk>> wrote: >> >> The *only* change in semantics I'm proposing is for users of >> IronPython 2.6 which is not even at final release yet. CPython >> users would be unaffected. >> >> >> Then why can't IronPython patch site.py to do what they want? I still feel >> uncomfortable changing site.py for this in a micro release. >> > > The IronPython team currently have legal issues distributing modified > versions of the standard library (Dino can correct me if I'm wrong here). > > That aside, I think that were a Python feature is implemented in such a way > as to preclude compatibility with other implementations then we should see > it as a bugfix to correct it. > > I can't see why you should feel uncomfortable (beyond general principle), > it seems clear that this change can't affect users of ClassicPython. It is > not only IronPython that will benefit but also Jython when they port to > Python 2.6. > It's just principle. I'm not going to fight this as no one else has spoken up with my view and both Dino and Frank would like this to happen. The one unfortunate thing about this proposal is how this is going to mean I have to install a package potentially four times if I want it available to all possible Python interpreters. Then again, the chances of anyone beyond the people on this mailing list using more than a single interpreter regularly is so small that bothering to add support for yet another user directory that works on any Python interpreter will not be worth it. -Brett > As a general rule the alternative implementations will only start a serious > port once a version is released and into maintenance mode. If we aren't able > to fix compatibility issues then the implementations will always have to > maintain their own patch sets that may or may not get merged back in (or in > the case of implementations with awkward legal departments they'll have to > ship bugs). > > All the best, > > Michael > > > > >> On 9 Oct 2009, at 19:00, Brett Cannon > > wrote: >> >> >>> >>> On Fri, Oct 9, 2009 at 04:53, Michael Foord >>> > wrote: >>> >>> Christian Heimes wrote: >>> >>> Michael Foord wrote: >>> >>> I really like this scheme. The important thing for >>> IronPython is that we can get it into Python 2.6 >>> (along with other fixes to make distutils compatible >>> with IronPython - like not attempting to >>> bytecode-compile when sys.dont_write_bytecode is True). >>> >>> >>> I don't think my proposal will land into 2.6. The changes >>> are too severe >>> for a bug fix release. >>> >>> >>> Right, certainly not adding umpteen new sys attributes. :-) >>> >>> The problem is that the alternative implementations run well >>> behind Python-trunk, indeed it doesn't really make sense for >>> them to put a lot of effort into implementing a version that >>> is still in development. The result is that they discover >>> incompatibilites after a version has gone into 'bugfix only' >>> mode. >>> >>> Whilst the fix you have described (add information to sys >>> that is used by site.py and distutils) is ideal it can only >>> go into 2.7. I would *still* like to see a fix in 2.6 - even >>> if it is simple logic in site.py using sys.platform (if >>> sys.platform == 'cli'; elif sys.platform == 'java' etc). That >>> way IronPython 2.6 is able to be compatible with Python 2.6. >>> This logic might need duplicating in distutils (I haven't >>> looked at how distutils works out where the user >>> site-packages folder is), but it is a 'maintenance only' fix. >>> >>> >>> But it's still a change in semantics. Tossing this into 2.6 would >>> mean that anyone who has worked around the current behaviour is >>> going to have a busted install. >>> >>> -Brett >>> >> >> >> > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin at bud.ca Sun Oct 11 00:16:22 2009 From: kevin at bud.ca (Kevin Teague) Date: Sat, 10 Oct 2009 15:16:22 -0700 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <4AD0C4E7.4030307@voidspace.org.uk> Message-ID: <45F3668A-AB95-4C65-8C86-83980CAB6C75@bud.ca> On Oct 10, 2009, at 2:17 PM, Brett Cannon wrote: > > The one unfortunate thing about this proposal is how this is going > to mean I have to install a package potentially four times if I want > it available to all possible Python interpreters. Then again, the > chances of anyone beyond the people on this mailing list using more > than a single interpreter regularly is so small that bothering to > add support for yet another user directory that works on any Python > interpreter will not be worth it. > Yeah, I imagine that the use-case of wanting to re-use a shared set of packages between multiple interpreter implementations is relatively small. But if you did want to do so, you could just export the location for your (PYTHONIMPLEMENTATIONNAME)PATH to the location you want to re-use and off you go ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Sun Oct 11 01:06:38 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 11 Oct 2009 00:06:38 +0100 Subject: [Python-Dev] PEP 370 and IronPython In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <4AD0C4E7.4030307@voidspace.org.uk> Message-ID: <4AD1137E.2030604@voidspace.org.uk> Brett Cannon wrote: > > > On Sat, Oct 10, 2009 at 10:31, Michael Foord > > wrote: > > Brett Cannon wrote: > > > > On Fri, Oct 9, 2009 at 11:32, Michael Foord > > >> wrote: > > The *only* change in semantics I'm proposing is for users of > IronPython 2.6 which is not even at final release yet. CPython > users would be unaffected. > > > Then why can't IronPython patch site.py to do what they want? > I still feel uncomfortable changing site.py for this in a > micro release. > > > The IronPython team currently have legal issues distributing > modified versions of the standard library (Dino can correct me if > I'm wrong here). > > That aside, I think that were a Python feature is implemented in > such a way as to preclude compatibility with other implementations > then we should see it as a bugfix to correct it. > > I can't see why you should feel uncomfortable (beyond general > principle), it seems clear that this change can't affect users of > ClassicPython. It is not only IronPython that will benefit but > also Jython when they port to Python 2.6. > > > It's just principle. I'm not going to fight this as no one else has > spoken up with my view and both Dino and Frank would like this to happen. > > The one unfortunate thing about this proposal is how this is going to > mean I have to install a package potentially four times if I want it > available to all possible Python interpreters. Then again, the chances > of anyone beyond the people on this mailing list using more than a > single interpreter regularly is so small that bothering to add support > for yet another user directory that works on any Python interpreter > will not be worth it. > It is unfortunate, there are quite a few modules that I use with both IronPython and ClassicPython. Unfortunately there are many others where I have to make modifications or replace components (especially C extensions). It is these modules that require changes that mean separate site-packages directories are needed. To be honest, at the moment I tend to bundle all third party dependencies in with my IronPython projects. As distutils support for IronPython improves I hope that will change. All the best, Michael > -Brett > > > > > As a general rule the alternative implementations will only start > a serious port once a version is released and into maintenance > mode. If we aren't able to fix compatibility issues then the > implementations will always have to maintain their own patch sets > that may or may not get merged back in (or in the case of > implementations with awkward legal departments they'll have to > ship bugs). > > All the best, > > Michael > > > > > On 9 Oct 2009, at 19:00, Brett Cannon > >> wrote: > > > > On Fri, Oct 9, 2009 at 04:53, Michael Foord > > >> wrote: > > Christian Heimes wrote: > > Michael Foord wrote: > > I really like this scheme. The important > thing for > IronPython is that we can get it into > Python 2.6 > (along with other fixes to make distutils > compatible > with IronPython - like not attempting to > bytecode-compile when > sys.dont_write_bytecode is True). > > > I don't think my proposal will land into 2.6. > The changes > are too severe > for a bug fix release. > > > Right, certainly not adding umpteen new sys > attributes. :-) > > The problem is that the alternative implementations > run well > behind Python-trunk, indeed it doesn't really make > sense for > them to put a lot of effort into implementing a > version that > is still in development. The result is that they > discover > incompatibilites after a version has gone into > 'bugfix only' > mode. > > Whilst the fix you have described (add information > to sys > that is used by site.py and distutils) is ideal it > can only > go into 2.7. I would *still* like to see a fix in > 2.6 - even > if it is simple logic in site.py using sys.platform (if > sys.platform == 'cli'; elif sys.platform == 'java' > etc). That > way IronPython 2.6 is able to be compatible with > Python 2.6. > This logic might need duplicating in distutils (I > haven't > looked at how distutils works out where the user > site-packages folder is), but it is a 'maintenance > only' fix. > > > But it's still a change in semantics. Tossing this into > 2.6 would > mean that anyone who has worked around the current > behaviour is > going to have a busted install. > > -Brett > > > > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From greg.ewing at canterbury.ac.nz Sun Oct 11 02:02:56 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Oct 2009 13:02:56 +1300 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4AD120B0.5080100@canterbury.ac.nz> Stephen J. Turnbull wrote: > In the > modern environment bandwidth is not really a consideration, That's no reason to squander it, though. Quoting the entire message every time makes the size of the thread grow as O(n**2), and makes things harder to read as well. That's just senseless. The only possible benefit is to allow latecomers to catch up on the thread without having to consult any archives. But I've never seen any complaints that people who *do* quote responsibly are making this hard, so I can only conclude that it's not a significant problem. -- Greg From stephen at xemacs.org Sun Oct 11 06:26:41 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 11 Oct 2009 13:26:41 +0900 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <4AD120B0.5080100@canterbury.ac.nz> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> Message-ID: <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> Greg Ewing writes: > That's no reason to squander it, though. Quoting the entire > message every time makes the size of the thread grow as > O(n**2), Agreed, but let's not exaggerate. Threads are finite, so big O is not really relevant. Spam OTOH is boundless, and that's where the bandwidth is going. > and makes things harder to read as well. Indeed, and that's why I thanked Michael. Trimming can be a PITA if you're using a crummy MUA, and for reasons I have no intention of even trying to remember, let alone understand, a lot of people are very attached to their crummmy MUAs. But adding a few words to at least make it a little readable (whether apologetic or factual) would make me a lot happier ... given that they're going to top post anyway. From fdrake at gmail.com Sun Oct 11 06:59:47 2009 From: fdrake at gmail.com (Fred Drake) Date: Sun, 11 Oct 2009 00:59:47 -0400 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <4AD120B0.5080100@canterbury.ac.nz> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> Message-ID: <9cee7ab80910102159x46ccbb39l5b147bb17d8c6bb9@mail.gmail.com> On Sat, Oct 10, 2009 at 8:02 PM, Greg Ewing wrote: > That's no reason to squander it, though. Quoting the entire > message every time makes the size of the thread grow as > O(n**2), and makes things harder to read as well. That's > just senseless. Most importantly, insufficient trimming makes many of us start to ignore threads we'd otherwise want to read more carefully or participate in, because the tedium of wading through all the quotes to make sure we catch all the content. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From steve at pearwood.info Sun Oct 11 12:01:36 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 11 Oct 2009 21:01:36 +1100 Subject: [Python-Dev] [OT] Re: Top-posting on this list In-Reply-To: <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACDCA5A.4090609@cheimes.de> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <200910112101.36823.steve@pearwood.info> On Sun, 11 Oct 2009 03:26:41 pm Stephen J. Turnbull wrote: > Indeed, and that's why I thanked Michael. Trimming can be a PITA if > you're using a crummy MUA, and for reasons I have no intention of > even trying to remember, let alone understand, a lot of people are > very attached to their crummmy MUAs. Perhaps I've never used a sufficiently crummy MUA, but I don't get this. Particularly for GUI apps on a PC, trimming just means: select text, press delete, repeat until done. How hard can it be? Surely even the crummiest MUA lets you delete text? This might be hard on a PDA or mobile phone, but on a PC with a keyboard and mouse? (I have a vague recollection that Lotus Notes inserts the quoted message *after* you've done editing your reply, so Notes users should be forgiven. Not to mention pitied.) -- Steven D'Aprano From ben+python at benfinney.id.au Sun Oct 11 13:26:02 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 11 Oct 2009 22:26:02 +1100 Subject: [Python-Dev] Top-posting on this list References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87fx9qrx2t.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > Trimming can be a PITA if you're using a crummy MUA How so? It merely requires the ability to navigate up and down by lines, and to select and delete text. I've used some very crummy MUAs, but the ability to trim quoted text has never been absent or difficult. Are there MUAs so crummy that this is a PITA, yet still used by any significant portion of email users? -- \ ?Simplicity is prerequisite for reliability.? ?Edsger W. | `\ Dijkstra | _o__) | Ben Finney From ben+python at benfinney.id.au Sun Oct 11 13:27:56 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 11 Oct 2009 22:27:56 +1100 Subject: [Python-Dev] Top-posting on this list References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <9cee7ab80910102159x46ccbb39l5b147bb17d8c6bb9@mail.gmail.com> Message-ID: <87bpkerwzn.fsf@benfinney.id.au> Fred Drake writes: > Most importantly, insufficient trimming makes many of us start to > ignore threads we'd otherwise want to read more carefully or > participate in, because the tedium of wading through all the quotes to > make sure we catch all the content. Absolutely. This is a significant reason to publicly, and frequently, respond to request inline-replied, suitably-trimmed messages: to help newcomers reading the list understand what constitutes an easy-to-read message, and to maximise their opportunity to write messages that won't simply be skipped as too much effort to read. -- \ ?The WWW is exciting because Microsoft doesn't own it, and | `\ therefore, there's a tremendous amount of innovation | _o__) happening.? ?Steve Jobs | Ben Finney From solipsis at pitrou.net Sun Oct 11 13:34:53 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 11:34:53 +0000 (UTC) Subject: [Python-Dev] Top-posting on this list References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > That's no reason to squander it, though. Quoting the entire > message every time makes the size of the thread grow as > O(n**2), and makes things harder to read as well. That's > just senseless. +1. It's always annoying to skim through a three-page quoted message just to read a two-line reply. The problem is not electronic bandwidth, but visual bandwidth. Regards Antoine. From ziade.tarek at gmail.com Sun Oct 11 13:36:44 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 11 Oct 2009 13:36:44 +0200 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <87fx9qrx2t.fsf@benfinney.id.au> References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> Message-ID: <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> On Sun, Oct 11, 2009 at 1:26 PM, Ben Finney wrote: > "Stephen J. Turnbull" writes: > >> Trimming can be a PITA if you're using a crummy MUA > > How so? It merely requires the ability to navigate up and down by lines, > and to select and delete text. I've used some very crummy MUAs, but the > ability to trim quoted text has never been absent or difficult. Are > there MUAs so crummy that this is a PITA, yet still used by any > significant portion of email users? You just can't do it on some mobile device mail clients. For instance Gmail's client on Android. It will just top-post and quote the whole mail for you AFAIK. Tarek From ben+python at benfinney.id.au Sun Oct 11 14:31:05 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 11 Oct 2009 23:31:05 +1100 Subject: [Python-Dev] Top-posting on this list References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> Message-ID: <877hv2ru2e.fsf@benfinney.id.au> Tarek Ziad? writes: > You just can't do it on some mobile device mail clients. For instance > Gmail's client on Android. > It will just top-post and quote the whole mail for you AFAIK. Wow, that *is* crummy. Perhaps a posse of users of that application can loudly request this basic feature from the vendor, to at least make it easy to avoid sending obnoxious messages. -- \ ?Ice water? Get some onions ? that'll make your eyes water!? | `\ ?Groucho Marx | _o__) | Ben Finney From masklinn at masklinn.net Sun Oct 11 15:04:34 2009 From: masklinn at masklinn.net (Masklinn) Date: Sun, 11 Oct 2009 15:04:34 +0200 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> Message-ID: <565817BF-F768-4E81-A099-5DF7474970BC@masklinn.net> On 11 Oct 2009, at 13:36 , Tarek Ziad? wrote: > On Sun, Oct 11, 2009 at 1:26 PM, Ben Finney > wrote: >> "Stephen J. Turnbull" writes: >> >>> Trimming can be a PITA if you're using a crummy MUA >> >> How so? It merely requires the ability to navigate up and down by >> lines, >> and to select and delete text. I've used some very crummy MUAs, but >> the >> ability to trim quoted text has never been absent or difficult. Are >> there MUAs so crummy that this is a PITA, yet still used by any >> significant portion of email users? > > You just can't do it on some mobile device mail clients. For instance > Gmail's client on Android. > It will just top-post and quote the whole mail for you AFAIK. I'll add the iPhone pre 3.0 Mail (also holds for the Touch): while trimming was possible, it was done using the delete key and character per character (jumping to word per word after a few seconds) because there was no selection. This made trimming long mails a fairly long-winded task already, but things were even worse due to the lack of undo: not stopping soon enough (and deleting part of the text you wanted to quote) meant you had to either re-type it manually or restart the whole operation from the start. This has thankfully been fixed (with the addition of a pretty good selection mechanism and an undo). Another iPhone mail "feature", (which hasn't been fixed and is actually just as broken in the desktop version of Mail) is that Mail is hardcoded to top-post: it's not possible to make the insertion point default to bottom the quote, it's always above it with a bunch of white-spaces for the message. Not much of an issue, but it still makes top-posting lower friction than bottom-posting or trimming. From mal at egenix.com Sun Oct 11 15:13:33 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 11 Oct 2009 15:13:33 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4AD0AB27.1040109@gmail.com> References: <1afaf6160910091651i3fa55fc0qf3e786af462b8e02@mail.gmail.com> <4ACFD44C.7040009@cheimes.de> <1afaf6160910091729j20f63d4ai389f1952716c58f9@mail.gmail.com> <4ACFDE5C.7080008@cheimes.de> <1afaf6160910091814p5e14c5e6o16620219378bca3a@mail.gmail.com> <4AD0650B.7080600@egenix.com> <4AD0AB27.1040109@gmail.com> Message-ID: <4AD1D9FD.3090206@egenix.com> Nick Coghlan wrote: > M.-A. Lemburg wrote: >> Benjamin Peterson wrote: >>> I forgot to ask before: Does this deprecate platform.python_implementation()? >> >> No, platform.py is meant to be portable across multiple Python >> versions and as such not really suitable for such deprecations. >> >> It'll also take a long while before all Python implementations >> expose a new sys module attribute along the proposed lines. > > However, the function could be updated to take advantage of the new > sys.implementation attribute when it was available. If it didn't find > it, it would fallback to figuring out the way it does now. Sure. It already does that for a couple of other APIs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 11 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Sun Oct 11 17:50:54 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 15:50:54 +0000 (UTC) Subject: [Python-Dev] Weak dict iterators are fragile Message-ID: Hello, In py3k, the weak dict methods keys(), values() and items() have been changed to return iterators (they returned lists in 2.x). However, it turns out that it makes these methods quite fragile, because a GC collection can occur whenever during iterating, destroy one of the weakref'ed objects, and trigger a resizing of the underlying dict, which in turn raises an exception ("RuntimeError: dictionary changed size during iteration"). This has just triggered (assuming the diagnosis is correct) a hang in test_multiprocessing: http://bugs.python.org/issue7060 I would like to propose some possible solutions against this: 1. Add the safe methods listkeys(), listitems(), listvalues() which would behave as the keys(), etc. methods from 2.x 2. Make it so that keys(), items(), values() atomically build a list of items internally, which makes them more costly for large weak dicts, but robust. What do you think? Regards Antoine. From python at mrabarnett.plus.com Sun Oct 11 18:07:29 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 11 Oct 2009 17:07:29 +0100 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <565817BF-F768-4E81-A099-5DF7474970BC@masklinn.net> References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> <565817BF-F768-4E81-A099-5DF7474970BC@masklinn.net> Message-ID: <4AD202C1.8060604@mrabarnett.plus.com> Masklinn wrote: > On 11 Oct 2009, at 13:36 , Tarek Ziad? wrote: >> On Sun, Oct 11, 2009 at 1:26 PM, Ben Finney >> wrote: >>> "Stephen J. Turnbull" writes: >>> >>>> Trimming can be a PITA if you're using a crummy MUA >>> >>> How so? It merely requires the ability to navigate up and down by lines, >>> and to select and delete text. I've used some very crummy MUAs, but the >>> ability to trim quoted text has never been absent or difficult. Are >>> there MUAs so crummy that this is a PITA, yet still used by any >>> significant portion of email users? >> >> You just can't do it on some mobile device mail clients. For instance >> Gmail's client on Android. >> It will just top-post and quote the whole mail for you AFAIK. > > I'll add the iPhone pre 3.0 Mail (also holds for the Touch): while > trimming was possible, it was done using the delete key and character > per character (jumping to word per word after a few seconds) because > there was no selection. > > This made trimming long mails a fairly long-winded task already, but > things were even worse due to the lack of undo: not stopping soon enough > (and deleting part of the text you wanted to quote) meant you had to > either re-type it manually or restart the whole operation from the start. > > This has thankfully been fixed (with the addition of a pretty good > selection mechanism and an undo). > > Another iPhone mail "feature", (which hasn't been fixed and is actually > just as broken in the desktop version of Mail) is that Mail is hardcoded > to top-post: it's not possible to make the insertion point default to > bottom the quote, it's always above it with a bunch of white-spaces for > the message. Not much of an issue, but it still makes top-posting lower > friction than bottom-posting or trimming. > Didn't the iPhone also lack cut-and-paste? From solipsis at pitrou.net Sun Oct 11 18:18:32 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 16:18:32 +0000 (UTC) Subject: [Python-Dev] Top-posting on this list References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> <565817BF-F768-4E81-A099-5DF7474970BC@masklinn.net> <4AD202C1.8060604@mrabarnett.plus.com> Message-ID: MRAB mrabarnett.plus.com> writes: > [snipped three pages of quoted messages before a one-liner] > Didn't the iPhone also lack cut-and-paste? Not to sound harsh, but your quoting was a perfect example of wasted visual bandwidth... (are you posting from an iPhone ? ;-)) Antoine. From daniel at stutzbachenterprises.com Sun Oct 11 18:22:41 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sun, 11 Oct 2009 11:22:41 -0500 Subject: [Python-Dev] Weak dict iterators are fragile In-Reply-To: References: Message-ID: On Sun, Oct 11, 2009 at 10:50 AM, Antoine Pitrou wrote: > In py3k, the weak dict methods keys(), values() and items() have been > changed to return iterators (they returned lists in 2.x). > However, it turns out that it makes these methods quite fragile, because > a GC collection can occur whenever during iterating, destroy one of the > weakref'ed objects, and trigger a resizing of the underlying dict, which > in turn raises an exception ("RuntimeError: dictionary changed size > during iteration"). > Ouch! The iterator from __iter__ is also affected. 1. Add the safe methods listkeys(), listitems(), listvalues() which would > behave as the keys(), etc. methods from 2.x > > 2. Make it so that keys(), items(), values() atomically build a list of > items internally, which makes them more costly for large weak dicts, but > robust. > -1 on 1. +0 on 2. It'd be nice if we could postpone the resize if there are active iterators, but I don't think there's a clean way to track the iterators. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Oct 11 18:26:21 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 16:26:21 +0000 (UTC) Subject: [Python-Dev] Weak dict iterators are fragile References: Message-ID: Antoine Pitrou pitrou.net> writes: > > 1. Add the safe methods listkeys(), listitems(), listvalues() which would > behave as the keys(), etc. methods from 2.x > > 2. Make it so that keys(), items(), values() atomically build a list of > items internally, which makes them more costly for large weak dicts, but > robust. And a third one (a bit more complicated implementation-wise): 3. Delay weak dict removals until any iteration has finished. Regards Antoine. From catch-all at masklinn.net Sun Oct 11 18:32:16 2009 From: catch-all at masklinn.net (Xavier Morel) Date: Sun, 11 Oct 2009 18:32:16 +0200 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <4AD202C1.8060604@mrabarnett.plus.com> References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <94bdd2610910110436y7b1b708ey6bdc1dbb7ac63492@mail.gmail.com> <565817BF-F768-4E81-A099-5DF7474970BC@masklinn.net> <4AD202C1.8060604@mrabarnett.plus.com> Message-ID: <75544DF4-D9AA-4BD5-B9A1-39694CEB0E19@masklinn.net> On 11 Oct 2009, at 18:07 , MRAB wrote: > > Didn't the iPhone also lack cut-and-paste? It did, but given text selection is a near-mandatory requirement to cutting text (and pasting isn't very useful if you can't put anything into the clipboard) those were implied consequences of the lack of selection. From solipsis at pitrou.net Sun Oct 11 18:37:38 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 16:37:38 +0000 (UTC) Subject: [Python-Dev] Weak dict iterators are fragile References: Message-ID: Daniel Stutzbach stutzbachenterprises.com> writes: > > -1 on 1.+0 on 2.It'd be nice if we could postpone the resize if there are active iterators, but I don't think there's a clean way to track the iterators. I've started experimenting, and it seems reasonably possible using a simple guard class and a set of weakrefs. Regards Antoine. From stephen at xemacs.org Sun Oct 11 18:38:15 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 12 Oct 2009 01:38:15 +0900 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <87fx9qrx2t.fsf@benfinney.id.au> References: <4ACDCA5A.4090609@cheimes.de> <4ACE0E99.7050102@voidspace.org.uk> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> Message-ID: <87r5t9khs8.fsf@uwakimon.sk.tsukuba.ac.jp> Ben Finney writes: > "Stephen J. Turnbull" writes: > > > Trimming can be a PITA if you're using a crummy MUA > > How so? It merely requires the ability to navigate up and down by lines, > and to select and delete text. I've used some very crummy MUAs, but the > ability to trim quoted text has never been absent or difficult. Not difficult, time-consuming. It can take many seconds to scroll to bottom in a 200-line quote with gmail and the webmail system used by the University of California at Santa Cruz. You bet I get down on my knees and give thanks that my other MUA is an Emacs every time I have to use webmail. Please note that I'm not defending the practice of top-posting; I'm trying to understand why anybody would do it, and address the readability issue in that context. A sentinel that says "the next 500 lines are whatever happened to be in the tail of the 16KiB block on the file system" does help. If others are willing to play bad cop, as Aahz did, I'd be very happy to accept the benefit of a cleaned-up list. But I'm not willing to do it myself. From p.f.moore at gmail.com Sun Oct 11 18:52:24 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2009 17:52:24 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACF6993.5040704@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> Message-ID: <79990c6b0910110952s266d1d0bj35a9a991420e7de9@mail.gmail.com> 2009/10/9 Michael Foord : > Many Windows users would be quite happy if the standard mechanism for > installing non-source distributions on Windows was via the wininst binaries. +1 I'm one of those people. > I wonder if it is going to be possible to make this compatible with the > upcoming distutils package management 'stuff' (querying for installed > packages, uninstallation etc) since installation/uninstallation goes through > the Windows system package management feature. ?I guess it would be > eminently possible but require some reasonably high level Windows-fu to do. I am working with Tarek to keep Windows issues (and in particular this one) on the agenda. It's quite hard at times, as getting a representative sample of Windows users' preferences/requirements is difficult at best (Windows users seem as a group to be either very quiet, or very easy to please. I'm an exception :-)) If any Windows users want to speak up and help, that would be immensely useful! (And please, identify yourself as such - it's often hard to determine who knows Windows, and who is a Unix user making assumptions or best guesses). Paul. From jnoller at gmail.com Sun Oct 11 18:55:47 2009 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 11 Oct 2009 12:55:47 -0400 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <87r5t9khs8.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACDCA5A.4090609@cheimes.de> <4ACF243F.4080408@voidspace.org.uk> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <87r5t9khs8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4222a8490910110955s5d9d504br428b0fa27b27c6f7@mail.gmail.com> On Sun, Oct 11, 2009 at 12:38 PM, Stephen J. Turnbull wrote: > If others are willing to play bad cop, as Aahz did, I'd be very happy > to accept the benefit of a cleaned-up list. ?But I'm not willing to do > it myself. Is it really that big of an issue that we have to discuss it ad-infinitum and potentially have a quoting cop? Sometimes top-posting happens. Sometimes people don't trim messages. Sometimes people argue about the color of a shed. jesse From steven.bethard at gmail.com Sun Oct 11 19:27:16 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 11 Oct 2009 10:27:16 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910110952s266d1d0bj35a9a991420e7de9@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <79990c6b0910110952s266d1d0bj35a9a991420e7de9@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 9:52 AM, Paul Moore wrote: > 2009/10/9 Michael Foord : >> Many Windows users would be quite happy if the standard mechanism for >> installing non-source distributions on Windows was via the wininst binaries. > > +1 I'm one of those people. +1 on installing packages on Windows through the native package manager (Add/Remove Programs). > I am working with Tarek to keep Windows issues (and in particular this > one) on the agenda. It's quite hard at times, as getting a > representative sample of Windows users' preferences/requirements is > difficult at best (Windows users seem as a group to be either very > quiet, or very easy to please. I'm an exception :-)) > > If any Windows users want to speak up and help, that would be > immensely useful! (And please, identify yourself as such - it's often > hard to determine who knows Windows, and who is a Unix user making > assumptions or best guesses). I'm a Windows user, and I've also spent a fair bit of time improving Python's bdist_msi support exactly so that we have better support for Windows native package management. However, I don't have enough time to keep up with the massive package management threads. If you want to CC me occasionally to get my feedback on a particular comment though, I'd be happy to speak up. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From solipsis at pitrou.net Sun Oct 11 19:31:10 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 17:31:10 +0000 (UTC) Subject: [Python-Dev] Weak dict iterators are fragile References: Message-ID: Antoine Pitrou pitrou.net> writes: > > Daniel Stutzbach stutzbachenterprises.com> writes: > > > > -1 on 1.+0 on 2.It'd be nice if we could postpone the resize if there are > active iterators, but I don't think there's a clean way to track the iterators. > > I've started experimenting, and it seems reasonably possible using a simple > guard class and a set of weakrefs. Open issue and proposed patch at http://bugs.python.org/issue7105 Antoine. From benjamin at python.org Sun Oct 11 19:37:58 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 11 Oct 2009 12:37:58 -0500 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <4222a8490910110955s5d9d504br428b0fa27b27c6f7@mail.gmail.com> References: <4ACDCA5A.4090609@cheimes.de> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <87r5t9khs8.fsf@uwakimon.sk.tsukuba.ac.jp> <4222a8490910110955s5d9d504br428b0fa27b27c6f7@mail.gmail.com> Message-ID: <1afaf6160910111037u10a4e818w1679a24266adf1bd@mail.gmail.com> 2009/10/11 Jesse Noller : > On Sun, Oct 11, 2009 at 12:38 PM, Stephen J. Turnbull > wrote: > >> If others are willing to play bad cop, as Aahz did, I'd be very happy >> to accept the benefit of a cleaned-up list. ?But I'm not willing to do >> it myself. > > Is it really that big of an issue that we have to discuss it > ad-infinitum and potentially have a quoting cop? Sometimes top-posting > happens. Sometimes people don't trim messages. Sometimes people argue > about the color of a shed. +1 This is really getting so meta that it's off topic. -- Regards, Benjamin From g.brandl at gmx.net Sun Oct 11 20:23:17 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 11 Oct 2009 20:23:17 +0200 Subject: [Python-Dev] Top-posting on this list In-Reply-To: <1afaf6160910111037u10a4e818w1679a24266adf1bd@mail.gmail.com> References: <4ACDCA5A.4090609@cheimes.de> <1A2CCCA5-E910-4AE9-9303-032BE939A9BE@voidspace.org.uk> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <87r5t9khs8.fsf@uwakimon.sk.tsukuba.ac.jp> <4222a8490910110955s5d9d504br428b0fa27b27c6f7@mail.gmail.com> <1afaf6160910111037u10a4e818w1679a24266adf1bd@mail.gmail.com> Message-ID: Benjamin Peterson schrieb: > 2009/10/11 Jesse Noller : >> On Sun, Oct 11, 2009 at 12:38 PM, Stephen J. Turnbull >> wrote: >> >>> If others are willing to play bad cop, as Aahz did, I'd be very happy >>> to accept the benefit of a cleaned-up list. But I'm not willing to do >>> it myself. >> >> Is it really that big of an issue that we have to discuss it >> ad-infinitum and potentially have a quoting cop? Sometimes top-posting >> happens. Sometimes people don't trim messages. Sometimes people argue >> about the color of a shed. > > +1 This is really getting so meta that it's off topic. quoting-SIG anyone? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From dickinsm at gmail.com Sun Oct 11 20:28:11 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 11 Oct 2009 19:28:11 +0100 Subject: [Python-Dev] Backport new float repr to Python 2.7? Message-ID: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> In a recent #python-dev IRC conversation, it was suggested that we should consider backporting the new-style float repr from py3k to trunk. I'd like to get people's opinions on this idea. To recap quickly, the algorithm for computing the repr of floats changed between Python 2.x and Python 3.x (well, actually between 3.0 and 3.1, but 3.0 is dead): - in Python 2.x, repr(x) computes 17 significant decimal digits, and then strips trailing zeros. In other words, it's pretty much identical to doing '%.17g' % x. The computation is done using the platform's *printf functions. - in Python 3.x, repr(x) returns the shortest decimal string that's guaranteed to evaluate back to the float x under correct rounding. The computation is done using David Gay's dtoa.c code, adapted for inclusion in Python (in file Python/dtoa.c). There are (in my view) many benefits to the new approach. Among them: - fewer newbie complaints and questions (on c.l.p, IRC, Stack Overflow, etc.) about Python 'rounding incorrectly'. Whether this is a good thing or not is the matter of some debate (I'm tempted to borrow the time machine and simply say 'see the replies to this message'!) - string to float *and* float to string conversions are both guaranteed correctly rounded in 3.x: David Gay's code implements the conversion in both directions, and having correctly rounded string -> float conversions is essential to ensure that eval(repr(x)) recovers x exactly. - the repr of round(x, n) really does have at most n digits after the point, giving the semi-illusion that x really has been rounded exactly, and eliminating one of the most common user complaints about the round function. - round(x, n) agrees exactly with '{:.{}f}'.format(x, n) (this isn't true in Python 2.x, and the difference is a cause of bug reports) - side effects like finding that float(x) rounds correctly for Decimal instances x. - the output from the new rule is more consistent: the 'strip trailing zeros' part of the old rule has some strange consequences: e.g., in 2.x right now (on a typical machine): >>> 0.02 0.02 >>> 0.03 0.029999999999999999 even though neither 0.02 nor 0.03 can be exactly represented in binary. 3.x gives '0.02' and '0.03'. - repr(x) is consistent across platforms (or at least across platforms with IEEE 754 doubles; in practice this seems to account for virtually all platforms currently running Python). - the float <-> string conversions are under our control, so any bugs found can be fixed in the Python source. There's no shortage of conversion bugs in the wild, and certainly bugs have been observed in OS X, Linux and Windows. (The ones I found in OS X 10.5 have been fixed in OS X 10.6, though.) Possible problems: - breaking docstrings in third party code. Though Eric reminded me that when we implemented this for 3.1, there were essentially no standard library test breakages resulting from the changed repr format. - some might argue that the new repr (and round) just allows users to remain ignorant of floating-point difficulties for longer, and that this is a bad thing. I don't really buy either of these points. - someone has to put in the work. As mentioned below, I'm happy to do this (and Eric's offered to help, without which this probably wouldn't be feasible at all), but it'll use cycles that I could also usefully be spending elsewhere. I'm mostly neutral on the backport idea: I'm very happy that this is in 3.x, but don't see any great need to backport it. But if there's majority (+BDFL) support, I'm willing to put the work in to do the backport. Masochists who are still reading by this point and who want more information about the new repr implementation can see the issue discussion: http://bugs.python.org/issue1580 Thoughts? Mark From python at rcn.com Sun Oct 11 21:44:01 2009 From: python at rcn.com (Raymond Hettinger) Date: Sun, 11 Oct 2009 12:44:01 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: [Mark Dickinson] > - string to float *and* float to string conversions are both guaranteed > correctly rounded in 3.x: David Gay's code implements the conversion > in both directions, and having correctly rounded string -> float > conversions is essential to ensure that eval(repr(x)) recovers x exactly. IMO, that is so important that it should be considered a bug fix. Recently, I lost a lot of time in a discussion about a piece of mathematical code returning a wrong answer. The problem wasn't the Python code; instead, it was the str to float conversion (that we inherit from libc) giving the wrong answer. The code worked correctly under Py3.1 but not Py2.6. IDLE 2.6.2 >>> float('-30000000000000000477086733292797541405082425691835844169343982673243516316511143182440409957570445312.00000000000000000009000000000000000258695827614150149965556917147323765827915398318737061345018446445465087890625') -2.9999999999999999e+100 Python 3.1.1 (r311:74483, Aug 17 2009, 17:02:12) [MSC v.1500 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. >>> float('-30000000000000000477086733292797541405082425691835844169343982673243516316511143182440409957570445312.00000000000000000009000000000000000258695827614150149965556917147323765827915398318737061345018446445465087890625') -3.0000000000000002e+100 > - the repr of round(x, n) really does have at most n digits after the > point, giving the semi-illusion that x really has been rounded exactly, > and eliminating one of the most common user complaints about the > round function. This is also an important improvement and makes round() do what people expect. > - side effects like finding that float(x) rounds correctly for > Decimal instances x. This is also important because other decimal calculations can often be done exactly and it's a bummer to have an accurate result thrown off by an incorrect rounding to float. > - the float <-> string conversions are under our control, so any bugs > found can be fixed in the Python source. There's no shortage of > conversion bugs in the wild, and certainly bugs have been observed in > OS X, Linux and Windows. (The ones I found in OS X 10.5 have > been fixed in OS X 10.6, though.) Nice win. > Thoughts? +1 I've worked with the 3.1 float reprs for a while and have been delighted. It was a great improvement. Raymond From guido at python.org Sun Oct 11 21:48:08 2009 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2009 12:48:08 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 11:28 AM, Mark Dickinson wrote: > In a recent #python-dev IRC conversation, it was suggested that we > should consider backporting the new-style float repr from py3k to > trunk. ?I'd like to get people's opinions on this idea. [...] > Possible problems: > > ?- breaking docstrings in third party code. ?Though Eric reminded me > ? that when we implemented this for 3.1, there were essentially no > ? standard library test breakages resulting from the changed repr > ? format. I think you mean doctests? These are the primary reason I've always been hesitant to change this in 2.x. > ?- some might argue that the new repr (and round) just allows users > ? to remain ignorant of floating-point difficulties for longer, and that > ? this is a bad thing. ?I don't really buy either of these points. If we bought that we wouldn't have fixed in 3.x. :-) > ?- someone has to put in the work. ?As mentioned below, I'm happy > ? to do this (and Eric's offered to help, without which this probably > ? wouldn't be feasible at all), but it'll use cycles that I could also > ? usefully be spending elsewhere. > > I'm mostly neutral on the backport idea: ?I'm very happy that this is > in 3.x, but don't see any great need to backport it. ?But if there's > majority (+BDFL) support, I'm willing to put the work in to do the > backport. I'm -0 -- mostly because of the 3rd party doctests and perhaps also because I'd like 3.x to have some carrots. (I've heard from at least one author who is very happy with 3.x for the next edition of his "programming for beginners" book.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Sun Oct 11 21:55:58 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 11 Oct 2009 21:55:58 +0200 Subject: [Python-Dev] http://www.pythonlabs.com/logos.html is gone Message-ID: Which I noticed since it's cited in the BeOpen license we still refer to in LICENSE. Since pythonlabs.com itself is still up, it probably isn't much work to make the logos.html URI work again, but I don't know who maintains that page. cheer, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From glyph at twistedmatrix.com Sun Oct 11 22:00:41 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 11 Oct 2009 16:00:41 -0400 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 3:48 PM, Guido van Rossum wrote: > I'm -0 -- mostly because of the 3rd party doctests and perhaps also > because I'd like 3.x to have some carrots. (I've heard from at least > one author who is very happy with 3.x for the next edition of his > "programming for beginners" book.) > This reasoning definitely makes sense to me; with all the dependency-migration issues 3.x could definitely use some carrots. However, I don't think I agree with it, because this doesn't feel like a big new feature, just some behavior which has changed. The carrots I'm interested in as a user are new possibilties, like new standard library features, a better debugger/profiler, or everybody's favorate bugaboo, multicore parallelism. (Although, to be fair, the removal of old-style classes qualifies.) I'd much rather have my doctests and float-repr'ing code break on 2.7 so I can deal with it as part of a minor-version upgrade than have it break on 3.x and have to deal with this at the same time as the unicode->str explosion. It feels like a backport of this behavior would make the 2->3 transition itself a little easier. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Oct 11 22:21:11 2009 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2009 13:21:11 -0700 Subject: [Python-Dev] http://www.pythonlabs.com/logos.html is gone In-Reply-To: References: Message-ID: On Sun, Oct 11, 2009 at 12:55 PM, Georg Brandl wrote: > Which I noticed since it's cited in the BeOpen license we still refer > to in LICENSE. ?Since pythonlabs.com itself is still up, it probably > isn't much work to make the logos.html URI work again, but I don't know > who maintains that page. I own the domain. I don't know what was on logos.html and http://web.archive.org won't show it to me because of a robots.txt file. I think it's fine to let it be a 404. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From robert.kern at gmail.com Sun Oct 11 22:43:31 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 11 Oct 2009 15:43:31 -0500 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: Guido van Rossum wrote: > On Sun, Oct 11, 2009 at 11:28 AM, Mark Dickinson wrote: >> In a recent #python-dev IRC conversation, it was suggested that we >> should consider backporting the new-style float repr from py3k to >> trunk. I'd like to get people's opinions on this idea. > [...] >> Possible problems: >> >> - breaking docstrings in third party code. Though Eric reminded me >> that when we implemented this for 3.1, there were essentially no >> standard library test breakages resulting from the changed repr >> format. > > I think you mean doctests? These are the primary reason I've always > been hesitant to change this in 2.x. I think that doctests inherently unsuited to testing floating point algorithms. Leaving aside the platform differences in actual floating point arithmetic that cause minute real differences in the results, Python 2.x relies on string conversion routines which give different doctest results on different platforms. Using a consistent routine would actually improve the ability to use doctests in that one regard. It certainly would make writing examples much more consistent, particularly for those of us that use infs and nans frequently. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From g.brandl at gmx.net Sun Oct 11 22:46:04 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 11 Oct 2009 22:46:04 +0200 Subject: [Python-Dev] http://www.pythonlabs.com/logos.html is gone In-Reply-To: References: Message-ID: Guido van Rossum schrieb: > On Sun, Oct 11, 2009 at 12:55 PM, Georg Brandl wrote: >> Which I noticed since it's cited in the BeOpen license we still refer >> to in LICENSE. Since pythonlabs.com itself is still up, it probably >> isn't much work to make the logos.html URI work again, but I don't know >> who maintains that page. > > I own the domain. I don't know what was on logos.html and > http://web.archive.org won't show it to me because of a robots.txt > file. I think it's fine to let it be a 404. Okay. Though I am fairly sure that Tim *would* remember ;) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ziade.tarek at gmail.com Sun Oct 11 22:59:21 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 11 Oct 2009 22:59:21 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910080556m1f2d4156v791193da56db40fb@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <79990c6b0910080156s2e540184r255624d2166cbae2@mail.gmail.com> <94bdd2610910080211q5ff4fd09wf62c44dd4a680a8f@mail.gmail.com> <79990c6b0910080556m1f2d4156v791193da56db40fb@mail.gmail.com> Message-ID: <94bdd2610910111359j7bfe2f0n82d1d6d13cb20c08@mail.gmail.com> On Thu, Oct 8, 2009 at 2:56 PM, Paul Moore wrote: > In this context, eggs are "merely" the first (and most important) [..] > example of a format extension, and so should drive the development of > a standard. > > To summarise: > > I believe that we need a statement of direction on the (zipped) egg > format. I see a number of options: > > 1. Retain it as-is in Distribute, but deprecated and no further development. > 2. Deprecate egg support in Distribute, and ultimately drop it. > 3. Develop into a reference example of an extensible architecture for > adding to PEP 376 support. > 4. Extend PEP 376 support to eggs by means of egg-specific approaches > hooking into distutils. [..] > I'm willing to help work on option (3). I think PEP 376 needs first to be finished with only one installation format, without thinking about adding an extensible architecture for various formats. That's what I meant when I said I wanted to reduce its scope. Besides, I think the same site-packages should stick with a single installation format for the distributions it contains. Although, I can picture that a site-packages could be stored in a totally different way, possibly in a database, or with only eggs, but I am still fuzzy about what would this mean for PEP 376, and the stdlib itself. (pkgutil+distutils) And as you said earlier, this could be done later in a second phase, once PEP 376 APIs (that are not directly related to the format, but rather to what people do with installed distributions metadata) are stabilized and proven to work. Tarek From ziade.tarek at gmail.com Sun Oct 11 23:02:29 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 11 Oct 2009 23:02:29 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <79990c6b0910110952s266d1d0bj35a9a991420e7de9@mail.gmail.com> Message-ID: <94bdd2610910111402k129fb244m197f1b3ce8606849@mail.gmail.com> On Sun, Oct 11, 2009 at 7:27 PM, Steven Bethard wrote: >> I am working with Tarek to keep Windows issues (and in particular this >> one) on the agenda. It's quite hard at times, as getting a >> representative sample of Windows users' preferences/requirements is >> difficult at best (Windows users seem as a group to be either very >> quiet, or very easy to please. I'm an exception :-)) >> >> If any Windows users want to speak up and help, that would be >> immensely useful! (And please, identify yourself as such - it's often >> hard to determine who knows Windows, and who is a Unix user making >> assumptions or best guesses). > > I'm a Windows user, and I've also spent a fair bit of time improving > Python's bdist_msi support exactly so that we have better support for > Windows native package management. However, I don't have enough time > to keep up with the massive package management threads. If you want to > CC me occasionally to get my feedback on a particular comment though, > I'd be happy to speak up. Thanks for the help guys ! Just to mention, these win32 discussions have now moved back to Distutils-SIG. Tarek From brett at python.org Sun Oct 11 23:16:44 2009 From: brett at python.org (Brett Cannon) Date: Sun, 11 Oct 2009 14:16:44 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 13:00, Glyph Lefkowitz wrote: > On Sun, Oct 11, 2009 at 3:48 PM, Guido van Rossum wrote: > > >> I'm -0 -- mostly because of the 3rd party doctests and perhaps also >> because I'd like 3.x to have some carrots. (I've heard from at least >> one author who is very happy with 3.x for the next edition of his >> "programming for beginners" book.) >> > > This reasoning definitely makes sense to me; with all the > dependency-migration issues 3.x could definitely use some carrots. However, > I don't think I agree with it, because this doesn't feel like a big new > feature, just some behavior which has changed. The carrots I'm interested > in as a user are new possibilties, like new standard library features, a > better debugger/profiler, or everybody's favorate bugaboo, multicore > parallelism. (Although, to be fair, the removal of old-style classes > qualifies.) > > Sure, but if people like Mark are having to spend their time backporting every bit of behaviour like this then we won't have the time and energy to add the bigger carrots to 3.x to help entice people to switch. > I'd much rather have my doctests and float-repr'ing code break on 2.7 so I > can deal with it as part of a minor-version upgrade than have it break on > 3.x and have to deal with this at the same time as the unicode->str > explosion. It feels like a backport of this behavior would make the 2->3 > transition itself a little easier. > Maybe, but as Mark pointed out, at least in the test suite for Python, there was no breakage. This will only be an issues if someone does:: >>> x 0.029999999 instead of:: >>> x == 0.03 True Plus it should be obvious when a doctest breaks with 0.03 != 0.0299999999999 what has happened. I'm with Guido: -0 on the backport, especially with Mark feeling neutral on wanting to put the effort in. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Oct 11 23:27:04 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 11 Oct 2009 17:27:04 -0400 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 5:16 PM, Brett Cannon wrote: > On Sun, Oct 11, 2009 at 13:00, Glyph Lefkowitz wrote: > >> The carrots I'm interested in as a user are new possibilties, like new >> standard library features, a better debugger/profiler, or everybody's >> favorate bugaboo, multicore parallelism. (Although, to be fair, the removal >> of old-style classes qualifies.) >> > Sure, but if people like Mark are having to spend their time backporting > every bit of behaviour like this then we won't have the time and energy to > add the bigger carrots to 3.x to help entice people to switch. > Okay, call me +0 then. Not one of the migration issues I'm really sweating about :). -------------- next part -------------- An HTML attachment was scrubbed... URL: From sparks.m at gmail.com Sun Oct 11 23:35:56 2009 From: sparks.m at gmail.com (Michael Sparks) Date: Sun, 11 Oct 2009 22:35:56 +0100 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: <20e5c9660910111435o4d2c48e2he3504a7750201fad@mail.gmail.com> On Sunday 11 October 2009 21:00:41 Glyph Lefkowitz wrote: > with all the > dependency-migration issues 3.x could definitely use some carrots. .. > everybody's favorate bugaboo, multicore parallelism. I know it's the upteen-thousandth time it's been discussed, but removal of the GIL in 3.x would probably be pretty big carrots for some. I know the arguments about performance hits on single core systems etc, and the simplifications it brings, but given every entry level machine these days is multicore, is it time to reconsider some of those points ? Not here perhaps - python-ideas or c.l.p, but if bigger carrots are wanted... Just saying. (As time goes on, lack of a GIL in Ironpython makes it more attractive for multicore work) Not suggesting this happens, but just noting it would probably be a big carrot. Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home From benjamin at python.org Sun Oct 11 23:40:45 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 11 Oct 2009 16:40:45 -0500 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: <20e5c9660910111435o4d2c48e2he3504a7750201fad@mail.gmail.com> References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <20e5c9660910111435o4d2c48e2he3504a7750201fad@mail.gmail.com> Message-ID: <1afaf6160910111440r660767e8o1d7745911cd95f2e@mail.gmail.com> 2009/10/11 Michael Sparks : > On Sunday 11 October 2009 21:00:41 Glyph Lefkowitz wrote: >> with all the >> dependency-migration issues 3.x could definitely use some carrots. > .. >> everybody's favorate bugaboo, multicore parallelism. > > I know it's the upteen-thousandth time it's been discussed, but > removal of the GIL in 3.x would probably be pretty big carrots for > some. I know the arguments about performance hits on single core > systems etc, and the simplifications it brings, but given every entry > level machine these days is multicore, is it time to reconsider some > of those points ? Not here perhaps - python-ideas or c.l.p, but if > bigger carrots are wanted... Just saying. (As time goes on, lack of a > GIL in Ironpython makes it more attractive for multicore work) Oh, I don't think the problem is that Python-dev is opposed to it in principle, but that someone has yet to submit a patch that doesn't affect single thread preformance and remains compatible with the C-API. -- Regards, Benjamin From solipsis at pitrou.net Sun Oct 11 23:41:13 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2009 21:41:13 +0000 (UTC) Subject: [Python-Dev] Backport new float repr to Python 2.7? References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <20e5c9660910111435o4d2c48e2he3504a7750201fad@mail.gmail.com> Message-ID: Michael Sparks gmail.com> writes: > > I know it's the upteen-thousandth time it's been discussed, but > removal of the GIL in 3.x would probably be pretty big carrots for > some. I know the arguments [...] Not before someone produces a patch anyway. It is certainly not as easy as you seem to think it is. Regards Antoine. From sparks.m at gmail.com Sun Oct 11 23:49:25 2009 From: sparks.m at gmail.com (Michael Sparks) Date: Sun, 11 Oct 2009 22:49:25 +0100 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <20e5c9660910111435o4d2c48e2he3504a7750201fad@mail.gmail.com> Message-ID: <20e5c9660910111449w7fea72aw28e9948e5e05595b@mail.gmail.com> On Sun, Oct 11, 2009 at 10:41 PM, Antoine Pitrou wrote: > Michael Sparks gmail.com> writes: >> >> I know it's the upteen-thousandth time it's been discussed, but >> removal of the GIL in 3.x would probably be pretty big carrots for >> some. I know the arguments [...] > > Not before someone produces a patch anyway. It is certainly not as easy as you > seem to think it is. You misunderstand me I think. I don't think it's easy, based on all the discussion I've seen over the years, and the various attempts that have been made, I think it's a Hard problem. I was just mentioning that if someone wanted a serious carrot, that would probably be it. I'll be quiet about it now though, since I *do* understand how contentious an issue it is. Michael. From fwierzbicki at gmail.com Mon Oct 12 01:31:06 2009 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Sun, 11 Oct 2009 19:31:06 -0400 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD86F.3010305@v.loewis.de> References: <4ACFD86F.3010305@v.loewis.de> Message-ID: <4dab5f760910111631j28e8c3b6g6e344a49506654a8@mail.gmail.com> On Fri, Oct 9, 2009 at 8:42 PM, "Martin v. L?wis" wrote: > I think it is important to confirm in advance that all the > implementations listed below agree to implement the PEP "soonish" after > it's adopted. "Required" sounds like a strong term - however, if an > implementation chooses not to implement the PEP, it can do whatever it > wants, including omission of required fields. Speaking for Jython, so far it looks like something we would adopt soonish after it was accepted (it looks pretty useful to me). > So I propose that the python.org version is identified as "python". I'll add my voice to the group that likes "cpython" and "CPython" as the identifier of the python.org implementation. This version has a long history, and "Classic Python" has a nice sound to it. :) -- also I hope (but won't hold my breath) that Python becomes more associated with the abstract language in time. -Frank From fwierzbicki at gmail.com Mon Oct 12 01:36:45 2009 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Sun, 11 Oct 2009 19:36:45 -0400 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD680.1080001@v.loewis.de> References: <1A472770E042064698CB5ADC83A12ACD04A96AA3@TK5EX14MBXC116.redmond.corp.microsoft.com> <4ACFD680.1080001@v.loewis.de> Message-ID: <4dab5f760910111636j72d916a0u7cc9aa5a4c7b007@mail.gmail.com> On Fri, Oct 9, 2009 at 8:34 PM, "Martin v. L?wis" wrote: > Also, why is it the name of the JIT compiler, and not the name of the > source language compiler? >From the Jython side it is easier to get the VM name compared to the source language compiler. Although there is a property on java.lang.System called "java.compiler" it is often empty. -Frank From guido at python.org Mon Oct 12 02:40:56 2009 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2009 17:40:56 -0700 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4dab5f760910111631j28e8c3b6g6e344a49506654a8@mail.gmail.com> References: <4ACFD86F.3010305@v.loewis.de> <4dab5f760910111631j28e8c3b6g6e344a49506654a8@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 4:31 PM, Frank Wierzbicki wrote: > On Fri, Oct 9, 2009 at 8:42 PM, "Martin v. L?wis" wrote: >> So I propose that the python.org version is identified as "python". > I'll add my voice to the group that likes "cpython" and "CPython" as > the identifier of the python.org implementation. ?This version has a > long history, and "Classic Python" has a nice sound to it. ?:) Regardless of the history, "CPython" is indeed how most people refer to this implementation *whenever the distinction matters*, so in a variable used to identify the implementation this makes sense to me, even though most people most of the time don't make this distinction. I'm not worried about other implementations written in C aspiring to the name CPython. > -- also > I hope (but won't hold my breath) that Python becomes more associated > with the abstract language in time. Me too (on both accounts :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Mon Oct 12 06:40:21 2009 From: python at rcn.com (Raymond Hettinger) Date: Sun, 11 Oct 2009 21:40:21 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: <9CE66A617E6740D48190BB16F0E0D49F@RaymondLaptop1> [Glyph Lefkowitz ] > This reasoning definitely makes sense to me; with all the dependency-migration > issues 3.x could definitely use some carrots. However, I don't think I agree with it, > because this doesn't feel like a big new feature, just some behavior which has changed. The carrots/incentives idea also sounds specious to me. First of all, I consider it to be more of a bug fix than a feature -- we've had plenty of bug reports and confusion surrounding the current implementation and at least one of my scripts is broken (occasionally giving wrong answers and the same also is true for decimal.__float__ method being similarly afflicted). Our current dependency on a badly implemented libc strtod() function is not a good thing (and not consistent across various Python builds). Second, as Glyph points out, the change is too small of an improvement to be a real carrot. One quick thought on the doctest issue. If the doctests are being used as originally intended (as part of validating examples in docstrings), then consider that the docstrings themselves would actually be improved with the shorter repr. IMO, it significantly detracts from examples if they are afflicted with floating point repr issues: def average(seq): """ Return the arithmetic mean of a sequence. >>> average([0.1, 0.5]) 0.29999999999999999 """ return sum(seq) / float(len(seq)) Wouldn't this example be much nicer if it returned 0.3 ? Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 12 08:36:05 2009 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2009 23:36:05 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: On Sun, Oct 11, 2009 at 12:44 PM, Raymond Hettinger wrote: > [Mark Dickinson] >> >> - string to float *and* float to string conversions are both guaranteed >> ?correctly rounded in 3.x: David Gay's code implements the conversion >> ?in both directions, and having correctly rounded string -> float >> ?conversions is essential to ensure that eval(repr(x)) recovers x exactly. > > IMO, that is so important that it should be considered a bug fix. It looks like the majority of posters so far is +0 to +1 on this, and I don't want my -0 to stand in the way of progress. But make no mistake, it *is* a feature, not a bug fix. It took a multidisciplinary team a year to implement it -- from existing portable code in C! Mark wrote easily a page in favor of introducing it. Those are signs of a Feature. The original behavior was no more a bug than the distinction between int and long, or the existence of classic and new classes, or the GIL. These were all deliberate design decisions made as compromises between different requirements. And make no mistake, I like this feature in 3.1. Just as I like the new int type, the new str type, the new classes, the new rich comparisons. The new f.p. repr() may look like a small feature -- but it's a feature that few other languages have! If I had a doctest that depended on the vagaries of f.p. repr(), I would probably change the doctest to use simpler numbers; getting 1./3.+1./3. to be exactly 2./3. is hard enough, but I might rewrite my test to use 1/2. + 1/2. = 1. instead. So the doctest argument isn't all too strong. Carrot-wise the feature is quite strong (see Mark's page of arguments in favor) -- but that works also in favor of inclusion on 2.7. In the end somebody needs to do the work. If it can reuse much of the work done for 3.1, then hopefully it will be done in much less time than the original was done. It looks like someone wants to do this work. I wish them well. But doing this work will mean not doing other work on 2.7 that may be more productive. PS. str(x) still seems to be using %.12g -- shouldn't it be made equal to repr() in 3.1 or 3.2? *That* I would call a bug, an oversight. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at egenix.com Mon Oct 12 10:25:35 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 Oct 2009 10:25:35 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4ACFD86F.3010305@v.loewis.de> References: <4ACFD86F.3010305@v.loewis.de> Message-ID: <4AD2E7FF.8070700@egenix.com> "Martin v. L?wis" wrote: >> id (required): >> lower case identifier, for example "cpython", "ironpython", "jython", >> "pypy" > > Doing some bike-shedding: I'd like to not use "cpython" as the name of > the python.org implementation. This term, I believe, was coined around > JPython, somehow making the implementation language the primary means > of distinction. However, there may be alternative implementations > written in C (e.g. unladen swallow), or otherwise be "C based" (e.g. > pypy). > > So I propose that the python.org version is identified as "python". The CPython name is the traditional and commonly used name of the python.org C implementation. Even though the name is inspired by using C as implementation language, it doesn't refer to CPython being the only implementation of Python in C. I don't think there's a need to do any bike shedding on well established names. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 12 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Oct 12 10:55:12 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 Oct 2009 10:55:12 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <4AD2EEF0.1080506@egenix.com> Christian Heimes wrote: > sys.implementation > ------------------ > > platform (required): > platform or language of the implementation, for example "C", ".NET", > "Java" I'd call this attribute "language". We already have sys.platform with a different meaning. Possible values would then be "C", "C#", "Java", "Python", etc. > runtime (required): > lower case runtime library of the implementation, for example "libc", > "msvcrt90", "java6", ".net" > > compiler (required): > verbose name of the compiler, for example "GCC 4.3.3", "Java > HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" If you include the compiler as well, it may make sense to also add the other creation details that platform.py parses from the sys.version string: branch, revision, buildno, builddate > ... > > > sys.userdirsuffix > ----------------- > > sys.userdirsuffix is an implementation and platform specific string that > is used to construct the path for the user site directory as explained > in PEP 370. The string contains the implementation name as well as the > version number of Python. Don't we already have this information available as site.getuserbase()/ site.getusersitepackages() ? > Examples: > python2.6 (CPython, Unix) > Python26 (CPython, Windows) > ironpython2.6 (IronPython, Unix) > IronPython26 (IronPython, Windows) > ... > Since the examples point to the name of the Python stdlib dir and this form is also used in a lot of other places to distinguish different Python versions (e.g. in distutils build dirs), perhaps we should use a more generic name for it like e.g. sys.versiondir ?! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 12 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From scott+python-dev at scottdial.com Mon Oct 12 11:38:56 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 12 Oct 2009 05:38:56 -0400 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: References: Message-ID: <4AD2F930.9050104@scottdial.com> Christian Heimes wrote: > id (required): > lower case identifier, for example "cpython", "ironpython", "jython", > "pypy" > > name (required): > mixed case name of the implementation, for example "CPython", > "IronPython", "Jython", "PyPy" Why both? Is it not true that the following is guaranteed? sys.implementation.id == sys.implementation.name.lower() Furthermore, why is a lower-case-only constant superior to a mixed-case identifier? I wonder this especially given that platform.python_implementation() already returns mixed-case constants. Why not include the version of the language? > platform (required): > platform or language of the implementation, for example "C", ".NET", > "Java" What should someone like PyPy say here? Do you have a use-case in mind? What is the relevance of this field? > runtime (required): > lower case runtime library of the implementation, for example "libc", > "msvcrt90", "java6", ".net" Same questions as above. Is the java-runtime not dependent on libc? $ ldd /opt/sun-jdk-1.6.0.15/jre/bin/java | grep libc libc.so.6 => /lib/libc.so.6 (0xb7d70000) > compiler (required): > verbose name of the compiler, for example "GCC 4.3.3", "Java > HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" I think other commenters have already asked what exactly is considered the "compiler"? This is especially interesting if you consider PyPy, as well. What is the item being compiled that you are interested in? What is the use-case for this info? It could vary based on module: some modules could've been built with a different compiled or even compiled to bytecode by a different implementation or etc. It seems like (platform, runtime, compiler) are guesses at what *might* be useful. But, in the absence of use-cases they are not useful, perhaps misguided, and create a maintenance burden for the future. A lot of this info is available via the platform module. And the only reason given to not use the platform module was to avoid adding dependencies for site.py. I don't see that as an open invitation to adding other fields that are already available via the platform module. Until they are critical to run-time performance, why not wait to add these extra things? The only thing that has been indicated as needed is the identifier for the python implementation. sys.vm or sys.implementation may very well fully support the use cases given merely by being a string. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From david.lyon at preisshare.net Mon Oct 12 14:08:12 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 12 Oct 2009 08:08:12 -0400 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACF6993.5040704@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> Message-ID: On Fri, 09 Oct 2009 17:49:23 +0100, Michael Foord > I wonder if it is going to be possible to make this compatible with the > upcoming distutils package management 'stuff' (querying for installed > packages, uninstallation etc) since installation/uninstallation goes > through the Windows system package management feature. I guess it would > be eminently possible but require some reasonably high level Windows-fu > to do. It's a good question. Searching on it turned up no easy way of doing control panel applications in python. A much easier way would be to just mount a package manager application in the "Programs" + "Python X.Y" menu. Users would surely find it there. Regards David From fetchinson at googlemail.com Mon Oct 12 14:52:12 2009 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Mon, 12 Oct 2009 14:52:12 +0200 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: >> My opinion is that this tool exists only because Python doesn't >> support the installation of multiple versions for the same >> distributions. > > This is not at all how I use virtualenv. For me virtualenv is a > sandbox so that I don't have to become root whenever I need to install > a Python package for testing purposes and to allow me to hop between > sets of installed Python packages while developing on multiple Python > projects. I also use it as a sandbox for build bots so that multiple > bots on the same machine can each build their own projects with just > the known dependencies installed. +1 I also don't use virtualenv for supporting multiple versions, but rather for sandboxing, testing and experimentation. To make sure that an app works with the exact dependencies I think it should work with. Also, it's important that the 'global' site-packages typically requires root privileges while installing to a virtualenv doesn't. Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From eric at trueblade.com Mon Oct 12 17:00:53 2009 From: eric at trueblade.com (Eric Smith) Date: Mon, 12 Oct 2009 11:00:53 -0400 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: <4AD344A5.6060605@trueblade.com> Glyph Lefkowitz wrote: > I'd much rather have my doctests and float-repr'ing code break on 2.7 so > I can deal with it as part of a minor-version upgrade than have it break > on 3.x and have to deal with this at the same time as the unicode->str > explosion. It feels like a backport of this behavior would make the > 2->3 transition itself a little easier. I'm +0 on making the change, primarily for this reason. We can reuse the vast majority of the code from 3.x. Eric. From dickinsm at gmail.com Mon Oct 12 20:41:44 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 12 Oct 2009 19:41:44 +0100 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> Message-ID: <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> [Guido] > I think you mean doctests? These are the primary reason I've always > been hesitant to change this in 2.x. Yes, sorry. I did of course mean doctests. It occurs to me that any doctests that depend on the precise form of repr(x) are, in a sense, already broken, since 2.x makes no guarantees about repr(x) being consistent across platforms. It's just an accident that repr(x) in 2.x pretty much *is* consistent across major platforms, so long as you steer clear of IEEE 754 oddities like subnormals, nans and infinities. [Glyph] > I'd much rather have my doctests and float-repr'ing code break on > 2.7 so I can deal with it as part of a minor-version upgrade than > have it break on 3.x and have to deal with this at the same time > as the unicode->str explosion. It feels like a backport of this > behavior would make the 2->3 transition itself a little easier. I hadn't really thought about this aspect. I find this quite convincing. [Guido] > PS. str(x) still seems to be using %.12g -- shouldn't it be made equal > to repr() in 3.1 or 3.2? *That* I would call a bug, an oversight. But str still has some value in py3k: it protects users from accumulated rounded errors produced by arithmetic operations: Python 3.2a0 (py3k:75216:75220, Oct 3 2009, 21:38:04) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 0.1 + 0.2 0.30000000000000004 >>> 1.23 * 4.64 5.707199999999999 >>> str(0.1 + 0.2) '0.3' >>> str(1.23 * 4.64) '5.7072' I don't know whether this makes it worth keeping str different from repr. Mark From guido at python.org Mon Oct 12 20:48:54 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Oct 2009 11:48:54 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> Message-ID: On Mon, Oct 12, 2009 at 11:41 AM, Mark Dickinson wrote: > [Guido] >> PS. str(x) still seems to be using %.12g -- shouldn't it be made equal >> to repr() in 3.1 or 3.2? *That* I would call a bug, an oversight. > > But str still has some value in py3k: ?it protects users from > accumulated rounded errors produced by arithmetic operations: [...] I know, but this is much more questionable now. Making str() and repr() different was an intentional choice when repr() would often show ugly values even if the input was pretty; but now str() just hides real differences (1.3 != 1.2 + 1.1), and the difference between str() and repr() is pretty subtle. (Show of hands for those who didn't even know there was a difference? :-). I think if we had to start from scratch with the 3.1 float repr() I would have made float's str() equal to its repr(). But this is at most a +1 feeling. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From dickinsm at gmail.com Mon Oct 12 20:54:44 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 12 Oct 2009 19:54:44 +0100 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> Message-ID: <5c6f2a5d0910121154w431b9f1aoa5b51dfdfabfec86@mail.gmail.com> On Mon, Oct 12, 2009 at 7:48 PM, Guido van Rossum wrote: > On Mon, Oct 12, 2009 at 11:41 AM, Mark Dickinson wrote: >> But str still has some value in py3k: ?it protects users from >> accumulated rounded errors produced by arithmetic operations: > [...] > > I know, but this is much more questionable now. [...] Would it be out of the question to make str = repr in 3.2, do you think? Mark From guido at python.org Mon Oct 12 21:09:36 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Oct 2009 12:09:36 -0700 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: <5c6f2a5d0910121154w431b9f1aoa5b51dfdfabfec86@mail.gmail.com> References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> <5c6f2a5d0910121154w431b9f1aoa5b51dfdfabfec86@mail.gmail.com> Message-ID: On Mon, Oct 12, 2009 at 11:54 AM, Mark Dickinson wrote: > On Mon, Oct 12, 2009 at 7:48 PM, Guido van Rossum wrote: >> On Mon, Oct 12, 2009 at 11:41 AM, Mark Dickinson wrote: >>> But str still has some value in py3k: ?it protects users from >>> accumulated rounded errors produced by arithmetic operations: >> [...] >> >> I know, but this is much more questionable now. [...] > > Would it be out of the question to make str = repr in 3.2, do you think? I don't think it is out of the question. (Though it would probably break a few more doctests.) I'd like to hear other voices. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Mon Oct 12 21:23:33 2009 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 12 Oct 2009 15:23:33 -0400 Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> Message-ID: <1f7befae0910121223i25539a1aqce7492b851e40b49@mail.gmail.com> [Mark Dickinson] > It occurs to me that any doctests that depend on the precise form of > repr(x) are, in a sense, already broken, since 2.x makes no guarantees > about repr(x) being consistent across platforms. The doctest documentation has warned about this forever (look near the end of the "Warnings" section, http://docs.python.org/library/doctest.html#warnings ). >?It's just an accident that repr(x) in 2.x pretty much *is* consistent across > major platforms, so long as you steer clear of IEEE 754 oddities like subnormals, > nans and infinities. If you don't consider Windows to be a major platform ;-) Besides that there's just no guessing what the Microsoft double->string routines will produce for the 17th digit, the MS routines always produce 3 digits for the exponent in scientific notation, while AFAIK all other platforms produce 2 digits when 3 aren't necessary. For example, on Windows under Python 2.x: >>> repr(1e47) '1e+047' >>> repr(1e-47) '9.9999999999999997e-048' and "everywhere else": >>> repr(1e47) '1e+47' >>> repr(1e-47) '9.9999999999999997e-48' The leading 0 in the Window's exponents is enough to break a naive doctest too -- and repr(x) does produce scientific notation for "almost all" finite doubles. Why people are obsessed with the relative handful of doubles between 1 and 1000 is beyond me ;-) As doctest's original author, I have no sympathy for users who burn themselves with float output. Of course it's open to question whether I have sympathy for anyone, ever, but that's not the issue here ;-) From eric at trueblade.com Mon Oct 12 21:42:52 2009 From: eric at trueblade.com (Eric Smith) Date: Mon, 12 Oct 2009 15:42:52 -0400 (EDT) Subject: [Python-Dev] Backport new float repr to Python 2.7? In-Reply-To: <1f7befae0910121223i25539a1aqce7492b851e40b49@mail.gmail.com> References: <5c6f2a5d0910111128j672381b3k507be922a7582a3f@mail.gmail.com> <5c6f2a5d0910121141u65da9668r73529489fd3591d2@mail.gmail.com> <1f7befae0910121223i25539a1aqce7492b851e40b49@mail.gmail.com> Message-ID: <24288.63.251.87.214.1255376572.squirrel@mail.trueblade.com> [Tim:] > If you don't consider Windows to be a major platform ;-) Besides that > there's just no guessing what the Microsoft double->string routines > will produce for the 17th digit, the MS routines always produce 3 > digits for the exponent in scientific notation, while AFAIK all other > platforms produce 2 digits when 3 aren't necessary. For example, on > Windows under Python 2.x: > >>>> repr(1e47) > '1e+047' >>>> repr(1e-47) > '9.9999999999999997e-048' > > and "everywhere else": > >>>> repr(1e47) > '1e+47' >>>> repr(1e-47) > '9.9999999999999997e-48' Not that it's germane to the discussion, but this 3 digit exponent on Windows was fixed in 2.6. > The leading 0 in the Window's exponents is enough to break a naive > doctest too -- and repr(x) does produce scientific notation for > "almost all" finite doubles. Why people are obsessed with the > relative handful of doubles between 1 and 1000 is beyond me ;-) > > As doctest's original author, I have no sympathy for users who burn > themselves with float output. Of course it's open to question whether > I have sympathy for anyone, ever, but that's not the issue here ;-) I agree, we shouldn't base the decision to change this based on doctests. From tseaver at palladion.com Mon Oct 12 23:36:49 2009 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 12 Oct 2009 17:36:49 -0400 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> Message-ID: <4AD3A171.6070906@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziad? wrote: > = Virtualenv and the multiple version support in Distribute = > > (I am not saying "We" here because this part was not discussed yet > with everyone) > > Virtualenv allows you to create an isolated environment to install > some distribution without polluting the > main site-packages, a bit like a user site-packages. > > My opinion is that this tool exists only because Python doesn't > support the installation of multiple versions for the same > distributions. > But if PEP 376 and PEP 386 support are added in Python, we're not far > from being able to provide multiple version support with > the help of importlib. I couldn't disagree more strongly: I don't *want* multiple version support: compared to the simplicity of a virtualenv, adding such a complex feature would be a huge backward step. The idea that the "system" python should be polluted with whatever-any- random-app-decides-to-throw-into-it is a source of endless pain, confusion, and hair loss. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrToW0ACgkQ+gerLs4ltQ7fKgCdEPxU6JxKOKCFWtpQSks606DQ XeAAoICgFZ8RURvFH4p3dLTzqq51ROqP =e71z -----END PGP SIGNATURE----- From greg at krypto.org Tue Oct 13 07:55:28 2009 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 12 Oct 2009 22:55:28 -0700 Subject: [Python-Dev] Top-posting on this list In-Reply-To: References: <4ACDCA5A.4090609@cheimes.de> <87r5tblkp0.fsf@uwakimon.sk.tsukuba.ac.jp> <4AD120B0.5080100@canterbury.ac.nz> <873a5qlfni.fsf@uwakimon.sk.tsukuba.ac.jp> <87fx9qrx2t.fsf@benfinney.id.au> <87r5t9khs8.fsf@uwakimon.sk.tsukuba.ac.jp> <4222a8490910110955s5d9d504br428b0fa27b27c6f7@mail.gmail.com> <1afaf6160910111037u10a4e818w1679a24266adf1bd@mail.gmail.com> Message-ID: <52dc1c820910122255r6328f44ax58e0dac22f36e4d2@mail.gmail.com> Those who feel diverse can top post. On Sun, Oct 11, 2009 at 11:23 AM, Georg Brandl wrote: >>> Is it really that big of an issue that we have to discuss it >>> ad-infinitum and potentially have a quoting cop? Sometimes top-posting >>> happens. Sometimes people don't trim messages. Sometimes people argue >>> about the color of a shed. >> >> +1 This is really getting so meta that it's off topic. > > quoting-SIG anyone? Those who don't can post below. From ncoghlan at gmail.com Tue Oct 13 11:49:44 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Oct 2009 19:49:44 +1000 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4AD2EEF0.1080506@egenix.com> References: <4AD2EEF0.1080506@egenix.com> Message-ID: <4AD44D38.1020608@gmail.com> M.-A. Lemburg wrote: >> sys.userdirsuffix >> ----------------- >> >> sys.userdirsuffix is an implementation and platform specific string that >> is used to construct the path for the user site directory as explained >> in PEP 370. The string contains the implementation name as well as the >> version number of Python. > > Don't we already have this information available as site.getuserbase()/ > site.getusersitepackages() ? No, the new attribute is intended to allow a VM implementation to customise the way site.py calculates the user directories so that CPython, Jython, IronPython, etc, don't have to share the same directory. CPython will stick with the directory as named in PEP 370 (since we already released a version with that behaviour), but the other implementations will be able to choose a name that doesn't conflict with it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From mal at egenix.com Tue Oct 13 12:27:57 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 12:27:57 +0200 Subject: [Python-Dev] PEP about sys.implementation and implementation specific user site directory In-Reply-To: <4AD44D38.1020608@gmail.com> References: <4AD2EEF0.1080506@egenix.com> <4AD44D38.1020608@gmail.com> Message-ID: <4AD4562D.9030104@egenix.com> Nick Coghlan wrote: > M.-A. Lemburg wrote: >>> sys.userdirsuffix >>> ----------------- >>> >>> sys.userdirsuffix is an implementation and platform specific string that >>> is used to construct the path for the user site directory as explained >>> in PEP 370. The string contains the implementation name as well as the >>> version number of Python. >> >> Don't we already have this information available as site.getuserbase()/ >> site.getusersitepackages() ? > > No, the new attribute is intended to allow a VM implementation to > customise the way site.py calculates the user directories so that > CPython, Jython, IronPython, etc, don't have to share the same directory. > > CPython will stick with the directory as named in PEP 370 (since we > already released a version with that behaviour), but the other > implementations will be able to choose a name that doesn't conflict with it. Ah, so it's intended to provide information to site.getuserbase()/ site.getusersitepackages() ? Still, since that path name component is used in other contexts as well, wouldn't it be better to have all those contexts use the name ?! Python has always used the (lib|include)/pythonX.Y/ path naming scheme on Unix, but it's not codified anywhere. A sys attribute or a set of such attributes would be good way to do so. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Tue Oct 13 14:23:35 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 08:23:35 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> Message-ID: On Oct 8, 2009, at 8:45 AM, Zvezdan Petkovic wrote: > On Oct 7, 2009, at 2:09 PM, Barry Warsaw wrote: >> I want us to be very careful about 2.6.4. This isn't a normal bug >> fix release, it's specifically there to remove the brown paper bag >> of 2.6.3 from our heads. So let's be conservative and fix this one >> in 2.6.5. > > Can we get the readline patch reviewed as well. It was applied to > trunk already: http://svn.python.org/view?view=rev&revision=74970 > > It's marked for backport to 2.6 as "needs review": > http://bugs.python.org/issue6877 > > It fixes a BusError crash that was just swiped under a rug by > disabling the build in setup.py. :-) I'm disinclined to approve this for 2.6.4 since it is not a regression in 2.6.3. It seems fine to me for 2.6.5 though. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Tue Oct 13 14:27:01 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 08:27:01 -0400 Subject: [Python-Dev] On track for Python 2.6.4 final this Sunday? Message-ID: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> Are we on track to release 2.6.4 final this Sunday or do we need another rc? Yesterday, Tarek committed another setuptools related fix and said that he was going to run a bunch of build tests locally. Tarek, how did that go? Please note that issue 7064 is still open as a release blocker. Can we close that (again) now? http://bugs.python.org/issue7064 I am still being very conservative about what goes in 2.6.4. Only regressions introduced in 2.6.3 are being accepted. There's plenty of time after that for patches to go in for 2.6.5. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From zvezdan at zope.com Tue Oct 13 14:31:00 2009 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Tue, 13 Oct 2009 08:31:00 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> Message-ID: On Oct 13, 2009, at 8:23 AM, Barry Warsaw wrote: > On Oct 8, 2009, at 8:45 AM, Zvezdan Petkovic wrote: > >> On Oct 7, 2009, at 2:09 PM, Barry Warsaw wrote: >>> I want us to be very careful about 2.6.4. This isn't a normal bug >>> fix release, it's specifically there to remove the brown paper bag >>> of 2.6.3 from our heads. So let's be conservative and fix this >>> one in 2.6.5. >> >> Can we get the readline patch reviewed as well. It was applied to >> trunk already: http://svn.python.org/view?view=rev&revision=74970 >> >> It's marked for backport to 2.6 as "needs review": >> http://bugs.python.org/issue6877 >> >> It fixes a BusError crash that was just swiped under a rug by >> disabling the build in setup.py. :-) > > I'm disinclined to approve this for 2.6.4 since it is not a > regression in 2.6.3. It seems fine to me for 2.6.5 though. Excellent. That's exactly what I meant. I quoted the part of the previous message where you proposed to review another patch for 2.6.5. I guess it was not very clear that I'm proposing to review this for 2.6.5 as well. Well, at least we're on the same page now it seems. :-) Thanks, Zvezdan From barry at python.org Tue Oct 13 15:48:41 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 09:48:41 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> Message-ID: <8F38C15F-3A1C-4825-A38D-6242CC04B56E@python.org> On Oct 13, 2009, at 8:31 AM, Zvezdan Petkovic wrote: > Excellent. That's exactly what I meant. > I quoted the part of the previous message where you proposed to > review another patch for 2.6.5. I guess it was not very clear that > I'm proposing to review this for 2.6.5 as well. Well, at least > we're on the same page now it seems. :-) :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mal at egenix.com Tue Oct 13 16:16:36 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 16:16:36 +0200 Subject: [Python-Dev] On track for Python 2.6.4 final this Sunday? In-Reply-To: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> Message-ID: <4AD48BC4.2060809@egenix.com> Barry Warsaw wrote: > Are we on track to release 2.6.4 final this Sunday or do we need another > rc? > > Yesterday, Tarek committed another setuptools related fix and said that > he was going to run a bunch of build tests locally. Tarek, how did that > go? > > Please note that issue 7064 is still open as a release blocker. Can we > close that (again) now? > > http://bugs.python.org/issue7064 > > I am still being very conservative about what goes in 2.6.4. Only > regressions introduced in 2.6.3 are being accepted. There's plenty of > time after that for patches to go in for 2.6.5. It would be nice to get this issue resolved out for 2.6.4: http://bugs.python.org/issue4120 The problem is that extensions built with 2.6.x will not work when used with a User-only installation of Python on machines that don't already have the MS VC90 CRT DLLs installed system-wide. There's a patch available that appears to work, but someone with more manifest-Windows-DLL-fu than me will need to have a look at it. One possible approach would be to use the patch in 2.6.4 and then continue digging deeper until 2.6.5 is released. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Tue Oct 13 16:41:01 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Oct 2009 22:41:01 +0800 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD48BC4.2060809@egenix.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> Message-ID: <4AD4917D.8090800@v.loewis.de> > It would be nice to get this issue resolved out for 2.6.4: > > http://bugs.python.org/issue4120 > > The problem is that extensions built with 2.6.x will not work > when used with a User-only installation of Python on machines that > don't already have the MS VC90 CRT DLLs installed system-wide. As this bug already exists in 2.6.2, I don't think the change is eligible for 2.6.4. In addition, I want to review it, which I won't be able to until Sunday. > One possible approach would be to use the patch in 2.6.4 and then > continue digging deeper until 2.6.5 is released. This also speaks against the patch. Anything being changed in this area ideally should be the final state of affairs for the rest of 2.6.x. Regards, Martin From pje at telecommunity.com Tue Oct 13 16:52:02 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 13 Oct 2009 10:52:02 -0400 Subject: [Python-Dev] On track for Python 2.6.4 final this Sunday? In-Reply-To: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> Message-ID: <20091013145211.8DB243A4068@sparrow.telecommunity.com> At 08:27 AM 10/13/2009 -0400, Barry Warsaw wrote: >Are we on track to release 2.6.4 final this Sunday or do we need >another rc? > >Yesterday, Tarek committed another setuptools related fix and said >that he was going to run a bunch of build tests locally. Tarek, how >did that go? FWIW, that change won't work to fix the problem on Windows if a package's setup.py is written with cross-platform paths (i.e., the usual case). Also, for the record, it's not really a setuptools-related fix; it's just attempting to fix collateral damage caused by the attempt to fix the problems caused by the original API change... which affected all build_ext customizations relying on the docstring of the changed method, not just the one in setuptools. From mal at egenix.com Tue Oct 13 17:01:05 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 17:01:05 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD4917D.8090800@v.loewis.de> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> Message-ID: <4AD49631.70607@egenix.com> "Martin v. L?wis" wrote: > >> It would be nice to get this issue resolved out for 2.6.4: >> >> http://bugs.python.org/issue4120 >> >> The problem is that extensions built with 2.6.x will not work >> when used with a User-only installation of Python on machines that >> don't already have the MS VC90 CRT DLLs installed system-wide. > > As this bug already exists in 2.6.2, I don't think the change is > eligible for 2.6.4. > > In addition, I want to review it, which I won't be able to until > Sunday. Then I'd suggest to wait another week with 2.6.4 to give you a chance to look at the patch. >> One possible approach would be to use the patch in 2.6.4 and then >> continue digging deeper until 2.6.5 is released. > > This also speaks against the patch. Anything being changed in this area > ideally should be the final state of affairs for the rest of 2.6.x. Of course, but we don't live in an ideal world, otherwise we would have noticed before releasing 2.6 :-) I guess the problem is that developer typically already have the MS VC90 CRT DLLs installed system-wide, so they simply don't notice. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Tue Oct 13 17:16:19 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 13 Oct 2009 17:16:19 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <20091013145211.8DB243A4068@sparrow.telecommunity.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> Message-ID: <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> On Tue, Oct 13, 2009 at 4:52 PM, P.J. Eby wrote: > At 08:27 AM 10/13/2009 -0400, Barry Warsaw wrote: >> >> Are we on track to release 2.6.4 final this Sunday or do we need >> another rc? >> >> Yesterday, Tarek committed another setuptools related fix and said >> that he was going to run a bunch of build tests locally. ?Tarek, how >> did that go? > I still need to do some more tests, I didn't have time to try the various projects under win32. It's planned to night. The tests are consisting of compiling and insatling a dozain of projects on linux and win32 (and bith when possible) > FWIW, that change won't work to fix the problem on Windows if a package's > setup.py is written with cross-platform paths (i.e., the usual case). > > Also, for the record, it's not really a setuptools-related fix; it's just > attempting to fix collateral damage caused by the attempt to fix the > problems caused by the original API change... ?which affected all build_ext > customizations relying on the docstring of the changed method, not just the > one in setuptools. Yes the doctest was pretty fuzzy about what is an extension name. It's will be improved in 2.7. The current code is doing os.path.join()'s to join the Extension name with the build path, leading to the collateral damage you are mentioning. To fix the problem, the API has to be as permissive as it was before. I have already three or four tests to fix that problem in test_build_ext. If you can take a look at them and think of any other test that would be missing, that would be appreciated. Another pair of eye is never enough. Tarek From martin at v.loewis.de Tue Oct 13 17:20:21 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Oct 2009 23:20:21 +0800 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD49631.70607@egenix.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> Message-ID: <4AD49AB5.9040703@v.loewis.de> >> As this bug already exists in 2.6.2, I don't think the change is >> eligible for 2.6.4. >> >> In addition, I want to review it, which I won't be able to until >> Sunday. > > Then I'd suggest to wait another week with 2.6.4 to give you a > chance to look at the patch. That won't make the change more eligible. > Of course, but we don't live in an ideal world, otherwise we would > have noticed before releasing 2.6 :-) Oh, I did notice. I had been cautioning for years that switching to a newer VS version (2005 or later) would cause severe problems. We are still trying to recover from the switch to VS 2008. That said, staying with VS 2003 really hadn't been an option, either. It's just said that Microsoft has created a new DLL hell in their attempt to fix the old one, and that they fail to admit it (assuming that everybody should be using .NET, anyway). Regards, Martin From pje at telecommunity.com Tue Oct 13 17:30:26 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 13 Oct 2009 11:30:26 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.co m> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> Message-ID: <20091013153037.3EB2A3A4068@sparrow.telecommunity.com> At 05:16 PM 10/13/2009 +0200, Tarek Ziad? wrote: >Yes the doctest was pretty fuzzy about what is an extension name. It's >will be improved in 2.7. > >The current code is doing os.path.join()'s to join the Extension name >with the build path, >leading to the collateral damage you are mentioning. To fix the >problem, the API has to be as permissive >as it was before. > >I have already three or four tests to fix that problem in >test_build_ext. If you can take a look at them and >think of any other test that would be missing, that would be >appreciated. Another pair of eye is never enough. One identical to test_build_ext_path_with_os_sep, but that explicitly uses a '/' (rather than os.sep) will identify the problem I'm referring to, when run on Windows. It's common practice to use /-separated paths in setup scripts, regardless of platform. So, your current fix (converting os.sep to '.') will work on Linux, Mac, etc., but fail on Windows when run with the same setup.py, since os.sep is a backslash there. (Just as a side note, if when you split off issue 7115 you'd said what the new issue number was in 7064, or copied me to the nosy-list on the new issue, I'd have been able to review this change and comment on it yesterday instead of today, and I'd have done it in the bug tracker rather than via Python-Dev.) From ziade.tarek at gmail.com Tue Oct 13 17:39:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 13 Oct 2009 17:39:10 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <20091013153037.3EB2A3A4068@sparrow.telecommunity.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <20091013153037.3EB2A3A4068@sparrow.telecommunity.com> Message-ID: <94bdd2610910130839jf0b653fg231d1ed4c5236c10@mail.gmail.com> On Tue, Oct 13, 2009 at 5:30 PM, P.J. Eby wrote: > > One identical to test_build_ext_path_with_os_sep, but that explicitly uses a > '/' (rather than os.sep) will identify the problem I'm referring to, when > run on Windows. > > It's common practice to use /-separated paths in setup scripts, regardless > of platform. ?So, your current fix (converting os.sep to '.') will work on > Linux, Mac, etc., but fail on Windows when run with the same setup.py, since > os.sep is a backslash there. Ok I will have a look at this tonite. Thanks. Notice that this changed I made to fix another bug opened a can of worms because Extension is not enough/clearly documented in this regards. The goal for 2.7/3.2 is to come up with one single way to define extensions, and I will probably just add deprecation warnings for the other forms. > > (Just as a side note, if when you split off issue 7115 you'd said what the > new issue number was in 7064, or copied me to the nosy-list on the new > issue, I'd have been able to review this change and comment on it yesterday > instead of today, and I'd have done it in the bug tracker rather than via > Python-Dev.) Sorry about that, I'll add a note on #7064. From mal at egenix.com Tue Oct 13 17:42:32 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 17:42:32 +0200 Subject: [Python-Dev] On track for Python 2.6.4 final this Sunday? In-Reply-To: <20091013145211.8DB243A4068@sparrow.telecommunity.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> Message-ID: <4AD49FE8.8050002@egenix.com> P.J. Eby wrote: > At 08:27 AM 10/13/2009 -0400, Barry Warsaw wrote: >> Are we on track to release 2.6.4 final this Sunday or do we need >> another rc? >> >> Yesterday, Tarek committed another setuptools related fix and said >> that he was going to run a bunch of build tests locally. Tarek, how >> did that go? > > FWIW, that change won't work to fix the problem on Windows if a > package's setup.py is written with cross-platform paths (i.e., the usual > case). > > Also, for the record, it's not really a setuptools-related fix; it's > just attempting to fix collateral damage caused by the attempt to fix > the problems caused by the original API change... which affected all > build_ext customizations relying on the docstring of the changed method, > not just the one in setuptools. Just to clarify a bit what all the fuzz is about: The API in question (build_ext.get_ext_filename()) was not changed at all. The doc-string also doesn't refer to *the* extension currently being built, but to just to some arbitrary extension: def get_ext_filename (self, ext_name): r"""Convert the name of an extension (eg. "foo.bar") into the name Convert the name of an extension (eg. "foo.bar") into the name of the file from which it will be loaded (eg. "foo/bar.so", or "foo\bar.pyd").""" What did change is the way this method is used by other methods in build_ext: the method is no longer being called with the full dotted name, but only with the last component. Now, setuptools and pywin32 both override the method, with the understanding that this method is the one and only way to convert the extensions (full) dotted name into a file name. Up until the change (which was needed to get inplace installations working), that assumption was true. However, neither the doc-string nor the implementation of the method ever implied this. distutils occasionally makes such assumptions necessary since there's no clean OO-way to override the functionality. That's not ideal, but works most of the time. However, authors of distutils add-ons cannot really expect such hacks to survive forever. Rather than relying on these, it's better to get distutils fixed to provide the necessary OO-hooks to properly implement the modified functionality. FWIW, I've had to change and tweak our mxSetup.py many many times to get it to work again with new distutils releases and I expect this to occur even more often now that distutils is finally being maintained again - and that's good, since I'll be able to get rid off the few remaining hacks ! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Oct 13 17:59:17 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 17:59:17 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD49AB5.9040703@v.loewis.de> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> <4AD49AB5.9040703@v.loewis.de> Message-ID: <4AD4A3D5.8030907@egenix.com> "Martin v. L?wis" wrote: > >>> As this bug already exists in 2.6.2, I don't think the change is >>> eligible for 2.6.4. >>> >>> In addition, I want to review it, which I won't be able to until >>> Sunday. >> >> Then I'd suggest to wait another week with 2.6.4 to give you a >> chance to look at the patch. > > That won't make the change more eligible. No, but chances increase :-) >> Of course, but we don't live in an ideal world, otherwise we would >> have noticed before releasing 2.6 :-) > > Oh, I did notice. I had been cautioning for years that switching to > a newer VS version (2005 or later) would cause severe problems. We > are still trying to recover from the switch to VS 2008. That said, > staying with VS 2003 really hadn't been an option, either. It's > just said that Microsoft has created a new DLL hell in their attempt > to fix the old one, and that they fail to admit it (assuming that > everybody should be using .NET, anyway). Well, hopefully we can work out those problems and then stay with VS 2008 a bit. MS is already working on VS 2010: http://www.microsoft.com/visualstudio/en-us/products/2010/default.mspx List of C/C++ changes: http://msdn.microsoft.com/en-us/library/dd465215(VS.100).aspx Not sure what this means: """ The libraries deployment model is no longer based on the fusion mechanism, and manifests are no longer used. Instead, the name of each dynamic link library contains its version number. """ Perhaps they finally realized that adding version number to DLLs is better than trying to match manifests. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Tue Oct 13 18:23:34 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 13 Oct 2009 12:23:34 -0400 Subject: [Python-Dev] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD49FE8.8050002@egenix.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <4AD49FE8.8050002@egenix.com> Message-ID: <20091013162342.BED653A4068@sparrow.telecommunity.com> At 05:42 PM 10/13/2009 +0200, M.-A. Lemburg wrote: >FWIW, I've had to change and tweak our mxSetup.py many many >times to get it to work again with new distutils releases and >I expect this to occur even more often now that distutils is >finally being maintained again - and that's good, since I'll >be able to get rid off the few remaining hacks ! Thanks for the nice (and neutral) explanation, for those not up to speed on the details. Of course, had the change been made in 2.7 rather than 2.6.3, there wouldn't have been a problem here. People understand that new major versions sometimes mean that existing packages will need to adapt and cope. Under normal circumstances, however, a dot release shouldn't break packages that worked with previous dot releases. (At least, that's the common expectation.) Now, I do have to admit that I'm still ignorant of what bug this was trying to fix in the first place. You say it "was needed to get inplace installations working", but I don't know what that means, since AFAIK inplace extension builds (build_ext -i) have been working since before the first versions of setuptools appeared in 2003. So, if there was a bug in build_ext that wasn't *introduced* in 2.6, it seems like 2.6.3 would be an odd place to fix that bug. This isn't intended as a criticism of anybody, mind you; I'm just saying that I still don't understand why the change was made in 2.6.3 in the first place, and it would be kind of nice to know. These kinds of situations can be useful for illustrating and improving Python policy, if the root causes and reasoning are understood by everyone. From barry at python.org Tue Oct 13 18:56:09 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 12:56:09 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD49631.70607@egenix.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> Message-ID: On Oct 13, 2009, at 11:01 AM, M.-A. Lemburg wrote: > Then I'd suggest to wait another week with 2.6.4 to give you a > chance to look at the patch. That's not a good option, IMO. We have a known broken 2.6.3 out there and we owe it to our users to correct our mistake and given them the release they should have gotten in the first place. We don't need to wait too long for 2.6.5 though. A few months would be appropriate. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Tue Oct 13 18:57:21 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 12:57:21 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> Message-ID: <52E997AF-A050-41F4-9E2A-36282590B230@python.org> On Oct 13, 2009, at 11:16 AM, Tarek Ziad? wrote: > I still need to do some more tests, I didn't have time to try the > various projects under win32. > It's planned to night. > > The tests are consisting of compiling and insatling a dozain of > projects on linux and win32 (and bith when possible) Great thanks, let us know how it goes. If there's any question at all about the fixes that have gone in since rc1, we should spin an rc2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From ssteinerx at gmail.com Tue Oct 13 19:04:41 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Tue, 13 Oct 2009 13:04:41 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <52E997AF-A050-41F4-9E2A-36282590B230@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> Message-ID: On Oct 13, 2009, at 12:57 PM, Barry Warsaw wrote: > On Oct 13, 2009, at 11:16 AM, Tarek Ziad? wrote: > >> I still need to do some more tests, I didn't have time to try the >> various projects under win32. >> It's planned to night. >> >> The tests are consisting of compiling and insatling a dozain of >> projects on linux and win32 (and bith when possible) > > Great thanks, let us know how it goes. If there's any question at > all about the fixes that have gone in since rc1, we should spin an > rc2. Just curious, how big a server does the linux portion of the test require memory-wise? Would, e.g. an Ubuntu 9.0x with 256MB do the job? I ask because those types of servers are available for short-run jobs for pennines per hour and I would be willing to work on the code to fire-up, load them with the test suite, do the run, and shut them down. I happen to have a bunch of code lying around for doing such things... S From mal at egenix.com Tue Oct 13 19:07:22 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 19:07:22 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> Message-ID: <4AD4B3CA.4000409@egenix.com> Barry Warsaw wrote: > On Oct 13, 2009, at 11:01 AM, M.-A. Lemburg wrote: > >> Then I'd suggest to wait another week with 2.6.4 to give you a >> chance to look at the patch. > > That's not a good option, IMO. We have a known broken 2.6.3 out there > and we owe it to our users to correct our mistake and given them the > release they should have gotten in the first place. > > We don't need to wait too long for 2.6.5 though. A few months would be > appropriate. Would it be reasonable to shorten that period, if the fix for the mentioned problem gets ready for prime time earlier ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Tue Oct 13 19:12:42 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 13:12:42 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD4B3CA.4000409@egenix.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> <4AD4B3CA.4000409@egenix.com> Message-ID: <66AEE52E-608D-4A55-A85B-B95C570950D0@python.org> On Oct 13, 2009, at 1:07 PM, M.-A. Lemburg wrote: > Would it be reasonable to shorten that period, if the fix for the > mentioned problem gets ready for prime time earlier ? I think there are many 2.6.x bugs queued up for after 2.6.4 is released. I'm not at all opposed to setting a date approximately 1 month from now for the first 2.6.5 rc. That should give people plenty of time to get their top fixes in. i-like-timed-releases-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From skip at pobox.com Tue Oct 13 19:33:03 2009 From: skip at pobox.com (skip at pobox.com) Date: Tue, 13 Oct 2009 12:33:03 -0500 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD4B3CA.4000409@egenix.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> <4AD4B3CA.4000409@egenix.com> Message-ID: <19156.47567.39139.469228@montanaro.dyndns.org> >> We don't need to wait too long for 2.6.5 though. A few months would be >> appropriate. MAL> Would it be reasonable to shorten that period, if the fix for the MAL> mentioned problem gets ready for prime time earlier ? I think it would be worthwhile to prioritize all outstanding bugs which have been mentioned in the context of 2.6.[345] and run a bug day with the top priority being to fix those bugs. If that task is completed, then move onto other stuff. Once those primary bugs are tackled schedule a 2.6.5 release. Skip From rdmurray at bitdance.com Tue Oct 13 19:41:49 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 13 Oct 2009 13:41:49 -0400 (EDT) Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <52E997AF-A050-41F4-9E2A-36282590B230@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> Message-ID: On Tue, 13 Oct 2009 at 12:57, Barry Warsaw wrote: > On Oct 13, 2009, at 11:16 AM, Tarek Ziad? wrote: >> I still need to do some more tests, I didn't have time to try the >> various projects under win32. >> It's planned to night. >> >> The tests are consisting of compiling and insatling a dozain of >> projects on linux and win32 (and bith when possible) > > Great thanks, let us know how it goes. If there's any question at all about > the fixes that have gone in since rc1, we should spin an rc2. I always thought that the idea of a release candidate was that if there were any fixes _at all_ that there would be a new rc. Only when no bugs needing fixed are found does the rc turn into the actual release. But I understand that this is an idealized rc/release policy :) --David (RDM) From barry at python.org Tue Oct 13 19:45:39 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 13:45:39 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> Message-ID: <39C0375D-0A34-45F8-90AA-A4AA089CC9C5@python.org> On Oct 13, 2009, at 1:41 PM, R. David Murray wrote: > I always thought that the idea of a release candidate was that if > there > were any fixes _at all_ that there would be a new rc. Only when no > bugs needing fixed are found does the rc turn into the actual release. > But I understand that this is an idealized rc/release policy :) I'll invoke the Umbrella Rule now. If we don't do one we'll wish we had. If we do, we won't have needed it. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mal at egenix.com Tue Oct 13 19:54:20 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Oct 2009 19:54:20 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <66AEE52E-608D-4A55-A85B-B95C570950D0@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <4AD48BC4.2060809@egenix.com> <4AD4917D.8090800@v.loewis.de> <4AD49631.70607@egenix.com> <4AD4B3CA.4000409@egenix.com> <66AEE52E-608D-4A55-A85B-B95C570950D0@python.org> Message-ID: <4AD4BECC.6010300@egenix.com> Barry Warsaw wrote: > On Oct 13, 2009, at 1:07 PM, M.-A. Lemburg wrote: > >> Would it be reasonable to shorten that period, if the fix for the >> mentioned problem gets ready for prime time earlier ? > > I think there are many 2.6.x bugs queued up for after 2.6.4 is > released. I'm not at all opposed to setting a date approximately 1 > month from now for the first 2.6.5 rc. That should give people plenty > of time to get their top fixes in. > > i-like-timed-releases-ly y'rs, Agreed. Adding some pressure that way helps :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 13 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Tue Oct 13 20:00:27 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Oct 2009 11:00:27 -0700 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <39C0375D-0A34-45F8-90AA-A4AA089CC9C5@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <39C0375D-0A34-45F8-90AA-A4AA089CC9C5@python.org> Message-ID: On Tue, Oct 13, 2009 at 10:45 AM, Barry Warsaw wrote: > On Oct 13, 2009, at 1:41 PM, R. David Murray wrote: > >> I always thought that the idea of a release candidate was that if there >> were any fixes _at all_ that there would be a new rc. ?Only when no >> bugs needing fixed are found does the rc turn into the actual release. >> But I understand that this is an idealized rc/release policy :) > > I'll invoke the Umbrella Rule now. ?If we don't do one we'll wish we had. > ?If we do, we won't have needed it. Traceback (most recent call last): File "umbrella_rules.py", line 6, in UmbrellaRule() File "unbrella_rules.py", line 4, in UmbrellaRule raise DuplicateRuleName('http://whatplanetisthis.com/2008/02/the-umbrella-rule-and-sharing-rules-in-general/') DuplicateRuleName: http://whatplanetisthis.com/2008/02/the-umbrella-rule-and-sharing-rules-in-general/ -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Tue Oct 13 20:06:29 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 14:06:29 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <39C0375D-0A34-45F8-90AA-A4AA089CC9C5@python.org> Message-ID: On Oct 13, 2009, at 2:00 PM, Guido van Rossum wrote: > Traceback (most recent call last): > File "umbrella_rules.py", line 6, in > UmbrellaRule() > File "unbrella_rules.py", line 4, in UmbrellaRule > raise DuplicateRuleName('http://whatplanetisthis.com/2008/02/the-umbrella-rule-and-sharing-rules-in-general/') > DuplicateRuleName: > http://whatplanetisthis.com/2008/02/the-umbrella-rule-and-sharing-rules-in-general/ MisinformedMeteorologistRule (probably more PC than stupid weatherman rule :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From martin at v.loewis.de Wed Oct 14 00:10:24 2009 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Oct 2009 06:10:24 +0800 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> Message-ID: <4AD4FAD0.1080807@v.loewis.de> > I always thought that the idea of a release candidate was that if there > were any fixes _at all_ that there would be a new rc. Only when no > bugs needing fixed are found does the rc turn into the actual release. This was also my understanding; that's the point of calling it "candidate". Since the code base of 2.6.4rc1 was not released as-is, we would need to consider another candidate. But then, Barry doesn't like release candidates in the first place. Regards, Martin From ziade.tarek at gmail.com Wed Oct 14 00:44:29 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 14 Oct 2009 00:44:29 +0200 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> Message-ID: <94bdd2610910131544h231d07cdqb07d3f6f65eee06d@mail.gmail.com> On Tue, Oct 13, 2009 at 5:16 PM, Tarek Ziad? wrote: > On Tue, Oct 13, 2009 at 4:52 PM, P.J. Eby wrote: >> At 08:27 AM 10/13/2009 -0400, Barry Warsaw wrote: >>> >>> Are we on track to release 2.6.4 final this Sunday or do we need >>> another rc? >>> >>> Yesterday, Tarek committed another setuptools related fix and said >>> that he was going to run a bunch of build tests locally. ?Tarek, how >>> did that go? >> > > I still need to do some more tests, I didn't have time to try the > various projects under win32. > It's planned to night. > > The tests are consisting of compiling and insatling a dozain of > projects on linux and win32 (and bith when possible) Barry, I'll continue my win32 smoke tests tomorrow, it's not over. So far, so good. windows XP (vmware) - VC9 - running setup.py build+install - zope.interface 3.5.2 + setuptools 0.6c9 - Twisted-8.2.0 + setuptools 0.6c9 - pythonwin32-314 +plain distutils: works but I had a failure on 'mc.exe' on one extension which is unrelated obviously. pywin32 builds fine otherwise. Still investigating. on this one Tarek From barry at python.org Wed Oct 14 01:06:38 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 19:06:38 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD4FAD0.1080807@v.loewis.de> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <4AD4FAD0.1080807@v.loewis.de> Message-ID: <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> On Oct 13, 2009, at 6:10 PM, Martin v. L?wis wrote: >> I always thought that the idea of a release candidate was that if >> there >> were any fixes _at all_ that there would be a new rc. Only when no >> bugs needing fixed are found does the rc turn into the actual >> release. > > This was also my understanding; that's the point of calling it > "candidate". Since the code base of 2.6.4rc1 was not released as-is, > we would need to consider another candidate. > > But then, Barry doesn't like release candidates in the first place. No, but let's do one anyway! So, we can either make Sunday's release rc2 and do the final release one week later, or I can try to get an rc2 out in the next day or two, with a final release mid-next week. Thoughts? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From martin at v.loewis.de Wed Oct 14 04:15:55 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Oct 2009 10:15:55 +0800 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <4AD4FAD0.1080807@v.loewis.de> <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> Message-ID: <4AD5345B.4080301@v.loewis.de> > So, we can either make Sunday's release rc2 and do the final release one > week later, or I can try to get an rc2 out in the next day or two, with > a final release mid-next week. > > Thoughts? I won't be able to produce Windows binaries until Saturday. Now sure how that would fit into the "within the next day or two" plan. Regards, Martin From barry at python.org Wed Oct 14 04:45:43 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Oct 2009 22:45:43 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD5345B.4080301@v.loewis.de> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <4AD4FAD0.1080807@v.loewis.de> <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> <4AD5345B.4080301@v.loewis.de> Message-ID: On Oct 13, 2009, at 10:15 PM, Martin v. L?wis wrote: >> So, we can either make Sunday's release rc2 and do the final >> release one >> week later, or I can try to get an rc2 out in the next day or two, >> with >> a final release mid-next week. >> >> Thoughts? > > I won't be able to produce Windows binaries until Saturday. Now sure > how > that would fit into the "within the next day or two" plan. That seems to argue for doing rc2 on Sunday the 18th. If I tag the release some time Saturday, you could have the binaries by Sunday right? Then we'll take one more week and release the final on the 25th. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From anthonybaxter at gmail.com Wed Oct 14 04:37:16 2009 From: anthonybaxter at gmail.com (Anthony Baxter) Date: Wed, 14 Oct 2009 13:37:16 +1100 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <4AD4FAD0.1080807@v.loewis.de> <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> Message-ID: I strongly urge another release candidate. But then, I am not doing the work, so take that advice for what it is... On Oct 14, 2009 10:18 AM, "Barry Warsaw" wrote: On Oct 13, 2009, at 6:10 PM, Martin v. L?wis wrote: >> I always thought that the idea of a release ... No, but let's do one anyway! So, we can either make Sunday's release rc2 and do the final release one week later, or I can try to get an rc2 out in the next day or two, with a final release mid-next week. Thoughts? -Barry _______________________________________________ python-committers mailing list python-committers at python.org http://mail.python.org/mailman/listinfo/python-committers -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Oct 14 14:06:58 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Oct 2009 20:06:58 +0800 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <4AD4FAD0.1080807@v.loewis.de> <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> <4AD5345B.4080301@v.loewis.de> Message-ID: <4AD5BEE2.2000405@v.loewis.de> > That seems to argue for doing rc2 on Sunday the 18th. If I tag the > release some time Saturday, you could have the binaries by Sunday > right? Correct. Regards, Martin From barry at python.org Wed Oct 14 14:48:01 2009 From: barry at python.org (Barry Warsaw) Date: Wed, 14 Oct 2009 08:48:01 -0400 Subject: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday? In-Reply-To: <4AD5BEE2.2000405@v.loewis.de> References: <430E0910-C2A1-48F7-B036-743BA5BA91A3@python.org> <20091013145211.8DB243A4068@sparrow.telecommunity.com> <94bdd2610910130816g42826010w3d20901683678ed3@mail.gmail.com> <52E997AF-A050-41F4-9E2A-36282590B230@python.org> <4AD4FAD0.1080807@v.loewis.de> <79DD5085-8D85-43F7-87B7-B378D76ECA64@python.org> <4AD5345B.4080301@v.loewis.de> <4AD5BEE2.2000405@v.loewis.de> Message-ID: <9185B478-1C1B-4F26-A20D-24834EBFD898@python.org> On Oct 14, 2009, at 8:06 AM, Martin v. L?wis wrote: >> That seems to argue for doing rc2 on Sunday the 18th. If I tag the >> release some time Saturday, you could have the binaries by Sunday >> right? > > Correct. Cool. I've updated the Python Release Schedule calendar to reflect the new dates. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From nas at arctrix.com Wed Oct 14 22:16:35 2009 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 14 Oct 2009 14:16:35 -0600 Subject: [Python-Dev] Better module shutdown procedure Message-ID: <20091014201635.GA28053@arctrix.com> The current shutdown code in pythonrun.c zaps module globals by setting them to None (an attempt to break reference cycles). That causes problems since __del__ methods can try to use the globals after they have been set to None. The procedure implemented by http://bugs.python.org/issue812369 seems to be a better idea. References to modules are replaced by weak references and the GC is allowed to cleanup reference cycles. I would like to commit this change but I'm not sure if it is a good time in the development cycle or if anyone has objections to the idea. Please speak up if you have input. Neil From exarkun at twistedmatrix.com Wed Oct 14 22:35:28 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 14 Oct 2009 20:35:28 -0000 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014201635.GA28053@arctrix.com> References: <20091014201635.GA28053@arctrix.com> Message-ID: <20091014203528.8004.1935283805.divmod.xquotient.44@localhost.localdomain> On 08:16 pm, nas at arctrix.com wrote: >The current shutdown code in pythonrun.c zaps module globals by >setting them to None (an attempt to break reference cycles). That >causes problems since __del__ methods can try to use the globals >after they have been set to None. > >The procedure implemented by http://bugs.python.org/issue812369 >seems to be a better idea. References to modules are replaced by >weak references and the GC is allowed to cleanup reference cycles. > >I would like to commit this change but I'm not sure if it is a good >time in the development cycle or if anyone has objections to the >idea. Please speak up if you have input. I notice that the patch doesn't include any unit tests for the feature being provided (it does change test_sys.py - although I don't understand how that change is related to this feature). I hope that regardless of whatever else is decided, if the change is to be made, it will be accompanied by new unit tests verify its proper operation. Jean-Paul From solipsis at pitrou.net Wed Oct 14 22:49:59 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Oct 2009 20:49:59 +0000 (UTC) Subject: [Python-Dev] Better module shutdown procedure References: <20091014201635.GA28053@arctrix.com> Message-ID: Neil Schemenauer arctrix.com> writes: > > I would like to commit this change but I'm not sure if it is a good > time in the development cycle or if anyone has objections to the > idea. Please speak up if you have input. Have you tried to remove the various hacks/workarounds in the stdlib that currently avoid module finalization problems? (take a look at e.g. threading.py) It would be a good validation that the patch improves on the current behaviour. Regards Antoine. From daniel at stutzbachenterprises.com Wed Oct 14 23:13:12 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 14 Oct 2009 16:13:12 -0500 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014201635.GA28053@arctrix.com> References: <20091014201635.GA28053@arctrix.com> Message-ID: On Wed, Oct 14, 2009 at 3:16 PM, Neil Schemenauer wrote: > The current shutdown code in pythonrun.c zaps module globals by > setting them to None (an attempt to break reference cycles). That > causes problems since __del__ methods can try to use the globals > after they have been set to None. > > The procedure implemented by http://bugs.python.org/issue812369 > seems to be a better idea. References to modules are replaced by > weak references and the GC is allowed to cleanup reference cycles. > Based on the description, it still resorts to zapping module globals by setting them to None. It zaps them to weakrefs first, which means that globals are more likely to be valid during __del__, but it still cannot make any guarantees and referencing globals from __del__ is still a bad idea. Is that a correct synopsis? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Wed Oct 14 23:42:30 2009 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 14 Oct 2009 15:42:30 -0600 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: References: <20091014201635.GA28053@arctrix.com> Message-ID: <20091014214230.GA28424@arctrix.com> On Wed, Oct 14, 2009 at 04:13:12PM -0500, Daniel Stutzbach wrote: > Based on the description, it still resorts to zapping module globals by > setting them to None. It zaps them to weakrefs first, which means that > globals are more likely to be valid during __del__, but it still cannot make > any guarantees and referencing globals from __del__ is still a bad idea. Is > that a correct synopsis? Yes, it does still resort to setting globals to None. However, the weakref step makes it much more likely that __del__ methods run before that happens. After this change, referencing global variables from __del__ methods is okay. What is not a good idea is creating __del__ methods that are part of a reference cycle (i.e. an island of references) and also refer to that cycle somehow. That has never been a good idea and those __del__ methods will never get run, before or after the proposed change. HTH, Neil From nas at arctrix.com Wed Oct 14 23:45:06 2009 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 14 Oct 2009 15:45:06 -0600 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014203528.8004.1935283805.divmod.xquotient.44@localhost.localdomain> References: <20091014201635.GA28053@arctrix.com> <20091014203528.8004.1935283805.divmod.xquotient.44@localhost.localdomain> Message-ID: <20091014214506.GB28424@arctrix.com> On Wed, Oct 14, 2009 at 08:35:28PM -0000, exarkun at twistedmatrix.com wrote: > I notice that the patch doesn't include any unit tests for the feature > being provided That's hard to do although I suppose not impossible. We would be trying to test the interpreter shutdown procedure. I suppose on plaforms that support it we could fork a subprocess and then inspect the output from __del__ methods. > (it does change test_sys.py - although I don't understand how > that change is related to this feature). That's a spurious change and will be left out of the real fix. Neil From glyph at twistedmatrix.com Thu Oct 15 00:25:35 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 14 Oct 2009 15:25:35 -0700 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014214506.GB28424@arctrix.com> References: <20091014201635.GA28053@arctrix.com> <20091014203528.8004.1935283805.divmod.xquotient.44@localhost.localdomain> <20091014214506.GB28424@arctrix.com> Message-ID: On Wed, Oct 14, 2009 at 2:45 PM, Neil Schemenauer wrote: > On Wed, Oct 14, 2009 at 08:35:28PM -0000, exarkun at twistedmatrix.com wrote: > > I notice that the patch doesn't include any unit tests for the feature > > being provided > > That's hard to do although I suppose not impossible. We would be > trying to test the interpreter shutdown procedure. I suppose on > plaforms that support it we could fork a subprocess and then inspect > the output from __del__ methods. Why not just expose the module-teardown procedure and call it on a module, then inspect the state there as it's being torn down? -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel at stutzbachenterprises.com Thu Oct 15 00:35:38 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 14 Oct 2009 17:35:38 -0500 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014214230.GA28424@arctrix.com> References: <20091014201635.GA28053@arctrix.com> <20091014214230.GA28424@arctrix.com> Message-ID: On Wed, Oct 14, 2009 at 4:42 PM, Neil Schemenauer wrote: > Yes, it does still resort to setting globals to None. However, the > weakref step makes it much more likely that __del__ methods run > before that happens. After this change, referencing global variables > from __del__ methods is okay. > The first and last sentences seem like a contradiction to me. If I cannot guarantee that globals will be valid when __del__ is executed, then I must not reference globals from __del__. I think I'm missing something here. Is there some way the programmer can determine that referencing a global variable in __del__ will be 100% safe? (not just "likely") -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Thu Oct 15 01:05:56 2009 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 14 Oct 2009 17:05:56 -0600 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: References: <20091014201635.GA28053@arctrix.com> <20091014214230.GA28424@arctrix.com> Message-ID: <20091014230556.GA28661@arctrix.com> On Wed, Oct 14, 2009 at 05:35:38PM -0500, Daniel Stutzbach wrote: > The first and last sentences seem like a contradiction to me. If I cannot > guarantee that globals will be valid when __del__ is executed, then I must > not reference globals from __del__. I think there is confusion about the word "reference". In the method: def __del__(): print sys.version the global variable reference to 'sys' is not a reference on the GC referencing counting sense. IOW, it does not result in a a Py_INCREF while the function is not being executed and therefore should be safe after the proposed change. Currently, it could result in 'None' being printed. HTH, Neil From daniel at stutzbachenterprises.com Thu Oct 15 01:27:37 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Wed, 14 Oct 2009 18:27:37 -0500 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014230556.GA28661@arctrix.com> References: <20091014201635.GA28053@arctrix.com> <20091014214230.GA28424@arctrix.com> <20091014230556.GA28661@arctrix.com> Message-ID: On Wed, Oct 14, 2009 at 6:05 PM, Neil Schemenauer wrote: > def __del__(): > print sys.version > > the global variable reference to 'sys' is not a reference on the GC > referencing counting sense. IOW, it does not result in a a Py_INCREF > while the function is not being executed and therefore should be > safe after the proposed change. Currently, it could result in 'None' > being printed. > Currently it throws an exception since "sys" is None. :-) Here is my understanding of the proposed procedure: 1. Replace modules in sys.modules with weakrefs 2. Run the garbage collector 3. Replace globals in any remaining modules with None 4. Run the garbage collector Is it possible for a __del__ method to be called in step 4 or not? I am still unclear on this point. :-) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Thu Oct 15 01:49:04 2009 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 14 Oct 2009 23:49:04 +0000 (UTC) Subject: [Python-Dev] Better module shutdown procedure References: <20091014201635.GA28053@arctrix.com> Message-ID: Antoine Pitrou wrote: > Have you tried to remove the various hacks/workarounds in the > stdlib that currently avoid module finalization problems? Good idea. These hacks are the best argument for applying the change, IMHO. I made a quick review of Lib modules using __del__: tarfile: looks not safe tempfile: goes through elaborate motions to avoid globals threading: yikes, I didn't attempt to decypher this one shelve: probably not safe dumbdb: avoids globals, easy to remove hacks _pyio: try/except can be removed (hopefully) urllib: avoids globals, easy to remove hacks Will try to cook up a patch in my copious free time. Neil From solipsis at pitrou.net Thu Oct 15 14:42:51 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Oct 2009 12:42:51 +0000 (UTC) Subject: [Python-Dev] Better module shutdown procedure References: <20091014201635.GA28053@arctrix.com> <20091014214230.GA28424@arctrix.com> <20091014230556.GA28661@arctrix.com> Message-ID: Le Wed, 14 Oct 2009 18:27:37 -0500, Daniel Stutzbach a ?crit?: > > Here is my understanding of the proposed procedure: > > 1. Replace modules in sys.modules with weakrefs 2. Run the garbage > collector > 3. Replace globals in any remaining modules with None 4. Run the garbage > collector > > Is it possible for a __del__ method to be called in step 4 or not? I am > still unclear on this point. :-) If an object is holding a reference to a module (e.g. self._sys = sys), then yes it should be possible because the reference has been creating a cycle. From jnoller at gmail.com Thu Oct 15 15:05:34 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 15 Oct 2009 09:05:34 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> Message-ID: <4222a8490910150605x395465csa68177492dd275bd@mail.gmail.com> On Wed, Oct 7, 2009 at 2:09 PM, Barry Warsaw wrote: > On Oct 7, 2009, at 1:26 PM, Scott Dial wrote: > >> I suspect this release is primarily to quench the problems with >> distutils, but.. >> >> http://bugs.python.org/issue5949 >> >> doesn't seem to have been addressed by you. And this seems like it would >> be another unfortunate loss of an opportunity. > > I want us to be very careful about 2.6.4. ?This isn't a normal bug fix > release, it's specifically there to remove the brown paper bag of 2.6.3 from > our heads. ?So let's be conservative and fix this one in 2.6.5. > > -Barry > Here's another one barry: http://bugs.python.org/issue7120 We should get this in - it's a regression I introduced awhile ago for environments without the multiprocessing module using logging. jesse From skip at pobox.com Thu Oct 15 13:39:30 2009 From: skip at pobox.com (skip at pobox.com) Date: Thu, 15 Oct 2009 06:39:30 -0500 (CDT) Subject: [Python-Dev] Can 3.1 still be built without complex? Message-ID: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> I notice that WITHOUT_COMPLEX still appears in Python.h and several .c files but nowhere else in the 2.6, 2.7 or 3.1 source, most particularly not in configure or pyconfig.h.in. Are builds --without-complex still supported? Has it been tested at any time in the recent past? Skip From barry at python.org Thu Oct 15 15:40:43 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 15 Oct 2009 09:40:43 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: <4222a8490910150605x395465csa68177492dd275bd@mail.gmail.com> References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> <4222a8490910150605x395465csa68177492dd275bd@mail.gmail.com> Message-ID: On Oct 15, 2009, at 9:05 AM, Jesse Noller wrote: > Here's another one barry: > > http://bugs.python.org/issue7120 > > We should get this in - it's a regression I introduced awhile ago for > environments without the multiprocessing module using logging. Was this a regression in 2.6.2 or 2.6.3? I think it might not matter, but let me know and I'll decide. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From eric at trueblade.com Thu Oct 15 15:48:34 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 15 Oct 2009 09:48:34 -0400 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> Message-ID: <4AD72832.90409@trueblade.com> skip at pobox.com wrote: > I notice that WITHOUT_COMPLEX still appears in Python.h and several .c files > but nowhere else in the 2.6, 2.7 or 3.1 source, most particularly not in > configure or pyconfig.h.in. Are builds --without-complex still supported? > Has it been tested at any time in the recent past? I haven't tested it, but I still see WITHOUT_COMPLEX in trunk and py3k branches. In py3k, it's referenced in: ./Include/Python.h ./Misc/HISTORY ./Objects/complexobject.c ./Objects/object.c ./Parser/tokenizer.c ./Python/ast.c ./Python/bltinmodule.c ./Python/getargs.c ./Python/marshal.c ./Python/modsupport.c I checked complexobject.c, and it looks like it's used correctly there. Eric. From jnoller at gmail.com Thu Oct 15 16:00:40 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 15 Oct 2009 10:00:40 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> <4222a8490910150605x395465csa68177492dd275bd@mail.gmail.com> Message-ID: <4222a8490910150700u358568b2l1cf8ba623f14c22f@mail.gmail.com> On Thu, Oct 15, 2009 at 9:40 AM, Barry Warsaw wrote: > On Oct 15, 2009, at 9:05 AM, Jesse Noller wrote: > >> Here's another one barry: >> >> http://bugs.python.org/issue7120 >> >> We should get this in - it's a regression I introduced awhile ago for >> environments without the multiprocessing module using logging. > > Was this a regression in 2.6.2 or 2.6.3? ?I think it might not matter, but > let me know and I'll decide. > > -Barry > > My original commit was on 2009-01-18, you tagged 2.6.2 on Tue Apr 14 13:17:30 2009, so it went into 2.6.2. The fix is minor, the bug affects things such as GAE which may use logging, but not have multiprocessing available. jesse From barry at python.org Thu Oct 15 16:14:46 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 15 Oct 2009 10:14:46 -0400 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: <4222a8490910150700u358568b2l1cf8ba623f14c22f@mail.gmail.com> References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> <4222a8490910150605x395465csa68177492dd275bd@mail.gmail.com> <4222a8490910150700u358568b2l1cf8ba623f14c22f@mail.gmail.com> Message-ID: On Oct 15, 2009, at 10:00 AM, Jesse Noller wrote: > On Thu, Oct 15, 2009 at 9:40 AM, Barry Warsaw > wrote: >> On Oct 15, 2009, at 9:05 AM, Jesse Noller wrote: >> >>> Here's another one barry: >>> >>> http://bugs.python.org/issue7120 >>> >>> We should get this in - it's a regression I introduced awhile ago >>> for >>> environments without the multiprocessing module using logging. >> >> Was this a regression in 2.6.2 or 2.6.3? I think it might not >> matter, but >> let me know and I'll decide. >> >> -Barry >> >> > > My original commit was on 2009-01-18, you tagged 2.6.2 on Tue Apr 14 > 13:17:30 2009, so it went into 2.6.2. > > The fix is minor, the bug affects things such as GAE which may use > logging, but not have multiprocessing available. Approved for 2.6.4rc2 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From skip at pobox.com Thu Oct 15 16:28:47 2009 From: skip at pobox.com (skip at pobox.com) Date: Thu, 15 Oct 2009 09:28:47 -0500 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: <4AD72832.90409@trueblade.com> References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> <4AD72832.90409@trueblade.com> Message-ID: <19159.12703.850168.592693@montanaro.dyndns.org> Eric> I haven't tested it, but I still see WITHOUT_COMPLEX in trunk and py3k Eric> branches. In py3k, it's referenced in: ... Sure, but is it ever exercised? A name like WITHOUT_COMPLEX suggests that it should be flipped on/off by configure using --without-complex, but that script doesn't know about it and it's not mentioned in pyconfig.h.in, where the various --with-* flags work their magic. Skip From eric at trueblade.com Thu Oct 15 16:35:30 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 15 Oct 2009 10:35:30 -0400 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: <19159.12703.850168.592693@montanaro.dyndns.org> References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> <4AD72832.90409@trueblade.com> <19159.12703.850168.592693@montanaro.dyndns.org> Message-ID: <4AD73332.50506@trueblade.com> skip at pobox.com wrote: > Eric> I haven't tested it, but I still see WITHOUT_COMPLEX in trunk and py3k > Eric> branches. In py3k, it's referenced in: > ... > > Sure, but is it ever exercised? A name like WITHOUT_COMPLEX suggests that > it should be flipped on/off by configure using --without-complex, but that > script doesn't know about it and it's not mentioned in pyconfig.h.in, where > the various --with-* flags work their magic. Ah, I misunderstood. I thought you were asking "does setting WITHOUT_COMPLEX work?" No idea about any configure option, sorry. Eric. From dickinsm at gmail.com Thu Oct 15 16:46:55 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 15 Oct 2009 15:46:55 +0100 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> Message-ID: <5c6f2a5d0910150746u41d163dby369a2f6d86dd8075@mail.gmail.com> [I originally sent this reply to Skip instead of to the list; apologies.] On Thu, Oct 15, 2009 at 12:39 PM, wrote: > I notice that WITHOUT_COMPLEX still appears in Python.h and several .c files > but nowhere else in the 2.6, 2.7 or 3.1 source, most particularly not in > configure or pyconfig.h.in. Are builds --without-complex still supported? > Has it been tested at any time in the recent past? Apparently not. :) I just tried the following with an svn checkout of trunk (r75433), on OS X 10.6: dickinsm$ CC='gcc -DWITHOUT_COMPLEX' ./configure && make The build fails with: gcc -DWITHOUT_COMPLEX -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/compile.o Python/compile.c Python/compile.c: In function ?compiler_add_o?: Python/compile.c:914: error: ?Py_complex? undeclared (first use in this function) Python/compile.c:914: error: (Each undeclared identifier is reported only once Python/compile.c:914: error: for each function it appears in.) Python/compile.c:914: error: expected ?;? before ?z? Python/compile.c:931: warning: implicit declaration of function ?PyComplex_Check? Python/compile.c:937: error: ?z? undeclared (first use in this function) Python/compile.c:937: warning: implicit declaration of function ?PyComplex_AsCComplex? make: *** [Python/compile.o] Error 1 Mark Postscript: the above compilation failure is easily fixed. The next failure is: gcc -DWITHOUT_COMPLEX -u _PyMac_Error -o python.exe \ Modules/python.o \ libpython2.7.a -ldl Undefined symbols: "_PyComplex_RealAsDouble", referenced from: __PyComplex_FormatAdvanced in libpython2.7.a(formatter_string.o) "_PyComplex_ImagAsDouble", referenced from: __PyComplex_FormatAdvanced in libpython2.7.a(formatter_string.o) ld: symbol(s) not found collect2: ld returned 1 exit status make: *** [python.exe] Error 1 From solipsis at pitrou.net Thu Oct 15 17:06:27 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Oct 2009 15:06:27 +0000 (UTC) Subject: [Python-Dev] Can 3.1 still be built without complex? References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > I notice that WITHOUT_COMPLEX still appears in Python.h and several .c files > but nowhere else in the 2.6, 2.7 or 3.1 source, most particularly not in > configure or pyconfig.h.in. Are builds --without-complex still supported? > Has it been tested at any time in the recent past? Is there any point in building without complex? Size reduction perhaps? If nobody uses it, we could remove that option. We have trouble staying compatible with lots of build options (see how --without-threads is little exercised). Regards Antoine. From fdrake at gmail.com Thu Oct 15 17:25:29 2009 From: fdrake at gmail.com (Fred Drake) Date: Thu, 15 Oct 2009 11:25:29 -0400 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> Message-ID: <9cee7ab80910150825g3118325fjc4c76293de86e1c6@mail.gmail.com> On Thu, Oct 15, 2009 at 11:06 AM, Antoine Pitrou wrote: > We have trouble staying > compatible with lots of build options (see how --without-threads is little > exercised). I don't know if folks deploying on embedded platforms are using these options (directly or indirectly), but expect that's the primary audience. Hopefully somebody who's doing that can tell us if they're using the defines, and what versions of Python they're currently using. Then we can determine whether the should be supported. If they should be, there should be buildbot's set up for relevant configurations. Otherwise removal is best. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From guido at python.org Thu Oct 15 19:57:32 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Oct 2009 10:57:32 -0700 Subject: [Python-Dev] Python 2.6.4rc1 In-Reply-To: References: <4ACCCF29.7030003@scottdial.com> <8F1F8793-79E0-459C-9D6A-55AB79CD9EEC@python.org> <4222a8490910150605x395465csa68177492dd275bd@mail.gmail.com> <4222a8490910150700u358568b2l1cf8ba623f14c22f@mail.gmail.com> Message-ID: Thanks all! On Thu, Oct 15, 2009 at 7:14 AM, Barry Warsaw wrote: > On Oct 15, 2009, at 10:00 AM, Jesse Noller wrote: > >> On Thu, Oct 15, 2009 at 9:40 AM, Barry Warsaw wrote: >>> >>> On Oct 15, 2009, at 9:05 AM, Jesse Noller wrote: >>> >>>> Here's another one barry: >>>> >>>> http://bugs.python.org/issue7120 >>>> >>>> We should get this in - it's a regression I introduced awhile ago for >>>> environments without the multiprocessing module using logging. >>> >>> Was this a regression in 2.6.2 or 2.6.3? ?I think it might not matter, >>> but >>> let me know and I'll decide. >>> >>> -Barry >>> >>> >> >> My original commit was on 2009-01-18, you tagged 2.6.2 on Tue Apr 14 >> 13:17:30 2009, so it went into 2.6.2. >> >> The fix is minor, the bug affects things such as GAE which may use >> logging, but not have multiprocessing available. > > Approved for 2.6.4rc2 > > -Barry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From dickinsm at gmail.com Thu Oct 15 21:11:25 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 15 Oct 2009 20:11:25 +0100 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> Message-ID: <5c6f2a5d0910151211g199f5efq7aa4ff8f92f07daf@mail.gmail.com> On Thu, Oct 15, 2009 at 4:06 PM, Antoine Pitrou wrote: > pobox.com> writes: >> >> I notice that WITHOUT_COMPLEX still appears in Python.h and several .c files >> but nowhere else in the 2.6, 2.7 or 3.1 source, most particularly not in >> configure or pyconfig.h.in. ?Are builds --without-complex still supported? >> Has it been tested at any time in the recent past? > > Is there any point in building without complex? Size reduction perhaps? > If nobody uses it, we could remove that option. We have trouble staying > compatible with lots of build options (see how --without-threads is little > exercised). Size reduction is the only point I can think of. There's one respect in which complex is slightly more tightly integrated in py3k than in trunk: raising a negative number to a non-integer power (e.g., (-1)**0.5) gives a complex result in py3k. In trunk this raises ValueError, which means that the only way to get a complex number in trunk is if you explicitly ask for one somehow (e.g., by invoking complex, or using the cmath module, or using imaginary literals, ...), so it makes slightly more sense to build without the complex type there. +1 for removing the WITHOUT_COMPLEX define from py3k. -0 for removing it from trunk. Mark From solipsis at pitrou.net Thu Oct 15 21:17:42 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Oct 2009 19:17:42 +0000 (UTC) Subject: [Python-Dev] Can 3.1 still be built without complex? References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> <5c6f2a5d0910151211g199f5efq7aa4ff8f92f07daf@mail.gmail.com> Message-ID: Mark Dickinson gmail.com> writes: > > There's one respect in which complex is slightly more tightly > integrated in py3k than in trunk: raising a negative number to a > non-integer power (e.g., (-1)**0.5) gives a complex result in py3k. >>> (-1)**.5 (6.123031769111886e-17+1j) Don't we have a precision problem here? 0.5 is supposed to be represented exactly, isn't it? From dickinsm at gmail.com Thu Oct 15 21:42:50 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 15 Oct 2009 20:42:50 +0100 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> <5c6f2a5d0910151211g199f5efq7aa4ff8f92f07daf@mail.gmail.com> Message-ID: <5c6f2a5d0910151242q2bc80fdbq3d71fb16ca6a3669@mail.gmail.com> On Thu, Oct 15, 2009 at 8:17 PM, Antoine Pitrou wrote: >>>> (-1)**.5 > (6.123031769111886e-17+1j) > > Don't we have a precision problem here? 0.5 is supposed to be represented > exactly, isn't it? 0.5 is represented exactly, but complex.__pow__ makes no pretence of being correctly rounded (and making it correctly rounded would likely be prohibitively expensive in terms of code size and complexity). It's using something like x**y = exp(y*log(x)) behind the scenes, at least for computing the argument of the result. For square roots, cmath.sqrt produces better results. Mark From skip.montanaro at gmail.com Fri Oct 16 01:47:14 2009 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 15 Oct 2009 18:47:14 -0500 Subject: [Python-Dev] Can 3.1 still be built without complex? In-Reply-To: <5c6f2a5d0910151242q2bc80fdbq3d71fb16ca6a3669@mail.gmail.com> References: <20091015113930.7B0D812C1BE3@montanaro.dyndns.org> <5c6f2a5d0910151211g199f5efq7aa4ff8f92f07daf@mail.gmail.com> <5c6f2a5d0910151242q2bc80fdbq3d71fb16ca6a3669@mail.gmail.com> Message-ID: <60bb7ceb0910151647k2b311e30x70fe6355c09a9f9a@mail.gmail.com> http://bugs.python.org/issue7147 Passes test (no big surprise there). Doesn't yet include any changes to documentation or Misc/NEWS entry. Should this be decided I will take a look at that. Skip From nas at arctrix.com Fri Oct 16 03:06:59 2009 From: nas at arctrix.com (Neil Schemenauer) Date: Thu, 15 Oct 2009 19:06:59 -0600 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091014201635.GA28053@arctrix.com> References: <20091014201635.GA28053@arctrix.com> Message-ID: <20091016010659.GA4189@arctrix.com> On Wed, Oct 14, 2009 at 02:16:35PM -0600, Neil Schemenauer wrote: > The procedure implemented by http://bugs.python.org/issue812369 > seems to be a better idea. After some experimentation I realize this idea is not ready yet. The main problem comes from references to Python objects that modules keep but don't expose to the garbage collector. For example, gcmodule.c has a static pointer "tmod" that is a reference to the "time" module. This reference prevents the "time" module from being freed during interpreter shutdown. Ideally, I suppose modules should be treated like any other object and have tp_traverse and tp_clear methods that deal with these sorts of pointers. They would have to delegated to the instance since each module would have its own implementation. Neil From amauryfa at gmail.com Fri Oct 16 10:01:07 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 16 Oct 2009 10:01:07 +0200 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: <20091016010659.GA4189@arctrix.com> References: <20091014201635.GA28053@arctrix.com> <20091016010659.GA4189@arctrix.com> Message-ID: 2009/10/16 Neil Schemenauer : > After some experimentation I realize this idea is not ready yet. > The main problem comes from references to Python objects that > modules keep but don't expose to the garbage collector. For example, > gcmodule.c has a static pointer "tmod" that is a reference to the > "time" module. This reference prevents the "time" module from being > freed during interpreter shutdown. > > Ideally, I suppose modules should be treated like any other object > and have tp_traverse and tp_clear methods that deal with these sorts > of pointers. They would have to delegated to the instance since each > module would have its own implementation. Note since python 3.0 (and PEP 3121), the PyModuleDef structure has some members like m_traverse, m_clear and m_free for this very purpose. So far, nobody cared to implement these methods for any module. Maybe one should start at least for static PyObject* that contain references to modules. -- Amaury Forgeot d'Arc From benjamin at python.org Fri Oct 16 16:11:17 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 16 Oct 2009 09:11:17 -0500 Subject: [Python-Dev] Better module shutdown procedure In-Reply-To: References: <20091014201635.GA28053@arctrix.com> <20091016010659.GA4189@arctrix.com> Message-ID: <1afaf6160910160711jd83c02ah1498d13c087cc9a6@mail.gmail.com> 2009/10/16 Amaury Forgeot d'Arc : > 2009/10/16 Neil Schemenauer : >> After some experimentation I realize this idea is not ready yet. >> The main problem comes from references to Python objects that >> modules keep but don't expose to the garbage collector. For example, >> gcmodule.c has a static pointer "tmod" that is a reference to the >> "time" module. This reference prevents the "time" module from being >> freed during interpreter shutdown. >> >> Ideally, I suppose modules should be treated like any other object >> and have tp_traverse and tp_clear methods that deal with these sorts >> of pointers. They would have to delegated to the instance since each >> module would have its own implementation. > > Note since python 3.0 (and PEP 3121), the PyModuleDef structure has > some members like m_traverse, m_clear and m_free for this very > purpose. > So far, nobody cared to implement these methods for any module. Maybe > one should start at least for static PyObject* that contain references > to modules. I believe the implementation is buggy because modules (and their states) can easily be finalized before the objects contained in them. For example, when I tried to convert the _io module to use a state, it resulted in segfaults when the dealloc methods of objects tried to use objects in a state, which had already been deallocated. -- Regards, Benjamin From status at bugs.python.org Fri Oct 16 18:07:37 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 16 Oct 2009 18:07:37 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20091016160737.B03477803E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (10/09/09 - 10/16/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2468 open (+37) / 16504 closed (+20) / 18972 total (+57) Open issues with patches: 982 Average duration of open issues: 669 days. Median duration of open issues: 423 days. Open Issues Breakdown open 2432 (+37) pending 35 ( +0) Issues Created Or Reopened (58) _______________________________ Python 2.6.3 / setuptools 0.6c9: extension module builds fail wi 10/12/09 CLOSED http://bugs.python.org/issue7064 reopened barry Add alternate float formatting styles to new-style formatting. 10/09/09 http://bugs.python.org/issue7094 created mark.dickinson Multiprocessing.Array(lock=False) fails 10/09/09 CLOSED http://bugs.python.org/issue7095 created schlesin test_curses fails on 3.1 when run under regrtest 10/10/09 http://bugs.python.org/issue7096 created r.david.murray urlparse.urljoin of simple "http://" and "somedomain.com" produc 10/10/09 CLOSED http://bugs.python.org/issue7097 created ssteiner g formatting for decimal types should always strip trailing zero 10/10/09 http://bugs.python.org/issue7098 created mark.dickinson patch Decimal.is_normal should return True even for numbers with expon 10/10/09 http://bugs.python.org/issue7099 created mark.dickinson test_xmlrpc: global name 'stop_serving' is not defined 10/10/09 http://bugs.python.org/issue7100 created pitrou tarfile: OSError with TarFile.add(..., recursive=True) about non 10/10/09 http://bugs.python.org/issue7101 created denis Problems building pyhton from source on Snow Leopard (Mac OS X 10/11/09 http://bugs.python.org/issue7102 created FredrikHedman Error in config parser example (w/ Patch) 10/11/09 CLOSED http://bugs.python.org/issue7103 created tcourbon patch test_descr uses __cmp__ which will never be called 10/11/09 CLOSED http://bugs.python.org/issue7104 created stutzbach weak dict iterators are fragile because of unpredictable GC runs 10/11/09 http://bugs.python.org/issue7105 created pitrou patch Python crashed in the middle of an argument between wx and boa_c 10/11/09 CLOSED http://bugs.python.org/issue7106 created bvn Missing uninstallation instructions for mac 10/11/09 http://bugs.python.org/issue7107 created gradha test_commands.py failing on OS X 10.5.7 due to '@' in ls output 10/11/09 http://bugs.python.org/issue7108 created minge patch broken link on news page 10/12/09 CLOSED http://bugs.python.org/issue7109 created mkiever Output test failures on stderr in regrtest.py 10/12/09 http://bugs.python.org/issue7110 created ezio.melotti core dump when stderr is moved 10/12/09 http://bugs.python.org/issue7111 created petere unicodetype_db.h warning: integer constant is too large for 'lon 10/12/09 CLOSED http://bugs.python.org/issue7112 created ocean-city ConfigParser load speedup 10/12/09 http://bugs.python.org/issue7113 created aioryi patch HTMLParser doesn't handle 10/12/09 http://bugs.python.org/issue7114 created ggbaker extension module builds fail when using paths in the extension n 10/12/09 CLOSED http://bugs.python.org/issue7115 created tarek str.join() should be documented as taking an iterable 10/12/09 CLOSED http://bugs.python.org/issue7116 created jess.austin patch Backport py3k float repr to trunk 10/13/09 http://bugs.python.org/issue7117 created mark.dickinson urllib2.urlopen() timeout argument ignored after redirect 10/13/09 http://bugs.python.org/issue7118 created gthaler email: msg.items() returns different values before and after ms 10/13/09 http://bugs.python.org/issue7119 created jnelson logging depends on multiprocessing 10/14/09 CLOSED http://bugs.python.org/issue7120 created gvanrossum ImportError of urllib.request & http.client under PYTHONHOME/Lib 10/14/09 CLOSED http://bugs.python.org/issue7121 created mikezp59 multiprocessing.Pool() problem in windows 10/14/09 CLOSED http://bugs.python.org/issue7122 created SirLalala Multiprocess Process does not always exit when run from a thread 10/14/09 http://bugs.python.org/issue7123 created pajs at fodder.org.uk idle.py -n : help() doesn't work in a reopened shell window 10/14/09 http://bugs.python.org/issue7124 created gregorlingl typo (English) in threading.py 10/14/09 CLOSED http://bugs.python.org/issue7125 created Yinon Contradictory documentation for os.putenv and os.system 10/14/09 CLOSED http://bugs.python.org/issue7126 created stutzbach regrtest -j fails when tests write to stderr 10/14/09 http://bugs.python.org/issue7127 created r.david.murray easy cPickle looking for non-existent package copyreg 10/14/09 http://bugs.python.org/issue7128 created joequant patch, easy 'filling' missing in __all__ ot turtle.py 10/14/09 CLOSED http://bugs.python.org/issue7129 created gregorlingl patch json uses sre_* modules which may not work on alternate implemen 10/14/09 CLOSED http://bugs.python.org/issue7130 created DinoV Extension module build fails for MinGW: missing vcvarsall.bat 10/14/09 http://bugs.python.org/issue7131 created dieterv Regexp: capturing groups in repetitions 10/14/09 http://bugs.python.org/issue7132 created verdy_p test_ssl failure 10/14/09 http://bugs.python.org/issue7133 created pitrou patch Add looping capability to regrtest 10/14/09 http://bugs.python.org/issue7134 created jnoller AttributeError: 'module' object has no attribute 'HTTPSConnectio 10/14/09 CLOSED http://bugs.python.org/issue7135 created awelch Idle File Menu Option Improvement 10/15/09 http://bugs.python.org/issue7136 created tjd Socket documentation not updated 10/15/09 http://bugs.python.org/issue7137 created yang elementtree segfaults on invalid xml declaration 10/15/09 http://bugs.python.org/issue7138 created whichlinden ElementTree: Incorrect serialization of end-of-line characters i 10/15/09 http://bugs.python.org/issue7139 created moriyoshi imp.new_module does not function correctly if the module is retu 10/15/09 CLOSED http://bugs.python.org/issue7140 created jkp 2to3 should add from __future__ import print_statement 10/15/09 http://bugs.python.org/issue7141 created stutzbach __repr__ of memoryview object has type unicode; should be str. 10/15/09 CLOSED http://bugs.python.org/issue7142 created mark.dickinson patch get_payload(decode=True) eats last newline 10/15/09 http://bugs.python.org/issue7143 created athomas imp.load_module in thread causes core dump on OSX 10.6 10/15/09 http://bugs.python.org/issue7144 created peterhunt UTF-16 BOM is not written by socket.makefile (but is expected by 10/15/09 http://bugs.python.org/issue7145 created yang [PATCH] platform.uname()[4] returns 'amd64' on Windows and 'x86- 10/15/09 http://bugs.python.org/issue7146 created zooko Remove WITHOUT_COMPLEX from 3.x trunk 10/15/09 http://bugs.python.org/issue7147 created skip.montanaro patch, patch, easy, needs review socket.py: r75412 broke build 10/16/09 CLOSED http://bugs.python.org/issue7148 created kbk 2.6.4rc1 regression: test_urllib2 fails on OS X with UnboundLoca 10/16/09 http://bugs.python.org/issue7149 created ned.deily datetime operations spanning MINYEAR give bad results 10/16/09 http://bugs.python.org/issue7150 created mark.leander Issues Now Closed (37) ______________________ sys.exit() called from optparse - bad, bad, bad 491 days http://bugs.python.org/issue3079 pfeldman at verizon.net easy distutils.util.get_platform() is wrong for universal builds on m 369 days http://bugs.python.org/issue4064 ned.deily patch, needs review ihooks module cannot handle absolute imports 359 days http://bugs.python.org/issue4152 nascheme PEP 383 implementation 163 days http://bugs.python.org/issue5915 loewis patch distutils builds extension modules to root package directory 103 days http://bugs.python.org/issue6403 srid ihooks support for relative imports 38 days http://bugs.python.org/issue6855 brett.cannon urllib2 doesn't respect "no_proxy" environment (python2.6.2) 29 days http://bugs.python.org/issue6894 orsenthil patch test_urllib: constructLocalFileUrl returns invalid URLs on Windo 11 days http://bugs.python.org/issue7043 orsenthil Automatic test___all__ 6 days http://bugs.python.org/issue7055 pitrou patch Python 2.6.3 / setuptools 0.6c9: extension module builds fail wi 1 days http://bugs.python.org/issue7064 tarek bytes.maketrans segfaults 9 days http://bugs.python.org/issue7065 pitrou patch inspect.isabstract to return boolean values only 9 days http://bugs.python.org/issue7069 benjamin.peterson patch logging.handlers.SysLogHandler with TCP support 2 days http://bugs.python.org/issue7086 vinay.sajip patch Distutils build ignores the --compiler command line option 5 days http://bugs.python.org/issue7091 tarek Multiprocessing.Array(lock=False) fails 0 days http://bugs.python.org/issue7095 jnoller urlparse.urljoin of simple "http://" and "somedomain.com" produc 0 days http://bugs.python.org/issue7097 brett.cannon Error in config parser example (w/ Patch) 0 days http://bugs.python.org/issue7103 r.david.murray patch test_descr uses __cmp__ which will never be called 0 days http://bugs.python.org/issue7104 benjamin.peterson Python crashed in the middle of an argument between wx and boa_c 0 days http://bugs.python.org/issue7106 bvn broken link on news page 0 days http://bugs.python.org/issue7109 mkiever unicodetype_db.h warning: integer constant is too large for 'lon 2 days http://bugs.python.org/issue7112 amaury.forgeotdarc extension module builds fail when using paths in the extension n 1 days http://bugs.python.org/issue7115 tarek str.join() should be documented as taking an iterable 2 days http://bugs.python.org/issue7116 georg.brandl patch logging depends on multiprocessing 3 days http://bugs.python.org/issue7120 vinay.sajip ImportError of urllib.request & http.client under PYTHONHOME/Lib 0 days http://bugs.python.org/issue7121 ezio.melotti multiprocessing.Pool() problem in windows 0 days http://bugs.python.org/issue7122 amaury.forgeotdarc typo (English) in threading.py 0 days http://bugs.python.org/issue7125 georg.brandl Contradictory documentation for os.putenv and os.system 0 days http://bugs.python.org/issue7126 georg.brandl 'filling' missing in __all__ ot turtle.py 0 days http://bugs.python.org/issue7129 georg.brandl patch json uses sre_* modules which may not work on alternate implemen 0 days http://bugs.python.org/issue7130 pitrou AttributeError: 'module' object has no attribute 'HTTPSConnectio 0 days http://bugs.python.org/issue7135 r.david.murray imp.new_module does not function correctly if the module is retu 0 days http://bugs.python.org/issue7140 benjamin.peterson __repr__ of memoryview object has type unicode; should be str. 0 days http://bugs.python.org/issue7142 mark.dickinson patch socket.py: r75412 broke build 0 days http://bugs.python.org/issue7148 kbk dict subclass breaks cPickle noload() 1736 days http://bugs.python.org/issue1101399 nascheme Digest Authentication not working in all cases 1499 days http://bugs.python.org/issue1285440 orsenthil patch, easy Tighter co_stacksize computation in stackdepth_walk 823 days http://bugs.python.org/issue1754094 nascheme patch Top Issues Most Discussed (10) ______________________________ 23 Regexp: capturing groups in repetitions 2 days open http://bugs.python.org/issue7132 19 logging depends on multiprocessing 3 days closed http://bugs.python.org/issue7120 13 Compilation error if configuref --with-computed-gotos 78 days open http://bugs.python.org/issue6603 11 Multiprocess Process does not always exit when run from a threa 2 days open http://bugs.python.org/issue7123 9 elementtree segfaults on invalid xml declaration 1 days open http://bugs.python.org/issue7138 7 Python 2.6.3 / setuptools 0.6c9: extension module builds fail w 1 days closed http://bugs.python.org/issue7064 7 test_multiprocessing dictionary changed size errors and hang 12 days open http://bugs.python.org/issue7060 7 Unicode case mappings are incorrect 311 days open http://bugs.python.org/issue4610 6 core dump when stderr is moved 4 days open http://bugs.python.org/issue7111 6 Test suite emits many DeprecationWarnings when -3 is enabled 7 days open http://bugs.python.org/issue7092 From gregor.lingl at aon.at Fri Oct 16 23:14:05 2009 From: gregor.lingl at aon.at (Gregor Lingl) Date: Fri, 16 Oct 2009 23:14:05 +0200 Subject: [Python-Dev] Tkinter script crashes Python 3.1 Message-ID: <4AD8E21D.5010307@aon.at> I've written a small Tkinter-Script, that crashes Python 3.1 (but not Python 2.6) without any specific rrror message. When started from within IDLE, the failing of the script also closes IDLE. (This is the case under Windows, Mac OS X and Linux (teted with Ubuntu 9.04)) Bug-Tracker Issue 6717 The script is attached to this issue I think, errors like this should definitely not occur. Instead a message like "recusion depth exceeded" should be displayed (and IDLE should remain functional, if used) Since I do not have the means availble to track down this bug, I'd like to draw your attention to it and to ask if someone else has the means and time to do so. I'd also suggest to increase the priority of this bug in the bugtracker. Regards, Gregor From vinay_sajip at yahoo.co.uk Sat Oct 17 09:23:50 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 17 Oct 2009 07:23:50 +0000 (UTC) Subject: [Python-Dev] A proposal: configuring logging using dictionaries Message-ID: A little while ago, I posted here a suggestion about a new way to configure logging, using dictionaries. This received some positive and no negative feedback, so I have thought some more about the details of how it might work. I present below the results of that thinking, in a PEP-style format. I don't know if an actual PEP is required for a change of this type, but I felt that it's still worth going through the exercise to try and achieve a reasonable level of rigour. (I hope I've succeeded.) I would welcome all your feedback on this proposal. If I hear no negative feedback, I propose to implement this feature as suggested. I thought about posting this on comp.lang.python as well, but possibly it's a little too much information for most of the folks there. I think it would be useful to get feedback from the wider community, though, and welcome any suggestions on how best to achieve this. Thanks and regards, Vinay Sajip ----------- PEP: XXX Title: Dictionary-Based Configuration For Logging Version: $Revision$ Last-Modified: $Date$ Author: Vinay Sajip Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Oct-2009 Python-Version: 2.7 and 3.2 Post-History: Abstract ======== This PEP describes a new way of configuring logging using a dictionary to hold configuration information. Rationale ========= The present means for configuring Python's logging package is either by using the logging API to configure logging programmatically, or else by means of ConfigParser-based configuration files. Programmatic configuration, while offering maximal control, fixes the configuration in Python code. This does not facilitate changing it easily at runtime, and, as a result, the ability to flexibly turn the verbosity of logging up and down for different parts of a using application is lost. This limits the usability of logging as an aid to diagnosing problems - and sometimes, logging is the only diagnostic aid available in production environments. The ConfigParser-based configuration system is usable, but does not allow its users to configure all aspects of the logging package. For example, Filters cannot be configured using this system. Furthermore, the ConfigParser format appears to engender dislike (sometimes strong dislike) in some quarters. Though it was chosen because it was the only configuration format supported in the Python standard at that time, many people regard it (or perhaps just the particular schema chosen for logging's configuration) as 'crufty' or 'ugly', in some cases apparently on purely aesthetic grounds. Recent versions of Python inlude JSON support in the standard library, and this is also usable as a configuration format. In other environments, such as Google App Engine, YAML is used to configure applications, and usually the configuration of logging would be considered an integral part of the application configuration. Although the standard library does not contain YAML support at present, support for both JSON and YAML can be provided in a common way because both of these serialization formats allow deserialization of Python dictionaries. By providing a way to configure logging by passing the configuration in a dictionary, logging will be easier to configure not only for users of JSON and/or YAML, but also for users of bespoke configuration methods, by providing a common format in which to describe the desired configuration. Another drawback of the current ConfigParser-based configuration system is that it does not support incremental configuration: a new configuration completely replaces the existing configuration. Although full flexibility for incremental configuration is difficult to provide in a multi-threaded environment, the new configuration mechanism will allow the provision of limited support for incremental configuration. Specification ============= The specification consists of two parts: the API and the format of the dictionary used to convey configuration information (i.e. the schema to which it must conform). Naming ------ Historically, the logging package has not been PEP-8 conformant. At some future time, this will be corrected by changing method and function names in the package in order to conform with PEP-8. However, in the interests of uniformity, the proposed additions to the API use a naming scheme which is consistent with the present scheme used by logging. API --- The logging.config module will have the following additions: * A class, called ``DictConfigurator``, whose constructor is passed the dictionary used for configuration, and which has a ``configure()`` method. * A callable, called ``dictConfigClass``, which will (by default) be set to ``DictConfigurator``. This is provided so that if desired, ``DictConfigurator`` can be replaced with a suitable user-defined implementation. * A function, called ``dictConfig()``, which takes a single argument - the dictionary holding the configuration. This function will call ``dictConfigClass`` passing the specified dictionary, and then call the ``configure()`` method on the returned object to actually put the configuration into effect:: def dictConfig(config): dictConfigClass(config).configure() Dictionary Schema - Overview ---------------------------- Before describing the schema in detail, it is worth saying a few words about object connections, support for user-defined objects and access to external objects. Object connections '''''''''''''''''' The schema is intended to describe a set of logging objects - loggers, handlers, formatters, filters - which are connected to each other in an object graph. Thus, the schema needs to represent connections between the objects. For example, say that, once configured, a particular logger has an attached to it a particular handler. For the purposes of this discussion, we can say that the logger represents the source, and the handler the destination, of a connection between the two. Of course in the configured objects this is represented by the logger holding a reference to the handler. In the configuration dict, this is done by giving each destination object an id which identifies it unambiguously, and then using the id in the source object's configuration to indicate that a connection exists between the source and the destination object with that id. So, for example, consider the following YAML snippet:: handers: h1: #This is an id # configuration of handler with id h1 goes here h2: #This is another id # configuration of handler with id h2 goes here loggers: foo.bar.baz: # other configuration for logger "foo.bar.baz" handlers: [h1, h2] (Note: YAML will be used in this document as it is more readable than the equivalent Python source form for the dictionary.) The ids for loggers are the logger names which would be used programmatically to obtain a reference to those loggers, e.g. ``foo.bar.baz``. The ids for other objects can be any string value (such as ``h1``, ``h2`` above) and they are transient, in that they are only meaningful for processing the configuration dictionary and used to determine connections between objects, and are not persisted anywhere when the configuration call is complete. The above snippet indicates that logger named ``foo.bar.baz`` should have two handlers attached to it, which are described by the handler ids ``h1`` and ``h2``. User-defined objects '''''''''''''''''''' The schema should support user-defined objects for handlers, filters and formatters. (Loggers do not need to have different types for different instances, so there is no support - in the configuration - for user-defined logger classes.) Objects to be configured will typically be described by dictionaries which detail their configuration. In some places, the logging system will be able to infer from the context how an object is to be instantiated, but when a user-defined object is to be instantiated, the system will not know how to do this. In order to provide complete flexibility for user-defined object instantiation, the user will need to provide a 'factory' - a callable which is called with a configuration dictionary and which returns the instantiated object. This will be signalled by the factory being made available under the special key ``'()'``. Here's a concrete example:: formatters: brief: format: '%(message)s' default: format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s' datefmt: '%Y-%m-%d %H:%M:%S' custom: (): my.package.customFormatterFactory bar: baz spam: 99.9 answer: 42 The above YAML snippet defines three formatters. The first, with id ``brief``, is a standard ``logging.Formatter`` instance with the specified format string. The second, with id ``default``, has a longer format and also defines the time format explicitly, and will result in a ``logging.Formatter`` initialized with those two format strings. Shown in Python source form, the ``brief`` and ``default`` formatters have have configuration sub-dictionaries:: { 'format' : '%(message)s' } and:: { 'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s', 'datefmt' : '%Y-%m-%d %H:%M:%S' } respectively, and as these dictionaries do not contain the special key ``'()'``, the instantiation is inferred from the context: as a result, standard ``logging.Formatter`` instances are created. The configuration sub-dictionary for the third formatter, with id ``custom``, is:: { '()' : 'my.package.customFormatterFactory', 'bar' : 'baz', 'spam' : 99.9, 'answer' : 42 } and this contains the special key ``'()'``, which means that user-defined instantiation is wanted. In this case, the specified factory callable will be located using normal import mechanisms and called with the *remaining* items in the configuration sub-dictionary as keyword arguments. In the above example, the formatter with id ``custom`` will be assumed to be returned by the call:: my.package.customFormatterFactory(bar="baz", spam=99.9, answer=42) The key ``'()'`` has been used as the special key because it is not a valid keyword parameter name, and so will not clash with the names of the keyword arguments used in the call. The ``'()'`` also serves as a mnemonic that the corresponding value is a callable. Access to external objects '''''''''''''''''''''''''' There are times where a configuration will need to refer to objects external to the configuration, for example ``sys.stderr``. If the configuration dict is constructed using Python code then this is straightforward, but a problem arises when the configuration is provided via a text file (e.g. JSON, YAML). In a text file, there is no standard way to distinguish ``sys.stderr`` from the literal string ``'sys.stderr'``. To facilitate this distinction, the configuration system will look for certain special prefixes in string values and treat them specially. For example, if the literal string ``'ext://sys.stderr'`` is provided as a value in the configuration, then the ``ext://`` will be stripped off and the remainder of the value processed using normal import mechanisms. The handling of such prefixes will be done in a way analogous to protocol handling: there will be a generic mechanism to look for prefixes which match the regular expression ``^(?P[a-z]+)://(?P.*)$`` whereby, if the ``prefix`` is recognised, the ``suffix`` is processed in a prefix- dependent manner and the result of the processing replaces the string value. If the prefix is not recognised, then the string value will be left as-is. The implementation will provide for a set of standard prefixes such as ``ext://`` but it will be possible to disable the mechanism completely or provide additional or different prefixes for special handling. Dictionary Schema - Detail -------------------------- The dictionary passed to ``dictConfig()`` must contain the following keys: * `version` - to be set to an integer value representing the schema version. The only valid value at present is 1, but having this key allows the schema to evolve while still preserving backwards compatibility. All other keys are optional, but if present they will be interpreted as described below. In all cases below where a 'configuring dict' is mentioned, it will be checked for the special ``'()'`` key to see if a custom instantiation is required. If so, the mechanism described above is used to instantiate; otherwise, the context is used to determine how to instantiate. * `formatters` - the corresponding value will be a dict in which each key is a formatter id and each value is a dict describing how to configure the corresponding Formatter instance. The configuring dict is searched for keys ``format`` and ``datefmt`` (with defaults of ``None``) and these are used to construct a ``logging.Formatter`` instance. * `filters` - the corresponding value will be a dict in which each key is a filter id and each value is a dict describing how to configure the corresponding Filter instance. The configuring dict is searched for key ``name`` (defaulting to the empty string) and this is used to construct a ``logging.Filter`` instance. * `handlers` - the corresponding value will be a dict in which each key is a handler id and each value is a dict describing how to configure the corresponding Handler instance. The configuring dict is searched for the following keys: * ``class`` (mandatory). This is the fully qualified name of the handler class. * ``level`` (optional). The level of the handler. * ``formatter`` (optional). The id of the formatter for this handler. * ``filters`` (optional). A list of ids of the filters for this handler. All *other* keys are passed through as keyword arguments to the handler's constructor. For example, given the snippet:: handlers: console: class : logging.StreamHandler formatter: brief level : INFO filters: [allow_foo] stream : ext://sys.stdout file: class : logging.handlers.RotatingFileHandler formatter: precise filename: logconfig.log maxBytes: 1024 backupCount: 3 the handler with id ``console`` is instantiated as a ``logging.StreamHandler``, using ``sys.stdout`` as the underlying stream. The handler with id ``file`` is instantiated as a ``logging.handlers.RotatingFileHandler`` with the keyword arguments ``filename="logconfig.log", maxBytes=1024, backupCount=3``. * `loggers` - the corresponding value will be a dict in which each key is a logger name and each value is a dict describing how to configure the corresponding Logger instance. The configuring dict is searched for the following keys: * ``level`` (optional). The level of the logger. * ``propagate`` (optional). The propagation setting of the logger. * ``filters`` (optional). A list of ids of the filters for this logger. * ``handlers`` (optional). A list of ids of the handlers for this logger. The specified loggers will be configured according to the level, propagation, filters and handlers specified. * `root` - this will be the configuration for the root logger. Processing of the configuration will be as for any logger, except that the ``propagate`` setting will not be applicable. * `incremental` - whether the configuration is to be interpreted as incremental to the existing configuration. This value defaults to False, which means that the specified configuration replaces the existing configuration with the same semantics as used by the existing ``fileConfig()`` API. If the specified value is True, the configuration is processed as described in the section on "Incremental Configuration", below. A Working Example ----------------- The following is an actual working configuration in YAML format (except that the email addresses are bogus):: formatters: brief: format: '%(levelname)-8s: %(name)-15s: %(message)s' precise: format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s' filters: allow_foo: name: foo handlers: console: class : logging.StreamHandler formatter: brief level : INFO stream : ext://sys.stdout filters: [allow_foo] file: class : logging.handlers.RotatingFileHandler formatter: precise filename: logconfig.log maxBytes: 1024 backupCount: 3 debugfile: class : logging.FileHandler formatter: precise filename: logconfig-detail.log mode: a email: class: logging.handlers.SMTPHandler mailhost: localhost fromaddr: my_app at domain.tld toaddrs: - support_team at domain.tld - dev_team at domain.tld subject: Houston, we have a problem. loggers: foo: level : ERROR handlers: [debugfile] spam: level : CRITICAL handlers: [debugfile] propagate: no bar.baz: level: WARNING root: level : DEBUG handlers : [console, file] Incremental Configuration ========================= It is difficult to provide complete flexibility for incremental configuration. For example, because objects such as handlers, filters and formatters are anonymous, once a configuration is set up, it is not possible to refer to such anonymous objects when augmenting a configuration. For example, if an initial call is made to configure the system where logger ``foo`` has a handler with id ``console`` attached, then a subsequent call to configure a logger ``bar`` with id ``console`` would create a new handler instance, as the id ``console`` from the first call isn't kept. Furthermore, there is not a compelling case for arbitrarily altering the object graph of loggers, handlers, filters, formatters at run-time, once a configuration is set up; the verbosity of loggers can be controlled just by setting levels (and perhaps propagation flags). Thus, when the ``incremental`` key of a configuration dict is present and is ``True``, the system will ignore the ``formatters``, ``filters``, ``handlers`` entries completely, and process only the ``level`` and ``propagate`` settings in the ``loggers`` and ``root`` entries. Configuration Errors ==================== If an error is encountered during configuration, the system will raise a ``ValueError`` or a ``TypeError`` with a suitably descriptive message. The following is a (possibly incomplete) list of conditions which will raise an error: * A ``level`` which is not a string or which is a string not corresponding to an actual logging level * A ``propagate`` value which is not a Boolean * An id which does not have a corresponding destination * An invalid logger name Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From ncoghlan at gmail.com Sat Oct 17 10:10:49 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Oct 2009 18:10:49 +1000 Subject: [Python-Dev] A proposal: configuring logging using dictionaries In-Reply-To: References: Message-ID: <4AD97C09.3070705@gmail.com> Vinay Sajip wrote: > A little while ago, I posted here a suggestion about a new way to configure > logging, using dictionaries. This received some positive and no negative > feedback, so I have thought some more about the details of how it might work. I > present below the results of that thinking, in a PEP-style format. I don't know > if an actual PEP is required for a change of this type, but I felt that it's > still worth going through the exercise to try and achieve a reasonable level of > rigour. (I hope I've succeeded.) Since part of the purpose of this new configuration option is to allow other projects to use it as a target for specifying a logging configuration, maintaining it as a formal PEP seems like a good idea to me. This is already done for a few other standard-ish interfaces (distutils metadata, DB-API, WSGI). > I would welcome all your feedback on this proposal. If I hear no negative > feedback, I propose to implement this feature as suggested. My only real objection is to the "incremental" flag in the dictionary. I would suggest having two different API methods instead - initDictConfig and updateDictConfig. Or is there some specific reason for having the choice of incremental update vs replacement configuration in the hands of the config author rather than the application developer? My other question is whether or not it would be worth having the logging module able to export it's *current* configuration in dictionary form. I don't have a use case other than that it might be useful for debugging logging configuration errors when attempting to combine multiple sources of logging data, but it's worth considering. > So, for example, consider the following YAML snippet:: > > handers: > h1: #This is an id > # configuration of handler with id h1 goes here > h2: #This is another id > # configuration of handler with id h2 goes here Typo in the header here: s/handers/handlers/ > Configuration Errors > ==================== > > If an error is encountered during configuration, the system will raise a > ``ValueError`` or a ``TypeError`` with a suitably descriptive message. The > following is a (possibly incomplete) list of conditions which will raise an > error: > > * A ``level`` which is not a string or which is a string not corresponding to > an actual logging level > > * A ``propagate`` value which is not a Boolean > > * An id which does not have a corresponding destination > > * An invalid logger name You may want to allow yourself AttributeError and ImportError here. Alternatively, you could commit to converting those to an appropriate ValueError that provides more detail on the config setting that contained the problematic reference. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Sat Oct 17 10:33:53 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 17 Oct 2009 08:33:53 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PY=5FSSIZE=5FT=5FCLEAN_in_py3k?= Message-ID: Hello, It turns out (*) that even in py3k, not all modules are PY_SSIZE_T_CLEAN. Should we try to remedy that, and make PY_SSIZE_T_CLEAN the default in future 3.x versions? As far I know, there's no reason not to be PY_SSIZE_T_CLEAN, except for having to convert old code. (*) http://bugs.python.org/issue7080 Regards Antoine. PS : no, I'm not volunteering to do it all myself :) From vinay_sajip at yahoo.co.uk Sat Oct 17 10:53:59 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 17 Oct 2009 08:53:59 +0000 (UTC) Subject: [Python-Dev] A proposal: configuring logging using dictionaries References: <4AD97C09.3070705@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > Since part of the purpose of this new configuration option is to allow > other projects to use it as a target for specifying a logging > configuration, maintaining it as a formal PEP seems like a good idea to > me. This is already done for a few other standard-ish interfaces > (distutils metadata, DB-API, WSGI). I'm happy with this. I've subscribed to the PEPs list and once the subscription comes through, will post there linking to the PEP draft. > My only real objection is to the "incremental" flag in the dictionary. I > would suggest having two different API methods instead - initDictConfig > and updateDictConfig. Or is there some specific reason for having the > choice of incremental update vs replacement configuration in the hands > of the config author rather than the application developer? I was thinking that in some environments, configuration changes might be sent as pickled dicts over the wire. In such cases it might be useful to have the "incremental" flag in the payload. But I haven't thought that through in detail, as it's somewhat peripheral to this feature. > My other question is whether or not it would be worth having the logging > module able to export it's *current* configuration in dictionary form. I > don't have a use case other than that it might be useful for debugging > logging configuration errors when attempting to combine multiple sources > of logging data, but it's worth considering. Another fair point - perhaps a getConfig() which returns a dict can be added. > You may want to allow yourself AttributeError and ImportError here. > Alternatively, you could commit to converting those to an appropriate > ValueError that provides more detail on the config setting that > contained the problematic reference. Perhaps TypeError, too. The idea is to definitely pinpoint where in the configuration the error occurred. I'll think about the detail of this a bit further down the line, but I just wanted to establish that I wasn't planning on introducing any logging-related exception classes. Thanks for the quick response and detailed feedback. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sat Oct 17 14:57:03 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 17 Oct 2009 12:57:03 +0000 (UTC) Subject: [Python-Dev] A proposal: configuring logging using dictionaries References: <4AD97C09.3070705@gmail.com> Message-ID: Vinay Sajip yahoo.co.uk> writes: > Perhaps TypeError, too. Sorry, brainfade there. I already mentioned TypeError in the original post :-) From lie.1296 at gmail.com Sun Oct 18 06:11:16 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 18 Oct 2009 15:11:16 +1100 Subject: [Python-Dev] transitioning from % to {} formatting In-Reply-To: <4ACA71EB.4050003@voidspace.org.uk> References: <261064E43172924BBADB00DA14BB4B672A13F6FC1C@XCH-NW-14V.nw.nos.boeing.com> <4ACA551F.9090309@gmail.com> <1afaf6160910051357t32301c3aq593ac1745c807290@mail.gmail.com> <4ACA71EB.4050003@voidspace.org.uk> Message-ID: Michael Foord wrote: > Benjamin Peterson wrote: >> 2009/10/5 Nick Coghlan : >> >>> So I would agree that method invocation on literals (particularly string >>> literals) is an already established language idiom. >>> >> >> And who hasn't ever used 4.56.as_integer_ratio()? :) > > I've tried 4.__add__ a few times (not for a while now though). That's a Syntax Error >>> 4.__add__ File "", line 1 4.__add__ ^ SyntaxError: invalid syntax however this works: >>> 4 .__add__ From vinay_sajip at yahoo.co.uk Sun Oct 18 13:50:23 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 18 Oct 2009 11:50:23 +0000 (UTC) Subject: [Python-Dev] A proposal: configuring logging using dictionaries References: <4AD97C09.3070705@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > My other question is whether or not it would be worth having the logging > module able to export it's *current* configuration in dictionary form. I > don't have a use case other than that it might be useful for debugging > logging configuration errors when attempting to combine multiple sources > of logging data, but it's worth considering. > One restriction on any output of the current configuration is that it may not be suitable for round-tripping, i.e. you may not be able to use the output dict to (re)configure the system. That's because in general, the configurator wants to know how to instantiate handlers, filters, formatters and so it essentially needs the constructor arguments in an incoming dict. However, there's no requirement for the handlers, filters and formatters to keep those constructor arguments around in the original form; and so, in the general case, they are not available to output in a dict describing the current configuration. Regards, Vinay Sajip From dsdale24 at gmail.com Sun Oct 18 16:50:51 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Sun, 18 Oct 2009 10:50:51 -0400 Subject: [Python-Dev] nonstandard behavior of reflected functions Message-ID: According to http://docs.python.org/reference/datamodel.html , the reflected operands functions like __radd__ "are only called if the left operand does not support the corresponding operation and the operands are of different types. [3] For instance, to evaluate the expression x - y, where y is an instance of a class that has an __rsub__() method, y.__rsub__(x) is called if x.__sub__(y) returns NotImplemented." Consider the following simple example: ========================== class Quantity(object): def __add__(self, other): return '__add__ called' def __radd__(self, other): return '__radd__ called' class UnitQuantity(Quantity): def __add__(self, other): return '__add__ called' def __radd__(self, other): return '__radd__ called' print 'Quantity()+Quantity()', Quantity()+Quantity() print 'UnitQuantity()+UnitQuantity()', UnitQuantity()+UnitQuantity() print 'UnitQuantity()+Quantity()', UnitQuantity()+Quantity() print 'Quantity()+UnitQuantity()', Quantity()+UnitQuantity() ========================== The output should indicate that __add__ was called in all four trials, but the last trial calls __radd__. Interestingly, if I comment out the definition of __radd__ in UnitQuantity, then the fourth trial calls __add__ like it should. I think this may be an important bug. I'm running Python 2.6.4rc1 (r264rc1:75270, Oct 13 2009, 17:02:06) an ubuntu Karmic. Is it a known issue, or am I misreading the documentation? Thanks, Darren From dsdale24 at gmail.com Sun Oct 18 17:03:10 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Sun, 18 Oct 2009 11:03:10 -0400 Subject: [Python-Dev] nonstandard behavior of reflected functions In-Reply-To: References: Message-ID: On Sun, Oct 18, 2009 at 10:50 AM, Darren Dale wrote: > According to http://docs.python.org/reference/datamodel.html , the > reflected operands functions like __radd__ "are only called if the > left operand does not support the corresponding operation and the > operands are of different types. [3] For instance, to evaluate the > expression x - y, where y is an instance of a class that has an > __rsub__() method, y.__rsub__(x) is called if x.__sub__(y) returns > NotImplemented." > > Consider the following simple example: > > ========================== > class Quantity(object): > > ? ?def __add__(self, other): > ? ? ? ?return '__add__ called' > > ? ?def __radd__(self, other): > ? ? ? ?return '__radd__ called' > > class UnitQuantity(Quantity): > > ? ?def __add__(self, other): > ? ? ? ?return '__add__ called' > > ? ?def __radd__(self, other): > ? ? ? ?return '__radd__ called' > > print 'Quantity()+Quantity()', Quantity()+Quantity() > print 'UnitQuantity()+UnitQuantity()', UnitQuantity()+UnitQuantity() > print 'UnitQuantity()+Quantity()', UnitQuantity()+Quantity() > print 'Quantity()+UnitQuantity()', Quantity()+UnitQuantity() > ========================== > > The output should indicate that __add__ was called in all four trials, > but the last trial calls __radd__. Interestingly, if I comment out the > definition of __radd__ in UnitQuantity, then the fourth trial calls > __add__ like it should. > > I think this may be an important bug. I'm running Python 2.6.4rc1 > (r264rc1:75270, Oct 13 2009, 17:02:06) an ubuntu Karmic. Is it a known > issue, or am I misreading the documentation? I'm sorry, I should have read further down the page in the documentation: "Note: If the right operand?s type is a subclass of the left operand?s type and that subclass provides the reflected method for the operation, this method will be called before the left operand?s non-reflected method. This behavior allows subclasses to override their ancestors? operations." From solipsis at pitrou.net Sun Oct 18 22:01:54 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Oct 2009 20:01:54 +0000 (UTC) Subject: [Python-Dev] SIGCHECK() in longobject.c Message-ID: Hello, In Objects/longobject.c, there's the SIGCHECK() macro which periodically checks for signals when doing long integer computations (divisions, multiplications). It does so by messing with the _Py_Ticker variable. It was added in 1991 under the title "Many small changes", and I suppose it was useful back then. However, nowadays long objects are ridiculously fast, witness for example: $ ./py3k/python -m timeit -s "a=eval('3'*10000+'5');b=eval('8'*6000+'7')" "str(a//b)" 1000 loops, best of 3: 1.47 msec per loop Can we remove this check, or are there people doing million-digits calculations they want to interrupt using Control-C ? Regards Antoine. From dickinsm at gmail.com Sun Oct 18 22:08:27 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 18 Oct 2009 21:08:27 +0100 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: References: Message-ID: <5c6f2a5d0910181308s32876221sd20f6df2be004ffb@mail.gmail.com> On Sun, Oct 18, 2009 at 9:01 PM, Antoine Pitrou wrote: > In Objects/longobject.c, there's the SIGCHECK() macro which periodically checks > for signals when doing long integer computations (divisions, multiplications). > It does so by messing with the _Py_Ticker variable. > > It was added in 1991 under the title "Many small changes", and I suppose it was > useful back then. > > However, nowadays long objects are ridiculously fast, witness for example: > > $ ./py3k/python -m timeit -s "a=eval('3'*10000+'5');b=eval('8'*6000+'7')" > "str(a//b)" > 1000 loops, best of 3: 1.47 msec per loop > > Can we remove this check, or are there people doing million-digits calculations > they want to interrupt using Control-C ? Yes, I suspect there are. Though you don't need millions of digits for a single operation to take a noticeable amount of time: try str(10**100000), for example. Is there a benefit to removing the check? Mark From solipsis at pitrou.net Sun Oct 18 22:23:41 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Oct 2009 20:23:41 +0000 (UTC) Subject: [Python-Dev] SIGCHECK() in longobject.c References: <5c6f2a5d0910181308s32876221sd20f6df2be004ffb@mail.gmail.com> Message-ID: Mark Dickinson gmail.com> writes: > > Yes, I suspect there are. Though you don't need millions of digits for a single > operation to take a noticeable amount of time: try str(10**100000), > for example. > > Is there a benefit to removing the check? Apart from casual cleanup, the reason I'm asking is that I'm experimenting with removing _Py_Ticker, and I want to know what I need to emulate :-) Antoine. From ehsanamiri at gmail.com Sun Oct 18 22:39:15 2009 From: ehsanamiri at gmail.com (Ehsan Amiri) Date: Sun, 18 Oct 2009 13:39:15 -0700 Subject: [Python-Dev] nonstandard behavior of reflected functions In-Reply-To: References: Message-ID: I see the same behaviour, moreover when I change class Quantity to a classic class (removing '(object)'), it works as expected. (i.e. Quanitity.__add__() is called after the fourth print. I run Python 2.6.2 on Vista. On Sun, Oct 18, 2009 at 7:50 AM, Darren Dale wrote: > According to http://docs.python.org/reference/datamodel.html , the > reflected operands functions like __radd__ "are only called if the > left operand does not support the corresponding operation and the > operands are of different types. [3] For instance, to evaluate the > expression x - y, where y is an instance of a class that has an > __rsub__() method, y.__rsub__(x) is called if x.__sub__(y) returns > NotImplemented." > > Consider the following simple example: > > ========================== > class Quantity(object): > > def __add__(self, other): > return '__add__ called' > > def __radd__(self, other): > return '__radd__ called' > > class UnitQuantity(Quantity): > > def __add__(self, other): > return '__add__ called' > > def __radd__(self, other): > return '__radd__ called' > > print 'Quantity()+Quantity()', Quantity()+Quantity() > print 'UnitQuantity()+UnitQuantity()', UnitQuantity()+UnitQuantity() > print 'UnitQuantity()+Quantity()', UnitQuantity()+Quantity() > print 'Quantity()+UnitQuantity()', Quantity()+UnitQuantity() > ========================== > > The output should indicate that __add__ was called in all four trials, > but the last trial calls __radd__. Interestingly, if I comment out the > definition of __radd__ in UnitQuantity, then the fourth trial calls > __add__ like it should. > > I think this may be an important bug. I'm running Python 2.6.4rc1 > (r264rc1:75270, Oct 13 2009, 17:02:06) an ubuntu Karmic. Is it a known > issue, or am I misreading the documentation? > > Thanks, > Darren > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ehsanamiri%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel at stutzbachenterprises.com Mon Oct 19 00:16:52 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sun, 18 Oct 2009 17:16:52 -0500 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: References: Message-ID: On Sun, Oct 18, 2009 at 3:01 PM, Antoine Pitrou wrote: > Can we remove this check, or are there people doing million-digits > calculations > they want to interrupt using Control-C ? > I sometimes do million-digits calculations that I want to interrupt using Control-C. (particularly when I didn't *intend* to do a million-digits calculation... ;) ) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Oct 19 00:46:04 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Oct 2009 22:46:04 +0000 (UTC) Subject: [Python-Dev] SIGCHECK() in longobject.c References: Message-ID: Daniel Stutzbach stutzbachenterprises.com> writes: > > I sometimes do million-digits calculations that I want to interrupt using Control-C.(particularly when I didn't *intend* to do a million-digits calculation... ;) )-- Sure, but it's no different than doing, e.g.: list(range(100000000)).sort() (don't try this, it just made by computer slow down to a crawl and I had to kill -9 the Python interpreter) The question is whether there is still any reason to special-case long objects and not, say, list.sort() or re.sub(). Regards Antoine. From daniel at stutzbachenterprises.com Mon Oct 19 01:05:54 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Sun, 18 Oct 2009 18:05:54 -0500 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: References: Message-ID: On Sun, Oct 18, 2009 at 5:46 PM, Antoine Pitrou wrote: > Daniel Stutzbach stutzbachenterprises.com> writes: > > I sometimes do million-digits calculations that I want to interrupt using > Control-C.(particularly when I didn't *intend* to do a million-digits > calculation... ;) )-- > > Sure, but it's no different than doing, e.g.: > list(range(100000000)).sort() > That's a good point, although I can't recall the last time I accidently created a painfully large list. I can recall the last time I started a painfully large integer computation. Being able to stop the interpretter with Control-C instead of kill -9 is a minor convenience, though. I could live without it. (Although I can't speak for everyone, of course) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Oct 19 03:15:38 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Oct 2009 11:15:38 +1000 Subject: [Python-Dev] nonstandard behavior of reflected functions In-Reply-To: References: Message-ID: <4ADBBDBA.1050002@gmail.com> Ehsan Amiri wrote: > I see the same behaviour, moreover when I change class Quantity to a > classic class (removing '(object)'), it works as expected. (i.e. > Quanitity.__add__() is called after the fourth print. I run Python 2.6.2 > on Vista. Darren found the explanation further down the page he was reading - the fact that the right operand is an instance of a subclass of the left operand's class makes a difference. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Mon Oct 19 03:20:49 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 18 Oct 2009 21:20:49 -0400 Subject: [Python-Dev] Python 2.6.4rc2 Message-ID: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> Hello everyone. The source tarballs and Windows installers for Python 2.6.4rc2 are now available: http://www.python.org/download/releases/2.6.4/ Please download them, install them, and try to use them with your projects and environments. Let's make 2.6.4 a rock solid release! If there are no more regressions found, we'll do the final release in one week, on 25-October. Enjoy, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From abhiram.casina at gmail.com Mon Oct 19 08:29:01 2009 From: abhiram.casina at gmail.com (Abhiram Kasina) Date: Mon, 19 Oct 2009 11:59:01 +0530 Subject: [Python-Dev] Proposal : Python Trusted Computing API Message-ID: Hi Trusted Computing (TC) is a technology developed and promoted by the Trusted Computing Group (TCG)[3]. So, basically the group came up with these chips called TPM chips which are present on most motherboards nowadays. The main purpose of it is to enhance security so that infected executables don't run. It also provides memory curtaining such that cryptographic keys won't be accessible and many other features. There was a criticism on this from the FOSS community as well that it enables DRM. No wonder, it is being pushed by Intel, Microsoft, AMD, etc.. But personally I think its a good idea from security point of view. So, currently there is an TSS (TCG Software Stack)[1] API written in C. And TrustedJava[2] is a project which ported it to Java and is going to be included in the standard API of Java soon. They have 2 versions of it. One is a simple wrapper on top of the API and the other is a whole implementation of the stack in Java. My proposal is we create an API for it in python. *Reason*: I am a developer in Umit and I think Python is a very good platform for developing applications. So, why not create an API which helps in developing secure applications? I would love to learn more and provide you with any more information. Please let me know what you guys think of it? Thanks in advance Cheers Abhiram [1] http://www.trustedcomputinggroup.org/resources/tcg_software_stack_tss_specification [2] http://trustedjava.sourceforge.net/index.php?item=jtss/about [3] http://www.trustedcomputinggroup.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Oct 19 09:59:05 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 19 Oct 2009 07:59:05 +0000 (UTC) Subject: [Python-Dev] Python 2.6.4rc2 References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> Message-ID: Barry Warsaw python.org> writes: > http://www.python.org/download/releases/2.6.4/ Good news, but just one little nit: the page above appears to link to the NEWS file for 2.6.4rc1. Regards, Vinay Sajip From dickinsm at gmail.com Mon Oct 19 11:28:34 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 19 Oct 2009 10:28:34 +0100 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: References: Message-ID: <5c6f2a5d0910190228r30b82068rdd22c0c905c3f7b4@mail.gmail.com> On Sun, Oct 18, 2009 at 11:46 PM, Antoine Pitrou wrote: > Sure, but it's no different than doing, e.g.: > ? ?list(range(100000000)).sort() > > (don't try this, it just made by computer slow down to a crawl and I had to kill > -9 the Python interpreter) Maybe you were running out of RAM? On a 64-bit machine, list(range(10**8)) takes over 3 Gb. For me, x = list(range(10**8)) takes around 6 seconds, and then x.sort() takes around 2 seconds. This is on a machine with 16 Gb of RAM. (Though I do seem to recall that timsort is extra fast for already sorted lists: O(n) rather than O(n log n)?) So I'd say that it *is* different. A million digit integer takes less than half a megabyte of RAM, so a single operation can be horribly slow long before memory limitations are reached. I'd prefer to keep the SIGCHECK unless there's a really compelling advantage to getting rid of it. Mark From barry at python.org Mon Oct 19 12:55:00 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 19 Oct 2009 06:55:00 -0400 Subject: [Python-Dev] Python 2.6.4rc2 In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> Message-ID: <594E6B4A-8D52-4A93-A368-E699FBA42535@python.org> On Oct 19, 2009, at 3:59 AM, Vinay Sajip wrote: > Barry Warsaw python.org> writes: > >> http://www.python.org/download/releases/2.6.4/ > > Good news, but just one little nit: the page above appears to link > to the NEWS > file for 2.6.4rc1. Ooops! Fixed, thanks. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From dickinsm at gmail.com Mon Oct 19 13:27:04 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 19 Oct 2009 12:27:04 +0100 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: References: Message-ID: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> On Sun, Oct 18, 2009 at 9:01 PM, Antoine Pitrou wrote: > Can we remove this check, or are there people doing million-digits calculations > they want to interrupt using Control-C ? By the way, here's an example of an *almost* real-life use of million digit calculations. For an elementary number theory course that I taught a while ago, there was an associated (optional) computer lab, where the students used Python to investigate various ideas, conjectures, examples, etc. One of the less open-ended questions might have been[1] something like: "On August 23rd, 2008 a computer at UCLA found the first example of a prime with more than 10 million digits: p = 2**43112609-1. Find (1) the exact number of digits in p, when written out in decimal (2) the last 100 digits of p (3) the first 100 digits of p." It's wonderfully easy to get answers to these questions with Python: >>> import math >>> e = 43112609 >>> p = 2**e - 1 >>> ndigits = int(math.ceil(math.log10(p))) >>> last_100_digits = '{:0100d}'.format(p % 10**100) >>> first_100_digits = str(p // 10**(ndigits-100)) Getting the first 100 digits takes a good few seconds; the other operations are faster. But if you (not unreasonably) try to compute str(p), you'll find it's impossibly slow, and it's very handy that it's possible to interrupt that calculation, attempt to understand *why* it's slow, and then try different methods. (And yes, there are better ways to get both the first and last hundred digits.) Mark [1] Might have been, but wasn't. Hence the *almost*. If I'd been teaching that course this semester I'd almost certainly have included something like this. From guido at python.org Mon Oct 19 21:35:59 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Oct 2009 12:35:59 -0700 Subject: [Python-Dev] Proposal : Python Trusted Computing API In-Reply-To: References: Message-ID: On Sun, Oct 18, 2009 at 11:29 PM, Abhiram Kasina wrote: > Trusted Computing (TC) is a technology developed and promoted by the Trusted > Computing Group (TCG)[3]. So, basically the group came up with these chips > called TPM chips which are present on most motherboards nowadays. The main > purpose of it is to enhance security so that infected executables don't run. > It also provides memory curtaining such that cryptographic keys won't be > accessible and many other features. There was a criticism on this from the > FOSS community as well that it enables DRM. No wonder, it is being pushed by > Intel, Microsoft, AMD, etc.. But personally I think its a good idea from > security point of view. Hm... Given that most infections these days are JavaScript based and run in the browser, how does this provide any protection? I'm presuming you're going to say that it doesn't but that there are other use cases where it *does* provide protection; but most likely those use cases are only relevant for Windows (since that's what most attackers attack anyway). > So, currently there is an TSS (TCG Software Stack)[1] API written in C. And > TrustedJava[2] is a project which ported it to Java and is going to be > included in the standard API of Java soon. They have 2 versions of it. One > is a simple wrapper on top of the API and the other is a whole > implementation of the stack in Java. Since this intefaces with the hardware, doesn't it require some kind of cooperation from the Linux kernel? And wouldn't it be better if Python was never allowed access to any of the protected resources in the first place? > My proposal is we create an API for it in python. > Reason: I am a developer in Umit Where/what is Umit? (Google gives several meanings but it's unclear which you might mean.) > and I think Python is a very good platform > for developing applications. So, why not create an API which helps in > developing secure applications? You'd first have to tell us more about the security model. What is a "secure application" and what does it protect against? And how? > I would love to learn more and provide you with any more information. Please > let me know what you guys think of it? This is better directed at python-ideas, so I've redirected this reply there and Bcc'ed the python-dev list. > Thanks in advance > > Cheers > Abhiram > > [1] > http://www.trustedcomputinggroup.org/resources/tcg_software_stack_tss_specification > [2] http://trustedjava.sourceforge.net/index.php?item=jtss/about > [3] http://www.trustedcomputinggroup.org/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Mon Oct 19 22:18:08 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Oct 2009 06:18:08 +1000 Subject: [Python-Dev] Proposal : Python Trusted Computing API In-Reply-To: References: Message-ID: <4ADCC980.9090106@gmail.com> Abhiram Kasina wrote: > I would love to learn more and provide you with any more information. > Please let me know what you guys think of it? This is really an off-topic question for python-dev. This list is just about developing the core interpreter and standard library - we have no control over the APIs that people choose to develop and publish on top of that. If you want to develop such an API and put it up on PyPI, then go right ahead. comp.lang.python (aka python-list) would be the place to ask for interest from other developers. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at barrys-emacs.org Mon Oct 19 22:39:06 2009 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 19 Oct 2009 21:39:06 +0100 Subject: [Python-Dev] Add const to python API - issue 6952 Message-ID: http://bugs.python.org/issue6952 Martin v. L?wis suggested that solutions to this issue should be discussed here. My goal is to avoid compiler warning and the need to cast to remove const when calling the python API. For example I see compiler messages like this on Mac OS X Snow Leopard g++ reports: example.cxx:633: warning: deprecated conversion from string constant to ?char*? The patch I developed for comment only adds const to the input parameters and used casts to allow output parameters to stay without the const. This is because adding the const to the input params will not break existing code, but adding const to output parameters may well require code changes for users of the Python API. What is the best approach to this problem that will be acceptable? Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon Oct 19 22:58:47 2009 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 19 Oct 2009 16:58:47 -0400 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> Message-ID: <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> [Mark Dickinson] > By the way, here's an example of an *almost* real-life use of million digit > calculations. > > For an elementary number theory course that I taught a while ago, there > was an associated (optional) computer lab, where the students used > Python to investigate various ideas, conjectures, examples, etc. ?One > of the less open-ended questions might have been[1] something like: > > "On August 23rd, 2008 a computer at UCLA found the first example > of a prime with more than 10 million digits: p = 2**43112609-1. ?Find > > (1) the exact number of digits in p, when written out in decimal > (2) the last 100 digits of p > (3) the first 100 digits of p." > > It's wonderfully easy to get answers to these questions with Python: ... > But if you (not unreasonably) try to compute str(p), > you'll find it's impossibly slow, and it's very handy > that it's possible to interrupt that calculation, attempt > to understand *why* it's slow, and then try different > methods. Don't want to hijack this thread, but this is the kind of use case justifying keeping the 3-argument pow in the decimal module. People "playing" with number theory questions can learn a bag of tricks to worm around that non-decimal arithmetic can make it inconvenient & slow to get answers about decimal digits, but most people -- and especially beginners -- find life easier when doing calculations in decimal to begin with. Then they can use obvious methods, and not get sidetracked by wondering why they take forever to finish. Although, to be fair, I started >>> decimal.getcontext().prec = 20000000 >>> p10 = pow(decimal.Decimal(2), 43112609) before I started typing this, and am still waiting for it to finish ;-) OTOH, >>> decimal.getcontext().prec = 200 >>> pow(decimal.Decimal(2), 43112609) # get the first 100 digits (& then some) and >>> pow(decimal.Decimal(2), 43112609, 10**100) - 1 # get the last 100 digits both appear to work instantaneously. From dalcinl at gmail.com Tue Oct 20 05:10:45 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 20 Oct 2009 01:10:45 -0200 Subject: [Python-Dev] Python 2.6.4rc2 In-Reply-To: <594E6B4A-8D52-4A93-A368-E699FBA42535@python.org> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <594E6B4A-8D52-4A93-A368-E699FBA42535@python.org> Message-ID: I'm getting this warning. It seems nothing is actually broken, but the fix is pretty easy. gcc -pthread -c -fno-strict-aliasing -g -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c Objects/unicodeobject.c: In function 'PyUnicodeUCS2_FromFormatV': Objects/unicodeobject.c:687: warning: pointer targets in passing argument 1 of 'strlen' differ in signedness /usr/include/string.h:397: note: expected 'const char *' but argument is of type 'unsigned char *' Objects/unicodeobject.c:687: warning: pointer targets in passing argument 1 of 'PyUnicodeUCS2_DecodeUTF8' differ in signedness Include/unicodeobject.h:752: note: expected 'const char *' but argument is of type 'unsigned char *' BTW, should Python build with a C++ compiler? It seems this is not possible with 2.6.4rc2 (and GCC 4.4.1) On Mon, Oct 19, 2009 at 8:55 AM, Barry Warsaw wrote: > On Oct 19, 2009, at 3:59 AM, Vinay Sajip wrote: > >> Barry Warsaw python.org> writes: >> >>> http://www.python.org/download/releases/2.6.4/ >> >> Good news, but just one little nit: the page above appears to link to the >> NEWS >> file for 2.6.4rc1. > > Ooops! ?Fixed, thanks. > -Barry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dalcinl%40gmail.com > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From martin at v.loewis.de Tue Oct 20 05:35:05 2009 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Oct 2009 05:35:05 +0200 Subject: [Python-Dev] Add const to python API - issue 6952 In-Reply-To: References: Message-ID: <4ADD2FE9.2010807@v.loewis.de> > The patch I developed for comment only adds const to the input > parameters and used casts to > allow output parameters to stay without the const. What specific APIs are you talking about here? Regards, Martin From dickinsm at gmail.com Tue Oct 20 10:39:31 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 20 Oct 2009 09:39:31 +0100 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> Message-ID: <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> On Mon, Oct 19, 2009 at 9:58 PM, Tim Peters wrote: > Don't want to hijack this thread, but this is the kind of use case > justifying keeping the 3-argument pow in the decimal module. ?People > "playing" with number theory questions can learn a bag of tricks to > worm around that non-decimal arithmetic can make it inconvenient & > slow to get answers about decimal digits, but most people -- and > especially beginners -- find life easier when doing calculations in > decimal to begin with. ?Then they can use obvious methods, and not get > sidetracked by wondering why they take forever to finish. That's true. I wouldn't relish the task of trying to teach the decimal module at the same time as introducing students to Python and to elementary number theory, though. It's not the most friendly module to get started with. > Although, to be fair, I started > >>>> decimal.getcontext().prec = 20000000 >>>> p10 = pow(decimal.Decimal(2), 43112609) > > before I started typing this, and am still waiting for it to finish ;-) Hmm, yes. The decimal module isn't (currently) well set up for this sort of calculation, mostly because almost every operation is implemented by converting to binary, invoking the appropriate int/long arithmetic operation, and converting back. Since the conversion is O(n^2), even addition and multiplication of Decimal instances end up being O(n^2) operations at the moment, instead of getting the linear and Karatsuba (resp.) running times that they deserve. (The exception is rounding operations, which don't do any decimal <-> binary operations, but operate directly on the coefficient in its representation as a string of decimal digits.) This high-precision inefficiency could easily be fixed by using a dedicated 'decimal natural number' extension type for the Decimal coefficient, stored internally in base a suitable power of 10. I think this may be worth considering seriously. I'm not proposing this as an alternative to the huge task of rewriting the entire decimal module in C, by the way; it would be more a stop-gap measure that would be easy to implement, would slightly increase efficiency for normal operations, and vastly increase efficiency for high-precision operations. > > OTOH, > >>>> decimal.getcontext().prec = 200 >>>> pow(decimal.Decimal(2), 43112609) ? # get the first 100 digits (& then some) > > and > >>>> pow(decimal.Decimal(2), 43112609, 10**100) - 1 ?# get the last 100 digits > > both appear to work instantaneously. > Right. Low precision Decimal operations should be fine. I don't really see the advantage of the second operation over a simple pow(2, 43112609, 10**100) though. By the way, despite the above use-case, and despite the fact that I use it frequently, I still don't believe 3-argument pow belongs in the core. But that's probably a discussion topic for Python 4000. Mark From ncoghlan at gmail.com Tue Oct 20 12:49:27 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Oct 2009 20:49:27 +1000 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> Message-ID: <4ADD95B7.8040304@gmail.com> Mark Dickinson wrote: > This high-precision inefficiency could easily be fixed by > using a dedicated 'decimal natural number' extension > type for the Decimal coefficient, stored internally in base > a suitable power of 10. I think this may be worth > considering seriously. I'm not proposing this as an alternative > to the huge task of rewriting the entire decimal module in C, > by the way; it would be more a stop-gap measure that would > be easy to implement, would slightly increase efficiency for > normal operations, and vastly increase efficiency for high-precision > operations. Didn't you already start work on that concept with the deccoeff patch? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From barry at python.org Tue Oct 20 12:57:45 2009 From: barry at python.org (Barry Warsaw) Date: Tue, 20 Oct 2009 06:57:45 -0400 Subject: [Python-Dev] Python 2.6.4rc2 In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <594E6B4A-8D52-4A93-A368-E699FBA42535@python.org> Message-ID: On Oct 19, 2009, at 11:10 PM, Lisandro Dalcin wrote: > I'm getting this warning. It seems nothing is actually broken, but the > fix is pretty easy. > > gcc -pthread -c -fno-strict-aliasing -g -Wall -Wstrict-prototypes -I. > -IInclude -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o > Objects/unicodeobject.c > Objects/unicodeobject.c: In function 'PyUnicodeUCS2_FromFormatV': > Objects/unicodeobject.c:687: warning: pointer targets in passing > argument 1 of 'strlen' differ in signedness > /usr/include/string.h:397: note: expected 'const char *' but argument > is of type 'unsigned char *' > Objects/unicodeobject.c:687: warning: pointer targets in passing > argument 1 of 'PyUnicodeUCS2_DecodeUTF8' differ in signedness > Include/unicodeobject.h:752: note: expected 'const char *' but > argument is of type 'unsigned char *' This isn't a regression in 2.6.3, nor is it critical enough, to be fixed for 2.6.4. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From dickinsm at gmail.com Tue Oct 20 13:22:33 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 20 Oct 2009 12:22:33 +0100 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <4ADD95B7.8040304@gmail.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> <4ADD95B7.8040304@gmail.com> Message-ID: <5c6f2a5d0910200422g71560d5eic6503da0d4fdc0fa@mail.gmail.com> On Tue, Oct 20, 2009 at 11:49 AM, Nick Coghlan wrote: > Mark Dickinson wrote: >> This high-precision inefficiency could easily be fixed by >> using a dedicated 'decimal natural number' extension >> type for the Decimal coefficient, stored internally in base >> a suitable power of 10. [...] > > Didn't you already start work on that concept with the deccoeff patch? I did: the code can be seen at: http://svn.python.org/view/sandbox/trunk/decimal/decimal_in_c/ This code defines a Deccoeff type as above, providing base 10 natural number arithmetic, along with a skeletal _Decimal type. The work on Deccoeff is pretty much complete, but the _Decimal type is only just a beginning; the original intent was to slowly translate the Decimal code into C and move it into the _Decimal type. The code was working a few months ago (with all Decimal tests passing), but there have been some changes and bugfixes since then. I might try to resurrect that code, dropping the _Decimal type and just concentrating on Deccoeff. Mark From amk at amk.ca Tue Oct 20 13:43:25 2009 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 20 Oct 2009 07:43:25 -0400 Subject: [Python-Dev] Volunteer needed to organize summits Message-ID: <20091020114325.GB6355@amk-desktop.matrixgroup.net> I'd like to turn over the organization of the VM and Python Language Summits at PyCon 2010 to someone else, one or two people. (The same person doesn't need to organize both of them.) Why: in November PyCon will be three months away, so the guest list needs to be finalized and the invitations need to be sent. Yet I can't pull together the motivation to work on them; I contemplate the task for two minutes and then go do something else. It's obviously better if the summit tasks are being actively worked on instead of just drifting, so I want to give it up now. What's required: chiefly it's just a matter of sending and replying to e-mail. Draw up a guest list (I can provide last year's lists); think of new people & projects to be added, or e-mail someone to ask for suggestions; send out invitations and requests for agenda items; collect the responses so we know how many people are coming. You can also help moderate the summits on the day of the events, but if that's not feasible someone else could do it, or the groups could manage themselves. (Also sent to pycon-organizers, psf-members.) --amk From stefan-usenet at bytereef.org Tue Oct 20 15:15:47 2009 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Tue, 20 Oct 2009 15:15:47 +0200 Subject: [Python-Dev] Interest in integrating C decimal module into Python? Message-ID: <20091020131547.GA615@mail.bytereef.org> Hi, as some of you know, recently I've released an arbitrary precision C library for decimal arithmetic together with a Python module: http://www.bytereef.org/libmpdec.html http://www.bytereef.org/fastdec.html Both the library and the module have been tested extensively. Fastdec currently differs from decimal.py in a couple of ways that could be fixed. The license is AGPL, but if there is interest in integrating it into Python I'd release it under a Python-compatible license. There have been several approaches towards getting C decimal arithmetic into Python: http://bugs.python.org/issue2486 Fastdec follows Raymond Hettinger's suggestion to provide wrappers for an independent C implementation. Arguments in favour of fastdec are: * Complete implementation of Mike Cowlishaw's specification * C library can be tested independently * Redundant arithmetic module for tests against decimal.py * Faster than Java BigDecimal * Compares relatively well in speed against gmpy To be clear, I would not want to _replace_ decimal.py. Rather I'd like to see a cdecimal module alongside decimal.py. I know that ultimately there should be a PEP for module inclusions. The purpose of this post is to gauge interest. Specifically: 1. Are you generally in favour of a C decimal module in Python? 2. Would fastdec - after achieving full decimal.py compatibility - be a serious candidate? 3. Could I use this list to settle a couple of questions, or would perhaps a Python developer be willing to work with me to make it compatible? I'm asking this to avoid doing work that would not find acceptance afterwards. Thanks, Stefan Krah From ssteinerx at gmail.com Tue Oct 20 15:27:33 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Tue, 20 Oct 2009 09:27:33 -0400 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <20091020131547.GA615@mail.bytereef.org> References: <20091020131547.GA615@mail.bytereef.org> Message-ID: Shouldn't this be on python-ideas? S On Oct 20, 2009, at 9:15 AM, Stefan Krah wrote: > Hi, > > as some of you know, recently I've released an arbitrary precision > C library for decimal arithmetic together with a Python module: > > http://www.bytereef.org/libmpdec.html > http://www.bytereef.org/fastdec.html > > > Both the library and the module have been tested extensively. Fastdec > currently differs from decimal.py in a couple of ways that could be > fixed. The license is AGPL, but if there is interest in integrating it > into Python I'd release it under a Python-compatible license. > > > There have been several approaches towards getting C decimal > arithmetic > into Python: > > http://bugs.python.org/issue2486 > > > Fastdec follows Raymond Hettinger's suggestion to provide wrappers for > an independent C implementation. Arguments in favour of fastdec are: > > * Complete implementation of Mike Cowlishaw's specification > > * C library can be tested independently > > * Redundant arithmetic module for tests against decimal.py > > * Faster than Java BigDecimal > > * Compares relatively well in speed against gmpy > > > To be clear, I would not want to _replace_ decimal.py. Rather I'd > like to > see a cdecimal module alongside decimal.py. > > > I know that ultimately there should be a PEP for module inclusions. > The > purpose of this post is to gauge interest. Specifically: > > > 1. Are you generally in favour of a C decimal module in Python? > > > 2. Would fastdec - after achieving full decimal.py compatibility - be > a serious candidate? > > > 3. Could I use this list to settle a couple of questions, or would > perhaps > a Python developer be willing to work with me to make it > compatible? I'm > asking this to avoid doing work that would not find acceptance > afterwards. > > > > Thanks, > > Stefan Krah > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ssteinerx%40gmail.com From rdmurray at bitdance.com Tue Oct 20 15:47:33 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 20 Oct 2009 09:47:33 -0400 (EDT) Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: References: <20091020131547.GA615@mail.bytereef.org> Message-ID: On Tue, 20 Oct 2009 at 09:27, ssteinerX at gmail.com wrote: > Shouldn't this be on python-ideas? IMO this question is appropriate for python-dev, not python-ideas. --David From stefan-usenet at bytereef.org Tue Oct 20 15:43:33 2009 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Tue, 20 Oct 2009 15:43:33 +0200 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: References: <20091020131547.GA615@mail.bytereef.org> Message-ID: <20091020134333.GA9122@yoda.bytereef.org> ssteinerX at gmail.com wrote: > Shouldn't this be on python-ideas? I found previous discussions about "Decimal in C" on python-dev, that's why used this list. Stefan Krah From ssteinerx at gmail.com Tue Oct 20 15:55:06 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Tue, 20 Oct 2009 09:55:06 -0400 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <20091020134333.GA9122@yoda.bytereef.org> References: <20091020131547.GA615@mail.bytereef.org> <20091020134333.GA9122@yoda.bytereef.org> Message-ID: On Oct 20, 2009, at 9:43 AM, Stefan Krah wrote: > ssteinerX at gmail.com wrote: >> Shouldn't this be on python-ideas? > > I found previous discussions about "Decimal in C" on python-dev, > that's why > used this list. python-ideas: This list is to contain discussion of speculative language ideas for Python for possible inclusion into the language. If an idea gains traction it can then be discussed and honed to the point of becoming a solid proposal to put to either python-dev or python-3000 as appropriate. I guess it's a fine line...and matter of opinion. No worries. S From chris at simplistix.co.uk Tue Oct 20 16:18:50 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 20 Oct 2009 15:18:50 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ACF6993.5040704@voidspace.org.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> Message-ID: <4ADDC6CA.5000809@simplistix.co.uk> [following up on distutils-sig] Michael Foord wrote: > Many Windows users would be quite happy if the standard mechanism for > installing non-source distributions on Windows was via the wininst > binaries. ...and many users may not be ;-) I know I'd be extremely unhappy if that were the case as I routines use multiple versions of packages with multiple versions of python on one machine... > I wonder if it is going to be possible to make this compatible with the > upcoming distutils package management 'stuff' (querying for installed > packages, uninstallation etc) since installation/uninstallation goes > through the Windows system package management feature. I guess it would > be eminently possible but require some reasonably high level Windows-fu > to do. I wouldn't have a problem if integrating with the windows package manager was an optional extra, but I think it's one of many types of package management that need to be worried about, so might be easier to get the others working and let anyone who wants anything beyond a pure-python packaging system that works across platforms, regardless of whether binary extensions are needed, do the work themselves... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Oct 20 16:22:11 2009 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 20 Oct 2009 15:22:11 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ADDC6CA.5000809@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <4ADDC6CA.5000809@simplistix.co.uk> Message-ID: <4ADDC793.9050507@simplistix.co.uk> Chris Withers wrote: > [following up on distutils-sig] ...FAIL, sorry for the noise. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From rdmurray at bitdance.com Tue Oct 20 16:40:05 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 20 Oct 2009 10:40:05 -0400 (EDT) Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: References: <20091020131547.GA615@mail.bytereef.org> <20091020134333.GA9122@yoda.bytereef.org> Message-ID: On Tue, 20 Oct 2009 at 09:55, ssteinerX at gmail.com wrote: > On Oct 20, 2009, at 9:43 AM, Stefan Krah wrote: >> ssteinerX at gmail.com wrote: >> > Shouldn't this be on python-ideas? >> >> I found previous discussions about "Decimal in C" on python-dev, that's why >> used this list. > > python-ideas: > > This list is to contain discussion of speculative language ideas for Python > for possible inclusion into the language. If an idea gains traction it can > then be discussed and honed to the point of becoming a solid proposal to put > to either python-dev or python-3000 as appropriate. > > I guess it's a fine line...and matter of opinion. No worries. In this case it isn't a language idea under discussion, but the possible inclusion of an implementation of an idea (and moreover an idea that is already included in Python in another, less efficient, form). (I'm expecting that both Mark Dickinson and Raymond Hettinger will comment on this thread eventually...) --David (RDM) From eric at trueblade.com Tue Oct 20 16:50:07 2009 From: eric at trueblade.com (Eric Smith) Date: Tue, 20 Oct 2009 10:50:07 -0400 (EDT) Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <5c6f2a5d0910200422g71560d5eic6503da0d4fdc0fa@mail.gmail.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> <4ADD95B7.8040304@gmail.com> <5c6f2a5d0910200422g71560d5eic6503da0d4fdc0fa@mail.gmail.com> Message-ID: <41504.63.251.87.214.1256050207.squirrel@mail.trueblade.com> > On Tue, Oct 20, 2009 at 11:49 AM, Nick Coghlan wrote: >> Mark Dickinson wrote: >>> This high-precision inefficiency could easily be fixed by >>> using a dedicated 'decimal natural number' extension >>> type for the Decimal coefficient, stored internally in base >>> a suitable power of 10. [...] >> >> Didn't you already start work on that concept with the deccoeff patch? > > I did: the code can be seen at: > > http://svn.python.org/view/sandbox/trunk/decimal/decimal_in_c/ > > This code defines a Deccoeff type as above, providing base 10 natural > number arithmetic, along with a skeletal _Decimal type. The work on > Deccoeff is pretty much complete, but the _Decimal type is only just > a beginning; the original intent was to slowly translate the Decimal > code into C and move it into the _Decimal type. > > The code was working a few months ago (with all Decimal tests > passing), but there have been some changes and bugfixes since > then. I might try to resurrect that code, dropping the _Decimal type and > just concentrating on Deccoeff. My only concern about this is the effect it would have on IronPython, Jython, PyPy, and other alternate implementations that use the stdlib. From dickinsm at gmail.com Tue Oct 20 16:57:13 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 20 Oct 2009 15:57:13 +0100 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <41504.63.251.87.214.1256050207.squirrel@mail.trueblade.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> <4ADD95B7.8040304@gmail.com> <5c6f2a5d0910200422g71560d5eic6503da0d4fdc0fa@mail.gmail.com> <41504.63.251.87.214.1256050207.squirrel@mail.trueblade.com> Message-ID: <5c6f2a5d0910200757s32b814aeod5b7d6d8c202df47@mail.gmail.com> On Tue, Oct 20, 2009 at 3:50 PM, Eric Smith wrote: >> The code was working a few months ago (with all Decimal tests >> passing), but there have been some changes and bugfixes since >> then. ?I might try to resurrect that code, dropping the _Decimal type and >> just concentrating on Deccoeff. > > My only concern about this is the effect it would have on IronPython, > Jython, PyPy, and other alternate implementations that use the stdlib. Yes, that worries me a bit, too. I have the same worry with the idea of rewriting the entire decimal module in C. The Deccoeff type is very simple, though. It would be easy to create a pure Python version of it, and then do something like: try: from _decimal import Deccoeff # try to get C version except ImportError: from deccoeff import Deccoeff # else use Python fallback code. Mark From p.f.moore at gmail.com Tue Oct 20 22:49:42 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Oct 2009 21:49:42 +0100 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <4ADDC6CA.5000809@simplistix.co.uk> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <4ADDC6CA.5000809@simplistix.co.uk> Message-ID: <79990c6b0910201349w6f0537f2q1eb91bb4fd8d3c8d@mail.gmail.com> 2009/10/20 Chris Withers : > I wouldn't have a problem if integrating with the windows package manager > was an optional extra, but I think it's one of many types of package > management that need to be worried about, so might be easier to get the > others working and let anyone who wants anything beyond a pure-python > packaging system that works across platforms, regardless of whether binary > extensions are needed, do the work themselves... There are many (I believe) Windows users for whom bdist_wininst is just what they want. For those people, where's the incentive to switch in what you propose? You're not providing the features they currently have, and frankly "do the work yourself" is no answer (not everyone can, often for entirely legitimate reasons). If such users could ignore the new offering, that would be fine - but they can't, if projects stop providing bdist_wininst installers. That's what's happening with eggs already, and yes, it *is* a pain. And I'm in a position where I *can* build my own bdist_wininst installer - but often, I'll just not bother using a package. So packages lose users - does this matter? Who can tell? In my view, the number one priority is to have a single distribution format. I really don't care what it is. But *one*. bdist_wininst used to be that on Windows, and for all its limitations it was a huge benefit. Eggs messed that up, essentially because they didn't provide all the features of bdist_wininst (uninstallers...) so they didn't replace bdist_wininst, they sat alongside. Now you're proposing to make the same mistake again. Can I repeat that in big letters? The key is a SINGLE DISTRIBUTION FORMAT. If you can persuade everyone to accept a format which ignores clearly stated user requirements, go for it. But if you can't, you're making the problem worse rather than helping. My money's on a solution that acknowledges and addresses user requirements instead. Paul. From brett at python.org Tue Oct 20 22:54:53 2009 From: brett at python.org (Brett Cannon) Date: Tue, 20 Oct 2009 13:54:53 -0700 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: <5c6f2a5d0910200757s32b814aeod5b7d6d8c202df47@mail.gmail.com> References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> <4ADD95B7.8040304@gmail.com> <5c6f2a5d0910200422g71560d5eic6503da0d4fdc0fa@mail.gmail.com> <41504.63.251.87.214.1256050207.squirrel@mail.trueblade.com> <5c6f2a5d0910200757s32b814aeod5b7d6d8c202df47@mail.gmail.com> Message-ID: On Tue, Oct 20, 2009 at 07:57, Mark Dickinson wrote: > On Tue, Oct 20, 2009 at 3:50 PM, Eric Smith wrote: > >> The code was working a few months ago (with all Decimal tests > >> passing), but there have been some changes and bugfixes since > >> then. I might try to resurrect that code, dropping the _Decimal type > and > >> just concentrating on Deccoeff. > > > > My only concern about this is the effect it would have on IronPython, > > Jython, PyPy, and other alternate implementations that use the stdlib. > > Yes, that worries me a bit, too. I have the same worry with the idea > of rewriting the entire decimal module in C. > > The Deccoeff type is very simple, though. It would be easy to create > a pure Python version of it, and then do something like: > > try: > from _decimal import Deccoeff # try to get C version > except ImportError: > from deccoeff import Deccoeff # else use Python fallback code. And this is why you shouldn't have to worry. As long as the tests exercise both the pure Python and C versions then the other VMs are covered. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Oct 20 23:00:12 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Oct 2009 22:00:12 +0100 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <20091020131547.GA615@mail.bytereef.org> References: <20091020131547.GA615@mail.bytereef.org> Message-ID: <79990c6b0910201400k24e5ad4aj3813dc69d5049c9a@mail.gmail.com> 2009/10/20 Stefan Krah : > Hi, > > as some of you know, recently I've released an arbitrary precision > C library for decimal arithmetic together with a Python module: > > http://www.bytereef.org/libmpdec.html > http://www.bytereef.org/fastdec.html > > > Both the library and the module have been tested extensively. Fastdec > currently differs from decimal.py in a couple of ways that could be > fixed. The license is AGPL, but if there is interest in integrating it > into Python I'd release it under a Python-compatible license. > > > There have been several approaches towards getting C decimal arithmetic > into Python: > > http://bugs.python.org/issue2486 > > > Fastdec follows Raymond Hettinger's suggestion to provide wrappers for > an independent C implementation. Arguments in favour of fastdec are: > > ?* Complete implementation of Mike Cowlishaw's specification > > ?* C library can be tested independently > > ?* Redundant arithmetic module for tests against decimal.py > > ?* Faster than Java BigDecimal > > ?* Compares relatively well in speed against gmpy > > > To be clear, I would not want to _replace_ decimal.py. Rather I'd like to > see a cdecimal module alongside decimal.py. Why? If it's 100% compatible with decimal.py, just replace it. All the user should see is improved speed. Let's not do another pickle/cpickle. > I know that ultimately there should be a PEP for module inclusions. The > purpose of this post is to gauge interest. Specifically: > > > 1. Are you generally in favour of a C decimal module in Python? Yes, although I have to admit my interest is fairly theoretical. > 2. Would fastdec - after achieving full decimal.py compatibility - be > ? a serious candidate? I don't see why not, if it was 100% compatible with decimal.py > 3. Could I use this list to settle a couple of questions, or would perhaps > ? a Python developer be willing to work with me to make it compatible? I'm > ? asking this to avoid doing work that would not find acceptance afterwards. I can't help much here, but I'd prefer to see discussions on python-dev so I'm +1 on keeping discussions here. Waiting eagerly to hear the experts' comments... Paul. From barry at barrys-emacs.org Tue Oct 20 23:03:39 2009 From: barry at barrys-emacs.org (Barry Scott) Date: Tue, 20 Oct 2009 22:03:39 +0100 Subject: [Python-Dev] Add const to python API - issue 6952 In-Reply-To: <4ADD2FE9.2010807@v.loewis.de> References: <4ADD2FE9.2010807@v.loewis.de> Message-ID: <74D1336F-AB87-431F-895C-885EE344CEBD@barrys-emacs.org> On 20 Oct 2009, at 04:35, Martin v. L?wis wrote: >> The patch I developed for comment only adds const to the input >> parameters and used casts to >> allow output parameters to stay without the const. > > What specific APIs are you talking about here? Checking my patch I have two functions that need to have output params changed to const to avoid casting. PyOS_strtoul - ptr PyLong_FromString - pend Barry From daniel at stutzbachenterprises.com Tue Oct 20 23:31:19 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Tue, 20 Oct 2009 16:31:19 -0500 Subject: [Python-Dev] Add const to python API - issue 6952 In-Reply-To: <74D1336F-AB87-431F-895C-885EE344CEBD@barrys-emacs.org> References: <4ADD2FE9.2010807@v.loewis.de> <74D1336F-AB87-431F-895C-885EE344CEBD@barrys-emacs.org> Message-ID: On Tue, Oct 20, 2009 at 4:03 PM, Barry Scott wrote: > Checking my patch I have two functions that need to > have output params changed to const to avoid casting. > > PyOS_strtoul - ptr > PyLong_FromString - pend > This is a no-win situation. If the string is const in the caller, they currently need to cast it. If you make the change, then if string is not const in the caller then they will need to cast it. I've provided a short example below and marked the lines that generate "an incompatible pointer type" warning with gcc. I suggest following POSIX's lead and omitted the const in these cases. void test_const(const char **foo); void test_noconst(char **foo); int main(void) { char *noconst_var; const char *const_var; test_const(&noconst_var); // generates a warning test_const(&const_var); test_noconst(&noconst_var); test_noconst(&const_var); // generates a warning return 0; } -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lyon at preisshare.net Tue Oct 20 23:40:07 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 20 Oct 2009 17:40:07 -0400 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910201349w6f0537f2q1eb91bb4fd8d3c8d@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081555h13c8b48cg8810377912633dc5@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <4ADDC6CA.5000809@simplistix.co.uk> <79990c6b0910201349w6f0537f2q1eb91bb4fd8d3c8d@mail.gmail.com> Message-ID: <52bce2be86dbbd753f612256602d2c70@preisshare.net> On Tue, 20 Oct 2009 21:49:42 +0100, Paul Moore wrote: > Can I repeat that in big letters? The key is a SINGLE DISTRIBUTION FORMAT. ok - but that pretty much exists.. > If you can persuade everyone to accept a format which ignores clearly > stated user requirements, go for it. But if you can't, you're making > the problemorse rather than helping. My money's on a solution that > acknowledges and addresses user requirements instead. What solution is that exactly? You don't say.. I would be happy to revamp pythonpkgmgr and backgrade to have a tk interface if it is felt that would help. That's not so difficult - certainly not on my side. I don't know which iceberg the people that want to keep command line interfaces forever still live on, but they seem very good at sinking any attempts of passage by forward moving boats.. Distutils roadmap for windows is to apply more freeze spray.. it's a cryogenic source code freezing iceberg... It's just there for windows users to crash their boats into.. Try and see for yourself.. David From fijall at gmail.com Wed Oct 21 00:49:28 2009 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 20 Oct 2009 16:49:28 -0600 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <79990c6b0910201400k24e5ad4aj3813dc69d5049c9a@mail.gmail.com> References: <20091020131547.GA615@mail.bytereef.org> <79990c6b0910201400k24e5ad4aj3813dc69d5049c9a@mail.gmail.com> Message-ID: <693bc9ab0910201549o101673fbpda9e7ff5131bfd2@mail.gmail.com> On Tue, Oct 20, 2009 at 3:00 PM, Paul Moore wrote: > 2009/10/20 Stefan Krah : >> Hi, >> >> as some of you know, recently I've released an arbitrary precision >> C library for decimal arithmetic together with a Python module: >> >> http://www.bytereef.org/libmpdec.html >> http://www.bytereef.org/fastdec.html >> >> >> Both the library and the module have been tested extensively. Fastdec >> currently differs from decimal.py in a couple of ways that could be >> fixed. The license is AGPL, but if there is interest in integrating it >> into Python I'd release it under a Python-compatible license. >> >> >> There have been several approaches towards getting C decimal arithmetic >> into Python: >> >> http://bugs.python.org/issue2486 >> >> >> Fastdec follows Raymond Hettinger's suggestion to provide wrappers for >> an independent C implementation. Arguments in favour of fastdec are: >> >> ?* Complete implementation of Mike Cowlishaw's specification >> >> ?* C library can be tested independently >> >> ?* Redundant arithmetic module for tests against decimal.py >> >> ?* Faster than Java BigDecimal >> >> ?* Compares relatively well in speed against gmpy >> >> >> To be clear, I would not want to _replace_ decimal.py. Rather I'd like to >> see a cdecimal module alongside decimal.py. > > Why? If it's 100% compatible with decimal.py, just replace it. All the > user should see is improved speed. Let's not do another > pickle/cpickle. For example other python implementations might decide to use python version as long as builtin version does not appear. Python versions are usually also better targets for jit than mixed versions. C level versions also usually have more bugs (just statistics), so some people might want to choose pure-python version. In general - some people have some reasons. Cheers, fijal From cournape at gmail.com Wed Oct 21 04:44:25 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 21 Oct 2009 11:44:25 +0900 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <79990c6b0910201349w6f0537f2q1eb91bb4fd8d3c8d@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <4ACE718D.8090400@cryptelium.net> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <4ADDC6CA.5000809@simplistix.co.uk> <79990c6b0910201349w6f0537f2q1eb91bb4fd8d3c8d@mail.gmail.com> Message-ID: <5b8d13220910201944h36577afeu15f96c4cbf86dd88@mail.gmail.com> On Wed, Oct 21, 2009 at 5:49 AM, Paul Moore wrote: > 2009/10/20 Chris Withers : >> I wouldn't have a problem if integrating with the windows package manager >> was an optional extra, but I think it's one of many types of package >> management that need to be worried about, so might be easier to get the >> others working and let anyone who wants anything beyond a pure-python >> packaging system that works across platforms, regardless of whether binary >> extensions are needed, do the work themselves... > > There are many (I believe) Windows users for whom bdist_wininst is > just what they want. For those people, where's the incentive to switch > in what you propose? You're not providing the features they currently > have, and frankly "do the work yourself" is no answer (not everyone > can, often for entirely legitimate reasons). I am not so familiar with msi or wininst internals, but isn't it possible to install w.r.t. a given prefix ? Basically, making it possible to use a wininst in a virtualenv if required (in which case I guess it would not register with the windows db - at least it should be possible to disable it). The main problem with bdist_wininst installers is that they don't work with setuptools dependency stuff (at least, that's the reason given by windows users for a numpy egg on windows, whereas we used to only provide an exe). But you could argue it is a setuptools pb as much as a wininst pb, I guess. David From steven.bethard at gmail.com Wed Oct 21 07:05:51 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 20 Oct 2009 22:05:51 -0700 Subject: [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip) In-Reply-To: <5b8d13220910201944h36577afeu15f96c4cbf86dd88@mail.gmail.com> References: <94bdd2610910080131j323b98d9i871bce43465f237a@mail.gmail.com> <94bdd2610910081621n502803e4l8efeabae3c5863ff@mail.gmail.com> <79990c6b0910090532y63169b60q8dd0a95f056c244d@mail.gmail.com> <4ACF6993.5040704@voidspace.org.uk> <4ADDC6CA.5000809@simplistix.co.uk> <79990c6b0910201349w6f0537f2q1eb91bb4fd8d3c8d@mail.gmail.com> <5b8d13220910201944h36577afeu15f96c4cbf86dd88@mail.gmail.com> Message-ID: On Tue, Oct 20, 2009 at 7:44 PM, David Cournapeau wrote: > On Wed, Oct 21, 2009 at 5:49 AM, Paul Moore wrote: >> 2009/10/20 Chris Withers : >> There are many (I believe) Windows users for whom bdist_wininst is >> just what they want. For those people, where's the incentive to switch >> in what you propose? You're not providing the features they currently >> have, and frankly "do the work yourself" is no answer (not everyone >> can, often for entirely legitimate reasons). > > I am not so familiar with msi or wininst internals, but isn't it > possible to install w.r.t. a given prefix ? This is definitely supported by the msi. In fact, you can do this at the command line (although I don't know if this is considered officially supported or not): $ argparse-1.0.2.win32-py2.6.msi /quiet TARGETDIR=C:\TempPython $ dir C:\TempPython\Lib\site-packages Volume in drive C has no label. Volume Serial Number is B6E6-4B4D Directory of C:\TempPython\Lib\site-packages 10/20/2009 10:04 PM . 10/20/2009 10:04 PM .. 10/20/2009 10:03 PM 1,584 argparse-1.0.2-py3.1.egg-info 10/09/2009 11:03 AM 86,620 argparse.py Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From martin at v.loewis.de Wed Oct 21 07:15:14 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 21 Oct 2009 07:15:14 +0200 Subject: [Python-Dev] Add const to python API - issue 6952 In-Reply-To: References: <4ADD2FE9.2010807@v.loewis.de> <74D1336F-AB87-431F-895C-885EE344CEBD@barrys-emacs.org> Message-ID: <4ADE98E2.3080707@v.loewis.de> > I suggest following POSIX's lead and omitted the const in these cases. Thanks, that sounds reasonable. Regards, Martin From kristjan at ccpgames.com Wed Oct 21 12:15:52 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 21 Oct 2009 10:15:52 +0000 Subject: [Python-Dev] [Python-ideas] Remove GIL with CAS instructions? In-Reply-To: <4ADE35F0.10803@molden.no> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> > -----Original Message----- > From: python-ideas-bounces+kristjan=ccpgames.com at python.org > [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On > Behalf Of Sturla Molden > Sent: 20. okt?ber 2009 22:13 > To: python-ideas at python.org > Subject: Re: [Python-ideas] Remove GIL with CAS instructions? > > > - The GIL has consequences on multicore CPUs that are overlooked: > thread > switches are usually missed at check intervals. This could be fixed > without removing the GIL: For example, there could be a wait-queue for > the GIL; a thread that request the GIL puts itself in the back. > This depends entirely on the platform and primitives used to implement the GIL. I'm interested in windows. There, I found this article: http://fonp.blogspot.com/2007/10/fairness-in-win32-lock-objects.html So, you may be on to something. Perhaps a simple C test is in order then? I did that. I found, on my dual-core vista machine, that running "release", that both Mutexes and CriticalSections behaved as you describe, with no "fairness". Using a "semaphore" seems to retain fairness, however. "fairness" was retained in debug builds too, strangely enough. Now, Python uses none of these. On windows, it uses an "Event" object coupled with an atomically updated counter. This also behaves fairly. The test application is attached. I think that you ought to sustantiate your claims better, maybe with a specific platform and using some test like the above. On the other hand, it shows that we must be careful what we use. There has been some talk of using CriticalSections for the GIL on windows. This test ought to show the danger of that. The GIL is different than a regular lock. It is a reverse-lock, really, and therefore may need to be implemented in its own special way, if we want very fast mutexes for the rest of the system (cc to python-dev) Cheers, Kristj?n -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: giltest.cpp URL: From ncoghlan at gmail.com Wed Oct 21 12:27:58 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Oct 2009 20:27:58 +1000 Subject: [Python-Dev] SIGCHECK() in longobject.c In-Reply-To: References: <5c6f2a5d0910190427v74a79cacrf4a472017713763@mail.gmail.com> <1f7befae0910191358k52496861r4a6b39dae58f5235@mail.gmail.com> <5c6f2a5d0910200139p4ef82df2k949eba8fa33365fb@mail.gmail.com> <4ADD95B7.8040304@gmail.com> <5c6f2a5d0910200422g71560d5eic6503da0d4fdc0fa@mail.gmail.com> <41504.63.251.87.214.1256050207.squirrel@mail.trueblade.com> <5c6f2a5d0910200757s32b814aeod5b7d6d8c202df47@mail.gmail.com> Message-ID: <4ADEE22E.2090002@gmail.com> Brett Cannon wrote: > On Tue, Oct 20, 2009 at 07:57, Mark Dickinson > wrote: > The Deccoeff type is very simple, though. It would be easy to create > a pure Python version of it, and then do something like: > > try: > from _decimal import Deccoeff # try to get C version > except ImportError: > from deccoeff import Deccoeff # else use Python fallback code. > > > And this is why you shouldn't have to worry. As long as the tests > exercise both the pure Python and C versions then the other VMs are covered. We need to worry at least a little bit, as the pure Python version doesn't exist at this point in time. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Oct 21 12:37:39 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Oct 2009 20:37:39 +1000 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <693bc9ab0910201549o101673fbpda9e7ff5131bfd2@mail.gmail.com> References: <20091020131547.GA615@mail.bytereef.org> <79990c6b0910201400k24e5ad4aj3813dc69d5049c9a@mail.gmail.com> <693bc9ab0910201549o101673fbpda9e7ff5131bfd2@mail.gmail.com> Message-ID: <4ADEE473.9020508@gmail.com> Maciej Fijalkowski wrote: > For example other python implementations might decide to use python > version as long as builtin version does not appear. Python versions are > usually also better targets for jit than mixed versions. C level versions also > usually have more bugs (just statistics), so some people might want to > choose pure-python version. > > In general - some people have some reasons. Although nobody has broken "sys.modules['_decimal'] = 0", so deliberately turning off optimisations is pretty easy if you really don't want them. There's a reason we moved to implicit import of optimised versions in Py3k - we're unlikely to revert to the old way of doing things. As far as decimal.py in particular goes, there are significant maintenance gains in keeping a lot of the non-performance critical context management code in pure Python. So we're likely to wait and see how much speed Mark can wring out of a simple C decimal coefficient object (that other implementations can also fairly easily provide natively) before looking seriously at a wholesale replacement of the module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Wed Oct 21 12:51:43 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 10:51:43 +0000 (UTC) Subject: [Python-Dev] GIL behaviour under Windows References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> Message-ID: Hello Kristjan, > This depends entirely on the platform and primitives used to implement the GIL. > I'm interested in windows. Could you try ccbench (*) under Windows? The only Windows system I have here is a qemu virtual machine and it wouldn't be very realistic to do concurrency measurements on it. (*) http://svn.python.org/view/sandbox/trunk/ccbench/ Thanks Antoine. From scott+python-dev at scottdial.com Wed Oct 21 14:28:48 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 21 Oct 2009 08:28:48 -0400 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> Message-ID: <4ADEFE80.3050002@scottdial.com> Antoine Pitrou wrote: > Could you try ccbench (*) under Windows? The only Windows system I have here is > a qemu virtual machine and it wouldn't be very realistic to do concurrency > measurements on it. > > (*) http://svn.python.org/view/sandbox/trunk/ccbench/ > I don't really know how this test works, so I won't claim to understand the results either. However, here you go: C:\>systeminfo OS Name: Microsoft Windows XP Professional OS Version: 5.1.2600 Service Pack 3 Build 2600 C:\>c:\Python26\python.exe Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32 C:\>start /B /HIGH c:\Python26\python.exe c:\ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 377 iterations/s. threads=2: 376 ( 99 %) threads=3: 380 ( 100 %) threads=4: 376 ( 99 %) regular expression (C) threads=1: 222 iterations/s. threads=2: 213 ( 95 %) threads=3: 223 ( 100 %) threads=4: 218 ( 97 %) bz2 compression (C) threads=1: 324 iterations/s. threads=2: 324 ( 99 %) threads=3: 327 ( 100 %) threads=4: 324 ( 100 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From solipsis at pitrou.net Wed Oct 21 14:52:40 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 14:52:40 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <4ADEFE80.3050002@scottdial.com> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <4ADEFE80.3050002@scottdial.com> Message-ID: <1256129560.5069.30.camel@localhost> > I don't really know how this test works, so I won't claim to understand > the results either. However, here you go: Thanks. Interesting results. I wonder what they would be like on a multi-core machine. The GIL seems to behave perfectly on your setup (no performance degradation due to concurrency, and zero latencies). For a quick explanation of what the benchmark does: - the "throughput" part launches N computational (CPU-bound) threads and measures the total work done per second, and then compares the result to the 1-thread result. It does so with three different workloads, which have different impacts on the GIL. 100% is the most you can get on a single-core machine. On a multi-core machine, you can get more than 100% with the workload that explicitly releases the GIL before taxing the CPU (bz2 compression). - the "latency" part launches N computational threads in the background, and the main thread listens for periodic ping messages on an UDP socket (the ping messages themselves are emitted from a separate Python process, so as to decouple it from the process under test). The latencies are the measured delay between the emission of the UDP message and the moment at which the recv() call returns in the main thread. This aims at reproducing the situation where a thread handles IO operations while one or several other threads perform heavy computations. Regards Antoine. > C:\>systeminfo > OS Name: Microsoft Windows XP Professional > OS Version: 5.1.2600 Service Pack 3 Build 2600 > > C:\>c:\Python26\python.exe > Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit > (Intel)] on win32 > > C:\>start /B /HIGH c:\Python26\python.exe c:\ccbench.py > --- Throughput --- > > Pi calculation (Python) > > threads=1: 377 iterations/s. > threads=2: 376 ( 99 %) > threads=3: 380 ( 100 %) > threads=4: 376 ( 99 %) > > regular expression (C) > > threads=1: 222 iterations/s. > threads=2: 213 ( 95 %) > threads=3: 223 ( 100 %) > threads=4: 218 ( 97 %) > > bz2 compression (C) > > threads=1: 324 iterations/s. > threads=2: 324 ( 99 %) > threads=3: 327 ( 100 %) > threads=4: 324 ( 100 %) > > --- Latency --- > > Background CPU task: Pi calculation (Python) > > CPU threads=0: 0 ms. (std dev: 0 ms.) > CPU threads=1: 0 ms. (std dev: 0 ms.) > CPU threads=2: 0 ms. (std dev: 0 ms.) > CPU threads=3: 0 ms. (std dev: 0 ms.) > CPU threads=4: 0 ms. (std dev: 0 ms.) > > Background CPU task: regular expression (C) > > CPU threads=0: 0 ms. (std dev: 0 ms.) > CPU threads=1: 0 ms. (std dev: 0 ms.) > CPU threads=2: 0 ms. (std dev: 0 ms.) > CPU threads=3: 0 ms. (std dev: 0 ms.) > CPU threads=4: 0 ms. (std dev: 0 ms.) > > Background CPU task: bz2 compression (C) > > CPU threads=0: 0 ms. (std dev: 0 ms.) > CPU threads=1: 0 ms. (std dev: 0 ms.) > CPU threads=2: 0 ms. (std dev: 0 ms.) > CPU threads=3: 0 ms. (std dev: 0 ms.) > CPU threads=4: 0 ms. (std dev: 0 ms.) > From scott+python-dev at scottdial.com Wed Oct 21 15:15:51 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 21 Oct 2009 09:15:51 -0400 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <1256129560.5069.30.camel@localhost> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <4ADEFE80.3050002@scottdial.com> <1256129560.5069.30.camel@localhost> Message-ID: <4ADF0987.9000500@scottdial.com> Antoine Pitrou wrote: > Interesting results. I wonder what they would be like on a multi-core > machine. The GIL seems to behave perfectly on your setup (no performance > degradation due to concurrency, and zero latencies). You are correct, my machine is a single-core system. I don't have any multi-core systems around to test it on, I'm still in the stone age. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From kristjan at ccpgames.com Wed Oct 21 15:15:59 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 21 Oct 2009 13:15:59 +0000 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF41FFC@exchis.ccp.ad.local> > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Antoine Pitrou > Sent: 21. okt?ber 2009 10:52 > To: python-dev at python.org > Subject: Re: [Python-Dev] GIL behaviour under Windows > > > Hello Kristjan, > > > This depends entirely on the platform and primitives used to > implement the GIL. > > I'm interested in windows. > > Could you try ccbench (*) under Windows? The only Windows system I have > here is > a qemu virtual machine and it wouldn't be very realistic to do > concurrency > measurements on it. > Hi, I just want to stress, that according to my test, the current GIL implementation works as intended on windows. But if we were to reimplement it, say using a CriticalSection, then yielding the GIL as we do in sys.checkinterval wouldn't work as intended anymore. Just something to keep in mind in case anyone is thinking along those lines. K From kristjan at ccpgames.com Wed Oct 21 15:24:15 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 21 Oct 2009 13:24:15 +0000 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> > -----Original Message----- > Could you try ccbench (*) under Windows? The only Windows system I have > here is > a qemu virtual machine and it wouldn't be very realistic to do > concurrency > measurements on it. > > (*) http://svn.python.org/view/sandbox/trunk/ccbench/ I?ve run it twice on my dual core machine. It hangs every time, but not in the same place: D:\pydev\python\trunk\PCbuild>python.exe \tmp\ccbench.py malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 --- Throughput --- Pi calculation (Python) threads=1: 514 iterations/s. threads=2: 403 ( 78 %) threads=3: 392 ( 76 %) threads=4: 364 ( 70 %) regular expression (C) threads=1: 443 iterations/s. threads=2: 474 ( 106 %) threads=3: 461 ( 104 %) threads=4: 466 ( 105 %) SHA1 hashing (C) threads=1: 983 iterations/s. threads=2: 1026 ( 104 %) ^C D:\pydev\python\trunk\PCbuild>python.exe \tmp\ccbench.py malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 malloc 262144 gave 262144, diff 0 --- Throughput --- Pi calculation (Python) threads=1: 506 iterations/s. threads=2: 405 ( 80 %) From solipsis at pitrou.net Wed Oct 21 15:34:47 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 15:34:47 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> Message-ID: <1256132087.5069.39.camel@localhost> > > (*) http://svn.python.org/view/sandbox/trunk/ccbench/ > > I?ve run it twice on my dual core machine. It hangs every time, but not in the same place: > D:\pydev\python\trunk\PCbuild>python.exe \tmp\ccbench.py Ah, you should report a bug then. ccbench is pure Python (and not particularly evil Python), it shouldn't be able to crash the interpreter. From dickinsm at gmail.com Wed Oct 21 16:51:41 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 21 Oct 2009 15:51:41 +0100 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <20091020131547.GA615@mail.bytereef.org> References: <20091020131547.GA615@mail.bytereef.org> Message-ID: <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> On Tue, Oct 20, 2009 at 2:15 PM, Stefan Krah wrote: > 1. Are you generally in favour of a C decimal module in Python? I'm certainly interested in the general idea; whether I'd be in favour of replacing decimal.py with a particular C version would depend on a lot of factors, with code quality, interchangeability with the current decimal module, and maintainability by Python core developers high on the list. There's also the question of what IronPython and Jython would do. > 2. Would fastdec - after achieving full decimal.py compatibility - be > ? a serious candidate? Definitely. As far as I know it's the only real candidate for a full C version of decimal right now. Other possibilities that I'm aware of: * I think Raymond Hettinger is working on a C version of decimal. I'm not sure what its current status is. Raymond? * There's a partially complete rewrite of decimal in C in the sandbox, dating from the Need for Speed sprint in 2006: * Wrapping the decNumber library directly would make some sense, but I think licensing issues rule this out. http://svn.python.org/view/sandbox/trunk/decimal-c/ Last time I looked at this it wasn't up to date with the decimal specification: I'm not sure that functions like exp, log and log10 are currently working. Georg Brandl might know better than I do. > 3. Could I use this list to settle a couple of questions, or would perhaps > ? a Python developer be willing to work with me to make it compatible? I'm > ? asking this to avoid doing work that would not find acceptance afterwards. I don't see why not. Working closely with one of the core developers on this sounds like a good idea, as well: if the code were to go into Python core, at least one (and preferably more than one) core dev should be familiar enough with it for maintenance. (BTW, I think that *if* fastdec went in it's likely you'd be granted commit privileges at some point.) It's difficult to gauge the likelihood of eventual acceptance in advance, though. Maybe writing a PEP would be an efficient use of time in this regard? There are certainly some open issues (e.g., what to do with the existing Python module; what should other Python implementations do). I think my biggest concern is maintenance: we'd be replacing 8500 lines of Python code in a single file, that several of the current core developers understand well, with 30000 (Stefan, is that about accurate?) lines of C in several files, that presumably none of the current core devs is familiar with right now. What happens when (e.g.,) the number-theoretic transform needs updating for one reason or another? Stefan, do you see yourself having time to spend on maintenance of this code for the forseeable future? BTW, does anyone know the current SLOC count for py3k? Mark From dickinsm at gmail.com Wed Oct 21 16:59:44 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 21 Oct 2009 15:59:44 +0100 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <4ADEE473.9020508@gmail.com> References: <20091020131547.GA615@mail.bytereef.org> <79990c6b0910201400k24e5ad4aj3813dc69d5049c9a@mail.gmail.com> <693bc9ab0910201549o101673fbpda9e7ff5131bfd2@mail.gmail.com> <4ADEE473.9020508@gmail.com> Message-ID: <5c6f2a5d0910210759i7b2bc12awbc694c9660f68a15@mail.gmail.com> On Wed, Oct 21, 2009 at 11:37 AM, Nick Coghlan wrote: > As far as decimal.py in particular goes, there are significant > maintenance gains in keeping a lot of the non-performance critical > context management code in pure Python. So we're likely to wait and see > how much speed Mark can wring out of a simple C decimal coefficient > object No need to wait for this. :-) I don't really expect to get much speed gain at all in normal, low-precision (= precisions less than 100 digits, say) use. Even doubling the speed would be way too much to hope for here. The only real gain from a decimal integer coefficient would be fixing the asymptotics for high-precision calculations. Mark From solipsis at pitrou.net Wed Oct 21 17:05:16 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 15:05:16 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Interest_in_integrating_C_decimal_module_i?= =?utf-8?q?nto=09Python=3F?= References: <20091020131547.GA615@mail.bytereef.org> <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> Message-ID: Mark Dickinson gmail.com> writes: > > There are certainly some open issues (e.g., what to do with > the existing Python module; what should other Python implementations > do). The existing module could be kept as a fallback. Also, the test suite should be careful to test both implementations (like what is currently done for the io module). > BTW, does anyone know the current SLOC count for py3k? Here you are, generated using David A. Wheeler's 'SLOCCount': SLOC Directory SLOC-by-Language (Sorted) 261496 Lib python=261451,sh=45 186769 Modules ansic=173279,asm=9561,sh=3929 53258 Objects ansic=53258 40257 Python ansic=40229,python=28 27220 Tools python=27117,ansic=67,sh=36 18892 Demo python=18511,ansic=377,sh=4 9168 PC ansic=8465,python=703 5840 Include ansic=5840 5799 Parser ansic=3914,python=1885 3485 Misc lisp=2948,python=242,sh=185,ansic=110 3101 Doc python=2306,ansic=795 3030 Mac python=2138,objc=775,sh=109,ansic=8 1666 top_dir python=1140,ansic=286,sh=240 349 PCbuild python=279,ansic=70 337 build ansic=295,python=42 0 Grammar (none) Totals grouped by language (dominant language first): python: 315842 (50.89%) ansic: 286993 (46.24%) asm: 9561 (1.54%) sh: 4548 (0.73%) lisp: 2948 (0.47%) objc: 775 (0.12%) Total Physical Source Lines of Code (SLOC) = 620,667 From john at arbash-meinel.com Wed Oct 21 17:15:56 2009 From: john at arbash-meinel.com (John Arbash Meinel) Date: Wed, 21 Oct 2009 10:15:56 -0500 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <1256129560.5069.30.camel@localhost> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <4ADEFE80.3050002@scottdial.com> <1256129560.5069.30.camel@localhost> Message-ID: <4ADF25AC.802@arbash-meinel.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antoine Pitrou wrote: >> I don't really know how this test works, so I won't claim to understand >> the results either. However, here you go: > > Thanks. > > Interesting results. I wonder what they would be like on a multi-core > machine. The GIL seems to behave perfectly on your setup (no performance > degradation due to concurrency, and zero latencies). > C:\downloads>C:\Python26\python.exe ccbench.py - --- Throughput --- Pi calculation (Python) threads=1: 691 iterations/s. threads=2: 400 ( 57 %) threads=3: 453 ( 65 %) threads=4: 467 ( 67 %) ^- seems to have some contention regular expression (C) threads=1: 592 iterations/s. threads=2: 598 ( 101 %) threads=3: 587 ( 99 %) threads=4: 586 ( 99 %) bz2 compression (C) threads=1: 536 iterations/s. threads=2: 1056 ( 196 %) threads=3: 1040 ( 193 %) threads=4: 1060 ( 197 %) ^- seems to properly show that I have 2 cores here. - --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 38 ms. (std dev: 18 ms.) CPU threads=2: 173 ms. (std dev: 77 ms.) CPU threads=3: 518 ms. (std dev: 264 ms.) CPU threads=4: 661 ms. (std dev: 343 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) John =;-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrfJawACgkQJdeBCYSNAANQlgCgwx0TCLh7YhLSJxkfOuMi1/YF XhkAoIONtdP0rR1YW0nDza+wpKpAlInd =L4WZ -----END PGP SIGNATURE----- From john at arbash-meinel.com Wed Oct 21 17:27:12 2009 From: john at arbash-meinel.com (John Arbash Meinel) Date: Wed, 21 Oct 2009 10:27:12 -0500 Subject: [Python-Dev] [Python-ideas] Remove GIL with CAS instructions? In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> Message-ID: <4ADF2850.1040304@arbash-meinel.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Kristj?n Valur J?nsson wrote: ... > This depends entirely on the platform and primitives used to implement the GIL. > I'm interested in windows. There, I found this article: > http://fonp.blogspot.com/2007/10/fairness-in-win32-lock-objects.html > So, you may be on to something. Perhaps a simple C test is in order then? > > I did that. I found, on my dual-core vista machine, that running "release", that both Mutexes and CriticalSections behaved as you describe, with no "fairness". Using a "semaphore" seems to retain fairness, however. > "fairness" was retained in debug builds too, strangely enough. > > Now, Python uses none of these. On windows, it uses an "Event" object coupled with an atomically updated counter. This also behaves fairly. > > The test application is attached. > > > I think that you ought to sustantiate your claims better, maybe with a specific platform and using some test like the above. > > On the other hand, it shows that we must be careful what we use. There has been some talk of using CriticalSections for the GIL on windows. This test ought to show the danger of that. The GIL is different than a regular lock. It is a reverse-lock, really, and therefore may need to be implemented in its own special way, if we want very fast mutexes for the rest of the system (cc to python-dev) > > Cheers, > > Kristj?n I can compile and run this, but I'm not sure I know how to interpret the results. If I understand it correctly, then everything but "Critical Sections" are fair on my Windows Vista machine. To run, I changed the line "#define EVENT" to EVENT, MUTEX, SEMAPHORE and CRIT. I then built and ran in "Release" environment (using VS 2008 Express) For all but CRIT, I saw things like: thread 5532 reclaims GIL thread 5532 working 51234 units thread 5532 worked 51234 units: 1312435761 thread 5532 flashing GIL thread 5876 reclaims GIL thread 5876 working 51234 units thread 5876 worked 51234 units: 1312435761 thread 5876 flashing GIL Where there would be 4 lines for one thread, then 4 lines for the other thread. for CRIT, I saw something more like 50 lines for one thread, and then 50 lines for the other thread. This is Vista Home Basic, and VS 2008 Express Edition, with a 2-core machine. John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrfKFAACgkQJdeBCYSNAANbuQCgudU0IChylofTwvUk/JglChBd 9gsAoIJHj63/CagKpduUsd68HV8pP3QX =CuUj -----END PGP SIGNATURE----- From dickinsm at gmail.com Wed Oct 21 17:28:50 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 21 Oct 2009 16:28:50 +0100 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: References: <20091020131547.GA615@mail.bytereef.org> <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> Message-ID: <5c6f2a5d0910210828g5dab0aa0qac54815741734e8f@mail.gmail.com> On Wed, Oct 21, 2009 at 4:05 PM, Antoine Pitrou wrote: > Mark Dickinson gmail.com> writes: >> BTW, does anyone know the current SLOC count for py3k? > > Here you are, generated using David A. Wheeler's 'SLOCCount': > [...] Thanks, Antoine! With SLOCCount I can revise my earlier numbers, as well: Here's Stefan Krah's mpdecimal, version 0.80: SLOC Directory SLOC-by-Language (Sorted) 21445 top_dir ansic=21267,sh=105,python=55,asm=18 6238 python python=6177,java=43,sh=18 1403 tests ansic=1356,sh=47 476 literature lisp=476 274 cmd ansic=274 11 tools sh=11 0 doc (none) Totals grouped by language (dominant language first): ansic: 22897 (76.71%) python: 6232 (20.88%) lisp: 476 (1.59%) sh: 181 (0.61%) java: 43 (0.14%) asm: 18 (0.06%) Lib/decimal.py: SLOC Directory SLOC-by-Language (Sorted) 2636 tmp python=2636 Totals grouped by language (dominant language first): python: 2636 (100.00%) So it looks like 2636 lines of Python versus 21000-ish lines of C. Mark From sturla at molden.no Wed Oct 21 17:10:54 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 21 Oct 2009 17:10:54 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <1256132087.5069.39.camel@localhost> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> Message-ID: <4ADF247E.8040401@molden.no> Antoine Pitrou skrev: >>> (*) http://svn.python.org/view/sandbox/trunk/ccbench/ >>> >> I?ve run it twice on my dual core machine. It hangs every time, but not in the same place: >> D:\pydev\python\trunk\PCbuild>python.exe \tmp\ccbench.py >> > > Ah, you should report a bug then. ccbench is pure Python (and not > particularly evil Python), it shouldn't be able to crash the > interpreter. > > It does not crash the interpreter, but it seems it can deadlock. Here is what I get con a quadcore (Vista, Python 2.6.3). D:\>ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 568 iterations/s. threads=2: 253 ( 44 %) threads=3: 274 ( 48 %) threads=4: 283 ( 49 %) regular expression (C) threads=1: 510 iterations/s. threads=2: 508 ( 99 %) threads=3: 503 ( 98 %) threads=4: 502 ( 98 %) bz2 compression (C) threads=1: 456 iterations/s. threads=2: 892 ( 195 %) threads=3: 1320 ( 289 %) threads=4: 1743 ( 382 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 37 ms. (std dev: 21 ms.) CPU threads=2: 379 ms. (std dev: 175 ms.) CPU threads=3: 625 ms. (std dev: 310 ms.) CPU threads=4: 718 ms. (std dev: 381 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 1 ms. (std dev: 3 ms.) D:\>ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 554 iterations/s. threads=2: 400 ( 72 %) threads=3: 273 ( 49 %) threads=4: 231 ( 41 %) regular expression (C) threads=1: 508 iterations/s. threads=2: 509 ( 100 %) threads=3: 509 ( 100 %) threads=4: 509 ( 100 %) bz2 compression (C) threads=1: 456 iterations/s. threads=2: 897 ( 196 %) threads=3: 1316 ( 288 %) DEADLOCK D:\>ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 559 iterations/s. threads=2: 397 ( 71 %) threads=3: 274 ( 49 %) threads=4: 238 ( 42 %) regular expression (C) threads=1: 507 iterations/s. threads=2: 499 ( 98 %) threads=3: 505 ( 99 %) threads=4: 495 ( 97 %) bz2 compression (C) threads=1: 455 iterations/s. threads=2: 896 ( 196 %) threads=3: 1320 ( 290 %) threads=4: 1736 ( 381 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 34 ms. (std dev: 21 ms.) CPU threads=2: 358 ms. (std dev: 174 ms.) CPU threads=3: 619 ms. (std dev: 312 ms.) CPU threads=4: 742 ms. (std dev: 382 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 6 ms. (std dev: 13 ms.) From sturla at molden.no Wed Oct 21 17:38:13 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 21 Oct 2009 17:38:13 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <4ADF247E.8040401@molden.no> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> Message-ID: <4ADF2AE5.5040100@molden.no> Sturla Molden skrev: > does not crash the interpreter, but it seems it can deadlock. > > Here is what I get con a quadcore (Vista, Python 2.6.3). > This what I get with affinity set to CPU 3. There are deadlocks happening at random locations in ccbench.py. It gets worse with affinity set to one processor. Sturla D:\>start /AFFINITY 3 /B ccbench.py D:\>--- Throughput --- Pi calculation (Python) threads=1: 554 iterations/s. threads=2: 257 ( 46 %) threads=3: 272 ( 49 %) threads=4: 280 ( 50 %) regular expression (C) threads=1: 501 iterations/s. threads=2: 505 ( 100 %) threads=3: 493 ( 98 %) threads=4: 507 ( 101 %) bz2 compression (C) threads=1: 455 iterations/s. threads=2: 889 ( 195 %) threads=3: 1309 ( 287 %) threads=4: 1710 ( 375 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 40 ms. (std dev: 22 ms.) CPU threads=2: 384 ms. (std dev: 179 ms.) CPU threads=3: 618 ms. (std dev: 314 ms.) CPU threads=4: 713 ms. (std dev: 379 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 3 ms.) CPU threads=4: 0 ms. (std dev: 1 ms.) From solipsis at pitrou.net Wed Oct 21 19:21:08 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 17:21:08 +0000 (UTC) Subject: [Python-Dev] GIL behaviour under Windows References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> Message-ID: Sturla Molden molden.no> writes: > > It does not crash the interpreter, but it seems it can deadlock. Kristj?n sent me a patch which I applied and is supposed to fix this. Anyway, thanks for the numbers. The GIL does seem to fare a bit better (zero latency with the Pi calculation in the background) than under Linux, although it may be caused by the limited resolution of time.time() under Windows. Regards Antoine. From g.brandl at gmx.net Wed Oct 21 19:21:07 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 21 Oct 2009 19:21:07 +0200 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> References: <20091020131547.GA615@mail.bytereef.org> <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> Message-ID: Mark Dickinson schrieb: > * There's a partially complete rewrite of decimal in C in the sandbox, > dating from the Need for Speed sprint in 2006: > > http://svn.python.org/view/sandbox/trunk/decimal-c/ > > Last time I looked at this it wasn't up to date with the decimal > specification: I'm not sure that functions like exp, log and log10 > are currently working. Georg Brandl might know better than I do. I haven't touched that code since the sprint, but a student named Mateusz Rukowicz worked on it for a past Summer of Code, I think. I never heard of the outcome of that particular project. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From john.arbash.meinel at gmail.com Wed Oct 21 19:42:51 2009 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 21 Oct 2009 12:42:51 -0500 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> Message-ID: <4ADF481B.90207@gmail.com> Antoine Pitrou wrote: > Sturla Molden molden.no> writes: >> It does not crash the interpreter, but it seems it can deadlock. > > Kristj?n sent me a patch which I applied and is supposed to fix this. > Anyway, thanks for the numbers. The GIL does seem to fare a bit better (zero > latency with the Pi calculation in the background) than under Linux, although it > may be caused by the limited resolution of time.time() under Windows. > > Regards > > Antoine. You can use time.clock() instead to get <15ms resolution. Changing all instances of 'time.time' to 'time.clock' gives me this result: (2-core machine, python 2.6.2) $ py ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 675 iterations/s. threads=2: 388 ( 57 %) threads=3: 374 ( 55 %) threads=4: 445 ( 65 %) regular expression (C) threads=1: 588 iterations/s. threads=2: 519 ( 88 %) threads=3: 511 ( 86 %) threads=4: 513 ( 87 %) bz2 compression (C) threads=1: 536 iterations/s. threads=2: 949 ( 176 %) threads=3: 900 ( 167 %) threads=4: 927 ( 172 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 24727 ms. (std dev: 0 ms.) CPU threads=1: 27930 ms. (std dev: 0 ms.) CPU threads=2: 31029 ms. (std dev: 0 ms.) CPU threads=3: 34170 ms. (std dev: 0 ms.) CPU threads=4: 37292 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 40454 ms. (std dev: 0 ms.) CPU threads=1: 43674 ms. (std dev: 21 ms.) CPU threads=2: 47100 ms. (std dev: 165 ms.) CPU threads=3: 50441 ms. (std dev: 304 ms.) CPU threads=4: 53707 ms. (std dev: 377 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 56138 ms. (std dev: 0 ms.) CPU threads=1: 59332 ms. (std dev: 0 ms.) CPU threads=2: 62436 ms. (std dev: 0 ms.) CPU threads=3: 66130 ms. (std dev: 0 ms.) CPU threads=4: 69859 ms. (std dev: 0 ms.) From solipsis at pitrou.net Wed Oct 21 19:55:26 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 19:55:26 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <4ADF481B.90207@gmail.com> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> Message-ID: <1256147726.5069.50.camel@localhost> Le mercredi 21 octobre 2009 ? 12:42 -0500, John Arbash Meinel a ?crit : > > You can use time.clock() instead to get <15ms resolution. Changing all > instances of 'time.time' to 'time.clock' gives me this result: [snip] > > --- Latency --- > > Background CPU task: Pi calculation (Python) > > CPU threads=0: 24727 ms. (std dev: 0 ms.) > CPU threads=1: 27930 ms. (std dev: 0 ms.) > CPU threads=2: 31029 ms. (std dev: 0 ms.) > CPU threads=3: 34170 ms. (std dev: 0 ms.) > CPU threads=4: 37292 ms. (std dev: 0 ms.) Well apparently time.clock() has a per-process time reference, which makes it unusable for this benchmark :-( (the numbers above are obviously incorrect) Regards Antoine. From john at arbash-meinel.com Wed Oct 21 20:00:54 2009 From: john at arbash-meinel.com (John Arbash Meinel) Date: Wed, 21 Oct 2009 13:00:54 -0500 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <1256147726.5069.50.camel@localhost> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> Message-ID: <4ADF4C56.3080607@arbash-meinel.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antoine Pitrou wrote: > Le mercredi 21 octobre 2009 ? 12:42 -0500, John Arbash Meinel a ?crit : >> You can use time.clock() instead to get <15ms resolution. Changing all >> instances of 'time.time' to 'time.clock' gives me this result: > [snip] >> --- Latency --- >> >> Background CPU task: Pi calculation (Python) >> >> CPU threads=0: 24727 ms. (std dev: 0 ms.) >> CPU threads=1: 27930 ms. (std dev: 0 ms.) >> CPU threads=2: 31029 ms. (std dev: 0 ms.) >> CPU threads=3: 34170 ms. (std dev: 0 ms.) >> CPU threads=4: 37292 ms. (std dev: 0 ms.) > > Well apparently time.clock() has a per-process time reference, which > makes it unusable for this benchmark :-( > (the numbers above are obviously incorrect) > > Regards > > Antoine. I believe that 'time.count()' is measured as seconds since the start of the process. So yeah, I think spawning a background process will reset this counter back to 0. John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrfTFYACgkQJdeBCYSNAAObWQCfRJsRENbcp6kuo2x1k+HvhYGZ ftsAn2PNnNHNj6D4esNBMhlSdH4IjeMA =1KWG -----END PGP SIGNATURE----- From guido at python.org Wed Oct 21 20:31:48 2009 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Oct 2009 11:31:48 -0700 Subject: [Python-Dev] Proposal: Moratorium on Python language changes Message-ID: In the python-ideas list I've proposed a moratorium on language changes. It seems to be gaining momentum; if you want to have a say, come over. You can watch the discussion in the archives starting here: http://mail.python.org/pipermail/python-ideas/2009-October/006305.html . (Eventually I'll move the proposal over to python-dev, but I thought it only fair to give early warning to those who don't read python-ideas.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kristjan at ccpgames.com Wed Oct 21 22:33:11 2009 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Wed, 21 Oct 2009 20:33:11 +0000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <1256147726.5069.50.camel@localhost> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> You are right, on windows time.clock() is based relative to its first call in the process. There is no such promise made on unix. QueryPerformanceCounter() (what time.clock uses()) is a robust high resolution timer that is processor/core independent. It should be possible to use it across different processes too, if this annoying rebasing wasn't there. I wonder if we should consider this a bug? If so, I see three remedies: 1) simply using the absolute value and stop creating this arbitrary zero point. this should be ok since the same is done on unix, but it would be a break from the documented behavior. Never the less, the absolute value of this timer is irrelevant, it is the deltas that matter. 2) Add a flag to time.clock() for it to return absolute value 3) Create yet another api, either something like time.rclock() returning the absolute value or something like time.clockbase() returning the base of the zeroed clock timer. If you just want to patch locally for your timing pleasure, change line 184 of timemodule.c to: diff = (double)(now.QuadPart); K > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Antoine Pitrou > --- > > > > Background CPU task: Pi calculation (Python) > > > > CPU threads=0: 24727 ms. (std dev: 0 ms.) > > CPU threads=1: 27930 ms. (std dev: 0 ms.) > > CPU threads=2: 31029 ms. (std dev: 0 ms.) > > CPU threads=3: 34170 ms. (std dev: 0 ms.) > > CPU threads=4: 37292 ms. (std dev: 0 ms.) > > Well apparently time.clock() has a per-process time reference, which > makes it unusable for this benchmark :-( > (the numbers above are obviously incorrect) > From solipsis at pitrou.net Wed Oct 21 22:51:03 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2009 20:51:03 +0000 (UTC) Subject: [Python-Dev] time.clock() on windows References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> Message-ID: Kristj?n Valur J?nsson ccpgames.com> writes: > > You are right, on windows time.clock() is based relative to its first call in the process. There is no such > promise made on unix. > QueryPerformanceCounter() (what time.clock uses()) is a robust high resolution timer that is > processor/core independent. It should be possible to use it across different processes too, if this > annoying rebasing wasn't there. Well, could we simply have a high-resolution time.time()? Or is Windows just too limited to provide this? Regards Antoine. From curt at hagenlocher.org Wed Oct 21 23:31:13 2009 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Wed, 21 Oct 2009 14:31:13 -0700 Subject: [Python-Dev] time.clock() on windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> Message-ID: On Wed, Oct 21, 2009 at 1:51 PM, Antoine Pitrou wrote: > > Kristj?n Valur J?nsson ccpgames.com> writes: >> >> You are right, on windows time.clock() is based relative to its first call >> in the process. ?There is no such promise made on unix. >> QueryPerformanceCounter() (what time.clock uses()) is a robust high >> resolution timer that is processor/core independent. ?It should be >> possible to use it across different processes too, if this >> annoying rebasing wasn't there. > > Well, could we simply have a high-resolution time.time()? > Or is Windows just too limited to provide this? Presumably you could fake something like this by combining output from an initial time(), an initial QueryPerformanceCounter() and the current QueryPerformanceCounter(). But it makes more sense to understand why someone chose to implement time.clock() on Windows the way they did -- this seems very broken to me, and I think it should be changed. Of course, there are no doubt people relying on the broken behavior... -- Curt Hagenlocher curt at hagenlocher.org From scott+python-dev at scottdial.com Wed Oct 21 23:52:26 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 21 Oct 2009 17:52:26 -0400 Subject: [Python-Dev] time.clock() on windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> Message-ID: <4ADF829A.7060509@scottdial.com> Curt Hagenlocher wrote: > But it makes more sense to > understand why someone chose to implement time.clock() on Windows the > way they did -- this seems very broken to me, and I think it should be > changed. Some SVN detective work takes this to all the way back to r7713 (1997-04-02). The original implementation checked by Guido and attributed to Mark Hammond. So, we should ask Mark why he did that. Can anyone honestly use it, as it is, without already having normalize it across platforms themselves? I don't know how much of an impact it is, but the current implementation of clock() does not require argument parsing, so the proposal to add a "absolute" boolean-flag argument is perhaps bad. This is generally a function used for performance timing and that proposal adds some amount of latency to the query. The proposal to add a clockbase() function is perhaps better because of this, you need only call it once, and you can cache the result for the life of your process. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From zookog at gmail.com Thu Oct 22 01:00:11 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 21 Oct 2009 17:00:11 -0600 Subject: [Python-Dev] Python 2.6.4rc2 In-Reply-To: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> Message-ID: Barry: Do you know anything about this alleged regression in 2.6.3 with regard to the __doc__ property? https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 Regards, Zooko From kristjan at ccpgames.com Thu Oct 22 01:05:58 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 21 Oct 2009 23:05:58 +0000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF4213E@exchis.ccp.ad.local> > Presumably you could fake something like this by combining output from > an initial time(), an initial QueryPerformanceCounter() and the > current QueryPerformanceCounter(). But it makes more sense to > understand why someone chose to implement time.clock() on Windows the > way they did -- this seems very broken to me, and I think it should be > changed. Yes. The problem with QPC is that although it has very high resolution, it is not precise in the long term. And GetSystemTimeAsFileTime() is high precision in the long term but only updated evey 20ms or so. In EVE Online we use a combination of the two for high resolution, long term precision. But I'm not happy with the way I'm doing it. It needs some sort of smoothing of course. I've even played with using Kalman filtering to do it... The idea is to use the low frequency timer to apply correction coefficients to the high frequency timer, yet keep the flow of time smooth (no backwards jumps because of corrections.). An optimal solution has so far eluded me. Cheers, K From barry at python.org Thu Oct 22 01:33:12 2009 From: barry at python.org (Barry Warsaw) Date: Wed, 21 Oct 2009 19:33:12 -0400 Subject: [Python-Dev] Python 2.6.4rc2 In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> Message-ID: <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> On Oct 21, 2009, at 7:00 PM, Zooko O'Whielacronx wrote: > Do you know anything about this alleged regression in 2.6.3 with > regard to the __doc__ property? > > https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 This is the first I've heard of it, but my guess is that it's caused by the fix for bug 5890. http://bugs.python.org/issue5890 It's a change in Python 2.6.3, but I don't believe it's a regression (as in, currently broken). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From david.lyon at preisshare.net Thu Oct 22 02:34:24 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 21 Oct 2009 20:34:24 -0400 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> Message-ID: Hi All, I started out some time ago and wrote a Python Package Manager with wxpython. It was an interesting learning experience for me. I was new to python. Some have pointed out that using wx was not a good idea for a tool to go into all the python distributions. Because it required the external dependency of wx. So, given that, I'm ready to have another go. Some have suggested using Tk. But I noticed python Tk isn't standard on ubuntu. A console mode tool is possible. That would be better than the nothing that we have under windows today. Vote [1] - console mode tool for this Alternatively, a web server based tool along the lines of the old python docs system might be possible. Vote [2] - web tool for this I have come to understand that python-dev is full of people that know the packages that they like, know how to install them by hand, and probably only ever go looking for upgrades. However, for new users, it is so much harder. It also seems to me that we have a lot of reasonable and talented people, who are committed and dedicated. To me, a relative outsider, it's not clear if this is a python central issue, or a distutils issue. It could be either. Advise me please. By June next year, some people including myself, want to go to europe for holidays - sorry conferences.. and we want to take some work with us to talk about. We can do the work.. But we need concessions... What can we work on that might make a difference ? If the current roadmap for distutils package management on windows for the next 8 months is nothing, I can live with that. But seriously, lets get this discussion going again. If a command line tool is all that windows users will allowed to have, then it would be better than the current plan which is for them to have nothing. It's not just nothing for now, it's nothing for years. That doesn't seem fair or nice. It's not just one person saying that, it is a handful who don't use windows on a regular basis. Why can't we have an advocate from each major platform? bring their issues, and try to come to a consensus. Even if that tool would simply allow them to choose: - PIP - Distribute - Easy Install - Python Package Manager >From there, users could explore each offer on it's own merits. I'm interested in getting all types of opinions and feedback. David From andrew.svetlov at gmail.com Thu Oct 22 02:44:04 2009 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Wed, 21 Oct 2009 20:44:04 -0400 Subject: [Python-Dev] PEP 384 Message-ID: <270ea8310910211744i250047cewc2d7b26a8be2c2f@mail.gmail.com> Is there some public branch for this PEP? I like the idea and glad to look on implementation. Thanks. From brett at python.org Thu Oct 22 02:48:42 2009 From: brett at python.org (Brett Cannon) Date: Wed, 21 Oct 2009 17:48:42 -0700 Subject: [Python-Dev] PEP 384 In-Reply-To: <270ea8310910211744i250047cewc2d7b26a8be2c2f@mail.gmail.com> References: <270ea8310910211744i250047cewc2d7b26a8be2c2f@mail.gmail.com> Message-ID: On Wed, Oct 21, 2009 at 17:44, Andrew Svetlov wrote: > Is there some public branch for this PEP? > I like the idea and glad to look on implementation. > Thanks. > If there was such a thing the PEP would mention it. But since the PEP is about not changing ABIs between releases I don't see why some branch would be needed for it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Oct 22 02:56:57 2009 From: brett at python.org (Brett Cannon) Date: Wed, 21 Oct 2009 17:56:57 -0700 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> Message-ID: On Wed, Oct 21, 2009 at 17:34, David Lyon wrote: > > Hi All, > > I started out some time ago and wrote a Python Package Manager > with wxpython. It was an interesting learning experience for > me. I was new to python. > > Some have pointed out that using wx was not a good idea for > a tool to go into all the python distributions. Because it > required the external dependency of wx. > > So, given that, I'm ready to have another go. > > I honestly don't see what this has to do with python-dev or the distutils-sig. If you want to write a package management tool that's great, but I don't see why python-dev should have input on that sort of thing. Asking what kind of GUI to use is a comp.lang.python question and not a python-dev question as I don't see anything like this going into the stdlib. If you want distutils to expose something to make it easier to write your tool then that belong on the distutils-sig. But otherwise this seems off-topic for python-dev. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lyon at preisshare.net Thu Oct 22 03:17:59 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 21 Oct 2009 21:17:59 -0400 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> Message-ID: <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> On Wed, 21 Oct 2009 17:56:57 -0700, Brett Cannon wrote: > but I don't see why python-dev should have input on that sort of thing. python-dev is the only place where we could get a change to the installation binary release. We'd need a change and the addition of a program shortcut. > If you want distutils to expose something to make it easier to write your > tool then that belong on the distutils-sig. But otherwise this seems > off-topic for python-dev. I didn't ask for that - because I know that getting that assistance on the distutils side is certainly possible. Distutils is simply just one of the many libraries within python. It doesn't have an external interface. The roadmap for distutils for windows doesn't include getting a shortcut or utility for windows so that's why I'm asking about it here. Surely logic says that if it's 'python' and 'development' and it's not in distutils then some discussion of it should be allowed here. What I am really talking about is the menu shortcuts in the cpython distribution for windows. And how they can be improved to help windows users. This is the only place that I can think to discuss that. Best Regards David From brett at python.org Thu Oct 22 03:38:26 2009 From: brett at python.org (Brett Cannon) Date: Wed, 21 Oct 2009 18:38:26 -0700 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> Message-ID: On Wed, Oct 21, 2009 at 18:17, David Lyon wrote: > On Wed, 21 Oct 2009 17:56:57 -0700, Brett Cannon wrote: > > > but I don't see why python-dev should have input on that sort of thing. > > python-dev is the only place where we could get a change to the > installation > binary release. We'd need a change and the addition of a program > shortcut. > > But that assumes you can get your tool into the stdlib. It would have been better to phrase the question as "is there interest in having a package manager tool included with Python" rather than ask us out of the blue what GUI you should use. > > If you want distutils to expose something to make it easier to write your > > tool then that belong on the distutils-sig. But otherwise this seems > > off-topic for python-dev. > > I didn't ask for that - because I know that getting that assistance on the > distutils side is certainly possible. > > Distutils is simply just one of the many libraries within python. It > doesn't > have an external interface. > > And that's on purpose. > The roadmap for distutils for windows doesn't include getting a shortcut > or utility for windows so that's why I'm asking about it here. > > Surely logic says that if it's 'python' and 'development' and it's > not in distutils then some discussion of it should be allowed here. > > No it surely does not. We do not need to discuss design decisions of pip, setuptools, virtualenv, buildout, or various other tools that involve the terms "python" and "development" and are not in distutils. > What I am really talking about is the menu shortcuts in the cpython > distribution for windows. And how they can be improved to help > windows users. This is the only place that I can think to discuss > that. > David, you are making a huge leap here thinking that we even want a package manager in the stdlib. You did not ask about menu shortcuts but whether a package manager should be written using Tk or a web front-end. Then you start discussing about wanting to add some UI to package management by default on Windows or add some tool that sounds like what the EU is going to have MS stick in front of users to get new user by browsers. This extends beyond adding some shortcut the Windows installer adds to someone's machine. I realize you are trying to help, David, but you are going in the wrong direction here and pushing rather hard. At the language summit we discussed opening up some APIs in distutils about making it easier for people to write package management tools, but we don't have a burning desire to get into the tool business. We make a language here. Distutils exists as a bootstrap mechanism for the package story and for our own building needs of the stdlib. But I doubt I am the only core developer who has no interest to be in charge of a package management tool when there already exists several good ones out there that people seem to find on their own without issue. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbk.lists at gmail.com Thu Oct 22 03:56:51 2009 From: mbk.lists at gmail.com (Mike Krell) Date: Wed, 21 Oct 2009 18:56:51 -0700 Subject: [Python-Dev] nonlocal keyword in 2.x? Message-ID: Is there any possibility of backporting support for the nonlocal keyword into a 2.x release? I see it's not in 2.6, but I don't know if that was an intentional design choice or due to a lack of demand / round tuits. I'm also not sure if this would fall under the scope of the proposed moratorium on new language features (although my first impression was that it could be allowed since it already exists in python 3. One of my motivations for asking is a recent blog post by Fernando Perez of IPython fame that describes an interesting decorator-based idiom inspired by Apple's Grand Central Dispatch which would allow many interesting possibilities for expressing parallelization and other manipulations of execution context for blocks of python code. Unfortunately, using the technique to its fullest extent requires the nonlocal keyword. The blog post is here: https://cirl.berkeley.edu/fperez/py4science/decorators.html Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lyon at preisshare.net Thu Oct 22 04:13:09 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 21 Oct 2009 22:13:09 -0400 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> Message-ID: <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> On Wed, 21 Oct 2009 18:38:26 -0700, Brett Cannon wrote: > But that assumes you can get your tool into the stdlib. No I'm not assuming that I can. I am actually assuming that I cannot.. So lets move forward.. > It would have been > better to phrase the question as "is there interest in having a package > manager tool included with Python" rather than ask us out of the blue what > GUI you should use. ok - but I think I know the answer to that.. you answer it next. > David, you are making a huge leap here thinking that we even want a package > manager in the stdlib. Well - who is 'we'? If it's python-dev people I can accept and respect that. If it's regular developers out there in developer land, I'm not so sure about your assertion. I'd even politely have to disagree from my experience. > You did not ask about menu shortcuts but whether a > package manager should be written using Tk or a web front-end. I was thinking about the issue on the fly... Menu shortcuts that link off to a a standard community web page would be an excellent compromise - in the case where some tool could not be added. That would be a tremendous improvement for windows users over what they are given at the moment. > Then you > start discussing about wanting to add some UI to package management by > default on Windows or add some tool that sounds like what the EU is going > to > have MS stick in front of users to get new user by browsers. This extends > beyond adding some shortcut the Windows installer adds to someone's > machine. That's going further than what I'm saying.. > I realize you are trying to help, David, but you are going in the wrong > direction here and pushing rather hard. On the counter side, others are pushing rather hard for 0 improvement for the windows platform for the user experience. While everything else on windows rushes ahead.. My direction is improving the developer experience for windows users. I can't do compiler writing. I'm not clever enough. > At the language summit we discussed > opening up some APIs in distutils about making it easier for people to > write > package management tools, but we don't have a burning desire to get into > the > tool business. ok - but nothing happened there... I'm not in the tools business either. I'm not doing it for money but that shouldn't be the point. > We make a language here. Distutils exists as a bootstrap > mechanism for the package story and for our own building needs of the > stdlib. I accept that it's a tool for building stdlib. No debate. > But I doubt I am the only core developer who has no interest to be > in charge of a package management tool when there already exists several > good ones out there that people seem to find on their own without issue. umm.. I disagree with the 'without issue' statement. I'm sure if I tralled the mailing lists I could find more than one.. Enough from me for now. Thanks Brett. David From sturla at molden.no Thu Oct 22 04:56:40 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Oct 2009 04:56:40 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> Message-ID: <4ADFC9E8.1010400@molden.no> Antoine Pitrou skrev: > Kristj?n sent me a patch which I applied and is supposed to fix this. > Anyway, thanks for the numbers. The GIL does seem to fare a bit better (zero > latency with the Pi calculation in the background) than under Linux, although it > may be caused by the limited resolution of time.time() under Windows. > > My critisism of the GIL on python-ideas was partly motivated by this: http://blip.tv/file/2232410 However, David Beazley is not talking about Windows. Since the GIL is apparently not a mutex on Windows, it could behave differently. So I wrote a small script that contructs a GIL battle, and record how often a check-interval results in a thread-switch or not. For monitoring check intervals, I used a small C extension to read _Py_Ticker from ceval.c. It is not declared static so I could easily hack into it. With two threads and a check interval og 100, only 61 of 100000 check intervals failed to produce a thread-switch in the interpreter. I'd call that rather fair. :-) And in case someone asks, the nthreads=1 case is just for verification. S.M. D:\>test.py check interval = 1 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=57809, missed=42191 nthreads=3, swiched=91535, missed=8465 nthreads=4, swiched=99751, missed=249 nthreads=5, swiched=95839, missed=4161 nthreads=6, swiched=100000, missed=0 D:\>test.py check interval = 10 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=99858, missed=142 nthreads=3, swiched=99992, missed=8 nthreads=4, swiched=100000, missed=0 nthreads=5, swiched=100000, missed=0 nthreads=6, swiched=100000, missed=0 D:\>test.py check interval = 100 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=99939, missed=61 nthreads=3, swiched=100000, missed=0 nthreads=4, swiched=100000, missed=0 nthreads=5, swiched=100000, missed=0 nthreads=6, swiched=100000, missed=0 D:\>test.py check interval = 1000 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=99999, missed=1 nthreads=3, swiched=100000, missed=0 nthreads=4, swiched=100000, missed=0 nthreads=5, swiched=100000, missed=0 nthreads=6, swiched=100000, missed=0 From sturla at molden.no Thu Oct 22 05:09:44 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Oct 2009 05:09:44 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: <4ADFC9E8.1010400@molden.no> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADFC9E8.1010400@molden.no> Message-ID: <4ADFCCF8.5010807@molden.no> Sturla Molden skrev: > > However, David Beazley is not talking about Windows. Since the GIL is > apparently not a mutex on Windows, it could behave differently. So I > wrote a small script that contructs a GIL battle, and record how often > a check-interval results in a thread-switch or not. For monitoring > check intervals, I used a small C extension to read _Py_Ticker from > ceval.c. It is not declared static so I could easily hack into it. Anyway, if anyone wants to run a GIL battle, here is the code I used. If it turns out the GIL is far worse with pthreads, as it is implemented with a mutex, it might be a good idea to reimplement it with an event object as it is on Windows. Sturla Molden ---------------- In python: from giltest import * from time import clock import threading import sys def thread(rank, battle, start): while not start.isSet(): if rank == 0: start.set() try: while 1: battle.record(rank) except: pass if __name__ == '__main__': sys.setcheckinterval(1000) print "check interval = %d" % sys.getcheckinterval() for nthreads in range(1,7): start = threading.Event() battle = GIL_Battle(100000) threads = [threading.Thread(target=thread, args=(i,battle,start)) for i in range(1,nthreads)] for t in threads: t.setDaemon(True) t.start() thread(0, battle, start) for t in threads: t.join() s,m = battle.report() print "nthreads=%d, swiched=%d, missed=%d" % (nthreads, s, m) In Cython or Pyrex: from exceptions import Exception cdef extern from *: ctypedef int vint "volatile int" vint _Py_Ticker class StopBattle(Exception): pass cdef class GIL_Battle: """ tests the fairness of the GIL """ cdef vint prev_tick, prev_rank, switched, missed cdef int trials def __cinit__(GIL_Battle self, int trials=100000): self.prev_tick = _Py_Ticker self.prev_rank = -1 self.missed = 0 self.switched = 0 self.trials = trials def record(GIL_Battle self, int rank): if self.trials == self.switched + self.missed: raise StopBattle if self.prev_rank == -1: self.prev_tick = _Py_Ticker self.prev_rank = rank else: if _Py_Ticker > self.prev_tick: if self.prev_rank == rank: self.missed += 1 else: self.switched += 1 self.prev_tick = _Py_Ticker self.prev_rank = rank else: self.prev_tick = _Py_Ticker def report(GIL_Battle self): return int(self.switched), int(self.missed) From skippy.hammond at gmail.com Thu Oct 22 06:21:13 2009 From: skippy.hammond at gmail.com (Mark Hammond) Date: Thu, 22 Oct 2009 15:21:13 +1100 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <4ADF829A.7060509@scottdial.com> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> Message-ID: <4ADFDDB9.3080707@gmail.com> On 22/10/2009 8:52 AM, Scott Dial wrote: > Curt Hagenlocher wrote: >> But it makes more sense to >> understand why someone chose to implement time.clock() on Windows the >> way they did -- this seems very broken to me, and I think it should be >> changed. > > Some SVN detective work takes this to all the way back to r7713 > (1997-04-02). The original implementation checked by Guido and > attributed to Mark Hammond. So, we should ask Mark why he did that. The thread seems to be at http://groups.google.com/group/comp.lang.python/browse_frm/thread/be32478a4b8e77b6/816d6228119a3474 (although I do seem to recall more discussion of the patch which I currently can't find). I'd be very surprised if any applications rely on the fact that each process starts counting at zero, so if someone can come up with a high-res counter which avoids that artifact I'd expect it could be used. Cheers, Mark From nnorwitz at gmail.com Thu Oct 22 06:46:54 2009 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 21 Oct 2009 21:46:54 -0700 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: References: Message-ID: On Wed, Oct 21, 2009 at 6:56 PM, Mike Krell wrote: > Is there any possibility of backporting support for the nonlocal keyword > into a? 2.x release?? I see it's not in 2.6, but I don't know if that was an > intentional design choice or due to a lack of demand / round tuits.? I'm > also not sure if this would fall under the scope of the proposed moratorium > on new language features (although my first impression was that it could be > allowed since it already exists in python 3. IIRC, Jeremy Hylton tried to backport it during a Pycon sprint a few years back. I think it was difficult and he dropped it. I don't know if there were any showstoppers or if Jeremy saved his work...or if my memory is even correct. :-) n From robertc at robertcollins.net Thu Oct 22 06:45:34 2009 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 22 Oct 2009 15:45:34 +1100 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <4ADFDDB9.3080707@gmail.com> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> Message-ID: <1256186734.2929.37.camel@lifeless-64> On Thu, 2009-10-22 at 15:21 +1100, Mark Hammond wrote: > I'd be very surprised if any applications rely on > the fact that each process starts counting at zero, so if someone can > come up with a high-res counter which avoids that artifact I'd expect it > could be used. Could you offset it by the system time on the first call? -Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From v+python at g.nevcal.com Thu Oct 22 07:35:49 2009 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 21 Oct 2009 22:35:49 -0700 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> Message-ID: <4ADFEF35.6000409@g.nevcal.com> On approximately 10/21/2009 7:13 PM, came the following characters from the keyboard of David Lyon: > On Wed, 21 Oct 2009 18:38:26 -0700, Brett Cannon wrote: >> We make a language here. Distutils exists as a bootstrap >> mechanism for the package story and for our own building needs of the >> stdlib. Maybe what David is missing is that since python-dev is uninterested in the package management issue, the only remaining way to include package management in a convenient, single installation, is to 1) Create the package management tool 2) Create a bundled installer that will install both Python, the package management tool, and any dependencies of the package management tool 3) Advertise the availability of the bundle, and its usefulness* 4) See how many users try it out, and as new versions of Python are created, how many users keep coming back, in preference to the package management tool free python.org distribution. *There does seem to be precedent for mentioning/linking to other distributions of Python from http://www.python.org/download/ so that would surely be an avenue that is open to David, although I have no idea what criteria must be met to obtain a listing on that page. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking From stephen at xemacs.org Thu Oct 22 08:54:39 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 22 Oct 2009 15:54:39 +0900 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> Message-ID: <87ljj3dikw.fsf@uwakimon.sk.tsukuba.ac.jp> David Lyon writes: > On Wed, 21 Oct 2009 18:38:26 -0700, Brett Cannon wrote: > > David, you are making a huge leap here thinking that we even want > > a package manager in the stdlib. > > Well - who is 'we'? If it's python-dev people I can accept and > respect that. Yes. The stdlib is a product of python-dev. > If it's regular developers out there in developer land, They are not participants in this decision. python-dev people *do* care about them. Nevertheless python-dev is manifestly uninterested in dealing with *this* issue on behalf of the "regular developers out there". (With good reasons IMHO, but you can find those reasons elsewhere.) To make progress here on python-dev, you need to get those regular developers involved in python-dev. I can think of two ways to do this that will help. (1) Turn them into committers via whatever aspect of the language they are interested in. This will take a while, and substantial contribution (ie, effort) on their part. Once that happens, they will be in a much stronger position to advocate the package manager and credibly promise to maintain it (or find a new, credible maintainer) indefinitely. (2) Put the package manager on PyPI and get a lot of people on the bandwagon. If it's a good product and recommended, the movers and shakers of python-dev will be very likely to try it and more likely to second your motion in the future. From solipsis at pitrou.net Thu Oct 22 11:09:34 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 09:09:34 +0000 (UTC) Subject: [Python-Dev] time.clock() on windows References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> <1256186734.2929.37.camel@lifeless-64> Message-ID: Robert Collins robertcollins.net> writes: > > Could you offset it by the system time on the first call? Two problems: - the two measurements are not done simultaneously, so the result *still* does not guarantee you have the same time reference in all processes (but gives you the illusion you do, which is perhaps worse) - adding a precise measure to an imprecise measure doesn't make the result precise, but imprecise (or, rather, precise but inexact); in other words, if the system time only gives a 0.01 second resolution, adding a high-resolution timer only gives you an illusion of accuracy Therefore it seems a very bad solution. The only way, AFAICT, to do this right is for Windows to provide a high-resolution system time. It sounds astonishing that this doesn't exist. Regards Antoine. From solipsis at pitrou.net Thu Oct 22 11:20:32 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 09:20:32 +0000 (UTC) Subject: [Python-Dev] GIL behaviour under Windows References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADFC9E8.1010400@molden.no> Message-ID: Sturla Molden molden.no> writes: > > With two threads and a check interval og 100, only 61 of 100000 check > intervals failed to produce a thread-switch in the interpreter. I'd call > that rather fair. This number lacks the elapsed time. 61 switches in one second is probably enough, the same amount of switches in 10 or 20 seconds is too small (at least for threads needing good responsivity, e.g. I/O threads). Also, "fair" has to take into account the average latency and its relative stability, which is why I wrote ccbench. Antoine. From kristjan at ccpgames.com Thu Oct 22 11:47:50 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 22 Oct 2009 09:47:50 +0000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <4ADFDDB9.3080707@gmail.com> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF42178@exchis.ccp.ad.local> > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Mark Hammond > The thread seems to be at > http://groups.google.com/group/comp.lang.python/browse_frm/thread/be324 > 78a4b8e77b6/816d6228119a3474 > (although I do seem to recall more discussion of the patch which I > currently can't find). I'd be very surprised if any applications rely > on > the fact that each process starts counting at zero, so if someone can > come up with a high-res counter which avoids that artifact I'd expect > it > could be used. The point in question seems to be this this (from the thread): * Need some sort of static "start value", which is set when the process starts, so I can return to Python in seconds. An easy hack is to set this the first time clock() is called, but then it wont reflect any sort of real time - but would be useful for relative times... But the argumentation is flawed. There is an implicit "start" value (technically, CPU powerup). The point concedes that no sort of real time is returned, and so the particular "start" time chosen is immaterial. K From kristjan at ccpgames.com Thu Oct 22 11:51:53 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 22 Oct 2009 09:51:53 +0000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <1256186734.2929.37.camel@lifeless-64> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> <1256186734.2929.37.camel@lifeless-64> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF4217A@exchis.ccp.ad.local> > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Robert Collins > > I'd be very surprised if any applications rely on the fact that each > > process starts counting at zero, so if someone can come up with a > > high-res counter which avoids that artifact I'd expect it could be > > used. > > Could you offset it by the system time on the first call? > Since system time has low granularity, it would negate our attemt at having time.clock() reflect the same time between processes. In my opinion, the simplest way is to simply stop setting choosing the first call as a zero base, and use whatever arbitrary time the system has chosen for us. The documentation could then read: "On Windows, this function returns wall-clock seconds elapsed since an arbitrary, system wited, epoch, as a floating point number, based on the Win32 function QueryPerformanceCounter. The resolution is typically better than one microsecond." K From phd at phd.pp.ru Thu Oct 22 12:08:09 2009 From: phd at phd.pp.ru (Oleg Broytman) Date: Thu, 22 Oct 2009 14:08:09 +0400 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> Message-ID: <20091022100809.GE13617@phd.pp.ru> On Wed, Oct 21, 2009 at 08:34:24PM -0400, David Lyon wrote: > I started out some time ago and wrote a Python Package Manager > with wxpython. David, your message appeared in the middle of another thread. Please don't use "Reply" button to start a new thread - send a new message. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From phd at phd.pp.ru Thu Oct 22 12:12:06 2009 From: phd at phd.pp.ru (Oleg Broytman) Date: Thu, 22 Oct 2009 14:12:06 +0400 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: <4ADFEF35.6000409@g.nevcal.com> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> <4ADFEF35.6000409@g.nevcal.com> Message-ID: <20091022101206.GF13617@phd.pp.ru> On Wed, Oct 21, 2009 at 10:35:49PM -0700, Glenn Linderman wrote: > Maybe what David is missing is that since python-dev is uninterested in > the package management issue, the only remaining way to include package > management in a convenient, single installation, is to > > 1) Create the package management tool > 2) Create a bundled installer that will install both Python, the package > management tool, and any dependencies of the package management tool > 3) Advertise the availability of the bundle, and its usefulness* > 4) See how many users try it out, and as new versions of Python are > created, how many users keep coming back, in preference to the package > management tool free python.org distribution. David, may be my memory betrays me, but I think you ask you question not for the first but for the second, if not the third time. And every time you got exactly this answer - "Create the package management tool and see how many users try it out". Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Thu Oct 22 12:20:03 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 10:20:03 +0000 (UTC) Subject: [Python-Dev] Python Package Management Roadmap in Python Releases References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> <4ADFEF35.6000409@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > > Maybe what David is missing is that since python-dev is uninterested in > the package management issue, [...] It's a bit strong to say we are uninterested. Most of us are not interested enough to tackle it ourselves (*), but we are certainly interested in good package management being available for Python. (*) Remember, however, that Tarek and work on Distribute, and also on bringing pieces of setuptools/Distribute functionality into distutils. It's true, however, that it is probably not the place to ask for advice with directions to follow for a yet unwritten piece of software. The only piece of advice (besides "forget Tk and use a modern toolkit" :-)) I could give would be to build on Distribute, rather than reinvent a whole bunch of mechanisms. Regards Antoine. From mal at egenix.com Thu Oct 22 12:57:32 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 22 Oct 2009 12:57:32 +0200 Subject: [Python-Dev] time.clock() on windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> Message-ID: <4AE03A9C.9020506@egenix.com> Curt Hagenlocher wrote: > On Wed, Oct 21, 2009 at 1:51 PM, Antoine Pitrou wrote: >> >> Kristj?n Valur J?nsson ccpgames.com> writes: >>> >>> You are right, on windows time.clock() is based relative to its first call >>> in the process. There is no such promise made on unix. >>> QueryPerformanceCounter() (what time.clock uses()) is a robust high >>> resolution timer that is processor/core independent. It should be >>> possible to use it across different processes too, if this >>> annoying rebasing wasn't there. >> >> Well, could we simply have a high-resolution time.time()? >> Or is Windows just too limited to provide this? > > Presumably you could fake something like this by combining output from > an initial time(), an initial QueryPerformanceCounter() and the > current QueryPerformanceCounter(). But it makes more sense to > understand why someone chose to implement time.clock() on Windows the > way they did -- this seems very broken to me, and I think it should be > changed. > > Of course, there are no doubt people relying on the broken behavior... I'm not sure I understand why time.clock() should be considered broken. time.clock() is used for measuring process CPU time and is usually only used in a relative way, ie. you remember that start value, do something, then subtract the end value from the start value. For absolute CPU time values associated with a process, it's usually better to rely on other APIs such as getrusage() on Unix. You might want to have a look at the systimes module that comes with pybench for some alternative timers: Tools/pybench/systimes.py This module tries to provide a cross-platform API for process and system time: """ systimes() user and system timer implementations for use by pybench. This module implements various different strategies for measuring performance timings. It tries to choose the best available method based on the platforma and available tools. On Windows, it is recommended to have the Mark Hammond win32 package installed. Alternatively, the Thomas Heller ctypes packages can also be used. On Unix systems, the standard resource module provides the highest resolution timings. Unfortunately, it is not available on all Unix platforms. If no supported timing methods based on process time can be found, the module reverts to the highest resolution wall-clock timer instead. The system time part will then always be 0.0. The module exports one public API: def systimes(): Return the current timer values for measuring user and system time as tuple of seconds (user_time, system_time). Copyright (c) 2006, Marc-Andre Lemburg (mal at egenix.com). See the documentation for further information on copyrights, or contact the author. All Rights Reserved. """ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 22 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mhammond at skippinet.com.au Thu Oct 22 12:57:53 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 22 Oct 2009 21:57:53 +1100 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F098FF42178@exchis.ccp.ad.local> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> <930F189C8A437347B80DF2C156F7EC7F098FF42178@exchis.ccp.ad.local> Message-ID: <4AE03AB1.5000000@skippinet.com.au> On 22/10/2009 8:47 PM, Kristj?n Valur J?nsson wrote: > The point in question seems to be this this (from the thread): > * Need some sort of static "start value", which is set when the > process starts, so I can return to Python in seconds. An easy hack > is to set this the first time clock() is called, but then it wont > reflect any sort of real time - but would be useful for relative > times... > > But the argumentation is flawed. It was made in the context of the APIs available to implement this. The code is short-and-sweet in timemodule.c, so please do go ahead and fix my flawed reasoning. For reference: #if defined(MS_WINDOWS) && !defined(__BORLANDC__) /* Due to Mark Hammond and Tim Peters */ static PyObject * time_clock(PyObject *self, PyObject *unused) { static LARGE_INTEGER ctrStart; static double divisor = 0.0; LARGE_INTEGER now; double diff; if (divisor == 0.0) { LARGE_INTEGER freq; QueryPerformanceCounter(&ctrStart); if (!QueryPerformanceFrequency(&freq) || freq.QuadPart == 0) { /* Unlikely to happen - this works on all intel machines at least! Revert to clock() */ return PyFloat_FromDouble(((double)clock()) / CLOCKS_PER_SEC); } divisor = (double)freq.QuadPart; } QueryPerformanceCounter(&now); diff = (double)(now.QuadPart - ctrStart.QuadPart); return PyFloat_FromDouble(diff / divisor); } Cheers, Mark. From mhammond at skippinet.com.au Thu Oct 22 13:04:16 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 22 Oct 2009 22:04:16 +1100 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <1256186734.2929.37.camel@lifeless-64> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> <1256186734.2929.37.camel@lifeless-64> Message-ID: <4AE03C30.6060500@skippinet.com.au> On 22/10/2009 3:45 PM, Robert Collins wrote: > On Thu, 2009-10-22 at 15:21 +1100, Mark Hammond wrote: > >> I'd be very surprised if any applications rely on >> the fact that each process starts counting at zero, so if someone can >> come up with a high-res counter which avoids that artifact I'd expect it >> could be used. > > Could you offset it by the system time on the first call? Off the top of my head, I guess that depends on the actual accuracy required (ie, how many clock ticks elapse between querying the time and the high-resolution timer). Starting at 0 works fine for profiling in a single process, the predominant use-case when this was done; I guess it depends on the specific requirements and time-intervals being dealt with in the cross-process case which determines how suitable that might be? Cheers, Mark From solipsis at pitrou.net Thu Oct 22 13:19:54 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 11:19:54 +0000 (UTC) Subject: [Python-Dev] time.clock() on windows References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4AE03A9C.9020506@egenix.com> Message-ID: M.-A. Lemburg egenix.com> writes: > > I'm not sure I understand why time.clock() should be considered > broken. Because in some cases you want comparable high-resolution numbers from distinct processes. > time.clock() is used for measuring process CPU time According to the docs, under Windows it measures wall-clock time rather than CPU time. Regards Antoine. From kristjan at ccpgames.com Thu Oct 22 15:39:21 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 22 Oct 2009 13:39:21 +0000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <4AE03AB1.5000000@skippinet.com.au> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <4ADF829A.7060509@scottdial.com> <4ADFDDB9.3080707@gmail.com> <930F189C8A437347B80DF2C156F7EC7F098FF42178@exchis.ccp.ad.local> <4AE03AB1.5000000@skippinet.com.au> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF42268@exchis.ccp.ad.local> > -----Original Message----- > From: Mark Hammond [mailto:mhammond at skippinet.com.au] > Sent: 22. okt?ber 2009 10:58 > To: Kristj?n Valur J?nsson > Cc: Scott Dial; python-dev at python.org > It was made in the context of the APIs available to implement this. > The > code is short-and-sweet in timemodule.c, so please do go ahead and fix > my flawed reasoning. ... I'm sorry, I don't want to start a flame war here, it just seems that if you need a zero point, and arbitrarily choose the first call to time.clock(), you could just as well use the implicit zero point already provided by the system. > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of M.-A. Lemburg > I'm not sure I understand why time.clock() should be considered > broken. Ah, well, not broken, but it could be even more useful: If it used the implicit system-wide epoch rather than the one based on the first call within each process, it could be useful for cross-process high-resolution timings. Anyway, it is simple enough to patch it on windows using ctypes if one needs that kind of behaviour: #nuclock.py import ctypes import time counter = ctypes.c_uint64() pcounter = ctypes.byref(counter) ctypes.windll.kernel32.QueryPerformanceFrequency(pcounter) frequency = float(counter.value) QPC = ctypes.windll.kernel32.QueryPerformanceCounter def nuclock(): QPC(pcounter) return float(counter.value)/frequency time.clock = nuclock Cheers, Kristjan From barry at python.org Thu Oct 22 16:44:46 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 22 Oct 2009 10:44:46 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 Message-ID: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> I'd like to get a second opinion on bug 7183: http://bugs.python.org/issue7183 The Boost folks have reported this as a regression in 2.6.3, making it a candidate for Python 2.6.4. IIUC, the latest version of Boost fixes the problem in their code, but if it really is a regression it could affect other projects and should probably be fixed. Robert Collins from the Bazaar team pinged me on IRC and originally thought that they'd been bitten by this problem too, though he looked around and determined it probably did /not/ affect them after all. So we have only anecdotal evidence of a problem, and no reproducible test case. If the Python 2.6.4 release is to be held up for this issue, we need to know today or tomorrow. Come the weekend, I'm going ahead with the tag and release. Holding up the release will mean another release candidate, and a wait of another week for the final. So does anybody else think bug 7183 should be a release blocker for 2.6.4 final, or is even a legitimate but that we need to fix? Thanks, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From benjamin at python.org Thu Oct 22 16:47:17 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 22 Oct 2009 09:47:17 -0500 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> 2009/10/22 Barry Warsaw : > So does anybody else think bug 7183 should be a release blocker for 2.6.4 > final, or is even a legitimate but that we need to fix? I think it cannot hold up a release with out a reproducible code snippet. -- Regards, Benjamin From stefan-usenet at bytereef.org Thu Oct 22 11:10:34 2009 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 22 Oct 2009 11:10:34 +0200 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <5c6f2a5d0910210828g5dab0aa0qac54815741734e8f@mail.gmail.com> References: <20091020131547.GA615@mail.bytereef.org> <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> <5c6f2a5d0910210828g5dab0aa0qac54815741734e8f@mail.gmail.com> Message-ID: <20091022091034.GA2812@mail.bytereef.org> Mark Dickinson wrote: > Thanks, Antoine! With SLOCCount I can revise my earlier numbers, as well: > Here's Stefan Krah's mpdecimal, version 0.80: > > SLOC Directory SLOC-by-Language (Sorted) > 21445 top_dir ansic=21267,sh=105,python=55,asm=18 > 6238 python python=6177,java=43,sh=18 > 1403 tests ansic=1356,sh=47 > 476 literature lisp=476 > 274 cmd ansic=274 > 11 tools sh=11 > 0 doc (none) I would say that the relevant code is less than that: The module code is counted twice (fastdec2.c, fastdec3.c), and sloccount counts the inline functions in the mpdecimal*.h header files. So, after removing fastdec3.c, mpdecimal32.h, mpdecimal32vc.h, mpdecimal64vc.h, I get: SLOC Directory SLOC-by-Language (Sorted) 13702 top_dir ansic=13524,sh=105,python=55,asm=18 6238 python python=6177,java=43,sh=18 1403 tests ansic=1356,sh=47 476 literature lisp=476 274 cmd ansic=274 11 tools sh=11 0 doc (none) Totals grouped by language (dominant language first): ansic: 15154 (68.56%) python: 6232 (28.19%) lisp: 476 (2.15%) sh: 181 (0.82%) java: 43 (0.19%) asm: 18 (0.08%) Therefore, my estimate is between 12660 and 13702 lines, depending on whether to count the remaining mpdecimal64.h (Most inline functions in this header file are signaling wrappers around the quiet functions and are not necessary for the module). If one takes out all library functions that the module does not use, I'm sure it could be condensed to ~11500 lines. This is comparable to the expat directory in Modules/: SLOC Directory SLOC-by-Language (Sorted) 11406 expat ansic=11406 Stefan Krah From stefan-usenet at bytereef.org Thu Oct 22 16:50:17 2009 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 22 Oct 2009 16:50:17 +0200 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> References: <20091020131547.GA615@mail.bytereef.org> <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> Message-ID: <20091022145017.GB2812@mail.bytereef.org> Mark Dickinson wrote: > > 2. Would fastdec - after achieving full decimal.py compatibility - be > > ? a serious candidate? > > Definitely. As far as I know it's the only real candidate for a full > C version of decimal right now. Other possibilities that I'm aware of: > > * I think Raymond Hettinger is working on a C version of decimal. > I'm not sure what its current status is. Raymond? > > * There's a partially complete rewrite of decimal in C in the sandbox, > dating from the Need for Speed sprint in 2006: > > * Wrapping the decNumber library directly would make some sense, > but I think licensing issues rule this out. I hope I'm not spreading misinformation, but I see two problems with the use of decNumber as an arbitrary precision library. First, the algorithms are not optimized for large numbers. Then, you have to define the maximum length of the coefficient *at compile time* by setting DECNUMDIGITS. > It's difficult to gauge the likelihood of eventual acceptance in advance, > though. Maybe writing a PEP would be an efficient use of time in this > regard? There are certainly some open issues (e.g., what to do with > the existing Python module; what should other Python implementations > do). Good, I'll do that. I mainly wanted to find out if there is fundamental opposition against C-decimal in the standard library, but this does not seem to be the case. > I think my biggest concern is maintenance: we'd be replacing > 8500 lines of Python code in a single file, that several of the > current core developers understand well, with 30000 (Stefan, is > that about accurate?) lines of C in several files, that presumably > none of the current core devs is familiar with right now. Ok, I think we could say that the sloc-counts are around 2600 lines of Python vs. around 11500-12500 lines of C. > What happens when (e.g.,) the number-theoretic transform needs > updating for one reason or another? Stefan, do you see yourself > having time to spend on maintenance of this code for the > forseeable future? Yes. Generally speaking, I expect the code to be low-maintenance. It would be even easier if all major compilers supported exact width C99 types and __uint128_t, but this is wishful thinking. Stefan Krah From barry at python.org Thu Oct 22 17:04:22 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 22 Oct 2009 11:04:22 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> Message-ID: <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> On Oct 22, 2009, at 10:47 AM, Benjamin Peterson wrote: > 2009/10/22 Barry Warsaw : >> So does anybody else think bug 7183 should be a release blocker for >> 2.6.4 >> final, or is even a legitimate but that we need to fix? > > I think it cannot hold up a release with out a reproducible code > snippet. It may not be reproducible in standard Python, see David's follow up to the issue. If that holds true and we can't reproduce it, I agree we should not hold up the release for this. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From fabiofz at gmail.com Thu Oct 22 17:32:37 2009 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Thu, 22 Oct 2009 12:32:37 -0300 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: References: Message-ID: On Wed, Oct 21, 2009 at 10:56 PM, Mike Krell wrote: > Is there any possibility of backporting support for the nonlocal keyword > into a? 2.x release?? I see it's not in 2.6, but I don't know if that was an > intentional design choice or due to a lack of demand / round tuits.? I'm > also not sure if this would fall under the scope of the proposed moratorium > on new language features (although my first impression was that it could be > allowed since it already exists in python 3. > > One of my motivations for asking is a recent blog post by Fernando Perez of > IPython fame that describes an interesting decorator-based idiom inspired by > Apple's Grand Central Dispatch which would allow many interesting > possibilities for expressing parallelization and other manipulations of > execution context for blocks of python code.? Unfortunately, using the > technique to its fullest extent requires the nonlocal keyword. > > The blog post is here: > https://cirl.berkeley.edu/fperez/py4science/decorators.html > Just as a note, the nonlocal there is not a requirement... You can just create a mutable object there and change that object (so, you don't need to actually rebind the object in the outer scope). E.g.: instead of creating a float in the context, create a list with a single float and change the float in the list (maybe the nonlocal would be nicer, but it's certainly still usable) Cheers, Fabio From g.brandl at gmx.net Thu Oct 22 17:43:04 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 22 Oct 2009 17:43:04 +0200 Subject: [Python-Dev] PyErr_NewExceptionWithDoc Message-ID: Hi, issue #7033 proposes a new C API that creates a new exception class with a docstring. Since exception classes should be documented, and adding the docstring after creating is not a one-liner, I would say it is a useful addition. What do you all think? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From stefan_ml at behnel.de Thu Oct 22 18:41:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 22 Oct 2009 18:41:03 +0200 Subject: [Python-Dev] Interest in integrating C decimal module into Python? In-Reply-To: <20091022145017.GB2812@mail.bytereef.org> References: <20091020131547.GA615@mail.bytereef.org> <5c6f2a5d0910210751k2851cb31qa0c61ed49545bef3@mail.gmail.com> <20091022145017.GB2812@mail.bytereef.org> Message-ID: Stefan Krah wrote: > Mark Dickinson wrote: >> I think my biggest concern is maintenance: we'd be replacing >> 8500 lines of Python code in a single file, that several of the >> current core developers understand well, with 30000 (Stefan, is >> that about accurate?) lines of C in several files, that presumably >> none of the current core devs is familiar with right now. > > Ok, I think we could say that the sloc-counts are around 2600 lines of > Python vs. around 11500-12500 lines of C. If maintenance is an issue, I would actually propose to compile the existing decimal.py using Cython (works with a few trivial modifications), add a few type decorators at the hot spots to make them really fast, and then use that as both the Python implementation and fast binary module. Stefan From solipsis at pitrou.net Thu Oct 22 18:45:00 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 16:45:00 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?readonly_=5F=5Fdoc=5F=5F?= References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: Speaking of the __doc__ property, I just noticed the following thing on py3k: >>> class C: pass ... >>> C.__doc__ = "hop" Traceback (most recent call last): File "", line 1, in AttributeError: attribute '__doc__' of 'type' objects is not writable Is this deliberate? Antoine. From fetchinson at googlemail.com Thu Oct 22 19:00:51 2009 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Thu, 22 Oct 2009 19:00:51 +0200 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: > Speaking of the __doc__ property, I just noticed the following thing on > py3k: > >>>> class C: pass > ... >>>> C.__doc__ = "hop" > Traceback (most recent call last): > File "", line 1, in > AttributeError: attribute '__doc__' of 'type' objects is not writable Happens also with new style classes in python 2.x: Python 2.5.1 (r251:54863, Oct 30 2007, 13:45:26) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class C(object): pass ... >>> C.__doc__ = 'hello' Traceback (most recent call last): File "", line 1, in AttributeError: attribute '__doc__' of 'type' objects is not writable >>> But not with old style classes: Python 2.5.1 (r251:54863, Oct 30 2007, 13:45:26) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class C: pass ... >>> C.__doc__ = 'hello' >>> Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From foom at fuhm.net Thu Oct 22 19:01:02 2009 From: foom at fuhm.net (James Y Knight) Date: Thu, 22 Oct 2009 13:01:02 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> Message-ID: <14794788-7FFC-4713-BB4C-8213CA746BA3@fuhm.net> On Oct 22, 2009, at 11:04 AM, Barry Warsaw wrote: > On Oct 22, 2009, at 10:47 AM, Benjamin Peterson wrote: > >> 2009/10/22 Barry Warsaw : >>> So does anybody else think bug 7183 should be a release blocker >>> for 2.6.4 >>> final, or is even a legitimate but that we need to fix? >> >> I think it cannot hold up a release with out a reproducible code >> snippet. > > It may not be reproducible in standard Python, see David's follow up > to the issue. If that holds true and we can't reproduce it, I agree > we should not hold up the release for this. >>> class Foo(property): ... __slots__=[] ... >>> x=Foo() >>> x.__doc__ = "asdf" Traceback (most recent call last): File "", line 1, in AttributeError: 'Foo' object attribute '__doc__' is read-only You can't add arbitrary attributes to instances, since some instances don't have the slot to put them in. Is that an equivalent demonstration to that which boost runs into? (except, it's using a C type not a python type). James From tseaver at palladion.com Thu Oct 22 19:05:50 2009 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 22 Oct 2009 13:05:50 -0400 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADFC9E8.1010400@molden.no> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antoine Pitrou wrote: > Sturla Molden molden.no> writes: >> With two threads and a check interval og 100, only 61 of 100000 check >> intervals failed to produce a thread-switch in the interpreter. I'd call >> that rather fair. > > This number lacks the elapsed time. 61 switches in one second is probably > enough, the same amount of switches in 10 or 20 seconds is too small (at least > for threads needing good responsivity, e.g. I/O threads). I read Sturla as saying there were 99,939 switches out of a possible 100,000, with sys.checkinterval set to 100. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrgkO4ACgkQ+gerLs4ltQ4yZgCfZyRfWKPBzRb52v7RtLAMlNts SOIAnjgxqJ1Wovas5cHoju8kwbvn/9Es =2SDx -----END PGP SIGNATURE----- From guido at python.org Thu Oct 22 19:12:09 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Oct 2009 10:12:09 -0700 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: On Thu, Oct 22, 2009 at 9:45 AM, Antoine Pitrou wrote: > > Speaking of the __doc__ property, I just noticed the following thing on py3k: > >>>> class C: pass > ... >>>> C.__doc__ = "hop" > Traceback (most recent call last): > ?File "", line 1, in > AttributeError: attribute '__doc__' of 'type' objects is not writable > > Is this deliberate? Yes. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From solipsis at pitrou.net Thu Oct 22 19:16:46 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 17:16:46 +0000 (UTC) Subject: [Python-Dev] GIL behaviour under Windows References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADFC9E8.1010400@molden.no> Message-ID: Tres Seaver palladion.com> writes: > > I read Sturla as saying there were 99,939 switches out of a possible > 100,000, with sys.checkinterval set to 100. Oops, you're right. But depending on the elapsed time (again :-)), it may be too high, because too many switches per second will add a lot of overhead and decrease performance. Regards Antoine. From tseaver at palladion.com Thu Oct 22 19:16:32 2009 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 22 Oct 2009 13:16:32 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On Oct 22, 2009, at 10:47 AM, Benjamin Peterson wrote: > >> 2009/10/22 Barry Warsaw : >>> So does anybody else think bug 7183 should be a release blocker for >>> 2.6.4 >>> final, or is even a legitimate but that we need to fix? >> I think it cannot hold up a release with out a reproducible code >> snippet. > > It may not be reproducible in standard Python, see David's follow up > to the issue. If that holds true and we can't reproduce it, I agree > we should not hold up the release for this. The fix for 5890 has a funny "smell" to me: copying __doc__ into the instance dict just feels wrong. How does that work with a pure Python class using slots? E.g.: class MyProp(property): pass class Foo(object): @property def bar(self): '''Get a bar.''' @MyProp def baz(self): '''Get a baz.''' print Foo.bar.__doc__ print Foo.baz.__doc__ class SlottedFoo(object): __slots__ = () @property def bar(self): '''Get a bar.''' @MyProp def baz(self): '''Get a baz.''' print SlottedFoo.bar.__doc__ print SlottedFoo.baz.__doc__ That being said, I can't this bug as a release blocker: people can either upgrade to super-current Boost, or stick with 2.6.2 until they can. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrgk3AACgkQ+gerLs4ltQ6n7gCgmV3zWTazLTHWfmD2zul0BTnF 9dkAn38HXKw4RPwWV8Ht4hUGBC00IB2N =ssdH -----END PGP SIGNATURE----- From ubershmekel at gmail.com Thu Oct 22 19:18:45 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Thu, 22 Oct 2009 19:18:45 +0200 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: <9d153b7c0910221018u60f2e87x7c2097d369fd9e6@mail.gmail.com> On Thu, Oct 22, 2009 at 7:12 PM, Guido van Rossum wrote: > Yes. Why? --yuv From solipsis at pitrou.net Thu Oct 22 19:25:30 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 17:25:30 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?readonly_=5F=5Fdoc=5F=5F?= References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: Guido van Rossum python.org> writes: > > On Thu, Oct 22, 2009 at 9:45 AM, Antoine Pitrou pitrou.net> wrote: > > > > Speaking of the __doc__ property, I just noticed the following thing on py3k: > > > >>>> class C: pass > > ... > >>>> C.__doc__ = "hop" > > Traceback (most recent call last): > > ?File "", line 1, in > > AttributeError: attribute '__doc__' of 'type' objects is not writable > > > > Is this deliberate? > > Yes. I might add why I was asking this question. I was trying to demonstrate the use of class decorators and the simplest example I found was to add a docstring to the class. And I was surprised when I saw the following fail with the aforementioned exception: def classdeco(cls): cls.__doc__ = "decorated!" return cls @classdeco class C: pass Regards Antoine. From phillip.sitbon+python-dev at gmail.com Thu Oct 22 19:51:01 2009 From: phillip.sitbon+python-dev at gmail.com (Phillip Sitbon) Date: Thu, 22 Oct 2009 10:51:01 -0700 Subject: [Python-Dev] [Python-ideas] Remove GIL with CAS instructions? In-Reply-To: <4ADF2850.1040304@arbash-meinel.com> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <4ADF2850.1040304@arbash-meinel.com> Message-ID: <536685ea0910221051p1897770xc517561dccc020f6@mail.gmail.com> I'd just like to point out some previous discussion about implementing the GIL as a critical section or semaphore on Windows since it's come up here (although not quite the center of the OP's proposal AFAICS): http://bugs.python.org/issue6132 http://mail.python.org/pipermail/python-dev/2009-May/089746.html Some of this is more low-level. I did see higher performance when using non-Event objects, although I have not had time to follow up and do a deeper analysis. The GIL flashing "problem" with critical sections can very likely be rectified with a call to Sleep(0) or YieldProcessor() for those who are worried about it. On the subject of fairness, I tested various forms of the GIL on my multi-threaded ISAPI extension, where every millisecond counts when under high concurrency, and fairness wasn't an issue for single- or multi-core systems. It may be anecdotal, but it also may be that the issue is somewhat over-blown. It seems like these discussions come up in one form or another a few times a year and don't really get anywhere - probably because many people find that it's easier to just run one instance of Python on each core/processor. IPC is cheap (cPickle rocks!), and Python's memory footprint is acceptable by today's standards. Still, it is an interesting topic to many, myself included. Also, many people keep talking about inefficiencies due to threads waking up to a locked GIL. I'd like to see examples of this- most of the time, the OS should know that the thread is contending on the lock object and it is skipped over. Granted, a thread may wake up just to release the GIL shortly thereafter, but that's why sys.setcheckinterval() is there for us to tinker with. Anyway, enough of my $0.02. - Phillip 2009/10/21 John Arbash Meinel : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Kristj?n Valur J?nsson wrote: > ... > >> This depends entirely on the platform and primitives used to implement the GIL. >> I'm interested in windows. ?There, I found this article: >> http://fonp.blogspot.com/2007/10/fairness-in-win32-lock-objects.html >> So, you may be on to something. ?Perhaps a simple C test is in order then? >> >> I did that. ?I found, on my dual-core vista machine, that running "release", that both Mutexes and CriticalSections behaved as you describe, with no "fairness". ?Using a "semaphore" seems to retain fairness, however. >> "fairness" was retained in debug builds too, strangely enough. >> >> Now, Python uses none of these. ?On windows, it uses an "Event" object coupled with an atomically updated counter. ?This also behaves fairly. >> >> The test application is attached. >> >> >> I think that you ought to sustantiate your claims better, maybe with a specific platform and using some test like the above. >> >> On the other hand, it shows that we must be careful what we use. ?There has been some talk of using CriticalSections for the GIL on windows. ?This test ought to show the danger of that. ?The GIL is different than a regular lock. ?It is a reverse-lock, really, and therefore may need to be implemented in its own special way, if we want very fast mutexes for the rest of the system (cc to python-dev) >> >> Cheers, >> >> Kristj?n > > I can compile and run this, but I'm not sure I know how to interpret the > results. If I understand it correctly, then everything but "Critical > Sections" are fair on my Windows Vista machine. > > To run, I changed the line "#define EVENT" to EVENT, MUTEX, SEMAPHORE > and CRIT. I then built and ran in "Release" environment (using VS 2008 > Express) > > For all but CRIT, I saw things like: > thread ? ? ? 5532 reclaims GIL > thread ? ? ? 5532 working 51234 units > thread ? ? ? 5532 worked 51234 units: 1312435761 > thread ? ? ? 5532 flashing GIL > thread ? ? ? 5876 reclaims GIL > thread ? ? ? 5876 working 51234 units > thread ? ? ? 5876 worked 51234 units: 1312435761 > thread ? ? ? 5876 flashing GIL > > Where there would be 4 lines for one thread, then 4 lines for the other > thread. > > for CRIT, I saw something more like 50 lines for one thread, and then 50 > lines for the other thread. > > This is Vista Home Basic, and VS 2008 Express Edition, with a 2-core > machine. > > John > =:-> > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (Cygwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkrfKFAACgkQJdeBCYSNAANbuQCgudU0IChylofTwvUk/JglChBd > 9gsAoIJHj63/CagKpduUsd68HV8pP3QX > =CuUj > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/phillip.sitbon%2Bpython-dev%40gmail.com > From guido at python.org Thu Oct 22 19:58:08 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Oct 2009 10:58:08 -0700 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: Well __doc__ isn't a normal attribute -- it doesn't follow inheritance rules. On Thu, Oct 22, 2009 at 10:25 AM, Antoine Pitrou wrote: > Guido van Rossum python.org> writes: >> >> On Thu, Oct 22, 2009 at 9:45 AM, Antoine Pitrou pitrou.net> > wrote: >> > >> > Speaking of the __doc__ property, I just noticed the following thing > on py3k: >> > >> >>>> class C: pass >> > ... >> >>>> C.__doc__ = "hop" >> > Traceback (most recent call last): >> > ?File "", line 1, in >> > AttributeError: attribute '__doc__' of 'type' objects is not writable >> > >> > Is this deliberate? >> >> Yes. > > I might add why I was asking this question. I was trying to demonstrate the use > of class decorators and the simplest example I found was to add a docstring to > the class. And I was surprised when I saw the following fail with the > aforementioned exception: > > def classdeco(cls): > ? ?cls.__doc__ = "decorated!" > ? ?return cls > > @classdeco > class C: > ? ?pass > > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From alexander.belopolsky at gmail.com Thu Oct 22 19:52:51 2009 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 22 Oct 2009 13:52:51 -0400 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: On Thu, Oct 22, 2009 at 1:25 PM, Antoine Pitrou wrote: .. > I might add why I was asking this question. I was trying to demonstrate the use > of class decorators and the simplest example I found was to add a docstring to > the class. I always thought that read-only __doc__ was an artifact of new classes evolving from (C-implemented) types. IIRC, NumPy project implemented a somewhat elaborate hack to allow docstrings to be added to C-implemented classes in a .py file. From ssteinerx at gmail.com Thu Oct 22 20:18:37 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 22 Oct 2009 14:18:37 -0400 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> Message-ID: <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> On Oct 22, 2009, at 1:58 PM, Guido van Rossum wrote: > Well __doc__ isn't a normal attribute -- it doesn't follow > inheritance rules. Maybe we could add a ticket to flag this in the docs. Is __doc__ not normal due to its general underscorishness, or is it not normal because it isn't? Any others that deserve special notice, while we're at it? Thanks, S From phillip.sitbon+python-dev at gmail.com Thu Oct 22 20:19:37 2009 From: phillip.sitbon+python-dev at gmail.com (Phillip Sitbon) Date: Thu, 22 Oct 2009 11:19:37 -0700 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADFC9E8.1010400@molden.no> Message-ID: <536685ea0910221119t66cc1b37ua940e9a8eda8294f@mail.gmail.com> I know I already posted some relevant threads to the other discussion, but I wanted to point out a couple of specific comments on GIL fairness from the discussion: http://mail.python.org/pipermail/python-dev/2009-May/089752.html http://mail.python.org/pipermail/python-dev/2009-May/089755.html - Phillip On Thu, Oct 22, 2009 at 10:16 AM, Antoine Pitrou wrote: > Tres Seaver palladion.com> writes: >> >> I read Sturla as saying there were 99,939 switches out of a possible >> 100,000, with sys.checkinterval set to 100. > > Oops, you're right. > But depending on the elapsed time (again :-)), it may be too high, because > too many switches per second will add a lot of overhead and decrease performance. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/phillip.sitbon%2Bpython-dev%40gmail.com > From brett at python.org Thu Oct 22 20:30:38 2009 From: brett at python.org (Brett Cannon) Date: Thu, 22 Oct 2009 11:30:38 -0700 Subject: [Python-Dev] PyErr_NewExceptionWithDoc In-Reply-To: References: Message-ID: On Thu, Oct 22, 2009 at 08:43, Georg Brandl wrote: > Hi, > > issue #7033 proposes a new C API that creates a new exception class with > a docstring. Since exception classes should be documented, and adding > the docstring after creating is not a one-liner, I would say it is a useful > addition. What do you all think? > > Seems fine to me. > Georg > > -- > Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. > Four shall be the number of spaces thou shalt indent, and the number of thy > indenting shall be four. Eight shalt thou not indent, nor either indent > thou > two, excepting that thou then proceed to four. Tabs are right out. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Oct 22 20:36:37 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 22 Oct 2009 14:36:37 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> Message-ID: <65DE256C-9EE8-4B72-A647-0C47EE4C27E1@python.org> On Oct 22, 2009, at 1:16 PM, Tres Seaver wrote: > That being said, I can't this bug as a release blocker: people can > either upgrade to super-current Boost, or stick with 2.6.2 until > they can. Agreed. I've knocked it down from release-blocker. -Barryu -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From brett at python.org Thu Oct 22 20:39:53 2009 From: brett at python.org (Brett Cannon) Date: Thu, 22 Oct 2009 11:39:53 -0700 Subject: [Python-Dev] readonly __doc__ In-Reply-To: <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> Message-ID: On Thu, Oct 22, 2009 at 11:18, ssteinerX at gmail.com wrote: > > On Oct 22, 2009, at 1:58 PM, Guido van Rossum wrote: > > Well __doc__ isn't a normal attribute -- it doesn't follow inheritance >> rules. >> > > Maybe we could add a ticket to flag this in the docs. > > Sure, go for it. > Is __doc__ not normal due to its general underscorishness, or is it not > normal because it isn't? > > I honestly don't follow that sentence. But __doc__ is special because of its use; documenting how to use of an object. In this case when you call something like help() on an instance of an object it skips the instance's value for __doc__ and goes straight to the defining class and stops there as you don't care how a subclass says to use itself as that is not what you are working with. > Any others that deserve special notice, while we're at it? > Don't know. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Thu Oct 22 20:44:05 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 22 Oct 2009 14:44:05 -0400 (EDT) Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> Message-ID: On Thu, 22 Oct 2009 at 13:16, Tres Seaver wrote: > The fix for 5890 has a funny "smell" to me: copying __doc__ into the > instance dict just feels wrong. How does that work with a pure Python > class using slots? E.g.: It doesn't. There's even a test to make sure it doesn't :) (It raises an attribute error.) __doc__ is a funny beast. If someone can come up with a better fix for 5890, that would be great. --David From barry at barrys-emacs.org Thu Oct 22 21:31:42 2009 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 22 Oct 2009 20:31:42 +0100 Subject: [Python-Dev] Add const to python API - issue 6952 In-Reply-To: <4ADE98E2.3080707@v.loewis.de> References: <4ADD2FE9.2010807@v.loewis.de> <74D1336F-AB87-431F-895C-885EE344CEBD@barrys-emacs.org> <4ADE98E2.3080707@v.loewis.de> Message-ID: On 21 Oct 2009, at 06:15, Martin v. L?wis wrote: >> I suggest following POSIX's lead and omitted the const in these >> cases. Ah... Yes I see this in strstr for example. > > Thanks, that sounds reasonable. This is basically what I have done in the patch I hope. I have updated the patch to include a comment explaining why output param is char ** and the need for a cast. Barry From ssteinerx at gmail.com Thu Oct 22 21:48:36 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Thu, 22 Oct 2009 15:48:36 -0400 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> Message-ID: <995C225C-8141-43D3-A840-75F7F336FA5E@gmail.com> On Oct 22, 2009, at 2:39 PM, Brett Cannon wrote: > > On Thu, Oct 22, 2009 at 11:18, ssteinerX at gmail.com > wrote: > > On Oct 22, 2009, at 1:58 PM, Guido van Rossum wrote: > > Well __doc__ isn't a normal attribute -- it doesn't follow > inheritance rules. > > Maybe we could add a ticket to flag this in the docs. > Sure, go for it. Filed: http://bugs.python.org/issue7186 > > Is __doc__ not normal due to its general underscorishness, or is it > not normal because it isn't? > > > I honestly don't follow that sentence. It means, is it special because it's a "special" i.e. __X__ type attribute or because of something special about __doc__ itself, independent of it being a "special attribute". > But __doc__ is special because of its use; documenting how to use of > an object. In this case when you call something like help() on an > instance of an object it skips the instance's value for __doc__ and > goes straight to the defining class and stops there as you don't > care how a subclass says to use itself as that is not what you are > working with. > > Any others that deserve special notice, while we're at it? > > Don't know. Ok, issue recorded. Thanks, S -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Thu Oct 22 21:47:26 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 22 Oct 2009 21:47:26 +0200 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> Message-ID: Tres Seaver schrieb: > Barry Warsaw wrote: >> On Oct 22, 2009, at 10:47 AM, Benjamin Peterson wrote: > >>> 2009/10/22 Barry Warsaw : >>>> So does anybody else think bug 7183 should be a release blocker for >>>> 2.6.4 >>>> final, or is even a legitimate but that we need to fix? >>> I think it cannot hold up a release with out a reproducible code >>> snippet. > >> It may not be reproducible in standard Python, see David's follow up >> to the issue. If that holds true and we can't reproduce it, I agree >> we should not hold up the release for this. > > The fix for 5890 has a funny "smell" to me: copying __doc__ into the > instance dict just feels wrong. How does that work with a pure Python > class using slots? E.g.: > > class MyProp(property): > pass > > class Foo(object): > @property > def bar(self): > '''Get a bar.''' > > @MyProp > def baz(self): > '''Get a baz.''' > > print Foo.bar.__doc__ > print Foo.baz.__doc__ > > class SlottedFoo(object): > __slots__ = () > @property > def bar(self): > '''Get a bar.''' > > @MyProp > def baz(self): > '''Get a baz.''' > > print SlottedFoo.bar.__doc__ > print SlottedFoo.baz.__doc__ Works just fine. I think you meant class MyProp(property): __slots__ = () Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From robertc at robertcollins.net Thu Oct 22 21:53:43 2009 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 23 Oct 2009 06:53:43 +1100 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> Message-ID: <1256241223.2929.56.camel@lifeless-64> On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: ... > That being said, I can't this bug as a release blocker: people can > either upgrade to super-current Boost, or stick with 2.6.2 until they can. Thats the challenge Ubuntu faces: https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 We've just announced our Karmic RC, boost 1.40 isn't released, and python 2.6.3 doesn't work with a released boost :( -Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From martin at v.loewis.de Thu Oct 22 22:24:35 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Oct 2009 22:24:35 +0200 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: References: Message-ID: <4AE0BF83.4080903@v.loewis.de> Mike Krell wrote: > Is there any possibility of backporting support for the nonlocal keyword > into a 2.x release? If so, only into 2.7. Can you please explain why it would be desirable to do that? 2.7 will likely be the last 2.x release, so only a fairly small portion of the applications would be actually able to use this (or any other new feature added to 2.7): most code supporting 2.x will also have to support 2.6, so the keyword won't be available to such code, anyway. Regards, Martin From p.f.moore at gmail.com Thu Oct 22 22:27:04 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 22 Oct 2009 21:27:04 +0100 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <1256241223.2929.56.camel@lifeless-64> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> <1256241223.2929.56.camel@lifeless-64> Message-ID: <79990c6b0910221327h37a678cfw4e566e4cb9f8083e@mail.gmail.com> 2009/10/22 Robert Collins : > On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: > ... >> That being said, I can't this bug as a release blocker: ?people can >> either upgrade to super-current Boost, or stick with 2.6.2 until they can. > > Thats the challenge Ubuntu faces: > https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 > > We've just announced our Karmic RC, boost 1.40 isn't released, and > python 2.6.3 doesn't work with a released boost :( ?? Boost 1.40 was released on 27th August, according to www.boost.org... Paul. From barry at python.org Thu Oct 22 22:29:18 2009 From: barry at python.org (Barry Warsaw) Date: Thu, 22 Oct 2009 16:29:18 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <1256241223.2929.56.camel@lifeless-64> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> <1256241223.2929.56.camel@lifeless-64> Message-ID: On Oct 22, 2009, at 3:53 PM, Robert Collins wrote: > On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: > ... >> That being said, I can't this bug as a release blocker: people can >> either upgrade to super-current Boost, or stick with 2.6.2 until >> they can. > > Thats the challenge Ubuntu faces: > https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 > > We've just announced our Karmic RC, boost 1.40 isn't released, and > python 2.6.3 doesn't work with a released boost :( Aren't we stuck either way though? :( -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From foom at fuhm.net Thu Oct 22 22:29:28 2009 From: foom at fuhm.net (James Y Knight) Date: Thu, 22 Oct 2009 16:29:28 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <1256241223.2929.56.camel@lifeless-64> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> <1256241223.2929.56.camel@lifeless-64> Message-ID: <0272128F-3703-411C-BA1B-2F7BF8AC63A4@fuhm.net> On Oct 22, 2009, at 3:53 PM, Robert Collins wrote: > On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: > ... >> That being said, I can't this bug as a release blocker: people can >> either upgrade to super-current Boost, or stick with 2.6.2 until >> they can. > > Thats the challenge Ubuntu faces: > https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 > > We've just announced our Karmic RC, boost 1.40 isn't released, and > python 2.6.3 doesn't work with a released boost :( If I were running a Linux distro, I'd revert the patch in 2.6.3. And if I were running a Python release process, I'd revert that patch for python 2.6.4, and reopen the bug that it fixed, so a less-breaky patch can be made. James From exarkun at twistedmatrix.com Thu Oct 22 22:46:21 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 22 Oct 2009 20:46:21 -0000 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <4AE0BF83.4080903@v.loewis.de> References: <4AE0BF83.4080903@v.loewis.de> Message-ID: <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> On 08:24 pm, martin at v.loewis.de wrote: >Mike Krell wrote: >>Is there any possibility of backporting support for the nonlocal >>keyword >>into a 2.x release? > >If so, only into 2.7. Can you please explain why it would be desirable >to do that? 2.7 will likely be the last 2.x release, so only a fairly >small portion of the applications would be actually able to use this >(or >any other new feature added to 2.7): most code supporting 2.x will also >have to support 2.6, so the keyword won't be available to such code, >anyway. For the same reason that it is desirable to backport all of the other changes from 3.x - because it makes the 2.x to 3.x transition easier. If Python 2.7 supports the nonlocal keyword, then 2.7 becomes that much better of a stepping stone towards 3.x. You've suggested that most 2.x code will have to support 2.6 and so won't be able to use the nonlocal keyword even if it is added to 2.7. This precise argument could be applied to all of the features in 2.6 which aim to bring it closer to 3.x. Any program which must retain Python 2.5 compatibility will not be able to use them. Yet 2.6 is a more useful stepping stone towards 3.x than 2.5 is. So yes, it would be quite desirable to see nonlocal and as many other 3.x features as possible backported for 2.7. And depending on how close 2.7 manages to get, it may make sense to backport anything that doesn't make it into 2.7 for a 2.8 release. The 3.x transition is *hard*. Anything that makes it easier is good. Jean-Paul From ncoghlan at gmail.com Thu Oct 22 22:49:19 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2009 06:49:19 +1000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F098FF4213E@exchis.ccp.ad.local> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF4213E@exchis.ccp.ad.local> Message-ID: <4AE0C54F.20705@gmail.com> Kristj?n Valur J?nsson wrote: > Yes. The problem with QPC is that although it has very high > resolution, it is not precise in the long term. And > GetSystemTimeAsFileTime() is high precision in the long term but only > updated evey 20ms or so. In EVE Online we use a combination of the > two for high resolution, long term precision. But I'm not happy with > the way I'm doing it. It needs some sort of smoothing of course. > I've even played with using Kalman filtering to do it... The idea is > to use the low frequency timer to apply correction coefficients to > the high frequency timer, yet keep the flow of time smooth (no > backwards jumps because of corrections.). An optimal solution has so > far eluded me. That sounds very similar to the problem spec for system time corrections in Network Time Protocol client implementations. Perhaps the time drifting algorithms in the NTP specs are relevant? Or are they too slow to correct discrepancies? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Oct 22 22:49:37 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 20:49:37 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?readonly_=5F=5Fdoc=5F=5F?= References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> Message-ID: Brett Cannon python.org> writes: > > > I honestly don't follow that sentence. But __doc__ is special because of its > use; documenting how to use of an object. In this case when you call > something like help() on an instance of an object it skips the instance's > value for __doc__ and goes straight to the defining class and stops there as > you don't care how a subclass says to use itself as that is not what you are > working with. I don't really understand how this explains the read-only __doc__. I am talking about modifying __doc__ on a class, not on an instance. (sure, a new-style class is also an instance of type, but still...) Antoine. From robertc at robertcollins.net Thu Oct 22 22:51:31 2009 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 23 Oct 2009 07:51:31 +1100 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <79990c6b0910221327h37a678cfw4e566e4cb9f8083e@mail.gmail.com> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> <1256241223.2929.56.camel@lifeless-64> <79990c6b0910221327h37a678cfw4e566e4cb9f8083e@mail.gmail.com> Message-ID: <1256244691.2929.69.camel@lifeless-64> On Thu, 2009-10-22 at 21:27 +0100, Paul Moore wrote: > 2009/10/22 Robert Collins : > > On Thu, 2009-10-22 at 13:16 -0400, Tres Seaver wrote: > > ... > >> That being said, I can't this bug as a release blocker: people can > >> either upgrade to super-current Boost, or stick with 2.6.2 until they can. > > > > Thats the challenge Ubuntu faces: > > https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 > > > > We've just announced our Karmic RC, boost 1.40 isn't released, and > > python 2.6.3 doesn't work with a released boost :( > > ?? Boost 1.40 was released on 27th August, according to www.boost.org... Oh bah, teach me to trust comments in bug reports :) Still, a full release is high risk, one of the Ubuntu devs is looking for the single patch they added to fix things. -Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From martin at v.loewis.de Thu Oct 22 22:57:32 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 22 Oct 2009 22:57:32 +0200 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> Message-ID: <4AE0C73C.9070805@v.loewis.de> exarkun at twistedmatrix.com wrote: > On 08:24 pm, martin at v.loewis.de wrote: >> Mike Krell wrote: >>> Is there any possibility of backporting support for the nonlocal keyword >>> into a 2.x release? >> >> If so, only into 2.7. Can you please explain why it would be desirable >> to do that? 2.7 will likely be the last 2.x release, so only a fairly >> small portion of the applications would be actually able to use this (or >> any other new feature added to 2.7): most code supporting 2.x will also >> have to support 2.6, so the keyword won't be available to such code, >> anyway. > > For the same reason that it is desirable to backport all of the other > changes from 3.x - because it makes the 2.x to 3.x transition easier. Hmm. Really? > If Python 2.7 supports the nonlocal keyword, then 2.7 becomes that much > better of a stepping stone towards 3.x. What use has such a stepping stone? Why, and (more importantly) when would anybody currently supporting 2.x give up 2.6 and earlier, and only support 2.7? And, if they chose to do so, why would they not move the code base to 3.x right away? > You've suggested that most 2.x code will have to support 2.6 and so > won't be able to use the nonlocal keyword even if it is added to 2.7. Correct. > This precise argument could be applied to all of the features in 2.6 > which aim to bring it closer to 3.x. Not so. One of the features added to 2.6 was the 3k warning. This warning can be used without any modification to the code. So code can run on 2.6 and use the feature, while running unmodified on 2.5 and earlier (not using it). As for actual language and library changes (such as any new future import): there was indeed little point adding them. However, given that the possible migration paths weren't as clear back then as they are now, it is understandable that people considered this a viable path. In addition, for 2.6, it's a bit more realistic to assume that people might drop 2.5 support and still support 2.x for some more time (in particular as people wouldn't rule out a 2.8 release back then, either). > Any program which must retain > Python 2.5 compatibility will not be able to use them. Yet 2.6 is a > more useful stepping stone towards 3.x than 2.5 is. I disagree fairly much (except that the 3k warnings probably *are* useful - even though I haven't ever used them myself). > So yes, it would be quite desirable to see nonlocal and as many other > 3.x features as possible backported for 2.7. And depending on how close > 2.7 manages to get, it may make sense to backport anything that doesn't > make it into 2.7 for a 2.8 release. There might not be a 2.8 release at all, though. > The 3.x transition is *hard*. Anything that makes it easier is good. I agree. I question whether backporting features actually makes the transition easier. In addition, in the *specific* case: the nonlocal keyword isn't necessary for a transition *at all*. Code that currently works without it won't need it when ported to 3.x. You may not be able to use it while maintaining 2.x and 3.x simultaneously, but you can certainly do the transition just fine without it. Regards, Martin From guido at python.org Thu Oct 22 22:59:17 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Oct 2009 13:59:17 -0700 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> Message-ID: On Thu, Oct 22, 2009 at 1:49 PM, Antoine Pitrou wrote: > Brett Cannon python.org> writes: >> I honestly don't follow that sentence. But __doc__ is special because of its >> use; documenting how to use of an object. In this case when you call >> something like help() on an instance of an object it skips the instance's >> value for __doc__ and goes straight to the defining class and stops there as >> you don't care how a subclass says to use itself as that is not what you are >> working with. > > I don't really understand how this explains the read-only __doc__. > I am talking about modifying __doc__ on a class, not on an instance. > (sure, a new-style class is also an instance of type, but still...) Antoine, it's not clear from the questions you're asking whether you've read the code in typobject.c that implements __doc__ or not. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From solipsis at pitrou.net Thu Oct 22 23:26:35 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2009 23:26:35 +0200 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> Message-ID: <1256246795.5828.0.camel@localhost> Le jeudi 22 octobre 2009 ? 13:59 -0700, Guido van Rossum a ?crit : > > I don't really understand how this explains the read-only __doc__. > > I am talking about modifying __doc__ on a class, not on an instance. > > (sure, a new-style class is also an instance of type, but still...) > > Antoine, it's not clear from the questions you're asking whether > you've read the code in typobject.c that implements __doc__ or not. Ah, well I haven't :-/ My bad. Regards Antoine. From fuzzyman at voidspace.org.uk Thu Oct 22 23:08:06 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 22 Oct 2009 22:08:06 +0100 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <4AE0C73C.9070805@v.loewis.de> References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> Message-ID: <4AE0C9B6.60109@voidspace.org.uk> Martin v. L?wis wrote: > exarkun at twistedmatrix.com wrote: > >> On 08:24 pm, martin at v.loewis.de wrote: >> >>> Mike Krell wrote: >>> >>>> Is there any possibility of backporting support for the nonlocal keyword >>>> into a 2.x release? >>>> >>> If so, only into 2.7. Can you please explain why it would be desirable >>> to do that? 2.7 will likely be the last 2.x release, so only a fairly >>> small portion of the applications would be actually able to use this (or >>> any other new feature added to 2.7): most code supporting 2.x will also >>> have to support 2.6, so the keyword won't be available to such code, >>> anyway. >>> >> For the same reason that it is desirable to backport all of the other >> changes from 3.x - because it makes the 2.x to 3.x transition easier. >> > > Hmm. Really? > > >> If Python 2.7 supports the nonlocal keyword, then 2.7 becomes that much >> better of a stepping stone towards 3.x. >> > > What use has such a stepping stone? Why, and (more importantly) when > would anybody currently supporting 2.x give up 2.6 and earlier, and > only support 2.7? And, if they chose to do so, why would they not move > the code base to 3.x right away? > From the Django roadmap for supporting 3.0, using 2.6 as a stepping stone (and if 2.7 was a *better* stepping stone then it would make it easier): http://groups.google.com/group/django-developers/msg/0888b1c8f2518059? * First half of 2009: Django 1.1 is released, with a notification that it will be the final release supporting Python 2.3. * Second half of 2009: Django 1.2 is released, drops Python 2.3 support and is the final release supporting Python 2.4. * First half of 2010: Django 1.3 is released, drops Python 2.4 support and is the final release supporting Python 2.5. * Second half of 2010: Django 1.4 is released and drops Python 2.5 support. This gets us to a situation where, about a year after the release of Python 3.0, Django will be ready to make the transition -- the only 2.x Python we'll be supporting is Python 2.6, and 2to3 plus manual effort and available supporting libraries should make it possible to also run Django on Python 3.0 either at that point or not long after. From there, 2.6 support can be dropped whenever convenient, and Django can move to running only on Python 3.x at whatever time is judged appropriate. All the best, Michael Foord > >> You've suggested that most 2.x code will have to support 2.6 and so >> won't be able to use the nonlocal keyword even if it is added to 2.7. >> > > Correct. > > >> This precise argument could be applied to all of the features in 2.6 >> which aim to bring it closer to 3.x. >> > > Not so. One of the features added to 2.6 was the 3k warning. This > warning can be used without any modification to the code. So code > can run on 2.6 and use the feature, while running unmodified on 2.5 > and earlier (not using it). > > As for actual language and library changes (such as any new future > import): there was indeed little point adding them. However, given > that the possible migration paths weren't as clear back then as they > are now, it is understandable that people considered this a viable > path. > > In addition, for 2.6, it's a bit more realistic to assume that people > might drop 2.5 support and still support 2.x for some more time (in > particular as people wouldn't rule out a 2.8 release back then, either). > > >> Any program which must retain >> Python 2.5 compatibility will not be able to use them. Yet 2.6 is a >> more useful stepping stone towards 3.x than 2.5 is. >> > > I disagree fairly much (except that the 3k warnings probably *are* > useful - even though I haven't ever used them myself). > > >> So yes, it would be quite desirable to see nonlocal and as many other >> 3.x features as possible backported for 2.7. And depending on how close >> 2.7 manages to get, it may make sense to backport anything that doesn't >> make it into 2.7 for a 2.8 release. >> > > There might not be a 2.8 release at all, though. > > >> The 3.x transition is *hard*. Anything that makes it easier is good. >> > > I agree. I question whether backporting features actually makes the > transition easier. > > In addition, in the *specific* case: the nonlocal keyword isn't > necessary for a transition *at all*. Code that currently works without > it won't need it when ported to 3.x. You may not be able to use it while > maintaining 2.x and 3.x simultaneously, but you can certainly do the > transition just fine without it. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From sturla at molden.no Thu Oct 22 23:33:22 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Oct 2009 23:33:22 +0200 Subject: [Python-Dev] GIL behaviour under Windows In-Reply-To: References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADFC9E8.1010400@molden.no> Message-ID: <4AE0CFA2.5070701@molden.no> Antoine Pitrou skrev: > This number lacks the elapsed time. 61 switches in one second is probably > enough, the same amount of switches in 10 or 20 seconds is too small (at least > for threads needing good responsivity, e.g. I/O threads). > > Also, "fair" has to take into account the average latency and its relative > stability, which is why I wrote ccbench. > Since I am a scientist and statistics interests me, let's do this properly :-) Here is a suggestion: _Py_Ticker is a circular variable. Thus, it can be transformed to an angle measured in radians, using: a = 2 * pi * _Py_Ticker / _Py_CheckInterval With simultaneous measurements of a, check interval count x, and time y (?s), we can fit the multiple regression: y = b0 + b1*cos(a) + b2*sin(a) + b3*x + err using some non-linear least squares solver. We can then extract all the statistics we need on interpreter latencies for "ticks" with and without periodic checks. On a Python setup with many missed thread switches (pthreads according to D. Beazley), we could just extend the model to take into account successful and unsccessful check intervals: y = b0 + b1*cos(a) + b2*sin(a) + b3*x1 + b4*x2 + err where x1 being successful thread switches and x2 being missed thread switches. But at least on Windows we can use the simpler model. The reason why multiple regression is needed, is that the record method of my GIL_Battle class is not called on every interpreter tick. I thus cannot measure precicesly each latency, which I could have done with a direct hook into ceval.c. So statistics to the rescue. But on the bright side, it reduces the overhead of the profiler. Would that help? Sturla Molden From sturla at molden.no Thu Oct 22 23:42:32 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Oct 2009 23:42:32 +0200 Subject: [Python-Dev] [Python-ideas] Remove GIL with CAS instructions? In-Reply-To: <536685ea0910221051p1897770xc517561dccc020f6@mail.gmail.com> References: <4ADE2AA9.4030604@molden.no> <4ADE35F0.10803@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF41F74@exchis.ccp.ad.local> <4ADF2850.1040304@arbash-meinel.com> <536685ea0910221051p1897770xc517561dccc020f6@mail.gmail.com> Message-ID: <4AE0D1C8.8050404@molden.no> Phillip Sitbon skrev: > Some of this is more low-level. I did see higher performance when > using non-Event objects, although I have not had time to follow up and > do a deeper analysis. The GIL flashing "problem" with critical > sections can very likely be rectified with a call to Sleep(0) or > YieldProcessor() for those who are worried about it. For those who don't know what Sleep(0) on Windows does: It returns the reminder of the current time-slice back to the system is a thread with equal or higher-priority is ready to run. Otherwise it does nothing. GIL flashing is a serious issue if it happens often; with the current event-based GIL on Windows, it never happens (61 cases of GIL flash in 100,000 periodic checks is as good as never). S.M. From martin at v.loewis.de Thu Oct 22 23:51:29 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 22 Oct 2009 23:51:29 +0200 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <4AE0C9B6.60109@voidspace.org.uk> References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <4AE0C9B6.60109@voidspace.org.uk> Message-ID: <4AE0D3E1.2010207@v.loewis.de> > From the Django roadmap for supporting 3.0, using 2.6 as a stepping > stone (and if 2.7 was a *better* stepping stone then it would make it > easier): > > http://groups.google.com/group/django-developers/msg/0888b1c8f2518059? Is that still a current plan? It's from November 2008. > This gets us to a situation where, about a year after the release of > Python 3.0, Django will be ready to make the transition -- the only > 2.x Python we'll be supporting is Python 2.6, and 2to3 plus manual > effort and available supporting libraries should make it possible to > also run Django on Python 3.0 either at that point or not long after. > > From there, 2.6 support can be dropped whenever convenient, and Django > can move to running only on Python 3.x at whatever time is judged > appropriate. I would claim that this specific plan was ignoring opportunities for migrating to Python 3.0. My Django port demonstrates that you can very well support 2.3 and 3.0 simultaneously: http://wiki.python.org/moin/PortingDjangoTo3k Regards, Martin From steve at pearwood.info Fri Oct 23 04:13:55 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Oct 2009 13:13:55 +1100 Subject: [Python-Dev] readonly __doc__ In-Reply-To: References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> Message-ID: <200910231313.55925.steve@pearwood.info> On Fri, 23 Oct 2009 05:39:53 am Brett Cannon wrote: > > Is __doc__ not normal due to its general underscorishness, or is it > > not normal because it isn't? > > I honestly don't follow that sentence. But __doc__ is special because > of its use; documenting how to use of an object. In this case when > you call something like help() on an instance of an object it skips > the instance's value for __doc__ and goes straight to the defining > class and stops there as you don't care how a subclass says to use > itself as that is not what you are working with. Classes don't normally inherit behaviour from their subclasses. Presumably you meant superclass. I expected __doc__ to be just like the other double-underscore objects: lookup skips the instance and goes to the class, then follows the normal rules of inheritance. Consequently I've just discovered that a few of my classes, which I assumed inherited __doc__, don't actually have a docstring. This has consequences beyond help(obj) not working as expected: doctests which I thought were running aren't. Magic behaviour like this is nasty because it breaks the Rule of Least Astonishment. I thought I understood Python's object model and it's rules of inheritance, but not only do double-underscore attributes follow a different rule for inheritance than other attributes, but __doc__ follows a different rule than other double-underscore attributes. -- Steven D'Aprano From brett at python.org Fri Oct 23 06:47:06 2009 From: brett at python.org (Brett Cannon) Date: Thu, 22 Oct 2009 21:47:06 -0700 Subject: [Python-Dev] language summit topic: issue tracker Message-ID: Another summit, another potential time to see if people want to change anything about the issue tracker. I would bring up: - Dropping Stage in favor of some keywords (e.g. 'needs unit test', 'needs docs') - Adding a freestyle text box to delineate which, if any, stdlib module is the cause of a bug and tie that into Misc/maintainers.rst; would potentially scale back the Component box -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Oct 23 07:51:03 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 23 Oct 2009 16:51:03 +1100 Subject: [Python-Dev] language summit topic: issue tracker References: Message-ID: <871vku7j5k.fsf@benfinney.id.au> Brett Cannon writes: > Another summit, another potential time to see if people want to change > anything about the issue tracker. It requires some coding, but I see OpenID authentication support to be important for lowering the barrier to getting bug reports. -- \ ?I was once walking through the forest alone and a tree fell | `\ right in front of me, and I didn't hear it.? ?Steven Wright | _o__) | Ben Finney From fperez.net at gmail.com Fri Oct 23 08:07:41 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 23 Oct 2009 06:07:41 +0000 (UTC) Subject: [Python-Dev] nonlocal keyword in 2.x? References: Message-ID: On Thu, 22 Oct 2009 12:32:37 -0300, Fabio Zadrozny wrote: > Just as a note, the nonlocal there is not a requirement... > > You can just create a mutable object there and change that object (so, > you don't need to actually rebind the object in the outer scope). > > E.g.: instead of creating a float in the context, create a list with a > single float and change the float in the list (maybe the nonlocal would > be nicer, but it's certainly still usable) Yup, that's what I meant by 'some slightly ugly solutions' in this note: http://mail.scipy.org/pipermail/ipython-dev/2009-September/005529.html in the thread that spawned those notes. nonlocal allows for this pattern to work without the ugliness of writing code like: s = [s] @somedeco def foo(): s[0] += 1 s = s[0] just to be able to 'change s' inside the foo() scope. I felt this was both obvious and ugly enough not to warrant too much explicit mention, but I probably should have left it there for the sake of completeness. Thanks for the feedback. Cheers, f ps - the above shouldn't be taken as either pro or con on the idea of nonlocal in 2.x, just a clarification on why I didn't add the mutable container trick to the original notes. From mbk.lists at gmail.com Fri Oct 23 09:30:00 2009 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 23 Oct 2009 00:30:00 -0700 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <4AE0BF83.4080903@v.loewis.de> References: <4AE0BF83.4080903@v.loewis.de> Message-ID: On Thu, Oct 22, 2009 at 1:24 PM, "Martin v. L?wis" wrote: > Can you please explain why it would be desirable > to [backport nonlocal]? 2.7 will likely be the last 2.x release, so only a > fairly > small portion of the applications would be actually able to use this (or > any other new feature added to 2.7): most code supporting 2.x will also > have to support 2.6, so the keyword won't be available to such code, > anyway. > Others have explained the rationale for the backport, so I won't bother repeating those arguments. I understand your point about code supporting 2.6, but as you say, that applies to any new features being added in 2.7. I'm therefore confused as to what the rationale for a 2.7 release is. It's a bump in minor release number, so it's purpose is to add new features, correct? Could someone please explain what new features are currently envisioned for 2.7? Why would those make the cut but a not backport of nonlocal? Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Fri Oct 23 10:15:21 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Oct 2009 10:15:21 +0200 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: References: <4AE0BF83.4080903@v.loewis.de> Message-ID: <4AE16619.2030100@v.loewis.de> > Others have explained the rationale for the backport, so I won't bother > repeating those arguments. > > I understand your point about code supporting 2.6, but as you say, that > applies to any new features being added in 2.7. I'm therefore confused > as to what the rationale for a 2.7 release is. It's a bump in minor > release number, so it's purpose is to add new features, correct? That, but not only - it's purpose is also to allow for certain incompatible changes, and to arrive at a version that will be maintained until 2015 (wrt. security fixes). > Could someone please explain what new features are currently envisioned > for 2.7? Read through Misc/NEWS for details of what has been added already. > Why would those make the cut but a not backport of nonlocal? Just to name a few changes: - Issue #1655: Make imaplib IPv6-capable. Patch by Derek Morr. - Issue #4915: Port sysmodule to Windows CE. - Issue #6101: A new opcode, SETUP_WITH, has been added to speed up the with statement and correctly lookup the __enter__ and __exit__ special methods. - Issue #1616979: Added the cp720 (Arabic DOS) encoding. In all of these cases, in order to use the new feature, no change to Python code will be necessary: the feature just becomes available transparently. This is the kind of new features I like to see in 2.7. The other kind (i.e. features that do require application changes in order to use them) I find questionable. Regards, Martin From ncoghlan at gmail.com Fri Oct 23 11:46:00 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2009 19:46:00 +1000 Subject: [Python-Dev] readonly __doc__ In-Reply-To: <200910231313.55925.steve@pearwood.info> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <198E579C-0FFF-4CF7-AC5A-9D0AB2CEA509@gmail.com> <200910231313.55925.steve@pearwood.info> Message-ID: <4AE17B58.1060306@gmail.com> Steven D'Aprano wrote: > On Fri, 23 Oct 2009 05:39:53 am Brett Cannon wrote: > >>> Is __doc__ not normal due to its general underscorishness, or is it >>> not normal because it isn't? >> I honestly don't follow that sentence. But __doc__ is special because >> of its use; documenting how to use of an object. In this case when >> you call something like help() on an instance of an object it skips >> the instance's value for __doc__ and goes straight to the defining >> class and stops there as you don't care how a subclass says to use >> itself as that is not what you are working with. > > Classes don't normally inherit behaviour from their subclasses. > Presumably you meant superclass. > > I expected __doc__ to be just like the other double-underscore objects: > lookup skips the instance and goes to the class, then follows the > normal rules of inheritance. Consequently I've just discovered that a > few of my classes, which I assumed inherited __doc__, don't actually > have a docstring. This has consequences beyond help(obj) not working as > expected: doctests which I thought were running aren't. > > Magic behaviour like this is nasty because it breaks the Rule of Least > Astonishment. I thought I understood Python's object model and it's > rules of inheritance, but not only do double-underscore attributes > follow a different rule for inheritance than other attributes, but > __doc__ follows a different rule than other double-underscore > attributes. This actually breaks a recipe I recently suggested in python-ideas, where I recommended inheriting from a namedtuple instance in order to insert additional argument validity checks in __new__. Probably not invalid for it to break there though - in the case of that recipe, the default docstring from the parent class really should be replaced with one that details the additional constraints being placed on the arguments. Then again, the parent class docstring wouldn't have been wrong - merely incomplete. The point you mentioned about doctests silently failing to run strikes me as a pretty major glitch due to this behaviour though. Having a base class that defines the doctests to run in its docstring and then creating subclasses to specialise the test execution sounds like a perfectly reasonable use case to me (even though I personally tend to write unittest based test rather than doctest based ones). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Fri Oct 23 11:49:06 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2009 19:49:06 +1000 Subject: [Python-Dev] language summit topic: issue tracker In-Reply-To: References: Message-ID: <4AE17C12.1080007@gmail.com> Brett Cannon wrote: > Another summit, another potential time to see if people want to change > anything about the issue tracker. I would bring up: > > - Dropping Stage in favor of some keywords (e.g. 'needs unit test', > 'needs docs') > - Adding a freestyle text box to delineate which, if any, stdlib module > is the cause of a bug and tie that into Misc/maintainers.rst; would > potentially scale back the Component box +lots on adding a module field (independent of automatically adding maintainers to the nosy list, it would assist in "I just did a major cleanup of module X, are there any old bugs I can kill off"). Of course, it will take a while for the field to be filled in on existing issues, but even having it be possible would be very nice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From w.richert at gmx.net Fri Oct 23 11:32:45 2009 From: w.richert at gmx.net (Willi Richert) Date: Fri, 23 Oct 2009 11:32:45 +0200 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it Message-ID: <200910231132.45504.w.richert@gmx.net> Hi, recently I wrote an algorithm, in which very often I had to get an arbitrary element from a set without removing it. Three possibilities came to mind: 1. x = some_set.pop() some_set.add(x) 2. for x in some_set: break 3. x = iter(some_set).next() Of course, the third should be the fastest. It nevertheless goes through all the iterator creation stuff, which costs some time. I wondered, why the builtin set does not provide a more direct and efficient way for retrieving some element without removing it. Is there any reason for this? I imagine something like x = some_set.get() or x = some_set.pop(False) and am thinking about providing a patch against setobject.c (preferring the .get() solution being a stripped down pop()). Before, I would like to know whether I have overlooked something or whether this can be done in an already existing way. Thanks, wr From w.richert at gmx.net Fri Oct 23 12:23:08 2009 From: w.richert at gmx.net (Willi Richert) Date: Fri, 23 Oct 2009 12:23:08 +0200 Subject: [Python-Dev] First shot at some_set.get() Message-ID: <200910231223.08470.w.richert@gmx.net> Hi, here is the first shot to provide a faster means of retrieving an arbitrary element from a set without removing it. The times for ========================= from timeit import * stat1 = "for i in xrange(100): iter(s).next()" stat2 = "for i in xrange(100): s.get()" for stat in [stat1, stat2]: t = Timer(stat, setup="s=set(range(10000))") # outside the try/except try: print t.timeit(number=1000) except: t.print_exc() ========================= are stat1: 0.0451760292053 stat2: 0.0148510932922 The patch is attached. Feel free to criticize. I would love to see something like that in the standard Python interpreter. Regards, wr -------------- next part -------------- A non-text attachment was scrubbed... Name: setobject_get.patch Type: text/x-patch Size: 2211 bytes Desc: not available URL: From phd at phd.pp.ru Fri Oct 23 12:37:09 2009 From: phd at phd.pp.ru (Oleg Broytman) Date: Fri, 23 Oct 2009 14:37:09 +0400 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <200910231223.08470.w.richert@gmx.net> References: <200910231223.08470.w.richert@gmx.net> Message-ID: <20091023103709.GA29267@phd.pp.ru> On Fri, Oct 23, 2009 at 12:23:08PM +0200, Willi Richert wrote: > The patch is attached. Patches should be put to the issue tracker. Thank you. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From steve at pearwood.info Fri Oct 23 12:54:52 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Oct 2009 21:54:52 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910231132.45504.w.richert@gmx.net> References: <200910231132.45504.w.richert@gmx.net> Message-ID: <200910232154.53188.steve@pearwood.info> On Fri, 23 Oct 2009 08:32:45 pm Willi Richert wrote: > Hi, > > recently I wrote an algorithm, in which very often I had to get an > arbitrary element from a set without removing it. > > Three possibilities came to mind: ... > Of course, the third should be the fastest. If you need one or more items chosen randomly without replacement, use a list: L = list(some_set) x = random.choice(L) If you don't need a randomly chosen item, merely an arbitrary item, you can still use a list but avoid the call to random.choice: x = list(some_set)[0] > It nevertheless goes > through all the iterator creation stuff, which costs some time. I > wondered, why the builtin set does not provide a more direct and > efficient way for retrieving some element without removing it. Is > there any reason for this? I can see it being useful to iterate over the entire set, without removing anything. I can see it being useful to pop an arbitrary item, and remove it. But I can't see the use for retrieving an arbitrary item, leaving it in the set. What do you use this for? > I imagine something like > > x = some_set.get() > > or > > x = some_set.pop(False) > > and am thinking about providing a patch against setobject.c > (preferring the .get() solution being a stripped down pop()). -1 on .pop(flag) +0 on .get() -- Steven D'Aprano From steve at pearwood.info Fri Oct 23 13:09:50 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Oct 2009 22:09:50 +1100 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <20091023103709.GA29267@phd.pp.ru> References: <200910231223.08470.w.richert@gmx.net> <20091023103709.GA29267@phd.pp.ru> Message-ID: <200910232209.50723.steve@pearwood.info> On Fri, 23 Oct 2009 09:37:09 pm Oleg Broytman wrote: > On Fri, Oct 23, 2009 at 12:23:08PM +0200, Willi Richert wrote: > > The patch is attached. > > Patches should be put to the issue tracker. Thank you. Is there any point? Even if accepted, it's too late to make it into 3.1, and with the overwhelming approval for a moratorium on changes to built-ins, it is likely to just sit in the tracker, forgotten, until 2013 or later. How likely is it that the patch will still be applicable? -- Steven D'Aprano From ubershmekel at gmail.com Fri Oct 23 13:19:18 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Fri, 23 Oct 2009 13:19:18 +0200 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <200910232209.50723.steve@pearwood.info> References: <200910231223.08470.w.richert@gmx.net> <20091023103709.GA29267@phd.pp.ru> <200910232209.50723.steve@pearwood.info> Message-ID: <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> On Fri, Oct 23, 2009 at 1:09 PM, Steven D'Aprano wrote: > Is there any point? Even if accepted, it's too late to make it into 3.1, > and with the overwhelming approval for a moratorium on changes to > built-ins, it is likely to just sit in the tracker, forgotten, until > 2013 or later. How likely is it that the patch will still be > applicable? +1 on throwing it away completely even if it's a good idea. I suggest Willi go invent a new language (and hope for it to become popular :-) if he wants to experiment. --yuv From kristjan at ccpgames.com Fri Oct 23 13:48:25 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Fri, 23 Oct 2009 11:48:25 +0000 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <4AE0C54F.20705@gmail.com> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF4213E@exchis.ccp.ad.local> <4AE0C54F.20705@gmail.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF4246E@exchis.ccp.ad.local> Thanks, I'll take a look in that direction. > -----Original Message----- > From: Nick Coghlan [mailto:ncoghlan at gmail.com] > I've even played with using Kalman filtering to do it... The idea is > > to use the low frequency timer to apply correction coefficients to > > the high frequency timer, yet keep the flow of time smooth (no > > backwards jumps because of corrections.). An optimal solution has so > > far eluded me. > > That sounds very similar to the problem spec for system time > corrections > in Network Time Protocol client implementations. Perhaps the time > drifting algorithms in the NTP specs are relevant? Or are they too slow > to correct discrepancies? K From solipsis at pitrou.net Fri Oct 23 14:24:46 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Oct 2009 12:24:46 +0000 (UTC) Subject: [Python-Dev] language summit topic: issue tracker References: Message-ID: Le Thu, 22 Oct 2009 21:47:06 -0700, Brett Cannon a ?crit?: > > - Dropping Stage in favor of some keywords (e.g. 'needs unit test', > 'needs docs') What would it bring? We don't have a very strict process and the current "stage" looks sufficient to me. Saying that unit tests or docs are lacking is part of the review and doesn't need a specific indicator IMO. Besides, the more keywords there are, the messier it is. From ncoghlan at gmail.com Fri Oct 23 14:47:22 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2009 22:47:22 +1000 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> References: <200910231223.08470.w.richert@gmx.net> <20091023103709.GA29267@phd.pp.ru> <200910232209.50723.steve@pearwood.info> <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> Message-ID: <4AE1A5DA.40603@gmail.com> Yuvgoog Greenle wrote: > On Fri, Oct 23, 2009 at 1:09 PM, Steven D'Aprano wrote: >> Is there any point? Even if accepted, it's too late to make it into 3.1, >> and with the overwhelming approval for a moratorium on changes to >> built-ins, it is likely to just sit in the tracker, forgotten, until >> 2013 or later. How likely is it that the patch will still be >> applicable? > > > +1 on throwing it away completely even if it's a good idea. I suggest > Willi go invent a new language (and hope for it to become popular :-) > if he wants to experiment. Careful folks - these kinds of throwaway comments may be clearly tongue in cheek for anyone following the moratorium discussion on python-ideas, but will be justifiably confusing to anyone else, especially newcomers. Willi - Oleg is right that patches should go on the issue tracker. They tend to get lost if they only exist on the mailing list. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From skip at pobox.com Fri Oct 23 15:00:25 2009 From: skip at pobox.com (skip at pobox.com) Date: Fri, 23 Oct 2009 08:00:25 -0500 Subject: [Python-Dev] language summit topic: issue tracker In-Reply-To: References: Message-ID: <19169.43241.70138.170239@montanaro.dyndns.org> Brett> Another summit, another potential time to see if people want to Brett> change anything about the issue tracker. I have no idea how hard this would be to implement and won't be at the language summit to formally present the idea, but it seems to me that some integration between the issue tracker and Rietveld would be beneficial. If someone attaches a patch to an issue the current next step is essentially a code review without the benefits provided by a code review tool. I'd envision a bit of workflow like this: * A patch is attached to an issue. * The user clicks the "Create Review" button. (Maybe not all patches require review?) * This generates a review request in Rietveld with all on the nosy list invited as reviewers. (Or should this be a side effect of attaching a patch?) * The "needs review" keyword is added to the selected keywords. * A link to the review request is added as a comment to the issue so other people not on the nosy list can evaluate the patch. * If an updated diff is uploaded the review request would be updated. That might necessitate adding a "Replace Patch" button next to all uploaded patches instead of adding a new one then deleting a previous one. Skip From stephen at xemacs.org Fri Oct 23 16:18:39 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 23 Oct 2009 23:18:39 +0900 Subject: [Python-Dev] language summit topic: issue tracker In-Reply-To: References: Message-ID: <87tyxqb3cw.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > Besides, the more keywords there are, the messier it is. That's what I've found in the XEmacs tracker. Keywords are a reasonable way (in the context of the Roundup implementation) to test new classifications before going to the effort of messing with the page templates. But if you don't think "stage" is needed, I'd say drop it entirely rather than demote it to keywords. Keywords themselves are rather confusing to non-tracker-hackers, anyway. A lot of people seem to think anything that occurs to them should be allowed. That's not true in Roundup, you need to register a keyword before using it. In the XEmacs tracker I don't allow non-committers to do that. I'm open to changing that policy, but as yet I haven't seen a keyword suggestion from a non-committer that either (1) helps the committers to do their work or (2) seems likely to help users find relevant bugs. The suggestions are always of the form "it would make the interface more complete/consistent." So I'm going to maintain editorial control for now.... From van.lindberg at gmail.com Fri Oct 23 17:49:40 2009 From: van.lindberg at gmail.com (VanL) Date: Fri, 23 Oct 2009 10:49:40 -0500 Subject: [Python-Dev] Volunteer needed to organize summits In-Reply-To: <20091020114325.GB6355@amk-desktop.matrixgroup.net> References: <20091020114325.GB6355@amk-desktop.matrixgroup.net> Message-ID: Have you had any bites? Thanks, Van From status at bugs.python.org Fri Oct 23 18:07:40 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 23 Oct 2009 18:07:40 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20091023160740.7468A7864F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (10/16/09 - 10/23/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2474 open (+27) / 16541 closed (+16) / 19015 total (+43) Open issues with patches: 987 Average duration of open issues: 673 days. Median duration of open issues: 428 days. Open Issues Breakdown open 2436 (+27) pending 37 ( +0) Issues Created Or Reopened (43) _______________________________ regrtest -j sometimes fails if output gets written to stderr by 10/16/09 CLOSED http://bugs.python.org/issue7151 created r.david.murray patch, patch urllib2.build_opener() skips ProxyHandler 10/16/09 http://bugs.python.org/issue7152 created barry add "start" arg to max and min functions 10/16/09 CLOSED http://bugs.python.org/issue7153 created phr urllib.request system proxy configuration lookup broken for OS X 10/16/09 http://bugs.python.org/issue7154 created ned.deily urllib2 and python 3 urllib do not document default use of syste 10/16/09 CLOSED http://bugs.python.org/issue7155 created ned.deily patch curses can't find _curses 10/17/09 CLOSED http://bugs.python.org/issue7156 created Archon Fix Download Current Documentation link 10/17/09 http://bugs.python.org/issue7157 created ryles os.path.basename/split fails 10/17/09 CLOSED http://bugs.python.org/issue7158 created kuiper Urllib2 authentication memory. 10/17/09 http://bugs.python.org/issue7159 created bradobro Crash when returning a 64-bit char pointer in Python 2.6.3 ctype 10/17/09 CLOSED http://bugs.python.org/issue7160 created creachadair raise of SyntaxError in codeop was ported incorrectly to Py3 10/17/09 CLOSED http://bugs.python.org/issue7161 created Trundle patch 2to3 does not convert __builtins__.file 10/17/09 http://bugs.python.org/issue7162 created joe.amenta IDLE suppresses sys.stdout.write() return value 10/17/09 http://bugs.python.org/issue7163 created tjreedy pickle test failure after test_imp/test_import (_make_stat_resul 10/18/09 CLOSED http://bugs.python.org/issue7164 created r.david.murray xmlrpc.server assumes sys.stdout will have a buffer attribute 10/18/09 http://bugs.python.org/issue7165 created ncoghlan patch IDLE (python 3.1.1) syntax coloring for b'bytestring' and u'unic 10/18/09 http://bugs.python.org/issue7166 created lieryan easy Smarter FTP passive mode 10/18/09 http://bugs.python.org/issue7167 created pitrou patch Document PyFloat_AsString and PyFloat_AsReprString as deprecated 10/19/09 CLOSED http://bugs.python.org/issue7168 created eric.smith patch zipfile leaves a file handle open if file is zero size 10/19/09 http://bugs.python.org/issue7169 created skelker subclasses of (some) built-in types are not weakref-able 10/19/09 CLOSED http://bugs.python.org/issue7170 created stutzbach Add inet_ntop and inet_pton support for Windows 10/19/09 http://bugs.python.org/issue7171 created jaraco BaseHTTPServer.BaseHTTPRequestHandler.responses[405] has a small 10/19/09 http://bugs.python.org/issue7172 created dalke Cython compiler run crashes CPython 3.1.1 and 3.2 10/20/09 http://bugs.python.org/issue7173 created scoder patch modeule: queue class: PriorityQueue 10/20/09 CLOSED http://bugs.python.org/issue7174 created asuiu unify pydistutils.cfg and distutils.cfg and use .local 10/20/09 http://bugs.python.org/issue7175 created tarek sum() doesn't work for lists. 10/20/09 CLOSED http://bugs.python.org/issue7176 created oggust Unclear warning for subprocess.call 10/20/09 http://bugs.python.org/issue7177 created azumanga open function's buffering parameter is not completely doc'ed 10/20/09 CLOSED http://bugs.python.org/issue7178 created kuhnsjohn Unportable test(1) construct 10/21/09 http://bugs.python.org/issue7179 created wiz patch "pydoc -k" can generate AttributeError on Mac OS X 10/21/09 http://bugs.python.org/issue7180 created kmgrant No logging when two loggers with same qualname 10/21/09 CLOSED http://bugs.python.org/issue7181 created cristiroma For non-debug builds, the cygwinccompiler.py should define NDEBU 10/21/09 http://bugs.python.org/issue7182 created stutzbach did 2.6.3 regress for some uses of the __doc__ property? 10/21/09 http://bugs.python.org/issue7183 created zooko build failures on Snow Leopard 10/22/09 http://bugs.python.org/issue7184 created richard csv reader utf-8 BOM error 10/22/09 http://bugs.python.org/issue7185 created W00D00 Document specialness of __doc__, and possibly other "special" at 10/22/09 http://bugs.python.org/issue7186 created ssteiner importlib/_bootstrap.py write_bytecode raises an IOError if it c 10/22/09 http://bugs.python.org/issue7187 created dmalcolm patch optionxform documentation confusing 10/23/09 CLOSED http://bugs.python.org/issue7188 created loewis struct.calcsize returns strange size 10/23/09 CLOSED http://bugs.python.org/issue7189 created igor.mikushkin TCP/IP? 10/23/09 http://bugs.python.org/issue7190 created jlr2383 Odd behaviour with zlib.decompressobj optional parameter "wbits" 10/23/09 http://bugs.python.org/issue7191 created pythonhacker webbrowser.get("firefox") does not work on Mac with installed Fi 10/23/09 http://bugs.python.org/issue7192 created dalke patch Popen blocks program from another thread 10/23/09 http://bugs.python.org/issue7193 created dgriff1 Issues Now Closed (41) ______________________ fromfd() and dup() for _socket on WIndows 716 days http://bugs.python.org/issue1378 planders patch GC optimization: don't track simple tuples and dicts 303 days http://bugs.python.org/issue4688 esam patch optimize bytecode for conditional branches 299 days http://bugs.python.org/issue4715 esam patch Bugs in _ssl object read() when a buffer is specified 278 days http://bugs.python.org/issue4967 pitrou patch "in" expression falls back to __iter__ before __getitem__ 122 days http://bugs.python.org/issue6324 georg.brandl Printing the 'The Python Tutorial' 76 days http://bugs.python.org/issue6670 brimac uuid creates zombies 42 days http://bugs.python.org/issue6882 eric.smith patch Metaclass doc (Ref 3.3.3) errors 36 days http://bugs.python.org/issue6927 georg.brandl Getopt documentation ambiguity 29 days http://bugs.python.org/issue6977 georg.brandl codecs error handlers lack documentation 20 days http://bugs.python.org/issue7035 georg.brandl Doc/reference/datamodel: Slots description needs update 20 days http://bugs.python.org/issue7036 georg.brandl patch urllib.urlopen crashes when used on Mac OS 10.6.1 through a prox 15 days http://bugs.python.org/issue7044 ronaldoussoren No docs for module 'IN' 17 days http://bugs.python.org/issue7062 georg.brandl Python 2.6.3 / setuptools 0.6c9: extension module builds fail wi 6 days http://bugs.python.org/issue7064 ned.deily Suggest better documentation of 2 versus 3 on 3.1.1 site. 17 days http://bugs.python.org/issue7067 srl locale.strxfrm raises MemoryError 12 days http://bugs.python.org/issue7080 pitrou patch New (in 2.6) functions in signal module are not documented as un 13 days http://bugs.python.org/issue7088 georg.brandl Problems building pyhton from source on Snow Leopard (Mac OS X 7 days http://bugs.python.org/issue7102 ronaldoussoren regrtest -j fails when tests write to stderr 4 days http://bugs.python.org/issue7127 r.david.murray easy test_ssl failure 5 days http://bugs.python.org/issue7133 pitrou patch Socket documentation not updated 7 days http://bugs.python.org/issue7137 georg.brandl Remove WITHOUT_COMPLEX from 3.x trunk 3 days http://bugs.python.org/issue7147 skip.montanaro patch, patch, easy, needs review 2.6.4rc1 regression: test_urllib2 fails on OS X with UnboundLoca 3 days http://bugs.python.org/issue7149 ned.deily regrtest -j sometimes fails if output gets written to stderr by 3 days http://bugs.python.org/issue7151 r.david.murray patch, patch add "start" arg to max and min functions 2 days http://bugs.python.org/issue7153 rhettinger urllib2 and python 3 urllib do not document default use of syste 1 days http://bugs.python.org/issue7155 orsenthil patch curses can't find _curses 6 days http://bugs.python.org/issue7156 georg.brandl os.path.basename/split fails 0 days http://bugs.python.org/issue7158 ezio.melotti Crash when returning a 64-bit char pointer in Python 2.6.3 ctype 2 days http://bugs.python.org/issue7160 creachadair raise of SyntaxError in codeop was ported incorrectly to Py3 0 days http://bugs.python.org/issue7161 benjamin.peterson patch pickle test failure after test_imp/test_import (_make_stat_resul 0 days http://bugs.python.org/issue7164 ncoghlan Document PyFloat_AsString and PyFloat_AsReprString as deprecated 1 days http://bugs.python.org/issue7168 eric.smith patch subclasses of (some) built-in types are not weakref-able 1 days http://bugs.python.org/issue7170 georg.brandl modeule: queue class: PriorityQueue 0 days http://bugs.python.org/issue7174 r.david.murray sum() doesn't work for lists. 0 days http://bugs.python.org/issue7176 rhettinger open function's buffering parameter is not completely doc'ed 0 days http://bugs.python.org/issue7178 benjamin.peterson No logging when two loggers with same qualname 0 days http://bugs.python.org/issue7181 vinay.sajip optionxform documentation confusing 0 days http://bugs.python.org/issue7188 georg.brandl struct.calcsize returns strange size 0 days http://bugs.python.org/issue7189 mark.dickinson os.path, %HOME% set: realpath contradicts expanduser on '~' 993 days http://bugs.python.org/issue1646838 ezio.melotti PyThreadState_SetAsyncExc and the main thread 788 days http://bugs.python.org/issue1779233 pitrou Top Issues Most Discussed (10) ______________________________ 10 add ftp-tls support to ftplib - RFC 4217 623 days open http://bugs.python.org/issue2054 9 did 2.6.3 regress for some uses of the __doc__ property? 2 days open http://bugs.python.org/issue7183 9 unify pydistutils.cfg and distutils.cfg and use .local 3 days open http://bugs.python.org/issue7175 9 Add gamma function, error functions and other C99 math.h functi 465 days open http://bugs.python.org/issue3366 6 add "start" arg to max and min functions 2 days closed http://bugs.python.org/issue7153 6 locale.strxfrm raises MemoryError 12 days closed http://bugs.python.org/issue7080 5 subclasses of (some) built-in types are not weakref-able 1 days closed http://bugs.python.org/issue7170 5 os.path.basename/split fails 0 days closed http://bugs.python.org/issue7158 5 2.6.4rc1 regression: test_urllib2 fails on OS X with UnboundLoc 3 days closed http://bugs.python.org/issue7149 5 deprecated conversion from string constant to char * 33 days open http://bugs.python.org/issue6952 From fuzzyman at voidspace.org.uk Fri Oct 23 18:49:58 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 23 Oct 2009 17:49:58 +0100 Subject: [Python-Dev] Volunteer needed to organize summits In-Reply-To: References: <20091020114325.GB6355@amk-desktop.matrixgroup.net> Message-ID: <4AE1DEB6.7070303@voidspace.org.uk> VanL wrote: > Have you had any bites? > I'm going to help Andrew with the invites and working out agendas. He's sent me a bunch of stuff to get me started. Michael > Thanks, > > Van > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From algorias at gmail.com Fri Oct 23 19:04:33 2009 From: algorias at gmail.com (Vitor Bosshard) Date: Fri, 23 Oct 2009 14:04:33 -0300 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910231132.45504.w.richert@gmx.net> References: <200910231132.45504.w.richert@gmx.net> Message-ID: <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> 2009/10/23 Willi Richert : > Hi, > > recently I wrote an algorithm, in which very often I had to get an arbitrary > element from a set without removing it. > > Three possibilities came to mind: > > 1. > x = some_set.pop() > some_set.add(x) > > 2. > for x in some_set: > ? ? ? ?break > > 3. > x = iter(some_set).next() > > > Of course, the third should be the fastest. It nevertheless goes through all > the iterator creation stuff, which costs some time. I wondered, why the builtin > set does not provide a more direct and efficient way for retrieving some element > without removing it. Is there any reason for this? > > I imagine something like > > x = some_set.get() > I see this as being useful for frozensets as well, where you can't get an arbitrary element easily due to the obvious lack of .pop(). I ran into this recently, when I had a frozenset that I knew had 1 element (it was the difference between 2 other sets), but couldn't get to that element easily (get the pun?) From sturla at molden.no Fri Oct 23 19:18:46 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 23 Oct 2009 19:18:46 +0200 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F098FF4246E@exchis.ccp.ad.local> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF4213E@exchis.ccp.ad.local> <4AE0C54F.20705@gmail.com> <930F189C8A437347B80DF2C156F7EC7F098FF4246E@exchis.ccp.ad.local> Message-ID: <4AE1E576.5000903@molden.no> Kristj?n Valur J?nsson skrev: > Thanks, I'll take a look in that direction. > > I have a suggestion, forgive me if I am totally ignorant. :-) Sturla Molden #include union __reftime { double us; __int64 bits; }; static volatile union __reftime __ref_perftime, __ref_filetime; double clock() { __int64 cnt, hz, init; double us; union __reftime ref_filetime; union __reftime ref_perftime; for (;;) { ref_filetime.bits = __ref_filetime.bits; ref_perftime.bits = __ref_perftime.bits; if(!QueryPerformanceCounter((LARGE_INTEGER*)&cnt)) goto error; if(!QueryPerformanceFrequency((LARGE_INTEGER*)&hz)) goto error; us = ref_filetime.us + ((double)(1000000*cnt)/(double)hz - ref_perftime.us); /* verify that values did not change */ init = InterlockedCompareExchange64((LONGLONG*)&__ref_filetime.bits, (LONGLONG)ref_filetime.bits, (LONGLONG)ref_filetime.bits); if (init != ref_filetime.bits) continue; init = InterlockedCompareExchange64((LONGLONG*)&__ref_perftime.bits, (LONGLONG)ref_perftime.bits, (LONGLONG)ref_perftime.bits); if (init == ref_perftime.bits) break; } return us; error: /* only if there is no performance counter */ return -1; /* or whatever */ } int periodic_reftime_check() { /* call this function at regular intervals, e.g. once every second */ __int64 cnt1, cnt2, hz; FILETIME systime; double ft; if(!QueryPerformanceFrequency((LARGE_INTEGER*)&hz)) goto error; if(!QueryPerformanceCounter((LARGE_INTEGER*)&cnt1)) goto error; GetSystemTimeAsFileTime(&systime); __ref_filetime.us = (double)(((((__int64)(systime.dwHighDateTime)) << 32) | ((__int64)(systime.dwLowDateTime)))/10); if(!QueryPerformanceCounter((LARGE_INTEGER*)&cnt2)) goto error; __ref_perftime.us = 500000*(cnt1 + cnt2)/((double)hz); return 0; error: /* only if there is no performance counter */ return -1; } From john.arbash.meinel at gmail.com Fri Oct 23 19:25:48 2009 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Fri, 23 Oct 2009 12:25:48 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> Message-ID: <4AE1E71C.7020806@gmail.com> Vitor Bosshard wrote: > 2009/10/23 Willi Richert : >> Hi, >> >> recently I wrote an algorithm, in which very often I had to get an arbitrary >> element from a set without removing it. >> >> Three possibilities came to mind: >> >> 1. >> x = some_set.pop() >> some_set.add(x) >> >> 2. >> for x in some_set: >> break >> >> 3. >> x = iter(some_set).next() >> >> >> Of course, the third should be the fastest. It nevertheless goes through all >> the iterator creation stuff, which costs some time. I wondered, why the builtin >> set does not provide a more direct and efficient way for retrieving some element >> without removing it. Is there any reason for this? >> >> I imagine something like >> >> x = some_set.get() >> > > > I see this as being useful for frozensets as well, where you can't get > an arbitrary element easily due to the obvious lack of .pop(). I ran > into this recently, when I had a frozenset that I knew had 1 element > (it was the difference between 2 other sets), but couldn't get to that > element easily (get the pun?) So in my testing (2) was actually the fastest. I assumed because .next() was a function call overhead, while: for x in some_set: break Was evaluated inline. It probably still has to call PyObject_GetIter, however it doesn't have to create a stack frame for it. This is what "timeit" tells me. All runs are of the form: python -m timeit -s "s = set([10])" ... 0.101us "for x in s: break; x" 0.130us "for x in s: pass; x" 0.234us -s "n = next; i = iter" "x = n(i(s)); x" 0.248us "x = next(iter(s)); x" 0.341us "x = iter(s).next(); x" So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x faster than (iter(s).next()). I was pretty surprised that it was 30% faster than "for x in s: pass". I assume it has something to do with a potential "else:" statement? Note that all of these are significantly < 1us. So this only matters if it is something you are doing often. I don't know your specific timings, but I would guess that: for x in s: break Is actually going to be faster than your s.get() Primarily because s.get() requires an attribute lookup. I would think your version might be faster for: stat2 = "g = s.get; for i in xrange(100): g()" However, that is still a function call, which may be treated differently by the interpreter than the for:break loop. I certainly suggest you try it and compare. John =:-> From kmtracey at gmail.com Fri Oct 23 20:05:49 2009 From: kmtracey at gmail.com (Karen Tracey) Date: Fri, 23 Oct 2009 14:05:49 -0400 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <4AE0D3E1.2010207@v.loewis.de> References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <4AE0C9B6.60109@voidspace.org.uk> <4AE0D3E1.2010207@v.loewis.de> Message-ID: On Thu, Oct 22, 2009 at 5:51 PM, "Martin v. L?wis" wrote: > > From the Django roadmap for supporting 3.0, using 2.6 as a stepping > > stone (and if 2.7 was a *better* stepping stone then it would make it > > easier): > > > > http://groups.google.com/group/django-developers/msg/0888b1c8f2518059 > ? > > Is that still a current plan? It's from November 2008. > > Django 1.1 came out in 2H09, not 1H09, and Django 1.2 is now looking to come out in 1H10, not 2H09, so the dates in that note have slipped out by 3-6 months. (The labeling of some features as must-have for a release has been dropped for 1.2, so as to hopefully prevent the kind of slip seen with 1.1.) Django 1.2 is still scheduled to drop Python 2.3 support. I think it's too early to say whether Python 2.4 support will be dropped with 1.3, nor do I have any good idea when supporting 3.x will become a priority. Karen -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 23 20:13:33 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Oct 2009 19:13:33 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE1E71C.7020806@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE1E71C.7020806@gmail.com> Message-ID: <79990c6b0910231113p11adc0e7g970116822090f9d6@mail.gmail.com> 2009/10/23 John Arbash Meinel : > I was pretty surprised that it was 30% faster than "for x in s: pass". I > assume it has something to do with a potential "else:" statement? I'd imagine it's actually because it has to call next() a second time and deal with the StopIteration exception - the loop has to end normally, whereas the break form exits prematurely. Paul. From stefan_ml at behnel.de Fri Oct 23 21:01:23 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 23 Oct 2009 21:01:23 +0200 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910231132.45504.w.richert@gmx.net> References: <200910231132.45504.w.richert@gmx.net> Message-ID: Willi Richert wrote: > recently I wrote an algorithm, in which very often I had to get an arbitrary > element from a set without removing it. See this discussion: http://comments.gmane.org/gmane.comp.python.ideas/5606 Stefan From tjreedy at udel.edu Fri Oct 23 21:04:12 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 23 Oct 2009 15:04:12 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE1E71C.7020806@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE1E71C.7020806@gmail.com> Message-ID: John Arbash Meinel wrote: > So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x > faster than (iter(s).next()). > I was pretty surprised that it was 30% faster than "for x in s: pass". I > assume it has something to do with a potential "else:" statement? for x in s: pass iterates through *all* the elements in s and leaves x bound to the arbritrary *last* one instead of the arbitrary *first* one. For a large set, this would be a lot slower, not just a little. fwiw, I think the use case for this is sufficiently rare that it does not need a separate method just for this purpose. tjr From barry at python.org Fri Oct 23 21:16:19 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 23 Oct 2009 15:16:19 -0400 Subject: [Python-Dev] Bug 7183 and Python 2.6.4 In-Reply-To: <0272128F-3703-411C-BA1B-2F7BF8AC63A4@fuhm.net> References: <92004188-2DD4-4708-8737-0AFD98201B65@python.org> <1afaf6160910220747v41b441f6h69e3778b906dc96c@mail.gmail.com> <25FBFDCF-7D07-464E-8117-12AA0C380484@python.org> <1256241223.2929.56.camel@lifeless-64> <0272128F-3703-411C-BA1B-2F7BF8AC63A4@fuhm.net> Message-ID: <484091E4-AC53-4314-BFDB-7067504AC8F1@python.org> While I think there is some risk of exposure on this bug, I haven't yet heard a compelling argument for delaying 2.6.4 final for it. I think we should go ahead and do the release this Sunday as planned with the code from 2.6.4rc2. If you strongly disagree, please private email me. Otherwise... there's always 2.6.5! :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From john.arbash.meinel at gmail.com Fri Oct 23 22:11:57 2009 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Fri, 23 Oct 2009 15:11:57 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE1E71C.7020806@gmail.com> Message-ID: <4AE20E0D.4050405@gmail.com> Terry Reedy wrote: > John Arbash Meinel wrote: >> So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x >> faster than (iter(s).next()). >> I was pretty surprised that it was 30% faster than "for x in s: pass". I >> assume it has something to do with a potential "else:" statement? > > for x in s: pass > > iterates through *all* the elements in s and leaves x bound to the > arbritrary *last* one instead of the arbitrary *first* one. For a large > set, this would be a lot slower, not just a little. > > fwiw, I think the use case for this is sufficiently rare that it does > not need a separate method just for this purpose. > > tjr The point of my test was that it was a set with a *single* item, and 'break' was 30% faster than 'pass'. Which was surprising. Certainly the difference is huge if there are 10k items in the set. John =:-> From w.richert at gmx.net Fri Oct 23 22:53:24 2009 From: w.richert at gmx.net (Willi Richert) Date: Fri, 23 Oct 2009 22:53:24 +0200 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE1E71C.7020806@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE1E71C.7020806@gmail.com> Message-ID: <200910232253.24375.w.richert@gmx.net> Hi, surprised about the performance of for/break provided by Vitor, I did some more testing. It revealed that indeed we can forget the get() (which was implemented as a stripped down pop()): from timeit import * stats = ["for i in xrange(1000): iter(s).next() ", "for i in xrange(1000): \n\tfor x in s: \n\t\tbreak ", "for i in xrange(1000): s.add(s.pop()) ", "for i in xrange(1000): s.get() "] for stat in stats: t = Timer(stat, setup="s=set(range(100))") try: print "Time for %s:\t %f"%(stat, t.timeit(number=1000)) except: t.print_exc() $ ./test_get.py Time for for i in xrange(1000): iter(s).next() : 0.433080 Time for for i in xrange(1000): for x in s: break : 0.148695 Time for for i in xrange(1000): s.add(s.pop()) : 0.317418 Time for for i in xrange(1000): s.get() : 0.146673 In some tests, for/break was even slightly faster then get(). As always, intuition regarding performance bottlenecks is flawed ;-) Anyway, thanks for all the helpful comments, especially to Stefan for the http://comments.gmane.org/gmane.comp.python.ideas/5606 link. Regards, wr Am Freitag, 23. Oktober 2009 19:25:48 schrieb John Arbash Meinel: > Vitor Bosshard wrote: > > 2009/10/23 Willi Richert : > >> Hi, > >> > >> recently I wrote an algorithm, in which very often I had to get an > >> arbitrary element from a set without removing it. > >> > >> Three possibilities came to mind: > >> > >> 1. > >> x = some_set.pop() > >> some_set.add(x) > >> > >> 2. > >> for x in some_set: > >> break > >> > >> 3. > >> x = iter(some_set).next() > >> > >> > >> Of course, the third should be the fastest. It nevertheless goes through > >> all the iterator creation stuff, which costs some time. I wondered, why > >> the builtin set does not provide a more direct and efficient way for > >> retrieving some element without removing it. Is there any reason for > >> this? > >> > >> I imagine something like > >> > >> x = some_set.get() > > > > I see this as being useful for frozensets as well, where you can't get > > an arbitrary element easily due to the obvious lack of .pop(). I ran > > into this recently, when I had a frozenset that I knew had 1 element > > (it was the difference between 2 other sets), but couldn't get to that > > element easily (get the pun?) > > So in my testing (2) was actually the fastest. I assumed because .next() > was a function call overhead, while: > for x in some_set: > break > > Was evaluated inline. It probably still has to call PyObject_GetIter, > however it doesn't have to create a stack frame for it. > > This is what "timeit" tells me. All runs are of the form: > python -m timeit -s "s = set([10])" ... > > 0.101us "for x in s: break; x" > 0.130us "for x in s: pass; x" > 0.234us -s "n = next; i = iter" "x = n(i(s)); x" > 0.248us "x = next(iter(s)); x" > 0.341us "x = iter(s).next(); x" > > So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x > faster than (iter(s).next()). > I was pretty surprised that it was 30% faster than "for x in s: pass". I > assume it has something to do with a potential "else:" statement? > > Note that all of these are significantly < 1us. So this only matters if > it is something you are doing often. > > I don't know your specific timings, but I would guess that: > for x in s: break > > Is actually going to be faster than your > s.get() > > Primarily because s.get() requires an attribute lookup. I would think > your version might be faster for: > stat2 = "g = s.get; for i in xrange(100): g()" > > However, that is still a function call, which may be treated differently > by the interpreter than the for:break loop. I certainly suggest you try > it and compare. > > John > =:-> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Oct 24 00:44:22 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Oct 2009 09:44:22 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE20E0D.4050405@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <4AE20E0D.4050405@gmail.com> Message-ID: <200910240944.22917.steve@pearwood.info> On Sat, 24 Oct 2009 07:11:57 am John Arbash Meinel wrote: > The point of my test was that it was a set with a *single* item, and > 'break' was 30% faster than 'pass'. Which was surprising. Not really. See below. > Certainly > the difference is huge if there are 10k items in the set. Earlier you suggested that the difference may have been because of a potential "else" statement. That won't be it: if there's no "else" in the code, it's not compiled in: >>> import dis >>> dis.dis(compile("for x in s: break", "", "exec")) 1 0 SETUP_LOOP 15 (to 18) 3 LOAD_NAME 0 (s) 6 GET_ITER >> 7 FOR_ITER 7 (to 17) 10 STORE_NAME 1 (x) 13 BREAK_LOOP 14 JUMP_ABSOLUTE 7 >> 17 POP_BLOCK >> 18 LOAD_CONST 0 (None) 21 RETURN_VALUE >>> >>> dis.dis(compile("for x in s: pass", "", "exec")) 1 0 SETUP_LOOP 14 (to 17) 3 LOAD_NAME 0 (s) 6 GET_ITER >> 7 FOR_ITER 6 (to 16) 10 STORE_NAME 1 (x) 13 JUMP_ABSOLUTE 7 >> 16 POP_BLOCK >> 17 LOAD_CONST 0 (None) 20 RETURN_VALUE The difference is likely to be this: for x in s: break retrieves the first (only) element of the set, then immediately breaks out of the loop. This is very different from: for x in s: pass which retrieves the first element of the set, then tries to retrieve a second element, which fails and raises StopIteration, which is then caught, ending the loop. That's much more expensive. -- Steven D'Aprano From steve at pearwood.info Sat Oct 24 00:46:48 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Oct 2009 09:46:48 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> Message-ID: <200910240946.48542.steve@pearwood.info> On Sat, 24 Oct 2009 06:04:12 am Terry Reedy wrote: > John Arbash Meinel wrote: > > So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x > > faster than (iter(s).next()). > > I was pretty surprised that it was 30% faster than "for x in s: > > pass". I assume it has something to do with a potential "else:" > > statement? > > for x in s: pass > > iterates through *all* the elements in s and leaves x bound to the > arbritrary *last* one instead of the arbitrary *first* one. For a > large set, this would be a lot slower, not just a little. > > fwiw, I think the use case for this is sufficiently rare that it does > not need a separate method just for this purpose. And yet it keeps coming up, again and again... obviously people using sets in code think it has a use-case. I did ask earlier for a use-case, and the OP hasn't replied, but Vitor Bosshard did point out one of his use-cases: he had the difference between two frozensets, which he knew had only one element, but due to the lack of pop() he had no straightforward way of finding out what that element was. The lack of get() in sets and frozensets is sounding more and more to me like the victory of purity over practicality. -- Steven D'Aprano From sturla at molden.no Sat Oct 24 00:47:28 2009 From: sturla at molden.no (Sturla Molden) Date: Sat, 24 Oct 2009 00:47:28 +0200 Subject: [Python-Dev] time.clock() on windows In-Reply-To: <4AE1E576.5000903@molden.no> References: <4ADE2AA9.4030604@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42001@exchis.ccp.ad.local> <1256132087.5069.39.camel@localhost> <4ADF247E.8040401@molden.no> <4ADF481B.90207@gmail.com> <1256147726.5069.50.camel@localhost> <930F189C8A437347B80DF2C156F7EC7F098FF42135@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F098FF4213E@exchis.ccp.ad.local> <4AE0C54F.20705@gmail.com> <930F189C8A437347B80DF2C156F7EC7F098FF4246E@exchis.ccp.ad.local> <4AE1E576.5000903@molden.no> Message-ID: <4AE23280.50209@molden.no> Sturla Molden skrev: > I have a suggestion, forgive me if I am totally ignorant. :-) > Ah, damn... Since there is a GIL, we don't need any of that crappy synchronization. And my code does not correct for the 20 ms time jitter in GetSystemTimeAsFileTime. Sorry! S.M. From steve at pearwood.info Sat Oct 24 00:49:38 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Oct 2009 09:49:38 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910232253.24375.w.richert@gmx.net> References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910232253.24375.w.richert@gmx.net> Message-ID: <200910240949.40150.steve@pearwood.info> On Sat, 24 Oct 2009 07:53:24 am Willi Richert wrote: > Hi, > > surprised about the performance of for/break provided by Vitor, I did > some more testing. It revealed that indeed we can forget the get() > (which was implemented as a stripped down pop()): I don't understand that conclusion. According to your tests, your implementation of get() is as fast as "for x in set: break", and it's certainly much, much more readable and straightforward. -- Steven D'Aprano From ben+python at benfinney.id.au Sat Oct 24 01:26:27 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 24 Oct 2009 10:26:27 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: <87skd966ak.fsf@benfinney.id.au> Steven D'Aprano writes: > On Sat, 24 Oct 2009 06:04:12 am Terry Reedy wrote: > > fwiw, I think the use case for this is sufficiently rare that it > > does not need a separate method just for this purpose. > > And yet it keeps coming up, again and again... obviously people using > sets in code think it has a use-case. The desire for this may be obvious, but what seems to be lacking is One Obvious Way to Do It. I agree that ?for x in foo_set: break? is not an Obvious Way. > The lack of get() in sets and frozensets is sounding more and more to > me like the victory of purity over practicality. What would be the input to ?set.get?? If it's the value, that makes it rather non-obvious; if I already know about ?dict.get? which takes the key, I'm not going to be thinking about the ?get? method if I don't have a key to feed it. Once I learn it, I'm going to forget it easily, because it's inconsistent with ?dict.get?. So I don't think that meets the ?obvious way? criterion. By analogy with ?list.pop?, the method that takes the ?top? item off the ?stack?, I would expect to see ?list.peek? and ?set.peek?, to see the item without altering the container. -- \ ?Odious ideas are not entitled to hide from criticism behind | `\ the human shield of their believers' feelings.? ?Richard | _o__) Stallman | Ben Finney From steve at pearwood.info Sat Oct 24 03:48:09 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Oct 2009 12:48:09 +1100 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <4AE1A5DA.40603@gmail.com> References: <200910231223.08470.w.richert@gmx.net> <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> <4AE1A5DA.40603@gmail.com> Message-ID: <200910241248.09530.steve@pearwood.info> On Fri, 23 Oct 2009 11:47:22 pm Nick Coghlan wrote: > Yuvgoog Greenle wrote: > > On Fri, Oct 23, 2009 at 1:09 PM, Steven D'Aprano wrote: > >> Is there any point? Even if accepted, it's too late to make it > >> into 3.1, and with the overwhelming approval for a moratorium on > >> changes to built-ins, it is likely to just sit in the tracker, > >> forgotten, until 2013 or later. How likely is it that the patch > >> will still be applicable? > > > > +1 on throwing it away completely even if it's a good idea. I > > suggest Willi go invent a new language (and hope for it to become > > popular :-) if he wants to experiment. > > Careful folks - these kinds of throwaway comments may be clearly > tongue in cheek for anyone following the moratorium discussion on > python-ideas, but will be justifiably confusing to anyone else, > especially newcomers. I'm not being tongue-in-cheek or sarcastic. My question was serious -- if there is a moratorium, is there any reason to bother submitting patches for functional changes to built-ins? A lot can change between now and 2013, and I for one wouldn't bother making a patch that I knew wouldn't even be considered for inclusion for four years, and would likely need to be re-done even if it was accepted. Guido has said that the purpose of the moratorium is to discourage changes to the language. If we're discouraging changes, shouldn't we actually discourage them rather than waste people's time and give them false hope by telling them to put them in the tracker? Nick, you made the comparison with C in another thread. I don't think that the comparison with C is apt, for various reasons, but putting that aside, given that C is so stable, what do they do with suggested changes to the language? Is there a C issue tracker with ten years of accumulated patches, or even proposals, in the queue? -- Steven D'Aprano From steve at pearwood.info Sat Oct 24 04:07:10 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Oct 2009 13:07:10 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <87skd966ak.fsf@benfinney.id.au> References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <87skd966ak.fsf@benfinney.id.au> Message-ID: <200910241307.10303.steve@pearwood.info> On Sat, 24 Oct 2009 10:26:27 am Ben Finney wrote: > Steven D'Aprano writes: > > On Sat, 24 Oct 2009 06:04:12 am Terry Reedy wrote: > > > fwiw, I think the use case for this is sufficiently rare that it > > > does not need a separate method just for this purpose. > > > > And yet it keeps coming up, again and again... obviously people > > using sets in code think it has a use-case. > > The desire for this may be obvious, but what seems to be lacking is > One Obvious Way to Do It. > > I agree that ?for x in foo_set: break? is not an Obvious Way. > > > The lack of get() in sets and frozensets is sounding more and more > > to me like the victory of purity over practicality. > > What would be the input to ?set.get?? It wouldn't take any input. > If it's the value, that makes it rather non-obvious; It makes it pointless if it takes the value. If you already have the value, why would you need to retrieve it from the set? > if I already > know about ?dict.get? which takes the key, I'm not going to be > thinking about the ?get? method if I don't have a key to feed it. > Once I learn it, I'm going to forget it easily, because it's > inconsistent with ?dict.get?. So I don't think that meets the > ?obvious way? criterion. "get" is such a generic term that I don't believe that is a problem. There are already quite a number of get() methods in the 2.6 std lib: $ grep "def get[(]" *.py _abcoll.py: def get(self, key, default=None): ConfigParser.py: def get(self, section, option): ConfigParser.py: def get(self, section, option, raw=False, vars=None): doctest.py: def get(self): mailbox.py: def get(self, key, default=None): os.py: def get(self, key, failobj=None): pickle.py: def get(self, i, pack=struct.pack): Queue.py: def get(self, block=True, timeout=None): shelve.py: def get(self, key, default=None): sre_parse.py: def get(self): threading.py: def get(self): UserDict.py: def get(self, key, failobj=None): UserDict.py: def get(self, key, default=None): weakref.py: def get(self, key, default=None): weakref.py: def get(self, key, default=None): webbrowser.py:def get(using=None): I think you over-estimate the degree of confusion people suffer from similar sounding methods. I don't think people are confused that dict[key] and list[index] have different semantics, and I don't see why dict.get(key[, default]) and set.get() would be any different. But if you don't like get(), spell it differently. There's plenty of opportunity for bike-shedding: getitem() getelement() get_arbitrary_element() peek() etc. > By analogy with ?list.pop?, the method that takes the ?top? item off > the ?stack?, I would expect to see ?list.peek? and ?set.peek?, to see > the item without altering the container. You don't need list.peek() because there already is an obvious way to retrieve an item from a list without removing it: list[index]. -- Steven D'Aprano From ben+python at benfinney.id.au Sat Oct 24 05:02:48 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 24 Oct 2009 14:02:48 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <87skd966ak.fsf@benfinney.id.au> <200910241307.10303.steve@pearwood.info> Message-ID: <87aazh5w9z.fsf@benfinney.id.au> Steven D'Aprano writes: > On Sat, 24 Oct 2009 10:26:27 am Ben Finney wrote: > > Steven D'Aprano writes: > > > The lack of get() in sets and frozensets is sounding more and more > > > to me like the victory of purity over practicality. > > > > What would be the input to ?set.get?? > > It wouldn't take any input. That is even less obvious. I would expect a parameter-less ?set.get? to get the set. Not terribly useful, but the name and function signature doesn't suggest anything else. > "get" is such a generic term that I don't believe that is a problem. The problem above is made less problematic by the fact that the function signature (e.g. ?foo_dict.get(key)?) clarifies the answer to the question ?get what??. Whereas ?foo_set.get()? doesn't communicate much at all to the reader. If we want a method that gets one item from a set, perhaps the name can make it clearer: name it ?set.getitem?. But which item should it get? The ?__getitem__? special method of lists and dictionaries requires an index or key as parameter. -- \ ?Roll dice!? ?Why?? ?Shut up! I don't need your fucking | `\ *input*, I need you to roll dice!? ?Luke Crane, demonstrating | _o__) his refined approach to play testing, 2009 | Ben Finney From steve at pearwood.info Sat Oct 24 05:38:52 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Oct 2009 14:38:52 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <87aazh5w9z.fsf@benfinney.id.au> References: <200910231132.45504.w.richert@gmx.net> <200910241307.10303.steve@pearwood.info> <87aazh5w9z.fsf@benfinney.id.au> Message-ID: <200910241438.53974.steve@pearwood.info> On Sat, 24 Oct 2009 02:02:48 pm Ben Finney wrote: > Steven D'Aprano writes: > > On Sat, 24 Oct 2009 10:26:27 am Ben Finney wrote: > > > Steven D'Aprano writes: > > > > The lack of get() in sets and frozensets is sounding more and > > > > more to me like the victory of purity over practicality. > > > > > > What would be the input to ?set.get?? > > > > It wouldn't take any input. > > That is even less obvious. I would expect a parameter-less ?set.get? > to get the set. Not terribly useful, but the name and function > signature doesn't suggest anything else. You already have the set. Why would you need a method that you call that returns the set you already have? A method called "get" obviously gets something, and if it takes no arguments the only thing is has access to is the set. The obvious things it could get are the set as a whole or some part of the set. Since getting the set as a whole would be pointless, and we're allowed to assume that the language designers aren't lunatics who would waste time creating pointless methods, the obvious answer is that it gets some part of the set. Since there's no obvious reason to choose one subset over another subset, the only useful thing it could return is a single arbitrary item. But if you're not willing to guess what it gets, you are permitted to read the docstring. > > "get" is such a generic term that I don't believe that is a > > problem. > > The problem above is made less problematic by the fact that the > function signature (e.g. ?foo_dict.get(key)?) clarifies the answer to > the question ?get what??. Whereas ?foo_set.get()? doesn't communicate > much at all to the reader. You are demanding a level of intuitiveness that few, if any, functions in the standard library would be able to meet. Consider list.index(x): would you expect it to return the value at index x, or the index of value x? Either description is compatible with the name, and in fact people sometimes mix them up. Or sorted() -- from the name alone, I'd expect this to work, but it doesn't: >>> sorted(1, 2, 3) Traceback (most recent call last): File "", line 1, in TypeError: 'int' object is not iterable That's not a problem with the built-in function. I took a guess about the API, and guessed wrong. I'll learn from this, and get it right next time. > If we want a method that gets one item from a set, perhaps the name > can make it clearer: name it ?set.getitem?. But which item should it > get? The ?__getitem__? special method of lists and dictionaries > requires an index or key as parameter. Neither of which is appropriate for sets -- sets are unordered, so elements don't have indexes, and they don't map keys to values. They just have elements. Sets are not lists, and they're not dicts. Analogies only go so far, and you can't reason about set methods by considering how lists or dicts will behave. -- Steven D'Aprano From ben+python at benfinney.id.au Sat Oct 24 06:06:28 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 24 Oct 2009 15:06:28 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it References: <200910231132.45504.w.richert@gmx.net> <200910241307.10303.steve@pearwood.info> <87aazh5w9z.fsf@benfinney.id.au> <200910241438.53974.steve@pearwood.info> Message-ID: <8763a55tbv.fsf@benfinney.id.au> Steven D'Aprano writes: > On Sat, 24 Oct 2009 02:02:48 pm Ben Finney wrote: > > I would expect a parameter-less ?set.get? to get the set. Not > > terribly useful, but the name and function signature doesn't suggest > > anything else. > > You already have the set. Why would you need a method that you call > that returns the set you already have? Exactly, hence the confusion. I think the method name ?set.get? is poor for that reason. > A method called "get" obviously gets something, and if it takes no > arguments the only thing is has access to is the set. The obvious > things it could get are the set as a whole or some part of the set. Which then raises the question ?what part of the set does it get??, which the function signature does nothing to answer. I'm proposing that a no-parameters ?set.get? is needlessly confusing to think about. > Since there's no obvious reason to choose one subset over another > subset, the only useful thing it could return is a single arbitrary > item. That's not at all obvious, IMO. The name doesn't give much help at all in getting to that conclusion, and isn't easily associated with that connotation. > You are demanding a level of intuitiveness that few, if any, functions > in the standard library would be able to meet. I'm not demanding intuitiveness; all programming interfaces are learned. I'm asking for ease of learning and later recall. > That's not a problem with the built-in function. I took a guess about > the API, and guessed wrong. I'll learn from this, and get it right > next time. Which is partly my point. Such a narrow use case (?get an arbitrary item from the set, without specifying anything about what that item might be?) is a mismatch for such a generic name. It makes it unnecessarily difficult to make the association. Since the use case is so specific, I would expect the name to be specific too, to better match the use case. (Since I know how your mind works, I can anticipate facetious fifty-character-long names bubbling up already in a response from you. Let's just assume that joke is already done :-) -- \ ?Holy tintinnabulation, Batman!? ?Robin | `\ | _o__) | Ben Finney From stephen at xemacs.org Sat Oct 24 08:04:33 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 24 Oct 2009 15:04:33 +0900 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <200910241248.09530.steve@pearwood.info> References: <200910231223.08470.w.richert@gmx.net> <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> <4AE1A5DA.40603@gmail.com> <200910241248.09530.steve@pearwood.info> Message-ID: <87fx99ba4u.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > I'm not being tongue-in-cheek or sarcastic. My question was serious -- > if there is a moratorium, is there any reason to bother submitting > patches for functional changes to built-ins? Yes. Python is open source. Private and public forks are possible and (at least in principle) encouraged where the core project decides that the proposed changes are inappropriate (or should be deferred, as here). Nevertheless, isn't the core Python project the obvious common point of contact for sharing such ideas, even if there is a moratorium on the code base itself? I know that a few patches to Mailman have lived for nearly a decade, beloved by their users and not (much) bothering the Mailman maintainers with their obvious pitfalls for naive users. Whether the tracker is the appropriate place is another question, but I think it's easier to search for "serious proposal with patch" than the mailing lists. From smiles at worksmail.net Sat Oct 24 08:04:25 2009 From: smiles at worksmail.net (C or L Smith) Date: Sat, 24 Oct 2009 11:49:25 +0545 Subject: [Python-Dev] tokenize string literal problem Message-ID: BACKGROUND I'm trying to modify the doctest DocTestParser so it will parse docstring code snippets out of a *py file. (Although doctest can parse these with another method out of *pyc, it is missing certain decorated functions and we would also like to insist of import of needed modules rather and that method automatically loads everything from the module containing the code.) PROBLEM I need to find code snippets which are located in docstrings. Docstrings, being string literals should be able to be parsed out with tokenize. But tokenize is giving the wrong results (or I am doing something wrong) for this (pathological) case: foo.py: +---- def bar(): """ A quoted triple quote is not a closing of this docstring: >>> print '"""' """ """ # <-- this is the closing quote pass +---- Here is how I tokenize the file: ### import re, tokenize DOCSTRING_START_RE = re.compile('\s+[ru]*("""|' + "''')") o=open('foo.py','r') for ti in tokenize.generate_tokens(o.next): typ = ti[0] text = ti[-1] if typ == tokenize.STRING: if DOCSTRING_START_RE.match(text): print "DOCSTRING:",repr(text) o.close() ### which outputs: DOCSTRING: ' """\n A quoted triple quote is not a closing\n of this docstring:\n >>> print \'"""\'\n' DOCSTRING: ' """\n """ # <-- this is the closing quote\n' There should be only one string tokenized, I believe. The PythonWin editor parses (and colorizes) this correctly, but tokenize (or I) are making an error. Thanks for any help, Chris From stephen at xemacs.org Sat Oct 24 08:22:25 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 24 Oct 2009 15:22:25 +0900 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <87aazh5w9z.fsf@benfinney.id.au> References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <87skd966ak.fsf@benfinney.id.au> <200910241307.10303.steve@pearwood.info> <87aazh5w9z.fsf@benfinney.id.au> Message-ID: <87d44db9b2.fsf@uwakimon.sk.tsukuba.ac.jp> Ben Finney writes: > Steven D'Aprano writes: > > "get" is such a generic term that I don't believe that is a problem. > > The problem above is made less problematic by the fact that the function > signature (e.g. 'foo_dict.get(key)') clarifies the answer to the > question "get what?". Whereas 'foo_set.get()' doesn't communicate much > at all to the reader. I agree. This is precisely why a couple of months ago people were proposing names like ".getany()" for this API. The problem brought up then was that pretty clearly people would then do things like x = foo.getany() y = foo.getany() expecting x and y to be different (ie, interpreting "any" as "random"). Pretty soon there was a whole family of proposals for such methods: .getfirst(), .getlast(), .getrandom(), .getonly(), .... I think it would be better to document the various ways of doing this, and let each program define its own oneliner for the MySet.get() that makes idiomatic sense in its use case. From smiles at worksmail.net Sat Oct 24 08:33:10 2009 From: smiles at worksmail.net (C or L Smith) Date: Sat, 24 Oct 2009 12:18:10 +0545 Subject: [Python-Dev] tokenize string literal problem Message-ID: C or L Smith wrote: > PROBLEM > I need to find code snippets which are located in docstrings. > Docstrings, being string literals should be able to be parsed out > with tokenize. But tokenize is giving the wrong results (or I am > doing something wrong) for this (pathological) case: > > foo.py: > +---- > def bar(): > """ > A quoted triple quote is not a closing > of this docstring: > >>> print '"""' > """ > """ # <-- this is the closing quote > pass > +---- > I now see that I've created a code snippet that is invalid. Myopia. The thing that pythonWin was doing correctly was displaying my sample STRING not code. I had delimited the code with triple-single-quotes so it showed up correctly. In fact, if entered as code it would show the need to delimit the docstring contents with ''' rather than """. Sorry! /c From ncoghlan at gmail.com Sat Oct 24 09:19:54 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Oct 2009 17:19:54 +1000 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <8763a55tbv.fsf@benfinney.id.au> References: <200910231132.45504.w.richert@gmx.net> <200910241307.10303.steve@pearwood.info> <87aazh5w9z.fsf@benfinney.id.au> <200910241438.53974.steve@pearwood.info> <8763a55tbv.fsf@benfinney.id.au> Message-ID: <4AE2AA9A.6090107@gmail.com> Ben Finney wrote: > Which then raises the question ?what part of the set does it get??, > which the function signature does nothing to answer. I'm proposing that > a no-parameters ?set.get? is needlessly confusing to think about. The fact that set.get() is just set.pop() without removing the result from the set seems perfectly straightforward. > Since the use case is so specific, I would expect the name to be > specific too, to better match the use case. The use case is no more specific than set.pop(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From gzlist at googlemail.com Sat Oct 24 11:12:26 2009 From: gzlist at googlemail.com (Martin (gzlist)) Date: Sat, 24 Oct 2009 10:12:26 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE2AA9A.6090107@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <200910241307.10303.steve@pearwood.info> <87aazh5w9z.fsf@benfinney.id.au> <200910241438.53974.steve@pearwood.info> <8763a55tbv.fsf@benfinney.id.au> <4AE2AA9A.6090107@gmail.com> Message-ID: On 24/10/2009, Nick Coghlan wrote: > Ben Finney wrote: >> Which then raises the question ?what part of the set does it get??, >> which the function signature does nothing to answer. I'm proposing that >> a no-parameters ?set.get? is needlessly confusing to think about. > > The fact that set.get() is just set.pop() without removing the result > from the set seems perfectly straightforward. There's a different proposed meaning for `set.get` that's been discussed on python-dev before: That one I've had cause for before and no clean and fast way of writing, this one I've always just done the for/break thing. Martin From steve at pearwood.info Sat Oct 24 16:20:37 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 25 Oct 2009 01:20:37 +1100 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <87fx99ba4u.fsf@uwakimon.sk.tsukuba.ac.jp> References: <200910231223.08470.w.richert@gmx.net> <200910241248.09530.steve@pearwood.info> <87fx99ba4u.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <200910250120.38748.steve@pearwood.info> On Sat, 24 Oct 2009 05:04:33 pm Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > ?> I'm not being tongue-in-cheek or sarcastic. My question was > serious -- > if there is a moratorium, is there any reason to bother > submitting > patches for functional changes to built-ins? > > Yes. ?Python is open source. ?Private and public forks are possible > and (at least in principle) encouraged where the core project decides > that the proposed changes are inappropriate (or should be deferred, > as here). ?Nevertheless, isn't the core Python project the obvious > common point of contact for sharing such ideas, even if there is a > moratorium on the code base itself? No. It's not obvious to me that the CPython tracker is the right place for patches for implementations that aren't for CPython. It's not even obvious that there would be a common point of contact for private and public forks, let alone that it would be CPython's tracker. There are, by my count, 14 active and defunct implementations of Python to date, apart from CPython itself. How many of them currently use the CPython tracker to share patches? If the answer is "None", why would you expect future implementations and forks to be any different? -- Steven D'Aprano From w.richert at gmx.net Sat Oct 24 16:29:32 2009 From: w.richert at gmx.net (Willi Richert) Date: Sat, 24 Oct 2009 16:29:32 +0200 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910240949.40150.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <200910232253.24375.w.richert@gmx.net> <200910240949.40150.steve@pearwood.info> Message-ID: <200910241629.32395.w.richert@gmx.net> Hi, I agree. But, here are the pros/cons collected from the recent list repsonses: Pro: - more readable - newbies will encounter one of the fastest solution (.get()) before trying slower "first solutions" like (iter(set).next()) Cons: - no name consensus. get() getany() arbitrary() ? - BDFL moratorium, which I find very wise (get() is, however, no language extension, but std lib extension, which Guido did not moratorize, didn't he?) - other classes should then also be extended, like frozenset Regards, wr PS: what is the correct verb form of moratorium? Am Samstag, 24. Oktober 2009 00:49:38 schrieb Steven D'Aprano: > On Sat, 24 Oct 2009 07:53:24 am Willi Richert wrote: > > Hi, > > > > surprised about the performance of for/break provided by Vitor, I did > > some more testing. It revealed that indeed we can forget the get() > > (which was implemented as a stripped down pop()): > > I don't understand that conclusion. According to your tests, your > implementation of get() is as fast as "for x in set: break", and it's > certainly much, much more readable and straightforward. > From steve at pearwood.info Sat Oct 24 17:18:28 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 25 Oct 2009 02:18:28 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE2AA9A.6090107@gmail.com> Message-ID: <200910250218.29454.steve@pearwood.info> On Sat, 24 Oct 2009 08:12:26 pm Martin (gzlist) wrote: > On 24/10/2009, Nick Coghlan wrote: > > Ben Finney wrote: > >> Which then raises the question ?what part of the set does it > >> get??, which the function signature does nothing to answer. I'm > >> proposing that a no-parameters ?set.get? is needlessly confusing > >> to think about. > > > > The fact that set.get() is just set.pop() without removing the > > result from the set seems perfectly straightforward. > > There's a different proposed meaning for `set.get` that's been > discussed on python-dev before: > > That thread has an example of a use-case for extracting an item from a set, given the item itself: interning. The current Python idiom for interning is to use a dict: # store the canonical version _cache[item] = item # retrieve the interned version item = _cache[item] It has been argued that such interning is better done by sets rather than dicts, since this will save a pointer per item (plus any additional overhead dicts may have over that of sets). If this argument is accepted, then we want a fast, efficient operation to perform this: def get(self, item): """Return an element of the set equal to item. Useful for retrieving the canonical version of an element and for interning. """ for x in self: if x == item: return x raise ValueError('item not in set') The above is O(N); ideally it should be O(1). Normally we don't care about identity, only equality, so if x and item above are equal we are indifferent about which one we use. But interning is a real example of a use-case where we do care about identity. Arguably it is inefficient and wasteful to use a dict for interning when sets could be just as fast while using less memory. The other proposed semantics for set.get() are to retrieve an arbitrary element. It must be arbitrary, because elements in a set are unordered and unkeyed. This gives us the semantics of pop() without removing the element: def get(self): """Return an arbitrary element of the set without removing it.""" for x in self: return x raise ValueError("empty set") This is a frequent request, but I still don't know what the use-case is. If you'll excuse me stating the obvious, both of these can easily be written as helper-functions. The main arguments for making them methods are: (1) The first one is needlessly O(N) when it could be O(1). (2) The second one is apparently a common need (although I personally have never needed it), forcing people to re-invent the wheel, sometimes incorrectly or inefficiently, e.g.: def get(aset): for element in aset: pass return element -- Steven D'Aprano From martin at v.loewis.de Sat Oct 24 17:28:58 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Oct 2009 17:28:58 +0200 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <200910241248.09530.steve@pearwood.info> References: <200910231223.08470.w.richert@gmx.net> <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> <4AE1A5DA.40603@gmail.com> <200910241248.09530.steve@pearwood.info> Message-ID: <4AE31D3A.6090207@v.loewis.de> > I'm not being tongue-in-cheek or sarcastic. My question was serious -- > if there is a moratorium, is there any reason to bother submitting > patches for functional changes to built-ins? A lot can change between > now and 2013, and I for one wouldn't bother making a patch that I knew > wouldn't even be considered for inclusion for four years, and would > likely need to be re-done even if it was accepted. Guido has said that > the purpose of the moratorium is to discourage changes to the language. I haven't been following the discussion, but I wouldn't expect that a moratorium on language changes would rule out adding a method to the set type. Regards, Martin From w.richert at gmx.net Sat Oct 24 17:41:18 2009 From: w.richert at gmx.net (Willi Richert) Date: Sat, 24 Oct 2009 17:41:18 +0200 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910232253.24375.w.richert@gmx.net> References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910232253.24375.w.richert@gmx.net> Message-ID: <200910241741.18388.w.richert@gmx.net> Hi, someone on this list mentioned that much of the s.get() time is spend on the name lookup for get(). That is indeed the case: =================== from timeit import * stats = ["for i in xrange(1000): iter(s).next() ", "for i in xrange(1000): \n\tfor x in s: \n\t\tbreak", "for i in xrange(1000): s.add(s.pop()) ", "for i in xrange(1000): s.get() ", "g=s.get;\nfor i in xrange(1000): g() "] for stat in stats: t = Timer(stat, setup="s=set(range(1000))") print "Time for %s:\t %f"%(stat, t.timeit(number=1000)) ================== Time for for i in xrange(1000): iter(s).next() : 0.448227 Time for for i in xrange(1000): for x in s: break: 0.141669 Time for for i in xrange(1000): s.add(s.pop()) : 0.348055 Time for for i in xrange(1000): s.get() : 0.148580 Time for g=s.get; for i in xrange(1000): g() : 0.080563 So, now set.get() is indeed the fastest and preferable solution if you need massive amounts of retrieving elements from a set without removing them. wr Am Freitag, 23. Oktober 2009 22:53:24 schrieb Willi Richert: > Hi, > > surprised about the performance of for/break provided by Vitor, I did some > more testing. It revealed that indeed we can forget the get() (which was > implemented as a stripped down pop()): > > from timeit import * > stats = ["for i in xrange(1000): iter(s).next() ", > "for i in xrange(1000): \n\tfor x in s: \n\t\tbreak ", > "for i in xrange(1000): s.add(s.pop()) ", > "for i in xrange(1000): s.get() "] > > for stat in stats: > t = Timer(stat, setup="s=set(range(100))") > try: > print "Time for %s:\t %f"%(stat, t.timeit(number=1000)) > except: > t.print_exc() > > > > $ ./test_get.py > Time for for i in xrange(1000): iter(s).next() : 0.433080 > Time for for i in xrange(1000): > for x in s: > break : 0.148695 > Time for for i in xrange(1000): s.add(s.pop()) : 0.317418 > Time for for i in xrange(1000): s.get() : 0.146673 > > > In some tests, for/break was even slightly faster then get(). > > As always, intuition regarding performance bottlenecks is flawed ;-) > > Anyway, thanks for all the helpful comments, especially to Stefan for the > http://comments.gmane.org/gmane.comp.python.ideas/5606 link. > > Regards, > wr > > Am Freitag, 23. Oktober 2009 19:25:48 schrieb John Arbash Meinel: > > Vitor Bosshard wrote: > > > 2009/10/23 Willi Richert : > > >> Hi, > > >> > > >> recently I wrote an algorithm, in which very often I had to get an > > >> arbitrary element from a set without removing it. > > >> > > >> Three possibilities came to mind: > > >> > > >> 1. > > >> x = some_set.pop() > > >> some_set.add(x) > > >> > > >> 2. > > >> for x in some_set: > > >> break > > >> > > >> 3. > > >> x = iter(some_set).next() > > >> > > >> > > >> Of course, the third should be the fastest. It nevertheless goes > > >> through all the iterator creation stuff, which costs some time. I > > >> wondered, why the builtin set does not provide a more direct and > > >> efficient way for retrieving some element without removing it. Is > > >> there any reason for this? > > >> > > >> I imagine something like > > >> > > >> x = some_set.get() > > > > > > I see this as being useful for frozensets as well, where you can't get > > > an arbitrary element easily due to the obvious lack of .pop(). I ran > > > into this recently, when I had a frozenset that I knew had 1 element > > > (it was the difference between 2 other sets), but couldn't get to that > > > element easily (get the pun?) > > > > So in my testing (2) was actually the fastest. I assumed because .next() > > was a function call overhead, while: > > for x in some_set: > > break > > > > Was evaluated inline. It probably still has to call PyObject_GetIter, > > however it doesn't have to create a stack frame for it. > > > > This is what "timeit" tells me. All runs are of the form: > > python -m timeit -s "s = set([10])" ... > > > > 0.101us "for x in s: break; x" > > 0.130us "for x in s: pass; x" > > 0.234us -s "n = next; i = iter" "x = n(i(s)); x" > > 0.248us "x = next(iter(s)); x" > > 0.341us "x = iter(s).next(); x" > > > > So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x > > faster than (iter(s).next()). > > I was pretty surprised that it was 30% faster than "for x in s: pass". I > > assume it has something to do with a potential "else:" statement? > > > > Note that all of these are significantly < 1us. So this only matters if > > it is something you are doing often. > > > > I don't know your specific timings, but I would guess that: > > for x in s: break > > > > Is actually going to be faster than your > > s.get() > > > > Primarily because s.get() requires an attribute lookup. I would think > > your version might be faster for: > > stat2 = "g = s.get; for i in xrange(100): g()" > > > > However, that is still a function call, which may be treated differently > > by the interpreter than the for:break loop. I certainly suggest you try > > it and compare. > > > > John > > =:-> > From alexander.belopolsky at gmail.com Sat Oct 24 19:34:48 2009 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 24 Oct 2009 13:34:48 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910240946.48542.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: On Fri, Oct 23, 2009 at 6:46 PM, Steven D'Aprano wrote: > On Sat, 24 Oct 2009 06:04:12 am Terry Reedy wrote: .. >> fwiw, I think the use case for this is sufficiently rare that it does >> not need a separate method just for this purpose. > > > And yet it keeps coming up, again and again... obviously people using > sets in code think it has a use-case. > This reminds me a debate I had with Martin several years ago: http://bugs.python.org/issue1507011 Here is the essence: AB> I disagree with Martin. I think interning is a set AB> operation and it is unfortunate that set API does not AB> support it directly. ML> I disagree with Alexander's last remark in several respects: ML> set is indeed a container, and there is a way to get the ML> elements back (namely, by enumerating them, or picking an ML> arbitrary element for removal). There is no operation to check, ML> on insertion of E, what the the prior element equal to E was ML> (if any); there is only a check to determine whether E is in the ML> set. The operation "give me the member equal but not identical ML> to E" is conceptually a lookup operation; the mathematical set ML> construct has no such operation, and the Python set models ML> it closely. IOW, set is *not* a dict with key==value. To me, however, a set seems to be a container that is a specialization of a dict with values and keys being the same. In this model, a get() method, or even a __getitem__ with s[k] is k, is only natural. From debatem1 at gmail.com Sat Oct 24 19:55:06 2009 From: debatem1 at gmail.com (geremy condra) Date: Sat, 24 Oct 2009 13:55:06 -0400 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: <4AE31D3A.6090207@v.loewis.de> References: <200910231223.08470.w.richert@gmx.net> <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> <4AE1A5DA.40603@gmail.com> <200910241248.09530.steve@pearwood.info> <4AE31D3A.6090207@v.loewis.de> Message-ID: On Sat, Oct 24, 2009 at 11:28 AM, "Martin v. L?wis" wrote: >> I'm not being tongue-in-cheek or sarcastic. My question was serious -- >> if there is a moratorium, is there any reason to bother submitting >> patches for functional changes to built-ins? A lot can change between >> now and 2013, and I for one wouldn't bother making a patch that I knew >> wouldn't even be considered for inclusion for four years, and would >> likely need to be re-done even if it was accepted. Guido has said that >> the purpose of the moratorium is to discourage changes to the language. > > I haven't been following the discussion, but I wouldn't expect that > a moratorium on language changes would rule out adding a method to the > set type. > > Regards, > Martin My understanding is that the moratorium would preclude changes to the builtins. Is that not the case here? Geremy Condra From guido at python.org Sat Oct 24 19:55:20 2009 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Oct 2009 10:55:20 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: On Sat, Oct 24, 2009 at 10:34 AM, Alexander Belopolsky wrote: > To me, however, a set seems to be a container that is a specialization > of a dict with values and keys being the same. That's absurd; the mapping provides nothing useful. -- --Guido van Rossum PS. My elbow needs a couple more weeks of rest. Limiting myself to ultra-short emails. From guido at python.org Sat Oct 24 20:01:21 2009 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Oct 2009 11:01:21 -0700 Subject: [Python-Dev] First shot at some_set.get() In-Reply-To: References: <200910231223.08470.w.richert@gmx.net> <9d153b7c0910230419t5e2e6af8n4ecd847a2c9ef329@mail.gmail.com> <4AE1A5DA.40603@gmail.com> <200910241248.09530.steve@pearwood.info> <4AE31D3A.6090207@v.loewis.de> Message-ID: On Sat, Oct 24, 2009 at 10:55 AM, geremy condra wrote: > My understanding is that the moratorium would preclude changes to > the builtins. Is that not the case here? It is as yet undecided. -- --Guido van Rossum PS. My elbow needs a couple more weeks of rest. Limiting myself to ultra-short emails. From solipsis at pitrou.net Sat Oct 24 20:02:30 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Oct 2009 18:02:30 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Clean_up_Python/thread=5F*=2Eh_=3F?= Message-ID: Hello, I am wondering which of the files in Python/thread_*.h are really necessary today. Looking at these files, I think most of them could perhaps be removed in py3k. I've identified three categories of potentially removable files: * The unused file: thread_wince.h Windows CE has actually been using thread_nt.h since January 2009 (see http://bugs.python.org/issue4893 ). thread_wince.h is still there, but it's unused and grep brings no instance of it being mentioned anywhere in the source tree. * The (unsupported, untested?) files: thread_atheos.h, thread_cthread.h, thread_lwp.h, thread_os2.h, thread_pth.h, thread_sgi.h. These files refer to architectures which we probably have stopped caring about. It is not even sure whether these files compile and work ok. Most of them have not seen maintenance for years, except for the OS/2 file. In any case, it is obvious they haven't received the level of attention and support that the pthreads and NT versions have. (also, according to http://www.lesstif.org/~amai/os2/html/porting.html, there is a pthreads interface available for OS/2) * The questionable files: thread_pth.h, thread_solaris.h. GNU pth is probably obsolete on all platforms and superseced by the pthreads API. According to http://www.gnu.org/software/pth/, the last pth version was released in 2006 (and the changelog is rather unimpressive). As for Solaris, according to the Sun website it should provide the pthreads API. It was already documented in a 1999 manpage for SunOS 5.9: http://docs.sun.com/app/docs/doc/816-0216/6m6ngupon?a=view . Making a decision and removing the files considered unnecessary would clarify what platforms still are/should be supported. Having less supported platforms would also make improvements easier. Right now making an addition to the core threading primitives is almost impossible since it would require to provide all those platform-specific versions which almost nobody is competent for. Regards Antoine. From solipsis at pitrou.net Sat Oct 24 20:13:44 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Oct 2009 18:13:44 +0000 (UTC) Subject: [Python-Dev] interning References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: Alexander Belopolsky gmail.com> writes: > > AB> I disagree with Martin. I think interning is a set > AB> operation and it is unfortunate that set API does not > AB> support it directly. > > ML> I disagree with Alexander's last remark in several respects: [...] > ML> The operation "give me the member equal but not identical > ML> to E" is conceptually a lookup operation; the mathematical set > ML> construct has no such operation, and the Python set models > ML> it closely. IOW, set is *not* a dict with key==value. This looks like a debate of purity vs. practicality. I don't have any opinion on a Python-facing API, but the interpreter's dict of interned strings could probably converted to a set as a simple optimization. Regards Antoine. From lists at cheimes.de Sat Oct 24 21:08:08 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 24 Oct 2009 21:08:08 +0200 Subject: [Python-Dev] Clean up Python/thread_*.h ? In-Reply-To: References: Message-ID: <4AE35098.6070306@cheimes.de> Antoine Pitrou wrote: > * The unused file: thread_wince.h > > Windows CE has actually been using thread_nt.h since January 2009 (see > http://bugs.python.org/issue4893 ). thread_wince.h is still there, but it's > unused and grep brings no instance of it being mentioned anywhere in the source > tree. What about older versions of Windows CE? Maybe they still require the wince thread header? > > * The (unsupported, untested?) files: thread_atheos.h, thread_cthread.h, > thread_lwp.h, thread_os2.h, thread_pth.h, thread_sgi.h. thread_atheos.h --------------- According to svn annotate the file was added on one revision and never touched afterwards. It was added by MvL. r27146 | loewis | 2002-06-11 08:22:31 +0200 (Tue, 11 Jun 2002) | 2 lines Patch #488073: AtheOS port. thread_cthread.h ---------------- Most lines were last modified or added in in revision 4018 (!), 12178, 16421 and 24967 by Guido, Thomas Wouters and MvL. The last change was: r24967 | loewis | 2002-01-01 19:41:33 +0100 (Di, 01. Jan 2002) | 2 Zeilen Patch #497098: build support for GNU/Hurd. thread_lwp.h ------------ The history of this file is almost identical to thread_cthread.h. No changes since r23732 (2001-10-16) thread_os2.h ------------ The file seems to be maintained by Andrew Macintyre. The last change was on r46919 (2006-06-13). We should ask him before we remove the file. thread_pth.h ------------ No changes since r15372 (2000-05-08), most of the files was last changed by Guido and Thomas Wouters. thread_sgi.h ------------ Same as thread_cthread.h I'mm +1 on removing the files. Let's ask Martin and Andrew before we remove the wince and os2 thread files. Christian From steven.bethard at gmail.com Sat Oct 24 21:10:34 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 24 Oct 2009 12:10:34 -0700 Subject: [Python-Dev] updated PEP 389: argparse Message-ID: Sorry for the delay, but I've finally updated PEP 389, the argparse PEP, based on all the feedback from python-dev. The full PEP is below, but in short, the important changes are: * The getopt module will *not* be deprecated. * In Python 2, the -3 flag will cause deprecation warnings to be issued for optparse. * No casually visible deprecation warnings for optparse are expected until Jun 2013. * Support for {}-format strings will be added when one of the converters is approved. This can be done at any point though, so it shouldn't hold up inclusion of argparse. Steve ---------------------------------------------------------------------- PEP: 389 Title: argparse - new command line parsing module Version: $Revision: 75674 $ Last-Modified: $Date: 2009-10-24 12:01:49 -0700 (Sat, 24 Oct 2009) $ Author: Steven Bethard Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 25-Sep-2009 Python-Version: 2.7 and 3.2 Post-History: 27-Sep-2009, 24-Oct-2009 Abstract ======== This PEP proposes inclusion of the argparse [1]_ module in the Python standard library in Python 2.7 and 3.2. Motivation ========== The argparse module is a command line parsing library which provides more functionality than the existing command line parsing modules in the standard library, getopt [2]_ and optparse [3]_. It includes support for positional arguments (not just options), subcommands, required options, options syntaxes like "/f" and "+rgb", zero-or-more and one-or-more style arguments, and many other features the other two lack. The argparse module is also already a popular third-party replacement for these modules. It is used in projects like IPython (the Scipy Python shell) [4]_, is included in Debian testing and unstable [5]_, and since 2007 has had various requests for its inclusion in the standard library [6]_ [7]_ [8]_. This popularity suggests it may be a valuable addition to the Python libraries. Why aren't getopt and optparse enough? ====================================== One argument against adding argparse is that thare are "already two different option parsing modules in the standard library" [9]_. The following is a list of features provided by argparse but not present in getopt or optparse: * While it is true there are two *option* parsing libraries, there are no full command line parsing libraries -- both getopt and optparse support only options and have no support for positional arguments. The argparse module handles both, and as a result, is able to generate better help messages, avoiding redundancies like the ``usage=`` string usually required by optparse. * The argparse module values practicality over purity. Thus, argparse allows required options and customization of which characters are used to identify options, while optparse explicitly states "the phrase 'required option' is self-contradictory" and that the option syntaxes ``-pf``, ``-file``, ``+f``, ``+rgb``, ``/f`` and ``/file`` "are not supported by optparse, and they never will be". * The argparse module allows options to accept a variable number of arguments using ``nargs='?'``, ``nargs='*'`` or ``nargs='+'``. The optparse module provides an untested recipe for some part of this functionality [10]_ but admits that "things get hairy when you want an option to take a variable number of arguments." * The argparse module supports subcommands, where a main command line parser dispatches to other command line parsers depending on the command line arguments. This is a common pattern in command line interfaces, e.g. ``svn co`` and ``svn up``. Why isn't the functionality just being added to optparse? ========================================================= Clearly all the above features offer improvements over what is available through optparse. A reasonable question then is why these features are not simply provided as patches to optparse, instead of introducing an entirely new module. In fact, the original development of argparse intended to do just that, but because of various fairly constraining design decisions of optparse, this wasn't really possible. Some of the problems included: * The optparse module exposes the internals of its parsing algorithm. In particular, ``parser.largs`` and ``parser.rargs`` are guaranteed to be available to callbacks [11]_. This makes it extremely difficult to improve the parsing algorithm as was necessary in argparse for proper handling of positional arguments and variable length arguments. For example, ``nargs='+'`` in argparse is matched using regular expressions and thus has no notion of things like ``parser.largs``. * The optparse extension APIs are extremely complex. For example, just to use a simple custom string-to-object conversion function, you have to subclass ``Option``, hack class attributes, and then specify your custom option type to the parser, like this:: class MyOption(Option): TYPES = Option.TYPES + ("mytype",) TYPE_CHECKER = copy(Option.TYPE_CHECKER) TYPE_CHECKER["mytype"] = check_mytype parser = optparse.OptionParser(option_class=MyOption) parser.add_option("-m", type="mytype") For comparison, argparse simply allows conversion functions to be used as ``type=`` arguments directly, e.g.:: parser = argparse.ArgumentParser() parser.add_option("-m", type=check_mytype) But given the baroque customization APIs of optparse, it is unclear how such a feature should interact with those APIs, and it is quite possible that introducing the simple argparse API would break existing custom Option code. * Both optparse and argparse parse command line arguments and assign them as attributes to an object returned by ``parse_args``. However, the optparse module guarantees that the ``take_action`` method of custom actions will always be passed a ``values`` object which provides an ``ensure_value`` method [12]_, while the argparse module allows attributes to be assigned to any object, e.g.:: foo_object = ... parser.parse_args(namespace=foo_object) foo_object.some_attribute_parsed_from_command_line Modifying optparse to allow any object to be passed in would be difficult because simply passing the ``foo_object`` around instead of a ``Values`` instance will break existing custom actions that depend on the ``ensure_value`` method. Because of issues like these, which made it unreasonably difficult for argparse to stay compatible with the optparse APIs, argparse was developed as an independent module. Given these issues, merging all the argparse features into optparse with no backwards incompatibilities seems unlikely. Deprecation of optparse ======================= Because all of optparse's features are available in argparse, the optparse module will be deprecated. The following very conservative deprecation strategy will be followed: * Python 2.7+ and 3.2+ -- The following note will be added to the optparse documentation: The optparse module is deprecated, and has been replaced by the argparse module. * Python 2.7+ -- If the Python 3 compatibility flag, ``-3``, is provided at the command line, then importing optparse will issue a DeprecationWarning. Otherwise no warnings will be issued. * Python 3.2 (estimated Jun 2010) -- Importing optparse will issue a PendingDeprecationWarning, which is not displayed by default. * Python 3.3 (estimated Jan 2012) -- Importing optparse will issue a PendingDeprecationWarning, which is not displayed by default. * Python 3.4 (estimated Jun 2013) -- Importing optparse will issue a DeprecationWarning, which *is* displayed by default. * Python 3.5 (estimated Jan 2015) -- The optparse module will be removed. Though this is slower than the usual deprecation process, with two releases of PendingDeprecationWarnings instead of the usual one, it seems prudent to avoid producing any casually visible Deprecation warnings until Python 3.X has had some additional time to attract developers. Updates to getopt documentation =============================== The getopt module will not be deprecated. However, its documentation will be updated to point to argparse in a couple of places. At the top of the module, the following note will be added: The getopt module is a parser for command line options whose API is designed to be familiar to users of the C getopt function. Users who are unfamiliar with the C getopt function or who would like to write less code and get better help and error messages should consider using the argparse module instead. Additionally, after the final getopt example, the following note will be added: Note that an equivalent command line interface could be produced with less code by using the argparse module:: import argparse if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('-o', '--output') parser.add_argument('-v', dest='verbose', action='store_true') args = parser.parse_args() # ... do something with args.output ... # ... do something with args.verbose .. Deferred: string formatting =========================== The argparse module supports Python from 2.3 up through 3.2 and as a result relies on traditional ``%(foo)s`` style string formatting. It has been suggested that it might be better to use the new style ``{foo}`` string formatting [13]_. There was some discussion about how best to do this for modules in the standard library [14]_ and several people are developing functions for automatically converting %-formatting to {}-formatting [15]_ [16]_. When one of these is added to the standard library, argparse will use them to support both formatting styles. Rejected: getopt compatibility methods ====================================== Previously, when this PEP was suggesting the deprecation of getopt as well as optparse, there was some talk of adding a method like:: ArgumentParser.add_getopt_arguments(options[, long_options]) However, this method will not be added for a number of reasons: * The getopt module is not being deprecated, so there is less need. * This method would not actually ease the transition for any getopt users who were already maintaining usage messages, because the API above gives no way of adding help messages to the arguments. * Some users of getopt consider it very important that only a single function call is necessary. The API above does not satisfy this requirement because both ``ArgumentParser()`` and ``parse_args()`` must also be called. Out of Scope: Various Feature Requests ====================================== Several feature requests for argparse were made in the discussion of this PEP: * Support argument defaults from environment variables * Support argument defaults from configuration files * Support "foo --help subcommand" in addition to the currently supported "foo subcommand --help" These are all reasonable feature requests for the argparse module, but are out of the scope of this PEP, and have been redirected to the argparse issue tracker. Discussion: sys.err and sys.exit ================================ There were some concerns that argparse by default always writes to ``sys.err`` and always calls ``sys.exit`` when invalid arguments are provided. This is the desired behavior for the vast majority of argparse use cases which revolve around simple command line interfaces. However, in some cases, it may be desirable to keep argparse from exiting, or to have it write its messages to something other than ``sys.err``. These use cases can be supported by subclassing ``ArgumentParser`` and overriding the ``exit`` or ``_print_message`` methods. The latter is an undocumented implementation detail, but could be officially exposed if this turns out to be a common need. References ========== .. [1] argparse (http://code.google.com/p/argparse/) .. [2] getopt (http://docs.python.org/library/getopt.html) .. [3] optparse (http://docs.python.org/library/optparse.html) .. [4] argparse in IPython (http://mail.scipy.org/pipermail/ipython-dev/2009-April/005102.html) .. [5] argparse in Debian (http://packages.debian.org/search?keywords=python-argparse) .. [6] 2007-01-03 request for argparse in the standard library (http://mail.python.org/pipermail/python-list/2007-January/592646.html) .. [7] 2009-06-09 request for argparse in the standard library (http://bugs.python.org/issue6247) .. [8] 2009-09-10 request for argparse in the standard library (http://mail.python.org/pipermail/stdlib-sig/2009-September/000342.html) .. [9] Fredrik Lundh response to [6]_ (http://mail.python.org/pipermail/python-list/2007-January/592675.html) .. [10] optparse variable args (http://docs.python.org/library/optparse.html#callback-example-6-variable-arguments) .. [11] parser.largs and parser.rargs (http://docs.python.org/library/optparse.html#how-callbacks-are-called) .. [12] take_action values argument (http://docs.python.org/library/optparse.html#adding-new-actions) .. [13] use {}-formatting instead of %-formatting (http://bugs.python.org/msg89279) .. [14] transitioning from % to {} formatting (http://mail.python.org/pipermail/python-dev/2009-September/092326.html) .. [15] Vinay Sajip's %-to-{} converter (http://gist.github.com/200936) .. [16] Benjamin Peterson's %-to-{} converter (http://bazaar.launchpad.net/~gutworth/+junk/mod2format/files) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From tjreedy at udel.edu Sat Oct 24 21:18:50 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Oct 2009 15:18:50 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <200910250218.29454.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <4AE2AA9A.6090107@gmail.com> <200910250218.29454.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Sat, 24 Oct 2009 08:12:26 pm Martin (gzlist) wrote: >> There's a different proposed meaning for `set.get` that's been >> discussed on python-dev before: >> >> For that case, I think the OP was mistaken mistaken to reject using a dict. He had objects with a key and wanted an index based on that key. > That thread has an example of a use-case for extracting an item from a > set, given the item itself: interning. > > The current Python idiom for interning is to use a dict: > > # store the canonical version > _cache[item] = item > > # retrieve the interned version > item = _cache[item] This is an interesting use case as it turns on the difference between mathematics, where value is identity, and informatics, where the two concepts split. The above is meaningless in math, but *is* meaningful in informatics, where different objects can have the same set-membership value. For Python's builtin sets, set-membership 'value' is based on hash comparisons followed by equality comparison, rather than identity. The purpose of this is to make them more like mathematical sets. But because of the value-identity distinction, they are not exactly like mathematical sets. Given that set()s implicitly map equivalence classes (which are not Python objects) to representitive members of that class (which are), then I agree and would even argue that there should be method of retrieving the one representative given any member. (The above assumes that .__eq__ is properly defined for potential members so that they are properly partitioned into equivalence classes. This is not true when Decimals are mixed with other number classes.) A counter-argument is that the implicit mapping is an implementation detail. A bit-mapped set class for a finite range of ints would not have this mapping. So an ABC for sets probably should not include such a method. > It has been argued that such interning is better done by sets rather > than dicts, since this will save a pointer per item (plus any > additional overhead dicts may have over that of sets). If this argument > is accepted, then we want a fast, efficient operation to perform this: > > def get(self, item): > """Return an element of the set equal to item. > Useful for retrieving the canonical version of an element > and for interning. > """ > for x in self: > if x == item: > return x > raise ValueError('item not in set') > > > The above is O(N); ideally it should be O(1). > > Normally we don't care about identity, only equality, so if x and item > above are equal we are indifferent about which one we use. But > interning is a real example of a use-case where we do care about > identity. Arguably it is inefficient and wasteful to use a dict for > interning when sets could be just as fast while using less memory. > > The other proposed semantics for set.get() are to retrieve an arbitrary > element. It must be arbitrary, because elements in a set are unordered > and unkeyed. This gives us the semantics of pop() without removing the > element: > > def get(self): > """Return an arbitrary element of the set without removing it.""" > for x in self: > return x > raise ValueError("empty set") > > This is a frequent request, but I still don't know what the use-case is. > > If you'll excuse me stating the obvious, both of these can easily be > written as helper-functions. The main arguments for making them methods > are: > > (1) The first one is needlessly O(N) when it could be O(1). > > (2) The second one is apparently a common need (although I personally > have never needed it), forcing people to re-invent the wheel, sometimes > incorrectly or inefficiently, e.g.: > > def get(aset): > for element in aset: > pass > return element Terry Jan Reedy From martin at v.loewis.de Sat Oct 24 21:24:47 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Oct 2009 21:24:47 +0200 Subject: [Python-Dev] Clean up Python/thread_*.h ? In-Reply-To: References: Message-ID: <4AE3547F.40800@v.loewis.de> > Making a decision and removing the files considered unnecessary would clarify > what platforms still are/should be supported. I think any such removal should go through the PEP 11 process. Put a #error into the files for 3.2, and a removal notice into the PEP, then remove them in 3.3. As for OS/2: this should probably stay. Regards, Martin From tjreedy at udel.edu Sat Oct 24 21:37:13 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Oct 2009 15:37:13 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: Guido van Rossum wrote: > On Sat, Oct 24, 2009 at 10:34 AM, Alexander Belopolsky > wrote: >> To me, however, a set seems to be a container that is a specialization >> of a dict with values and keys being the same. > > That's absurd; the mapping provides nothing useful. Given Alexander's premise, I agree with your response. But his premise is wrong. Python's current builtin set class maps abstract equivalence classes to representative members. And this *is* useful. Mapping arbitrary members of such classes to representative members, sometimes called 'canonical', is a standard problem/goal in math. String interning is an application of this idea. Terry Jan Reedy From tjreedy at udel.edu Sat Oct 24 21:37:27 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Oct 2009 15:37:27 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: Alexander Belopolsky wrote: > On Fri, Oct 23, 2009 at 6:46 PM, Steven D'Aprano wrote: >> On Sat, 24 Oct 2009 06:04:12 am Terry Reedy wrote: > .. >>> fwiw, I think the use case for this is sufficiently rare that it does >>> not need a separate method just for this purpose. >> >> And yet it keeps coming up, again and again... obviously people using >> sets in code think it has a use-case. >> > This reminds me a debate I had with Martin several years ago: > > http://bugs.python.org/issue1507011 > > Here is the essence: > > AB> I disagree with Martin. I think interning is a set > AB> operation and it is unfortunate that set API does not > AB> support it directly. > > ML> I disagree with Alexander's last remark in several respects: > ML> set is indeed a container, and there is a way to get the > ML> elements back (namely, by enumerating them, or picking an > ML> arbitrary element for removal). There is no operation to check, > ML> on insertion of E, what the the prior element equal to E was > ML> (if any); there is only a check to determine whether E is in the > ML> set. The operation "give me the member equal but not identical > ML> to E" is conceptually a lookup operation; the mathematical set > ML> construct has no such operation, and the Python set models > ML> it closely. IOW, set is *not* a dict with key==value. > > To me, however, a set seems to be a container that is a specialization > of a dict with values and keys being the same. As I explained in response to Steven, set()s *implicitly* map of abstract, non-object equivalence classes to a concrete, representative object/member of that class. > In this model, a > get() method, or even a __getitem__ with s[k] is k, is only natural. No, if key and value were the same thing, the get method would be nonesense, as Guido said. But since they are not, since the implict key is an abstract class, retrieving the representative, perhaps canonical object given *any* member *is* natural. Being able to do so is also a standard practice in mathematics. Terry Jan Reedy From solipsis at pitrou.net Sat Oct 24 21:41:54 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Oct 2009 19:41:54 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Clean_up_Python/thread=5F*=2Eh_=3F?= References: <4AE3547F.40800@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > Making a decision and removing the files considered unnecessary would clarify > > what platforms still are/should be supported. > > I think any such removal should go through the PEP 11 process. Put a > #error into the files for 3.2, and a removal notice into the PEP, then > remove them in 3.3. Ok, thanks! I see that AtheOS and BeOS are already marked unsupported in PEP 11. What is your opinion about the Solaris-specific thread support? Is it still necessary? Regards Antoine. From rhamph at gmail.com Sat Oct 24 22:09:41 2009 From: rhamph at gmail.com (Adam Olsen) Date: Sat, 24 Oct 2009 14:09:41 -0600 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> Message-ID: On Fri, Oct 23, 2009 at 11:04, Vitor Bosshard wrote: > I see this as being useful for frozensets as well, where you can't get > an arbitrary element easily due to the obvious lack of .pop(). I ran > into this recently, when I had a frozenset that I knew had 1 element > (it was the difference between 2 other sets), but couldn't get to that > element easily (get the pun?) item, = set_of_one -- Adam Olsen, aka Rhamphoryncus From ben+python at benfinney.id.au Sun Oct 25 01:34:35 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 25 Oct 2009 10:34:35 +1100 Subject: [Python-Dev] updated PEP 389: argparse References: Message-ID: <871vks5ptg.fsf@benfinney.id.au> Steven Bethard writes: > Discussion: sys.err and sys.exit > ================================ > There were some concerns that argparse by default always writes to > ``sys.err`` [?] Unless, I'm missing something, this should replace ?sys.err? with ?sys.stderr? throughout. -- \ ?Pinky, are you pondering what I'm pondering?? ?I think so, | `\ Brain, but why would anyone want to see Snow White and the | _o__) Seven Samurai?? ?_Pinky and The Brain_ | Ben Finney From steven.bethard at gmail.com Sun Oct 25 01:43:24 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 24 Oct 2009 16:43:24 -0700 Subject: [Python-Dev] updated PEP 389: argparse In-Reply-To: <871vks5ptg.fsf@benfinney.id.au> References: <871vks5ptg.fsf@benfinney.id.au> Message-ID: On Sat, Oct 24, 2009 at 4:34 PM, Ben Finney wrote: > Steven Bethard writes: > >> Discussion: sys.err and sys.exit >> ================================ >> There were some concerns that argparse by default always writes to >> ``sys.err`` > [?] > > Unless, I'm missing something, this should replace ?sys.err? with > ?sys.stderr? throughout. Yep, that's a typo. Thanks for the catch. It's fixed in SVN and the website: http://www.python.org/dev/peps/pep-0389/ Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From john.meinel at canonical.com Sun Oct 25 02:47:42 2009 From: john.meinel at canonical.com (John Arbash Meinel) Date: Sat, 24 Oct 2009 20:47:42 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> Message-ID: <4AE3AE3E.80402@canonical.com> Adam Olsen wrote: > On Fri, Oct 23, 2009 at 11:04, Vitor Bosshard wrote: >> I see this as being useful for frozensets as well, where you can't get >> an arbitrary element easily due to the obvious lack of .pop(). I ran >> into this recently, when I had a frozenset that I knew had 1 element >> (it was the difference between 2 other sets), but couldn't get to that >> element easily (get the pun?) > > item, = set_of_one > > Interesting. It depends a bit on the speed of tuple unpacking, but presumably that is quite fast. On my system it is pretty darn good: 0.101us "for x in s: break" 0.112us "x, = s" 0.122us "for x in s: pass" So not quite as good as the for loop, but quite close. John =:-> From aleaxit at gmail.com Sun Oct 25 05:22:03 2009 From: aleaxit at gmail.com (Alex Martelli) Date: Sat, 24 Oct 2009 21:22:03 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE3AE3E.80402@canonical.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> Message-ID: Next(s) would seem good... Alex Sent from my iPhone On Oct 24, 2009, at 6:47 PM, John Arbash Meinel wrote: > Adam Olsen wrote: >> On Fri, Oct 23, 2009 at 11:04, Vitor Bosshard >> wrote: >>> I see this as being useful for frozensets as well, where you can't >>> get >>> an arbitrary element easily due to the obvious lack of .pop(). I ran >>> into this recently, when I had a frozenset that I knew had 1 element >>> (it was the difference between 2 other sets), but couldn't get to >>> that >>> element easily (get the pun?) >> >> item, = set_of_one >> >> > > Interesting. It depends a bit on the speed of tuple unpacking, but > presumably that is quite fast. On my system it is pretty darn good: > > 0.101us "for x in s: break" > 0.112us "x, = s" > 0.122us "for x in s: pass" > > So not quite as good as the for loop, but quite close. > > John > =:-> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com From ncoghlan at gmail.com Sun Oct 25 06:48:54 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Oct 2009 15:48:54 +1000 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: <4AE3E6C6.4010008@gmail.com> Terry Reedy wrote: >> In this model, a >> get() method, or even a __getitem__ with s[k] is k, is only natural. > > No, if key and value were the same thing, the get method would be > nonesense, as Guido said. But since they are not, since the implict key > is an abstract class, retrieving the representative, perhaps canonical > object given *any* member *is* natural. Being able to do so is also a > standard practice in mathematics. To formalise this invariant to some degree: given a set "s" and two items "k1" and "k2", such that "k1 in s" and "k2 in s" and "k1 == k2", there is a single item "k" in s such that "k1 == k" and "k2 == k". If __getitem__ were to be provided by sets, then the last part of that invariant could be written "s[k1] is s[k2]". If you actually care about that aspect of the semantics for a particular application, it would be far clearer to use a full dict() and live with the additional memory usage. While I can see how saving that pointer-per-entry might make sense in some circumstances, for most I would see it as a needlessly confusing micro-optimisation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Oct 25 07:00:14 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Oct 2009 16:00:14 +1000 Subject: [Python-Dev] updated PEP 389: argparse In-Reply-To: References: Message-ID: <4AE3E96E.9030506@gmail.com> Steven Bethard wrote: > Sorry for the delay, but I've finally updated PEP 389, the argparse > PEP, based on all the feedback from python-dev. The full PEP is below, > but in short, the important changes are: > > * The getopt module will *not* be deprecated. > * In Python 2, the -3 flag will cause deprecation warnings to be > issued for optparse. > * No casually visible deprecation warnings for optparse are expected > until Jun 2013. > * Support for {}-format strings will be added when one of the > converters is approved. This can be done at any point though, so it > shouldn't hold up inclusion of argparse. The new version provides a very good summary of and response to the feedback on the previous version of the PEP. +1 on adding the module :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tjreedy at udel.edu Sun Oct 25 07:50:06 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Oct 2009 02:50:06 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> Message-ID: Alex Martelli wrote: > Next(s) would seem good... That does not work. It has to be next(iter(s)), and that has been tried and eliminated because it is significantly slower. >> Interesting. It depends a bit on the speed of tuple unpacking, but >> presumably that is quite fast. On my system it is pretty darn good: >> >> 0.101us "for x in s: break" >> 0.112us "x, = s" From dickinsm at gmail.com Sun Oct 25 09:54:46 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 25 Oct 2009 08:54:46 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots Message-ID: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> Would it be worth spending some time discussing the buildbot situation at the PyCon 2010 language summit? In the past, I've found the buildbots to be an incredibly valuable resource; especially when working with aspects of Python or C that tend to vary significantly from platform to platform (for me, this usually means floating-point, and platform math libraries, but there are surely many other things it applies to). But more recently there seem to have been some difficulties keeping a reasonable number of buildbots up and running. A secondary problem is that it can be awkward to debug some of the more obscure test failures on buildbots without having direct access to the machine. From conversations on IRC, I don't think I'm alone in wanting to find ways to make the buildbots more useful. So the question is: how best to invest time and possibly money to improve the buildbot situation (and as a result, I hope, improve the quality of Python)? What could be done to make maintenance of build slaves easier? Or to encourage interested third parties to donate hardware and time? Are there good alternatives to Buildbot that might make a difference? What do other projects do? These are probably the wrong questions; I'm hoping that a discussion would help produce the right questions, and possibly some answers. Mark From martin at v.loewis.de Sun Oct 25 10:47:10 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 10:47:10 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> Message-ID: <4AE41E9E.4050706@v.loewis.de> Mark Dickinson wrote: > Would it be worth spending some time discussing the buildbot situation > at the PyCon 2010 language summit? In the past, I've found the > buildbots to be an incredibly valuable resource; especially when > working with aspects of Python or C that tend to vary significantly > from platform to platform (for me, this usually means floating-point, > and platform math libraries, but there are surely many other things it > applies to). But more recently there seem to have been some > difficulties keeping a reasonable number of buildbots up and running. > A secondary problem is that it can be awkward to debug some of the > more obscure test failures on buildbots without having direct access > to the machine. From conversations on IRC, I don't think I'm alone in > wanting to find ways to make the buildbots more useful. These are actually two issues: a) where do we get buildbot hardware and operators? b) how can we reasonably debug problems occurring on buildbots For a), I think we can solve this only by redundancy, i.e. create more build slaves, hoping that a sufficient number would be up at any point in time. So: what specific kinds of buildbots do you think are currently lacking? A call for volunteers will likely be answered quickly. > So the question is: how best to invest time and possibly money to > improve the buildbot situation (and as a result, I hope, improve the > quality of Python)? I don't think money will really help (I'm skeptical in general that money helps in open source projects). As for time: "buildbot scales", meaning that the buildbot slave admins will all share the load, being responsible only for their own slaves. On the master side: would you be interested in tracking slave admins? > What could be done to make maintenance of build > slaves easier? This is something that only the slave admins can answer. I don't think it's difficult - it's just that people are really unlikely to contribute to the same thing over a period of five years at a steady rate. So we need to make sure to find replacements when people drop out. > Or to encourage interested third parties to donate > hardware and time? Again: I think a call for volunteers would do (Steve, if you are reading this, please hold back just a few days before actually making such a call :-) > Are there good alternatives to Buildbot that might > make a difference? I think people have started working on such a thing. There are certainly alternatives; I'm fairly skeptical that they are *good* alternatives (but then, I'm the one who set up the buildbot installation in the first place). > What do other projects do? I think that's really difficult to compare, since their testing often has a very different scope. I think CruiseControl is widely used. > These are probably the wrong questions; I'm hoping that a discussion > would help produce the right questions, and possibly some answers. I think these are good questions - just not for the summit. Setting up such a system is, conceptually, easy. It's also just a little work to set it up initially; the difficult part then is to keep it running (and no, a system where anybody can just post test results at any time without prior registration is *still* difficult to keep running). The source of the problem is that such a system can degrade without anybody taking action. If the web server's hard disk breaks down, people panic and look for a solution quickly. If the source control is down, somebody *will* "volunteer" to fix it. If the automated build system produces results less useful, people will worry, but not take action. Regards, Martin From andymac at bullseye.apana.org.au Sun Oct 25 12:00:03 2009 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Sun, 25 Oct 2009 22:00:03 +1100 Subject: [Python-Dev] Clean up Python/thread_*.h ? In-Reply-To: <4AE3547F.40800@v.loewis.de> References: <4AE3547F.40800@v.loewis.de> Message-ID: <4AE42FB3.9090606@bullseye.andymac.org> Martin v. L?wis wrote: >> Making a decision and removing the files considered unnecessary would clarify >> what platforms still are/should be supported. > > I think any such removal should go through the PEP 11 process. Put a > #error into the files for 3.2, and a removal notice into the PEP, then > remove them in 3.3. > > As for OS/2: this should probably stay. Yes please (to keeping the OS/2 thread support as-is). Andrew. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac at pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From solipsis at pitrou.net Sun Oct 25 13:16:26 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Oct 2009 12:16:26 +0000 (UTC) Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> Message-ID: > For a), I think we can solve this only by redundancy, i.e. create more > build slaves, hoping that a sufficient number would be up at any point > in time. We are already doing this, aren't we? http://www.python.org/dev/buildbot/3.x/ It doesn't seem to work very well, it's a bit like a Danaides vessel. > The source of the problem is that such a system can degrade without > anybody taking action. If the web server's hard disk breaks down, people > panic and look for a solution quickly. If the source control is down, > somebody *will* "volunteer" to fix it. If the automated build system > produces results less useful, people will worry, but not take action. Well, to be fair, buildbots breaking also happens much more frequently (perhaps one or two orders of magnitude) than the SVN server or the Web site going down. Maintaining them looks like a Sisyphean task, and nobody wants that. I don't know what kind of machines are the current slaves, but if they are 24/7 servers, isn't it a bit surprising that the slaves would go down so often? Is the buildbot software fragile? Does it require a lot of (maintenance, repair) work from the slave owners? From ctb at msu.edu Sun Oct 25 13:48:16 2009 From: ctb at msu.edu (C. Titus Brown) Date: Sun, 25 Oct 2009 05:48:16 -0700 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> Message-ID: <20091025124816.GA20329@idyll.org> On Sun, Oct 25, 2009 at 08:54:46AM +0000, Mark Dickinson wrote: > Would it be worth spending some time discussing the buildbot situation > at the PyCon 2010 language summit? In the past, I've found the > buildbots to be an incredibly valuable resource; especially when > working with aspects of Python or C that tend to vary significantly > from platform to platform (for me, this usually means floating-point, > and platform math libraries, but there are surely many other things it > applies to). But more recently there seem to have been some > difficulties keeping a reasonable number of buildbots up and running. > A secondary problem is that it can be awkward to debug some of the > more obscure test failures on buildbots without having direct access > to the machine. From conversations on IRC, I don't think I'm alone in > wanting to find ways to make the buildbots more useful. > > So the question is: how best to invest time and possibly money to > improve the buildbot situation (and as a result, I hope, improve the > quality of Python)? What could be done to make maintenance of build > slaves easier? Or to encourage interested third parties to donate > hardware and time? Are there good alternatives to Buildbot that might > make a difference? What do other projects do? > > These are probably the wrong questions; I'm hoping that a discussion > would help produce the right questions, and possibly some answers. [ x-posting to testing-in-python; please redirect followups to one list or the other! ] Hi Mark, a few bits of information... --- I have a set of VM machines running some "core" build archs -- Linux, Mac OS X, Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose. I am happy to give out remote admin access to a few people. This infrastructure is also going to increase slowly as I build up my lab's internal network. I'm giving Tarek an account on my Linux box later today to serve as a build slave for Distribute. -- More machines, and more esoteric machines, will be coming online as Snakebite unfolds. Trent Nelson (Snakepit master) has been finishing up some consulting work and is going to dedicate his time to it starting in November. This means more 64 bit, bigmem, and "weird" archs, with full login access. --- I've also been working on a buildbot alternative that I'm calling pony-build. pony-build is based on a client-push architecture in which client machines do builds and push results to the master, which then acts as a record-keeper rather than a slave driver. The result is a less coordinated but (AFAICT) much less fragile continuous integration system. I'm hoping to give a talk at PyCon on the differences, and there will be a sprint on pony-build + pyhton-dev at PyCon, regardless. The current status of pony-build is "functional but ugly inside". In particular, the data model is horrible, and the internal API needs much more fleshing out. Nonetheless, my server has been running for two months or so, and you can look at the results here, http://lyorn.idyll.org/ctb/pb-dev/ The most fully-fleshed out set of build clients is for pygr, http://lyorn.idyll.org/ctb/pb-dev/pygr/ but you can view daily build results for Python 2.7 at http://lyorn.idyll.org/ctb/pb-dev/python/ with an exhaustive list here http://lyorn.idyll.org/ctb/pb-dev/python/show_all (and why the heck am I sorting in reverse date order, anyway?!) The most interesting (?) part of pony-build right now is the client config, which I'm struggling to make simple and potentially universal enough to serve under buildbot as well: http://github.com/ctb/pony-build/blob/master/client/build-cpython (see 'commands' list). The most *exciting* part of pony-build, apart from the always-riveting spectacle of "titus rediscovering problems that buildbot solved 5 years ago", is the loose coupling of recording server to the build slaves and build reporters. My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) -- If you're interested in bashing on, contributing to, or helping figure out what color the pony-build main screen should be, contact me off-list; I'm reluctant to spam up python-dev or testing-in-python. That having been said, the results of taking it and trying it out -- you can post results to my own recording server at http://lyorn.idyll.org/ctb/pb-dev/xmlrpc -- would be most welcome. Once I fix the data model, code collaboration will be much more feasible, too. --- cheers, --titus -- C. Titus Brown, ctb at msu.edu From exarkun at twistedmatrix.com Sun Oct 25 14:50:18 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 25 Oct 2009 13:50:18 -0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> Message-ID: <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> On 12:16 pm, solipsis at pitrou.net wrote: > >>For a), I think we can solve this only by redundancy, i.e. create more >>build slaves, hoping that a sufficient number would be up at any point >>in time. > >We are already doing this, aren't we? >http://www.python.org/dev/buildbot/3.x/ > >It doesn't seem to work very well, it's a bit like a Danaides vessel. >>The source of the problem is that such a system can degrade without >>anybody taking action. If the web server's hard disk breaks down, >>people >>panic and look for a solution quickly. If the source control is down, >>somebody *will* "volunteer" to fix it. If the automated build system >>produces results less useful, people will worry, but not take action. > >Well, to be fair, buildbots breaking also happens much more frequently >(perhaps one or two orders of magnitude) than the SVN server or the Web >site going down. Maintaining them looks like a Sisyphean task, and >nobody >wants that. Perhaps this is a significant portion of the problem. Maintaining a build slave is remarkably simple and easy. I maintain about half a dozen slaves and spend at most a few minutes a month operating them. Actually setting one up in the first place might take a bit longer, since it involves installing the necessary software and making sure everything's set up right, but the actual slave configuration itself is one command: buildbot create-slave Perhaps this will help dispel the idea that it is a serious undertaking to operate a slave. The real requirement which some people may find challenging is that the slave needs to operate on a host which is actually online almost all of the time. If you don't such a machine, then there's little point offering to host a slave. >I don't know what kind of machines are the current slaves, but if they >are 24/7 servers, isn't it a bit surprising that the slaves would go >down >so often? Is the buildbot software fragile? Does it require a lot of >(maintenance, repair) work from the slave owners? As I have no specific experience maintaining any of the CPython build slaves, I can't speak to any maintenance issues which these slaves have encountered. I would expect that they are as minimal as the issues I have encountered maintaining slaves for other projects, but perhaps this is wrong. I do recall that there were some win32 issues (discussed on this list, I think) quite a while back, but I think those were resolved. I haven't heard of any other issues since then. If there are some, perhaps the people who know about them could raise them and we could try to figure out how to resolve them. Jean-Paul From exarkun at twistedmatrix.com Sun Oct 25 15:05:12 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 25 Oct 2009 14:05:12 -0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE41E9E.4050706@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> Message-ID: <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> On 09:47 am, martin at v.loewis.de wrote: >Mark Dickinson wrote: >>Would it be worth spending some time discussing the buildbot situation >>at the PyCon 2010 language summit? In the past, I've found the >>buildbots to be an incredibly valuable resource; especially when >>working with aspects of Python or C that tend to vary significantly >>from platform to platform (for me, this usually means floating-point, >>and platform math libraries, but there are surely many other things it >>applies to). But more recently there seem to have been some >>difficulties keeping a reasonable number of buildbots up and running. >>A secondary problem is that it can be awkward to debug some of the >>more obscure test failures on buildbots without having direct access >>to the machine. From conversations on IRC, I don't think I'm alone in >>wanting to find ways to make the buildbots more useful. > >These are actually two issues: >a) where do we get buildbot hardware and operators? >b) how can we reasonably debug problems occurring on buildbots > >For a), I think we can solve this only by redundancy, i.e. create >more build slaves, hoping that a sufficient number would be up at >any point in time. > >So: what specific kinds of buildbots do you think are currently >lacking? >A call for volunteers will likely be answered quickly. >>So the question is: how best to invest time and possibly money to >>improve the buildbot situation (and as a result, I hope, improve the >>quality of Python)? > >I don't think money will really help (I'm skeptical in general that >money helps in open source projects). As for time: "buildbot scales", >meaning that the buildbot slave admins will all share the load, being >responsible only for their own slaves. I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. Second, there are a number of active BuildBot developers. One of them has even recently taken a contract from Mozilla to implement some non- trivial BuildBot enhancements. I think it very likely that he would consider taking such a contract from the PSF for whatever enhancements would help out the CPython buildbot. >On the master side: would you be interested in tracking slave admins? >>What could be done to make maintenance of build >>slaves easier? > >This is something that only the slave admins can answer. I don't think >it's difficult - it's just that people are really unlikely to >contribute >to the same thing over a period of five years at a steady rate. So we >need to make sure to find replacements when people drop out. This is a good argument for VMs. It's certainly *possible* to chase an ever changing set of platforms, but it strikes me as something of a waste of time. >The source of the problem is that such a system can degrade without >anybody taking action. If the web server's hard disk breaks down, >people panic and look for a solution quickly. If the source control >is down, somebody *will* "volunteer" to fix it. If the automated >build system produces results less useful, people will worry, but >not take action. To me, that raises the question of why people aren't more concerned with the status of the build system. Shouldn't developers care if the code they're writing works or not? Jean-Paul From jnoller at gmail.com Sun Oct 25 15:10:31 2009 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 25 Oct 2009 10:10:31 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025124816.GA20329@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> Message-ID: <4222a8490910250710i6e3d2959s5da2ddf6f53a9fa4@mail.gmail.com> On Sun, Oct 25, 2009 at 8:48 AM, C. Titus Brown wrote: > [ x-posting to testing-in-python; please redirect followups to one list or > the other! ] > > Hi Mark, > > a few bits of information... > > --- > > I have a set of VM machines running some "core" build archs -- Linux, Mac OS X, > Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose. > I am happy to give out remote admin access to a few people. ?This > infrastructure is also going to increase slowly as I build up my lab's internal > network. > > I'm giving Tarek an account on my Linux box later today to serve as a build > slave for Distribute. Just to add to what titus said; I'm trying to price out a decent Desktop machine with enough ram/disk/cpu to run VMWare ESXi and serve a variety of virtual machines. I had planned on (ab)using the free MSDN license Microsoft provided to get a variety of platforms up and running. The end goal (since I have excess bandwidth and cooling where I live) would be to maintain this box as a series of buildslaves python-dev would have near unlimited shell/remote desktop access to. The nice thing about this would be that once the initial cost was sunk for the machine itself, and making all the virtual machines, in theory the machine could run a set of "common" virtual machines all the time, with a set of virtual machines on standby if someone needed to debug a less common problem. Yes, this is a mini-snakebite concept. Right now the main blocker is funding and time - that and I need to spec something that doesn't sound like a jet engine ;) jesse From exarkun at twistedmatrix.com Sun Oct 25 15:13:17 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 25 Oct 2009 14:13:17 -0000 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: <20091025124816.GA20329@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> Message-ID: <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> On 12:48 pm, ctb at msu.edu wrote: > >[snip] > >The most *exciting* part of pony-build, apart from the always-riveting >spectacle of "titus rediscovering problems that buildbot solved 5 years >ago", >is the loose coupling of recording server to the build slaves and build >reporters. My plan is to enable a simple and lightweight XML-RPC >and/or >REST-ish interface for querying the recording server from scripts or >other Web >sites. This has Brett aquiver with anticipation, I gather -- no more >visual >inspection of buildbot waterfall pages ;) BuildBot has an XML-RPC interface. So Brett can probably do what he wants with BuildBot right now. Jean-Paul From ssteinerx at gmail.com Sun Oct 25 15:27:55 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 25 Oct 2009 10:27:55 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE41E9E.4050706@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> Message-ID: <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> On Oct 25, 2009, at 5:47 AM, Martin v. L?wis wrote: > > These are actually two issues: > a) where do we get buildbot hardware and operators? I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of having buildbots come to life, report to the mother ship, do the build, then go away 'till next time they're required. S From ssteinerx at gmail.com Sun Oct 25 15:35:14 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 25 Oct 2009 10:35:14 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> Message-ID: <110EE89E-8313-4E08-8861-7772D0F2D722@gmail.com> On Oct 25, 2009, at 9:50 AM, exarkun at twistedmatrix.com wrote: > Actually setting one up in the first place might take a bit longer, > since it involves installing the necessary software and making sure > everything's set up right, but the actual slave configuration itself > is one command: > > buildbot create-slave password> I have written a Fabric script for the distutils-buildbot project (on bitbucket, under Tarek) that puts everything necessary up onto an Ubuntu server, runs all the build steps, and fires up the buildbot. Obviously it will have to be modified to correctly configure other types of servers but the implementation should be fairly trivial for someone who could have done it by hand in the first place. Once it's done, it's in the script and may require an occasional tweak but not much more. The next step is to have the slaves themselves created in the cloud, fired up and then report to the mother ship that they're available. This last step is the one that doesn't seem to be supported by the current system. Thanks, S From solipsis at pitrou.net Sun Oct 25 15:36:51 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Oct 2009 14:36:51 +0000 (UTC) Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> Message-ID: twistedmatrix.com> writes: > > To me, that raises the question of why people aren't more concerned with > the status of the build system. Shouldn't developers care if the code > they're writing works or not? The fact that we ask questions and publicly express worries should hint that we /are/ concerned :-) However, being mostly developers rather than system admins, and not knowing anything about the details of how buildbot does its work (not to mention the details of this or that particular buildslave and slave owner), makes us (at least me) quite clueless when faced with a buildbot-not-working-as-expected problem. Regards Antoine. From ssteinerx at gmail.com Sun Oct 25 15:37:41 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 25 Oct 2009 10:37:41 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> Message-ID: On Oct 25, 2009, at 10:05 AM, exarkun at twistedmatrix.com wrote: > First, there are now a multitude of cloud hosting providers which > will operate a slave machine for you. BuildBot has even begun to > support this deployment use-case by allowing you to start up and > shut down vms on demand to save on costs. Amazon's EC2 service is > supported out of the box in the latest release. I have been working to expand this support to Rackspace's Cloud Servers as well. S From foom at fuhm.net Sun Oct 25 18:09:11 2009 From: foom at fuhm.net (James Y Knight) Date: Sun, 25 Oct 2009 13:09:11 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> Message-ID: <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> On Oct 25, 2009, at 2:50 AM, Terry Reedy wrote: > Alex Martelli wrote: >> Next(s) would seem good... > > That does not work. It has to be next(iter(s)), and that has been > tried and eliminated because it is significantly slower. But who cares about the speed of getting an arbitrary element from a set? How can it *possibly* be a problem in a real program? If you want to optimize python, this operation is certainly not the right place to start... James From p.f.moore at gmail.com Sun Oct 25 18:47:54 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 25 Oct 2009 17:47:54 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> Message-ID: <79990c6b0910251047s3edb0d73mfdad918b04c68b1@mail.gmail.com> 2009/10/25 : > Perhaps this is a significant portion of the problem. ?Maintaining a build > slave is remarkably simple and easy. ?I maintain about half a dozen slaves > and spend at most a few minutes a month operating them. Actually setting one > up in the first place might take a bit longer, since it involves installing > the necessary software and making sure everything's set up right, but the > actual slave configuration itself is one command: > > ?buildbot create-slave > > Perhaps this will help dispel the idea that it is a serious undertaking to > operate a slave. > > The real requirement which some people may find challenging is that the > slave needs to operate on a host which is actually online almost all of the > time. ?If you don't such a machine, then there's little point offering to > host a slave. I have been seriously considering setting up one or more buildslaves for a while now. However, my biggest issue is that they would be running as VMs on my normal PC, which means that it's the issue of keeping them continually online that hurts me. If I could (say) just fire the slaves up for a set period, or fire them up, have them do a build and report back, and then shut down, that would make my life easier (regular activities rather than ongoing sysadmin works better for me). It sounds like a buildslave isn't really what I should be looking at. Maybe Titus' push model pony-build project would make more sense for me. Paul. From exarkun at twistedmatrix.com Sun Oct 25 18:59:47 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 25 Oct 2009 17:59:47 -0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <79990c6b0910251047s3edb0d73mfdad918b04c68b1@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <79990c6b0910251047s3edb0d73mfdad918b04c68b1@mail.gmail.com> Message-ID: <20091025175947.11571.1887104266.divmod.xquotient.1569@localhost.localdomain> On 05:47 pm, p.f.moore at gmail.com wrote: >2009/10/25 : >>Perhaps this is a significant portion of the problem. ?Maintaining a >>build >>slave is remarkably simple and easy. ?I maintain about half a dozen >>slaves >>and spend at most a few minutes a month operating them. Actually >>setting one >>up in the first place might take a bit longer, since it involves >>installing >>the necessary software and making sure everything's set up right, but >>the >>actual slave configuration itself is one command: >> >>?buildbot create-slave >password> >> >>Perhaps this will help dispel the idea that it is a serious >>undertaking to >>operate a slave. >> >>The real requirement which some people may find challenging is that >>the >>slave needs to operate on a host which is actually online almost all >>of the >>time. ?If you don't such a machine, then there's little point offering >>to >>host a slave. > >I have been seriously considering setting up one or more buildslaves >for a while now. However, my biggest issue is that they would be >running as VMs on my normal PC, which means that it's the issue of >keeping them continually online that hurts me. > >If I could (say) just fire the slaves up for a set period, or fire >them up, have them do a build and report back, and then shut down, >that would make my life easier (regular activities rather than ongoing >sysadmin works better for me). > >It sounds like a buildslave isn't really what I should be looking at. >Maybe Titus' push model pony-build project would make more sense for >me. Maybe. I wonder if Titus' "push model" (I don't really understand this term in this context) makes sense for continuous integration at all, though. As a developer, I don't want to have access to build results across multiple platforms when someone else feels like it. I want access when *I* feel like it. Anyway, BuildBot is actually perfectly capable of dealing with this. I failed to separate my assumptions about how everyone would want to use the system from what the system is actually capable of. If you run a build slave and it's offline when a build is requested, the build will be queued and run when the slave comes back online. So if the CPython developers want to work this way (I wouldn't), then we don't need pony-build; BuildBot will do just fine. Jean-Paul From p.f.moore at gmail.com Sun Oct 25 19:15:38 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 25 Oct 2009 18:15:38 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025175947.11571.1887104266.divmod.xquotient.1569@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <79990c6b0910251047s3edb0d73mfdad918b04c68b1@mail.gmail.com> <20091025175947.11571.1887104266.divmod.xquotient.1569@localhost.localdomain> Message-ID: <79990c6b0910251115x3983334bh12d618d3016a720f@mail.gmail.com> 2009/10/25 : > If you run a build slave and it's offline when a build is requested, the > build will be queued and run when the slave comes back online. ?So if the > CPython developers want to work this way (I wouldn't), then we don't need > pony-build; BuildBot will do just fine. OK, sounds useful. If I'm offline for a while, do multiple builds get queued, or only the "last" one? If the former, I can imagine coming back to a pretty huge load if the slave breaks while I'm on holiday :-( I should look all of this up somewhere. Is there a reference to buildbot for slave maintainers? Are there any specifics for Python slaves that I should refer to? (From what I've been able to find, it seems to me that setting up a slave first requires getting things sorted with the admins, which sadly precludes experimenting to find things out - I can understand why the python admins don't want people "playing" on the live buildbot infrastructure, though :-)) Paul. From fw at deneb.enyo.de Sun Oct 25 18:57:07 2009 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 25 Oct 2009 18:57:07 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> (James Y. Knight's message of "Sun, 25 Oct 2009 13:09:11 -0400") References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> Message-ID: <87ljizxsp8.fsf@mid.deneb.enyo.de> * James Y. Knight: > On Oct 25, 2009, at 2:50 AM, Terry Reedy wrote: > >> Alex Martelli wrote: >>> Next(s) would seem good... >> >> That does not work. It has to be next(iter(s)), and that has been >> tried and eliminated because it is significantly slower. > > But who cares about the speed of getting an arbitrary element from a > set? How can it *possibly* be a problem in a real program? Hmm, perhaps when using sets as work queues? From martin at v.loewis.de Sun Oct 25 19:32:52 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 19:32:52 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> Message-ID: <4AE499D4.4010104@v.loewis.de> > I've been trying to get some feedback about firing up buildbots on Cloud > Servers for a while now and haven't had much luck. I'd love to find a > way of having buildbots come to life, report to the mother ship, do the > build, then go away 'till next time they're required. I'm not quite sure whom you have been trying to get feedback from, and can't quite picture your proposed setup from above description. In any case, ISTM that your approach isn't compatible with how buildbot works today (not sure whether you are aware of that): a build slave needs to stay connected all the time, so that the build master can trigger a build when necessary. So if your setup requires the slaves to shut down after a build, I don't think this can possibly work. Regards, Martin From ctb at msu.edu Sun Oct 25 19:38:59 2009 From: ctb at msu.edu (C. Titus Brown) Date: Sun, 25 Oct 2009 11:38:59 -0700 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE499D4.4010104@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> Message-ID: <20091025183858.GA7460@idyll.org> On Sun, Oct 25, 2009 at 07:32:52PM +0100, "Martin v. L?wis" wrote: > > I've been trying to get some feedback about firing up buildbots on Cloud > > Servers for a while now and haven't had much luck. I'd love to find a > > way of having buildbots come to life, report to the mother ship, do the > > build, then go away 'till next time they're required. > > I'm not quite sure whom you have been trying to get feedback from, and > can't quite picture your proposed setup from above description. > > In any case, ISTM that your approach isn't compatible with how buildbot > works today (not sure whether you are aware of that): a build slave > needs to stay connected all the time, so that the build master can > trigger a build when necessary. Hi Martin, it shouldn't be difficult to cobble together a build script that spins up a buildslave on EC2 and runs the tests there; I wrote something similar a few years ago for an infrequently connected home machine. --titus -- C. Titus Brown, ctb at msu.edu From martin at v.loewis.de Sun Oct 25 19:43:52 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 19:43:52 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> Message-ID: <4AE49C68.9090108@v.loewis.de> > Given Alexander's premise, I agree with your response. But his premise > is wrong. Python's current builtin set class maps abstract equivalence > classes to representative members. And this *is* useful. Mapping > arbitrary members of such classes to representative members, sometimes > called 'canonical', is a standard problem/goal in math. String interning > is an application of this idea. Right. However, this is conceptually a function. In some cases (like string interning), it is mutable (over time) and finite (at any point in time). We already have a data type that can perfectly represent mutable finite funcitons, namely dictionaries. And indeed, interning is implemented by a dictionary. What Alexander wants is that the set type also becomes useful for storing canonical representations. I find that inappropriate, since dicts already provide the one obvious way to do it. Regards, Martin From martin at v.loewis.de Sun Oct 25 19:45:54 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 19:45:54 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE3E6C6.4010008@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> Message-ID: <4AE49CE2.5050502@v.loewis.de> > If you actually care about that aspect of the semantics for a particular > application, it would be far clearer to use a full dict() and live with > the additional memory usage. While I can see how saving that > pointer-per-entry might make sense in some circumstances, for most I > would see it as a needlessly confusing micro-optimisation. That's exactly my view. Regards, Martin From exarkun at twistedmatrix.com Sun Oct 25 19:45:53 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 25 Oct 2009 18:45:53 -0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE499D4.4010104@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> Message-ID: <20091025184553.31526.2015752889.divmod.xquotient.2@localhost.localdomain> On 06:32 pm, martin at v.loewis.de wrote: >>I've been trying to get some feedback about firing up buildbots on >>Cloud >>Servers for a while now and haven't had much luck. I'd love to find a >>way of having buildbots come to life, report to the mother ship, do >>the >>build, then go away 'till next time they're required. > >I'm not quite sure whom you have been trying to get feedback from, and >can't quite picture your proposed setup from above description. > >In any case, ISTM that your approach isn't compatible with how buildbot >works today (not sure whether you are aware of that): a build slave >needs to stay connected all the time, so that the build master can >trigger a build when necessary. > >So if your setup requires the slaves to shut down after a build, I >don't >think this can possibly work. This is supported in recent versions of BuildBot with a special kind of slave: http://djmitche.github.com/buildbot/docs/0.7.11/#On_002dDemand- _0028_0022Latent_0022_0029-Buildslaves Jean-Paul From martin at v.loewis.de Sun Oct 25 19:51:26 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 19:51:26 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <87ljizxsp8.fsf@mid.deneb.enyo.de> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> Message-ID: <4AE49E2E.3090506@v.loewis.de> > Hmm, perhaps when using sets as work queues? A number of comments: - it's somewhat confusing to use a set as a *queue*, given that it won't provide FIFO semantics. - there are more appropriate and direct container structures available, including a dedicated Queue module (even though this might be doing to much with its thread-safety). - if you absolutely want to use a set as a work queue, then the .pop() method should be sufficient, right? Regards, Martin From martin at v.loewis.de Sun Oct 25 19:57:48 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 19:57:48 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> Message-ID: <4AE49FAC.70207@v.loewis.de> Antoine Pitrou wrote: >> For a), I think we can solve this only by redundancy, i.e. create more >> build slaves, hoping that a sufficient number would be up at any point >> in time. > > We are already doing this, aren't we? > http://www.python.org/dev/buildbot/3.x/ > > It doesn't seem to work very well, it's a bit like a Danaides vessel. Both true. However, it seems that Mark is unhappy with the current set of systems, so we probably need to do it again. > Well, to be fair, buildbots breaking also happens much more frequently > (perhaps one or two orders of magnitude) than the SVN server or the Web > site going down. Maintaining them looks like a Sisyphean task, and nobody > wants that. It only looks so. It is like any server management task - it takes constant effort. However, it is not Sisyphean (feeling Greek today, ain't you :-); since you actually achieve something. It's not hard to restart a buildbot when it has crashed, and it gives a warm feeling of having achieved something. > I don't know what kind of machines are the current slaves, but if they > are 24/7 servers, isn't it a bit surprising that the slaves would go down > so often? Is the buildbot software fragile? Not really. It sometimes happens that the slaves don't reconnect after a master restart, but more often, it is just a change on the slave side that breaks it (such as a reboot done to the machine, and not having the machine configured to restart the slave after the reboot). > Does it require a lot of > (maintenance, repair) work from the slave owners? On Unix, not really. On Windows, there is still the issue that sometimes, some error message pops up which you need to click away. Over several builds, you may find that you have to click away dozens of such messages. This could use some improvement. Regards, Martin From martin at v.loewis.de Sun Oct 25 20:06:49 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:06:49 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> Message-ID: <4AE4A1C9.5010002@v.loewis.de> > As I have no specific experience maintaining any of the CPython build > slaves, I can't speak to any maintenance issues which these slaves have > encountered. I would expect that they are as minimal as the issues I > have encountered maintaining slaves for other projects, but perhaps this > is wrong. I do recall that there were some win32 issues (discussed on > this list, I think) quite a while back, but I think those were resolved. > I haven't heard of any other issues since then. Only partially. One old issue was that previous builds would not complete, keeping the executable files open, preventing further runs. Buildbot is supposed to kill a build, but only kills the parent process (as it is really difficult to kill the entire process tree (*)). We work around this by explicitly killing any stale Python processes at the beginning of a new build. The remaining issue is the popups; if a process still has a popup, you can't even terminate it properly. There are two kinds of popups: system-generated ones, and CRT-generated ones. For the CRT ones, we once had a way to turn them off, but I'm not sure whether that mechanism might have been removed. For the system messages, there is a way to turn them off in the parent process. David Bolen (IIRC) had developed a patch, but I think this patch only runs on his system(s). Regards, Martin (*) it may help if Buildbot would create a Win32 job object, and then use TerminateJobObject. Contributions are welcome. From brett at python.org Sun Oct 25 20:12:05 2009 From: brett at python.org (Brett Cannon) Date: Sun, 25 Oct 2009 12:12:05 -0700 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> Message-ID: On Sun, Oct 25, 2009 at 07:13, wrote: > On 12:48 pm, ctb at msu.edu wrote: > >> >> [snip] >> >> >> The most *exciting* part of pony-build, apart from the always-riveting >> spectacle of "titus rediscovering problems that buildbot solved 5 years >> ago", >> is the loose coupling of recording server to the build slaves and build >> reporters. My plan is to enable a simple and lightweight XML-RPC and/or >> REST-ish interface for querying the recording server from scripts or other >> Web >> sites. This has Brett aquiver with anticipation, I gather -- no more >> visual >> inspection of buildbot waterfall pages ;) >> > > BuildBot has an XML-RPC interface. So Brett can probably do what he wants > with BuildBot right now. > Brett actually wants web hooks so pony-build will ping an App Engine web app when there is more data, ala PubSubHubbub. Or hell, just have pony-build have an Atom feed with updates and simply use PuSH. In other words I want to be told when there is an update, not have to poll to find out. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Oct 25 20:15:09 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:15:09 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <79990c6b0910251115x3983334bh12d618d3016a720f@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <79990c6b0910251047s3edb0d73mfdad918b04c68b1@mail.gmail.com> <20091025175947.11571.1887104266.divmod.xquotient.1569@localhost.localdomain> <79990c6b0910251115x3983334bh12d618d3016a720f@mail.gmail.com> Message-ID: <4AE4A3BD.8010002@v.loewis.de> > OK, sounds useful. If I'm offline for a while, do multiple builds get > queued, or only the "last" one? IIRC, it will only build the last one, then with a huge blame list. > If the former, I can imagine coming > back to a pretty huge load if the slave breaks while I'm on holiday > :-( If it's offline too often, I'm skeptical that it would be useful. If you report breakage after a day, then it will be difficult to attribute this to a specific commit. It is most useful to have continuous integration if error reports are instantaneous. > I should look all of this up somewhere. Is there a reference to > buildbot for slave maintainers? Are there any specifics for Python > slaves that I should refer to? Hmm. I thought I had send you this before: http://wiki.python.org/moin/BuildbotOnWindows > (From what I've been able to find, it seems to me that setting up a > slave first requires getting things sorted with the admins, which > sadly precludes experimenting to find things out - I can understand > why the python admins don't want people "playing" on the live buildbot > infrastructure, though :-)) That's really not an issue. Feel free to play as much as you want, with the live infrastructure. You can't break anything doing so (perhaps except for spamming the mailing list with faulty reports). If you then decide to withdraw your offer, that's fine as well (just make sure to notify us instead of just silently taking the slave down). Regards, Martin From solipsis at pitrou.net Sun Oct 25 21:22:20 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Oct 2009 21:22:20 +0100 Subject: [Python-Dev] Reworking the GIL Message-ID: <1256502140.5621.163.camel@localhost> Hello there, The last couple of days I've been working on an experimental rewrite of the GIL. Since the work has been turning out rather successful (or, at least, not totally useless and crashing!) I thought I'd announce it here. First I want to stress this is not about removing the GIL. There still is a Global Interpreter Lock which serializes access to most parts of the interpreter. These protected parts haven't changed either, so Python doesn't become really better at extracting computational parallelism out of several cores. Goals ----- The new GIL (which is also the name of the sandbox area I've committed it in, "newgil") addresses the following issues : 1) Switching by opcode counting. Counting opcodes is a very crude way of estimating times, since the time spent executing a single opcode can very wildly. Litterally, an opcode can be as short as a handful of nanoseconds (think something like "... is not None") or as long as a fraction of second, or even longer (think calling a heavy non-GIL releasing C function, such as re.search()). Therefore, releasing the GIL every 100 opcodes, regardless of their length, is a very poor policy. The new GIL does away with this by ditching _Py_Ticker entirely and instead using a fixed interval (by default 5 milliseconds, but settable) after which we ask the main thread to release the GIL and let another thread be scheduled. 2) GIL overhead and efficiency in contended situations. Apparently, some OSes (OS X mainly) have problems with lock performance when the lock is already taken: the system calls are heavy. This is the "Dave Beazley effect", where he took a very trivial loop, therefore made of very short opcodes and therefore releasing the GIL very often (probably 100000 times a second), and runs it in one or two threads on an OS with poor lock performance (OS X). He sees a 50% increase in runtime when using two threads rather than one, in what is admittedly a pathological case. Even on better platforms such as Linux, eliminating the overhead of many GIL acquires and releases (since the new GIL is released on a fixed time basis rather than on an opcode counting basis) yields slightly better performance (read: a smaller performance degradation :-)) when there are several pure Python computation threads running. 3) Thread switching latency. The traditional scheme merely releases the GIL for a couple of CPU cycles, and reacquires it immediately. Unfortunately, this doesn't mean the OS will automatically switch to another, GIL-awaiting thread. In many situations, the same thread will continue running. This, with the opcode counting scheme, is the reason why some people have been complaining about latency problems when an I/O thread competes with a computational thread (the I/O thread wouldn't be scheduled right away when e.g. a packet arrives; or rather, it would be scheduled by the OS, but unscheduled immediately when trying to acquire the GIL, and it would be scheduled again only much later). The new GIL improves on this by combinating two mechanisms: - forced thread switching, which means that when the switching interval is terminated (mentioned in 1) and the GIL is released, we will force any of the threads waiting on the GIL to be scheduled instead of the formerly GIL-holding thread. Which thread exactly is an OS decision, however: the goal here is not to have our own scheduler (this could be discussed but I wanted the design to remain simple :-) After all, man-years of work have been invested in scheduling algorithms by kernel programming teams). - priority requests, which is an option for a thread requesting the GIL to be scheduled as soon as possible, and forcibly (rather than any other threads). This is meant to be used by GIL-releasing methods such as read() on files and sockets. The scheme, again, is very simple: when a priority request is done by a thread, the GIL is released as soon as possible by the thread holding it (including in the eval loop), and then the thread making the priority request is forcibly scheduled (by making all other GIL-awaiting threads wait in the meantime). Implementation -------------- The new GIL is implemented using a couple of mutexes and condition variables. A {mutex, condition} pair is used to protect the GIL itself, which is a mere variable named `gil_locked` (there are a couple of other variables for bookkeeping). Another {mutex, condition} pair is used for forced thread switching (described above). Finally, a separate mutex is used for priority requests (described above). The code is in the sandbox: http://svn.python.org/view/sandbox/trunk/newgil/ The file of interest is Python/ceval_gil.h. Changes in other files are very minimal, except for priority requests which have been added at strategic places (some methods of I/O modules). Also, the code remains rather short, while of course being less trivial than the old one. NB : this is a branch of py3k. There should be no real difficulty porting it back to trunk, provided someone wants to do the job. Platforms --------- I've implemented the new GIL for POSIX and Windows (tested under Linux and Windows XP (running in a VM)). Judging by what I can read in the online MSDN docs, the Windows support should include everything from Windows 2000, and probably recent versions of Windows CE. Other platforms aren't implemented, because I don't have access to the necessary hardware. Besides, I must admit I'm not very motivated in working on niche/obsolete systems. I've e-mailed Andrew MacIntyre in private to ask him if he'd like to do the OS/2 support. Supporting a new platform is not very difficult: it's a matter of writing the 50-or-so lines of necessary platform-specific macros at the beginning of Python/ceval_gil.h. The reason I couldn't use the existing thread support (Python/thread_*.h) is that these abstractions are too poor. Mainly, they don't provide: - events, conditions or an equivalent thereof - the ability to acquire a resource with a timeout Measurements ------------ Before starting this work, I wrote ccbench (*), a little benchmark script ("ccbench" being a shorthand for "concurrency benchmark") which measures two things: - computation throughput with one or several concurrent threads - latency to external events (I use an UDP socket) when there is zero, one, or several background computation threads running (*) http://svn.python.org/view/sandbox/trunk/ccbench/ The benchmark involves several computation workloads with different GIL characteristics. By default there are 3 of them: A- one pure Python workload (computation of a number of digits of pi): that is, something which spends its time in the eval loop B- one mostly C workload where the C implementation doesn't release the GIL (regular expression matching) C- one mostly C workload where the implementation does release the GIL (bz2 compression) In the ccbench directory you will find benchmark results, under Linux, for two different systems I have here. The new GIL shows roughly similar but slightly better throughput results than the old one. And it is much better in the latency tests, especially in workload B (going down from almost a second of average latency with the old GIL, to a couple of milliseconds with the new GIL). This is the combined result of using a time-based scheme (rather than opcode-based) and of forced thread switching (rather than relying on the OS to actually switch threads when we speculatively release the GIL). As a sidenote, I might mention that single-threaded performance is not degraded at all. It is, actually, theoretically a bit better because the old ticker check in the eval loop becomes simpler; however, this goes mostly unnoticed. Now what remains to be done? Having other people test it would be fine. Even better if you have an actual multi-threaded py3k application. But ccbench results for other OSes would be nice too :-) (I get good results under the Windows XP VM but I feel that a VM is not an ideal setup for a concurrency benchmark) Of course, studying and reviewing the code is welcome. As for integrating it into the mainline py3k branch, I guess we have to answer these questions: - is the approach interesting? (we could decide that it's just not worth it, and that a good GIL can only be a dead (removed) GIL) - is the patch good, mature and debugged enough? - how do we deal with the unsupported platforms (POSIX and Windows support should cover most bases, but the fate of OS/2 support depends on Andrew)? Regards Antoine. From ssteinerx at gmail.com Sun Oct 25 20:19:05 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 25 Oct 2009 15:19:05 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE499D4.4010104@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> Message-ID: On Oct 25, 2009, at 2:32 PM, Martin v. L?wis wrote: >> I've been trying to get some feedback about firing up buildbots on >> Cloud >> Servers for a while now and haven't had much luck. I'd love to >> find a >> way of having buildbots come to life, report to the mother ship, do >> the >> build, then go away 'till next time they're required. > > I'm not quite sure whom you have been trying to get feedback from, and > can't quite picture your proposed setup from above description. I posted a couple of messages on testing-in-python, and have sent around some queries to others that I know are using buildbot type setups with various tools/platforms, not necessarily Python. > In any case, ISTM that your approach isn't compatible with how > buildbot > works today (not sure whether you are aware of that): a build slave > needs to stay connected all the time, so that the build master can > trigger a build when necessary. > > So if your setup requires the slaves to shut down after a build, I > don't > think this can possibly work. It can't possibly work within the way the Python buildbot structure currently works, as I understand it so far. What I'm implementing is less of a 'continuous integration' tool like you would use for something like Python itself, and more of a testing tool for things that have to be installed on multiple versions of Python, on multiple platforms. I don't need to know that it works on every checkin, just every once in a while I'd like to start from scratch and make sure everything still works on all supported versions of Python on all the platforms I test on and cloud servers are great for this since I'll usually only need them for an hour or so. S From solipsis at pitrou.net Sun Oct 25 20:23:07 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Oct 2009 19:23:07 +0000 (UTC) Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <79990c6b0910251047s3edb0d73mfdad918b04c68b1@mail.gmail.com> <20091025175947.11571.1887104266.divmod.xquotient.1569@localhost.localdomain> <79990c6b0910251115x3983334bh12d618d3016a720f@mail.gmail.com> <4AE4A3BD.8010002@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > If it's offline too often, I'm skeptical that it would be useful. If > you report breakage after a day, then it will be difficult to attribute > this to a specific commit. It is most useful to have continuous > integration if error reports are instantaneous. Not only, but it's useful to have a stable set of buildbots, so that when a test fails, you know whether it's a new problem caused by a recent revision, or rather an already existing problem on this platform. Regards Antoine. From martin at v.loewis.de Sun Oct 25 20:24:12 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:24:12 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> Message-ID: <4AE4A5DC.2010906@v.loewis.de> > I think that money can help in two ways in this case. > > First, there are now a multitude of cloud hosting providers which will > operate a slave machine for you. BuildBot has even begun to support > this deployment use-case by allowing you to start up and shut down vms > on demand to save on costs. Amazon's EC2 service is supported out of > the box in the latest release. Here I'm skeptical. I think we can find people donating always-online machines still; no need to throw donated money to Amazon. > Second, there are a number of active BuildBot developers. One of them > has even recently taken a contract from Mozilla to implement some non- > trivial BuildBot enhancements. I think it very likely that he would > consider taking such a contract from the PSF for whatever enhancements > would help out the CPython buildbot. That could indeed be interesting, assuming we had a clear requirement. But then, most of us can "easily" fix things in buildbot themselves - this is python-dev, after all. >> This is something that only the slave admins can answer. I don't think >> it's difficult - it's just that people are really unlikely to contribute >> to the same thing over a period of five years at a steady rate. So we >> need to make sure to find replacements when people drop out. > > This is a good argument for VMs. Not really - you still would need somebody to manage them. > It's certainly *possible* to chase an > ever changing set of platforms, but it strikes me as something of a > waste of time. Hmm - can you really get "strange" operating systems "in the cloud"? Some of the operating systems that we would like to test don't even support VMs. > To me, that raises the question of why people aren't more concerned with > the status of the build system. Shouldn't developers care if the code > they're writing works or not? I think there are two issues here: 1. some developers actually *don't* care to much whether their code works in all cases. If it fails on some strange platform they never heard of (such as "Solaris", or "Windows"), they are not bothered by the failure. Or, if they care, they still don't know what to do about the failure. 2. the buildbots sometimes report false positives. Some tests fail in a non-repeatable fashion, only on selected systems. So when you change something, the tests break, and you cannot really see how this could be possibly related to your change. Then you start ignoring these reports - both the bogus ones, and the real ones. Regards, Martin From martin at v.loewis.de Sun Oct 25 20:27:39 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:27:39 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025183858.GA7460@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> <20091025183858.GA7460@idyll.org> Message-ID: <4AE4A6AB.8020801@v.loewis.de> > it shouldn't be difficult to cobble together a build script that spins up a > buildslave on EC2 and runs the tests there; I wrote something similar a > few years ago for an infrequently connected home machine. Ok - so it would be the master running this script? Sounds reasonable to me. As for EC2 (and other cloud providers); I'm somewhat skeptical about platform coverage, also. How many different processors and operating systems can they possibly support? Regards, Martin From martin at v.loewis.de Sun Oct 25 20:33:31 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:33:31 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025184553.31526.2015752889.divmod.xquotient.2@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> <20091025184553.31526.2015752889.divmod.xquotient.2@localhost.localdomain> Message-ID: <4AE4A80B.1050107@v.loewis.de> > This is supported in recent versions of BuildBot with a special kind of > slave: > > http://djmitche.github.com/buildbot/docs/0.7.11/#On_002dDemand- > _0028_0022Latent_0022_0029-Buildslaves Interesting. Coming back to "PSF may spend money", let me say this: If somebody would volunteer to set up slaves in EC2, and operate them, I'm fairly certain that the PSF would pay the bill. It might be useful to have two people operating them, so that the knowledge and the load is shared. Regards, Martin From martin at v.loewis.de Sun Oct 25 20:35:22 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:35:22 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> Message-ID: <4AE4A87A.1060602@v.loewis.de> > I don't need to know that it works on every checkin For us, that is a fairly important requirement, though. Reports get more and more useless if they aren't instantaneous. Sometimes, people check something in just to see how the build slaves react. Regards, Martin From ssteinerx at gmail.com Sun Oct 25 20:38:00 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 25 Oct 2009 15:38:00 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE4A87A.1060602@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> <4AE4A87A.1060602@v.loewis.de> Message-ID: On Oct 25, 2009, at 3:35 PM, Martin v. L?wis wrote: >> I don't need to know that it works on every checkin > > For us, that is a fairly important requirement, though. > Reports get more and more useless if they aren't instantaneous. > Sometimes, people check something in just to see how the build > slaves react. Understood -- that's why I mentioned it. This is a different use-case. It may still have some use for Python itself, but my idea is more for testing things like libraries where the developer may only work on or have one platform and may want to test installing on other platforms and Python versions during development and/or before release. S From martin at v.loewis.de Sun Oct 25 20:42:11 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 20:42:11 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091025124816.GA20329@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> Message-ID: <4AE4AA13.8010903@v.loewis.de> > The most *exciting* part of pony-build, apart from the always-riveting > spectacle of "titus rediscovering problems that buildbot solved 5 years ago", > is the loose coupling of recording server to the build slaves and build > reporters. My plan is to enable a simple and lightweight XML-RPC and/or > REST-ish interface for querying the recording server from scripts or other Web > sites. This has Brett aquiver with anticipation, I gather -- no more visual > inspection of buildbot waterfall pages ;) If that's something that people want to have, then buildbot could also provide it, of course. Do you have a spec of the interface somewhere? Regards, Martin From martin at v.loewis.de Sun Oct 25 21:10:41 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 25 Oct 2009 21:10:41 +0100 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> Message-ID: <4AE4B0C1.5060706@v.loewis.de> > Brett actually wants web hooks so pony-build will ping an App Engine web > app when there is more data, ala PubSubHubbub. Or hell, just have > pony-build have an Atom feed with updates and simply use PuSH. In other > words I want to be told when there is an update, not have to poll to > find out. Not sure what exactly it is that Brett wants to do, but perhaps Brett could take a look at http://www.python.org/dev/buildbot/all/atom As JP says, there is also XML-RPC (at all/xmlrpc) For a true push notifications: buildbot sends messages into an IRC channel - not sure whether an App Engine App could listen to that. Regards, Martin From martin at v.loewis.de Sun Oct 25 21:19:13 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 21:19:13 +0100 Subject: [Python-Dev] Buildbot alternate renderings Message-ID: <4AE4B2C1.7080504@v.loewis.de> Triggered by the recent discussion, I looked at the changes in status display that the buildbot people have implemented recently. I added a few links into alternate renderings; see http://www.python.org/dev/buildbot/ Regards, Martin From p.f.moore at gmail.com Sun Oct 25 22:17:40 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 25 Oct 2009 21:17:40 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE499D4.4010104@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> Message-ID: <79990c6b0910251417l515d29b7iaedd2a338eb0b9d2@mail.gmail.com> 2009/10/25 "Martin v. L?wis" : >> I've been trying to get some feedback about firing up buildbots on Cloud >> Servers for a while now and haven't had much luck. ?I'd love to find a >> way of having buildbots come to life, report to the mother ship, do the >> build, then go away 'till next time they're required. > > I'm not quite sure whom you have been trying to get feedback from, and > can't quite picture your proposed setup from above description. Sorry, feedback was the wrong word. I've been digging round the documentation I've been able to find and looking into what's needed to set up a slave. > In any case, ISTM that your approach isn't compatible with how buildbot > works today (not sure whether you are aware of that): a build slave > needs to stay connected all the time, so that the build master can > trigger a build when necessary. > > So if your setup requires the slaves to shut down after a build, I don't > think this can possibly work. It's not so much that I *require* the slave to shut down, more that I'm not sure how well I'll be able to ensure that it's up all the time, and I'm trying to understand the implications of that. My basic impression is that it's not really going to work, unfortunately. Paul. From glyph at twistedmatrix.com Sun Oct 25 22:36:08 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 25 Oct 2009 17:36:08 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE4A1C9.5010002@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> Message-ID: On Oct 25, 2009, at 3:06 PM, Martin v. L?wis wrote: > (*) it may help if Buildbot would create a Win32 job object, and > then use TerminateJobObject. Contributions are welcome. Some work has already been done on this, but it needs help. At the root it's a Twisted issue: http://twistedmatrix.com/trac/ticket/2726 -------------- next part -------------- An HTML attachment was scrubbed... URL: From db3l.net at gmail.com Sun Oct 25 22:36:31 2009 From: db3l.net at gmail.com (David Bolen) Date: Sun, 25 Oct 2009 17:36:31 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > The remaining issue is the popups; if a process still has a popup, > you can't even terminate it properly. There are two kinds of popups: > system-generated ones, and CRT-generated ones. For the CRT ones, we > once had a way to turn them off, but I'm not sure whether that mechanism > might have been removed. For the system messages, there is a way to > turn them off in the parent process. David Bolen (IIRC) had developed > a patch, but I think this patch only runs on his system(s). Yes, process-stopping dialogs have probably been the single most annoying issue over time running a Windows build slave - certainly from my perspective in terms of maintenance and detection they have taken up the largest amount of time. I believe the CRT disabling is still active in the 3.x branches (the "-n" flag to regrtest in the buildbot's test.bat), after you restored it this past March (it had regressed during an earlier py3k branch set of patches and caused a bunch of problems for a bit) but not in the 2.x branches or trunk. So there's still a bit of exposure there and I'd certainly be in favor of porting the regrtest -n support over to all current development branches. I think the other issue most likely to cause a perceived "downtime" with the Windows build slave that I've had a handful of cases over the past two years where the build slave appears to be operating properly, but the master seems to just queue up jobs as if it were down. The slave still shows an established TCP link to the master, so I generally only catch this when I happen to peek at the status web page, or catch a remark here on python-dev, so that can reduce availability. My build slave (based on 0.7.5 I think) runs with local patches to: 1. Protect again Win32 pop-up error boxes in child processes. 2. A fix for a recursive chaining of Twisted Deferreds during uploads which could break for large file transfers. This only came up when my build slave was generating daily MSI builds and uploading them to the master. 3. Handle clock jumps. It's a general flaw in the presence of system clock adjustments, but I only encountered it while in my FreeBSD build slave under VMWare with a Linux host. (2) and (3) are useful, but not likely to be an issue with most build slaves in normal operation. (2) probably isn't needed on my end any more now that the daily MSI builds aren't run, and it's possible that it got corrected in later buildbot updates, since I did report it on the development list at the time. (1) is a pretty trivial patch, but added a dependency on pywin32, so passing it back up to the buildbot maintainers (back in 2007) stalled while determining if that was ok, and I don't I ever closed the loop. I did later make it fall back to ctypes if pywin32 was missing, but then I think buildbot was using a minimum of Python 2.3 at the time, so even ctypes was a new dependency. Anyway, it became less crucial when Python's regrtest started executing similar code, though the buildbot patch covers anything run under it and not just the python process. I'd of course be happy to pass along the patch to anyone interested. I believe that Thomas Heller had run his Windows buildbot with some similar local code, but implemented with a modified buildbot script for building Python, rather than tweaking buildbot itself. Of course, the patch only protects against system pop-ups - it can't control the CRT assertion dialogs when Python is built in debug mode, which is why I've argued in the past that the test process for Python should ensure those are disabled. The CRT errors themselves are still important, but can be redirected to stderr rather than a blocking GUI dialog. -- David From martin at v.loewis.de Sun Oct 25 22:43:03 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Oct 2009 22:43:03 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <79990c6b0910251417l515d29b7iaedd2a338eb0b9d2@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> <79990c6b0910251417l515d29b7iaedd2a338eb0b9d2@mail.gmail.com> Message-ID: <4AE4C667.5030305@v.loewis.de> > It's not so much that I *require* the slave to shut down, more that > I'm not sure how well I'll be able to ensure that it's up all the > time, and I'm trying to understand the implications of that. My basic > impression is that it's not really going to work, unfortunately. There is a significant difference between "not able to ensure that it is up all the time", and "it is down most of the time, and only up once a day for a short period of time". Regular and irregular short downtimes are no problem at all. When the slave comes back, it will pick up pending work. Longer scheduled downtimes (e.g. for vacations) are acceptable. Only turning on the slave occasionally makes it useless. Regards, Martin From ssteinerx at gmail.com Sun Oct 25 22:50:50 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Sun, 25 Oct 2009 17:50:50 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE4C667.5030305@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> <79990c6b0910251417l515d29b7iaedd2a338eb0b9d2@mail.gmail.com> <4AE4C667.5030305@v.loewis.de> Message-ID: <09C914A0-C9F2-46B4-9133-BDC9B67C828F@gmail.com> On Oct 25, 2009, at 5:43 PM, Martin v. L?wis wrote: > Only turning on the slave occasionally makes it useless. For certain use cases; not mine. S From solipsis at pitrou.net Sun Oct 25 22:56:42 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Oct 2009 21:56:42 +0000 (UTC) Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <73E21491-E39E-4CFA-8C77-2E9AFDFCF3BE@gmail.com> <4AE499D4.4010104@v.loewis.de> <79990c6b0910251417l515d29b7iaedd2a338eb0b9d2@mail.gmail.com> <4AE4C667.5030305@v.loewis.de> <09C914A0-C9F2-46B4-9133-BDC9B67C828F@gmail.com> Message-ID: ssteinerX gmail.com gmail.com> writes: > On Oct 25, 2009, at 5:43 PM, Martin v. L?wis wrote: > > > Only turning on the slave occasionally makes it useless. > > For certain use cases; not mine. Let's say that for the use case we are talking here (this is python-dev), Martin's statement holds true. From brett at python.org Sun Oct 25 22:57:21 2009 From: brett at python.org (Brett Cannon) Date: Sun, 25 Oct 2009 14:57:21 -0700 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <1256502140.5621.163.camel@localhost> References: <1256502140.5621.163.camel@localhost> Message-ID: > > [SNIP - a lot of detail on what sounds like a good design] > > Now what remains to be done? > > Having other people test it would be fine. Even better if you have an > actual multi-threaded py3k application. But ccbench results for other > OSes would be nice too :-) > (I get good results under the Windows XP VM but I feel that a VM is not > an ideal setup for a concurrency benchmark) > > Of course, studying and reviewing the code is welcome. As for > integrating it into the mainline py3k branch, I guess we have to answer > these questions: > - is the approach interesting? (we could decide that it's just not worth > it, and that a good GIL can only be a dead (removed) GIL) > I think it's worth it. Removal of the GIL is a totally open-ended problem with no solution in sight. This, on the other hand, is a performance benefit now. I say move forward with this. If it happens to be short-lived because some actually figures out how to remove the GIL then great, but is that really going to happen between now and Python 3.2? I doubt it. > - is the patch good, mature and debugged enough? > - how do we deal with the unsupported platforms (POSIX and Windows > support should cover most bases, but the fate of OS/2 support depends on > Andrew)? > > It's up to Andrew to get the support in. While I have faith he will, this is why we have been scaling back the support for alternative OSs for a while and will continue to do so. I suspect the day Andrew stops keeping up will be the day we push to have OS/2 be externally maintained. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 26 00:17:03 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Oct 2009 10:17:03 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE49C68.9090108@v.loewis.de> References: <200910231132.45504.w.richert@gmx.net> <4AE49C68.9090108@v.loewis.de> Message-ID: <200910261017.04518.steve@pearwood.info> On Mon, 26 Oct 2009 05:43:52 am Martin v. L?wis wrote: > What Alexander wants is that the set type also becomes useful for > storing canonical representations. I find that inappropriate, since > dicts already provide the one obvious way to do it. Only because dicts came first in Python rather than sets. There's nothing inherently obvious about storing the canonical representation of an object *twice* (although I'm not Dutch). A more natural representation would be to store the canonical representation once. An abstract intern operation might be written: if s equals an element in the data store return the element which s equals otherwise insert s into the data store and return s Notice we don't say: if s equals an element in the data store return the value associated with the element which s equals which is what the dict-based solution does. We can implement that abstract algorithm using lists. The algorithm is exactly the same as it is for dicts, except that search/retrieval becomes two operations rather than one: _cache = ['x', 'spam', 'parrot'] def intern(s): try: s = _cache[ _cache.index(s) ] except ValueError: _cache.append(s) return s We don't do this because O(N) searching is too expensive. Using a dict {s: s} is a workaround for the lack of a data structure that combines O(1) searching and ability to retrieve the element just found. Memory is cheap, but not so cheap that doubling the size of a data structure (two pointers per element for {s:s} versus one for {s}) is insignificant. Part of the reason we intern in the first place is to save memory. We shouldn't dismiss this out of hand based on the argument that retrieval from a set is insufficiently pure. As soon as you allow iteration over sets, you have allowed retrieval. -- Steven D'Aprano From python at mrabarnett.plus.com Mon Oct 26 00:54:09 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 25 Oct 2009 23:54:09 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> Message-ID: <4AE4E521.600@mrabarnett.plus.com> David Bolen wrote: [snip] > I think the other issue most likely to cause a perceived "downtime" > with the Windows build slave that I've had a handful of cases over the > past two years where the build slave appears to be operating properly, > but the master seems to just queue up jobs as if it were down. The > slave still shows an established TCP link to the master, so I > generally only catch this when I happen to peek at the status web > page, or catch a remark here on python-dev, so that can reduce > availability. > [snip] Couldn't you write a script to check the status periodically? From exarkun at twistedmatrix.com Mon Oct 26 01:58:15 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 26 Oct 2009 00:58:15 -0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> Message-ID: <20091026005815.31526.130687717.divmod.xquotient.88@localhost.localdomain> On 25 Oct, 09:36 pm, db3l.net at gmail.com wrote: > >I think the other issue most likely to cause a perceived "downtime" >with the Windows build slave that I've had a handful of cases over the >past two years where the build slave appears to be operating properly, >but the master seems to just queue up jobs as if it were down. The >slave still shows an established TCP link to the master, so I >generally only catch this when I happen to peek at the status web >page, or catch a remark here on python-dev, so that can reduce >availability. This sounds like something that should be reported upstream. Particularly if you know how to reproduce it. Has it been? Jean-Paul From db3l.net at gmail.com Mon Oct 26 02:28:33 2009 From: db3l.net at gmail.com (David Bolen) Date: Sun, 25 Oct 2009 21:28:33 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <20091026005815.31526.130687717.divmod.xquotient.88@localhost.localdomain> Message-ID: exarkun at twistedmatrix.com writes: > This sounds like something that should be reported > upstream. Particularly if you know how to reproduce it. Has it been? No, largely because I can't reproduce it at all. It's happened maybe 4-5 times in the past 2 years or so. All that I see is that my end looks good yet the master end seems not to be dispatching jobs (it never shows an explicit disconnect for my slave though). My best guess is that something disrupted the TCP connection, and that the slave isn't doing anything that would let it know its connection was dropped. Although I thought there were periodic pings even from the slave side. Given the frequency, it's not quite high priority to me, though having the master let the owner of a slave know when it's down would help cut down on lost availability due to this case, so I suppose I could suggest that feature to the buildbot developers. -- David From db3l.net at gmail.com Mon Oct 26 02:31:55 2009 From: db3l.net at gmail.com (David Bolen) Date: Sun, 25 Oct 2009 21:31:55 -0400 Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <4AE4E521.600@mrabarnett.plus.com> Message-ID: MRAB writes: > Couldn't you write a script to check the status periodically? Sure, I suppose scraping the web status page would work. If it happened frequently I'd probably be forced to do something like that, but it's relatively low frequency (though I guess it does have a big impact in terms of availability) makes it hard to dedicate time to that compared to my "real world" work :-) -- David From exarkun at twistedmatrix.com Mon Oct 26 02:56:50 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 26 Oct 2009 01:56:50 -0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <20091026005815.31526.130687717.divmod.xquotient.88@localhost.localdomain> Message-ID: <20091026015650.31526.600593069.divmod.xquotient.94@localhost.localdomain> On 01:28 am, db3l.net at gmail.com wrote: >exarkun at twistedmatrix.com writes: >>This sounds like something that should be reported >>upstream. Particularly if you know how to reproduce it. Has it been? > >No, largely because I can't reproduce it at all. It's happened maybe >4-5 times in the past 2 years or so. All that I see is that my end >looks good yet the master end seems not to be dispatching jobs (it >never shows an explicit disconnect for my slave though). > >My best guess is that something disrupted the TCP connection, and that >the slave isn't doing anything that would let it know its connection >was dropped. Although I thought there were periodic pings even from >the slave side. > >Given the frequency, it's not quite high priority to me, though having >the master let the owner of a slave know when it's down would help cut >down on lost availability due to this case, so I suppose I could >suggest >that feature to the buildbot developers. This feature exists, at least. BuildBot can email people when slaves are offline for more than some configured time limit. I'm not sure if the CPython master is configured to do this or not. It's easy to set up if not, the BuildSlave initializer accepts a list of email addresses that will be notified when that slave goes offline, notify_on_missing: http://buildbot.net/apidocs/buildbot.buildslave.AbstractBuildSlave- class.html#__init__ Jean-Paul From john at arbash-meinel.com Mon Oct 26 03:20:51 2009 From: john at arbash-meinel.com (John Arbash Meinel) Date: Sun, 25 Oct 2009 21:20:51 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE49E2E.3090506@v.loewis.de> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> <4AE49E2E.3090506@v.loewis.de> Message-ID: <4AE50783.90303@arbash-meinel.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >> Hmm, perhaps when using sets as work queues? > > A number of comments: > > - it's somewhat confusing to use a set as a *queue*, given > that it won't provide FIFO semantics. > - there are more appropriate and direct container structures > available, including a dedicated Queue module (even though > this might be doing to much with its thread-safety). > - if you absolutely want to use a set as a work queue, > then the .pop() method should be sufficient, right? > > Regards, > Martin We were using sets to track the tips of a graph, and to compare whether one node was an ancestor of another one. We were caching that answer into frozensets, since that made them immutable. If res = heads(node1, node2) if len(res) == 1: # What is the 'obvious' way to get the node out? I posit that there *isn't* an obvious way to get the single item out of a 1-entry frozenset. for x in res: break list(res)[0] set(res).pop() iter(res).next() [x for x in res][0] x, = res # I didn't think of this one before recently Are all answers, but none of them I would consider *obvious*. At the least, none of them are obviously better than another, so you look into the performance characteristics to give you a reason to pick one over the other. res.get() would be a fairly obvious way to do it. Enough that I would probably never have gone searching for any of the other answers. Though personally, I think I would call it "set.peek()", but the specific name doesn't really matter to me. John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrlB4MACgkQJdeBCYSNAAN0lQCgrtyXWlqIbjj01qB4AKPhKrMq QH8An0z2gCWZHoceEJsqRJOUdEl/VLTB =fJXI -----END PGP SIGNATURE----- From debatem1 at gmail.com Mon Oct 26 03:46:31 2009 From: debatem1 at gmail.com (geremy condra) Date: Sun, 25 Oct 2009 22:46:31 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE50783.90303@arbash-meinel.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> <4AE49E2E.3090506@v.loewis.de> <4AE50783.90303@arbash-meinel.com> Message-ID: > Martin v. L?wis wrote: >>> Hmm, perhaps when using sets as work queues? >> >> A number of comments: >> >> - it's somewhat confusing to use a set as a *queue*, given >> ? that it won't provide FIFO semantics. >> - there are more appropriate and direct container structures >> ? available, including a dedicated Queue module (even though >> ? this might be doing to much with its thread-safety). >> - if you absolutely want to use a set as a work queue, >> ? then the .pop() method should be sufficient, right? >> >> Regards, >> Martin > > We were using sets to track the tips of a graph, and to compare whether > one node was an ancestor of another one. We were caching that answer > into frozensets, since that made them immutable. If > > res = heads(node1, node2) > if len(res) == 1: > ?# What is the 'obvious' way to get the node out? > > I posit that there *isn't* an obvious way to get the single item out of > a 1-entry frozenset. > > for x in res: break > list(res)[0] > set(res).pop() > iter(res).next() > [x for x in res][0] > x, = res ? # I didn't think of this one before recently > > Are all answers, but none of them I would consider *obvious*. At the > least, none of them are obviously better than another, so you look into > the performance characteristics to give you a reason to pick one over > the other. > > res.get() would be a fairly obvious way to do it. Enough that I would > probably never have gone searching for any of the other answers. Though > personally, I think I would call it "set.peek()", but the specific name > doesn't really matter to me. > > John > When I first wrote Graphine (graph library), I did something very similar to the last solution. The code has since been rewritten to avoid the issue, but it would be nice if it didn't have to be the next time it comes up- the comma is easy to miss, and while the results are common-sense once you get what the line is doing, it doesn't lend itself to immediate comprehension. Geremy Condra From tjreedy at udel.edu Mon Oct 26 06:07:26 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Oct 2009 01:07:26 -0400 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <1256502140.5621.163.camel@localhost> References: <1256502140.5621.163.camel@localhost> Message-ID: Antoine Pitrou wrote: > Hello there, > > The last couple of days I've been working on an experimental rewrite of > the GIL. Since the work has been turning out rather successful (or, at > least, not totally useless and crashing!) I thought I'd announce it > here. I am curious as to whether the entire mechanism is or can be turned off when not needed -- when there are not threads (other than the main, starting thread)? tjr From martin at v.loewis.de Mon Oct 26 06:53:30 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 26 Oct 2009 06:53:30 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE50783.90303@arbash-meinel.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> <4AE49E2E.3090506@v.loewis.de> <4AE50783.90303@arbash-meinel.com> Message-ID: <4AE5395A.7010902@v.loewis.de> > We were using sets to track the tips of a graph, and to compare whether > one node was an ancestor of another one. We were caching that answer > into frozensets, since that made them immutable. If > > res = heads(node1, node2) > if len(res) == 1: > # What is the 'obvious' way to get the node out? What specifically is that that you want to do in this case? IIUC, the possible result could be that either node1 is an ancestor of node2, or vice versa. So I would write that as if len(res) == 1: if node1 in res: # action to take if node1 is the head else: # action to take if node2 is the head else: # they are unrelated In any case, this is a use case different from "work queue" (AFAICT), so I'm unsure why you responded to my message where Florian and me were talking about work queues. > res.get() would be a fairly obvious way to do it. Enough that I would > probably never have gone searching for any of the other answers. Though > personally, I think I would call it "set.peek()", but the specific name > doesn't really matter to me. Somebody proposed to call it .any(); this I like best (except that one might expect that any takes a predicate as the argument). Regards, Martin From Scott.Daniels at Acm.Org Mon Oct 26 07:14:06 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sun, 25 Oct 2009 23:14:06 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE50783.90303@arbash-meinel.com> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> <4AE49E2E.3090506@v.loewis.de> <4AE50783.90303@arbash-meinel.com> Message-ID: John Arbash Meinel wrote: > res = heads(node1, node2) > if len(res) == 1: > # What is the 'obvious' way to get the node out? > > I posit that there *isn't* an obvious way to get the single item out of > a 1-entry frozenset. > > for x in res: break > list(res)[0] > set(res).pop() > iter(res).next() > [x for x in res][0] > x, = res # I didn't think of this one before recently > > Are all answers, but none of them I would consider *obvious*. And from my SQL-hacking experience: x = min(s) --Scott David Daniels Scott.Daniels at Acm.Org From python at rcn.com Mon Oct 26 08:29:36 2009 From: python at rcn.com (Raymond Hettinger) Date: Mon, 26 Oct 2009 00:29:36 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE1E71C.7020806@gmail.com> <4AE20E0D.4050405@gmail.com> Message-ID: [John Arbash Meine] > So 'for x in s: break' is about 2x faster than next(iter(s)) and 3x > faster than (iter(s).next()). The first version uses direct calls for the for-loop opcodes. The other two have to do functions/method look-up and make a pure python function call (neither is very fast). FWIW, a "choose" method had been previously proposed, discussed and rejected. I don't find the optimization issue to be very interesting in the case of retrieving an arbitrary element. This isn't the kind of thing that typically appears in an inner-loop; afterall, if you've retrieved an arbitrary element without removing it, then successive calls to "choose" could potentially retrieve the exact same element again and again. Raymond From stephen at xemacs.org Mon Oct 26 08:48:39 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 26 Oct 2009 16:48:39 +0900 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE5395A.7010902@v.loewis.de> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> <4AE49E2E.3090506@v.loewis.de> <4AE50783.90303@arbash-meinel.com> <4AE5395A.7010902@v.loewis.de> Message-ID: <873a56a948.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > res.get() would be a fairly obvious way to do it. Enough that I would > > probably never have gone searching for any of the other answers. Though > > personally, I think I would call it "set.peek()", but the specific name > > doesn't really matter to me. > > Somebody proposed to call it .any(); this I like best (except that one > might expect that any takes a predicate as the argument). Why not allow that? def any(self, predicate=lambda x: True, default=None) for a in self: if predicate(a): break else: return default return a From martin at v.loewis.de Mon Oct 26 09:02:10 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Oct 2009 09:02:10 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <873a56a948.fsf@uwakimon.sk.tsukuba.ac.jp> References: <200910231132.45504.w.richert@gmx.net> <2987c46d0910231004p216e68d3tdc93a68e9462b4f6@mail.gmail.com> <4AE3AE3E.80402@canonical.com> <3699AEA0-B268-4DDE-8739-8CC9CE7C7FB1@fuhm.net> <87ljizxsp8.fsf@mid.deneb.enyo.de> <4AE49E2E.3090506@v.loewis.de> <4AE50783.90303@arbash-meinel.com> <4AE5395A.7010902@v.loewis.de> <873a56a948.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4AE55782.3070805@v.loewis.de> > Why not allow that? > > def any(self, predicate=lambda x: True, default=None) > for a in self: > if predicate(a): > break > else: > return default > return a I'm +0 (given that I'm still skeptical about the need to have something like this). Also setting aside the moratorium here, which may disallow that kind of change for a foreseeable feature in any form. Regards, Martin From w.richert at gmx.net Mon Oct 26 09:22:37 2009 From: w.richert at gmx.net (Willi Richert) Date: Mon, 26 Oct 2009 09:22:37 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE20E0D.4050405@gmail.com> Message-ID: <200910260922.37375.w.richert@gmx.net> Hi, I totally agree regarding the efficiency. Code that relies on a fast "non- removing .pop()" probably has other worse bottlenecks that should be targetted first. This would, however, relief every programmer who needs this the first time in his Python experience, to research how this could be most reasonably done. And it is more of a style question. I find the "for x in some_set: break" rather ugly. wr Am Montag, 26. Oktober 2009 08:29:36 schrieben Sie: > I don't find the optimization issue to be very interesting in > the case of retrieving an arbitrary element. This isn't > the kind of thing that typically appears in an inner-loop; > afterall, if you've retrieved an arbitrary element without > removing it, then successive calls to "choose" could > potentially retrieve the exact same element again and again. > > > Raymond From martin at v.loewis.de Mon Oct 26 09:25:41 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Oct 2009 09:25:41 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <4AE4E521.600@mrabarnett.plus.com> Message-ID: <4AE55D05.3050305@v.loewis.de> David Bolen wrote: > MRAB writes: > >> Couldn't you write a script to check the status periodically? > > Sure, I suppose scraping the web status page would work. If it > happened frequently I'd probably be forced to do something like that, > but it's relatively low frequency (though I guess it does have a big > impact in terms of availability) makes it hard to dedicate time to > that compared to my "real world" work :-) In addition, if it was happening frequently, we would rather investigate the problem and fix it, than working around. Regards, Martin From martin at v.loewis.de Mon Oct 26 09:30:49 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 26 Oct 2009 09:30:49 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091026015650.31526.600593069.divmod.xquotient.94@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <20091026005815.31526.130687717.divmod.xquotient.88@localhost.localdomain> <20091026015650.31526.600593069.divmod.xquotient.94@localhost.localdomain> Message-ID: <4AE55E39.20000@v.loewis.de> >>> This sounds like something that should be reported >>> upstream. Particularly if you know how to reproduce it. Has it been? >> >> No, largely because I can't reproduce it at all. It's happened maybe >> 4-5 times in the past 2 years or so. All that I see is that my end >> looks good yet the master end seems not to be dispatching jobs (it >> never shows an explicit disconnect for my slave though). It's not really reproducible. I think it sometimes happens when I restart the master; sometimes, some clients fail to reconnect (properly). > It's easy to set up if not, the BuildSlave initializer accepts a list of > email addresses that will be notified when that slave goes offline, > notify_on_missing: > > http://buildbot.net/apidocs/buildbot.buildslave.AbstractBuildSlave- > class.html#__init__ I tried that out a couple of weeks ago, but never received any email. I didn't have the time to look into this further since. Regards, Martin From solipsis at pitrou.net Mon Oct 26 11:19:00 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Oct 2009 10:19:00 +0000 (UTC) Subject: [Python-Dev] Reworking the GIL References: <1256502140.5621.163.camel@localhost> Message-ID: Terry Reedy udel.edu> writes: > > I am curious as to whether the entire mechanism is or can be turned off > when not needed -- when there are not threads (other than the main, > starting thread)? It is an implicit feature: when no thread is waiting on the GIL, the GIL-holding thread isn't notified and doesn't try to release it at all (in the eval loop, that is; GIL-releasing C extensions still release it). Note that "no thread is waiting on the GIL" can mean one of two things: - either there is only one Python thread - or the other Python threads are doing things with the GIL released (zlib/bz2 compression, waiting on I/O, sleep()ing, etc.) So, yes, it automatically "turns itself off". Regards Antoine. From eckhardt at satorlaser.com Mon Oct 26 12:10:13 2009 From: eckhardt at satorlaser.com (Ulrich Eckhardt) Date: Mon, 26 Oct 2009 12:10:13 +0100 Subject: [Python-Dev] Clean up Python/thread_*.h ? In-Reply-To: <4AE35098.6070306@cheimes.de> References: <4AE35098.6070306@cheimes.de> Message-ID: <200910261210.13736.eckhardt@satorlaser.com> On Saturday 24 October 2009, Christian Heimes wrote: > Antoine Pitrou wrote: > > * The unused file: thread_wince.h > > > > Windows CE has actually been using thread_nt.h since January 2009 (see > > http://bugs.python.org/issue4893 ). thread_wince.h is still there, but > > it's unused and grep brings no instance of it being mentioned anywhere in > > the source tree. > > What about older versions of Windows CE? Maybe they still require the > wince thread header? Windows CE support in recent 2.x (and probably 3.x) is broken due to heavy bitrot. I can't imagine any CE platform this code would compile against. Further, I don't see many people using CE < 4, with CE 6 being current, in fact they are _very_ rare judging from Usenet activities. For current CE versions, porting the NT-threads code is the easier way out. +1 for removal of thread_wince.{h,c} Uli -- Sator Laser GmbH Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Sator Laser GmbH, Fangdieckstra?e 75a, 22547 Hamburg, Deutschland Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at ************************************************************************************** Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden. E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich. ************************************************************************************** From andymac at bullseye.apana.org.au Mon Oct 26 13:37:14 2009 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Mon, 26 Oct 2009 23:37:14 +1100 Subject: [Python-Dev] Reworking the GIL In-Reply-To: References: <1256502140.5621.163.camel@localhost> Message-ID: <4AE597FA.7070701@bullseye.andymac.org> Brett Cannon wrote: > It's up to Andrew to get the support in. While I have faith he will, > this is why we have been scaling back the support for alternative OSs > for a while and will continue to do so. I suspect the day Andrew stops > keeping up will be the day we push to have OS/2 be externally maintained. Notwithstanding my desire to keep OS/2 supported in the Python tree, keeping up has been more difficult of late: - OS/2 is unquestionably a "legacy" environment, with system APIs different in flavour and semantics from the current mainstream (though surprisingly capable in many ways despite its age). - The EMX runtime my OS/2 port currently relies on to abstract the system API to a Posix-ish API is itself a legacy package, essentially unmaintained for some years :-( This has been a source of increasing pain as Python has moved with the mainstream... with regard to Unicode support and threads in conjunction with multi-processing, in particular. Real Life hasn't been favourably disposed either... I have refrained from applying the extensive patches required to make the port feature complete for 2.6 and later while I investigate an alternate Posix emulating runtime (derived from FreeBSD's C library, and which is used by Mozilla on OS/2), which would allow me to dispense with most of these patches. But it has an issue or two of its own... The cost in effort has been compounded by effectively having to try and maintain two ports - 2.x and 3.x. And the 3.x port has suffered more as its demands are higher. So while I asked to keep the OS/2 thread support alive, if a decision were to be taken to remove OS/2 support from the Python 3.x sources I could live with that. A completed migration to Mercurial might well make future port maintenance easier for me. Regards, Andrew. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac at pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From sturla at molden.no Mon Oct 26 14:43:04 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 26 Oct 2009 14:43:04 +0100 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <1256502140.5621.163.camel@localhost> References: <1256502140.5621.163.camel@localhost> Message-ID: <4AE5A768.6080900@molden.no> Antoine Pitrou skrev: > - priority requests, which is an option for a thread requesting the GIL > to be scheduled as soon as possible, and forcibly (rather than any other > threads). So Python threads become preemptive rather than cooperative? That would be great. :-) time.sleep should generate a priority request to re-acquire the GIL; and so should all other blocking standard library functions with a time-out. S.M. From kristjan at ccpgames.com Mon Oct 26 15:09:43 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Mon, 26 Oct 2009 14:09:43 +0000 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <4AE5A768.6080900@molden.no> References: <1256502140.5621.163.camel@localhost> <4AE5A768.6080900@molden.no> Message-ID: <930F189C8A437347B80DF2C156F7EC7F098FF42768@exchis.ccp.ad.local> > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com at python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf > Of Sturla Molden > time.sleep should generate a priority request to re-acquire the GIL; > and > so should all other blocking standard library functions with a time- > out. I don't agree. You have to be very careful with priority. time.sleep() does not promise to wake up in any timely manner, and neither do the timeout functions. Rather, the timeout is a way to prevent infinite wait. In my experience (from stackless python) using priority wakeup for IO can result in very erratic scheduling when there is much IO going on, every IO trumping another. You should stick to round robin except for very special and carefully analysed cases. K From ssteinerx at gmail.com Mon Oct 26 15:25:25 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Mon, 26 Oct 2009 10:25:25 -0400 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F098FF42768@exchis.ccp.ad.local> References: <1256502140.5621.163.camel@localhost> <4AE5A768.6080900@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42768@exchis.ccp.ad.local> Message-ID: On Oct 26, 2009, at 10:09 AM, Kristj?n Valur J?nsson wrote: > > >> -----Original Message----- >> From: python-dev-bounces+kristjan=ccpgames.com at python.org >> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On >> Behalf >> Of Sturla Molden >> time.sleep should generate a priority request to re-acquire the GIL; >> and >> so should all other blocking standard library functions with a time- >> out. > > I don't agree. You have to be very careful with priority. > time.sleep() does not promise to wake up in any timely manner, and > neither do the timeout functions. Rather, the timeout is a way to > prevent infinite wait. > > In my experience (from stackless python) using priority wakeup for > IO can result in very erratic scheduling when there is much IO going > on, every IO trumping another. You should stick to round robin > except for very special and carefully analysed cases. All the IO tasks can also go in their own round robin so that CPU time is correctly shared among all waiting IO tasks. IOW, to make sure that all IO tasks get a fair share *in relation to all other IO tasks*. Tasks can be put into the IO round robin when they "pull the IO alarm" so to speak, so there's no need to decide before-hand which task goes in which round robin pool. I'm not familiar with this particular code in Python, but I've used this in other systems for years to make sure that IO tasks don't starve the rest of the system and that the most "insistent" IO task doesn't starve all the others. S From alexander.belopolsky at gmail.com Mon Oct 26 16:04:06 2009 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 26 Oct 2009 11:04:06 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <4AE3E6C6.4010008@gmail.com> References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> Message-ID: On Sun, Oct 25, 2009 at 1:48 AM, Nick Coghlan wrote: > Terry Reedy wrote: >>> In this model, a >>> get() method, or even a __getitem__ with s[k] is k, is only natural. >> >> No, if key and value were the same thing, the get method would be >> nonesense, as Guido said. But since they are not, since the implict key >> is an abstract class, retrieving the representative, perhaps canonical >> object given *any* member *is* natural. Being able to do so is also a >> standard practice in mathematics. > > To formalise this invariant to some degree: given a set "s" and two > items "k1" and "k2", such that "k1 in s" and "k2 in s" and "k1 == k2", > there is a single item "k" in s such that "k1 == k" and "k2 == k". > > If __getitem__ were to be provided by sets, then the last part of that > invariant could be written "s[k1] is s[k2]". > Thanks, Nick and Terry for a much more precise formulation of the point that I was trying to present. When I wrote s[k] is k, I had in mind for k stored is the set or among the keys of an equivalent dict. Formally alltrue(s[k] is k for k in s). Nick's invariant is of course a better expression of the same idea. I believe interning is a worthwhile use case for Python to have "one obvious way to do it." Martin suggests that such a way already exists and it involves storing interned objects as both keys and values in a dictionary. While this may have been obvious before sets became available, I agree with Steven that in post 2.4 python users are likely to look at set first and will only turn to dictionary after discovering that the functionality that they are looking for is not available in set. Even if you know to use a dictionary, there are still some non-obvious tricks to learn about. First, you need to learn about setdefault, a much criticized method that led to a new class being introduced to the standard library: http://mail.python.org/pipermail/python-dev/2003-February/033321.html http://docs.python.org/library/collections.html?highlight=defaultdict#collections.defaultdict Second, there is no obvious way to pre-seed the dictionary, i.e., given a list l of keys, d = {} for k in l: d[k] = k When I was looking for a dict method to accomplish that, I discovered dict.fromkeys, which of course was not the right method. I still don't know if a better solution exists than calling setdefault(k, k) in a loop. As experience with defaultdict has shown, it may be more appropriate to introduce a specialized and properly named class in collections rather than overloading set with new methods, but such approach would lead to steep learning curve. Collections.defaultdict, could be cross-referenced from dict.setdefault, but a hypothetical collections.interningset would most likely remain undiscovered for the majority of users. Here is an alternative idea on how storing interned objects in a set can be supported. Currently set.add method returns None and had no effect when set already has an object equal to the one being added. I propose to consider changing that behavior to make set.add return the added object or the set member that is equal to the object being added. It is unlikely that many programs rely on the return value being None (with doctests being a probable exception), so adding this feature is unlikely to cause much grief. From sturla at molden.no Mon Oct 26 16:46:34 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 26 Oct 2009 16:46:34 +0100 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <1256502140.5621.163.camel@localhost> References: <1256502140.5621.163.camel@localhost> Message-ID: <4AE5C45A.9090900@molden.no> Antoine Pitrou skrev: > - priority requests, which is an option for a thread requesting the GIL > to be scheduled as soon as possible, and forcibly (rather than any other > threads). T Should a priority request for the GIL take a priority number? - If two threads make a priority requests for the GIL, the one with the higher priority should get the GIL first. - If a thread with a low priority make a priority request for the GIL, it should not be allowed to "preempt" (take the GIL away from) a higher-priority thread, in which case the priority request would be ignored. Related issue: Should Python threads have priorities? They are after all real OS threads. S.M. From solipsis at pitrou.net Mon Oct 26 16:58:49 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Oct 2009 15:58:49 +0000 (UTC) Subject: [Python-Dev] Reworking the GIL References: <1256502140.5621.163.camel@localhost> <4AE5C45A.9090900@molden.no> Message-ID: Sturla Molden molden.no> writes: > > Antoine Pitrou skrev: > > - priority requests, which is an option for a thread requesting the GIL > > to be scheduled as soon as possible, and forcibly (rather than any other > > threads). T > Should a priority request for the GIL take a priority number? Er, I prefer to keep things simple. If you have lots of I/O you should probably use an event loop rather than separate threads. > Related issue: Should Python threads have priorities? They are after all > real OS threads. Well, precisely they are OS threads, and the OS already assigns them (static or dynamic) priorities. No need to replicate this. (to answer another notion expressed in another message, there's no "round-robin" scheduling either) Regards Antoine. From tjreedy at udel.edu Mon Oct 26 17:02:05 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Oct 2009 12:02:05 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> Message-ID: Alexander Belopolsky wrote: > Here is an alternative idea on how storing interned objects in a set > can be supported. Currently set.add method returns None and had no > effect when set already has an object equal to the one being added. I > propose to consider changing that behavior to make set.add return the > added object or the set member that is equal to the object being > added. It is unlikely that many programs rely on the return value > being None (with doctests being a probable exception), so adding this > feature is unlikely to cause much grief. I had exactly the same idea, but did not post because it violates the general rule that mutators return None. On the other hand, the returned value here would not be the mutated collection, so no chaining is possible. And 'add' is clearly intended to change something. On the other hand, frozensets do not have an add method. Terry Jan Reedy From daniel at stutzbachenterprises.com Mon Oct 26 17:18:59 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Mon, 26 Oct 2009 11:18:59 -0500 Subject: [Python-Dev] Reworking the GIL In-Reply-To: References: <1256502140.5621.163.camel@localhost> <4AE5C45A.9090900@molden.no> Message-ID: On Mon, Oct 26, 2009 at 10:58 AM, Antoine Pitrou wrote: > Er, I prefer to keep things simple. If you have lots of I/O you should > probably > use an event loop rather than separate threads. > On Windows, sometimes using a single-threaded event loop is sometimes impossible. WaitForMultipleObjects(), which is the Windows equivalent to select() or poll(), can handle a maximum of only 64 objects. Do we really need priority requests at all? They seem counter to your desire for simplicity and allowing the operating system's scheduler to do its work. That said, if a thread's time budget is merely paused during I/O rather than reset, then a thread making frequent (but short) I/O requests cannot starve the system. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 26 17:38:13 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Oct 2009 09:38:13 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> Message-ID: All this talk of equivalence classes makes me dizzy. :-) - If sets were to grow an API to non-destructively access the object stored in the set for a particular key, then dicts should have such a method too. - Ditto for an API to non-destructively get an arbitrary element. - I'm far from convinced that we urgently need either API. But I'm also not convinced it's unneeded. - I still wish we could go back in time and unify sets and dicts, if only to find out how that experiment would turn out. -- --Guido van Rossum PS. My elbow needs a couple more weeks of rest. Limiting myself to ultra-short emails. From alexander.belopolsky at gmail.com Mon Oct 26 17:43:09 2009 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 26 Oct 2009 12:43:09 -0400 Subject: [Python-Dev] One obvious way to do interning [Was: Retrieve an arbitrary element from a set without removing it] Message-ID: Changing the subject to reflect branched discussion and forwarding to python-ideas where it probably belongs. On Mon, Oct 26, 2009 at 12:02 PM, Terry Reedy wrote: > Alexander Belopolsky wrote: > >> Here is an alternative idea on how storing interned objects in a set >> can be supported. ?Currently set.add method returns None and has no >> effect when set already has an object equal to the one being added. ?I >> propose to consider changing that behavior to make set.add return the >> added object or the set member that is equal to the object being >> added. ?It is unlikely that many programs rely on the return value >> being None (with doctests being a probable exception), so adding this >> feature is unlikely to cause much grief. > > I had exactly the same idea, but did not post because it violates the > general rule that mutators return None. Is there such a rule? What about set/dict pop? > On the other hand, the returned > value here would not be the mutated collection, so no chaining is possible. I assume you refer to chaining as in s.add(1).add(2) which would be enabled if s.add(..) returned s. My proposal would enable a different type of "chaining" which I would find useful, but ready to hear objections: v = compute_value() s.add(v) # use v can, with my proposal, be rewritten as v = s.add(compute_value()) with an added benefit that v that is used is the "canonical" value. > And 'add' is clearly intended to change something. > Is this an argument for or against the proposal? > On the other hand, frozensets do not have an add method. However, PySet_Add "works with frozenset instances (like PyTuple_SetItem() it can be used to fill-in the values of brand new frozensets before they are exposed to other code). " I will experiment with changing PySet_Add to see how much it would break and whether it will be helpful in implementing set-based interning of python strings. From solipsis at pitrou.net Mon Oct 26 19:10:12 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Oct 2009 18:10:12 +0000 (UTC) Subject: [Python-Dev] Reworking the GIL References: <1256502140.5621.163.camel@localhost> <4AE5C45A.9090900@molden.no> Message-ID: Daniel Stutzbach stutzbachenterprises.com> writes: > > Do we really need priority requests at all?? They seem counter to your > desire for simplicity and allowing the operating system's scheduler to do > its work. No, they can be disabled (removed) if we prefer. With priority requests disabled, latency results becomes less excellent but still quite good. Running ccbench on a dual core machine gives the following latency results, first with then without priority requets. --- Latency --- (with prio requests) Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 2 ms.) CPU threads=2: 0 ms. (std dev: 2 ms.) CPU threads=3: 0 ms. (std dev: 2 ms.) CPU threads=4: 0 ms. (std dev: 2 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 3 ms. (std dev: 2 ms.) CPU threads=2: 3 ms. (std dev: 2 ms.) CPU threads=3: 3 ms. (std dev: 2 ms.) CPU threads=4: 4 ms. (std dev: 3 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 2 ms.) CPU threads=1: 0 ms. (std dev: 2 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 2 ms.) CPU threads=4: 0 ms. (std dev: 1 ms.) --- Latency --- (without prio requests) Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 2 ms.) CPU threads=1: 5 ms. (std dev: 0 ms.) CPU threads=2: 3 ms. (std dev: 3 ms.) CPU threads=3: 9 ms. (std dev: 7 ms.) CPU threads=4: 22 ms. (std dev: 23 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 1 ms.) CPU threads=1: 8 ms. (std dev: 2 ms.) CPU threads=2: 5 ms. (std dev: 4 ms.) CPU threads=3: 21 ms. (std dev: 32 ms.) CPU threads=4: 19 ms. (std dev: 26 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 1 ms.) CPU threads=1: 0 ms. (std dev: 2 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) From chris at subtlety.com Mon Oct 26 19:54:03 2009 From: chris at subtlety.com (Chris Bergstresser) Date: Mon, 26 Oct 2009 13:54:03 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> Message-ID: <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> On Mon, Oct 26, 2009 at 11:38 AM, Guido van Rossum wrote: > - If sets were to grow an API to non-destructively access the object > stored in the set for a particular key, then dicts should have such a > method too. > > - Ditto for an API to non-destructively get an arbitrary element. > > - I'm far from convinced that we urgently need either API. But I'm > also not convinced it's unneeded. These clearly aren't urgently needed, but I do think they're needed and useful. For those who want a use-case for getting an arbitrary element from a set, I've run into the need several times over the last year, and each time I'm a little surprised I had the need and a little surprised there wasn't an good way of going about it. In the most recent example, I was writing some UI code. I had a set of all the open window references so I could clean them up at the end of the program. I needed to call some API function that required a window reference as the first argument, but it returned a global value common to all the window references. I like the proposed set.get() method, personally. list.get(index) gets the item at that index, dict.get(key) gets the item associated with that key, set.get() gets an item, but doesn't place any guarantees on which item is returned. Makes sense to me. I also like the idea there aren't any guarantees about which item is returned--it allows subclasses to implement different behavior (so one might always return the last item placed in the set, another could always return a random item, another could implement some round-robin behavior, and all would fulfill the contract of the set class). The existing methods aren't great for accomplishing this, mainly because they're obfuscatory. "iter(s).next()" is probably clearest, and at least will throw a StopIteration exception if the set is empty. "for x in s: break" is just confusing, easy to accidentally confuse with "for x in s: pass", and causes an unrevealing NameError if the set is empty. Add in all the other idioms for accomplishing the same thing ("x, = s", etc.) and I think there's a good argument for adding the method to sets, if only to provide a single obvious way of doing it--and throwing a single, appropriate exception if the set is empty. -- Chris From tjreedy at udel.edu Mon Oct 26 20:20:55 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Oct 2009 15:20:55 -0400 Subject: [Python-Dev] One obvious way to do interning [Was: Retrieve an arbitrary element from a set without removing it] In-Reply-To: References: Message-ID: Alexander Belopolsky wrote: > Changing the subject to reflect branched discussion and forwarding to > python-ideas where it probably belongs. > > On Mon, Oct 26, 2009 at 12:02 PM, Terry Reedy wrote: >> Alexander Belopolsky wrote: >> >>> Here is an alternative idea on how storing interned objects in a set >>> can be supported. Currently set.add method returns None and has no >>> effect when set already has an object equal to the one being added. I >>> propose to consider changing that behavior to make set.add return the >>> added object or the set member that is equal to the object being >>> added. It is unlikely that many programs rely on the return value >>> being None (with doctests being a probable exception), so adding this >>> feature is unlikely to cause much grief. >> I had exactly the same idea, but did not post because it violates the >> general rule that mutators return None. > > Is there such a rule? What about set/dict pop? The rule perhaps should be restated as 'Collection mutators return None or possible an item but not the collection.' >> On the other hand, the returned >> value here would not be the mutated collection, so no chaining is possible. >> And 'add' is clearly intended to change something. >> > Is this an argument for or against the proposal? Mildly for. >> On the other hand, frozensets do not have an add method. > > However, PySet_Add "works with frozenset instances (like > PyTuple_SetItem() it can be used to fill-in the values of brand new > frozensets before they are exposed to other code). " > > > I will experiment with changing PySet_Add to see how much it would > break and whether it will be helpful in implementing set-based > interning of python strings. Terry Jan Reedy From python at rcn.com Mon Oct 26 20:23:31 2009 From: python at rcn.com (Raymond Hettinger) Date: Mon, 26 Oct 2009 12:23:31 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it References: <200910231132.45504.w.richert@gmx.net><4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> Message-ID: <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> Chris Bergstresser] > I like the proposed set.get() method, personally. Sets have been implemented in many languages, but I've only seen one that implemented this functionality (the "arb" function in SETL). For the most part, it seems that this is an uncommon need. Also consider that there is value to keeping the set-api as simple as possible. Right now, anyone who was exposed to the basics of sets in school can understand the set-api with a near zero learning curve. Some of that would be lost if you add methods that make identity/equality distinctions or that fetch the same arbitrary value over and over again. Besides, it is trivial to write a short function that encapsulates this behavior if it is part of your personal style of expression. > The existing methods aren't great for accomplishing this, mainly > because they're obfuscatory. "iter(s).next()" is probably clearest, > and at least will throw a StopIteration exception if the set is empty. > "for x in s: break" is just confusing ... A short comment would do wonders. Raymond P.S. It is weird that this thread is gaining traction at the same time as the moratorium thread. Go figure :-) From guido at python.org Mon Oct 26 20:56:02 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Oct 2009 12:56:02 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> Message-ID: On Mon, Oct 26, 2009 at 12:23 PM, Raymond Hettinger wrote: > P.S. ?It is weird that this thread is gaining traction at the same > time as the moratorium thread. ?Go figure :-) I'm beginning to think maybe adding a method to a built-in object type *should* fall under the moratorium. -- --Guido van Rossum PS. My elbow needs a couple more weeks of rest. Limiting myself to ultra-short emails. From python at rcn.com Mon Oct 26 20:56:54 2009 From: python at rcn.com (Raymond Hettinger) Date: Mon, 26 Oct 2009 12:56:54 -0700 Subject: [Python-Dev] Set methods for mapping views Message-ID: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> [GvR] > I still wish we could go back in time and unify sets and dicts, if > only to find out how that experiment would turn out. I'm curious about the outcome of another experiment along those lines. Is anyone seeing uptake for the set methods on mapping views in Py3.x? I haven't seen any code using it, nor any bug reports or documentation requests, nor any code in the ASPN cookbook, nor mention of it on the newsgroup or python-help. Has anyone here seen any hints about how this is faring in the wild? Raymond From collinw at gmail.com Mon Oct 26 21:01:34 2009 From: collinw at gmail.com (Collin Winter) Date: Mon, 26 Oct 2009 13:01:34 -0700 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <1256502140.5621.163.camel@localhost> References: <1256502140.5621.163.camel@localhost> Message-ID: <43aa6ff70910261301l2961a513m817796019ee1d13f@mail.gmail.com> On Sun, Oct 25, 2009 at 1:22 PM, Antoine Pitrou wrote: > Having other people test it would be fine. Even better if you have an > actual multi-threaded py3k application. But ccbench results for other > OSes would be nice too :-) My results for an 2.4 GHz Intel Core 2 Duo MacBook Pro (OS X 10.5.8): Control (py3k @ r75723) --- Throughput --- Pi calculation (Python) threads=1: 633 iterations/s. threads=2: 468 ( 74 %) threads=3: 443 ( 70 %) threads=4: 442 ( 69 %) regular expression (C) threads=1: 281 iterations/s. threads=2: 282 ( 100 %) threads=3: 282 ( 100 %) threads=4: 282 ( 100 %) bz2 compression (C) threads=1: 379 iterations/s. threads=2: 735 ( 193 %) threads=3: 733 ( 193 %) threads=4: 724 ( 190 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 1 ms. (std dev: 1 ms.) CPU threads=2: 1 ms. (std dev: 2 ms.) CPU threads=3: 3 ms. (std dev: 6 ms.) CPU threads=4: 2 ms. (std dev: 3 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 975 ms. (std dev: 577 ms.) CPU threads=2: 1035 ms. (std dev: 571 ms.) CPU threads=3: 1098 ms. (std dev: 556 ms.) CPU threads=4: 1195 ms. (std dev: 557 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 2 ms.) CPU threads=2: 4 ms. (std dev: 5 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 1 ms. (std dev: 4 ms.) Experiment (newgil branch @ r75723) --- Throughput --- Pi calculation (Python) threads=1: 651 iterations/s. threads=2: 643 ( 98 %) threads=3: 637 ( 97 %) threads=4: 625 ( 95 %) regular expression (C) threads=1: 298 iterations/s. threads=2: 296 ( 99 %) threads=3: 288 ( 96 %) threads=4: 287 ( 96 %) bz2 compression (C) threads=1: 378 iterations/s. threads=2: 720 ( 190 %) threads=3: 724 ( 191 %) threads=4: 718 ( 189 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 1 ms.) CPU threads=2: 0 ms. (std dev: 1 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 1 ms. (std dev: 5 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 1 ms. (std dev: 0 ms.) CPU threads=2: 2 ms. (std dev: 1 ms.) CPU threads=3: 2 ms. (std dev: 2 ms.) CPU threads=4: 2 ms. (std dev: 1 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 2 ms. (std dev: 3 ms.) CPU threads=3: 0 ms. (std dev: 1 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) I also ran this through Unladen Swallow's threading microbenchmark, which is a straight copy of what David Beazley was experimenting with (simply iterating over 1000000 ints in pure Python) [1]. "iterative_count" is doing the loops one after the other, "threaded_count" is doing the loops in parallel using threads. The results below are benchmarking py3k as the control, newgil as the experiment. When it says "x% faster", that is a measure of newgil's performance over py3k's. With two threads: iterative_count: Min: 0.336573 -> 0.387782: 13.21% slower # I've run this configuration multiple times and gotten the same slowdown. Avg: 0.338473 -> 0.418559: 19.13% slower Significant (t=-38.434785, a=0.95) threaded_count: Min: 0.529859 -> 0.397134: 33.42% faster Avg: 0.581786 -> 0.429933: 35.32% faster Significant (t=70.100445, a=0.95) With four threads: iterative_count: Min: 0.766617 -> 0.734354: 4.39% faster Avg: 0.771954 -> 0.751374: 2.74% faster Significant (t=22.164103, a=0.95) Stddev: 0.00262 -> 0.00891: 70.53% larger threaded_count: Min: 1.175750 -> 0.829181: 41.80% faster Avg: 1.224157 -> 0.867506: 41.11% faster Significant (t=161.715477, a=0.95) Stddev: 0.01900 -> 0.01120: 69.65% smaller With eight threads: iterative_count: Min: 1.527794 -> 1.447421: 5.55% faster Avg: 1.536911 -> 1.479940: 3.85% faster Significant (t=35.559595, a=0.95) Stddev: 0.00394 -> 0.01553: 74.61% larger threaded_count: Min: 2.424553 -> 1.677180: 44.56% faster Avg: 2.484922 -> 1.723093: 44.21% faster Significant (t=184.766131, a=0.95) Stddev: 0.02874 -> 0.02956: 2.78% larger I'd be interested in multithreaded benchmarks with less-homogenous workloads. Collin Winter [1] - http://code.google.com/p/unladen-swallow/source/browse/tests/performance/bm_threading.py From jnoller at gmail.com Mon Oct 26 21:06:29 2009 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 26 Oct 2009 16:06:29 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> Message-ID: <4222a8490910261306p4d72bb54u403d60ea843c962f@mail.gmail.com> On Mon, Oct 26, 2009 at 3:56 PM, Guido van Rossum wrote: > On Mon, Oct 26, 2009 at 12:23 PM, Raymond Hettinger wrote: >> P.S. ?It is weird that this thread is gaining traction at the same >> time as the moratorium thread. ?Go figure :-) > > I'm beginning to think maybe adding a method to a built-in object type > *should* fall under the moratorium. So far, fiddling with the PEP, I'm on the fence - adding a method to a built-in object type is sort of a grey area (at least in my mind). It depends on if doing so significantly alters the language/syntax. jesse From debatem1 at gmail.com Mon Oct 26 21:08:45 2009 From: debatem1 at gmail.com (geremy condra) Date: Mon, 26 Oct 2009 16:08:45 -0400 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> Message-ID: On Mon, Oct 26, 2009 at 3:56 PM, Raymond Hettinger wrote: > [GvR] >> >> I still wish we could go back in time and unify sets and dicts, if >> only to find out how that experiment would turn out. > > I'm curious about the outcome of another experiment along those lines. > Is anyone seeing uptake for the set methods on mapping views in Py3.x? > > I haven't seen any code using it, nor any bug reports or documentation > requests, nor any code in the ASPN cookbook, nor mention of it on the > newsgroup or python-help. > > Has anyone here seen any hints about how this is faring in the wild? > > > Raymond > Next version of Graphine will use them heavily for graph merges and set operations on graphs. Geremy Condra From debatem1 at gmail.com Mon Oct 26 21:14:58 2009 From: debatem1 at gmail.com (geremy condra) Date: Mon, 26 Oct 2009 16:14:58 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> Message-ID: On Mon, Oct 26, 2009 at 3:56 PM, Guido van Rossum wrote: > On Mon, Oct 26, 2009 at 12:23 PM, Raymond Hettinger wrote: >> P.S. ?It is weird that this thread is gaining traction at the same >> time as the moratorium thread. ?Go figure :-) > > I'm beginning to think maybe adding a method to a built-in object type > *should* fall under the moratorium. > > -- > --Guido van Rossum Seems like any changes requiring support from uninvolved developers should fall under the moratorium. If the proposal is to add a set.get() function to the set API, then this clearly falls under that heading. Geremy Condra From solipsis at pitrou.net Mon Oct 26 21:19:22 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Oct 2009 20:19:22 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Retrieve_an_arbitrary_element_from_a_set?= =?utf-8?q?=09withoutremoving_it?= References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> <4222a8490910261306p4d72bb54u403d60ea843c962f@mail.gmail.com> Message-ID: Jesse Noller gmail.com> writes: > > So far, fiddling with the PEP, I'm on the fence - adding a method to a > built-in object type is sort of a grey area (at least in my mind). It > depends on if doing so significantly alters the language/syntax. We have recently added things like float.fromhex() which IMHO shouldn't be blocked by the moratorium (*), although they technically add a method to a built-in. (*) it is a minor new feature aimed at catching up with some established standard for an exact, non-ambiguous string representation of floats Regards Antoine. From python at rcn.com Mon Oct 26 21:20:24 2009 From: python at rcn.com (Raymond Hettinger) Date: Mon, 26 Oct 2009 13:20:24 -0700 Subject: [Python-Dev] Set methods for mapping views References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> Message-ID: >> I'm curious about the outcome of another experiment along those lines. >> Is anyone seeing uptake for the set methods on mapping views in Py3.x? > Next version of Graphine will use them heavily for graph > merges and set operations on graphs. That's great! Thanks for the data point. I look forward to reading the code. Raymond From dickinsm at gmail.com Mon Oct 26 21:26:51 2009 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 26 Oct 2009 20:26:51 +0000 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> Message-ID: <5c6f2a5d0910261326x518a8892w86f9b76ee33b9f9d@mail.gmail.com> On Mon, Oct 26, 2009 at 7:56 PM, Guido van Rossum wrote: > I'm beginning to think maybe adding a method to a built-in object type > *should* fall under the moratorium. Another possible test case for this decision is the int.from_bytes and int.to_bytes methods, proposed a while ago, and implemented by Alexandre Vassalotti. See: http://bugs.python.org/issue1023290 Mark From guido at python.org Mon Oct 26 21:32:32 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Oct 2009 13:32:32 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <55A46E12112349EFAE8C2C8E7E6532B6@RaymondLaptop1> <4222a8490910261306p4d72bb54u403d60ea843c962f@mail.gmail.com> Message-ID: On Mon, Oct 26, 2009 at 1:19 PM, Antoine Pitrou wrote: > Jesse Noller gmail.com> writes: >> >> So far, fiddling with the PEP, I'm on the fence - adding a method to a >> built-in object type is sort of a grey area (at least in my mind). It >> depends on if doing so significantly alters the language/syntax. > > We have recently added things like float.fromhex() which IMHO shouldn't be > blocked by the moratorium (*), although they technically add a method to a > built-in. > > (*) it is a minor new feature aimed at catching up with some established > standard for an exact, non-ambiguous string representation of floats Okay, so it remains a gray area. -- --Guido van Rossum PS. My elbow needs a couple more weeks of rest. Limiting myself to ultra-short emails. From w.richert at gmx.net Mon Oct 26 22:14:11 2009 From: w.richert at gmx.net (Willi Richert) Date: Mon, 26 Oct 2009 22:14:11 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> Message-ID: <200910262214.11662.w.richert@gmx.net> For those of you who want to tinker with it, I posted the patch against the current trunk at http://bugs.python.org/issue7212 Have fun, wr Am Montag, 26. Oktober 2009 21:32:32 schrieb Guido van Rossum: > On Mon, Oct 26, 2009 at 1:19 PM, Antoine Pitrou wrote: > > Jesse Noller gmail.com> writes: > >> So far, fiddling with the PEP, I'm on the fence - adding a method to a > >> built-in object type is sort of a grey area (at least in my mind). It > >> depends on if doing so significantly alters the language/syntax. > > > > We have recently added things like float.fromhex() which IMHO shouldn't > > be blocked by the moratorium (*), although they technically add a method > > to a built-in. > > > > (*) it is a minor new feature aimed at catching up with some established > > standard for an exact, non-ambiguous string representation of floats > > Okay, so it remains a gray area. > From solipsis at pitrou.net Mon Oct 26 22:43:50 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Oct 2009 21:43:50 +0000 (UTC) Subject: [Python-Dev] Reworking the GIL References: <1256502140.5621.163.camel@localhost> <43aa6ff70910261301l2961a513m817796019ee1d13f@mail.gmail.com> Message-ID: Collin Winter gmail.com> writes: > > My results for an 2.4 GHz Intel Core 2 Duo MacBook Pro (OS X 10.5.8): Thanks! [the Dave Beazley benchmark] > The results below are benchmarking py3k as the control, newgil as the > experiment. When it says "x% faster", that is a measure of newgil's > performance over py3k's. > > With two threads: > > iterative_count: > Min: 0.336573 -> 0.387782: 13.21% slower # I've run this > configuration multiple times and gotten the same slowdown. > Avg: 0.338473 -> 0.418559: 19.13% slower Those numbers are not very in line with the other "iterative_count" results. Since iterative_count just runs the loop N times in a row, results should be proportional to the number N ("number of threads"). Besides, there's no reason for single-threaded performance to be degraded since the fast path of the eval loop actually got a bit streamlined (there is no volatile ticker to decrement). > I'd be interested in multithreaded benchmarks with less-homogenous workloads. So would I. Regards Antoine. From collinw at gmail.com Mon Oct 26 22:50:14 2009 From: collinw at gmail.com (Collin Winter) Date: Mon, 26 Oct 2009 14:50:14 -0700 Subject: [Python-Dev] Reworking the GIL In-Reply-To: References: <1256502140.5621.163.camel@localhost> <43aa6ff70910261301l2961a513m817796019ee1d13f@mail.gmail.com> Message-ID: <43aa6ff70910261450m23e66c4er654e7e425ddda31b@mail.gmail.com> On Mon, Oct 26, 2009 at 2:43 PM, Antoine Pitrou wrote: > Collin Winter gmail.com> writes: > [the Dave Beazley benchmark] >> The results below are benchmarking py3k as the control, newgil as the >> experiment. When it says "x% faster", that is a measure of newgil's >> performance over py3k's. >> >> With two threads: >> >> iterative_count: >> Min: 0.336573 -> 0.387782: 13.21% slower ?# I've run this >> configuration multiple times and gotten the same slowdown. >> Avg: 0.338473 -> 0.418559: 19.13% slower > > Those numbers are not very in line with the other "iterative_count" results. > Since iterative_count just runs the loop N times in a row, results should be > proportional to the number N ("number of threads"). > > Besides, there's no reason for single-threaded performance to be degraded since > the fast path of the eval loop actually got a bit streamlined (there is no > volatile ticker to decrement). I agree those numbers are out of line with the others and make no sense. I've run it with two threads several times and the results are consistent on this machine. I'm digging into it a bit more. Collin From ncoghlan at gmail.com Mon Oct 26 22:50:56 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Oct 2009 07:50:56 +1000 Subject: [Python-Dev] One obvious way to do interning [Was: Retrieve an arbitrary element from a set without removing it] In-Reply-To: References: Message-ID: <4AE619C0.803@gmail.com> Terry Reedy wrote: > Alexander Belopolsky wrote: >> Terry Reedy wrote: >>> I had exactly the same idea, but did not post because it violates the >>> general rule that mutators return None. >> >> Is there such a rule? What about set/dict pop? > > The rule perhaps should be restated as 'Collection mutators return None > or possible an item but not the collection.' And to clarify the rationale for that guideline: it is to make it clear that the mutator is changing the container in place and *not* creating a new container object. myset.pop() # No new container, returns popped object mylist.sort() # No new container, returns None sorted(mylist) # New container, so return it mystr.lower() # Creates new string, so return it Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Oct 26 22:54:52 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Oct 2009 07:54:52 +1000 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> Message-ID: <4AE61AAC.6020806@gmail.com> Raymond Hettinger wrote: > [GvR] >> I still wish we could go back in time and unify sets and dicts, if >> only to find out how that experiment would turn out. > > I'm curious about the outcome of another experiment along those lines. > Is anyone seeing uptake for the set methods on mapping views in Py3.x? > > I haven't seen any code using it, nor any bug reports or documentation > requests, nor any code in the ASPN cookbook, nor mention of it on the > newsgroup or python-help. > > Has anyone here seen any hints about how this is faring in the wild? If anyone is looking for further explanation as to why Guido's moratorium on core language changes is a good idea, allowing a chance for answers to questions like Raymond's above a chance to evolve naturally is what I see as the most important rationale. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From exarkun at twistedmatrix.com Mon Oct 26 23:45:25 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 26 Oct 2009 22:45:25 -0000 Subject: [Python-Dev] Reworking the GIL In-Reply-To: References: <1256502140.5621.163.camel@localhost> <4AE5C45A.9090900@molden.no> Message-ID: <20091026224525.31526.1835917647.divmod.xquotient.110@localhost.localdomain> On 04:18 pm, daniel at stutzbachenterprises.com wrote: >On Mon, Oct 26, 2009 at 10:58 AM, Antoine Pitrou >wrote: >>Er, I prefer to keep things simple. If you have lots of I/O you should >>probably >>use an event loop rather than separate threads. > >On Windows, sometimes using a single-threaded event loop is sometimes >impossible. WaitForMultipleObjects(), which is the Windows equivalent >to >select() or poll(), can handle a maximum of only 64 objects. This is only partially accurate. For one thing, WaitForMultipleObjects calls are nestable. For another thing, Windows also has I/O completion ports which are not limited to 64 event sources. The situation is actually better than on a lot of POSIXes. >Do we really need priority requests at all? They seem counter to your >desire for simplicity and allowing the operating system's scheduler to >do >its work. Despite what I said above, however, I would also take a default position against adding any kind of more advanced scheduling system here. It would, perhaps, make sense to expose the APIs for controlling the platform scheduler, though. Jean-Paul From barry at python.org Mon Oct 26 23:48:46 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 26 Oct 2009 18:48:46 -0400 Subject: [Python-Dev] RELEASED Python 2.6.4 Message-ID: On behalf of the Python community, I'm happy to announce the availability of Python 2.6.4. This is the latest production-ready version in the Python 2.6 series. We had a little trouble with the Python 2.6.3 release; a number of unfortunate regressions were introduced. I take responsibility for rushing it out, but the good news is that Python 2.6.4 fixes the known regressions in 2.6.3. We've had a lengthy release candidate cycle this time, and are confident that 2.6.4 is a solid release. We highly recommend you upgrade to Python 2.6.4. Please see the NEWS file for all the gory details. http://www.python.org/download/releases/2.6.4/NEWS.txt Source tarballs and the Windows installers can be downloaded from the Python 2.6.4 page. The Mac OS X disk image will be uploaded soon. http://www.python.org/download/releases/2.6.4/ For more information on Python 2.6 in general, please see http://docs.python.org/whatsnew/2.6.html Please report bugs for any Python version in the Python tracker. http://bugs.python.org Enjoy, -Barry Barry Warsaw barry at python.org Python 2.6 Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From steve at pearwood.info Mon Oct 26 23:59:43 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Oct 2009 09:59:43 +1100 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: <4AE61AAC.6020806@gmail.com> References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> <4AE61AAC.6020806@gmail.com> Message-ID: <200910270959.44539.steve@pearwood.info> On Tue, 27 Oct 2009 08:54:52 am Nick Coghlan wrote: > Raymond Hettinger wrote: > > [GvR] > > > >> I still wish we could go back in time and unify sets and dicts, if > >> only to find out how that experiment would turn out. > > > > I'm curious about the outcome of another experiment along those > > lines. Is anyone seeing uptake for the set methods on mapping views > > in Py3.x? > > > > I haven't seen any code using it, nor any bug reports or > > documentation requests, nor any code in the ASPN cookbook, nor > > mention of it on the newsgroup or python-help. > > > > Has anyone here seen any hints about how this is faring in the > > wild? > > If anyone is looking for further explanation as to why Guido's > moratorium on core language changes is a good idea, allowing a chance > for answers to questions like Raymond's above a chance to evolve > naturally is what I see as the most important rationale. I don't understand that rationale. Let's take a concrete example. The new `yield from` syntax was accepted but now will be delayed by the moratorium. How would the addition of `yield from` delay or prevent people using set methods on mapping views? If a proposed feature directly competes with a feature in 3.x, then it might delay usage of the 3.x feature -- but if that were the case, the proposal would almost certainly be rejected on the basis that 3.x already has a feature to do that very thing. -- Steven D'Aprano From solipsis at pitrou.net Tue Oct 27 00:06:15 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Oct 2009 23:06:15 +0000 (UTC) Subject: [Python-Dev] Set methods for mapping views References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> <4AE61AAC.6020806@gmail.com> <200910270959.44539.steve@pearwood.info> Message-ID: Steven D'Aprano pearwood.info> writes: > > I don't understand that rationale. Let's take a concrete example. The > new `yield from` syntax was accepted Was it? From guido at python.org Tue Oct 27 00:09:14 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Oct 2009 16:09:14 -0700 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> <4AE61AAC.6020806@gmail.com> <200910270959.44539.steve@pearwood.info> Message-ID: On Mon, Oct 26, 2009 at 4:06 PM, Antoine Pitrou wrote: > Steven D'Aprano pearwood.info> writes: >> >> I don't understand that rationale. Let's take a concrete example. The >> new `yield from` syntax was accepted > > Was it? No. -- --Guido van Rossum PS. My elbow needs a couple more weeks of rest. Limiting myself to ultra-short emails. From steve at pearwood.info Tue Oct 27 00:29:58 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Oct 2009 10:29:58 +1100 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> Message-ID: <200910271029.58591.steve@pearwood.info> On Tue, 27 Oct 2009 10:09:14 am Guido van Rossum wrote: > On Mon, Oct 26, 2009 at 4:06 PM, Antoine Pitrou wrote: > > Steven D'Aprano pearwood.info> writes: > >> I don't understand that rationale. Let's take a concrete example. > >> The new `yield from` syntax was accepted > > > > Was it? > > No. I thought it had been. My mistake. Serves me right for not checking the PEP first. But my point still stands. -- Steven D'Aprano From ssteinerx at gmail.com Tue Oct 27 01:02:03 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Mon, 26 Oct 2009 20:02:03 -0400 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <20091026224525.31526.1835917647.divmod.xquotient.110@localhost.localdomain> References: <1256502140.5621.163.camel@localhost> <4AE5C45A.9090900@molden.no> <20091026224525.31526.1835917647.divmod.xquotient.110@localhost.localdomain> Message-ID: On Oct 26, 2009, at 6:45 PM, exarkun at twistedmatrix.com wrote: >> Despite what I said above, however, I would also take a default >> position against adding any kind of more advanced scheduling system >> here. It would, perhaps, make sense to expose the APIs for >> controlling the platform scheduler, though. I would also like to have an exposed API and optional profiling (optional as in conditional compilation, not as in some sort of profiling 'flag' that slows down non-profiling runs) so that you can see what's going on well enough to use the API to tune a particular platform for a particular workload. S From cs at zip.com.au Tue Oct 27 02:09:53 2009 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 27 Oct 2009 12:09:53 +1100 Subject: [Python-Dev] Reworking the GIL In-Reply-To: <20091026224525.31526.1835917647.divmod.xquotient.110@localhost.localdomain> References: <20091026224525.31526.1835917647.divmod.xquotient.110@localhost.localdomain> Message-ID: <20091027010953.GA19484@cskk.homeip.net> On 26Oct2009 22:45, exarkun at twistedmatrix.com wrote: | On 04:18 pm, daniel at stutzbachenterprises.com wrote: | >On Mon, Oct 26, 2009 at 10:58 AM, Antoine Pitrou write: | >Do we really need priority requests at all? They seem counter to your | >desire for simplicity and allowing the operating system's | >scheduler to do | >its work. | | Despite what I said above, however, I would also take a default | position against adding any kind of more advanced scheduling system | here. It would, perhaps, make sense to expose the APIs for | controlling the platform scheduler, though. +1 to both sentences from me. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Plague, Famine, Pestilence, and C++ stalk the land. We're doomed! Doomed! - Simon E Spero From brett at python.org Tue Oct 27 04:13:28 2009 From: brett at python.org (Brett Cannon) Date: Mon, 26 Oct 2009 20:13:28 -0700 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: <200910270959.44539.steve@pearwood.info> References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> <4AE61AAC.6020806@gmail.com> <200910270959.44539.steve@pearwood.info> Message-ID: On Mon, Oct 26, 2009 at 15:59, Steven D'Aprano wrote: > On Tue, 27 Oct 2009 08:54:52 am Nick Coghlan wrote: > > Raymond Hettinger wrote: > > > [GvR] > > > > > >> I still wish we could go back in time and unify sets and dicts, if > > >> only to find out how that experiment would turn out. > > > > > > I'm curious about the outcome of another experiment along those > > > lines. Is anyone seeing uptake for the set methods on mapping views > > > in Py3.x? > > > > > > I haven't seen any code using it, nor any bug reports or > > > documentation requests, nor any code in the ASPN cookbook, nor > > > mention of it on the newsgroup or python-help. > > > > > > Has anyone here seen any hints about how this is faring in the > > > wild? > > > > If anyone is looking for further explanation as to why Guido's > > moratorium on core language changes is a good idea, allowing a chance > > for answers to questions like Raymond's above a chance to evolve > > naturally is what I see as the most important rationale. > > I don't understand that rationale. Let's take a concrete example. The > new `yield from` syntax was accepted but now will be delayed by the > moratorium. How would the addition of `yield from` delay or prevent > people using set methods on mapping views? > > It doesn't, but the point is we have already added several things to the language in Python 3 that have gone mostly unused from the community thus far. We do not need to continue to pile on the new features when we already have a stack that we need to see if they pan out. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 27 11:58:47 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Oct 2009 10:58:47 +0000 (UTC) Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <20091026005815.31526.130687717.divmod.xquotient.88@localhost.localdomain> <20091026015650.31526.600593069.divmod.xquotient.94@localhost.localdomain> <4AE55E39.20000@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > It's not really reproducible. I think it sometimes happens when I > restart the master; sometimes, some clients fail to reconnect > (properly). Another common problem is that some buildbot fails in the middle of the test suite, with the following kind of message: command timed out: 1800 seconds without output, killing pid 12325 process killed by signal 9 program finished with exit code -1 elapsedTime=10910.362981 See for example : http://www.python.org/dev/buildbot/trunk.stable/builders/ia64%20Ubuntu%20trunk/builds/73/steps/test/logs/stdio (notice, by the way, the elapsed time (10910s, that is, close to 3 hours...)) Regards Antoine. From zookog at gmail.com Tue Oct 27 16:41:42 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Tue, 27 Oct 2009 09:41:42 -0600 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <4AE4A5DC.2010906@v.loewis.de> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <4AE4A5DC.2010906@v.loewis.de> Message-ID: Right, how do developers benefit from a buildbot? >From my experience (five large buildbots with many developers plus two with only a couple of developers), a buildbot does little good unless the tests are reliable and not too noisy. "Reliable" is best achieved by having tests be deterministic and reproducible. "Not too noisy" means that the builders are all green all the time (at least for a "supported" subset of the buildslaves). Beyond that, then I think there has to be a culture change where the development team decides that it is really, really not okay to leave a builder red after you turned it red, and that instead you need to revert the patch that made it go from green to red before you do anything else. It has taken me a long time to acculturate to that and I wouldn't expect most people to do it quickly or easily. (It is interesting to think of what would happen if that policy were automated -- any patch which caused any "supported" builder to go from green to red would be automatically be reverted.) Also, of course, this is mostly meaningless unless the code that is being changed by the patches is well-covered by tests. Regards, Zooko From g.brandl at gmx.net Tue Oct 27 17:06:44 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 27 Oct 2009 17:06:44 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> Message-ID: Chris Bergstresser schrieb: > I like the proposed set.get() method, personally. list.get(index) > gets the item at that index, dict.get(key) gets the item associated > with that key, set.get() gets an item, but doesn't place any > guarantees on which item is returned. Sorry to nitpick, but there is no list.get(). Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From chris at subtlety.com Tue Oct 27 18:33:29 2009 From: chris at subtlety.com (Chris Bergstresser) Date: Tue, 27 Oct 2009 12:33:29 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> Message-ID: <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> On Tue, Oct 27, 2009 at 11:06 AM, Georg Brandl wrote: > Sorry to nitpick, but there is no list.get(). No? How ... odd. I guess it wouldn't have come up, but I was sure there was a .get method which took an optional default parameter if the index didn't exist, mirroring the dict method. Still, I think my point stands--it's a clear extrapolation from the existing dict.get(). -- Chris From g.brandl at gmx.net Tue Oct 27 18:40:45 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 27 Oct 2009 18:40:45 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: Chris Bergstresser schrieb: > On Tue, Oct 27, 2009 at 11:06 AM, Georg Brandl wrote: >> Sorry to nitpick, but there is no list.get(). > > No? How ... odd. I guess it wouldn't have come up, but I was sure > there was a .get method which took an optional default parameter if > the index didn't exist, mirroring the dict method. Still, I think my > point stands--it's a clear extrapolation from the existing dict.get(). I don't see that. Both dict.get() and your hypothetical list.get() are variants of the [] subscription operator, i.e. __getitem__, that have a default value, defaulting to None. The [] operator retrieves an element from the object at the specified "position", be it dict key, list index or some other abstract position. Contrary to that, sets don't support subscript access, there is no notion of a value at a specified position, so I would find the set.get() naming confusing. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From python at rcn.com Tue Oct 27 18:47:59 2009 From: python at rcn.com (Raymond Hettinger) Date: Tue, 27 Oct 2009 10:47:59 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it References: <200910231132.45504.w.richert@gmx.net><200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: [Chris Bergstresser] Still, I think my > point stands--it's a clear extrapolation from the existing dict.get(). Not really. One looks-up a key and supplies a default value if not found. The other, set.get(), doesn't have a key to lookup. A dict.get() can be meaningfully used in a loop (because the key can vary). A set.get() returns the same value over and over again (because there is no key). Raymond From alexander.belopolsky at gmail.com Tue Oct 27 18:50:56 2009 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 27 Oct 2009 13:50:56 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: On Tue, Oct 27, 2009 at 1:33 PM, Chris Bergstresser wrote: > On Tue, Oct 27, 2009 at 11:06 AM, Georg Brandl wrote: >> Sorry to nitpick, but there is no list.get(). > > ? No? ?How ... odd. Odd indeed. My first reaction was: it is not needed because lists support slicing, but when I tried to construct a list.get() using slicing the best I could come up with was the following hack >>> def lget(l, i, v): return (l[i:] or [v])[0] ... >>> lget(range(10), 20, 200) 200 >>> lget(range(10), 5, 50) 5 Yet for some reason I never missed this functionality ... From solipsis at pitrou.net Tue Oct 27 18:59:00 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Oct 2009 17:59:00 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Retrieve_an_arbitrary_element_from_a_set?= =?utf-8?q?=09withoutremoving_it?= References: <200910231132.45504.w.richert@gmx.net><200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: Raymond Hettinger rcn.com> writes: > > [Chris Bergstresser] > Still, I think my > > point stands--it's a clear extrapolation from the existing dict.get(). > > Not really. One looks-up a key and supplies a default value if not found. > The other, set.get(), doesn't have a key to lookup. set.getone() then ? From debatem1 at gmail.com Tue Oct 27 19:20:04 2009 From: debatem1 at gmail.com (geremy condra) Date: Tue, 27 Oct 2009 14:20:04 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: On Tue, Oct 27, 2009 at 1:59 PM, Antoine Pitrou wrote: > Raymond Hettinger rcn.com> writes: >> >> [Chris Bergstresser] >> ? Still, I think my >> > point stands--it's a clear extrapolation from the existing dict.get(). >> >> Not really. ?One looks-up a key and supplies a default value if not found. >> The other, set.get(), doesn't have a key to lookup. > > set.getone() then ? ugh- other spellings much preferred. Geremy Condra From phd at phd.pp.ru Tue Oct 27 19:30:30 2009 From: phd at phd.pp.ru (Oleg Broytman) Date: Tue, 27 Oct 2009 21:30:30 +0300 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <20091027183030.GA29335@phd.pp.ru> On Tue, Oct 27, 2009 at 02:20:04PM -0400, geremy condra wrote: > On Tue, Oct 27, 2009 at 1:59 PM, Antoine Pitrou wrote: > > Raymond Hettinger rcn.com> writes: > > set.getone() then ? > > ugh- other spellings much preferred. set[] ? (Just kidding, really.) Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From tjreedy at udel.edu Tue Oct 27 19:50:28 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Oct 2009 14:50:28 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net><200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: Raymond Hettinger wrote: > > A dict.get() can be meaningfully used in a loop (because the key can vary). > A set.get() returns the same value over and over again (because there is > no key). There are two ideas of set.get floating about: 1) get an arbitrary object 2) get the object in the set with the same 'value'(hash+eq) as an input arg (the intern case). In this case, there is a 'key', even if it is somewhat abstract rather that being an object. Both could be done with the same method, depending on whether an arg is passed or not. The former is not useful in a loop; the latter could be. If there is no match in case 2, the method could a) raise an exception, b) return None (which by its nature would never sensibly be looked up), or c) return an object specified by 'default=xxx' keyword arg. Terry Jan Reedy From chris at subtlety.com Tue Oct 27 20:02:18 2009 From: chris at subtlety.com (Chris Bergstresser) Date: Tue, 27 Oct 2009 14:02:18 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <7b0045ed0910271202h4e8f762cqdd48373c52af8f0a@mail.gmail.com> On Tue, Oct 27, 2009 at 12:47 PM, Raymond Hettinger wrote: > [Chris Bergstresser] > ?Still, I think my >> >> point stands--it's a clear extrapolation from the existing dict.get(). > > Not really. ?One looks-up a key and supplies a default value if not found. > The other, set.get(), doesn't have a key to lookup. Right, which is why dict.get() takes a key as an argument, while the proposed set.get() wouldn't. > A dict.get() can be meaningfully used in a loop (because the key can vary). > A set.get() returns the same value over and over again (because there is no > key). I don't think "can it be used meaningfully in a loop?" is an especially interesting or useful way of evaluating language features. Besides, why would set.get() necessarily return the same value over and over again? I thought it would be defined to return an arbitrary value--which could be the same value over and over again, but could just as easily be defined to return a round-robin value, or the minimum value, or some *other* value as the container defined it. The fact is, set.get() succinctly describes an action which is otherwise obscure in Python. It doesn't come up all that frequently, but when it does the alternatives are poor at best. Add in the uncertainty about which is optimized (imagine the situation where the set you're calling is actually a proxy for an object across the network, and constructing an iterator is expensive) and you start to see the value. -- Chris From ssteinerx at gmail.com Tue Oct 27 20:06:14 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Tue, 27 Oct 2009 15:06:14 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net><200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: On Oct 27, 2009, at 2:50 PM, Terry Reedy wrote more and more and more and more and more and more and more and more and more: This topic needs its own flippin' newsgroup. S From tseaver at palladion.com Tue Oct 27 20:10:25 2009 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 27 Oct 2009 15:10:25 -0400 Subject: [Python-Dev] RELEASED Python 2.6.4 In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > On behalf of the Python community, I'm happy to announce the > availability of Python 2.6.4. This is the latest production-ready > version in the Python 2.6 series. > > We had a little trouble with the Python 2.6.3 release; a number of > unfortunate regressions were introduced. I take responsibility for > rushing it out, but the good news is that Python 2.6.4 fixes the known > regressions in 2.6.3. We've had a lengthy release candidate cycle > this time, and are confident that 2.6.4 is a solid release. We highly > recommend you upgrade to Python 2.6.4. > > Please see the NEWS file for all the gory details. > > http://www.python.org/download/releases/2.6.4/NEWS.txt > > Source tarballs and the Windows installers can be downloaded from the > Python 2.6.4 page. The Mac OS X disk image will be uploaded soon. > > http://www.python.org/download/releases/2.6.4/ > > For more information on Python 2.6 in general, please see > > http://docs.python.org/whatsnew/2.6.html > > Please report bugs for any Python version in the Python tracker. > > http://bugs.python.org Congratulations, and thaks for working to correct the issues with the earlier release so quickly. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrnRaEACgkQ+gerLs4ltQ5yHACgt9CqCyIE8HfkKCNjSEAU3MB1 A70AoL/yJB9lcI+RrBe3/BlqChCweXvr =K/lR -----END PGP SIGNATURE----- From jnoller at gmail.com Tue Oct 27 20:13:23 2009 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 27 Oct 2009 15:13:23 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <4222a8490910271213w7966f501l5456798ff9858bc2@mail.gmail.com> On Tue, Oct 27, 2009 at 3:06 PM, ssteinerX at gmail.com wrote: > > On Oct 27, 2009, at 2:50 PM, Terry Reedy wrote more and more and more and > more and more and more and more and more and more: > > This topic needs its own flippin' newsgroup. > > S Don't like it? Mute the conversation (thank you gmail) or unsub. From guido at python.org Tue Oct 27 20:13:13 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Oct 2009 12:13:13 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: On Tue, Oct 27, 2009 at 11:50 AM, Terry Reedy wrote: > There are two ideas of set.get floating about: > 1) get an arbitrary object > 2) get the object in the set with the same 'value'(hash+eq) as an input arg > (the intern case). In this case, there is a 'key', even if it is somewhat > abstract rather that being an object. > > Both could be done with the same method, depending on whether an arg is > passed or not. My gut tells me it is bad API design to collapse these two use cases. Probably because the implementations won't have much in common: (1) should just pick the first valid element, while (2) should use the normal hash lookup algorithm (shared with 'in', .add() etc.). -- --Guido van Rossum (python.org/~guido) From debatem1 at gmail.com Tue Oct 27 20:27:50 2009 From: debatem1 at gmail.com (geremy condra) Date: Tue, 27 Oct 2009 15:27:50 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: On Tue, Oct 27, 2009 at 3:13 PM, Guido van Rossum wrote: > On Tue, Oct 27, 2009 at 11:50 AM, Terry Reedy wrote: >> There are two ideas of set.get floating about: >> 1) get an arbitrary object >> 2) get the object in the set with the same 'value'(hash+eq) as an input arg >> (the intern case). In this case, there is a 'key', even if it is somewhat >> abstract rather that being an object. >> >> Both could be done with the same method, depending on whether an arg is >> passed or not. > > My gut tells me it is bad API design to collapse these two use cases. > Probably because the implementations won't have much in common: (1) > should just pick the first valid element, while (2) should use the > normal hash lookup algorithm (shared with 'in', .add() etc.). > > -- > --Guido van Rossum (python.org/~guido) Was it ever decided whether this would fall under the moratorium? Geremy Condra From python at rcn.com Tue Oct 27 20:49:01 2009 From: python at rcn.com (Raymond Hettinger) Date: Tue, 27 Oct 2009 12:49:01 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it References: <200910231132.45504.w.richert@gmx.net><7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com><7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <9D7624CE452343719A7ED512B2FD17F5@RaymondLaptop1> [geremy condra] > Was it ever decided whether this would fall under the moratorium? Decided isn't the right word: http://mail.python.org/pipermail/python-dev/2009-October/093373.html FWIW, I'm a strong -1 on both proposals. Just add a short get_one() function and a get_equivalent() recipe to your utils directory. That will get the job done (thought I don't expect that you will *ever* make much use of either one). No need to complexify a type that is currently very simple. Raymond P.S. get_equivalent: http://code.activestate.com/recipes/499299/ get_one = lambda s, default=None: next(iter(s), default) The first works with all type that defines __contains__. The second works for any iterable. Neither of these concepts are specific to set objects. From ziade.tarek at gmail.com Tue Oct 27 22:55:53 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Oct 2009 22:55:53 +0100 Subject: [Python-Dev] Refactoring installation schemes Message-ID: <94bdd2610910271455s7854e6a9u94e3aeeeacb69c02@mail.gmail.com> Hello, Since the addition of PEP 370, (per-user site packages), site.py and distutils/command/install.py are *both* providing the various installation directories for Python, depending on the system and the Python version. We have also started to discuss lately in various Mailing Lists the addition of new schemes for IronPython and Jython, meaning that we might add some more in both places. I would like to suggest a simplification by adding a dedicated module to manage these installation schemes in one single place in the stdlib. This new independant module would be used by site.py and distutils and would also make it easier for third party code to work with these schemes. Of course this new module would be rather simple and not add any new import statement to avoid any overhead when Python starts and loads site.py Regards Tarek From tjreedy at udel.edu Wed Oct 28 04:02:06 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Oct 2009 23:02:06 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net><200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: ssteinerX at gmail.com wrote: > > On Oct 27, 2009, at 2:50 PM, Terry Reedy wrote more and more and more > and more and more and more and more and more and more: Actually, I wrote 7 succinct lines that summarized and made a proposal. In general, I snip when quoting and write concisely as possible. > This topic needs its own flippin' newsgroup. You need to learn politeness. You could have said just that, appropriate or not, without dumping on anyone in particular. Terry Jan Reedy From ssteinerx at gmail.com Wed Oct 28 04:27:00 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Tue, 27 Oct 2009 23:27:00 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net><200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <5CA39441-BE18-40C3-BEA5-330FF38D2AAE@gmail.com> On Oct 27, 2009, at 11:02 PM, Terry Reedy wrote: > ssteinerX at gmail.com wrote: > This topic needs its own flippin' newsgroup. > You could have said just that, appropriate or not, without dumping > on anyone in particular. I was not trying to dump on you in particular, I picked a random message of the trillions that went by and happened to get you. I apologize if you felt dumped on. No offense intended to you in particular but this thread has gone on and on and on and on and on and on... S From debatem1 at gmail.com Wed Oct 28 04:34:34 2009 From: debatem1 at gmail.com (geremy condra) Date: Tue, 27 Oct 2009 23:34:34 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it In-Reply-To: <9D7624CE452343719A7ED512B2FD17F5@RaymondLaptop1> References: <200910231132.45504.w.richert@gmx.net> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> <9D7624CE452343719A7ED512B2FD17F5@RaymondLaptop1> Message-ID: On Tue, Oct 27, 2009 at 3:49 PM, Raymond Hettinger wrote: > > [geremy condra] >> >> Was it ever decided whether this would fall under the moratorium? > > Decided isn't the right word: > http://mail.python.org/pipermail/python-dev/2009-October/093373.html > I'm unclear- does that imply that this is this going to be decided on a case-by-case basis in the future or have the details for the big M just not been hammered out yet? Geremy Condra From david.lyon at preisshare.net Wed Oct 28 06:13:21 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 28 Oct 2009 01:13:21 -0400 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> <4ADFEF35.6000409@g.nevcal.com> Message-ID: <5e4f7ac667b4bd5ffdd40f2b45d969df@preisshare.net> On Thu, 22 Oct 2009 10:20:03 +0000 (UTC), Antoine Pitrou wrote: > (*) Remember, however, that Tarek and work on Distribute, and also on > bringing pieces of setuptools/Distribute functionality into distutils. But if that's the case then why not work on any third party tool..? like pip or setuptools? It seems are very longwinded process if the only way to work on python is to work on distutils but before doing that you have to first work on distribute and then wait for all the changes to work their way back up the chain.. Actually, I have finally worked out what I want. That is shell support in the python windows distribution so that you can right click an .egg and install it. I don't see how that can be achieved by following the workprocess that you describe above. David From regebro at gmail.com Wed Oct 28 07:30:44 2009 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 28 Oct 2009 07:30:44 +0100 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <4AE0C73C.9070805@v.loewis.de> References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> Message-ID: <319e029f0910272330me6847d4i5f9a4edb8f9359fc@mail.gmail.com> 2009/10/22 "Martin v. L?wis" : > What use has such a stepping stone? Why, and (more importantly) when > would anybody currently supporting 2.x give up 2.6 and earlier, and > only support 2.7? And, if they chose to do so, why would they not move > the code base to 3.x right away? Python 2 support is not only about supporting old versions of Python, but also supporting users of Python2-only modules. If you have a module that runs only on Python 2.7, the people who for some reason can not move to Python 3 yet, can still use it, by running Python 2.7. For that your module doesn't have to run on 2.6 or 2.5. So 2.7 support will for the most part be a case not of supporting Python versions, but Python *users*. Which is a good thing. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Wed Oct 28 07:40:08 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2009 16:40:08 +1000 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <4AE7E748.9060400@gmail.com> Guido van Rossum wrote: > My gut tells me it is bad API design to collapse these two use cases. > Probably because the implementations won't have much in common: (1) > should just pick the first valid element, while (2) should use the > normal hash lookup algorithm (shared with 'in', .add() etc.). As a side note, a fairly obvious method name for "add if missing and then return canonical representation" did occur to me: s.intern(x) I'd be -0 on expanding the set API for this though (since the cookbook recipe overloading __eq__ already provides an efficient solution). As far as the moratorium in general goes, perhaps we should draw a line between API changes that affect the ABCs in collections and numbers and new convenience methods on the builtin types themselves? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Oct 28 07:42:40 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2009 16:42:40 +1000 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <7b0045ed0910261154t558dfe97h80b0ccd513890d77@mail.gmail.com> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <4AE7E7E0.50602@gmail.com> Alexander Belopolsky wrote: > Odd indeed. My first reaction was: it is not needed because lists > support slicing, but when I tried to construct a list.get() using > slicing the best I could come up with was the following hack > >>>> def lget(l, i, v): return (l[i:] or [v])[0] > ... >>>> lget(range(10), 20, 200) > 200 >>>> lget(range(10), 5, 50) > 5 > > Yet for some reason I never missed this functionality ... People tend to do length checks on lists to decide which items are and are not present. You can't do that with a dict. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Oct 28 08:00:44 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2009 17:00:44 +1000 Subject: [Python-Dev] Set methods for mapping views In-Reply-To: <200910271029.58591.steve@pearwood.info> References: <617D5DBD3D6549958A4AF1EBA25C163C@RaymondLaptop1> <200910271029.58591.steve@pearwood.info> Message-ID: <4AE7EC1C.809@gmail.com> Steven D'Aprano wrote: > On Tue, 27 Oct 2009 10:09:14 am Guido van Rossum wrote: >> On Mon, Oct 26, 2009 at 4:06 PM, Antoine Pitrou > wrote: >>> Steven D'Aprano pearwood.info> writes: >>>> I don't understand that rationale. Let's take a concrete example. >>>> The new `yield from` syntax was accepted >>> Was it? >> No. > > I thought it had been. My mistake. Serves me right for not checking the > PEP first. But my point still stands. Rather than going through and saying "oh, change X touches on area Y, where we're waiting to see how change Z plays out, so we shouldn't do X yet" it is simpler to declare the core of the language off limits for a release or two. We made a lot of big changes to fairly core parts of the language in a relatively short period. Giving those a chance to settle down before we start fiddling further is unlikely to be detrimental in the long run. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Wed Oct 28 10:12:14 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Oct 2009 09:12:14 +0000 (UTC) Subject: [Python-Dev] nonlocal keyword in 2.x? References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <319e029f0910272330me6847d4i5f9a4edb8f9359fc@mail.gmail.com> Message-ID: Lennart Regebro gmail.com> writes: > > So 2.7 support will for the most part be a case not of supporting > Python versions, but Python *users*. That's still not a good reason to backport nonlocal. The same reasoning could be used to backport new features to the 2.6 branch after all. Regards Antoine. From mal at egenix.com Wed Oct 28 11:43:54 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 28 Oct 2009 11:43:54 +0100 Subject: [Python-Dev] Refactoring installation schemes In-Reply-To: <94bdd2610910271455s7854e6a9u94e3aeeeacb69c02@mail.gmail.com> References: <94bdd2610910271455s7854e6a9u94e3aeeeacb69c02@mail.gmail.com> Message-ID: <4AE8206A.9020108@egenix.com> Tarek Ziad? wrote: > Hello, > > Since the addition of PEP 370, (per-user site packages), site.py and > distutils/command/install.py are *both* providing the various > installation directories for Python, > depending on the system and the Python version. > > We have also started to discuss lately in various Mailing Lists the > addition of new schemes for IronPython and Jython, meaning that we > might add some more in both places. > > I would like to suggest a simplification by adding a dedicated module > to manage these installation schemes in one single place in the > stdlib. > > This new independant module would be used by site.py and distutils and > would also make it easier for third party code to work with these > schemes. > Of course this new module would be rather simple and not add any new > import statement to avoid any overhead when Python starts and loads > site.py +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 28 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fetchinson at googlemail.com Wed Oct 28 12:01:36 2009 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Wed, 28 Oct 2009 12:01:36 +0100 Subject: [Python-Dev] Python Package Management Roadmap in Python Releases In-Reply-To: <5e4f7ac667b4bd5ffdd40f2b45d969df@preisshare.net> References: <42E841ED-E8C3-4F97-9E56-8DFDCB687386@python.org> <1998C055-4E0F-46F0-A0EE-142866C1B114@python.org> <13a7dd22a702e20a2526b48bd932e5a8@preisshare.net> <0841bc282e204bc4dadfbdb0e87feb51@preisshare.net> <4ADFEF35.6000409@g.nevcal.com> <5e4f7ac667b4bd5ffdd40f2b45d969df@preisshare.net> Message-ID: >> (*) Remember, however, that Tarek and work on Distribute, and also on >> bringing pieces of setuptools/Distribute functionality into distutils. > > But if that's the case then why not work on any third party tool..? like > pip or setuptools? > > It seems are very longwinded process if the only way to work on > python is to work on distutils but before doing that you have to > first work on distribute and then wait for all the changes to work > their way back up the chain.. > > Actually, I have finally worked out what I want. That is shell support > in the python windows distribution so that you can right click an > .egg and install it. > > I don't see how that can be achieved by following the workprocess > that you describe above. As has been said by many, you are entirely welcome to work on whatever tool you think is useful. Once you are done you are again welcome to distribute your tool or application to users and see how many users are happy with it. Once you are done with this step as well, you are again encouraged to come back to python-dev and say: "In the last X months my app/tool became very popular in the python community. There are Y developers working on the app/tool and there are Z happy users. I'd suggest including it in the python stdlib or I'd suggest coordinating the releases of my app/tool with that of python." At this point a useful conversation can start. Please note that a similarly useful conversation is impossible to take place before all the above steps have been completed. HTH, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From fuzzyman at voidspace.org.uk Wed Oct 28 12:02:50 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 28 Oct 2009 11:02:50 +0000 Subject: [Python-Dev] Refactoring installation schemes In-Reply-To: <4AE8206A.9020108@egenix.com> References: <94bdd2610910271455s7854e6a9u94e3aeeeacb69c02@mail.gmail.com> <4AE8206A.9020108@egenix.com> Message-ID: <4AE824DA.4090701@voidspace.org.uk> M.-A. Lemburg wrote: > Tarek Ziad? wrote: > >> Hello, >> >> Since the addition of PEP 370, (per-user site packages), site.py and >> distutils/command/install.py are *both* providing the various >> installation directories for Python, >> depending on the system and the Python version. >> >> We have also started to discuss lately in various Mailing Lists the >> addition of new schemes for IronPython and Jython, meaning that we >> might add some more in both places. >> >> I would like to suggest a simplification by adding a dedicated module >> to manage these installation schemes in one single place in the >> stdlib. >> >> This new independant module would be used by site.py and distutils and >> would also make it easier for third party code to work with these >> schemes. >> Of course this new module would be rather simple and not add any new >> import statement to avoid any overhead when Python starts and loads >> site.py >> > > +1 > > Also +1. It seems like this would make things easier for the alternative implementations. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From solipsis at pitrou.net Wed Oct 28 12:36:27 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Oct 2009 11:36:27 +0000 (UTC) Subject: [Python-Dev] Reworking the GIL References: <1256502140.5621.163.camel@localhost> <4AE5A768.6080900@molden.no> <930F189C8A437347B80DF2C156F7EC7F098FF42768@exchis.ccp.ad.local> Message-ID: Kristj?n Valur J?nsson ccpgames.com> writes: > > In my experience (from stackless python) using priority wakeup for IO can result in very erratic > scheduling when there is much IO going on, every IO trumping another. I whipped up a trivial multithreaded HTTP server using socketserver.ThreadingMixin and wsgiref, and used apachebench against it with a reasonable concurrency level (10 requests at once). Enabling/disabling priority requests doesn't seem to make a difference. Regards Antoine. From skip at pobox.com Wed Oct 28 14:35:54 2009 From: skip at pobox.com (skip at pobox.com) Date: Wed, 28 Oct 2009 08:35:54 -0500 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <319e029f0910272330me6847d4i5f9a4edb8f9359fc@mail.gmail.com> Message-ID: <19176.18618.662587.613771@montanaro.dyndns.org> >> So 2.7 support will for the most part be a case not of supporting >> Python versions, but Python *users*. Antoine> That's still not a good reason to backport nonlocal. The same Antoine> reasoning could be used to backport new features to the 2.6 Antoine> branch after all. No, because 2.6 is in feature freeze (bug fixes only). 2.7 is the current version of 2.x where new features are allowed to be added. Skip From ssteinerx at gmail.com Wed Oct 28 15:30:29 2009 From: ssteinerx at gmail.com (ssteinerX@gmail.com) Date: Wed, 28 Oct 2009 10:30:29 -0400 Subject: [Python-Dev] Refactoring installation schemes In-Reply-To: <4AE824DA.4090701@voidspace.org.uk> References: <94bdd2610910271455s7854e6a9u94e3aeeeacb69c02@mail.gmail.com> <4AE8206A.9020108@egenix.com> <4AE824DA.4090701@voidspace.org.uk> Message-ID: On Oct 28, 2009, at 7:02 AM, Michael Foord wrote: > M.-A. Lemburg wrote: >> Tarek Ziad? wrote: >> >>> Hello, >>> >>> Since the addition of PEP 370, (per-user site packages), site.py and >>> distutils/command/install.py are *both* providing the various >>> installation directories for Python, >>> depending on the system and the Python version. >>> >>> We have also started to discuss lately in various Mailing Lists the >>> addition of new schemes for IronPython and Jython, meaning that we >>> might add some more in both places. >>> >>> I would like to suggest a simplification by adding a dedicated >>> module >>> to manage these installation schemes in one single place in the >>> stdlib. >>> >>> This new independant module would be used by site.py and distutils >>> and >>> would also make it easier for third party code to work with these >>> schemes. >>> Of course this new module would be rather simple and not add any new >>> import statement to avoid any overhead when Python starts and loads >>> site.py +1 This would help unpollute ~/ which is getting quite full of 'stuff.' S From solipsis at pitrou.net Wed Oct 28 15:49:47 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Oct 2009 14:49:47 +0000 (UTC) Subject: [Python-Dev] nonlocal keyword in 2.x? References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <319e029f0910272330me6847d4i5f9a4edb8f9359fc@mail.gmail.com> <19176.18618.662587.613771@montanaro.dyndns.org> Message-ID: pobox.com> writes: > > >> So 2.7 support will for the most part be a case not of supporting > >> Python versions, but Python *users*. > > Antoine> That's still not a good reason to backport nonlocal. The same > Antoine> reasoning could be used to backport new features to the 2.6 > Antoine> branch after all. > > No, because 2.6 is in feature freeze (bug fixes only). 2.7 is the current > version of 2.x where new features are allowed to be added. That was precisely my point. There are development practices which mitigate the idea that backporting is always helpful to the user. Regards Antoine. From dreamingforward at gmail.com Wed Oct 28 17:22:46 2009 From: dreamingforward at gmail.com (average) Date: Wed, 28 Oct 2009 09:22:46 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE1E71C.7020806@gmail.com> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> Message-ID: <913f9f570910280922v5687db91m8bb0557ef2480c56@mail.gmail.com> [Guido wrote:] > - If sets were to grow an API to non-destructively access the object > stored in the set for a particular key, then dicts should have such a > method too. > - I still wish we could go back in time and unify sets and dicts, if > only to find out how that experiment would turn out. +5. If Python3 were to have this feature it would make it worth migrating to (and nearly worthy of the intent that was behind the idea of python3k). FWIW ... :) marcos From dreamingforward at gmail.com Wed Oct 28 17:34:45 2009 From: dreamingforward at gmail.com (average) Date: Wed, 28 Oct 2009 09:34:45 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <913f9f570910280931p3da32519kd56f251264ff867@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <200910240946.48542.steve@pearwood.info> <4AE3E6C6.4010008@gmail.com> <913f9f570910280922v5687db91m8bb0557ef2480c56@mail.gmail.com> <913f9f570910280931p3da32519kd56f251264ff867@mail.gmail.com> Message-ID: <913f9f570910280934n4030f54u19e865c9fbed0c88@mail.gmail.com> [I wrote:] >?If Python3 were to have this feature it would make it worth > migrating to Sorry that may have sounded more harsh than I expected. ?If I had more resources, I'd propose (and volunteer) a python3000 branch where any and all who were disappointed at the *lack* of compatability changes could continue working on the core language. ?(Moratorium controversy solved and quaranteened--any desireable features for the working branch would be back-propagated via the time machine as BDFL permits.) marcos From ziade.tarek at gmail.com Wed Oct 28 19:05:06 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 28 Oct 2009 19:05:06 +0100 Subject: [Python-Dev] Refactoring installation schemes In-Reply-To: References: <94bdd2610910271455s7854e6a9u94e3aeeeacb69c02@mail.gmail.com> <4AE8206A.9020108@egenix.com> <4AE824DA.4090701@voidspace.org.uk> Message-ID: <94bdd2610910281105q328bc3a5qaed020ef3afe28df@mail.gmail.com> M.-A. Lemburg wrote: > +1 On Oct 28, 2009, at 7:02 AM, Michael Foord wrote: > Also +1. It seems like this would make things easier for the alternative implementations. On Wed, Oct 28, 2009 at 3:30 PM, ssteinerX at gmail.com wrote: > +1 Ok then I'll work on a patch for that change and start an issue about it soon. > This would help unpollute ~/ which is getting quite full of 'stuff.' Notice that there's a discussion going on about that at http://bugs.python.org/issue7175 Regards Tarek From debatem1 at gmail.com Wed Oct 28 19:59:52 2009 From: debatem1 at gmail.com (geremy condra) Date: Wed, 28 Oct 2009 14:59:52 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <913f9f570910280934n4030f54u19e865c9fbed0c88@mail.gmail.com> References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <913f9f570910280922v5687db91m8bb0557ef2480c56@mail.gmail.com> <913f9f570910280931p3da32519kd56f251264ff867@mail.gmail.com> <913f9f570910280934n4030f54u19e865c9fbed0c88@mail.gmail.com> Message-ID: On Wed, Oct 28, 2009 at 12:34 PM, average wrote: > [I wrote:] >>?If Python3 were to have this feature it would make it worth >> migrating to > > Sorry that may have sounded more harsh than I expected. ?If I had more > resources, I'd propose (and volunteer) a python3000 branch where any > and all who were disappointed at the *lack* of compatability changes > could continue working on the core language. ?(Moratorium controversy > solved and quaranteened--any desireable features for the working > branch would be back-propagated via the time machine as BDFL permits.) > > marcos This is effectively the sandbox idea I proposed, except with backporting. Geremy Condra From olemis at gmail.com Wed Oct 28 22:28:30 2009 From: olemis at gmail.com (Olemis Lang) Date: Wed, 28 Oct 2009 16:28:30 -0500 Subject: [Python-Dev] Where is `ctypes.com` ? Message-ID: <24ea26600910281428n26497103v48e96a45b464d804@mail.gmail.com> Hello ! Recently I found a code snippet [1]_ illustrating integration between Python and COM technology in Win32 systems. I tried to reproduce it and I can't import module `ctypes.com`. Q: - Is it (`ctypes.com`) distributed with stdlib ? If that's true then I'm affraid that those py files were removed somehow (accidentally ? who knows ? ) ... Thnx in advance ! .. [1] ctypes.win32.com.server @ Py (http://svn.python.org/projects/ctypes/tags/release_0_6_2/ctypes/win32/com/server.py) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: [visualization-api] ANN: TracGViz 1.3.4 released: msg#00244 ... - http://feedproxy.google.com/~r/TracGViz-full/~3/OBfKGS_sy2E/msg00244.html From theller at ctypes.org Wed Oct 28 22:39:32 2009 From: theller at ctypes.org (Thomas Heller) Date: Wed, 28 Oct 2009 22:39:32 +0100 Subject: [Python-Dev] Where is `ctypes.com` ? In-Reply-To: <24ea26600910281428n26497103v48e96a45b464d804@mail.gmail.com> References: <24ea26600910281428n26497103v48e96a45b464d804@mail.gmail.com> Message-ID: Olemis Lang schrieb: > Hello ! > > Recently I found a code snippet [1]_ illustrating integration between > Python and COM technology in Win32 systems. I tried to reproduce it > and I can't import module `ctypes.com`. First, the python-dev mailing list is used for developing the Python language itself, not developing WITH Python. For questions about developing with Python, please use the newsgroup comp.lang.python. > Q: > > - Is it (`ctypes.com`) distributed with stdlib ? > > If that's true then I'm affraid that those py files were removed > somehow (accidentally ? who knows ? ) ... ctypes.com is very old and has been superseeded by the comtypes package: http://starship.python.net/crew/theller/comtypes/ An even more popular package for using COM with python is pywin32: http://sourceforge.net/projects/pywin32/ -- Thanks, Thomas From olemis at gmail.com Wed Oct 28 22:48:56 2009 From: olemis at gmail.com (Olemis Lang) Date: Wed, 28 Oct 2009 16:48:56 -0500 Subject: [Python-Dev] Where is `ctypes.com` ? In-Reply-To: References: <24ea26600910281428n26497103v48e96a45b464d804@mail.gmail.com> Message-ID: <24ea26600910281448x7f586efdpf72171eb4972cbe9@mail.gmail.com> On Wed, Oct 28, 2009 at 4:39 PM, Thomas Heller wrote: > Olemis Lang schrieb: >> Hello ! >> >> Recently I found a code snippet [1]_ illustrating integration between >> Python and COM technology in Win32 systems. I tried to reproduce it >> and I can't import module `ctypes.com`. > > First, the python-dev mailing list is used for developing the Python language > itself, not developing WITH Python. ?For questions about developing with Python, > please use the newsgroup comp.lang.python. > I apologize if needed ... but I didn't ask about ?how to program things using `ctypes.com` ?. I asked here because you are the ones who should know about modules in stdlib . That's why I thought that it was not ?very? OT >> Q: >> >> ? - Is it (`ctypes.com`) distributed with stdlib ? >> >> If that's true then I'm affraid that those py files were removed >> somehow (accidentally ? who knows ? ) ... > > ctypes.com is very old and has been superseeded by the comtypes > package: http://starship.python.net/crew/theller/comtypes/ > Thnx ! > An even more popular package for using COM with python is > pywin32: http://sourceforge.net/projects/pywin32/ > Yes, I've just tried that one, but the fact is that I'm experiencing a problem with it (that I won't detail in here because this is not the list to talk about such subjects , isn't it ? ... just learning fast ... :P ) and I thought I could switch to that one ... Thnx very much for your reply ! -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Suggestions: Wave (private) Groups, integration - Google Wave API ... - http://feedproxy.google.com/~r/TracGViz-full/~3/cuwdwGkX1WA/90bf35ca0c38caf0 From stephen at xemacs.org Thu Oct 29 03:31:54 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 29 Oct 2009 11:31:54 +0900 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <913f9f570910280922v5687db91m8bb0557ef2480c56@mail.gmail.com> <913f9f570910280931p3da32519kd56f251264ff867@mail.gmail.com> <913f9f570910280934n4030f54u19e865c9fbed0c88@mail.gmail.com> Message-ID: <87d4476icl.fsf@uwakimon.sk.tsukuba.ac.jp> geremy condra writes: > On Wed, Oct 28, 2009 at 12:34 PM, average wrote: > > [I wrote:] > >>?If Python3 were to have this feature it would make it worth > >> migrating to > > > > Sorry that may have sounded more harsh than I expected. ?If I had more > > resources, > This is effectively the sandbox idea I proposed, Um, guys? Python 3000 was a project involving fairly concentrated effort by about a dozen senior, very skilled, developers plus many others over a period of several (3 or 4?) years. The "sandbox" is going to be played in by people without the resources to create such an industrial-strength "code workshop," and as such is not going to be given anywhere near the credibility that the py3k effort got. As evidence of how difficult an effort like "Python 3000" is, consider: where is Perl 6? There are hundreds, perhaps thousands, of developers saying "we want Perl 6" but the resources aren't there to *finish* it. On the other hand, "this feature" is already present in Python in great generality. You just have to spell it for x in container: break (or any of several other ways). The multiple spellings (of different degrees of efficiency) and the lack of an obvious spelling are indeed warts IMO, but "this feature would make Python 3 worth migrating to" and "+5" are hard for me to understand except as extreme exaggeration. The Python 3000 effort succeeded because the resources were sufficient to the goal. Part of that is because some very powerful developers found some itches they needed to scratch: Guido is clearly happy with P3tL (Python 3 the Language), both its current state and the fact that all that effort was devoted to it. But now he (inter alia) wants to focus effort (mostly not his own ;-) on P3tSL (Python 3 the Standard Library) which still needs substantial cleanup though it's already very usable for many applications, and on P3S3PA (Python 3 Supported 3rd Party Applications). The resources available for language evolution have dried up; *you* who want more language evolution are going to have to supply them. OTOH, there are at present few attractive improvements. set.getany() is an example of the kind of decidedly unattractive improvement currently available. It has an efficient if not obvious spelling as above; it has several less efficient but more transparent spellings as have been posted several times. The *need* is not infinitesimal; some good programmers have testified to being stumped by this one. But it's not big, either. The *resources* required are surprisingly large: although we already have a satisfactory implementation, about half of this thread has been devoted to discussing the various names proposed, and no conclusion reached in this thread or the previous one (of similar length). From debatem1 at gmail.com Thu Oct 29 04:23:06 2009 From: debatem1 at gmail.com (geremy condra) Date: Wed, 28 Oct 2009 23:23:06 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it In-Reply-To: <87d4476icl.fsf@uwakimon.sk.tsukuba.ac.jp> References: <200910231132.45504.w.richert@gmx.net> <4AE3E6C6.4010008@gmail.com> <913f9f570910280922v5687db91m8bb0557ef2480c56@mail.gmail.com> <913f9f570910280931p3da32519kd56f251264ff867@mail.gmail.com> <913f9f570910280934n4030f54u19e865c9fbed0c88@mail.gmail.com> <87d4476icl.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: [Stephen Turbull] >?The resources available for language evolution have dried up; > *you* who want more language evolution are going to have to > supply them. That's the idea I've been advocating- a code-first-jaw-later approach and the sandbox to support it. Its not even that hard to find where I've said exactly that in the moratorium discussion. Geremy Condra From holdenweb at gmail.com Thu Oct 29 04:34:25 2009 From: holdenweb at gmail.com (Steve Holden) Date: Wed, 28 Oct 2009 23:34:25 -0400 Subject: [Python-Dev] MSDN subscribers: Using Visual Studio? Message-ID: <4AE90D41.8080107@holdenweb.com> I just wondered, with the recent flood of new MSDN subscriptions loosed on the developer community, how many people have installed the required version of Visual Studio and built Python for Windows from source? Not being that familiar with the process myself I was hoping for some advice from the inexperienced who have overcome any hurdles there might be. I have a pristine virtual machine just waiting for the right pieces ... Also what other uses have you found for the licenses? It would be good to get some information about how useful the licenses have been, and how they have helped people to improve Python's quality or ease of distribution (if they have). I'm sure Microsoft would also appreciate some positive feedback to their generosity, and I'll undertake to provide that if this message elicits much by way of reply. Since I'm pretty much too busy to follow the dev list right now I'd appreciate direct Cc's. regards Steve PS: If any further core developers need licenses, I plan to apply to Microsoft again in the new year. I'll be sending out a message then, I don't intend to keep a waiting list. -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Watch PyCon on video now! http://pycon.blip.tv/ From regebro at gmail.com Thu Oct 29 09:15:47 2009 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 29 Oct 2009 09:15:47 +0100 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <319e029f0910272330me6847d4i5f9a4edb8f9359fc@mail.gmail.com> <19176.18618.662587.613771@montanaro.dyndns.org> Message-ID: <319e029f0910290115y49be844co78da6636452445f@mail.gmail.com> 2009/10/28 Antoine Pitrou : > pobox.com> writes: >> >> ? ? >> So 2.7 support will for the most part be a case not of supporting >> ? ? >> Python versions, but Python *users*. >> >> ? ? Antoine> That's still not a good reason to backport nonlocal. The same >> ? ? Antoine> reasoning could be used to backport new features to the 2.6 >> ? ? Antoine> branch after all. >> >> No, because 2.6 is in feature freeze (bug fixes only). ?2.7 is the current >> version of 2.x where new features are allowed to be added. > > That was precisely my point. Then I don't understand what you are saying. Obviously we shouldn't backport to the 2.6 branch, it's in bugfix mode. This is about 2.7. I don't see what 2.6 has to do with it. > There are development practices which mitigate the > idea that backporting is always helpful to the user. And those are? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Thu Oct 29 10:23:36 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Oct 2009 19:23:36 +1000 Subject: [Python-Dev] nonlocal keyword in 2.x? In-Reply-To: <319e029f0910290115y49be844co78da6636452445f@mail.gmail.com> References: <4AE0BF83.4080903@v.loewis.de> <20091022204621.11571.1616253771.divmod.xquotient.1454@localhost.localdomain> <4AE0C73C.9070805@v.loewis.de> <319e029f0910272330me6847d4i5f9a4edb8f9359fc@mail.gmail.com> <19176.18618.662587.613771@montanaro.dyndns.org> <319e029f0910290115y49be844co78da6636452445f@mail.gmail.com> Message-ID: <4AE95F18.4040807@gmail.com> Lennart Regebro wrote: > 2009/10/28 Antoine Pitrou : >> pobox.com> writes: >>> >> So 2.7 support will for the most part be a case not of supporting >>> >> Python versions, but Python *users*. >>> >>> Antoine> That's still not a good reason to backport nonlocal. The same >>> Antoine> reasoning could be used to backport new features to the 2.6 >>> Antoine> branch after all. >>> >>> No, because 2.6 is in feature freeze (bug fixes only). 2.7 is the current >>> version of 2.x where new features are allowed to be added. >> That was precisely my point. > > Then I don't understand what you are saying. Obviously we shouldn't > backport to the 2.6 branch, it's in bugfix mode. This is about 2.7. I > don't see what 2.6 has to do with it. > >> There are development practices which mitigate the >> idea that backporting is always helpful to the user. > > And those are? You said it above yourself: "bugfix mode" That's all Antoine's point was - backporting of new features to previous branches is not automatically a good idea. In the case of 3.2 -> 2.7 backports, there are issues with the initial development time investment to do the backport, future double-keying of additional maintenance issues, consideration of possible poor interaction with legacy features in the 2.x series. It's a bunch of additional work that isn't going to happen without someone volunteering to do it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Thu Oct 29 12:57:49 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Oct 2009 12:57:49 +0100 Subject: [Python-Dev] MSDN subscribers: Using Visual Studio? In-Reply-To: <4AE90D41.8080107@holdenweb.com> Message-ID: <1256817469.5534.4.camel@localhost> Steve Holden gmail.com> writes: > > I just wondered, with the recent flood of new MSDN subscriptions loosed > on the developer community, how many people have installed the required > version of Visual Studio and built Python for Windows from source? Not > being that familiar with the process myself I was hoping for some advice > from the inexperienced who have overcome any hurdles there might be. I > have a pristine virtual machine just waiting for the right pieces ... If it's just a matter of building and testing it you don't need any MSDN subscription, you just need Visual Studio Express which is free (as in free beer...). I don't know if it allows you to build an installer though. I won't reply to the other part of the message since I haven't asked for (nor received) any MSDN subscription :) Regards Antoine. From solipsis at pitrou.net Thu Oct 29 15:30:05 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Oct 2009 14:30:05 +0000 (UTC) Subject: [Python-Dev] "Buildbot" category on the tracker Message-ID: Hello, What do you think of creating a "buildbot" category in the tracker? There are often problems on specific buildbots which would be nice to track, but there's nowhere to do so. Regards Antoine. From daniel at stutzbachenterprises.com Thu Oct 29 18:15:36 2009 From: daniel at stutzbachenterprises.com (Daniel Stutzbach) Date: Thu, 29 Oct 2009 12:15:36 -0500 Subject: [Python-Dev] MSDN subscribers: Using Visual Studio? In-Reply-To: <1256817469.5534.4.camel@localhost> References: <4AE90D41.8080107@holdenweb.com> <1256817469.5534.4.camel@localhost> Message-ID: On Thu, Oct 29, 2009 at 6:57 AM, Antoine Pitrou wrote: > If it's just a matter of building and testing it you don't need any MSDN > subscription, you just need Visual Studio Express which is free (as in > free beer...). I don't know if it allows you to build an installer > though. > It does. If I recall correctly, in addition to Visual Studio Express, I also needed the Windows SDK (which is also free as in beer). -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at cheimes.de Thu Oct 29 19:21:50 2009 From: lists at cheimes.de (Christian Heimes) Date: Thu, 29 Oct 2009 19:21:50 +0100 Subject: [Python-Dev] MSDN subscribers: Using Visual Studio? In-Reply-To: References: <4AE90D41.8080107@holdenweb.com> <1256817469.5534.4.camel@localhost> Message-ID: Daniel Stutzbach wrote: > It does. If I recall correctly, in addition to Visual Studio Express, I > also needed the Windows SDK (which is also free as in beer). The VS 2008 Express Edition is sufficient to build X86 binaries on Windows. The express edition doesn't support X64_86. though. Christian From martin at v.loewis.de Thu Oct 29 23:05:16 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 29 Oct 2009 23:05:16 +0100 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: References: Message-ID: <4AEA119C.5070707@v.loewis.de> > What do you think of creating a "buildbot" category in the tracker? There are > often problems on specific buildbots which would be nice to track, but there's > nowhere to do so. Do you have any specific reports that you would want to classify with this category? Regards, Martin From Bruno.Harbulot at manchester.ac.uk Thu Oct 29 19:32:29 2009 From: Bruno.Harbulot at manchester.ac.uk (Bruno Harbulot) Date: Thu, 29 Oct 2009 18:32:29 +0000 Subject: [Python-Dev] ssl module Message-ID: Hello, I would like to ask a few questions and suggestions regarding the ssl module (in Python 2.6). (I gather from [1] that there is some effort going on to enhance the ssl API, but I'm not sure if this is the right place to discuss it.) Like other Python users, I was a bit surprised by the lack of verification of httplib/urllib2 (hence I started to write a small library a while back, only published today [2]), but the following points are not HTTP-specific. 1. Hostname checking. From what I gather by reading the archives on this list, the issue of hostname checking seems controversial [3]. It seems widely admitted by browser communities nowadays to check that the hostname the CN field of the subject DN or the DNS entries of subjectAltName. I'd feel more comfortable if this was the default behaviour of the client in Python's SSL module, although having a mechanism to override this would be useful indeed. It's more or less a basic security requirement to check the identity of the server before doing anything else. 2. Cipher suite selection. It's useful to restrict the list of cipher suites that can be used, not just for speed (as mentioned in [1]), but also because some cipher suites may be considered insecure by some institutions. This would be a good feature to have indeed. 3. Full chain of certificates. The PyOpenSSL module is able to take a callback function that verifies each certificate in the chain (using depth). According to the documentation, the ssl module only exposes the first certificate in the chain (no CA). In some applications, it is useful to verify certain policies according to attributes further up in the chain. I'd like to suggest having an "SSLSocket.getpeercerts(binary_form=False)" (plural) that returns a list of certificates in the verification chain. Is there a place where the status of the ssl module is summarized, or a better place to discuss this? I could try to provide contributions or further details if appropriate. Best wishes, Bruno. [1] http://mail.python.org/pipermail/python-dev/2009-September/091636.html [2] http://code.google.com/p/python-httpclient [2] http://bugs.python.org/issue1589 From solipsis at pitrou.net Thu Oct 29 23:21:31 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Oct 2009 22:21:31 +0000 (UTC) Subject: [Python-Dev] (no subject) References: <4AEA119C.5070707@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > What do you think of creating a "buildbot" category in the tracker? There are > > often problems on specific buildbots which would be nice to track, but there's > > nowhere to do so. > > Do you have any specific reports that you would want to classify with > this category? I was thinking of http://bugs.python.org/issue4970 Regards Antoine. From exarkun at twistedmatrix.com Fri Oct 30 00:04:43 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 29 Oct 2009 23:04:43 -0000 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: References: Message-ID: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> On 02:30 pm, solipsis at pitrou.net wrote: > >Hello, > >What do you think of creating a "buildbot" category in the tracker? >There are >often problems on specific buildbots which would be nice to track, but >there's >nowhere to do so. Is your idea that this would be for tracking issues with the *bots* themselves? That is, not just for tracking cases where some test method fails on a particular bot, but for tracking cases where, say, a bot's host has run out of disk space and cannot run the tests at all? For the case where a test is failing because of some platform or environment issue, it seems more sensible to track the ticket as relating to that platform or environment, or track it in relation to the feature it affects. Of course, tickets could move between these classifications as investigation reveals new information about the problem. Jean-Paul From jnoller at gmail.com Fri Oct 30 00:41:59 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 29 Oct 2009 19:41:59 -0400 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> Message-ID: <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> On Thu, Oct 29, 2009 at 7:04 PM, wrote: > On 02:30 pm, solipsis at pitrou.net wrote: >> >> Hello, >> >> What do you think of creating a "buildbot" category in the tracker? There >> are >> often problems on specific buildbots which would be nice to track, but >> there's >> nowhere to do so. > > Is your idea that this would be for tracking issues with the *bots* > themselves? ?That is, not just for tracking cases where some test method > fails on a particular bot, but for tracking cases where, say, a bot's host > has run out of disk space and cannot run the tests at all? > > For the case where a test is failing because of some platform or environment > issue, it seems more sensible to track the ticket as relating to that > platform or environment, or track it in relation to the feature it affects. > > Of course, tickets could move between these classifications as investigation > reveals new information about the problem. Then again, I know for a fact certain tests fail ONLY on certain buildbots because of the way they're configured. For example, certain multiprocessing tests will fail if /dev/shm isn't accessible on Linux, and several of the buildbosts are in tight chroot jails and don't have that exposed. Is it a bug in that buildbot, a platform specific bug, etc? jesse From janssen at parc.com Fri Oct 30 00:55:03 2009 From: janssen at parc.com (Bill Janssen) Date: Thu, 29 Oct 2009 16:55:03 PDT Subject: [Python-Dev] ssl module In-Reply-To: References: Message-ID: <48451.1256860503@parc.com> Bruno Harbulot wrote: > Hello, > > I would like to ask a few questions and suggestions regarding the ssl > module (in Python 2.6). (I gather from [1] that there is some effort > going on to enhance the ssl API, but I'm not sure if this is the right > place to discuss it.) > > Is there a place where the status of the ssl module is summarized, or > a better place to discuss this? I could try to provide contributions > or further details if appropriate. Yes, please look at the issues in the issue tracker already, and contribute there. We could particularly use more test cases to support patches that people have already submitted. > Like other Python users, I was a bit surprised by the lack of > verification of httplib/urllib2 (hence I started to write a small > library a while back, only published today [2]), but the following > points are not HTTP-specific. > > 1. Hostname checking. > > From what I gather by reading the archives on this list, the issue > of hostname checking seems controversial [3]. It seems widely admitted > by browser communities nowadays to check that the hostname the CN > field of the subject DN or the DNS entries of subjectAltName. I'd feel > more comfortable if this was the default behaviour of the client in > Python's SSL module, although having a mechanism to override this > would be useful indeed. It's more or less a basic security requirement > to check the identity of the server before doing anything else. I don't think it should happen by default in the ssl module client code, but I agree it makes sense to do that in various application uses of SSL, such as the httplib support for https, since that behavior is (annoyingly) called for in the (experimental, not standard) RFC which defines HTTP over SSL, and, as you say, it's widely done in Web browsers when https is being used. If you check the issues, you'll see that I think there should be a helper function in the ssl module to do this checking. Would you like to contribute one? Please either attach it to an already open issue, or start a new feature request issue. > 2. Cipher suite selection. > > It's useful to restrict the list of cipher suites that can be used, > not just for speed (as mentioned in [1]), but also because some cipher > suites may be considered insecure by some institutions. This would be > a good feature to have indeed. Yes, I agree. > 3. Full chain of certificates. > > The PyOpenSSL module is able to take a callback function that > verifies each certificate in the chain (using depth). According to the > documentation, the ssl module only exposes the first certificate in > the chain (no CA). In some applications, it is useful to verify > certain policies according to attributes further up in the chain. > I'd like to suggest having an > "SSLSocket.getpeercerts(binary_form=False)" (plural) that returns a > list of certificates in the verification chain. Feel free to use PyOpenSSL for more complicated applications, like the one you mention, and to file more issues on the Python issue tracker about this. Though, we were striving to have something small and simple in the ssl module, not a feature-full binding to OpenSSL. Oh, and there's also a stdlib-sig, which (since the ssl module is part of the stdlib) might be an appropriate place to discuss it. Bill From rdmurray at bitdance.com Fri Oct 30 01:31:58 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 29 Oct 2009 20:31:58 -0400 (EDT) Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> Message-ID: On Thu, 29 Oct 2009 at 19:41, Jesse Noller wrote: > Then again, I know for a fact certain tests fail ONLY on certain > buildbots because of the way they're configured. For example, certain > multiprocessing tests will fail if /dev/shm isn't accessible on Linux, > and several of the buildbosts are in tight chroot jails and don't have > that exposed. > > Is it a bug in that buildbot, a platform specific bug, etc? I'd say that particular one is a bug in the tests. If /dev/shm is not available and is required, then the tests should be skipped with an appropriate message. It would also secondarily be an issue with the buildbot fleet, since multiprocessing would then not be getting thoroughly tested by those buildbots. IMO a buildbot category might be useful for bugs that show up in a buildbot but no one can (currently) reproduce, or problems with the buildbots themselves. I don't think we currently have any bugs filed that fall in the second category, but multiprocessing not getting completely tested because of lack of /dev/shm would fall into that category. Issue 4970 was in the first category until recently. But the real reason for having a buildbot category (or at least a keyword) would be to be able to tag all bugs that are currently making buildbots fail that are _not_ the result of a recent checkin. This would make the task of finding the bugs that need to be cleaned up to stabilize the buildbot fleet easier. I'm currently aware of issues 4970, 3892, and 6462 in this category, and there are a few more that we can/will file if we continue to pay attention to the failure reports now arriving on the irc channel. --David (RDM) From exarkun at twistedmatrix.com Fri Oct 30 01:58:05 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 30 Oct 2009 00:58:05 -0000 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> Message-ID: <20091030005805.12668.1011941081.divmod.xquotient.13@localhost.localdomain> On 29 Oct, 11:41 pm, jnoller at gmail.com wrote: >On Thu, Oct 29, 2009 at 7:04 PM, wrote: >>On 02:30 pm, solipsis at pitrou.net wrote: >>> >>>Hello, >>> >>>What do you think of creating a "buildbot" category in the tracker? >>>There >>>are >>>often problems on specific buildbots which would be nice to track, >>>but >>>there's >>>nowhere to do so. >> >>Is your idea that this would be for tracking issues with the *bots* >>themselves? ?That is, not just for tracking cases where some test >>method >>fails on a particular bot, but for tracking cases where, say, a bot's >>host >>has run out of disk space and cannot run the tests at all? >> >>For the case where a test is failing because of some platform or >>environment >>issue, it seems more sensible to track the ticket as relating to that >>platform or environment, or track it in relation to the feature it >>affects. >> >>Of course, tickets could move between these classifications as >>investigation >>reveals new information about the problem. > >Then again, I know for a fact certain tests fail ONLY on certain >buildbots because of the way they're configured. For example, certain >multiprocessing tests will fail if /dev/shm isn't accessible on Linux, >and several of the buildbosts are in tight chroot jails and don't have >that exposed. > >Is it a bug in that buildbot, a platform specific bug, etc? It's a platform configuration that can exist. If you're rejecting that configuration and saying that CPython will not support it, then it's silly to have a buildbot set up that way, and presumably that should be changed. The point is that this isn't about buildbot. It's about CPython and what platforms it supports. Categorizing it by "buildbot" is not useful, because no one is going to be cruising along looking for multiprocessing issues to fix by browsing tickets by the "buildbot" category. If, on the other hand, (sticking with this example) /dev/shm-less systems are not a platform that CPython wants to support, then having a buildbot running on one is a bit silly. It will probably always fail, and all it does is contribute another column of red. Who does that help? Jean-Paul From jnoller at gmail.com Fri Oct 30 02:58:48 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 29 Oct 2009 21:58:48 -0400 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> Message-ID: <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> On Thu, Oct 29, 2009 at 8:31 PM, R. David Murray wrote: > I'd say that particular one is a bug in the tests. ?If /dev/shm is > not available and is required, then the tests should be skipped with > an appropriate message. ?It would also secondarily be an issue with > the buildbot fleet, since multiprocessing would then not be getting > thoroughly tested by those buildbots. Fwiw: The tests skip on those platforms; but multiprocessing can't function properly on platforms like that. From steve at pearwood.info Fri Oct 30 03:58:16 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 30 Oct 2009 13:58:16 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> Message-ID: <200910301358.17752.steve@pearwood.info> On Wed, 28 Oct 2009 04:47:59 am Raymond Hettinger wrote: > A dict.get() can be meaningfully used in a loop (because the key can > vary). A set.get() returns the same value over and over again > (because there is no key). I don't believe anyone has requested those semantics. The suggested semantics for set.get() with no arguments, as I understand them, are: (1) it will only fail if the set is empty; (2) it should be efficient; (3) if you call it repeatedly on a set without modifying the set, you will cycle through each element in turn in some unspecified arbitrary order. To clarify point 3, given: x = set.get() y = set.get() then x and y will only be the same element if set has length one. However, given: x = set.get() set.add(el) set.remove(el) y = set.get() there are no guarantees about x and y being different. I believe that the patch supplied by Willi Richart implemented these behaviours. http://bugs.python.org/issue7212 -- Steven D'Aprano From python at rcn.com Fri Oct 30 05:00:27 2009 From: python at rcn.com (Raymond Hettinger) Date: Thu, 29 Oct 2009 21:00:27 -0700 Subject: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it References: <200910231132.45504.w.richert@gmx.net><7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> <200910301358.17752.steve@pearwood.info> Message-ID: <2585AF2DFDDE4C61B3596633F5001945@RaymondLaptop1> [Steven D'Aprano] > The suggested > semantics for set.get() with no arguments, as I understand them, are: > >(1) it will only fail if the set is empty; Just like next() except that next() gives you the option to supply a default and can be used on any iterator (perhaps iter(s) or itertools.cycle(s) etc). > (2) it should be efficient; Is this about optimization? I wouldn't expect "x=s.get()" to beat "for x in s: break". Attribute lookup and method calls usually are slower than equivalents using built-in syntax with specific opcodes. > (3) if you call it repeatedly on a set without modifying the set, you > will cycle through each element in turn in some unspecified arbitrary > order. What's wrong with using next()? That is what it's for. What about this proposal is specific to sets, i.e. why don't you want the same thing for lists. tuples, strings, file objects, or any other iterable? Does this proposal pass the test of being self-descriptive? Can you write a code fragment that exercises the cycling behavior, show it to another programmer, and have them correctly deduce what the code does (i.e. that different values are returned, that it fails when the set it empty, that it wraps around and never terminates)? Can they readily differentiate it for dict.get() which has decidedly different semantics? > To clarify point 3, given: > > x = set.get() > y = set.get() > > then x and y will only be the same element if set has length one. So, it can't even be used for looping through a set because there is no termination? > I believe that the patch supplied by Willi Richart implemented these > behaviours. > > http://bugs.python.org/issue7212 So you want to introduce additional, hidden state to sets? (to make sure that successive invocations return different values) Do you want a thread local version too? (so that two threads can call gets without stomping on each other's guarantees that successive calls will produce distinct elements) Do you have any real-world use-cases where next(), for-loops, or itertools wouldn't suffice? Is there a precedent in *any* other language you've ever seen? (setl has an "arb" function but it makes no promises about returning different values on consequetive calls; otherwise, I've never seen an equivalent in any other set implementation). Do you think the return-different-values-on-successive-calls semantics is self-evident and non-magical as compared to a straight for-loop or next(it)? ISTM, that when streams have non-destructive getters with self-advancing pointers, they also have a seek() function so that it can be controlled. Will this proposal need a seek() method too? Sorry for so many questions, but I honestly think there are too many unresolved design issues. We've seen no real-world source code that would be improved fwith the proposal. I think it sounds conceptually tempting and is fun to theorize about, but it actual implementation it will make sets more difficult to learn and it would quickly become a piece of rarely used, poorly understood cruft. Raymond From sumerc at gmail.com Fri Oct 30 08:07:57 2009 From: sumerc at gmail.com (=?ISO-8859-1?Q?S=FCmer_Cip?=) Date: Fri, 30 Oct 2009 09:07:57 +0200 Subject: [Python-Dev] yappi - any thoughts? Message-ID: <65b2175c0910300007i2cf614dq30893434fdd9721a@mail.gmail.com> Hi there, yappi(Yet Another Python Profiler) is a multithreaded profiler for 2.x series(do not know if it will work on 3k, not tested yet). I have done my best to make it efficient as possible.(details are under the technical bullet in the website). It is aimed for long-running programs, to attach/detach profiler and retrieve stats on the fly. The current cProfile module needs substantial amount of work to accomplish this task and also have some problems with recursive functions, basically that is why I have written a profiler from scratch. yappi is currently(v0.2) modifying the profilefunc variable fo the ThreadState object to profile an exsiting thread. The newly created threads are catched via threading.setprofile() call(from VM to our extension). And for the deleted threads, we rely on the recycling of the ThreadState objects.(Please see the website for details.) I have tested it under a game server (100k players per-day) for about a month for any issues and now releasing it. Please see: http://code.google.com/p/yappi/ Any thoughts/comments/ideas, anything?:) -- Sumer Cip -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.richert at gmx.net Fri Oct 30 09:16:02 2009 From: w.richert at gmx.net (Willi Richert) Date: Fri, 30 Oct 2009 09:16:02 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: <200910301358.17752.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <200910301358.17752.steve@pearwood.info> Message-ID: <200910300916.02924.w.richert@gmx.net> Am Freitag, 30. Oktober 2009 03:58:16 schrieb Steven D'Aprano: > To clarify point 3, given: > > x = set.get() > y = set.get() > > then x and y will only be the same element if set has length one. > However, given: > > x = set.get() > set.add(el) > set.remove(el) > y = set.get() > > there are no guarantees about x and y being different. > > I believe that the patch supplied by Willi Richart implemented these > behaviours. > > http://bugs.python.org/issue7212 > Actually, no. The patch makes no assumption about x and y being different. It does not try to extend the core functionality of set nor does it need to maintain any state that would be necessary for that. It is just a more obvious and cleaner way for saying "for x in some_set: break". So, as Raymond says, it is the Pythonic version of "arb" in setl. wr From martin at v.loewis.de Fri Oct 30 09:53:19 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Oct 2009 09:53:19 +0100 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> Message-ID: <4AEAA97F.7060300@v.loewis.de> Jesse Noller wrote: > On Thu, Oct 29, 2009 at 8:31 PM, R. David Murray wrote: > >> I'd say that particular one is a bug in the tests. If /dev/shm is >> not available and is required, then the tests should be skipped with >> an appropriate message. It would also secondarily be an issue with >> the buildbot fleet, since multiprocessing would then not be getting >> thoroughly tested by those buildbots. > > Fwiw: The tests skip on those platforms; but multiprocessing can't > function properly on platforms like that. I'm confused: first you said they fail, now you say they get skipped. Which one is it? I agree with R. David's analysis: if they fail, it's a multiprocessing bug, if they get skipped, it's a flaw in the build slave configuration (but perhaps only slightly so, because it is good if both cases are tested - and we do have machines also that provide /dev/shm). Regards, Martin From martin at v.loewis.de Fri Oct 30 09:57:47 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Oct 2009 09:57:47 +0100 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> Message-ID: <4AEAAA8B.3000007@v.loewis.de> > But the real reason for having a buildbot category (or at least a keyword) > would be to be able to tag all bugs that are currently making buildbots > fail that are _not_ the result of a recent checkin. This would make > the task of finding the bugs that need to be cleaned up to stabilize > the buildbot fleet easier. I'm currently aware of issues 4970, 3892, > and 6462 in this category, and there are a few more that we can/will file > if we continue to pay attention to the failure reports now arriving on > the irc channel. That's convincing; I've created a "buildbot" keyword. I gave it the description "indicates that tests fail on a buildslave for uncertain reasons" If that is indeed the intended purpose of this classification, please keep it in mind when assigning the tag. If I misunderstood the purpose of the keyword, please provide a better description. Regards, Martin From martin at v.loewis.de Fri Oct 30 10:00:40 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Oct 2009 10:00:40 +0100 Subject: [Python-Dev] ssl module In-Reply-To: References: Message-ID: <4AEAAB38.6090609@v.loewis.de> > Is there a place where the status of the ssl module is summarized The documentation of the ssl module should describe its features correctly and precisely. > or a better place to discuss this? I could try to provide contributions or > further details if appropriate. For contributions, this is indeed the right place for discussion. For mere inquiries for help, other places are better (such as python-list). For specific feature requests, use the bug tracker. Regards, Martin From martin at v.loewis.de Fri Oct 30 10:02:12 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Oct 2009 10:02:12 +0100 Subject: [Python-Dev] yappi - any thoughts? In-Reply-To: <65b2175c0910300007i2cf614dq30893434fdd9721a@mail.gmail.com> References: <65b2175c0910300007i2cf614dq30893434fdd9721a@mail.gmail.com> Message-ID: <4AEAAB94.5060006@v.loewis.de> > Any thoughts/comments/ideas, anything?:) Announce it to comp.lang.python.announce if you haven't done so, yet. Regards, Martin From jnoller at gmail.com Fri Oct 30 13:55:08 2009 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 30 Oct 2009 08:55:08 -0400 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4AEAA97F.7060300@v.loewis.de> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> <4AEAA97F.7060300@v.loewis.de> Message-ID: <4222a8490910300555g5d2ba753o7237171c3af1aad9@mail.gmail.com> On Fri, Oct 30, 2009 at 4:53 AM, "Martin v. L?wis" wrote: > I'm confused: first you said they fail, now you say they get skipped. > Which one is it? I agree with R. David's analysis: if they fail, it's > a multiprocessing bug, if they get skipped, it's a flaw in the build > slave configuration (but perhaps only slightly so, because it is good > if both cases are tested - and we do have machines also that provide > /dev/shm). They failed until we had the tests skip those platforms - at the time, I felt that it was more of a bug with the build slave configuration than a multiprocessing issue, I don't like skipping tests unless the platform fundamentally isn't supported (e.g. broken semaphores for some actions on OS/X) - linux platforms support this functionality just fine - except when in locked-down chroot jails. The only reason I brought it up was to point out the a buildbot configuration on a given host can make tests fail even if those tests would normally pass on that operating system. jesse From exarkun at twistedmatrix.com Fri Oct 30 15:15:35 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 30 Oct 2009 14:15:35 -0000 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4222a8490910300555g5d2ba753o7237171c3af1aad9@mail.gmail.com> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> <4AEAA97F.7060300@v.loewis.de> <4222a8490910300555g5d2ba753o7237171c3af1aad9@mail.gmail.com> Message-ID: <20091030141535.12668.1812365877.divmod.xquotient.85@localhost.localdomain> On 12:55 pm, jnoller at gmail.com wrote: >On Fri, Oct 30, 2009 at 4:53 AM, "Martin v. L?wis" >wrote: >>I'm confused: first you said they fail, now you say they get skipped. >>Which one is it? I agree with R. David's analysis: if they fail, it's >>a multiprocessing bug, if they get skipped, it's a flaw in the build >>slave configuration (but perhaps only slightly so, because it is good >>if both cases are tested - and we do have machines also that provide >>/dev/shm). > >They failed until we had the tests skip those platforms - at the time, >I felt that it was more of a bug with the build slave configuration >than a multiprocessing issue, I don't like skipping tests unless the >platform fundamentally isn't supported (e.g. broken semaphores for >some actions on OS/X) - linux platforms support this functionality >just fine - except when in locked-down chroot jails. > >The only reason I brought it up was to point out the a buildbot >configuration on a given host can make tests fail even if those tests >would normally pass on that operating system. Just as a build slave can be run in a chroot, so can any other Python program. This is a real shortcoming of the multiprocessing module. It's entirely possible that people will want to run Python software in chroots sometimes. So it's proper to acknowledge that this is an unsupported environment. The fact that the kernel in use is the same as the kernel in use on another supported platform is sort of irrelevant. The kernel is just one piece of the system, there are many other important pieces. Jean-Paul From rdmurray at bitdance.com Fri Oct 30 15:35:22 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Oct 2009 10:35:22 -0400 (EDT) Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4222a8490910300555g5d2ba753o7237171c3af1aad9@mail.gmail.com> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> <4AEAA97F.7060300@v.loewis.de> <4222a8490910300555g5d2ba753o7237171c3af1aad9@mail.gmail.com> Message-ID: On Fri, 30 Oct 2009 at 08:55, Jesse Noller wrote: > On Fri, Oct 30, 2009 at 4:53 AM, "Martin v. L?wis" wrote: >> I'm confused: first you said they fail, now you say they get skipped. >> Which one is it? I agree with R. David's analysis: if they fail, it's >> a multiprocessing bug, if they get skipped, it's a flaw in the build >> slave configuration (but perhaps only slightly so, because it is good >> if both cases are tested - and we do have machines also that provide >> /dev/shm). > > They failed until we had the tests skip those platforms - at the time, > I felt that it was more of a bug with the build slave configuration > than a multiprocessing issue, I don't like skipping tests unless the > platform fundamentally isn't supported (e.g. broken semaphores for > some actions on OS/X) - linux platforms support this functionality > just fine - except when in locked-down chroot jails. As Martin pointed out, Python supports both configurations (chroot and non-chroot), and needs to be tested in both. Somewhere we should probably have a list of what tests are getting skipped on what buildslaves so we can inspect the buildbot fleet for complete coverage, but I'm not sure who is going to volunteer to create and maintain that list :) > The only reason I brought it up was to point out the a buildbot > configuration on a given host can make tests fail even if those tests > would normally pass on that operating system. Yes, and that's a kind of ticket that should end up getting tagged with the new 'buildbot' keyword (thanks, Martin), IMO. --David From jnoller at gmail.com Fri Oct 30 15:44:59 2009 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 30 Oct 2009 10:44:59 -0400 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <20091030141535.12668.1812365877.divmod.xquotient.85@localhost.localdomain> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4222a8490910291858sc1caa57k7f46e03ab56262ca@mail.gmail.com> <4AEAA97F.7060300@v.loewis.de> <4222a8490910300555g5d2ba753o7237171c3af1aad9@mail.gmail.com> <20091030141535.12668.1812365877.divmod.xquotient.85@localhost.localdomain> Message-ID: <4222a8490910300744q61414bc9m1a82047c89e741fe@mail.gmail.com> On Fri, Oct 30, 2009 at 10:15 AM, wrote: > On 12:55 pm, jnoller at gmail.com wrote: >> >> On Fri, Oct 30, 2009 at 4:53 AM, "Martin v. L?wis" >> wrote: >>> >>> I'm confused: first you said they fail, now you say they get skipped. >>> Which one is it? I agree with R. David's analysis: if they fail, it's >>> a multiprocessing bug, if they get skipped, it's a flaw in the build >>> slave configuration (but perhaps only slightly so, because it is good >>> if both cases are tested - and we do have machines also that provide >>> /dev/shm). >> >> They failed until we had the tests skip those platforms - at the time, >> I felt that it was more of a bug with the build slave configuration >> than a multiprocessing issue, I don't like skipping tests unless the >> platform fundamentally isn't supported (e.g. broken semaphores for >> some actions on OS/X) - linux platforms support this functionality >> just fine - except when in locked-down chroot jails. >> >> The only reason I brought it up was to point out the a buildbot >> configuration on a given host can make tests fail even if those tests >> would normally pass on that operating system. > > Just as a build slave can be run in a chroot, so can any other Python > program. ?This is a real shortcoming of the multiprocessing module. It's > entirely possible that people will want to run Python software in chroots > sometimes. ?So it's proper to acknowledge that this is an unsupported > environment. ?The fact that the kernel in use is the same as the kernel in > use on another supported platform is sort of irrelevant. The kernel is just > one piece of the system, there are many other important pieces. > > Jean-Paul I'm well aware of that. From steve at pearwood.info Fri Oct 30 16:11:32 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 31 Oct 2009 02:11:32 +1100 Subject: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it In-Reply-To: <2585AF2DFDDE4C61B3596633F5001945@RaymondLaptop1> References: <200910231132.45504.w.richert@gmx.net> <200910301358.17752.steve@pearwood.info> <2585AF2DFDDE4C61B3596633F5001945@RaymondLaptop1> Message-ID: <200910310211.33590.steve@pearwood.info> On Fri, 30 Oct 2009 03:00:27 pm Raymond Hettinger wrote: [skipping to the last paragraph] > Sorry for so many questions Don't be sorry. These are good questions, and I'll try to answer them. But keep in mind that this isn't my proposal -- I vary between +0 and +1 on the proposal depending on the time of the day *wink* > >(1) it will only fail if the set is empty; > > Just like next() except that next() gives you the option to supply a > default and can be used on any iterator (perhaps iter(s) or > itertools.cycle(s) etc). Yes. I had considered suggesting that sets become their own iterator, so you could do the following: >>> s = set([1,2]) >>> next(s) 2 but that leads to the problem that if you accept a set from somewhere else, you have no idea whether next(s) will succeed or raise StopIteration. It may have already been exhausted before it reached you. > > (2) it should be efficient; > > Is this about optimization? Not primarily. Perhaps I should have said it should not be inefficient. It's primarily about there being One Obvious Way to extract an arbitrary item from a set -- this is a reoccurring question on comp.lang.python. Being efficient is a bonus, but it shouldn't be inefficient. > I wouldn't expect "x=s.get()" to beat "for x in s: break". > Attribute lookup and method calls usually are slower > than equivalents using built-in syntax with specific opcodes. Obviously any hypothetical solution would need to be benchmarked, but I wouldn't be concerned if s.get() was marginally slower than for `x in s: break`. When people are left to come up with their own ad hoc solutions, we've seen some poor solutions. I've seen, possibly in this very thread, the following O(N) suggestions: for x in s: pass x = list(s)[0] The usual technique people tend to come up with is: x = s.pop() s.add(x) Which strikes many people (including myself) as inelegant. Surely "get an element" is a more fundamental operation than "get an element and remove it"? > > (3) if you call it repeatedly on a set without modifying the set, > > you will cycle through each element in turn in some unspecified > > arbitrary order. > > What's wrong with using next()? That is what it's for. You can't call next() on a set. You have to call next(iter(set)). From Python 3.0: >>> next(set([1,2])) Traceback (most recent call last): File "", line 1, in TypeError: set object is not an iterator >>> next(iter(set([1,2]))) 1 Unless I missed something, this is so unobvious that nobody has suggested it in the thread yet. > What about this proposal is specific to sets, i.e. why don't you want > the same thing for lists. tuples, strings, file objects, or any other > iterable? Sequences such as lists, tuples and strings have easy random access. It's easy to get an arbitrary element: pick an index i by whatever method you like, and get seq[i]. Many file objects are similar, you have random access with seek(). It is unpractical and an over-generalisation to apply this to general iterables, for reasons I'm sure I don't need to go into. But sets and frozensets aren't lazily generated streams, they actually do store the elements inside their data structure in the same way that lists do. > Does this proposal pass the test of being self-descriptive? Can you > write a code fragment that exercises the cycling behavior, show it to > another programmer, and have them correctly deduce what the code does > (i.e. that different values are returned, that it fails when the set > it empty, that it wraps around and never terminates)? Can they > readily differentiate it for dict.get() which has decidedly different > semantics? I don't expect ConfigParser.get() to have the same semantics as dict.get(), and they're much more closely related than sets and dicts. I don't understand why so many people apparently have a problem with the name get(), but that's okay, I'm not wedded to the idea either. If folks prefer a different name, by all means suggest it. I like the name suggested by Wikipedia, "pick". As for being self-descriptive, I don't know, I haven't tried the experiment. > > To clarify point 3, given: > > > > x = set.get() > > y = set.get() > > > > then x and y will only be the same element if set has length one. > > So, it can't even be used for looping through a set because there is > no termination? That's a feature, not a bug! This is not intended to be a replacement for standard set iteration. Anyone writing the following: for _ in len(s): process(s.get()) instead of for x in s: process(x) will be taken out and slapped with a large fish. My reasoning for the given behaviour is as follows: * If you want the same element twice from a set, the One Obvious Way is to get an element from the set (somehow!) and assign it to a local variable: x = s.get() process(x) # later... process(x) So having get() return the same value each time is not useful or necessary. * If you want a random element on each call, the One Obvious Way is to call random.choice(list(s)). * Since sets aren't mappings, or sequences, there's no sensible way to look up an index or key, so any variation of "get element #1" are out. * Which leaves returning the elements in some unspecified arbitrary order. The most obvious arbitrary order is to cycle through them, which conveniently is exactly what iter(set) does, but we don't need to guarantee that order. > > I believe that the patch supplied by Willi Richart implemented > > these behaviours. > > > > http://bugs.python.org/issue7212 > > So you want to introduce additional, hidden state to sets? (to make > sure that successive invocations return different values) If you can think of any other way to efficiently cycle over the elements in a set, I'm all for it :) Presumably this is no different from what dict views do, except they don't wrap around when exhausted. > Do you want a thread local version too? (so that two threads can call > gets without stomping on each other's guarantees that successive > calls will produce distinct elements) I think that's overkill. I wouldn't think we need to guarantee that two threads don't see the same element, or that each thread will see each element in the same order. We need only the much weaker guarantee that two subsequent calls to get() in the one thread with no modification to the set between the calls won't retrieve the same element each time (unless there is only one element to retrieve), and that each element will eventually be seen. > Do you have any real-world use-cases where next(), for-loops, or > itertools wouldn't suffice? Someone in this thread -- forgive me, I don't recall who -- had the case where they had a frozenset which they knew only had one element, but no obvious way to retrieve that element. To my mind, the major rationalisation for set.get() is not that there's no way to get a single element from a set, but that there is no obvious, easily discoverable way to retrieve a single element. > Is there a precedent in *any* other language you've ever seen? (setl > has an "arb" function but it makes no promises about returning > different values on consequetive calls; otherwise, I've never seen an > equivalent in any other set implementation). I can't say I've seen one in any other languages, but Wikipedia lists "pick" as a fundamental set operation: pick(S): returns an arbitrary element of S. http://en.wikipedia.org/wiki/Set_(computer_science) This page claims that Icon has an operator that returns a random element of a set: ? set( [1, 2, 3, 4, 5] ) http://stackoverflow.com/questions/124671/picking-a-random-element-from-a-set but I've never used Icon myself and so can't confirm that. > Do you think the return-different-values-on-successive-calls > semantics is self-evident and non-magical as compared to a straight > for-loop or next(it)? I'm going to sit on the fence on that and say "maybe". > ISTM, that when streams have non-destructive getters with > self-advancing pointers, they also have a seek() function so that it > can be controlled. Will this proposal need a seek() method too? No. Since sets are unordered, there's no way to seek to a specific element. > Sorry for so many questions, but I honestly think there are too many > unresolved design issues. We've seen no real-world source code that > would be improved fwith the proposal. I think it sounds conceptually > tempting and is fun to theorize about, but it actual implementation > it will make sets more difficult to learn and it would quickly become > a piece of rarely used, poorly understood cruft. -- Steven D'Aprano From solipsis at pitrou.net Fri Oct 30 16:35:33 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Oct 2009 15:35:33 +0000 (UTC) Subject: [Python-Dev] Retrieve an arbitrary element from a set without removing it References: <200910231132.45504.w.richert@gmx.net> <200910301358.17752.steve@pearwood.info> <2585AF2DFDDE4C61B3596633F5001945@RaymondLaptop1> <200910310211.33590.steve@pearwood.info> Message-ID: Steven D'Aprano pearwood.info> writes: > > If you can think of any other way to efficiently cycle over the elements > in a set, I'm all for it :) How about "for x in s"? Or if you want to cycle: >>> s = set('abc') >>> it = itertools.cycle(s) >>> next(it) 'a' >>> next(it) 'c' >>> next(it) 'b' >>> next(it) 'a' Or if you don't want the overhead of itertools.cycle() keeping a copy of the set's elements: >>> s = set('abc') >>> it = itertools.chain.from_iterable(itertools.cycle([s])) >>> next(it) 'a' >>> next(it) 'c' >>> next(it) 'b' >>> next(it) 'a' >>> next(it) 'c' >>> next(it) 'b' > I can't say I've seen one in any other languages, but Wikipedia > lists "pick" as a fundamental set operation: > > pick(S): returns an arbitrary element of S. Well, it's an arbitrary element. It isn't specified that it will try to return different results in a row to satisfy the developer's aesthetical preferences... > This page claims that Icon has an operator that returns a random element > of a set: > > ? set( [1, 2, 3, 4, 5] ) random != arbitrary != weak-guaranteedly distinct From ncoghlan at gmail.com Fri Oct 30 16:46:01 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 31 Oct 2009 01:46:01 +1000 Subject: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it In-Reply-To: <200910310211.33590.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <200910301358.17752.steve@pearwood.info> <2585AF2DFDDE4C61B3596633F5001945@RaymondLaptop1> <200910310211.33590.steve@pearwood.info> Message-ID: <4AEB0A39.2070707@gmail.com> Steven D'Aprano wrote: >> So you want to introduce additional, hidden state to sets? (to make >> sure that successive invocations return different values) > > If you can think of any other way to efficiently cycle over the elements > in a set, I'm all for it :) for x in itertools.cycle(s): # this is an infinite loop Having a pick() or get() method that returns an arbitrary member of a set makes sense to me. Having any state on the set that guarantees successive calls to get will return different values feels wrong - creating an object with that extra state is what iter(s) is for. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Fri Oct 30 17:21:06 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Oct 2009 16:21:06 +0000 (UTC) Subject: [Python-Dev] Possible language summit topic: buildbots References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> Message-ID: Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ?crit?: > I think that money can help in two ways in this case. > > First, there are now a multitude of cloud hosting providers which will > operate a slave machine for you. BuildBot has even begun to support > this deployment use-case by allowing you to start up and shut down vms > on demand to save on costs. Amazon's EC2 service is supported out of > the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. After all, big companies often like sponsoring open-source projects, especially when the project is well-known and the donation is cheap for them. This would have several advantages: - the machines are administered by the provider: we don't have to worry about failed hardware, connectivity loss etc. - any Python core developer could get ssh access to the VMs to run tests directly, since they would be dedicated buildbot instances - they are not tied to a particular owner when it comes to fixing system problems, which means we eliminate a single point of failure: if a volunteer gets demotivated/bored/missing in action, someone can replace him/her easily - there are a number of various OS images available (of course, we still need competent people to install the required software -- buildbot, etc.) Since I've never used any such service ("cloud"-based VMs), I'm not sure what the downsides would be. But it seems to be that it would be at least worth trying. Right now we have around 15 buildbots but two thirds of them are down, the others sometimes fail or disconnect in erratic ways and it's difficult for "regular" core developers to be aware of what's precisely going on. Of course this could also be a broken idea, for whatever reason I'm not aware of. Regards Antoine. From ctb at msu.edu Fri Oct 30 17:31:28 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 30 Oct 2009 09:31:28 -0700 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> Message-ID: <20091030163128.GA30191@idyll.org> On Fri, Oct 30, 2009 at 04:21:06PM +0000, Antoine Pitrou wrote: > > Hello, > > Sorry for the little redundancy, I would like to underline Jean-Paul's > suggestion here: > > Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ??crit??: > > I think that money can help in two ways in this case. > > > > First, there are now a multitude of cloud hosting providers which will > > operate a slave machine for you. BuildBot has even begun to support > > this deployment use-case by allowing you to start up and shut down vms > > on demand to save on costs. Amazon's EC2 service is supported out of > > the box in the latest release. > > I'm not a PSF member, but it seems to me that the PSF could ask Amazon > (or any other virtual machine business anyway) to donate a small number > of permanent EC2 instances in order to run buildslaves on. [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Others have made similar offers. The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). What we really need (IMO) is someone to dig into the tests to figure out which tests fail randomly and why, and to fix them on specific architectures that most of us don't personally use. This is hard work that is neither glamorous nor popular. I think the idea of paying a dedicated developer to make the CPython+buildbot tests reliable is better, although I would still be -0 on it (I don't think the PSF should be paying for this kind of thing at all). cheers, --titus -- C. Titus Brown, ctb at msu.edu From solipsis at pitrou.net Fri Oct 30 17:41:39 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Oct 2009 17:41:39 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091030163128.GA30191@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> Message-ID: <1256920899.5472.5.camel@localhost> Le vendredi 30 octobre 2009 ? 09:31 -0700, C. Titus Brown a ?crit : > [ ... ] > > I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux > ia64; Linux x86; and Mac OS X. Others have made similar offers. The > architectures supported by the cloud services don't really add anything > (and generally don't have Mac OS X support, AFAIK). Well these VMs would have to run buildslaves on them, then. Are you ready to host some and connect them to the current buildbot infrastructure? (VMs without buildslaves are less interesting IMO) > What we really need (IMO) is someone to dig into the tests to figure out which > tests fail randomly and why, and to fix them on specific architectures that > most of us don't personally use. This is hard work that is neither glamorous > nor popular. I'm sure some of us are ready to do so (*). The situation has already improved quite a lot in the recent times. But fixing platform- or, worse, setup-specific issues often requires shell access to the target system, otherwise you spend too much time trying fixes on the SVN and waiting for the buildbot to react. (*) After all, if we weren't, we wouldn't even care about buildbots, we'd be content with running the test suite on our own machines > I think the idea of paying a dedicated developer to make the CPython+buildbot > tests reliable is better, although I would still be -0 on it (I don't think > the PSF should be paying for this kind of thing at all). Paying developers in volunteer communities is always more contentious than paying for other kinds of resources. (It's generally more expensive too) From olemis at gmail.com Fri Oct 30 17:42:30 2009 From: olemis at gmail.com (Olemis Lang) Date: Fri, 30 Oct 2009 11:42:30 -0500 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> Message-ID: <24ea26600910300942j3790e8a0m83c011ac17e57bd9@mail.gmail.com> On Sun, Oct 25, 2009 at 9:13 AM, wrote: > On 12:48 pm, ctb at msu.edu wrote: >> >> [snip] >> >> The most *exciting* part of pony-build, apart from the always-riveting >> spectacle of "titus rediscovering problems that buildbot solved 5 years >> ago", >> is the loose coupling of recording server to the build slaves and build >> reporters. ?My plan is to enable a simple and lightweight XML-RPC and/or >> REST-ish interface for querying the recording server from scripts or other >> Web >> sites. ?This has Brett aquiver with anticipation, I gather -- no more >> visual >> inspection of buildbot waterfall pages ;) > > BuildBot has an XML-RPC interface. ?So Brett can probably do what he wants > with BuildBot right now. > ... but pony-build follows a different model ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From ctb at msu.edu Fri Oct 30 17:43:31 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 30 Oct 2009 09:43:31 -0700 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <1256920899.5472.5.camel@localhost> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> <1256920899.5472.5.camel@localhost> Message-ID: <20091030164331.GA19953@idyll.org> On Fri, Oct 30, 2009 at 05:41:39PM +0100, Antoine Pitrou wrote: > Le vendredi 30 octobre 2009 ?? 09:31 -0700, C. Titus Brown a ??crit : > > [ ... ] > > > > I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux > > ia64; Linux x86; and Mac OS X. Others have made similar offers. The > > architectures supported by the cloud services don't really add anything > > (and generally don't have Mac OS X support, AFAIK). > > Well these VMs would have to run buildslaves on them, then. Are you > ready to host some and connect them to the current buildbot > infrastructure? > (VMs without buildslaves are less interesting IMO) No, I'm not willing to spend the time to install and maintain buildbot. But I'm happy to give the necessary access to those who are interested and willing. (...and let me tell you, getting these !#%!#$! Windows VMs up and running already took an immense amount of effort ;) > > What we really need (IMO) is someone to dig into the tests to figure out which > > tests fail randomly and why, and to fix them on specific architectures that > > most of us don't personally use. This is hard work that is neither glamorous > > nor popular. > > I'm sure some of us are ready to do so (*). The situation has already > improved quite a lot in the recent times. But fixing platform- or, > worse, setup-specific issues often requires shell access to the target > system, otherwise you spend too much time trying fixes on the SVN and > waiting for the buildbot to react. > > (*) After all, if we weren't, we wouldn't even care about buildbots, > we'd be content with running the test suite on our own machines I look forward to it! cheers, --titus -- C. Titus Brown, ctb at msu.edu From ctb at msu.edu Fri Oct 30 17:45:49 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 30 Oct 2009 09:45:49 -0700 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: <24ea26600910300942j3790e8a0m83c011ac17e57bd9@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> <24ea26600910300942j3790e8a0m83c011ac17e57bd9@mail.gmail.com> Message-ID: <20091030164549.GC19953@idyll.org> On Fri, Oct 30, 2009 at 11:42:30AM -0500, Olemis Lang wrote: > On Sun, Oct 25, 2009 at 9:13 AM, wrote: > > On 12:48 pm, ctb at msu.edu wrote: > >> > >> [snip] > >> > >> The most *exciting* part of pony-build, apart from the always-riveting > >> spectacle of "titus rediscovering problems that buildbot solved 5 years > >> ago", > >> is the loose coupling of recording server to the build slaves and build > >> reporters. ?My plan is to enable a simple and lightweight XML-RPC and/or > >> REST-ish interface for querying the recording server from scripts or other > >> Web > >> sites. ?This has Brett aquiver with anticipation, I gather -- no more > >> visual > >> inspection of buildbot waterfall pages ;) > > > > BuildBot has an XML-RPC interface. ?So Brett can probably do what he wants > > with BuildBot right now. > > ... but pony-build follows a different model I'd rather not get into discussions of why my vaporware is going to be way, way better than anything anyone else could possibly do -- that's flamebait and not very friendly, in the end. Let's just say that I'm wasting my own time on it to scratch my own itch and leave it at that! thanks, --titus -- C. Titus Brown, ctb at msu.edu From p.f.moore at gmail.com Fri Oct 30 17:49:51 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Oct 2009 16:49:51 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091030163128.GA30191@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> Message-ID: <79990c6b0910300949ybc11b29q11900f08a79cdb9e@mail.gmail.com> 2009/10/30 C. Titus Brown : > On Fri, Oct 30, 2009 at 04:21:06PM +0000, Antoine Pitrou wrote: >> >> Hello, >> >> Sorry for the little redundancy, I would like to underline Jean-Paul's >> suggestion here: >> >> Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ??crit??: >> > I think that money can help in two ways in this case. >> > >> > First, there are now a multitude of cloud hosting providers which will >> > operate a slave machine for you. ?BuildBot has even begun to support >> > this deployment use-case by allowing you to start up and shut down vms >> > on demand to save on costs. ?Amazon's EC2 service is supported out of >> > the box in the latest release. >> >> I'm not a PSF member, but it seems to me that the PSF could ask Amazon >> (or any other virtual machine business anyway) to donate a small number >> of permanent EC2 instances in order to run buildslaves on. > > [ ... ] > > I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux > ia64; Linux x86; and Mac OS X. ?Others have made similar offers. ?The > architectures supported by the cloud services don't really add anything > (and generally don't have Mac OS X support, AFAIK). As Antione pointed out, it's not clear (at least, it isn't to me) what that leaves to be done. As a counter-offer: Given remote access to however many Windows VMs you want to provide, I'll get them up and running with buildslaves on them. If that requires software such as Visual Studio, I have copies via the MSDN licenses that I am happy to provide. Once things are up and running, I'll be prepared to do basic care and feeding of the buildslave, but as my time is limited, it would be nice if others would pitch in to help. In other words, if it's setup effort that's lacking, I'll provide it. As long as someone else can cover systems admin, and we get some level of volunteers to cover ongoing support, that should give us better Windows coverage on the buildbots. Paul. From ctb at msu.edu Fri Oct 30 17:56:15 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 30 Oct 2009 09:56:15 -0700 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <79990c6b0910300949ybc11b29q11900f08a79cdb9e@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> <79990c6b0910300949ybc11b29q11900f08a79cdb9e@mail.gmail.com> Message-ID: <20091030165615.GB5132@idyll.org> On Fri, Oct 30, 2009 at 04:49:51PM +0000, Paul Moore wrote: > 2009/10/30 C. Titus Brown : > > On Fri, Oct 30, 2009 at 04:21:06PM +0000, Antoine Pitrou wrote: > >> > >> Hello, > >> > >> Sorry for the little redundancy, I would like to underline Jean-Paul's > >> suggestion here: > >> > >> Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ??crit??: > >> > I think that money can help in two ways in this case. > >> > > >> > First, there are now a multitude of cloud hosting providers which will > >> > operate a slave machine for you. ?BuildBot has even begun to support > >> > this deployment use-case by allowing you to start up and shut down vms > >> > on demand to save on costs. ?Amazon's EC2 service is supported out of > >> > the box in the latest release. > >> > >> I'm not a PSF member, but it seems to me that the PSF could ask Amazon > >> (or any other virtual machine business anyway) to donate a small number > >> of permanent EC2 instances in order to run buildslaves on. > > > > [ ... ] > > > > I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux > > ia64; Linux x86; and Mac OS X. ?Others have made similar offers. ?The > > architectures supported by the cloud services don't really add anything > > (and generally don't have Mac OS X support, AFAIK). > > As Antione pointed out, it's not clear (at least, it isn't to me) what > that leaves to be done. Great! We've solved the problem ;) > As a counter-offer: Given remote access to however many Windows VMs > you want to provide, I'll get them up and running with buildslaves on > them. If that requires software such as Visual Studio, I have copies > via the MSDN licenses that I am happy to provide. I, too, have MSDN licenses, and I have functioning build environments on all of the VMs (I think -- I've only tested Win XP currently: http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8276 ) I also have an OS X 10.5 machine that I can let you into through a firewall; it's building Python 2.7 quite nicely: http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8229 > Once things are up and running, I'll be prepared to do basic care and > feeding of the buildslave, but as my time is limited, it would be nice > if others would pitch in to help. I would be somewhat unhappy about giving more than three or four people admin access, but am prepared to lie back and think of England. --titus -- C. Titus Brown, ctb at msu.edu From exarkun at twistedmatrix.com Fri Oct 30 17:58:29 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 30 Oct 2009 16:58:29 -0000 Subject: [Python-Dev] Cloud build slaves (was Re: Possible language summit topic: buildbots) In-Reply-To: <20091030163128.GA30191@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> Message-ID: <20091030165829.12668.1546393452.divmod.xquotient.118@localhost.localdomain> On 04:31 pm, ctb at msu.edu wrote: >On Fri, Oct 30, 2009 at 04:21:06PM +0000, Antoine Pitrou wrote: >> >>Hello, >> >>Sorry for the little redundancy, I would like to underline Jean-Paul's >>suggestion here: >> >>Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ??crit??: >> > I think that money can help in two ways in this case. >> > >> > First, there are now a multitude of cloud hosting providers which >>will >> > operate a slave machine for you. BuildBot has even begun to support >> > this deployment use-case by allowing you to start up and shut down >>vms >> > on demand to save on costs. Amazon's EC2 service is supported out >>of >> > the box in the latest release. >> >>I'm not a PSF member, but it seems to me that the PSF could ask Amazon >>(or any other virtual machine business anyway) to donate a small >>number >>of permanent EC2 instances in order to run buildslaves on. > >[ ... ] > >I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); >Linux >ia64; Linux x86; and Mac OS X. Okay, let's move on this. Martin has, I believe, said that potential slave operators only need to contact him to get credentials for new slaves. Can you make sure to follow up with him to get slaves running on these machines? Or would you rather give out access to someone else and have them do the build slave setup? >Others have made similar offers. I'll similarly encourage them to take action, then. Do you happen to remember who? >The >architectures supported by the cloud services don't really add anything >(and generally don't have Mac OS X support, AFAIK). That's not entirely accurate. Currently, CPython has slaves on these platforms: - x86 - FreeBSD - Windows XP - Gentoo Linux - OS X - ia64 - Ubuntu Linux - Alpha - Debian Linux So, assuming we don't want to introduce any new OS, Amazon could fill in the following holes: - x86 - Ubuntu Linux - ia64 - FreeBSD - Windows XP - Gentoo Linux So very modestly, that's 4 currently missing slaves which Amazon's cloud service *does* add. It's easy to imagine further additions it could make as well. >What we really need (IMO) is someone to dig into the tests to figure >out which >tests fail randomly and why, and to fix them on specific architectures >that >most of us don't personally use. This is hard work that is neither >glamorous >nor popular. Sure. That's certainly necessary. I don't think anyone is suggesting that it's not. Fortunately, adding more build slaves is not mutually exclusive with a developer fixing bugs in CPython. >I think the idea of paying a dedicated developer to make the >CPython+buildbot >tests reliable is better, although I would still be -0 on it (I don't >think >the PSF should be paying for this kind of thing at all). I hope everyone is on board with the idea of fixing bugs in CPython, either in the actual implementation of features or in the tests for those features. That being the case, the discussion of whether or not the PSF should try to fund such a task is perhaps best discussed on the PSF members list. Jean-Paul From exarkun at twistedmatrix.com Fri Oct 30 18:01:29 2009 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 30 Oct 2009 17:01:29 -0000 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: <24ea26600910300942j3790e8a0m83c011ac17e57bd9@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> <24ea26600910300942j3790e8a0m83c011ac17e57bd9@mail.gmail.com> Message-ID: <20091030170129.12668.399242170.divmod.xquotient.122@localhost.localdomain> On 04:42 pm, olemis at gmail.com wrote: >On Sun, Oct 25, 2009 at 9:13 AM, wrote: >>On 12:48 pm, ctb at msu.edu wrote: >>> >>>[snip] >>> >>>The most *exciting* part of pony-build, apart from the always- >>>riveting >>>spectacle of "titus rediscovering problems that buildbot solved 5 >>>years >>>ago", >>>is the loose coupling of recording server to the build slaves and >>>build >>>reporters. ?My plan is to enable a simple and lightweight XML-RPC >>>and/or >>>REST-ish interface for querying the recording server from scripts or >>>other >>>Web >>>sites. ?This has Brett aquiver with anticipation, I gather -- no more >>>visual >>>inspection of buildbot waterfall pages ;) >> >>BuildBot has an XML-RPC interface. ?So Brett can probably do what he >>wants >>with BuildBot right now. > >... but pony-build follows a different model But BuildBot exists, is deployed, and can be used now, without waiting. (Sorry, I don't really understand what point you were hoping to make with your message, so I just thought I'd follow up in the same style and hope that you'll understand my message even if I don't understand yours :). Jean-Paul From status at bugs.python.org Fri Oct 30 18:07:15 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 30 Oct 2009 18:07:15 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20091030170715.D6AED781CD@psf.upfronthosting.co.za> ACTIVITY SUMMARY (10/23/09 - 10/30/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2483 open (+31) / 16582 closed (+19) / 19065 total (+50) Open issues with patches: 989 Average duration of open issues: 675 days. Median duration of open issues: 430 days. Open Issues Breakdown open 2447 (+31) pending 35 ( +0) Issues Created Or Reopened (53) _______________________________ distutils and IronPython compatibility 10/27/09 CLOSED http://bugs.python.org/issue7071 reopened michael.foord 26backport Popen blocks program from another thread 10/23/09 CLOSED http://bugs.python.org/issue7193 reopened dgriff1 test_thread is flaky 10/23/09 CLOSED http://bugs.python.org/issue7194 created pitrou Value error 'path is on drive c: start on drive d:' in os.path.r 10/23/09 http://bugs.python.org/issue7195 created jaraco Clarify str.split() behavior 10/24/09 http://bugs.python.org/issue7196 created gagenellina patch test_multiprocessing crashes under Windows when run in verbose m 10/24/09 http://bugs.python.org/issue7197 created pitrou patch csv.writer 10/24/09 CLOSED http://bugs.python.org/issue7198 created zacktu Doc: Logging level order seems inconsistent 10/24/09 CLOSED http://bugs.python.org/issue7199 created tjreedy multiprocessing deadlock on Mac OS X when queue collected before 10/24/09 http://bugs.python.org/issue7200 created bquinlan double Endian problem and more on arm 10/24/09 http://bugs.python.org/issue7201 created mancausoft "python setup.py MYCOMMAND --verbose" does not yield an unrecogn 10/25/09 http://bugs.python.org/issue7202 created zooko fixer for map(None, ...) needs to consider multi-argument case 10/25/09 CLOSED http://bugs.python.org/issue7203 created georg.brandl Strange shebang line in test_pep263 10/25/09 CLOSED http://bugs.python.org/issue7204 created wiz bz2file deadlock 10/25/09 CLOSED http://bugs.python.org/issue7205 created rbcollins patch 64 bit python fails on Windows 7 10/25/09 http://bugs.python.org/issue7206 created richo test_telnetlib fails on OS X 10.6 10/26/09 CLOSED http://bugs.python.org/issue7207 created mark.dickinson patch, needs review Getpass echo's password to screen on 2.6, but not on 2.5 or 3.1 10/26/09 http://bugs.python.org/issue7208 created pajs at fodder.org.uk patch Prevents uint_t from being used on QNX 10/26/09 http://bugs.python.org/issue7209 created kraai patch Proposed Syntax Checks in Test Suite 10/26/09 http://bugs.python.org/issue7210 created ChuckRhode patch select module - kevent ident field 64 bit issue 10/26/09 http://bugs.python.org/issue7211 created mbroughton patch Retrieve an arbitrary element from a set without removing it 10/26/09 http://bugs.python.org/issue7212 created wrichert patch Popen.subprocess change close_fds default to True 10/26/09 http://bugs.python.org/issue7213 created milko.krachounov TreeBuilder.end(tag) differs between cElementTree and ElementTre 10/26/09 http://bugs.python.org/issue7214 created merrellb TreeBuilder.end(tag) differs between cElementTree and ElementTre 10/26/09 CLOSED http://bugs.python.org/issue7215 created merrellb low performance of zipfile readline() 10/27/09 http://bugs.python.org/issue7216 created volker_siepmann IDLE Subprocess Startup Error 10/27/09 http://bugs.python.org/issue7217 created irdb test_site failure under Windows 10/27/09 CLOSED http://bugs.python.org/issue7218 created pitrou Unhelpful error message when a distutils package install fails d 10/27/09 http://bugs.python.org/issue7219 created michael.foord 26backport python -m no longer executes packages directly 10/27/09 CLOSED http://bugs.python.org/issue7220 created Martijn DispatcherWithSendTests_UsePoll with test_asyncore does nothing 10/27/09 http://bugs.python.org/issue7221 created pitrou thread reaping is imperfect 10/27/09 http://bugs.python.org/issue7222 created pitrou patch Simplify and improve Thread.join() 10/27/09 CLOSED http://bugs.python.org/issue7223 created pitrou patch One obvious way to do interning 10/28/09 http://bugs.python.org/issue7224 created belopolsky patch fwrite() compiler warnings 10/28/09 http://bugs.python.org/issue7225 created benjamin.peterson IDLE right-clicks don't work on Mac OS 10.5 10/28/09 http://bugs.python.org/issue7226 created MLModel Shell Support for installation of Python Packages (.EGG) 10/28/09 CLOSED http://bugs.python.org/issue7227 created djlyon %lld for PyErr_Format (Modules/_io/bufferedio.c) 10/28/09 http://bugs.python.org/issue7228 created ocean-city Manual entry for time.daylight can be misleading 10/28/09 http://bugs.python.org/issue7229 created napik test_hotshot fails on solaris 10/28/09 http://bugs.python.org/issue7230 created csernazs Windows installer does not add \Scripts folder to the path 10/28/09 http://bugs.python.org/issue7231 created sorin Support of 'with' statement fo TarFile class 10/29/09 http://bugs.python.org/issue7232 created jaime.buelta patch decimal.py: two rotate() issues 10/29/09 CLOSED http://bugs.python.org/issue7233 created skrah Complete your registration to Python tracker -- keyRVZuTYMju0Lif 10/29/09 CLOSED http://bugs.python.org/issue7234 created kaka IO libary have some error 10/29/09 CLOSED http://bugs.python.org/issue7235 created kaka the reply's additional "Re:" is ok 10/29/09 CLOSED http://bugs.python.org/issue7236 created kaka Syntax error with not 10/29/09 CLOSED http://bugs.python.org/issue7237 created LambertDW frame.f_lineno doesn't get updated after local trace function as 10/29/09 http://bugs.python.org/issue7238 created eggy Error when running a code 10/29/09 CLOSED http://bugs.python.org/issue7239 created MNilson subprocess.Popen.stdout.flush fails os OS-X 10.6.1 10/30/09 http://bugs.python.org/issue7240 created petegibson tkinter fails to import 10/30/09 CLOSED http://bugs.python.org/issue7241 reopened python.noob Forking in a thread raises RuntimeError 10/30/09 http://bugs.python.org/issue7242 created csernazs mac binary download link for 2.6.4 broken 10/30/09 http://bugs.python.org/issue7243 created beard 0.0 and -0.0 identified, with surprising results 10/24/09 http://bugs.python.org/issue1678380 reopened mark.dickinson patch Issues Now Closed (47) ______________________ Option to ignore or substitute ~/.pydistutils.cfg 769 days http://bugs.python.org/issue1180 tarek patch, easy Extension module build fails for MinGW: missing vcvarsall.bat 550 days http://bugs.python.org/issue2698 tarek py3k warn on use of frame.f_exc* 486 days http://bugs.python.org/issue3223 benjamin.peterson easy 'dictionary changed size' error in test_multiprocessing 433 days http://bugs.python.org/issue3578 pitrou tarfile keeps excessive dir structure in compressed files 307 days http://bugs.python.org/issue4750 lars.gustaebel patch tarfile: path problem in arcname under windows 226 days http://bugs.python.org/issue5500 lars.gustaebel Incorrect DST transition 217 days http://bugs.python.org/issue5582 acummings Fix reference leak in io.StringIO 138 days http://bugs.python.org/issue6242 pitrou patch zipfile: Invalid argument when opening zero-sized files 102 days http://bugs.python.org/issue6511 skelker PEP 314 inconsistency (authors/author/maintainer) 28 days http://bugs.python.org/issue6992 tarek utf-8 encoding error 25 days http://bugs.python.org/issue7045 benjamin.peterson 'g'/'G' format docs need a little more explanation 24 days http://bugs.python.org/issue7051 mark.dickinson Automatic test___all__ 23 days http://bugs.python.org/issue7055 pitrou patch Add some test execution environment protection to regrtest 23 days http://bugs.python.org/issue7058 r.david.murray patch archive_util.make_archive doesn't restore the cwd if an error is 19 days http://bugs.python.org/issue7066 tarek distutils and IronPython compatibility 0 days http://bugs.python.org/issue7071 r.david.murray 26backport struct help in the interpreter does not explain about the fmt op 20 days http://bugs.python.org/issue7078 mark.dickinson patch, easy Patch for get_filename in email.message when content-disposition 19 days http://bugs.python.org/issue7082 r.david.murray patch printing a list releases the GIL carelessly 19 days http://bugs.python.org/issue7084 pitrou patch Decimal.is_normal should return True even for numbers with expon 17 days http://bugs.python.org/issue7099 mark.dickinson Extension module build fails for MinGW: missing vcvarsall.bat 12 days http://bugs.python.org/issue7131 tarek Fix Download Current Documentation link 10 days http://bugs.python.org/issue7157 georg.brandl zipfile leaves a file handle open if file is zero size 9 days http://bugs.python.org/issue7169 r.david.murray sum() doesn't work for lists. 6 days http://bugs.python.org/issue7176 ezio.melotti Popen blocks program from another thread 0 days http://bugs.python.org/issue7193 dgriff1 test_thread is flaky 5 days http://bugs.python.org/issue7194 ned.deily csv.writer 0 days http://bugs.python.org/issue7198 zacktu Doc: Logging level order seems inconsistent 4 days http://bugs.python.org/issue7199 vinay.sajip fixer for map(None, ...) needs to consider multi-argument case 2 days http://bugs.python.org/issue7203 segfaulthunter Strange shebang line in test_pep263 0 days http://bugs.python.org/issue7204 georg.brandl bz2file deadlock 2 days http://bugs.python.org/issue7205 pitrou patch test_telnetlib fails on OS X 10.6 1 days http://bugs.python.org/issue7207 mark.dickinson patch, needs review TreeBuilder.end(tag) differs between cElementTree and ElementTre 0 days http://bugs.python.org/issue7215 r.david.murray test_site failure under Windows 0 days http://bugs.python.org/issue7218 tarek python -m no longer executes packages directly 0 days http://bugs.python.org/issue7220 benjamin.peterson Simplify and improve Thread.join() 0 days http://bugs.python.org/issue7223 pitrou patch Shell Support for installation of Python Packages (.EGG) 0 days http://bugs.python.org/issue7227 tarek decimal.py: two rotate() issues 0 days http://bugs.python.org/issue7233 mark.dickinson Complete your registration to Python tracker -- keyRVZuTYMju0Lif 0 days http://bugs.python.org/issue7234 eric.smith IO libary have some error 0 days http://bugs.python.org/issue7235 kaka the reply's additional "Re:" is ok 0 days http://bugs.python.org/issue7236 eric.smith Syntax error with not 0 days http://bugs.python.org/issue7237 benjamin.peterson Error when running a code 0 days http://bugs.python.org/issue7239 MNilson tkinter fails to import 0 days http://bugs.python.org/issue7241 python.noob bdist_deb - Debian packager 251 days http://bugs.python.org/issue1054967 tarek patch long int bitwise ops speedup (patch included) 1773 days http://bugs.python.org/issue1087418 mark.dickinson patch Thread shutdown exception in Thread.notify() 630 days http://bugs.python.org/issue1722344 pitrou patch Top Issues Most Discussed (10) ______________________________ 19 Backport py3k float repr to trunk 17 days open http://bugs.python.org/issue7117 12 %lld for PyErr_Format (Modules/_io/bufferedio.c) 2 days open http://bugs.python.org/issue7228 11 unify pydistutils.cfg and distutils.cfg and use .local 10 days open http://bugs.python.org/issue7175 10 64 bit python fails on Windows 7 5 days open http://bugs.python.org/issue7206 10 long int bitwise ops speedup (patch included) 1773 days closed http://bugs.python.org/issue1087418 9 select module - kevent ident field 64 bit issue 4 days open http://bugs.python.org/issue7211 9 bz2file deadlock 2 days closed http://bugs.python.org/issue7205 9 Python 2.5 64 bit compile fails on Solaris 10/gcc 4.1.1 1029 days open http://bugs.python.org/issue1628484 7 Proposed Syntax Checks in Test Suite 4 days open http://bugs.python.org/issue7210 7 fixer for map(None, ...) needs to consider multi-argument case 2 days closed http://bugs.python.org/issue7203 From olemis at gmail.com Fri Oct 30 18:27:14 2009 From: olemis at gmail.com (Olemis Lang) Date: Fri, 30 Oct 2009 12:27:14 -0500 Subject: [Python-Dev] [TIP] Possible language summit topic: buildbots In-Reply-To: <20091030164549.GC19953@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <20091025124816.GA20329@idyll.org> <20091025141317.11571.1000888001.divmod.xquotient.1552@localhost.localdomain> <24ea26600910300942j3790e8a0m83c011ac17e57bd9@mail.gmail.com> <20091030164549.GC19953@idyll.org> Message-ID: <24ea26600910301027j3724ab79x80de968a118047e8@mail.gmail.com> On Fri, Oct 30, 2009 at 11:45 AM, C. Titus Brown wrote: > On Fri, Oct 30, 2009 at 11:42:30AM -0500, Olemis Lang wrote: >> On Sun, Oct 25, 2009 at 9:13 AM, ? wrote: >> > On 12:48 pm, ctb at msu.edu wrote: >> >> >> >> [snip] >> >> >> >> The most *exciting* part of pony-build, apart from the always-riveting >> >> spectacle of "titus rediscovering problems that buildbot solved 5 years >> >> ago", >> >> is the loose coupling of recording server to the build slaves and build >> >> reporters. ?My plan is to enable a simple and lightweight XML-RPC and/or >> >> REST-ish interface for querying the recording server from scripts or other >> >> Web >> >> sites. ?This has Brett aquiver with anticipation, I gather -- no more >> >> visual >> >> inspection of buildbot waterfall pages ;) >> > >> > BuildBot has an XML-RPC interface. ?So Brett can probably do what he wants >> > with BuildBot right now. >> >> ... but pony-build follows a different model > that was just a brief comment to mention that even if both (buildbot + pony-build) support RPC, they are not just ?the same? . > I'd rather not get into discussions of why my vaporware is going to be > way, way better than anything anyone else could possibly do +1 ... I'll be the first one that won't follow it since I have no time for that and my intention was not to suggest that ?pb is better than bb? (but if you follow, please do it in a separate thread ;o) @exarkun at twistedmatrix.com > But BuildBot exists, is deployed, and can be used now, without waiting. > +1 as I mentioned before I was not talking about eliminating buildbots > (Sorry, I don't really understand what point you were hoping to make with your > message, so I just thought I'd follow up in the same style and hope that you'll > understand my message even if I don't understand yours :). well, I understand that you don't understand, since I barely understand that I will never be able to understand myself ... :) The only thing I can say so far is that if pb is seriously considered as an option ... then I could give a hand (... and I'll possibly do it anyway , once I have time :-/ ) ... turning myself off ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From g.brandl at gmx.net Fri Oct 30 21:37:36 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 30 Oct 2009 21:37:36 +0100 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: <200910301358.17752.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> <200910301358.17752.steve@pearwood.info> Message-ID: Steven D'Aprano schrieb: > On Wed, 28 Oct 2009 04:47:59 am Raymond Hettinger wrote: > >> A dict.get() can be meaningfully used in a loop (because the key can >> vary). A set.get() returns the same value over and over again >> (because there is no key). > > I don't believe anyone has requested those semantics. The suggested > semantics for set.get() with no arguments, as I understand them, are: > > (1) it will only fail if the set is empty; > > (2) it should be efficient; > > (3) if you call it repeatedly on a set without modifying the set, you > will cycle through each element in turn in some unspecified arbitrary > order. I don't like this. It gives a set object a hidden state, something that AFAICS no other builtin has. Iterating over an iterable is what iterators are for. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ctb at msu.edu Fri Oct 30 20:42:45 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 30 Oct 2009 12:42:45 -0700 Subject: [Python-Dev] Cloud build slaves (was Re: Possible language summit topic: buildbots) In-Reply-To: <20091030165829.12668.1546393452.divmod.xquotient.118@localhost.localdomain> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> <20091030165829.12668.1546393452.divmod.xquotient.118@localhost.localdomain> Message-ID: <20091030194244.GA18135@idyll.org> On Fri, Oct 30, 2009 at 04:58:29PM -0000, exarkun at twistedmatrix.com wrote: > On 04:31 pm, ctb at msu.edu wrote: >> On Fri, Oct 30, 2009 at 04:21:06PM +0000, Antoine Pitrou wrote: >>> >>> Hello, >>> >>> Sorry for the little redundancy, I would like to underline Jean-Paul's >>> suggestion here: >>> >>> Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ??crit??: >>> > I think that money can help in two ways in this case. >>> > >>> > First, there are now a multitude of cloud hosting providers which >>> will >>> > operate a slave machine for you. BuildBot has even begun to support >>> > this deployment use-case by allowing you to start up and shut down >>> vms >>> > on demand to save on costs. Amazon's EC2 service is supported out >>> of >>> > the box in the latest release. >>> >>> I'm not a PSF member, but it seems to me that the PSF could ask Amazon >>> (or any other virtual machine business anyway) to donate a small >>> number >>> of permanent EC2 instances in order to run buildslaves on. >> >> [ ... ] >> >> I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); >> Linux >> ia64; Linux x86; and Mac OS X. > > Okay, let's move on this. Martin has, I believe, said that potential > slave operators only need to contact him to get credentials for new > slaves. Can you make sure to follow up with him to get slaves running > on these machines? Or would you rather give out access to someone else > and have them do the build slave setup? I think we crossed threads here; I can provide the VMs, and access to them, but I won't (empirically, don't have the regular time available to ;) maintain buildbot buildslaves. You or Antoine or others are welcome to contact me off-list. Just give me an account name and ssh key, and I'll give you login access via tunneled Remote Desktop to the Windows machines. >> Others have made similar offers. > > I'll similarly encourage them to take action, then. Do you happen to > remember who? Every few months this thread seems to pop up and then fizzles when people realize the level of work and attention involved (<- he says cynically) in exploiting the offer of resources; I hope that anyone interested in offering resources will pop their head up again to look around. > I hope everyone is on board with the idea of fixing bugs in CPython, > either in the actual implementation of features or in the tests for > those features. That being the case, the discussion of whether or not > the PSF should try to fund such a task is perhaps best discussed on the > PSF members list. Sure. --titus -- C. Titus Brown, ctb at msu.edu From p.f.moore at gmail.com Fri Oct 30 20:46:34 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Oct 2009 19:46:34 +0000 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <20091030165615.GB5132@idyll.org> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> <79990c6b0910300949ybc11b29q11900f08a79cdb9e@mail.gmail.com> <20091030165615.GB5132@idyll.org> Message-ID: <79990c6b0910301246p60dcd5e5s4c1e7b9c0f3662db@mail.gmail.com> 2009/10/30 C. Titus Brown : >> As a counter-offer: Given remote access to however many Windows VMs >> you want to provide, I'll get them up and running with buildslaves on >> them. If that requires software such as Visual Studio, I have copies >> via the MSDN licenses that I am happy to provide. > > I, too, have MSDN licenses, and I have functioning build environments > on all of the VMs (I think -- I've only tested Win XP currently: > > http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8276 OK, so I guess it's just setting the buildbot stuff up. > I also have an OS X 10.5 machine that I can let you into through a firewall; > it's building Python 2.7 quite nicely: > > http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8229 Sorry, I've no experience with OS X at all. >> Once things are up and running, I'll be prepared to do basic care and >> feeding of the buildslave, but as my time is limited, it would be nice >> if others would pitch in to help. > > I would be somewhat unhappy about giving more than three or four people > admin access, but am prepared to lie back and think of England. Greetings from England... :-) I doubt it'll be a huge issue, I just didn't want to end up doing nothing more than delivering 5 more red boxes on the buildbot status page. We can see how it goes. I just think that maintaining the buildbots as more of a community effort means that there's a better chance of issues being fixed quickly. Paul. From rdmurray at bitdance.com Fri Oct 30 20:51:01 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Oct 2009 15:51:01 -0400 (EDT) Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: <4AEAAA8B.3000007@v.loewis.de> References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4AEAAA8B.3000007@v.loewis.de> Message-ID: On Fri, 30 Oct 2009 at 09:57, "Martin v. L?wis" wrote: >> But the real reason for having a buildbot category (or at least a keyword) >> would be to be able to tag all bugs that are currently making buildbots >> fail that are _not_ the result of a recent checkin. This would make >> the task of finding the bugs that need to be cleaned up to stabilize >> the buildbot fleet easier. I'm currently aware of issues 4970, 3892, >> and 6462 in this category, and there are a few more that we can/will file >> if we continue to pay attention to the failure reports now arriving on >> the irc channel. > > That's convincing; I've created a "buildbot" keyword. I gave it the > description > > "indicates that tests fail on a buildslave for uncertain reasons" > > If that is indeed the intended purpose of this classification, please > keep it in mind when assigning the tag. If I misunderstood the purpose > of the keyword, please provide a better description. How about: "indicates that related test failures are causing buildbot instability" My thought is that sometimes we more-or-less know the reasons for the failure, but for one reason or another we can't fix it immediately, and I'd like to keep such a bug visible when looking at buildbot related issues. IMO it would be no bad thing for this tag to be applied to any issue that is created as a result of an observed test failure on a buildbot. Such an issue should only get created if the person who did the checkin that caused it can't reproduce the problem themselves (ie: they ran the test suite and on their platform it was clean). Now, we know that in practice some bugs show up on buildbots because a committer forgot to run the test suite prior to check in (we all make mistakes), but if such a bug gets tagged 'buildbot' I think that's fine, since it will still be affecting the stability of the buildbots. --David From rdmurray at bitdance.com Fri Oct 30 21:08:14 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Oct 2009 16:08:14 -0400 (EDT) Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: <79990c6b0910301246p60dcd5e5s4c1e7b9c0f3662db@mail.gmail.com> References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <20091030163128.GA30191@idyll.org> <79990c6b0910300949ybc11b29q11900f08a79cdb9e@mail.gmail.com> <20091030165615.GB5132@idyll.org> <79990c6b0910301246p60dcd5e5s4c1e7b9c0f3662db@mail.gmail.com> Message-ID: On Fri, 30 Oct 2009 at 19:46, Paul Moore wrote: > 2009/10/30 C. Titus Brown : >>> Once things are up and running, I'll be prepared to do basic care and >>> feeding of the buildslave, but as my time is limited, it would be nice >>> if others would pitch in to help. >> >> I would be somewhat unhappy about giving more than three or four people >> admin access, but am prepared to lie back and think of England. > > Greetings from England... :-) > > I doubt it'll be a huge issue, I just didn't want to end up doing > nothing more than delivering 5 more red boxes on the buildbot status > page. We can see how it goes. I just think that maintaining the > buildbots as more of a community effort means that there's a better > chance of issues being fixed quickly. I'd be happy to help with keeping these (or any other) buildbots running, but I'm not much of a Windows geek (I can work with it, but I know a _lot_ more about Linux). Same goes for OS/X, though since that is unix based I'm a little more comfortable with it. I guess what I'm saying is, if you don't get responses from more Windows-savvy developers, then let me know and I'll be glad to help. That said, the idea of EC2 buildslaves seems pretty attractive... --David From amk at amk.ca Fri Oct 30 21:27:22 2009 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 30 Oct 2009 16:27:22 -0400 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: References: <200910231132.45504.w.richert@gmx.net> <7b0045ed0910271033h69fcc06aybbff0fa3192bb060@mail.gmail.com> <200910301358.17752.steve@pearwood.info> Message-ID: <20091030202722.GA23370@amk-desktop.matrixgroup.net> On Fri, Oct 30, 2009 at 09:37:36PM +0100, Georg Brandl wrote: > I don't like this. It gives a set object a hidden state, something that > AFAICS no other builtin has. Iterating over an iterable is what iterators > are for. It also makes the object thread-unsafe; there's no way for two threads to iterate over it at the same time. It's a terrible idea to introduce new things that won't work under threaded usage. --amk From solipsis at pitrou.net Fri Oct 30 23:04:12 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Oct 2009 22:04:12 +0000 (UTC) Subject: [Python-Dev] "Buildbot" category on the tracker References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> Message-ID: twistedmatrix.com> writes: > > Is your idea that this would be for tracking issues with the *bots* > themselves? That is, not just for tracking cases where some test method > fails on a particular bot, but for tracking cases where, say, a bot's > host has run out of disk space and cannot run the tests at all? Well the general situation would be slightly easier to appreciate if there was a public medium where buildbot info was exchanged, announcements done, and problems tracked. Some kind of tracker, tracker keyword, mailing-list, or anything else. Regards Antoine. From martin at v.loewis.de Fri Oct 30 23:36:32 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 30 Oct 2009 23:36:32 +0100 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> Message-ID: <4AEB6A70.7090007@v.loewis.de> > Well the general situation would be slightly easier to appreciate if there was a > public medium where buildbot info was exchanged, announcements done, and > problems tracked. Some kind of tracker, tracker keyword, mailing-list, or > anything else. As for the tracker keyword - I created one (in case you didn't notice). As for exchanging info: if you talk about the specific Python buildbot installation (instead of info about buildbot, the software), then python-dev is the proper place to exchange info about it. Regards, Martin From martin at v.loewis.de Fri Oct 30 23:43:18 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 30 Oct 2009 23:43:18 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> Message-ID: <4AEB6C06.3070907@v.loewis.de> > Since I've never used any such service ("cloud"-based VMs), I'm not sure > what the downsides would be. But it seems to be that it would be at least > worth trying. Not sure whether it's still relevant after the offers of individually donated hardware. However, if you want to look into this, feel free to set up EC2 slaves. When it comes to the point of actually having to pay money, please send a request to psf-board at python.org (make sure you don't pay anything until the request is approved). Exact processing would have to be decided, then, traditionally, it would be most simple if you could invoice the PSF (IIUC). Giving the PSF as the billing address would probably also work (check with the treasurer). Regards, Martin From steve at pearwood.info Sat Oct 31 02:29:28 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 31 Oct 2009 12:29:28 +1100 Subject: [Python-Dev] =?iso-8859-1?q?Retrieve_an_arbitrary_element_from_a_?= =?iso-8859-1?q?set=09withoutremoving_it?= In-Reply-To: <20091030202722.GA23370@amk-desktop.matrixgroup.net> References: <200910231132.45504.w.richert@gmx.net> <20091030202722.GA23370@amk-desktop.matrixgroup.net> Message-ID: <200910311229.29250.steve@pearwood.info> On Sat, 31 Oct 2009 07:27:22 am A.M. Kuchling wrote: > On Fri, Oct 30, 2009 at 09:37:36PM +0100, Georg Brandl wrote: > > I don't like this. It gives a set object a hidden state, something > > that AFAICS no other builtin has. All objects have a reference count field which is hidden from Python code. The C API for objects has a flags field which specifies whether objects are read-only or read/write from Python code. As of Python 2.6, type objects have an internal method cache. C code can clear it with PyType_ClearCache(), Python codes can't even see it. Lists and dicts pre-allocate extra space, and record hidden state of how much of the space is actually in use. Sets may do the same. File objects may use internal buffers, with all the state that implies. > > Iterating over an iterable is > > what iterators are for. set.get(), or set.pick() as Wikipedia calls it, isn't for iterating over sets. It is for getting an arbitrary element from the set. If the requirement that get/pick() cycles through the sets elements is the killer objection to this proposal, I'd be willing to drop it. I thought that was part of the OP's request, but apparently it isn't. I would argue that it's hardly "arbitrary" if you get the same element every time you call the method, but if others are happy with that behaviour, I'm not going to stand in the way. > It also makes the object thread-unsafe; there's no way for two > threads to iterate over it at the same time. It's a terrible idea to > introduce new things that won't work under threaded usage. I would agree if the purpose of get/pick() was to iterate over the elements of the set, but that's not the purpose. The purpose is to return an arbitrary item each time it is called. If two threads get the same element, that's not a problem; if one thread misses an element because another thread grabbed it first, that's not a problem either. If people prefer a random element instead, I have no problem with that -- personally I think that's overkill, but maybe that's just me. -- Steven D'Aprano From benjamin at python.org Sat Oct 31 02:38:17 2009 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 30 Oct 2009 20:38:17 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: <200910311229.29250.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <20091030202722.GA23370@amk-desktop.matrixgroup.net> <200910311229.29250.steve@pearwood.info> Message-ID: <1afaf6160910301838u37a12582h2e0a8ece2b3e6a57@mail.gmail.com> 2009/10/30 Steven D'Aprano : > On Sat, 31 Oct 2009 07:27:22 am A.M. Kuchling wrote: >> On Fri, Oct 30, 2009 at 09:37:36PM +0100, Georg Brandl wrote: >> > I don't like this. ?It gives a set object a hidden state, something >> > that AFAICS no other builtin has. > > All objects have a reference count field which is hidden from Python > code. The C API for objects has a flags field which specifies whether > objects are read-only or read/write from Python code. > > As of Python 2.6, type objects have an internal method cache. C code can > clear it with PyType_ClearCache(), Python codes can't even see it. > > Lists and dicts pre-allocate extra space, and record hidden state of how > much of the space is actually in use. Sets may do the same. File > objects may use internal buffers, with all the state that implies. Those are all examples of states which are implementation details. The proposed get() semantics would require that allow implementations keep this state around. -- Regards, Benjamin From chris at subtlety.com Sat Oct 31 02:43:31 2009 From: chris at subtlety.com (Chris Bergstresser) Date: Fri, 30 Oct 2009 20:43:31 -0500 Subject: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it In-Reply-To: <200910311229.29250.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <20091030202722.GA23370@amk-desktop.matrixgroup.net> <200910311229.29250.steve@pearwood.info> Message-ID: <7b0045ed0910301843y7059d316s8a851142b78b7aeb@mail.gmail.com> On Fri, Oct 30, 2009 at 8:29 PM, Steven D'Aprano wrote: >> > Iterating over an iterable is >> > what iterators are for. > > set.get(), or set.pick() as Wikipedia calls it, isn't for iterating over > sets. It is for getting an arbitrary element from the set. > > If the requirement that get/pick() cycles through the sets elements is > the killer objection to this proposal, I'd be willing to drop it. I > thought that was part of the OP's request, but apparently it isn't. I > would argue that it's hardly "arbitrary" if you get the same element > every time you call the method, but if others are happy with that > behaviour, I'm not going to stand in the way. It's arbitrary in the sense that the API doesn't make any assurances which item the caller will get, not that it's arbitrary for any particular * implementation*. > The purpose is to > return an arbitrary item each time it is called. If two threads get the > same element, that's not a problem; if one thread misses an element > because another thread grabbed it first, that's not a problem either. > If people prefer a random element instead, I have no problem with > that -- personally I think that's overkill, but maybe that's just me. I also think returning a random one is overkill, in the base set. And I'd have the base implementation do the simplest thing possible: return a fixed element (either the first returned when iterated over, or the last stored, or whatever's easiest based on the internals). For me, leaving the choice of *which* element to return on each call is a feature. It allows subclasses to change the behavior to support other use cases, like a random or round-robin behavior. -- Chris From stephen at xemacs.org Sat Oct 31 03:42:46 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 31 Oct 2009 11:42:46 +0900 Subject: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it In-Reply-To: <200910310211.33590.steve@pearwood.info> References: <200910231132.45504.w.richert@gmx.net> <200910301358.17752.steve@pearwood.info> <2585AF2DFDDE4C61B3596633F5001945@RaymondLaptop1> <200910310211.33590.steve@pearwood.info> Message-ID: <874opgi8rd.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > The usual technique people tend to come up with is: > > x = s.pop() > s.add(x) > > Which strikes many people (including myself) as inelegant. Surely "get > an element" is a more fundamental operation than "get an element and > remove it"? Not in a literal urn or bag. See "sampling with replacement" in any statistics text. Note that in order to "have your cake and eat it too" there's an implicit copy operation (although in Python it will be a cheap ref copy rather than an expensive object copy). I'm afraid that the various conflicting intuitions here are all correct. That is going to make design impossible without some compelling use cases. > > What's wrong with using next()? That is what it's for. > > You can't call next() on a set. You have to call next(iter(set)). > [...] > Unless I missed something, this is so unobvious that nobody has > suggested it in the thread yet. I've seen it at least twice (Nick suggested it IIRC), although that may have been on Python-Ideas (which is where this thread belongs IMHO). > If folks prefer a different name, by all means suggest it. I like > the name suggested by Wikipedia, "pick". "Pick" has a connotation of removal in many contexts: "Pick a card, any card". "Pick it up off the floor" (it's not in the set of things on the floor any more). Like get, to me it has a flavor of "according to some criterion": "a band of picked men". I would expect a pick or get operation to take an optional predicate. But then TOOWTDI is for x in container: if predicate(x): break else: x = default_x # or perhaps raise in cases where there # theoretically should be an element > My reasoning for the given behaviour is as follows: > > * If you want the same element twice from a set, Nobody is asking for that AFAICS. The use case I have so far observed people to have in mind is a cast from an equivalence class to a representative member. They don't care whether the member is the same or not. If they wanted iterator behavior, getting it would not be a problem. next(iter(foo_set)) is TOOWTDI. If they wanted cyclical behavior, itertools.cycle. If they wanted random behavior, make a list and choose from it. In one leading subcase, the equivalence class is a singleton. In this use case what people really want, I suspect, is behavior like Python strings and characters: a singleton set should provide the same operations as its unique element and vice versa. > So having get() return the same value each time is not useful or > necessary. Which is not the point. The point is that having get return a different value if possible is not useful or necessary in at least some of the leading use cases. OTOH, we don't know enough about "weird" use cases where having a different value returned is important to decide how that should be done, where "weird" means "outside of the list of already Obvious Ways". > No. Since sets are unordered, there's no way to seek to a specific > element. I think people will realize that in fact *these* sets are ordered and they will demand a seek function for various practical purposes. > > Sorry for so many questions, but I honestly think there are too many > > unresolved design issues. I have to agree with Raymond. From barry at barrys-emacs.org Sat Oct 31 11:07:12 2009 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 31 Oct 2009 10:07:12 +0000 Subject: [Python-Dev] Add const to python API - issue 6952 In-Reply-To: References: <4ADD2FE9.2010807@v.loewis.de> <74D1336F-AB87-431F-895C-885EE344CEBD@barrys-emacs.org> <4ADE98E2.3080707@v.loewis.de> Message-ID: <38B5518D-20C5-438E-8300-D0E92066B391@barrys-emacs.org> On 22 Oct 2009, at 20:31, Barry Scott wrote: > > On 21 Oct 2009, at 06:15, Martin v. L?wis wrote: > >>> I suggest following POSIX's lead and omitted the const in these >>> cases. > > Ah... Yes I see this in strstr for example. > >> >> Thanks, that sounds reasonable. > > > This is basically what I have done in the patch I hope. > I have updated the patch to include a comment explaining > why output param is char ** and the need for a cast. Have I address the issues sufficiently for the patch to be accepted? Should I go ahead a create the python 3 version of the patch? Barry From martin at v.loewis.de Sat Oct 31 15:57:31 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 31 Oct 2009 15:57:31 +0100 Subject: [Python-Dev] "Buildbot" category on the tracker In-Reply-To: References: <20091029230443.15635.1202637346.divmod.xquotient.10@localhost.localdomain> <4222a8490910291641jaa66329uf8ff59f0e13c708b@mail.gmail.com> <4AEAAA8B.3000007@v.loewis.de> Message-ID: <4AEC505B.8040701@v.loewis.de> > How about: > "indicates that related test failures are causing buildbot > instability" Ok, changed! Martin From martin at v.loewis.de Sat Oct 31 19:24:20 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 31 Oct 2009 19:24:20 +0100 Subject: [Python-Dev] Possible language summit topic: buildbots In-Reply-To: References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025135018.11571.560646061.divmod.xquotient.1516@localhost.localdomain> <4AE4A1C9.5010002@v.loewis.de> <20091026005815.31526.130687717.divmod.xquotient.88@localhost.localdomain> <20091026015650.31526.600593069.divmod.xquotient.94@localhost.localdomain> <4AE55E39.20000@v.loewis.de> Message-ID: <4AEC80D4.5090004@v.loewis.de> Antoine Pitrou wrote: > Martin v. L?wis v.loewis.de> writes: >> It's not really reproducible. I think it sometimes happens when I >> restart the master; sometimes, some clients fail to reconnect >> (properly). > > Another common problem is that some buildbot fails in the middle of the test > suite, with the following kind of message: > > command timed out: 1800 seconds without output, killing pid 12325 > process killed by signal 9 > program finished with exit code -1 > elapsedTime=10910.362981 > > See for example : > http://www.python.org/dev/buildbot/trunk.stable/builders/ia64%20Ubuntu%20trunk/builds/73/steps/test/logs/stdio > > (notice, by the way, the elapsed time (10910s, that is, close to 3 hours...)) That's not really a challenge to the buildbot operator, though - the buildbot will continue just fine afterwards, right? For some reason, the test suite stopped producing output, and eventually, the buildbot decided to kill the build process. Most likely, the machine ran severely out of memory, so everything stopped working. Regards, Martin From solipsis at pitrou.net Sat Oct 31 21:13:57 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 31 Oct 2009 20:13:57 +0000 (UTC) Subject: [Python-Dev] EC2 buildslaves References: <5c6f2a5d0910250154nfcdabafyc051389c956ef258@mail.gmail.com> <4AE41E9E.4050706@v.loewis.de> <20091025140512.11571.1513702964.divmod.xquotient.1543@localhost.localdomain> <4AEB6C06.3070907@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > Not sure whether it's still relevant after the offers of individually > donated hardware. We'll see, indeed. > However, if you want to look into this, feel free to > set up EC2 slaves. I only know to setup mainstream Linux distros though (Debian- or Redhat-lookalikes :-)). I've just played a bit and, after the hassle of juggling with a bunch of different keys and credentials, setting up an instance and saving an image for future use is quite easy. Regards Antoine. From brett at python.org Sat Oct 31 22:47:43 2009 From: brett at python.org (Brett Cannon) Date: Sat, 31 Oct 2009 14:47:43 -0700 Subject: [Python-Dev] thanks to everyone cleaning up the tests Message-ID: Just wanted to publicly thank everyone who has been causing all the checkins to fix and stabilize the test suite (I think it's mostly Antoine and Mark, but I could be missing somebody; I'm under a deadline so only have marginal higher brain functionality). -Brett From solipsis at pitrou.net Sat Oct 31 23:03:55 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 31 Oct 2009 22:03:55 +0000 (UTC) Subject: [Python-Dev] thanks to everyone cleaning up the tests References: Message-ID: Brett Cannon python.org> writes: > > Just wanted to publicly thank everyone who has been causing all the > checkins to fix and stabilize the test suite (I think it's mostly > Antoine and Mark, but I could be missing somebody; I'm under a > deadline so only have marginal higher brain functionality). A lot of work was done by David (RDM). Thanks ! Antoine.