From ziade.tarek at gmail.com Sat Jan 10 11:48:37 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 10 Jan 2009 11:48:37 +0100 Subject: [Python-ideas] Make getpass.getpass use system password keyrings ? Message-ID: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com> Hello, What about having an option in getpass to store and reuse passwords in system keyrings ? getpass(prompt[, stream]) would become: getpass(prompt[, stream, keyring]) where keyring would be a callable that can be use to retrieve the password from a keyring system and store it the first time. The getpass module could provide some keyring support for: - ssh-agent under Linux - keychain under Mac OS X - ... And let the developers use their own keyring system by providing a callable. Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From gerald.britton at gmail.com Sat Jan 10 16:35:49 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Sat, 10 Jan 2009 10:35:49 -0500 Subject: [Python-ideas] Add "while" clauses to generator expressions Message-ID: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> I've been using Python generators for a while now. e.g. a=(i for i in range(10)) a.next() a.next() ...etc. I also find the "if" clause handy: a = (i for i in range(10) if i%2==0) (I know that range(0,12,2) will do the same thing, but it's the idea I like, especially for more complex predicates.) I would like to know if anyone has thought of adding a "while" clause as well, like this: a = (i for i in range(100) while i <=50) (Again, this could be done with range(0,51) but then the predicate can be more complicated.) Why would this be helpful? Consider the "in ..." part of the generator. You could be referring to something that is ordered -- sorted names for example. Then you might want to stop your iterator when you reach "Morgan" so you would like to write: name = (n for n in names while n <= "Morgan") name.next() ...etc... Of course, name = (n for n in names if n <= "Morgan") will work, but it will look at every item in "names." Since "names" is sorted, this is a waste of time. Imagine you want to stop at "Baker". Your "if" clause will look at and discard most names in the list, assuming a normal distribution of English names. Now, you could do the same thing with a generator function: def leMorgan(names): i = 0 while names[i] <= "Morgan": yield names[i] i+=1 and use it like this: name=leMorgan(names) name.next() ...etc... but I think that adding a while clause to the generator expression is simpler and clearer (and it keeps it all in one place!) I know that this is functionally equivalent to the takewhile function in itertools and my motivation is the same. I just think that this could be done nicely within the context of the existing syntax. This is also convenient when the "in ..." clause refers to another (possibly infinite) generator. For a simple example, suppose I want to run through some natural numbers. I can write an infinte generator function like this: def genN(n=0): while 1: yield n n+=1 Then, I might use this in another generator: p = (n for n in genN() if prime(n) while n <= 100) to get the prime numbers under 100 (assuming I have a predicate "prime" that works as one would hope). Of course you could do this with range(101) instead of genN, but this is just an example to demonstrate the idea. Without the "while" clause, this will not work: p = (n for n in genN() if prime(n) if n <= 100) This will actually NEVER terminate, since EVERY item in genN() (which is an infinite generator) will be tested for <= 100. So...What do others think? Is this a loony idea? Is there a better way? Also, can anyone think of a similarly syntax to replicate the dropwhile function? From grosser.meister.morti at gmx.net Sat Jan 10 18:06:08 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sat, 10 Jan 2009 18:06:08 +0100 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> Message-ID: <4968D580.4030904@gmx.net> Gerald Britton schrieb: > I've been using Python generators for a while now. e.g. > > a=(i for i in range(10)) > > a.next() > a.next() > ...etc. > > I also find the "if" clause handy: > > a = (i for i in range(10) if i%2==0) > > (I know that range(0,12,2) will do the same thing, but it's the idea I > like, especially for more complex predicates.) > > I would like to know if anyone has thought of adding a "while" clause > as well, like this: > > a = (i for i in range(100) while i <=50) > from itertools takewhile a = takewhile(lambda i: i <= 50, xrange(100)) From grosser.meister.morti at gmx.net Sat Jan 10 18:07:43 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sat, 10 Jan 2009 18:07:43 +0100 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <4968D580.4030904@gmx.net> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> <4968D580.4030904@gmx.net> Message-ID: <4968D5DF.7070202@gmx.net> Mathias Panzenb?ck schrieb: > Gerald Britton schrieb: >> I've been using Python generators for a while now. e.g. >> >> a=(i for i in range(10)) >> >> a.next() >> a.next() >> ...etc. >> >> I also find the "if" clause handy: >> >> a = (i for i in range(10) if i%2==0) >> >> (I know that range(0,12,2) will do the same thing, but it's the idea I >> like, especially for more complex predicates.) >> >> I would like to know if anyone has thought of adding a "while" clause >> as well, like this: >> >> a = (i for i in range(100) while i <=50) >> > > from itertools takewhile > a = takewhile(lambda i: i <= 50, xrange(100)) typo: from itertools import takewhile a = takewhile(lambda i: i <= 50, xrange(100)) From gerald.britton at gmail.com Sat Jan 10 20:06:02 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Sat, 10 Jan 2009 14:06:02 -0500 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <4968D5DF.7070202@gmx.net> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> <4968D580.4030904@gmx.net> <4968D5DF.7070202@gmx.net> Message-ID: <5d1a32000901101106x31cd15e7v8d1a0a352baec52e@mail.gmail.com> Yes of course. However I am advocating adding this to the syntax rather than using the takewhile function, thus making it part of the generator expression proper and avoiding the function call and module include. From the little I know about the implementation of the corresponding "if" clause, this should be relatively easy. On 1/10/09, Mathias Panzenb?ck wrote: > Mathias Panzenb?ck schrieb: >> Gerald Britton schrieb: >>> I've been using Python generators for a while now. e.g. >>> >>> a=(i for i in range(10)) >>> >>> a.next() >>> a.next() >>> ...etc. >>> >>> I also find the "if" clause handy: >>> >>> a = (i for i in range(10) if i%2==0) >>> >>> (I know that range(0,12,2) will do the same thing, but it's the idea I >>> like, especially for more complex predicates.) >>> >>> I would like to know if anyone has thought of adding a "while" clause >>> as well, like this: >>> >>> a = (i for i in range(100) while i <=50) >>> >> >> from itertools takewhile >> a = takewhile(lambda i: i <= 50, xrange(100)) > > typo: > from itertools import takewhile > a = takewhile(lambda i: i <= 50, xrange(100)) > -- Sent from my mobile device From steven.bethard at gmail.com Sat Jan 10 20:33:25 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 10 Jan 2009 11:33:25 -0800 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> Message-ID: On Sat, Jan 10, 2009 at 7:35 AM, Gerald Britton wrote: > I would like to know if anyone has thought of adding a "while" clause > as well, like this: > > a = (i for i in range(100) while i <=50) I think this could end up being confusing. Current generator expressions turn into an equivalent generator function by simply indenting the clauses and adding a yield, for example: (i for i in range(100) if i % 2 == 0) is equivalent to: def gen(): for i in range(100): if i % 2 == 0: yield i Now you're proposing syntax that would no longer work like this. Taking your example: (i for i in range(100) while i <= 50) I would expect this to mean: def gen(): for i in range(100): while i <= 50: yield i In short, -1. You're proposing to use an existing keyword in a new way that doesn't match how generator expressions are evaluated. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From gerald.britton at gmail.com Sat Jan 10 23:34:24 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Sat, 10 Jan 2009 17:34:24 -0500 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> Message-ID: <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com> I guess I don't think it would be confusing. On the contrary, I believe that the expressions would read naturally and be a nice simplification. Of course it won't work just like "if" but that is just the point! I can (and do) accomplish the same thing with "takewhile", but if the same thing can be sone with a little addition to the generator expression, why not do it? On 1/10/09, Steven Bethard wrote: > On Sat, Jan 10, 2009 at 7:35 AM, Gerald Britton > wrote: >> I would like to know if anyone has thought of adding a "while" clause >> as well, like this: >> >> a = (i for i in range(100) while i <=50) > > I think this could end up being confusing. Current generator > expressions turn into an equivalent generator function by simply > indenting the clauses and adding a yield, for example: > > (i for i in range(100) if i % 2 == 0) > > is equivalent to: > > def gen(): > for i in range(100): > if i % 2 == 0: > yield i > > Now you're proposing syntax that would no longer work like this. > Taking your example: > > (i for i in range(100) while i <= 50) > > I would expect this to mean: > > def gen(): > for i in range(100): > while i <= 50: > yield i > > In short, -1. You're proposing to use an existing keyword in a new way > that doesn't match how generator expressions are evaluated. > > Steve > -- > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > tiny blip on the distant coast of sanity. > --- Bucky Katt, Get Fuzzy > -- Sent from my mobile device From steven.bethard at gmail.com Sat Jan 10 23:45:59 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 10 Jan 2009 14:45:59 -0800 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com> Message-ID: [Fixing the top-posting. Note that only Guido is allowed to top-post around here. ;-)] On 1/10/09, Steven Bethard wrote: > I think this could end up being confusing. Current generator > expressions turn into an equivalent generator function by simply > indenting the clauses and adding a yield, for example: > > (i for i in range(100) if i % 2 == 0) > > is equivalent to: > > def gen(): > for i in range(100): > if i % 2 == 0: > yield i > > Now you're proposing syntax that would no longer work like this. > Taking your example: > > (i for i in range(100) while i <= 50) > > I would expect this to mean: > > def gen(): > for i in range(100): > while i <= 50: > yield i On Sat, Jan 10, 2009 at 2:34 PM, Gerald Britton wrote: > I guess I don't think it would be confusing. On the contrary, I > believe that the expressions would read naturally and be a nice > simplification. Of course it won't work just like "if" but that is > just the point! I can (and do) accomplish the same thing with > "takewhile", but if the same thing can be sone with a little addition > to the generator expression, why not do it? I'm probably just repeating myself here, but the reason not to do it is that the current generator expressions translate almost directly into the corresponding generator statements. Using "while" in the way you've suggested breaks this symmetry, and would make Python harder to learn. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From gerald.britton at gmail.com Sun Jan 11 00:20:40 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Sat, 10 Jan 2009 18:20:40 -0500 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com> Message-ID: <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com> Hmmm. I can't really see what you are saying. Your example doesn't quite get the intention. The "while" clause in the example would translate like this: def gen(): for i in range(100): if i <= 50: yield i else: break Also, I think that for a new python user, (n for n in range(100) while n*n < 50) is easier to understand and use than: (n for n in takewhile(lambda n:n*n < 50, range(100))) My proposed version is shorter (less chance for typos), has one less set of parentheses (always a good thing) and reads naturally. Also, it is directly analogous to: (n for n in range(100) if n*n < 50) except that the "while" version stops when n reaches 8. The "if" version doesn't stop until n reaches 99. On Sat, Jan 10, 2009 at 5:45 PM, Steven Bethard wrote: > [Fixing the top-posting. Note that only Guido is allowed to top-post > around here. ;-)] > > On 1/10/09, Steven Bethard wrote: >> I think this could end up being confusing. Current generator >> expressions turn into an equivalent generator function by simply >> indenting the clauses and adding a yield, for example: >> >> (i for i in range(100) if i % 2 == 0) >> >> is equivalent to: >> >> def gen(): >> for i in range(100): >> if i % 2 == 0: >> yield i >> >> Now you're proposing syntax that would no longer work like this. >> Taking your example: >> >> (i for i in range(100) while i <= 50) >> >> I would expect this to mean: >> >> def gen(): >> for i in range(100): >> while i <= 50: >> yield i > > On Sat, Jan 10, 2009 at 2:34 PM, Gerald Britton > wrote: >> I guess I don't think it would be confusing. On the contrary, I >> believe that the expressions would read naturally and be a nice >> simplification. Of course it won't work just like "if" but that is >> just the point! I can (and do) accomplish the same thing with >> "takewhile", but if the same thing can be sone with a little addition >> to the generator expression, why not do it? > > I'm probably just repeating myself here, but the reason not to do it > is that the current generator expressions translate almost directly > into the corresponding generator statements. Using "while" in the way > you've suggested breaks this symmetry, and would make Python harder to > learn. > > Steve > -- > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > tiny blip on the distant coast of sanity. > --- Bucky Katt, Get Fuzzy > From grosser.meister.morti at gmx.net Sun Jan 11 00:44:02 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sun, 11 Jan 2009 00:44:02 +0100 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com> <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com> Message-ID: <496932C2.2070507@gmx.net> I've got an idea: x = (n for n in xrange(100) if n < 22 else break) but I don't mean that seriously (giving it -1). :P -panzi From g.brandl at gmx.net Sun Jan 11 10:14:41 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 11 Jan 2009 10:14:41 +0100 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com> <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com> Message-ID: Gerald Britton schrieb: > Hmmm. I can't really see what you are saying. Your example doesn't > quite get the intention. The "while" clause in the example would > translate like this: > > def gen(): > for i in range(100): > if i <= 50: > yield i > else: > break > > Also, I think that for a new python user, > > (n for n in range(100) while n*n < 50) > > is easier to understand and use than: > > (n for n in takewhile(lambda n:n*n < 50, range(100))) > > My proposed version is shorter (less chance for typos), has one less > set of parentheses (always a good thing) and reads naturally. The other is needlessly complicated, though. takewhile(lambda n: n*n < 50, range(100)) is just as fine, and only 3 characters longer than the proposed while-exp. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From bborcic at gmail.com Sun Jan 11 13:05:18 2009 From: bborcic at gmail.com (Boris Borcic) Date: Sun, 11 Jan 2009 13:05:18 +0100 Subject: [Python-ideas] Add "while" clauses to generator expressions In-Reply-To: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com> Message-ID: Gerald Britton wrote: > as well, like this: > > a = (i for i in range(100) while i <=50) there was a longish thread on the subject back in august '08, where I pointed out > you can do something like > >>>> def yet(b) : > if b : raise StopIteration > > >>>> list(x for x in range(0,10) if not yet(x>4)) > [0, 1, 2, 3, 4] but I was then answered that this "worked by accident" which apparently meant that 3.0 pedagogical doctrine wants [x for x...] to be strictly equivalent to list(x for x...), while this equivalence in fact doesn't extend to similar uses of StopIteration. Cheers, BB From greg at krypto.org Mon Jan 12 08:59:08 2009 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 11 Jan 2009 23:59:08 -0800 Subject: [Python-ideas] Make getpass.getpass use system password keyrings ? In-Reply-To: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com> References: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com> Message-ID: <52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com> On Sat, Jan 10, 2009 at 2:48 AM, Tarek Ziad? wrote: > Hello, > > What about having an option in getpass to store and reuse passwords in > system keyrings ? > > getpass(prompt[, stream]) > > would become: > > getpass(prompt[, stream, keyring]) > > where keyring would be a callable that can be use to retrieve the > password from a keyring system > and store it the first time. > > The getpass module could provide some keyring support for: > > - ssh-agent under Linux > - keychain under Mac OS X > - ... > > And let the developers use their own keyring system by providing a > callable. > > Regards > Tarek That sounds like a nice idea. I suggest starting by writing a library keyring get/put functions for the various keyrings out there and releasing it on its own. (ssh-agent, gnome and kde keychains, mac os x keychain, does windows have an api for this?, others?, pwsafe?) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Mon Jan 12 15:53:27 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 12 Jan 2009 15:53:27 +0100 Subject: [Python-ideas] Make getpass.getpass use system password keyrings ? In-Reply-To: <52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com> References: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com> <52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com> Message-ID: <94bdd2610901120653g27013a52sfbb7daa7dba04ee5@mail.gmail.com> On Mon, Jan 12, 2009 at 8:59 AM, Gregory P. Smith wrote: > > On Sat, Jan 10, 2009 at 2:48 AM, Tarek Ziad? wrote: >> >> Hello, >> >> What about having an option in getpass to store and reuse passwords in >> system keyrings ? >> >> getpass(prompt[, stream]) >> >> would become: >> >> getpass(prompt[, stream, keyring]) >> >> where keyring would be a callable that can be use to retrieve the >> password from a keyring system >> and store it the first time. >> >> The getpass module could provide some keyring support for: >> >> - ssh-agent under Linux >> - keychain under Mac OS X >> - ... >> >> And let the developers use their own keyring system by providing a >> callable. >> >> Regards >> Tarek > > That sounds like a nice idea. I suggest starting by writing a library > keyring get/put functions for the various keyrings out there and releasing > it on its own. Right, sounds like the best approach (My use case is for Distutils 's register/upload commands) > (ssh-agent, gnome and kde keychains, mac os x keychain, does windows have an > api for this?, others?, pwsafe?) I don't know about Windows, I need to digg, I'll set a list of existing keyrings. Cheers Tarek > -gps -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From mail at timgolden.me.uk Mon Jan 12 16:06:31 2009 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 12 Jan 2009 15:06:31 +0000 Subject: [Python-ideas] Make getpass.getpass use system password keyrings ? In-Reply-To: <94bdd2610901120653g27013a52sfbb7daa7dba04ee5@mail.gmail.com> References: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com> <52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com> <94bdd2610901120653g27013a52sfbb7daa7dba04ee5@mail.gmail.com> Message-ID: <496B5C77.1010402@timgolden.me.uk> Tarek Ziad? wrote: > I don't know about Windows, I need to digg, Assuming I understand what the requirement is, you need the authentication functions. Specifically, I think things like CredRead, CredWrite and CredUICmdLinePromptForCredentials would be useful. (getpass may be using these already; I haven't checked). http://msdn.microsoft.com/en-us/library/aa374804(VS.85).aspx http://msdn.microsoft.com/en-us/library/aa375187(VS.85).aspx http://msdn.microsoft.com/en-us/library/aa375171(VS.85).aspx TJG From sturla at molden.no Tue Jan 13 21:07:05 2009 From: sturla at molden.no (Sturla Molden) Date: Tue, 13 Jan 2009 21:07:05 +0100 Subject: [Python-ideas] allow tuples in a with statement? Message-ID: <496CF469.5010605@molden.no> Sometimes multiple arguments in a with statement would be convinient. with open('in.txt','rt'), \ open('out.txt','wt'),\ open('err.txt','wt') \ as fstdin, fstdout, fstderr: But today this has to be written as: with open('in.txt','rt') as fstdin: with open('out.txt','wt') as fstdout: with open('err.txt','wt') as fstderr: with the effect of causing multiple redundant levels of intendation. Regards, Sturla Molden From mail at timgolden.me.uk Tue Jan 13 21:43:32 2009 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 13 Jan 2009 20:43:32 +0000 Subject: [Python-ideas] allow tuples in a with statement? In-Reply-To: <496CF469.5010605@molden.no> References: <496CF469.5010605@molden.no> Message-ID: <496CFCF4.5040009@timgolden.me.uk> Sturla Molden wrote: > Sometimes multiple arguments in a with statement would be convinient. > > with open('in.txt','rt'), \ > open('out.txt','wt'),\ > open('err.txt','wt') \ > as fstdin, fstdout, fstderr: > > But today this has to be written as: > > with open('in.txt','rt') as fstdin: > with open('out.txt','wt') as fstdout: > with open('err.txt','wt') as fstderr: > > with the effect of causing multiple redundant levels of intendation. contextlib.nested ? TJG From fredrik.johansson at gmail.com Tue Jan 13 21:57:12 2009 From: fredrik.johansson at gmail.com (Fredrik Johansson) Date: Tue, 13 Jan 2009 21:57:12 +0100 Subject: [Python-ideas] allow tuples in a with statement? In-Reply-To: <496CF469.5010605@molden.no> References: <496CF469.5010605@molden.no> Message-ID: <3d0cebfb0901131257g6a441cb5u9752c8acf3231817@mail.gmail.com> On Tue, Jan 13, 2009 at 9:07 PM, Sturla Molden wrote: > > Sometimes multiple arguments in a with statement would be convinient. This has occurred to me as well. In sympy, we have considered using the with statement for assumptions on variables. Tuples would provide a very natural syntax for multiple assumptions: with x>0, y>0: simplify(sqrt((x+y)**2)) Fredrik From carl at carlsensei.com Wed Jan 14 08:12:34 2009 From: carl at carlsensei.com (Carl Johnson) Date: Tue, 13 Jan 2009 21:12:34 -1000 Subject: [Python-ideas] With-expression Message-ID: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> From the department of doomed ideas: data = f.read() with open("file.txt") as f Has the idea of a with-expression already been preemptively shot down? I guess the objection is that the above could easily be rewritten as data = read_and_close("file.txt") if one defines the appropriate function. The counter-objection is that why do I need to write a micro-function when the with-statement already does what I want, except it takes up two lines when it would still be clear as one line. But then the counter-counter-objection is that you can write with open("file.txt") as f: data = f.read() today. To which the counter-counter-counter-objection is, yeah, but that looks cluttered and ugly in a way that the first example doesn't... Except the first example is sort of cluttered. I dunno, what do other people think about this? Doomed or super-doomed? -- Carl From pyideas at rebertia.com Wed Jan 14 09:24:33 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Wed, 14 Jan 2009 00:24:33 -0800 Subject: [Python-ideas] With-expression In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> Message-ID: <50697b2c0901140024o21639c93k8d592f4dcbe2f35d@mail.gmail.com> On Tue, Jan 13, 2009 at 11:12 PM, Carl Johnson wrote: > From the department of doomed ideas: > > data = f.read() with open("file.txt") as f > > Has the idea of a with-expression already been preemptively shot down? > > I guess the objection is that the above could easily be rewritten as > > data = read_and_close("file.txt") > > if one defines the appropriate function. The counter-objection is that why > do I need to write a micro-function when the with-statement already does > what I want, except it takes up two lines when it would still be clear as > one line. But then the counter-counter-objection is that you can write > > with open("file.txt") as f: data = f.read() > > today. To which the counter-counter-counter-objection is, yeah, but that > looks cluttered and ugly in a way that the first example doesn't... Except > the first example is sort of cluttered. > > I dunno, what do other people think about this? Doomed or super-doomed? Fairly doomed, IMHO. To be consistent at all, Python would also have to allow the related Perlisms of: x **= 2 while x < y x += y for y in z x.sort() if should_sort which seem pretty gross to me as the control structure gets somewhat hidden. And as you pointed out, it's just plain unnecessary since such code can already be written as a one-liner, with the advantage of less reformatting if it later becomes a multiliner. Also, having a consistent order of block-statement before block-body enhances the uniformity of the language. About the only good examples of block bodies coming before block statements which I can think of offhand are do-while and Haskell's `where` clause, which both seem to me to be sufficiently special cases as to not have bearing here. That's my 2? anyway. Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com From tjreedy at udel.edu Wed Jan 14 12:20:08 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Jan 2009 06:20:08 -0500 Subject: [Python-ideas] With-expression In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> Message-ID: Carl Johnson wrote: > From the department of doomed ideas: > > data = f.read() with open("file.txt") as f > > Has the idea of a with-expression already been preemptively shot down? > > I guess the objection is that the above could easily be rewritten as > > data = read_and_close("file.txt") > > if one defines the appropriate function. The counter-objection is that > why do I need to write a micro-function when the with-statement already > does what I want, except it takes up two lines when it would still be > clear as one line. But then the counter-counter-objection is that you > can write > > with open("file.txt") as f: data = f.read() > > today. To which the counter-counter-counter-objection is, yeah, but that > looks cluttered and ugly in a way that the first example doesn't... > Except the first example is sort of cluttered. > > I dunno, what do other people think about this? Doomed or super-doomed? Twisted and super-doomed. From grosser.meister.morti at gmx.net Wed Jan 14 12:33:02 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Wed, 14 Jan 2009 12:33:02 +0100 Subject: [Python-ideas] With-expression In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> Message-ID: <496DCD6E.30009@gmx.net> strippedLines = [sline for line in lines with line.strip() as sline if sline] or something like that. :P (-1) -panzi Carl Johnson schrieb: > From the department of doomed ideas: > > data = f.read() with open("file.txt") as f > > Has the idea of a with-expression already been preemptively shot down? > > I guess the objection is that the above could easily be rewritten as > > data = read_and_close("file.txt") > > if one defines the appropriate function. The counter-objection is that > why do I need to write a micro-function when the with-statement already > does what I want, except it takes up two lines when it would still be > clear as one line. But then the counter-counter-objection is that you > can write > > with open("file.txt") as f: data = f.read() > > today. To which the counter-counter-counter-objection is, yeah, but that > looks cluttered and ugly in a way that the first example doesn't... > Except the first example is sort of cluttered. > > I dunno, what do other people think about this? Doomed or super-doomed? > > -- Carl From zac256 at gmail.com Sun Jan 18 02:35:48 2009 From: zac256 at gmail.com (Zac Burns) Date: Sat, 17 Jan 2009 17:35:48 -0800 Subject: [Python-ideas] defaultdict factory argument Message-ID: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> I think defaultdict would be more useful if one could initialize it with the option to have the key that is being looked up passed to default_factory. -- Zachary Burns (407)590-4814 Aim - Zac256FL Production Engineer (Digital Overlord) Zindagi Games From pyideas at rebertia.com Sun Jan 18 02:44:44 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Sat, 17 Jan 2009 17:44:44 -0800 Subject: [Python-ideas] defaultdict factory argument In-Reply-To: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> Message-ID: <50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com> On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns wrote: > I think defaultdict would be more useful if one could initialize it > with the option to have the key that is being looked up passed to > default_factory. While I don't disagree in principle, giving an example use-case would probably help. Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com From zac256 at gmail.com Sun Jan 18 03:01:44 2009 From: zac256 at gmail.com (Zac Burns) Date: Sat, 17 Jan 2009 18:01:44 -0800 Subject: [Python-ideas] defaultdict factory argument In-Reply-To: <50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com> References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> <50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com> Message-ID: <333edbe80901171801n7de3f2c1s79e5cf378a24bb6b@mail.gmail.com> For one, it could be used to cache function calls. I'm not sure how well defaultdict and WeakValueDictionary work together (when combined/subclassed), but I can also imagine using a class as the default_factory and using a defaultdict to return a new instance of a class if none yet existed instantiated with the same input parameters or returning the already existing one if one did exist with those parameters. -- Zachary Burns (407)590-4814 Aim - Zac256FL Production Engineer (Digital Overlord) Zindagi Games On Sat, Jan 17, 2009 at 5:44 PM, Chris Rebert wrote: > On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns wrote: >> I think defaultdict would be more useful if one could initialize it >> with the option to have the key that is being looked up passed to >> default_factory. > > While I don't disagree in principle, giving an example use-case would > probably help. > > Cheers, > Chris > > -- > Follow the path of the Iguana... > http://rebertia.com > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From steven.bethard at gmail.com Sun Jan 18 03:32:28 2009 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 17 Jan 2009 18:32:28 -0800 Subject: [Python-ideas] defaultdict factory argument In-Reply-To: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> Message-ID: On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns wrote: > I think defaultdict would be more useful if one could initialize it > with the option to have the key that is being looked up passed to > default_factory. Are you sure you can't get what you want by using __missing__? http://www.python.org/doc/2.5.2/lib/defaultdict-objects.html Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From zac256 at gmail.com Sun Jan 18 06:22:05 2009 From: zac256 at gmail.com (Zac Burns) Date: Sat, 17 Jan 2009 21:22:05 -0800 Subject: [Python-ideas] defaultdict factory argument In-Reply-To: References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> Message-ID: <333edbe80901172122v5675a70cs7c1984331c9fd40f@mail.gmail.com> Ah, excellent. -- Zachary Burns (407)590-4814 Aim - Zac256FL Production Engineer (Digital Overlord) Zindagi Games On Sat, Jan 17, 2009 at 6:32 PM, Steven Bethard wrote: > On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns wrote: >> I think defaultdict would be more useful if one could initialize it >> with the option to have the key that is being looked up passed to >> default_factory. > > Are you sure you can't get what you want by using __missing__? > > http://www.python.org/doc/2.5.2/lib/defaultdict-objects.html > > Steve > -- > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > tiny blip on the distant coast of sanity. > --- Bucky Katt, Get Fuzzy > From tjreedy at udel.edu Sun Jan 18 06:46:06 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 18 Jan 2009 00:46:06 -0500 Subject: [Python-ideas] defaultdict factory argument In-Reply-To: <333edbe80901171801n7de3f2c1s79e5cf378a24bb6b@mail.gmail.com> References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com> <50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com> <333edbe80901171801n7de3f2c1s79e5cf378a24bb6b@mail.gmail.com> Message-ID: Continuing the OP's top posting style.. if not hasattr(d,key): d[key] = return d[key] or variations, including try:...except:.. work for that Zac Burns wrote: > For one, it could be used to cache function calls. >> While I don't disagree in principle, giving an example use-case would >> probably help. From ziade.tarek at gmail.com Sun Jan 18 13:40:40 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 18 Jan 2009 13:40:40 +0100 Subject: [Python-ideas] More things in the standard library equipped with __enter__ and __exit__ Message-ID: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com> Hello, I have entered this yesterday : http://bugs.python.org/issue4972. about the fact that ftplib.FTP could be used in a "with" loop. But I spotted that imaplib.IMAP4/IMAP4_SSL and smtplib.SMTP could be equipped as well, so an __exit__ call takes care of closing or quiting. This change could be done together with some examples in the documentation. If this sounds like a good idea, I can start a patch in #4972, Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ironfroggy at gmail.com Sun Jan 18 14:31:07 2009 From: ironfroggy at gmail.com (Calvin Spealman) Date: Sun, 18 Jan 2009 08:31:07 -0500 Subject: [Python-ideas] More things in the standard library equipped with __enter__ and __exit__ In-Reply-To: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com> References: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com> Message-ID: <76fd5acf0901180531r7cee8373n4fe4bf2be2a2acba@mail.gmail.com> +1 on both On Sun, Jan 18, 2009 at 7:40 AM, Tarek Ziad? wrote: > Hello, > > I have entered this yesterday : http://bugs.python.org/issue4972. > about the fact that ftplib.FTP could be > used in a "with" loop. > > But I spotted that imaplib.IMAP4/IMAP4_SSL and smtplib.SMTP could be > equipped as well, > so an __exit__ call takes care of closing or quiting. > > This change could be done together with some examples in the documentation. > > If this sounds like a good idea, I can start a patch in #4972, > > Regards > Tarek > > -- > Tarek Ziad? | Association AfPy | www.afpy.org > Blog FR | http://programmation-python.org > Blog EN | http://tarekziade.wordpress.com/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From brett at python.org Sun Jan 18 23:32:06 2009 From: brett at python.org (Brett Cannon) Date: Sun, 18 Jan 2009 14:32:06 -0800 Subject: [Python-ideas] More things in the standard library equipped with __enter__ and __exit__ In-Reply-To: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com> References: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com> Message-ID: On Sun, Jan 18, 2009 at 04:40, Tarek Ziad? wrote: > Hello, > > I have entered this yesterday : http://bugs.python.org/issue4972. > about the fact that ftplib.FTP could be > used in a "with" loop. > > But I spotted that imaplib.IMAP4/IMAP4_SSL and smtplib.SMTP could be > equipped as well, > so an __exit__ call takes care of closing or quiting. > > This change could be done together with some examples in the documentation. > > If this sounds like a good idea, I can start a patch in #4972, They should be separate patches, but yes, more context managers are a good thing (we are not calling them "'with' loops" since they don't loop =). -Brett From leif.walsh at gmail.com Sun Jan 18 23:41:15 2009 From: leif.walsh at gmail.com (Leif Walsh) Date: Sun, 18 Jan 2009 17:41:15 -0500 Subject: [Python-ideas] With-expression In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> Message-ID: On Wed, Jan 14, 2009 at 2:12 AM, Carl Johnson wrote: > From the department of doomed ideas: > > data = f.read() with open("file.txt") as f "Who do you think we are, Ruby?" Just kidding, but -1. -- Cheers, Leif From josiah.carlson at gmail.com Tue Jan 20 02:46:27 2009 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Mon, 19 Jan 2009 17:46:27 -0800 Subject: [Python-ideas] With-expression In-Reply-To: References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> Message-ID: On Sun, Jan 18, 2009 at 2:41 PM, Leif Walsh wrote: > On Wed, Jan 14, 2009 at 2:12 AM, Carl Johnson wrote: >> From the department of doomed ideas: >> >> data = f.read() with open("file.txt") as f > > "Who do you think we are, Ruby?" > > Just kidding, but -1. I always preferred: data = open("file.txt").read() Which works all the way back as far as I can remember. And as for: strippedLines = [sline for line in lines with line.strip() as sline if sline] Try: strippedLines = [sline for sline in (line.strip() for line in lines) if sline] The request gets a big fat -1 from me. - Josiah From greg at krypto.org Tue Jan 20 02:47:56 2009 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 19 Jan 2009 17:47:56 -0800 Subject: [Python-ideas] With-expression In-Reply-To: References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> Message-ID: <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com> On Mon, Jan 19, 2009 at 5:46 PM, Josiah Carlson wrote: > On Sun, Jan 18, 2009 at 2:41 PM, Leif Walsh wrote: > > On Wed, Jan 14, 2009 at 2:12 AM, Carl Johnson > wrote: > >> From the department of doomed ideas: > >> > >> data = f.read() with open("file.txt") as f > > > > "Who do you think we are, Ruby?" > > > > Just kidding, but -1. > > I always preferred: > > data = open("file.txt").read() > > Which works all the way back as far as I can remember. > In C Python yes that "works" but it makes absolutely no guarantees about when that file is actually closed. Don't write code that depends on a particular GC behavior. Use with. :) > > And as for: > strippedLines = [sline for line in lines with line.strip() as sline if > sline] > > Try: > strippedLines = [sline for sline in (line.strip() for line in lines) if > sline] > > The request gets a big fat -1 from me. > > - Josiah > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Jan 20 14:00:31 2009 From: sturla at molden.no (Sturla Molden) Date: Tue, 20 Jan 2009 14:00:31 +0100 Subject: [Python-ideas] With-expression In-Reply-To: <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com> Message-ID: <4975CAEF.4090304@molden.no> On 1/20/2009 2:47 AM, Gregory P. Smith wrote: > I always preferred: > > data = open("file.txt").read() > > Which works all the way back as far as I can remember. > > > In C Python yes that "works" but it makes absolutely no guarantees about > when that file is actually closed. Don't write code that depends on a > particular GC behavior. Then why not introduce that guarantee? Indeterministic GC behaviour is a Jython and IronPython problem. And if you are worried about not closing the file, why not close it manually? S.M. From ironfroggy at gmail.com Tue Jan 20 15:38:09 2009 From: ironfroggy at gmail.com (Calvin Spealman) Date: Tue, 20 Jan 2009 09:38:09 -0500 Subject: [Python-ideas] With-expression In-Reply-To: <4975CAEF.4090304@molden.no> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com> <4975CAEF.4090304@molden.no> Message-ID: <76fd5acf0901200638h20135aearaeeee4fe785e40e2@mail.gmail.com> On Tue, Jan 20, 2009 at 8:00 AM, Sturla Molden wrote: > On 1/20/2009 2:47 AM, Gregory P. Smith wrote: > >> I always preferred: >> >> data = open("file.txt").read() >> >> Which works all the way back as far as I can remember. >> >> >> In C Python yes that "works" but it makes absolutely no guarantees about >> when that file is actually closed. Don't write code that depends on a >> particular GC behavior. > > Then why not introduce that guarantee? Indeterministic GC behaviour is a > Jython and IronPython problem. We cannot and should not define aspects of the language which do (especially on purpose) specifically deny other implementations from respecting that change. > And if you are worried about not closing the file, why not close it > manually? > > S.M. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From grosser.meister.morti at gmx.net Tue Jan 20 21:25:53 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Tue, 20 Jan 2009 21:25:53 +0100 Subject: [Python-ideas] With-expression In-Reply-To: <76fd5acf0901200638h20135aearaeeee4fe785e40e2@mail.gmail.com> References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com> <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com> <4975CAEF.4090304@molden.no> <76fd5acf0901200638h20135aearaeeee4fe785e40e2@mail.gmail.com> Message-ID: <49763351.5040202@gmx.net> Calvin Spealman schrieb: > On Tue, Jan 20, 2009 at 8:00 AM, Sturla Molden wrote: >> On 1/20/2009 2:47 AM, Gregory P. Smith wrote: >> >>> I always preferred: >>> >>> data = open("file.txt").read() >>> >>> Which works all the way back as far as I can remember. >>> >>> >>> In C Python yes that "works" but it makes absolutely no guarantees about >>> when that file is actually closed. Don't write code that depends on a >>> particular GC behavior. >> Then why not introduce that guarantee? Indeterministic GC behaviour is a >> Jython and IronPython problem. > > We cannot and should not define aspects of the language which do > (especially on purpose) specifically deny other implementations from > respecting that change. > And who knows, maybe someday Pythons GC will change its strategy to something more like Java? Then this would break a lot of code. (Well, most likely it would anyway.) -panzi From ironfroggy at gmail.com Wed Jan 21 02:32:21 2009 From: ironfroggy at gmail.com (Calvin Spealman) Date: Tue, 20 Jan 2009 20:32:21 -0500 Subject: [Python-ideas] min_fields argument to str.split() Message-ID: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> This might be a silly idea, but I was wondering about forcing split() to return at least X number of items. For example, I might be getting a comma separated string and i want to split it up into names, but I might have less than all of them. If it is just 0, 1, or 2, I can use partition(), but any more and that doesn't work. Besides, I don't care if the separator is there, just to get the values. Might also make sense to give the values to give by default. Example of implementing this: def split(self, sep=None, max_splits=None, min_items=None): parts = self.split(sep, max_splits) if len(parts) < min_items: parts.extend([None] * (min_items - len(parts))) return parts Use would be like this: a, b, c, d = "1,2,3".split(',', None, 4) Probably not a great idea, but I'm tossing it out there, anyway. -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From algorias at yahoo.com Wed Jan 21 02:50:51 2009 From: algorias at yahoo.com (Vitor Bosshard) Date: Tue, 20 Jan 2009 17:50:51 -0800 (PST) Subject: [Python-ideas] min_fields argument to str.split() References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> Message-ID: <162626.90154.qm@web54406.mail.yahoo.com> ----- Mensaje original ---- > De: Calvin Spealman > Para: python-ideas at python.org > Enviado: martes, 20 de enero, 2009 22:32:21 > Asunto: [Python-ideas] min_fields argument to str.split() > > This might be a silly idea, but I was wondering about forcing split() > to return at least X number of items. For example, I might be getting > a comma separated string and i want to split it up into names, but I > might have less than all of them. If it is just 0, 1, or 2, I can use > partition(), but any more and that doesn't work. Besides, I don't care > if the separator is there, just to get the values. Might also make > sense to give the values to give by default. > > Example of implementing this: > > def split(self, sep=None, max_splits=None, min_items=None): > parts = self.split(sep, max_splits) > if len(parts) < min_items: > parts.extend([None] * (min_items - len(parts))) > return parts > > Use would be like this: > > a, b, c, d = "1,2,3".split(',', None, 4) How about this? def splitter(string,sep=None,max_splits=-1, min_items=None): parts = string.split(sep,max_splits) for part in parts: yield part for i in range(min_items-len(parts)): yield a, b, c, d = splitter("1,2,3", ",", -1, 4) Vitor ?Todo sobre la Liga Mexicana de f?tbol! Estadisticas, resultados, calendario, fotos y m?s:< http://espanol.sports.yahoo.com/ From charlie137 at gmail.com Wed Jan 21 03:36:05 2009 From: charlie137 at gmail.com (Guillaume Chereau) Date: Wed, 21 Jan 2009 10:36:05 +0800 Subject: [Python-ideas] standard library proposal for "tasklets" using generators Message-ID: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com> Hello, Thanks to the introduction of new generator features, as described in PEP 0342, it is possible to write what I would call "tasklet" (I am no sure what should be the term exactly). tasklet are a way to write callback based code that looks like thread. Example : (assuming we have a function sleep(sec, callback) that will call the callback after `sec` seconds) @tasklet task1(): print "sleep" yield Wait(sleep, 5) print "called after sleep" yield 10 # returned value @tasklet task2(): for i in range(10): value = yield task1() (the same thing using callback only would be quite difficult to read) The library should provide -the names are not very important- a Tasklet class, plus a tasklet decorator to turn any generator into a tasklet, plus a Wait function to turn a callback function into a tasklet, plus maybe a few other functions. In many cases this kind of library would be very useful. I am currently using something very similar for a project I am working on [0]. the kiwi project also have a similar library [1] (but the syntax is not really nice because they don't take advantage of the PEP 0342 features.) I guess more and more people may try to use similar library in the future, but the problem is that the implementation of it is really not trivial, so I would like to suggest having such a library in the standard set of python libraries. Do you think this is a good idea ? cheers, Guillaume [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html -- http://charlie137.blogspot.com/ From guido at python.org Wed Jan 21 03:51:37 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2009 18:51:37 -0800 Subject: [Python-ideas] min_fields argument to str.split() In-Reply-To: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> Message-ID: On Tue, Jan 20, 2009 at 5:32 PM, Calvin Spealman wrote: > This might be a silly idea, but I was wondering about forcing split() > to return at least X number of items. For example, I might be getting > a comma separated string and i want to split it up into names, but I > might have less than all of them. If it is just 0, 1, or 2, I can use > partition(), but any more and that doesn't work. Besides, I don't care > if the separator is there, just to get the values. Might also make > sense to give the values to give by default. > > Example of implementing this: > > def split(self, sep=None, max_splits=None, min_items=None): > parts = self.split(sep, max_splits) > if len(parts) < min_items: > parts.extend([None] * (min_items - len(parts))) > return parts > > Use would be like this: > > a, b, c, d = "1,2,3".split(',', None, 4) > > Probably not a great idea, but I'm tossing it out there, anyway. Doesn't strike me as easy to use; the None values returned would most likely cause a traceback a few lines down. Returning '' might be better, but I still think it's very marginal functionality and makes the API ugly due to the large number of optional arguments. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Jan 21 03:56:48 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2009 18:56:48 -0800 Subject: [Python-ideas] standard library proposal for "tasklets" using generators In-Reply-To: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com> References: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com> Message-ID: The pattern of invoking all sorts of stuff that may wait indefinitely using yield has been proposed before, but I find it creates rather awkward code compared to using plain threads. Not something I'd like to encourage by having standard decorators and other APIs around this idea. I believe Twisted has something like this, and that's probably a good place to isolate such patterns; in their case they really want/need this, so they put up with the awkwardness. Putting the necessary decorators/APIs in the language doesn't do much about the awkwardness, so I don't think we should do this. On Tue, Jan 20, 2009 at 6:36 PM, Guillaume Chereau wrote: > Hello, > Thanks to the introduction of new generator features, as described in > PEP 0342, it is possible to write what I would call "tasklet" (I am no > sure what should be the term exactly). > tasklet are a way to write callback based code that looks like thread. Example : > > (assuming we have a function sleep(sec, callback) that will call the > callback after `sec` seconds) > > @tasklet > task1(): > print "sleep" > yield Wait(sleep, 5) > print "called after sleep" > yield 10 # returned value > > @tasklet > task2(): > for i in range(10): > value = yield task1() > > (the same thing using callback only would be quite difficult to read) > > The library should provide -the names are not very important- a > Tasklet class, plus a tasklet decorator to turn any generator into a > tasklet, plus a Wait function to turn a callback function into a > tasklet, plus maybe a few other functions. > > In many cases this kind of library would be very useful. I am > currently using something very similar for a project I am working on > [0]. the kiwi project also have a similar library [1] (but the syntax > is not really nice because they don't take advantage of the PEP 0342 > features.) > > I guess more and more people may try to use similar library in the > future, but the problem is that the implementation of it is really not > trivial, so I would like to suggest having such a library in the > standard set of python libraries. > > Do you think this is a good idea ? > > cheers, > > Guillaume > > [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py > [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html > > -- > http://charlie137.blogspot.com/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From taleinat at gmail.com Wed Jan 21 11:29:17 2009 From: taleinat at gmail.com (Tal Einat) Date: Wed, 21 Jan 2009 12:29:17 +0200 Subject: [Python-ideas] min_fields argument to str.split() In-Reply-To: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> Message-ID: <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com> Calvin Spealman wrote: > This might be a silly idea, but I was wondering about forcing split() > to return at least X number of items. For example, I might be getting > a comma separated string and i want to split it up into names, but I > might have less than all of them. If it is just 0, 1, or 2, I can use > partition(), but any more and that doesn't work. Besides, I don't care > if the separator is there, just to get the values. Might also make > sense to give the values to give by default. > > Example of implementing this: > > def split(self, sep=None, max_splits=None, min_items=None): > parts = self.split(sep, max_splits) > if len(parts) < min_items: > parts.extend([None] * (min_items - len(parts))) > return parts > > Use would be like this: > > a, b, c, d = "1,2,3".split(',', None, 4) > > Probably not a great idea, but I'm tossing it out there, anyway. This is not a common thing to do, and it is easy enough to implement in a single assignment even without a helper function: a, b, c, d = ("1,2,3".split(',', 4) + [None] * 4)[:4] This also has the advantage of giving you explicit control of the filler value for missing pieces. (Note: be careful not to use a _mutable_ filler value in this manner!) - Tal From charlie137 at gmail.com Wed Jan 21 11:45:55 2009 From: charlie137 at gmail.com (Guillaume Chereau) Date: Wed, 21 Jan 2009 18:45:55 +0800 Subject: [Python-ideas] Fwd: standard library proposal for "tasklets" using generators In-Reply-To: <8e9327d40901202000o34908c1dv27e7ef8ac969a14c@mail.gmail.com> References: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com> <8e9327d40901202000o34908c1dv27e7ef8ac969a14c@mail.gmail.com> Message-ID: <8e9327d40901210245v63dbbc9bn2132626858e6811a@mail.gmail.com> Just realized i clicked "reply" instead of 'reply to all" ---------- Forwarded message ---------- From: Guillaume Chereau Date: Wed, Jan 21, 2009 at 12:00 PM Subject: Re: [Python-ideas] standard library proposal for "tasklets" using generators To: Guido van Rossum Thanks for the answer, I understand the concern here. I still think having such a library, if not as a standard python lib, but at least as an independent one, that could then be used by all the project using such pattern (twisted, kiwi, gobject, etc.) would be a good idea. I will try to contact those different projects for suggestions. I also have to say one of my secret hope was that the pypy interpreter could eventually understand this kind of 'tasklet' generators and generate C or other language code out of it, since pypy already has support for tasklets, but using a syntax that can't be interpreted by cpython. - Guillaume On Wed, Jan 21, 2009 at 10:56 AM, Guido van Rossum wrote: > The pattern of invoking all sorts of stuff that may wait indefinitely > using yield has been proposed before, but I find it creates rather > awkward code compared to using plain threads. Not something I'd like > to encourage by having standard decorators and other APIs around this > idea. I believe Twisted has something like this, and that's probably a > good place to isolate such patterns; in their case they really > want/need this, so they put up with the awkwardness. Putting the > necessary decorators/APIs in the language doesn't do much about the > awkwardness, so I don't think we should do this. > > On Tue, Jan 20, 2009 at 6:36 PM, Guillaume Chereau wrote: >> Hello, >> Thanks to the introduction of new generator features, as described in >> PEP 0342, it is possible to write what I would call "tasklet" (I am no >> sure what should be the term exactly). >> tasklet are a way to write callback based code that looks like thread. Example : >> >> (assuming we have a function sleep(sec, callback) that will call the >> callback after `sec` seconds) >> >> @tasklet >> task1(): >> print "sleep" >> yield Wait(sleep, 5) >> print "called after sleep" >> yield 10 # returned value >> >> @tasklet >> task2(): >> for i in range(10): >> value = yield task1() >> >> (the same thing using callback only would be quite difficult to read) >> >> The library should provide -the names are not very important- a >> Tasklet class, plus a tasklet decorator to turn any generator into a >> tasklet, plus a Wait function to turn a callback function into a >> tasklet, plus maybe a few other functions. >> >> In many cases this kind of library would be very useful. I am >> currently using something very similar for a project I am working on >> [0]. the kiwi project also have a similar library [1] (but the syntax >> is not really nice because they don't take advantage of the PEP 0342 >> features.) >> >> I guess more and more people may try to use similar library in the >> future, but the problem is that the implementation of it is really not >> trivial, so I would like to suggest having such a library in the >> standard set of python libraries. >> >> Do you think this is a good idea ? >> >> cheers, >> >> Guillaume >> >> [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py >> [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html >> >> -- >> http://charlie137.blogspot.com/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- http://charlie137.blogspot.com/ -- http://charlie137.blogspot.com/ From arnodel at googlemail.com Wed Jan 21 14:32:56 2009 From: arnodel at googlemail.com (Arnaud Delobelle) Date: Wed, 21 Jan 2009 13:32:56 +0000 Subject: [Python-ideas] min_fields argument to str.split() In-Reply-To: <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com> References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com> Message-ID: <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com> 2009/1/21 Tal Einat : > Calvin Spealman wrote: >> This might be a silly idea, but I was wondering about forcing split() >> to return at least X number of items. For example, I might be getting >> a comma separated string and i want to split it up into names, but I >> might have less than all of them. If it is just 0, 1, or 2, I can use >> partition(), but any more and that doesn't work. Besides, I don't care >> if the separator is there, just to get the values. Might also make >> sense to give the values to give by default. >> >> Example of implementing this: >> >> def split(self, sep=None, max_splits=None, min_items=None): >> parts = self.split(sep, max_splits) >> if len(parts) < min_items: >> parts.extend([None] * (min_items - len(parts))) >> return parts >> >> Use would be like this: >> >> a, b, c, d = "1,2,3".split(',', None, 4) >> >> Probably not a great idea, but I'm tossing it out there, anyway. > > This is not a common thing to do, and it is easy enough to implement > in a single assignment even without a helper function: > > a, b, c, d = ("1,2,3".split(',', 4) + [None] * 4)[:4] > > This also has the advantage of giving you explicit control of the > filler value for missing pieces. How about this? >>> from itertools import * >>> def fixlen(iterable, len, item=None): ... return islice(chain(iterable, repeat(item)), len) ... >>> list(fixlen([1, 2, 3], 5)) [1, 2, 3, None, None] -- Arnaud From grosser.meister.morti at gmx.net Wed Jan 21 14:55:08 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Wed, 21 Jan 2009 14:55:08 +0100 Subject: [Python-ideas] min_fields argument to str.split() In-Reply-To: <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com> References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com> <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com> Message-ID: <4977293C.5080103@gmx.net> Arnaud Delobelle schrieb: > > How about this? > >>>> from itertools import * >>>> def fixlen(iterable, len, item=None): > ... return islice(chain(iterable, repeat(item)), len) > ... >>>> list(fixlen([1, 2, 3], 5)) > [1, 2, 3, None, None] > I like this. Maybe an addition to itertools? +1 From algorias at yahoo.com Wed Jan 21 15:14:57 2009 From: algorias at yahoo.com (Vitor Bosshard) Date: Wed, 21 Jan 2009 06:14:57 -0800 (PST) Subject: [Python-ideas] min_fields argument to str.split() References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com> <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com> <4977293C.5080103@gmx.net> Message-ID: <439790.10501.qm@web54406.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable ----- Mensaje original ----=0A> De: Mathias Panzenb=F6ck =0A> Para: Arnaud Delobelle =0A> CC: p= ython-ideas at python.org=0A> Enviado: mi=E9rcoles, 21 de enero, 2009 10:55:08= =0A> Asunto: Re: [Python-ideas] min_fields argument to str.split()=0A> =0A>= Arnaud Delobelle schrieb:=0A> >=0A> > How about this?=0A> >=0A> >>>> from = itertools import *=0A> >>>> def fixlen(iterable, len, item=3DNone):=0A> > .= ..=A0 =A0 return islice(chain(iterable, repeat(item)), len)=0A> > ...=0A> >= >>> list(fixlen([1, 2, 3], 5))=0A> > [1, 2, 3, None, None]=0A> >=0A> =0A> I= like this. Maybe an addition to itertools?=0A> =0A> +1=0A=0A+1 from me too= .=A0Way better than the generator I proposed. Having this as a function als= o makes it=A0applicable to much more than just one use case.=0A=0AIf you se= e the following code:=0A=0Aa,b,c,d =3D fixlen(str.split(),4)=0A=0Ayou immed= iately know it's not a regular unpacking operation; it will be the first pl= ace you look if something goes wrong.=0A=0A=0AVitor=0A=0A=0A =A1Todo s= obre la Liga Mexicana de f=FAtbol! Estadisticas, resultados, calendario, fo= tos y m=E1s:<=0Ahttp://espanol.sports.yahoo.com/ From brett at python.org Wed Jan 21 21:22:58 2009 From: brett at python.org (Brett Cannon) Date: Wed, 21 Jan 2009 12:22:58 -0800 Subject: [Python-ideas] Fwd: standard library proposal for "tasklets" using generators In-Reply-To: <8e9327d40901210245v63dbbc9bn2132626858e6811a@mail.gmail.com> References: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com> <8e9327d40901202000o34908c1dv27e7ef8ac969a14c@mail.gmail.com> <8e9327d40901210245v63dbbc9bn2132626858e6811a@mail.gmail.com> Message-ID: On Wed, Jan 21, 2009 at 02:45, Guillaume Chereau wrote: > Just realized i clicked "reply" instead of 'reply to all" > > > ---------- Forwarded message ---------- > From: Guillaume Chereau > Date: Wed, Jan 21, 2009 at 12:00 PM > Subject: Re: [Python-ideas] standard library proposal for "tasklets" > using generators > To: Guido van Rossum > > > Thanks for the answer, > > I understand the concern here. > > I still think having such a library, if not as a standard python lib, > but at least as an independent one, that could then be used by all the > project using such pattern (twisted, kiwi, gobject, etc.) would be a > good idea. > I will try to contact those different projects for suggestions. > If the package really takes off and it turns out there is a good design that leads to clean code this can be re-evaluated. But getting a third-party library out there that has wide usage is a good first step. -Brett > I also have to say one of my secret hope was that the pypy interpreter > could eventually understand this kind of 'tasklet' generators and > generate C or other language code out of it, since pypy already has > support for tasklets, but using a syntax that can't be interpreted by > cpython. > > - Guillaume > > On Wed, Jan 21, 2009 at 10:56 AM, Guido van Rossum wrote: >> The pattern of invoking all sorts of stuff that may wait indefinitely >> using yield has been proposed before, but I find it creates rather >> awkward code compared to using plain threads. Not something I'd like >> to encourage by having standard decorators and other APIs around this >> idea. I believe Twisted has something like this, and that's probably a >> good place to isolate such patterns; in their case they really >> want/need this, so they put up with the awkwardness. Putting the >> necessary decorators/APIs in the language doesn't do much about the >> awkwardness, so I don't think we should do this. >> >> On Tue, Jan 20, 2009 at 6:36 PM, Guillaume Chereau wrote: >>> Hello, >>> Thanks to the introduction of new generator features, as described in >>> PEP 0342, it is possible to write what I would call "tasklet" (I am no >>> sure what should be the term exactly). >>> tasklet are a way to write callback based code that looks like thread. Example : >>> >>> (assuming we have a function sleep(sec, callback) that will call the >>> callback after `sec` seconds) >>> >>> @tasklet >>> task1(): >>> print "sleep" >>> yield Wait(sleep, 5) >>> print "called after sleep" >>> yield 10 # returned value >>> >>> @tasklet >>> task2(): >>> for i in range(10): >>> value = yield task1() >>> >>> (the same thing using callback only would be quite difficult to read) >>> >>> The library should provide -the names are not very important- a >>> Tasklet class, plus a tasklet decorator to turn any generator into a >>> tasklet, plus a Wait function to turn a callback function into a >>> tasklet, plus maybe a few other functions. >>> >>> In many cases this kind of library would be very useful. I am >>> currently using something very similar for a project I am working on >>> [0]. the kiwi project also have a similar library [1] (but the syntax >>> is not really nice because they don't take advantage of the PEP 0342 >>> features.) >>> >>> I guess more and more people may try to use similar library in the >>> future, but the problem is that the implementation of it is really not >>> trivial, so I would like to suggest having such a library in the >>> standard set of python libraries. >>> >>> Do you think this is a good idea ? >>> >>> cheers, >>> >>> Guillaume >>> >>> [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py >>> [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html >>> >>> -- >>> http://charlie137.blogspot.com/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> http://mail.python.org/mailman/listinfo/python-ideas >>> >> >> >> >> -- >> --Guido van Rossum (home page: http://www.python.org/~guido/) >> > > > > -- > http://charlie137.blogspot.com/ > > > > -- > http://charlie137.blogspot.com/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From algorias at gmail.com Thu Jan 22 02:58:06 2009 From: algorias at gmail.com (Vitor Bosshard) Date: Wed, 21 Jan 2009 22:58:06 -0300 Subject: [Python-ideas] min_fields argument to str.split() In-Reply-To: <2987c46d0901211748n6b2559fcoe0d67c45f38f6a6f@mail.gmail.com> References: <2987c46d0901211748n6b2559fcoe0d67c45f38f6a6f@mail.gmail.com> Message-ID: <2987c46d0901211758m1e99d76bo879960e8eee6c079@mail.gmail.com> NOTE: It was brought to my attention that my last e-mail message arrived completely garbled. Here it is again, as I meant to send it. My Apologies to everyone. > > > > How about this? > > > >>>> from itertools import * > >>>> def fixlen(iterable, len, item=None): > > ... return islice(chain(iterable, repeat(item)), len) > > ... > >>>> list(fixlen([1, 2, 3], 5)) > > [1, 2, 3, None, None] > > > > I like this. Maybe an addition to itertools? > > +1 +1 from me too. Way better than the generator I proposed. Having this as a function also makes it applicable to much more than just one use case. If you see the following code: a,b,c,d = fixlen(str.split(),4) You immediately know it's not a regular unpacking operation; it will be the first place you look if something goes wrong. Vitor -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Thu Jan 22 11:53:13 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Jan 2009 11:53:13 +0100 Subject: [Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired) Message-ID: <49785019.5070606@molden.no> I have lately been using OpenMP to write parallel C and Fortran code. I must admit I am impressed. OpenMP is a much better abstraction for writing concurrent code than using Win32/posix threads directly (or similarly threading.Thread i Java or Python). What matters most is that code can be written as sequential, tested, and then parallelised using compiler pragmas. This is much easier than writing code intended to be parallel from the start. Not only is the abstraction more easy to apply, it also leads to fewer problems with deadlocks, race conditions, livelocks, etc. I was thinking something similar could be created for Python, e.g. on top of the existing thread or threading modules, and possibly multiprocessing. I believe a context manager could be used for this purpose. What I have in mind is an API that would look approximately like this (OpenMP pragmas for C on top, proposed Python equivalent below): #pragma omp parallel with pymp.Pool() as pool: #pragma omp for for item in pool.parallel(): #pragma omp for shedule(guided) for item in pool.parallel(, shed='guided'): #pragma omp parallel for with pymp.Pool() as pool: for item in pool.parallel(): #pragma omp barrier pool.barrier() #pragma omp section pool.section(fun, *args, **kwargs) #pragma omp parallel sections with pymp.Pool() as pool: pool.section(fun1, *args, **kwargs) pool.section(fun2, *args, **kwargs) #pragma omp master if pool.master: #pragma omp critical #pragma omp atomic with pool.lock: #pragma omp single with pool.single(): #pragma omp ordered with pool.ordered(): This is all trivial to program, except for the context manager on top. It has somehow to get access to the code block below, spawn multiple threads, and execute that block in each of the threads. I am not sure how to grab the next executable block as a Python object (so I could pass it to eval or exec), so a little help would be appreciated :) Regards, Sturla Molden From jnoller at gmail.com Thu Jan 22 15:17:28 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 22 Jan 2009 09:17:28 -0500 Subject: [Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired) In-Reply-To: <49785019.5070606@molden.no> References: <49785019.5070606@molden.no> Message-ID: <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com> On Thu, Jan 22, 2009 at 5:53 AM, Sturla Molden wrote: > > I have lately been using OpenMP to write parallel C and Fortran code. I must > admit I am impressed. OpenMP is a much better abstraction for writing > concurrent code than using Win32/posix threads directly (or similarly > threading.Thread i Java or Python). What matters most is that code can be > written as sequential, tested, and then parallelised using compiler pragmas. > This is much easier than writing code intended to be parallel from the > start. > > Not only is the abstraction more easy to apply, it also leads to fewer > problems with deadlocks, race conditions, livelocks, etc. > > I was thinking something similar could be created for Python, e.g. on top of > the existing thread or threading modules, and possibly multiprocessing. I > believe a context manager could be used for this purpose. What I have in > mind is an API that would look approximately like this (OpenMP pragmas for C > on top, proposed Python equivalent below): > > > #pragma omp parallel > with pymp.Pool() as pool: > > #pragma omp for > for item in pool.parallel(): > > #pragma omp for shedule(guided) > for item in pool.parallel(, shed='guided'): > > #pragma omp parallel for > with pymp.Pool() as pool: > for item in pool.parallel(): > > #pragma omp barrier > pool.barrier() > > #pragma omp section > pool.section(fun, *args, **kwargs) > > #pragma omp parallel sections > with pymp.Pool() as pool: > pool.section(fun1, *args, **kwargs) > pool.section(fun2, *args, **kwargs) > > #pragma omp master > if pool.master: > > #pragma omp critical > #pragma omp atomic > with pool.lock: > > #pragma omp single > with pool.single(): > > #pragma omp ordered > with pool.ordered(): > > > This is all trivial to program, except for the context manager on top. It > has somehow to get access to the code block below, spawn multiple threads, > and execute that block in each of the threads. I am not sure how to grab the > next executable block as a Python object (so I could pass it to eval or > exec), so a little help would be appreciated :) > Hi Sturla, Interesting that you bring this up - while I'm not in the know about openMP - I have been sketching out some improvements to threading and multiprocessing that follow some of this thinking. The first batch would be adding context managers where appropriate to the multiprocessing module (e.g. pool and managers/etc). The second is adding a library of decorators where you could say: @multiprocessing.Pool(5, 'apply') def myfunc() .... Or something along those lines. In the case of getting the function object back, decorators make more sense for some of the use cases. Now, for threading - I personally feel that there's some improvements to be made by adding a series of abstractions and utilities on top of threading ala multiprocessing.pool and the like. See java.util.concurrent for some good examples of threading abstractions. The same decorators and contextmanagers could be used for any threading abstractions added. Just some thoughts - right now I've sworn off writing new features until I zero out the multiprocessing bug queue, so these might be done in 2025. -jesse From sturla at molden.no Thu Jan 22 16:03:06 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Jan 2009 16:03:06 +0100 Subject: [Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired) In-Reply-To: <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com> References: <49785019.5070606@molden.no> <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com> Message-ID: <49788AAA.2040406@molden.no> On 1/22/2009 3:17 PM, Jesse Noller wrote: > Just some thoughts - right now I've sworn off writing new features > until I zero out the multiprocessing bug queue, so these might be done > in 2025. I was actually thinking about writing it myself. :) I think the OpenMP style of abstracting concurrency fits better with the mind, and therefore is more 'pythonic' than the Java inspired threading module. Not just for the sake of speed on SMPs, but also for other concurrent tasks for which threads (or processes) can be used. A skeleton of such a context manager would look roughly like this. Anyway I am stuck with what to exec in the static method Pool._threadproc, in order to execute the correct code object. I somehow have to exec (or eval) the code object in the frame from f_lineno or f_lasti. I have noticed the Queue bug. Very annoying indeed. import thread import sys import os from math import ceil, log class Pool(object): def __init__(self, nthreads=None, masterpool=None, tid=0): self.nthreads = nthreads self.masterpool = masterpool self.tid = tid if nthreads is None: self.nthreads = self._num_processors() def __enter__(self): if self.masterpool is None: frame = sys._getframe().f_back code = frame.f_code lasti = frame.f_lasti localdict = frame.f_locals globaldict = frame.f_globals # find name of self in frame for name in localdict: if localdict[name] is self: selfname = name break # spawn threads for tid in range(1,self.nthreads): pool = Pool(nthreads=self.nthreads, masterpool=self, tid=tid) _localdict = localdict.copy() _globaldict = globaldict.copy() _localdict[selfname] = pool thread.start_new_thread(self._threadproc, (code,lasti,_globaldict,_localdict)) def __exit__(self, exc_type, exc_val, exc_tb): self.barrier() if self.masterpool is not None: thread.exit() # ++ more cleanup @staticmethod def _threadproc(code, lasti, globals, locals): # we must somehow execute the code object from # the last instruction (lasti) # Something like 'exec code in globals, locals' @staticmethod def _num_processors(): if os.name == 'nt': return int(os.getenv('NUMBER_OF_PROCESSORS')) elif sys.platform == 'linux2': # Linux ... I think this works # each cpu is separated by a blank line, and there # is a blank line at the end retv = 0 with open('/proc/cpuinfo','rt') as cpuinfo: for line in cpuinfo: if not len(line): retv += 1 return retv else: raise RuntimeError, 'unknown platform' def barrier(self): ''' dissemination barrier ''' nt = self.nthreads if nt == 1: return tid = self.tid log2 = lambda x : for k in range(int(ceil(log(nt)/log(2)))): # send event to thread (tid + 2**k) % nt # wait for event from thread (tid - 2**k) % nt Regards, S.M. From sturla at molden.no Thu Jan 22 16:26:25 2009 From: sturla at molden.no (Sturla Molden) Date: Thu, 22 Jan 2009 16:26:25 +0100 Subject: [Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired) In-Reply-To: <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com> References: <49785019.5070606@molden.no> <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com> Message-ID: <49789021.2080101@molden.no> On 1/22/2009 3:17 PM, Jesse Noller wrote: > Interesting that you bring this up - while I'm not in the know about > openMP - I have been sketching out some improvements to threading and > multiprocessing that follow some of this thinking. Here is a toy example of what I have in mind. Say you would want to compute the DFT of some signal (real apps would use an FFT in C for this, but never mind). In Python using an O(n**2) algorithm, this would look like somthing like this: def real_dft(x): ''' DFT for a real valued sequence x ''' r = [] N = len(x) M = N//2 + 1 if N%2 else N//2 for n in range(M): s = 0j for k in range(N): tmp = 2*pi*k*n/N s += x[k] * (cos(tmp) - 1j*sin(tmp)) r.append(s) return r Then, one could 'magically' transform this algorithm into to a parallel one simply by inserting directives from the 'pymp' module: def real_dft(x): ''' DFT for a real valued sequence x ''' ''' parallelized ''' r = [] N = len(x) M = N//2 + 1 if N%2 else N//2 with Pool() as pool: for n in pool.parallel(range(M)): s = 0j for k in range(N): tmp = 2*pi*k*n/N s += x[k] * (cos(tmp) - 1j*sin(tmp)) with pool.ordered(): r.append(s) return r The idea is that 'parallelizing' a sequential algorithm like this is much easier than writing a parallel one from scratch using the abstractions in threading or multiprocessing. Sturla Molden From stargaming at gmail.com Thu Jan 22 19:25:49 2009 From: stargaming at gmail.com (Robert Lehmann) Date: Thu, 22 Jan 2009 18:25:49 +0000 (UTC) Subject: [Python-ideas] min_fields argument to str.split() References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com> <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com> <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com> <4977293C.5080103@gmx.net> Message-ID: On Wed, 21 Jan 2009 14:55:08 +0100, Mathias Panzenb?ck wrote: > Arnaud Delobelle schrieb: >> >> How about this? >> >>>>> from itertools import * >>>>> def fixlen(iterable, len, item=None): >> ... return islice(chain(iterable, repeat(item)), len) ... >>>>> list(fixlen([1, 2, 3], 5)) >> [1, 2, 3, None, None] >> >> > I like this. Maybe an addition to itertools? > > +1 I opened an issue_ at the Python bug tracker. .. _issue: http://bugs.python.org/issue5034 -- Robert "Stargaming" Lehmann From joeedh at gmail.com Fri Jan 23 07:59:01 2009 From: joeedh at gmail.com (joe) Date: Thu, 22 Jan 2009 23:59:01 -0700 Subject: [Python-ideas] optimized VM ideas Message-ID: So, I've been kicking around some ideas for an optimized python VM. I freely admit I'm an amateur at this, but I find the problem of making python code run faster fascinating. My ideas arose from observing that google V8's JIT compiler and type system are much simpler compared to TraceMonkey, but is also faster, and also learning that SquirrelFish is allegedy faster than V8, even though it doesn't compile to native code at all (which V8 and I believe TraceMonkey both do). This leads me to believe that relatively simple, more general concepts in VM design can have a bigger impact then specific, highly complicated JIT solutions, in the context of dynamic languages that can't be easily typed at compile time. So I've thought of a few ideas for a more (new) streamlined python VM: * Simplify the cpython object model as much as possible, while still allowing most of the power of the current model. * Either keep referencing counting, or experiment with some of the newer techniques such as pointer escaping. Object models that exclusively rely on cyclic GC's have many issues and are hard to get right. * Possibly modify the bytecode to be register-based, as in SquirrelFish. Not sure if this is worth it with python code. * Use direct threading (which is basically optimizing switch statements to be only one or two instructions) for the bytecode loop. * Remove string lookups for member access entirely, and replaced with a system of unique identifyers. The idea is you would use a hash in the types to map a member id to an index. Hashing ints is faster then strings, and I've even thought about experimenting with using collapsed arrays instead of hashes. Of course, the design would still need to support string lookups when necessary. I've thought about this a lot, and I think you'd need the same general idea as V8's hidden classes for this to work right (though instead of classes, it'd just be member/unique id lookup maps). I'm not sure I'll have the time to anytime soon to prototype these ideas, but I thought I'd kick them out there and see what people say. Note, I'm in no way suggesting any sort of change to the existing cpython VM (it's way, way too early for that kind of talk). references: v8's design: http://code.google.com/apis/v8/design.html squirrelfish's design: http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/ Joe From joeedh at gmail.com Fri Jan 23 08:20:22 2009 From: joeedh at gmail.com (joe) Date: Fri, 23 Jan 2009 00:20:22 -0700 Subject: [Python-ideas] optimized VM ideas In-Reply-To: References: Message-ID: Ah, should've done more fact checking *before* posting the first mail. Actually SquirrelFish does compile to native code nowadays, in a nicely simple way. On Thu, Jan 22, 2009 at 11:59 PM, joe wrote: > So, I've been kicking around some ideas for an optimized python VM. I freely > admit I'm an amateur at this, but I find the problem of making python code > run faster fascinating. My ideas arose from observing that google V8's JIT > compiler and type system are much simpler compared to TraceMonkey, but is > also faster, and also learning that SquirrelFish is allegedy faster than V8, > even though it doesn't compile to native code at all (which V8 and I > believe TraceMonkey both do). > > This leads me to believe that relatively simple, more general concepts in > VM design can have a bigger impact then specific, highly complicated JIT > solutions, in the context of dynamic languages that can't be easily typed > at compile time. > > So I've thought of a few ideas for a more (new) streamlined python VM: > > * Simplify the cpython object model as much as possible, while still allowing > most of the power of the current model. > > * Either keep referencing counting, or experiment with some of the newer > techniques such as pointer escaping. Object models that exclusively rely > on cyclic GC's have many issues and are hard to get right. > > * Possibly modify the bytecode to be register-based, as in SquirrelFish. > Not sure if this is worth it with python code. > > * Use direct threading (which is basically optimizing switch statements to > be only one or two instructions) for the bytecode loop. > > * Remove string lookups for member access entirely, and replaced with a > system of unique identifyers. The idea is you would use a hash in the > types to map a member id to an index. Hashing ints is faster then strings, > and I've even thought about experimenting with using collapsed arrays instead > of hashes. Of course, the design would still need to support string lookups > when necessary. I've thought about this a lot, and I think you'd need the > same general idea as V8's hidden classes for this to work right (though > instead of classes, it'd just be member/unique id lookup maps). > > I'm not sure I'll have the time to anytime soon to prototype these ideas, but I > thought I'd kick them out there and see what people say. Note, I'm in no way > suggesting any sort of change to the existing cpython VM (it's way, way too > early for that kind of talk). > > references: > > v8's design: http://code.google.com/apis/v8/design.html > squirrelfish's design: > http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/ > > Joe > From greg at krypto.org Fri Jan 23 09:17:15 2009 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 23 Jan 2009 00:17:15 -0800 Subject: [Python-ideas] optimized VM ideas In-Reply-To: References: Message-ID: <52dc1c820901230017r5021294el34e1726079f551c5@mail.gmail.com> On Thu, Jan 22, 2009 at 10:59 PM, joe wrote: ... > > * Use direct threading (which is basically optimizing switch statements to > be only one or two instructions) for the bytecode loop. fyi - http://bugs.python.org/issue4753 does this (at least when using gcc). The optimization is not about removing instructions per se. but in getting rid of the single switch statement's unpredictable branch that causes modern cpus to stall while determining the correct place to jump rather than speculatively guessing correctly a significant portion of the time. ... > > * Remove string lookups for member access entirely, and replaced with a > system of unique identifyers. The idea is you would use a hash in the > types to map a member id to an index. Hashing ints is faster then > strings, > and I've even thought about experimenting with using collapsed arrays > instead > of hashes. Of course, the design would still need to support string > lookups > when necessary. I've thought about this a lot, and I think you'd need the > same general idea as V8's hidden classes for this to work right (though > instead of classes, it'd just be member/unique id lookup maps). Python strings are already immutable, their hash is computed only once. strings used in code as attributes are interned so that all occurances of that string are the same object making the most common lookups for attribute accesses a table lookup with a pointer equality check. > I'm not sure I'll have the time to anytime soon to prototype these ideas, > but I > thought I'd kick them out there and see what people say. Note, I'm in no > way > suggesting any sort of change to the existing cpython VM (it's way, way too > early for that kind of talk). Anyways, the ideas are great and are definately all things people are considering. This should be an great year for python performance and all language VM performance in general. happy hacking, Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From santagada at gmail.com Fri Jan 23 10:36:52 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Fri, 23 Jan 2009 07:36:52 -0200 Subject: [Python-ideas] optimized VM ideas In-Reply-To: References: Message-ID: <1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com> On Jan 23, 2009, at 4:59 AM, joe wrote: > So, I've been kicking around some ideas for an optimized python VM. > I freely > admit I'm an amateur at this, but I find the problem of making > python code > run faster fascinating. My ideas arose from observing that google > V8's JIT > compiler and type system are much simpler compared to TraceMonkey, > but is > also faster, and also learning that SquirrelFish is allegedy faster > than V8, > even though it doesn't compile to native code at all (which V8 and I > believe TraceMonkey both do). I don't think TraceMonkey is slower than V8 I believe the last time I looked TraceMonkey was faster than V8 and it is becoming even faster at each interaction. The way TraceMonkey works reminds me a bit of Psyco, although I might be mixing it with the PyPy JIT. But talking about Psyco, why people don't go help Psyco if all they want is a JIT? It is not like the idea of having a JIT on Python is even new... Psyco was optimizing code even before Webkit/V8 existed. > This leads me to believe that relatively simple, more general > concepts in > VM design can have a bigger impact then specific, highly complicated > JIT > solutions, in the context of dynamic languages that can't be easily > typed > at compile time. I believe in the exact oposite, and TraceMonkey is probably one of the proofs of that... > So I've thought of a few ideas for a more (new) streamlined python VM: > > * Simplify the cpython object model as much as possible, while still > allowing > most of the power of the current model. This would modify the language, so it might be interesting, but would generate something which is not Python. > * Either keep referencing counting, or experiment with some of the > newer > techniques such as pointer escaping. Object models that exclusively > rely > on cyclic GC's have many issues and are hard to get right. Don't know, but a good GC is way faster than what CPython is doing already, but maybe it is a good idea to explore some others perspectives on this. > * Possibly modify the bytecode to be register-based, as in > SquirrelFish. > Not sure if this is worth it with python code. Maybe it would help a bit. I don't think it would help more than 10% tops (but I am completely guessing here) > * Use direct threading (which is basically optimizing switch > statements to > be only one or two instructions) for the bytecode loop. The problem with this is (besides the error someone has already stated about your phrasing) that python has really complex bytecodes, so this would also only gain around 10% and it only works with compilers that accept goto labels which the MSVC for example does not (maybe there are more compilers that also doesn't). > * Remove string lookups for member access entirely, and replaced > with a > system of unique identifyers. The idea is you would use a hash in > the > types to map a member id to an index. Hashing ints is faster then > strings, > and I've even thought about experimenting with using collapsed > arrays instead > of hashes. Of course, the design would still need to support > string lookups > when necessary. I've thought about this a lot, and I think you'd > need the > same general idea as V8's hidden classes for this to work right > (though > instead of classes, it'd just be member/unique id lookup maps). A form of hidden classes is already part of PyPy (but I think that only the jit does this). But you can simply remove string lookups as people can implement special methods to track this on the current Python. As I said before I don't believe changing the semantics of python for the sake of performance is even possible. > I'm not sure I'll have the time to anytime soon to prototype these > ideas, but I > thought I'd kick them out there and see what people say. Note, I'm > in no way > suggesting any sort of change to the existing cpython VM (it's way, > way too > early for that kind of talk). If you are not talking about changing CPython VM why not look at Psyco and PyPy? :) > references: > > v8's design: http://code.google.com/apis/v8/design.html > squirrelfish's design: > http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/ > > Joe -- Leonardo Santagada santagada at gmail.com From solipsis at pitrou.net Fri Jan 23 13:19:39 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Jan 2009 12:19:39 +0000 (UTC) Subject: [Python-ideas] optimized VM ideas References: <1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com> Message-ID: Leonardo Santagada writes: > > The way TraceMonkey works reminds me a bit of Psyco, although I might > be mixing it with the PyPy JIT. I'm not sure it works like Psyco. Read the paper on trace trees, it is quite interesting, and it may be doable to retrofit at least the trace construction part in the current bytecode execution loop. http://www.ics.uci.edu/~franz/Site/pubs-pdf/ICS-TR-06-16.pdf > But talking about Psyco, why people > don't go help Psyco if all they want is a JIT? It is not like the idea > of having a JIT on Python is even new... Psyco was optimizing code > even before Webkit/V8 existed. Three problems with Psyco: 1) it is not 100% compatible with Python semantics 2) it only works on i386 (not even x86-64) 3) it's (roughly) unmaintained > Don't know, but a good GC is way faster than what CPython is doing > already, but maybe it is a good idea to explore some others > perspectives on this. Without changing how the GC works, it's probably possible to improve things a bit. See e.g. http://bugs.python.org/issue4688 > > * Possibly modify the bytecode to be register-based, as in > > SquirrelFish. > > Not sure if this is worth it with python code. > > Maybe it would help a bit. I don't think it would help more than 10% > tops (but I am completely guessing here) I had thought about this and it seems a problem would be reference counting. You have to DECREF a register as soon as it isn't used anymore, which adds some bookkeeping. > The problem with this is (besides the error someone has already stated > about your phrasing) that python has really complex bytecodes, so this > would also only gain around 10% and it only works with compilers that > accept goto labels which the MSVC for example does not (maybe there > are more compilers that also doesn't). Well the bytecodes are not that complex. A bunch of them are completely implemented inline the evaluation loop. Also, since it is a stack machine, there are many trivial bytecodes such as LOAD_FAST, POP_TOP etc. The threaded code patch in http://bugs.python.org/issue4753 has given between 0% and 20% speedups on pybench totals amongst the various posters (and a 5% slowdown in one case). cheers Antoine. From jnoller at gmail.com Fri Jan 23 15:14:28 2009 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 23 Jan 2009 09:14:28 -0500 Subject: [Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired) In-Reply-To: <49789021.2080101@molden.no> References: <49785019.5070606@molden.no> <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com> <49789021.2080101@molden.no> Message-ID: <4222a8490901230614y5485e371p8f6e546563740794@mail.gmail.com> On Thu, Jan 22, 2009 at 10:26 AM, Sturla Molden wrote: > On 1/22/2009 3:17 PM, Jesse Noller wrote: > >> Interesting that you bring this up - while I'm not in the know about >> openMP - I have been sketching out some improvements to threading and >> multiprocessing that follow some of this thinking. > > Here is a toy example of what I have in mind. Say you would want to compute > the DFT of some signal (real apps would use an FFT in C for this, but never > mind). In Python using an O(n**2) algorithm, this would look like somthing > like this: > > def real_dft(x): > ''' DFT for a real valued sequence x ''' > r = [] > N = len(x) > M = N//2 + 1 if N%2 else N//2 > for n in range(M): > s = 0j > for k in range(N): > tmp = 2*pi*k*n/N > s += x[k] * (cos(tmp) - 1j*sin(tmp)) > r.append(s) > return r > > Then, one could 'magically' transform this algorithm into to a parallel one > simply by inserting directives from the 'pymp' module: > > def real_dft(x): > ''' DFT for a real valued sequence x ''' > ''' parallelized ''' > r = [] > N = len(x) > M = N//2 + 1 if N%2 else N//2 > with Pool() as pool: > for n in pool.parallel(range(M)): > s = 0j > for k in range(N): > tmp = 2*pi*k*n/N > s += x[k] * (cos(tmp) - 1j*sin(tmp)) > with pool.ordered(): > r.append(s) > return r > > The idea is that 'parallelizing' a sequential algorithm like this is much > easier than writing a parallel one from scratch using the abstractions in > threading or multiprocessing. > > > Sturla Molden Interesting - this is a slightly more extreme series of changes then I was thinking, a good way to approach this would not to be a patch against python-core (unless you find bugs) but rather as a separate package hosted outside of core and posted to pypi. I think it has merit - but would require more use/eyeballs on it than just a few of us. -jesse From santagada at gmail.com Sat Jan 24 22:08:27 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 24 Jan 2009 19:08:27 -0200 Subject: [Python-ideas] optimized VM ideas In-Reply-To: References: <1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com> Message-ID: <715FC3DA-61A5-45C0-A679-C63BFB9073E6@gmail.com> You should probably at least use a recognized python test suite like pystones or richards, or use some real world applications for this. look at the benchmarks for the pypy project for example: http://tuatara.cs.uni-duesseldorf.de/benchmark.html Good luck on you endeavor On Jan 24, 2009, at 2:45 AM, joe wrote: > Ok, I ran a little test script through CodeAnalyst. basically it is > just a loop that does member lookups and function calls (this is in > py2.5 btw, will have to install 3.x and test that too). > > The results were kindof interesting. CodeAnalyst is sample-based, so > I don't know if it's completely reliable. It is usually reasonably > correct though. > > Much of the time was spent in PyEval_EvalFrameEx (which makes sense, > it being the code that interprets and executes the bytecode). 14633 > samples were spent in it, of which around 2000 were spent in what > appears to be the switch jump lookup (I'm not sure, I'm not the best > at reading assembly code). After PyEval_EvalFrameEx, the four next > most expensive calls were PyWrapper_New (at 2879 samples) > PyInstance_New (at 1418) PyDict_GetItem (at 1298 samples) and > PyObject_SetAttr (at 1155). > > Here's the (rather hastily thrown together) script: > > class A: > t = 0 > > class Bleh: > a = 0 > b = 0 > c = 0 > > def __init__(self): > self.a = A() > self.b = [] > self.c = "" > > def doit(self, no=0): > if no: return > > while 1: > a = self.a > b = self.b > c = self.c > self.a = a > self.b = b > self.c = c > self.doit(1) > b = Bleh() > b.doit() > > So, it looks like indirect or direct threading could help (if I'm > reading the results and asm code right). Also the attribute lookups > did appear to take a significant portion of the time, as did > PyInstance_New and PyWrapper_New (need to look up what those last two > do). This is just preliminary though, that isn't a particularly good > script for profiling, I suspect. > > Joe -- Leonardo Santagada santagada at gmail.com From grosser.meister.morti at gmx.net Sun Jan 25 00:48:20 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-15?Q?Mathias_Panzenb=F6ck?=) Date: Sun, 25 Jan 2009 00:48:20 +0100 Subject: [Python-ideas] contextmanager for locale Message-ID: <497BA8C4.7030702@gmx.net> When I generate certain files I have to write a date formated as english text, but the error messages (if there are any) I'd like to keep in the system language. I propose to add this (or a similar) contextmanager: @contextmanager def localectx(category, newlocale=None): oldlocale = locale.getlocale(category) locale.setlocale(category, newlocale) try: yield finally: locale.setlocale(category, oldlocale) Example usage: with localectx(locale.LC_TIME,('en','UTF8')): sdate = time.strftime('%a %b %d %Y') spec_file.write(''' %%changelog * %s Maintainer Name ''' % sdate) -panzi From grosser.meister.morti at gmx.net Sun Jan 25 00:53:59 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sun, 25 Jan 2009 00:53:59 +0100 Subject: [Python-ideas] contextmanager for locale In-Reply-To: <497BA8C4.7030702@gmx.net> References: <497BA8C4.7030702@gmx.net> Message-ID: <497BAA17.20108@gmx.net> Off topic: Here would be a with-expression or inline with (like the inline if) handy: spec_file.write(''' %%changelog * %s Maintainer Name ''' % (time.strftime('%a %b %d %Y') with localectx(locale.LC_TIME,('en','UTF8')))) However, this would be a very long line. Not very clean code. -panzi From santagada at gmail.com Sat Jan 24 22:04:35 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 24 Jan 2009 19:04:35 -0200 Subject: [Python-ideas] optimized VM ideas In-Reply-To: References: <1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com> Message-ID: <40C4AE64-4D37-49FC-93FC-87915B135E57@gmail.com> On Jan 24, 2009, at 1:53 AM, joe wrote: > On Fri, Jan 23, 2009 at 2:36 AM, Leonardo Santagada > wrote: >> >> I don't think TraceMonkey is slower than V8 I believe the last time >> I looked >> TraceMonkey was faster than V8 and it is becoming even faster at each >> interaction. >> >> The way TraceMonkey works reminds me a bit of Psyco, although I >> might be >> mixing it with the PyPy JIT. But talking about Psyco, why people >> don't go >> help Psyco if all they want is a JIT? It is not like the idea of >> having a >> JIT on Python is even new... Psyco was optimizing code even before >> Webkit/V8 >> existed. >> >> >>> This leads me to believe that relatively simple, more general >>> concepts in >>> VM design can have a bigger impact then specific, highly >>> complicated JIT >>> solutions, in the context of dynamic languages that can't be >>> easily typed >>> at compile time. >> >> I believe in the exact oposite, and TraceMonkey is probably one of >> the >> proofs of that... >> > Well, perhaps there's validity to both sides; V8 is still pretty fast. > I didn't know they'd improved tracemonkey that much, interesting. > Trace trees struck me as fairly different from psyco's design (it's > been a while since I looked at the papers for both though); Psyco > essentially was designed to optimized specific forms of code, while > trace trees is more generalized. Yes I confused the design of pypy jit with the psyco one, sorry. >>> So I've thought of a few ideas for a more (new) streamlined python >>> VM: >>> >>> * Simplify the cpython object model as much as possible, while still >>> allowing >>> most of the power of the current model. >> >> This would modify the language, so it might be interesting, but would >> generate something which is not Python. > > I don't think so. I didn't mean modify the python object model, I > meant use a modified version of cpython's implementation of it. The > cpython C API isn't part of the language standard, after all, and it's > kindof inefficient and complex, imho. I never messed much with the cpython implementation of it, but maybe. If I remember correctly the idea for it is to be reasonably simple so that it is easy to write extension modules and incorporate it with c programs. >> >>> * Either keep referencing counting, or experiment with some of the >>> newer >>> techniques such as pointer escaping. Object models that >>> exclusively rely >>> on cyclic GC's have many issues and are hard to get right. >> >> Don't know, but a good GC is way faster than what CPython is doing >> already, >> but maybe it is a good idea to explore some others perspectives on >> this. >> > > I disagree. I dislike GCs as a generalized solution, you don't always > need the overhead of a full GC. Reference counting + a GC seems like > a better compromise, since there's less work for it to do (or rather, > the work is spread over time in smaller amounts). A generational GC does exactly that, it spread the work of a collection around, not as fine grained as refcounting but maybe simpler to implement if in the end you also want to get rid of the GIL... but yes this is still a point that is not a decided matter... >>> * Possibly modify the bytecode to be register-based, as in >>> SquirrelFish. >>> Not sure if this is worth it with python code. >> >> Maybe it would help a bit. I don't think it would help more than >> 10% tops >> (but I am completely guessing here) > > Ah, 10% sounds like it would be worth it, actually. The simple code > generation SquirrelFish does is interesting too, it essentially > compiles the opcodes to native code. The code for simpler opcodes > (and simple execution paths in the more complex ones) are inlined in > the native code stream, while more complex opcodes are called as > functions. There is a tool on pypy to do this also, you should look at it (I always forget the name, but it would be easy to find on pypy site). >>> * Use direct threading (which is basically optimizing switch >>> statements to >>> be only one or two instructions) for the bytecode loop. >> >> The problem with this is (besides the error someone has already >> stated about >> your phrasing) that python has really complex bytecodes, so this >> would also >> only gain around 10% and it only works with compilers that accept >> goto >> labels which the MSVC for example does not (maybe there are more >> compilers >> that also doesn't). > > Python's bytecode isn't all that complex, when I looked at it. It's > not that much worse then Squirrelfish's original bytecode > specification (which I need to look at again, btw, not sure what > they're doing now). I was kindof surprised, thought it'd be much > worse. There was a discussion about this on pypy-dev only a week ago, you might have some fun looking at the archives. >>> * Remove string lookups for member access entirely, and replaced >>> with a >>> system of unique identifyers. The idea is you would use a hash in >>> the >>> types to map a member id to an index. Hashing ints is faster then >>> strings, >>> and I've even thought about experimenting with using collapsed >>> arrays >>> instead >>> of hashes. Of course, the design would still need to support string >>> lookups >>> when necessary. I've thought about this a lot, and I think you'd >>> need >>> the >>> same general idea as V8's hidden classes for this to work right >>> (though >>> instead of classes, it'd just be member/unique id lookup maps). >> >> A form of hidden classes is already part of PyPy (but I think that >> only the >> jit does this). But you can simply remove string lookups as people >> can >> implement special methods to track this on the current Python. As I >> said >> before I don't believe changing the semantics of python for the >> sake of >> performance is even possible. > > Obviously you'd have to go through the effort to *not* change the > language semantics, which would mean still allowing things like > __get/setattr__ and __get/setattribute__, a working __dict__, etc. Thats one of the reason I think that a JIT is probably the only answer for performance in python. At any time you could appen any of the special methods to a class... >>> I'm not sure I'll have the time to anytime soon to prototype these >>> ideas, >>> but I >>> thought I'd kick them out there and see what people say. Note, >>> I'm in no >>> way >>> suggesting any sort of change to the existing cpython VM (it's >>> way, way >>> too >>> early for that kind of talk). >> >> If you are not talking about changing CPython VM why not look at >> Psyco and >> PyPy? :) > > I looked at Psyco. It didn't look like it had much further potential > to me. It only optimizes certain situations; it's not very > generalized. PyPy looks interesting though. Psyco optimizes a lot of situations, the next version I gathered is going to optimize generator expressions, but yes, pypy I guess is the long term answer to performance and python. ps1: I thought you just forgot to send this to the list so I am replying to it, hope you don't mind. ps2: I'm not part of pypy core team or anything, so my view is just, well, my view. -- Leonardo Santagada santagada at gmail.com From ben+python at benfinney.id.au Sun Jan 25 23:11:10 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 26 Jan 2009 09:11:10 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library Message-ID: <87wscj11fl.fsf@benfinney.id.au> Howdy all, I am preparing a PEP, and corresponding reference implementation, for a standard implementation of the steps needed to turn a program into a well-behaved Unix daemon. This message is a call for comments, support, and suggestions, prior to submitting this PEP to the editor. :PEP: XXX :Title: Standard daemon process library :Version: 0.1 :Last-Modified: 2009-01-26 08:44 :Author: Ben Finney :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 2009-01-26 :Python-Version: 3.1 :Post-History: ======== Abstract ======== Writing a program to become a well-behaved Unix daemon is somewhat complex and tricky to get right, yet the steps are largely similar for any daemon regardless of what else the program may need to do. This PEP introduces a module to the Python standard library that provides a simple interface to the task of becoming a daemon process. .. contents:: .. Table of Contents: Abstract Specification Example usage Interface ``Daemon`` objects ``DaemonError`` objects Motivation Rationale Correct daemon behaviour Reference Implementation References Copyright ============= Specification ============= Example usage ============= Simple example of usage:: import daemon from spam import do_main_program this_daemon = daemon.Daemon() this_daemon.start() do_main_program() More complex example usage:: import os import grp import signal import daemon from spam import ( initial_program_setup, do_main_program, program_cleanup, reload_program_config, ) initial_program_setup() important_file = open('spam.data', 'w') interesting_file = open('eggs.data', 'w') this_daemon = daemon.Daemon() this_daemon.files_preserve = [important_file, interesting_file] this_daemon.working_directory = '/var/lib/foo' this_daemon.umask = 0o002 mail_gid = grp.getgrnam('mail').gr_gid this_daemon.gid = mail_gid this_daemon.terminate_callback = program_cleanup this_daemon.reload_callback = reload_program_config this_daemon.reload_signals = [signal.SIGHUP, signal.SIGUSR1] this_daemon.start() do_main_program() Interface ========= A new module, `daemon`, is added to the standard library. The module defines a class, `Daemon`, used to represent the settings for a daemon process. An exception class, `DaemonError`, is defined for exceptions raised from the module. ``Daemon`` objects ================== A `Daemon` instance represents the behaviour settings for the process when it becomes a daemon. The behaviour is customised by setting attributes on the instance, before calling the `start` method. The following attributes are defined. `files_preserve` :Default: ``None`` List of files that should *not* be closed when starting the daemon. If ``None``, all open file descriptors will be closed. Elements of the list are file descriptors (as returned by a file object's `fileno()` method) or Python `file` objects. Each specifies a file that is not to be closed during daemon start. `chroot_directory` :Default: ``None`` Full path to a directory to set as the effective root directory of the process. If ``None``, specifies that the root directory is not to be changed. `working_directory` :Default: ``'/'`` Full path of the working directory to which the process should change on daemon start. Since a filesystem cannot be unmounted if a process has its current working directory on that filesystem, this should either be left at default or set to a directory that is a sensible ?home directory? for the daemon while it is running. `lockfile_directory` :Default: ``'/var/run'`` Absolute directory path to contain the daemon's lockfile. If ``None``, the lockfile behaviour for this daemon is skipped. `lockfile_name` :Default: ``None`` Base name of the lockfile for this daemon, without directory or extension. If ``None``, the name is derived from the process command line. `umask` :Default: ``0`` File access creation mask (?umask?) to set for the process on daemon start. Since a process inherits its umask from its parent process, starting the daemon will reset the umask to this value so that files are created by the daemon with access modes as it expects. `ignore_signals` :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]`` List of signals that the process should ignore (by setting the signal action to ``signal.SIG_IGN``) on daemon start. `terminate_signals` :Default: ``[signal.SIGTERM]`` List of signals that the process should interpret as a request to terminate cleanly. `terminate_callback` :Default: ``None`` Callable to invoke when the process receives any of the `terminate_signals` signals, before then terminating the process. `reload_signals` :Default: ``[signal.SIGHUP]`` List of signals that the process should interpret as a request to reload runtime configuration. `reload_callback` :Default: ``None`` Callable to invoke when the process receives any of the `reload_signals` signals. `uid` :Default: ``None`` The user ID (?uid?) value to switch the process to on daemon start. `gid` :Default: ``None`` The group ID (?gid?) value to switch the process to on daemon start. `prevent_core` :Default: ``True`` If true, prevents the generation of core files, in order to avoid leaking sensitive information from daemons run as `root`. `stdout` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stdout`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stdout` is not re-bound. `stderr` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stderr`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stderr` is not re-bound. The following methods are defined. `start()` :Return: ``None`` Start the daemon. This performs the following steps: * If the `chroot_directory` attribute is not ``None``: * Set the effective root directory of the process to that directory (via `os.chroot`). This allows running the daemon process inside a ?chroot gaol? as a means of limiting the system's exposure to rogue behaviour by the process. * If the `lockfile_directory` attribute is not ``None``: * Look in that directory for a file named '`lockfile_name`.pid'; if it exists, raise a `DaemonError` to prevent multiple instances of the daemon process. * Close all open file descriptors, excluding those listed in the `files_preserve` attribute. * Change current working directory to the path specified by the `working_directory` attribute. * Reset the file access creation mask to the value specified by the `umask` attribute. * Detach the current process into its own process group, and disassociate from any controlling terminal. This step is skipped if it is determined to be redundant: if the process was started by `init`, by `initd`, or by `inetd`. * Set signal handlers as specified by the `ignore_signals`, `terminate_signals`, `terminate_callback`, `reload_signals`, `reload_callback` attributes. * If the `prevent_core` attribute is true: * Set the resource limits for the process to prevent any core dump from the process. * Set the process uid and gid to the true uid and gid of the process, to relinquish any elevated privilege. * If the `lockfile_directory` attribute is not ``None``: * Create the lockfile for this daemon in that directory, by writing a text line containing the current process ID (?pid?) to a file named '`lockfile_name`.pid'. * If either of the attributes `uid` or `gid` are not ``None``: * Set the process uid and/or gid to the specified values. * If either of the attributes `stdout` or `stderr` are not ``None``: * Bind the names `sys.stdout` and/or `sys.stderr` to the corresponding objects. `reload()` :Return: ``None`` Reload the daemon configuration. The meaning of this is entirely defined by the customisation of this daemon: if the `reload_callback` attribute is not ``None``, call that object. The return value is discarded. `stop()` :Return: ``None`` Stop the daemon. This performs the following steps: * If the `terminate_callback` attribute is not ``None``: * Call that object. The return value is discarded. * If the `lockfile_directory` attribute is not ``None``: * Delete the lockfile for this daemon. * Raise a `SystemExit` exception. ``DaemonError`` objects ======================= The `DaemonError` class inherits from `Exception`. The module implementation will raise an instance of `DaemonError` when an error occurs in processing daemon behaviour. ========== Motivation ========== The majority of programs written to be Unix daemons either implement behaviour very similar to that in the `Specification`_, or are poorly-behaved daemons by the `Rationale`_. Since these steps should be much the same in most implementations but are very particular and easy to omit or implement incorrectly, they are a prime target for a standard well-tested implementation in the standard library. ========= Rationale ========= Correct daemon behaviour ======================== According to Stevens in [stevens]_ ?2.6, a program should perform the following steps to become a Unix daemon process. * Close all open file descriptors. * Change current working directory. * Reset the file access creation mask. * Run in the background. * Disassociate from process group. * Ignore terminal I/O signals. * Disassociate from control terminal. * Don't reacquire a control terminal. * Correctly handle the following circumstances: * Started by System V `init` process. * Daemon termination by ``SIGTERM`` signal. * Children generate ``SIGCLD`` signal. The `daemon` package [daemon]_ lists (in its summary of features) behaviour that should be performed when turning a program into a well-behaved Unix daemon process. The following are appropriate for a daemon started once the program is already running: * Sets up the correct process context for a daemon. * Behaves sensibly when started by `initd(8)` or `inetd(8)`. * Revokes any suid or sgid privileges to reduce security risks in case daemon is incorrectly installed with special privileges. * Prevents the generation of core files to prevent leaking sensitive information from daemons run as root (optional). * Names the daemon by creating and locking a PID file to guarantee that only one daemon with the given name can execute at any given time (optional). * Sets the user and group under which to run the daemon (optional, root only). * Creates a chroot gaol (optional, root only). * Captures the daemon's stdout and stderr and directs them to syslog (optional). ======================== Reference Implementation ======================== The `python-daemon` package [python-daemon]_. As of 2009-01-26, the package is under active development and is not yet a full implementation of this PEP. ========== References ========== .. [stevens] `Unix Network Programming`, W. Richard Stevens, 1994 Prentice Hall. .. [daemon] The (non-Python) `daemon` package, ``_ by ?raf? . .. [python-daemon] The `python-daemon` package at the Python Package Index, ``_. This is the successor to [bda.daemon]_. .. [bda.daemon] The `bda.daemon` package at the Python Package Index, ``_. This is an implementation of [cookbook-66012]_. .. [cookbook-66012] Many good ideas were contributed by the community to Python cookbook recipe 66012, ?Fork a daemon process on Unix? ``_. ========= Copyright ========= This work is hereby placed in the public domain. To the extent that placing a work in the public domain is not legally possible, the copyright holder hereby grants to all recipients of this work all rights and freedoms that would otherwise be restricted by copyright. -- \ ?Pinky, are you pondering what I'm pondering?? ?Well, I think | `\ so (hiccup), but Kevin Costner with an English accent?? ?_Pinky | _o__) and The Brain_ | Ben Finney From tjreedy at udel.edu Mon Jan 26 01:05:32 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jan 2009 19:05:32 -0500 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87wscj11fl.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: Ben Finney wrote: > Howdy all, > > I am preparing a PEP, and corresponding reference implementation, for > a standard implementation of the steps needed to turn a program into a > well-behaved Unix daemon. [complete spec omitted] > ======================== > Reference Implementation > ======================== > > The `python-daemon` package [python-daemon]_. > > As of 2009-01-26, the package is under active development and is not > yet a full implementation of this PEP. If I were wanting to make a *nix daemon, I would love to have this. As I understand things, you should finish the implementation, post it to PyPI, and get people to use and test it, (all of which require no approvals). When stable, propose it for the stdlib along with a credible promise to maintain it for at least 3 years. The question will then be whether this unix-only module is generally useful enough to be in the stdlib. Many good modules are not. tjr From janssen at parc.com Mon Jan 26 04:38:35 2009 From: janssen at parc.com (Bill Janssen) Date: Sun, 25 Jan 2009 19:38:35 PST Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <8637.1232941115@pippin.parc.xerox.com> Terry Reedy wrote: > The question will then be whether this unix-only module is generally > useful enough to be in the stdlib. Many good modules are not. Right. Personally, I think it's pretty easy to make a daemon on Unix. If this module supported both Unix and Windows, though, that would be interesting. Bill From andrew-pythonideas at puzzling.org Mon Jan 26 06:01:29 2009 From: andrew-pythonideas at puzzling.org (Andrew Bennetts) Date: Mon, 26 Jan 2009 16:01:29 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <8637.1232941115@pippin.parc.xerox.com> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> Message-ID: <20090126050129.GB11101@steerpike.home.puzzling.org> Bill Janssen wrote: > Terry Reedy wrote: > > > The question will then be whether this unix-only module is generally > > useful enough to be in the stdlib. Many good modules are not. > > Right. Personally, I think it's pretty easy to make a daemon on Unix. It's pretty easy, but still fiddly enough that I'd rather use a library than worry about the details myself. > If this module supported both Unix and Windows, though, that would be > interesting. That would be lovely, but even a Unix-only module would be worthwhile, IMO. -Andrew. From pyideas at rebertia.com Mon Jan 26 06:23:14 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 25 Jan 2009 21:23:14 -0800 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <20090126050129.GB11101@steerpike.home.puzzling.org> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> Message-ID: <50697b2c0901252123x7e7f2e83r72e24125bb09ec59@mail.gmail.com> On Sun, Jan 25, 2009 at 9:01 PM, Andrew Bennetts wrote: > Bill Janssen wrote: >> Terry Reedy wrote: >> >> > The question will then be whether this unix-only module is generally >> > useful enough to be in the stdlib. Many good modules are not. >> >> Right. Personally, I think it's pretty easy to make a daemon on Unix. > > It's pretty easy, but still fiddly enough that I'd rather use a library > than worry about the details myself. I strongly second this sentiment. It's such an oft-used recipe that it deserves a canonical, fully correct library implementation. >> If this module supported both Unix and Windows, though, that would be >> interesting. > > That would be lovely, but even a Unix-only module would be worthwhile, > IMO. Though it would probably improve its chances of getting added to the std lib. Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com From phd at phd.pp.ru Mon Jan 26 09:30:53 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Mon, 26 Jan 2009 11:30:53 +0300 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87wscj11fl.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <20090126083052.GB12811@phd.pp.ru> On Mon, Jan 26, 2009 at 09:11:10AM +1100, Ben Finney wrote: > .. [cookbook-66012] > > Many good ideas were contributed by the community to Python > cookbook recipe 66012, ???Fork a daemon process on Unix??? > ``_. I use this one (with some local modifications) and I'm pretty happy about it. Having such a module in the std lib would be a big plus. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From skip at pobox.com Mon Jan 26 14:04:13 2009 From: skip at pobox.com (skip at pobox.com) Date: Mon, 26 Jan 2009 07:04:13 -0600 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: References: Message-ID: <18813.46285.532475.389582@montanaro.dyndns.org> > I am preparing a PEP, and corresponding reference implementation, for > a standard implementation of the steps needed to turn a program into a > well-behaved Unix daemon. I missed Ben's original post and am now reading mail offline so I can't dredge it up, but I've used a module called daemon from PyPI. Might well be worth a look as an existing proof of concept if nothing else. -- Skip Montanaro - skip at pobox.com - http://smontanaro.dyndns.org/ From denis.spir at free.fr Mon Jan 26 19:42:08 2009 From: denis.spir at free.fr (spir) Date: Mon, 26 Jan 2009 19:42:08 +0100 Subject: [Python-ideas] import imports Message-ID: <20090126194208.47f33aad@o> Hello, Here are some ideas around import. I wish to receive comments and to learn whether they fit python. As I am not a highly experimented programmer, there may be some features or practices I do know know or do not know well. The following ideas are not dependant of each other, but as they all concern the import topic, I thought they could be grouped in a single message anyway. === import transmission === The module attribute __all__ allows defining names to be exported. I always use it for my own modules, and also sometimes define it in external modules, when absent, or adapt it to my needs. Then I'm happy with a secure "import *". I find nevertheless an issue in the common case of import "transmission" from module to module, where all imported names usually have to repeated to export. This is not only ugly, imo, but also unpracticle when a single change in an name export list has to be corrected anywhere else; there can easily be a whole chain or even a 'tree' of imports. I have found a way to deal with that, illustrated in below in a case where module M0 imports M1 and M2, which itself imports M3: *** M3 *** M3_names = [actual list of relevant names] __all__ = ["M3_names"] + M3_names *** M2 *** from M3 import * M2_names = [actual list of relevant /local/ names] + M3_names __all__ = ["M2_names"] + M2_names *** M1 *** M1_names = [actual list of relevant names] __all__ = ["M1_names"] + M1_names *** M0 *** from M1 import * from M2 import * M0_names = [actual list of relevant /local/ names] + M1_names + M2_names __all__ = ["M0_names"] + M0_names This has the advantage to avoid repetition, and allow a change in a name export list with no need of "chain correction". Still, it is only a personal trick. I wonder whether such a practice could be officialised with a kind of __imported_names__ module attribute, then understood and usable by all: from Mx import * __all__ = __imported_names__ + [actual list of relevant /local/ names] === semantics of "import *" === "import *" is not recommanded for good reasons, as a blind and risky practice. Now, when this is done consciously, kwowing that the imported module defines __all__, then the meaning is totally different, if not opposite. As a consequence, I am not happy at all with the present ambiguity. My proposal would be: * Either a slightly different syntactic form, meaning "import all names defined for export". [I would like "import **", for '*' and '**' are associated in other cases and this is all about named things. But this point is no issue.] As an additional advantage, imo important, trying this kind of import from a module that does not define __all__ would raise an exception; for instance: ImportError("Module does not define any name list for export.") * Or change the semantics of "import *" to require __all__ to be defined in the pointed module. Obviously, this raises compatibility issues that may or may not be adressed using warnings (and/or "from __future__ import import" ;-}) in a change path. === package module lookup === There is a similar issue at the package/application level: I do not know of any practice or feature that simply allows changing a filename, a directory name, or the organisation of the package's directory tree, without having then to correct all dependance information inside the concerned modules. I would love a kind of import lookup to be based on the following assumption: "The main/startup module resides in the package's root dir." Then any import lookup would automatically explore local, relative, sub directories. E basta! This would also encourage the good practice of never writing absolute imports -- which is a plague. There are probably many good ways to implement that feature clearly and efficiently. Here is proposed an example that only aims at clarifying the feature's meaning & usefulness; but does not pretend to be good. The purpose is python to know not only which modules exist, or which modules are intended to be imported, but also where actually they are -- or should be. This does not only address import from client code but also intra-module dependances. A file called __init__.py is used to identify a package as package and provide information about it. I propose to add a __dir__ attribute to a package, defined inside its __init__.py, that would hold a module directory (in the sense of 'index'). This could be implemented as a python dict holding a set of name:path bindings -- while actually it is pure information. It would be either a complement, or an alternative, to a package's __all__ attribute that lists the module names: __all__ = ["main", "compute", "userinput"] __dir__ = ["main":"/", "compute":"/tools", "userinput":"/interface/dialogs"] Note that the pathes are *relative*. The main point is that __dir__'s value would be set by python itself when it is absent from __init__.py or when it needs beeing updated after any change in the package's tree. So that the process of module lookup would be as follows (provided I properly understand it): * If current module is not part of a package: usual module lookup is performed. * If it is part of a package which __dir__ is not yet defined, perform a local tree traversal starting from the module's root dir to reference the modules, then write the result into __dir__. Use it to import. * If __dir__ is defined, try and import using it. If import fails (maybe the package's organisation has changed), while the module is referenced by __all__, update __dir__ as necessary. Then, try import again. * When intra-package import finally fails, use PYTHONPATH to try external import, as usual. Obviously, as long as the package's organisation does not change, a single traversal is necessary for all further imports during all further program executions. Intra-package imports may be faster, even if it is not the primary purpose of the proposal. Actually, this seems so simple that python aware editors may even provide this automatically: meaning maintain __dir__ inside an __init__.py file. (This is close to the notion of "project" that many editors provide.) Denis ------ la vida e estranya From guido at python.org Mon Jan 26 19:45:43 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Jan 2009 10:45:43 -0800 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87wscj11fl.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote: > I am preparing a PEP, and corresponding reference implementation, for > a standard implementation of the steps needed to turn a program into a > well-behaved Unix daemon. > > This message is a call for comments, support, and suggestions, prior > to submitting this PEP to the editor. I would like to note that there are many existing packages that provide support for daemonization (including one I wrote for Zope long ago, which still appears to be around under the name zdaemon). Twisted also seems to have one. I also recall hearing a talk at PyCon a few years ago about a descendant of my zdaemon on steroids. I have an inkling that this is one of those problems where the many solutions that are hard to compare, because they have different APIs or make different assumptions about the ultimate goals. The basics are very simple (just follow the recipe from the Stevens book), but that's a bit rough, and beyond that everyone seems to have a different idea on what use cases need to be support and what the most flexible API is to support all desirable use cases. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ben+python at benfinney.id.au Mon Jan 26 23:40:42 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Jan 2009 09:40:42 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <87ocxt1yj9.fsf@benfinney.id.au> Guido van Rossum writes: > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote: > > I am preparing a PEP, and corresponding reference implementation, > > for a standard implementation of the steps needed to turn a > > program into a well-behaved Unix daemon. > > I would like to note that there are many existing packages that > provide support for daemonization [?] Good point, thanks (and thanks to others in this thread for drawing my attention to some more that I hadn't yet found). I will make note, in the next revision, of the existing equivalent Python works of which I'm aware, and how their implementation differs from the intent of this PEP. If anyone feels I'm missing any, I would be happy to be informed. > I have an inkling that this is one of those problems where the many > solutions that are hard to compare, because they have different APIs > or make different assumptions about the ultimate goals. This suspicion could well be true. On the other hand, there already seems to be a good amount of positive support for the PEP, so I'd like to attempt to come to a sensible API that covers most use cases. Is ?python-ideas? the right forum to hash this out? Or is now the right time to subject it to the inevitable by posting to ?comp.lang.python?? -- \ ?Pinky, are you pondering what I'm pondering?? ?Well, I think | `\ so, Brain, but it's a miracle that this one grew back.? ?_Pinky | _o__) and The Brain_ | Ben Finney From guido at python.org Mon Jan 26 23:50:14 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Jan 2009 14:50:14 -0800 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87ocxt1yj9.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> <87ocxt1yj9.fsf@benfinney.id.au> Message-ID: On Mon, Jan 26, 2009 at 2:40 PM, Ben Finney wrote: > Guido van Rossum writes: > >> On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote: >> > I am preparing a PEP, and corresponding reference implementation, >> > for a standard implementation of the steps needed to turn a >> > program into a well-behaved Unix daemon. >> >> I would like to note that there are many existing packages that >> provide support for daemonization [?] > > Good point, thanks (and thanks to others in this thread for drawing my > attention to some more that I hadn't yet found). > > I will make note, in the next revision, of the existing equivalent > Python works of which I'm aware, and how their implementation differs > from the intent of this PEP. If anyone feels I'm missing any, I would > be happy to be informed. > >> I have an inkling that this is one of those problems where the many >> solutions that are hard to compare, because they have different APIs >> or make different assumptions about the ultimate goals. > > This suspicion could well be true. > > On the other hand, there already seems to be a good amount of positive > support for the PEP, so I'd like to attempt to come to a sensible API > that covers most use cases. > > Is 'python-ideas' the right forum to hash this out? Or is now the > right time to subject it to the inevitable by posting to > 'comp.lang.python'? Depends on how much bikeshedding you want. I expect that you'll get enough of that without involving c.k.py... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jseutter at gmail.com Mon Jan 26 23:55:31 2009 From: jseutter at gmail.com (Jerry Seutter) Date: Mon, 26 Jan 2009 15:55:31 -0700 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87ocxt1yj9.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> <87ocxt1yj9.fsf@benfinney.id.au> Message-ID: <2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com> It would be nice if this functionality could work on windows machine as well as unix flavors. I wrote the package at http://daemonize.sourceforge.net in frustration after seeing daemons built poorly several times. Some of the instances had been done incorrectly even after the developer had been pointed to the ActiveState page. Understanding why a daemon should be written the way it is requires a fair bit of historical knowledge that younger people occasionally do not have. Some of the recent grads hired at my workplace did not know what a daemon was. "Oh, like a web container?" Jerry Seutter On Mon, Jan 26, 2009 at 3:40 PM, Ben Finney > wrote: > Guido van Rossum writes: > > > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote: > > > I am preparing a PEP, and corresponding reference implementation, > > > for a standard implementation of the steps needed to turn a > > > program into a well-behaved Unix daemon. > > > > I would like to note that there are many existing packages that > > provide support for daemonization [?] > > Good point, thanks (and thanks to others in this thread for drawing my > attention to some more that I hadn't yet found). > > I will make note, in the next revision, of the existing equivalent > Python works of which I'm aware, and how their implementation differs > from the intent of this PEP. If anyone feels I'm missing any, I would > be happy to be informed. > > > I have an inkling that this is one of those problems where the many > > solutions that are hard to compare, because they have different APIs > > or make different assumptions about the ultimate goals. > > This suspicion could well be true. > > On the other hand, there already seems to be a good amount of positive > support for the PEP, so I'd like to attempt to come to a sensible API > that covers most use cases. > > Is 'python-ideas' the right forum to hash this out? Or is now the > right time to subject it to the inevitable by posting to > 'comp.lang.python'? > > -- > \ "Pinky, are you pondering what I'm pondering?" "Well, I think | > `\ so, Brain, but it's a miracle that this one grew back." ?_Pinky | > _o__) and The Brain_ | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 27 00:02:52 2009 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Jan 2009 15:02:52 -0800 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com> References: <87wscj11fl.fsf@benfinney.id.au> <87ocxt1yj9.fsf@benfinney.id.au> <2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com> Message-ID: Was that you in the YouTube video that was one of the first hits for "PyCon daemon" ? That was hilarious. :-) On Mon, Jan 26, 2009 at 2:55 PM, Jerry Seutter wrote: > It would be nice if this functionality could work on windows machine as well > as unix flavors. > > I wrote the package at http://daemonize.sourceforge.net in frustration after > seeing daemons built poorly several times. Some of the instances had been > done incorrectly even after the developer had been pointed to the > ActiveState page. Understanding why a daemon should be written the way it > is requires a fair bit of historical knowledge that younger people > occasionally do not have. > Some of the recent grads hired at my workplace did not know what a daemon > was. "Oh, like a web container?" > Jerry Seutter > On Mon, Jan 26, 2009 at 3:40 PM, Ben Finney > wrote: >> >> Guido van Rossum writes: >> >> > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote: >> > > I am preparing a PEP, and corresponding reference implementation, >> > > for a standard implementation of the steps needed to turn a >> > > program into a well-behaved Unix daemon. >> > >> > I would like to note that there are many existing packages that >> > provide support for daemonization [?] >> >> Good point, thanks (and thanks to others in this thread for drawing my >> attention to some more that I hadn't yet found). >> >> I will make note, in the next revision, of the existing equivalent >> Python works of which I'm aware, and how their implementation differs >> from the intent of this PEP. If anyone feels I'm missing any, I would >> be happy to be informed. >> >> > I have an inkling that this is one of those problems where the many >> > solutions that are hard to compare, because they have different APIs >> > or make different assumptions about the ultimate goals. >> >> This suspicion could well be true. >> >> On the other hand, there already seems to be a good amount of positive >> support for the PEP, so I'd like to attempt to come to a sensible API >> that covers most use cases. >> >> Is 'python-ideas' the right forum to hash this out? Or is now the >> right time to subject it to the inevitable by posting to >> 'comp.lang.python'? >> >> -- >> \ "Pinky, are you pondering what I'm pondering?" "Well, I think | >> `\ so, Brain, but it's a miracle that this one grew back." ?_Pinky | >> _o__) and The Brain_ | >> Ben Finney >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jseutter at gmail.com Tue Jan 27 00:08:18 2009 From: jseutter at gmail.com (Jerry Seutter) Date: Mon, 26 Jan 2009 16:08:18 -0700 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: References: <87wscj11fl.fsf@benfinney.id.au> <87ocxt1yj9.fsf@benfinney.id.au> <2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com> Message-ID: <2c8d48d70901261508w3183a3a9r8ff979f173cb2157@mail.gmail.com> Yeah, that was me. :) On Mon, Jan 26, 2009 at 4:02 PM, Guido van Rossum wrote: > Was that you in the YouTube video that was one of the first hits for > "PyCon daemon" ? That was hilarious. :-) > > On Mon, Jan 26, 2009 at 2:55 PM, Jerry Seutter wrote: > > It would be nice if this functionality could work on windows machine as > well > > as unix flavors. > > > > I wrote the package at http://daemonize.sourceforge.net in frustration > after > > seeing daemons built poorly several times. Some of the instances had > been > > done incorrectly even after the developer had been pointed to the > > ActiveState page. Understanding why a daemon should be written the way > it > > is requires a fair bit of historical knowledge that younger people > > occasionally do not have. > > Some of the recent grads hired at my workplace did not know what a daemon > > was. "Oh, like a web container?" > > Jerry Seutter > > On Mon, Jan 26, 2009 at 3:40 PM, Ben Finney > > > > wrote: > >> > >> Guido van Rossum writes: > >> > >> > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote: > >> > > I am preparing a PEP, and corresponding reference implementation, > >> > > for a standard implementation of the steps needed to turn a > >> > > program into a well-behaved Unix daemon. > >> > > >> > I would like to note that there are many existing packages that > >> > provide support for daemonization [?] > >> > >> Good point, thanks (and thanks to others in this thread for drawing my > >> attention to some more that I hadn't yet found). > >> > >> I will make note, in the next revision, of the existing equivalent > >> Python works of which I'm aware, and how their implementation differs > >> from the intent of this PEP. If anyone feels I'm missing any, I would > >> be happy to be informed. > >> > >> > I have an inkling that this is one of those problems where the many > >> > solutions that are hard to compare, because they have different APIs > >> > or make different assumptions about the ultimate goals. > >> > >> This suspicion could well be true. > >> > >> On the other hand, there already seems to be a good amount of positive > >> support for the PEP, so I'd like to attempt to come to a sensible API > >> that covers most use cases. > >> > >> Is 'python-ideas' the right forum to hash this out? Or is now the > >> right time to subject it to the inevitable by posting to > >> 'comp.lang.python'? > >> > >> -- > >> \ "Pinky, are you pondering what I'm pondering?" "Well, I think | > >> `\ so, Brain, but it's a miracle that this one grew back." ?_Pinky | > >> _o__) and The Brain_ | > >> Ben Finney > >> > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> http://mail.python.org/mailman/listinfo/python-ideas > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Tue Jan 27 01:45:53 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Jan 2009 11:45:53 +1100 Subject: [Python-ideas] Draft PEP (version 0.2): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <87ab9d1sqm.fsf@benfinney.id.au> Significant changes: new section ?Other daemon implementations? with brief descriptions of their relationship to this PEP. :PEP: XXX :Title: Standard daemon process library :Version: 0.2 :Last-Modified: 2009-01-27 11:28 :Author: Ben Finney :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 2009-01-26 :Python-Version: 3.1 :Post-History: ======== Abstract ======== Writing a program to become a well-behaved Unix daemon is somewhat complex and tricky to get right, yet the steps are largely similar for any daemon regardless of what else the program may need to do. This PEP introduces a module to the Python standard library that provides a simple interface to the task of becoming a daemon process. .. contents:: .. Table of Contents: Abstract Specification Example usage Interface ``Daemon`` objects ``DaemonError`` objects Motivation Rationale Correct daemon behaviour Reference Implementation Other daemon implementations References Copyright ============= Specification ============= Example usage ============= Simple example of usage:: import daemon from spam import do_main_program this_daemon = daemon.Daemon() this_daemon.start() do_main_program() More complex example usage:: import os import grp import signal import daemon from spam import ( initial_program_setup, do_main_program, program_cleanup, reload_program_config, ) initial_program_setup() important_file = open('spam.data', 'w') interesting_file = open('eggs.data', 'w') this_daemon = daemon.Daemon() this_daemon.files_preserve = [important_file, interesting_file] this_daemon.working_directory = '/var/lib/foo' this_daemon.umask = 0o002 mail_gid = grp.getgrnam('mail').gr_gid this_daemon.gid = mail_gid this_daemon.terminate_callback = program_cleanup this_daemon.reload_callback = reload_program_config this_daemon.reload_signals = [signal.SIGHUP, signal.SIGUSR1] this_daemon.start() do_main_program() Interface ========= A new module, `daemon`, is added to the standard library. The module defines a class, `Daemon`, used to represent the settings for a daemon process. An exception class, `DaemonError`, is defined for exceptions raised from the module. ``Daemon`` objects ================== A `Daemon` instance represents the behaviour settings for the process when it becomes a daemon. The behaviour is customised by setting attributes on the instance, before calling the `start` method. The following attributes are defined. `files_preserve` :Default: ``None`` List of files that should *not* be closed when starting the daemon. If ``None``, all open file descriptors will be closed. Elements of the list are file descriptors (as returned by a file object's `fileno()` method) or Python `file` objects. Each specifies a file that is not to be closed during daemon start. `chroot_directory` :Default: ``None`` Full path to a directory to set as the effective root directory of the process. If ``None``, specifies that the root directory is not to be changed. `working_directory` :Default: ``'/'`` Full path of the working directory to which the process should change on daemon start. Since a filesystem cannot be unmounted if a process has its current working directory on that filesystem, this should either be left at default or set to a directory that is a sensible ?home directory? for the daemon while it is running. `lockfile_directory` :Default: ``'/var/run'`` Absolute directory path to contain the daemon's lockfile. If ``None``, the lockfile behaviour for this daemon is skipped. `lockfile_name` :Default: ``None`` Base name of the lockfile for this daemon, without directory or suffix. If ``None``, the name is derived from the process command line. `umask` :Default: ``0`` File access creation mask (?umask?) to set for the process on daemon start. Since a process inherits its umask from its parent process, starting the daemon will reset the umask to this value so that files are created by the daemon with access modes as it expects. `ignore_signals` :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]`` List of signals that the process should ignore (by setting the signal action to ``signal.SIG_IGN``) on daemon start. `terminate_signals` :Default: ``[signal.SIGTERM]`` List of signals that the process should interpret as a request to terminate cleanly. `terminate_callback` :Default: ``None`` Callable to invoke when the process receives any of the `terminate_signals` signals, before then terminating the process. `reload_signals` :Default: ``[signal.SIGHUP]`` List of signals that the process should interpret as a request to reload runtime configuration. `reload_callback` :Default: ``None`` Callable to invoke when the process receives any of the `reload_signals` signals. `uid` :Default: ``None`` The user ID (?uid?) value to switch the process to on daemon start. `gid` :Default: ``None`` The group ID (?gid?) value to switch the process to on daemon start. `prevent_core` :Default: ``True`` If true, prevents the generation of core files, in order to avoid leaking sensitive information from daemons run as `root`. `stdout` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stdout`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stdout` is not re-bound. `stderr` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stderr`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stderr` is not re-bound. The following methods are defined. `start()` :Return: ``None`` Start the daemon. This performs the following steps: * If the `chroot_directory` attribute is not ``None``: * Set the effective root directory of the process to that directory (via `os.chroot`). This allows running the daemon process inside a ?chroot gaol? as a means of limiting the system's exposure to rogue behaviour by the process. * If the `lockfile_directory` attribute is not ``None``: * Look in that directory for a file named '`lockfile_name`.pid'; if it exists, raise a `DaemonError` to prevent multiple instances of the daemon process. * Close all open file descriptors, excluding those listed in the `files_preserve` attribute. * Change current working directory to the path specified by the `working_directory` attribute. * Reset the file access creation mask to the value specified by the `umask` attribute. * Detach the current process into its own process group, and disassociate from any controlling terminal. This step is skipped if it is determined to be redundant: if the process was started by `init`, by `initd`, or by `inetd`. * Set signal handlers as specified by the `ignore_signals`, `terminate_signals`, `reload_signals` attributes. * If the `prevent_core` attribute is true: * Set the resource limits for the process to prevent any core dump from the process. * Set the process uid and gid to the true uid and gid of the process, to relinquish any elevated privilege. * If the `lockfile_directory` attribute is not ``None``: * Create the lockfile for this daemon in that directory, by writing a text line containing the current process ID (?pid?) to a file named '`lockfile_name`.pid'. * If either of the attributes `uid` or `gid` are not ``None``: * Set the process uid and/or gid to the specified values. * If either of the attributes `stdout` or `stderr` are not ``None``: * Bind the names `sys.stdout` and/or `sys.stderr` to the corresponding objects. `reload()` :Return: ``None`` Reload the daemon configuration. The meaning of this is entirely defined by the customisation of this daemon: if the `reload_callback` attribute is not ``None``, call that object. The return value is discarded. `stop()` :Return: ``None`` Stop the daemon. This performs the following steps: * If the `terminate_callback` attribute is not ``None``: * Call that object. The return value is discarded. * If the `lockfile_directory` attribute is not ``None``: * Delete the lockfile for this daemon. * Raise a `SystemExit` exception. ``DaemonError`` objects ======================= The `DaemonError` class inherits from `Exception`. The module implementation will raise an instance of `DaemonError` when an error occurs in processing daemon behaviour. ========== Motivation ========== The majority of programs written to be Unix daemons either implement behaviour very similar to that in the `Specification`_, or are poorly-behaved daemons by the `Correct daemon behaviour`_. Since these steps should be much the same in most implementations but are very particular and easy to omit or implement incorrectly, they are a prime target for a standard well-tested implementation in the standard library. ========= Rationale ========= Correct daemon behaviour ======================== According to Stevens in [stevens]_ ?2.6, a program should perform the following steps to become a Unix daemon process. * Close all open file descriptors. * Change current working directory. * Reset the file access creation mask. * Run in the background. * Disassociate from process group. * Ignore terminal I/O signals. * Disassociate from control terminal. * Don't reacquire a control terminal. * Correctly handle the following circumstances: * Started by System V `init` process. * Daemon termination by ``SIGTERM`` signal. * Children generate ``SIGCLD`` signal. The `daemon` tool [slack-daemon]_ lists (in its summary of features) behaviour that should be performed when turning a program into a well-behaved Unix daemon process. It differs from this PEP's intent in that it invokes a *separate* program as a daemon process. The following features are appropriate for a daemon that starts itself once the program is already running: * Sets up the correct process context for a daemon. * Behaves sensibly when started by `initd(8)` or `inetd(8)`. * Revokes any suid or sgid privileges to reduce security risks in case daemon is incorrectly installed with special privileges. * Prevents the generation of core files to prevent leaking sensitive information from daemons run as root (optional). * Names the daemon by creating and locking a PID file to guarantee that only one daemon with the given name can execute at any given time (optional). * Sets the user and group under which to run the daemon (optional, root only). * Creates a chroot gaol (optional, root only). * Captures the daemon's stdout and stderr and directs them to syslog (optional). ======================== Reference Implementation ======================== The `python-daemon` package [python-daemon]_. As of 2009-01-26, the package is under active development and is not yet a full implementation of this PEP. Other daemon implementations ============================ Prior to this PEP, several existing third-party Python libraries or tools implemented some of this PEP's `correct daemon behaviour`_. The `reference implementation`_ is a fairly direct successor from the following implementations: * Many good ideas were contributed by the community to Python cookbook recipe 66012 [cookbook-66012]_. * The `bda.daemon` library [bda.daemon]_ is an implementation (by Robert Niederreiter et al) of [cookbook-66012]_. It is the predecessor of [python-daemon]_. Other Python daemon implementations that differ from this PEP: * The `zdaemon` tool [zdaemon]_ was written for the Zope project. Like [slack-daemon]_, it differs from this specification because it is used to run another program as a daemon process. * The Python library `daemon` [clapper-daemon]_ is (according to its homepage) no longer maintained. As of version 1.0.1, it implements the basic steps from [stevens]_. * The `daemonize` library [seutter-daemonize]_ also implements the basic steps from [stevens]_. * Twisted [twisted]_ includes, perhaps unsurprisingly, an implementation of a process daemonisation API that is integrated with the rest of the Twisted framework; it differs significantly from the API in this PEP. * The Python `initd` library [dagitses-initd]_, which uses [clapper-daemon]_, implements an equivalent of Unix `initd(8)` for controlling a daemon process. ========== References ========== .. [stevens] `Unix Network Programming`, W. Richard Stevens, 1994 Prentice Hall. .. [slack-daemon] The (non-Python) ?libslack? implementation of a `daemon` tool ``_ by ?raf? . .. [python-daemon] The `python-daemon` library ``_ by Ben Finney et al. .. [cookbook-66012] Python Cookbook recipe 66012, ?Fork a daemon process on Unix? ``_. .. [bda.daemon] The `bda.daemon` library ``_ by Robert Niederreiter et al. .. [zdaemon] The `zdaemon` tool ``_ by Guido van Rossum et al. .. [clapper-daemon] The `daemon` library ``_ by Brian Clapper. .. [seutter-daemonize] The `daemonize` library ``_ by Jerry Seutter. .. [twisted] The `Twisted` application framework ``_ by Glyph Lefkowitz et al. .. [dagitses-initd] The Python `initd` library ``_ by Michael Andreas Dagitses. ========= Copyright ========= This work is hereby placed in the public domain. To the extent that placing a work in the public domain is not legally possible, the copyright holder hereby grants to all recipients of this work all rights and freedoms that would otherwise be restricted by copyright. -- \ ?I went to a garage sale. ?How much for the garage?? ?It's not | `\ for sale.?? ?Steven Wright | _o__) | Ben Finney From ben+python at benfinney.id.au Tue Jan 27 01:51:16 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Jan 2009 11:51:16 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> Message-ID: <8763k11shn.fsf@benfinney.id.au> Andrew Bennetts writes: > Bill Janssen wrote: > > If this module supported both Unix and Windows, though, that would > > be interesting. > > That would be lovely, but even a Unix-only module would be > worthwhile, IMO. Jerry Seutter writes: > It would be nice if this functionality could work on windows machine > as well as unix flavors. My knowledge of programming on Windows is quite limited. AIUI, the steps required to make a Windows program into a daemon process are markedly different from the canonical Stevens procedure, correct? In that case, how would an implementation on Windows need to change the API given in this PEP? -- \ ?I wish there was a knob on the TV to turn up the intelligence. | `\ There's a knob called ?brightness? but it doesn't work.? | _o__) ?Eugene P. Gallagher | Ben Finney From lie.1296 at gmail.com Mon Jan 26 09:41:33 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Mon, 26 Jan 2009 08:41:33 +0000 (UTC) Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <50697b2c0901252123x7e7f2e83r72e24125bb09ec59@mail.gmail.com> Message-ID: On Sun, 25 Jan 2009 21:23:14 -0800, Chris Rebert wrote: >>> If this module supported both Unix and Windows, though, that would be >>> interesting. >> >> That would be lovely, but even a Unix-only module would be worthwhile, >> IMO. > > Though it would probably improve its chances of getting added to the std > lib. > > Cheers, > Chris I think you could have a much better chance if you revise the PEP proposal by also considering Windows' daemon (i.e. Windows Service). Make it so that Unix's and Unix-like's daemon and Windows Service have as similar interface as possible (then create the implementation, caveat: "it's not going to be me"). From ben+python at benfinney.id.au Tue Jan 27 01:58:13 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Jan 2009 11:58:13 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <871vup1s62.fsf@benfinney.id.au> Terry Reedy writes: > If I were wanting to make a *nix daemon, I would love to have this. I'm glad to have many comments like this in support of this PEP. Thanks, everyone! > As I understand things, you should finish the implementation, post > it to PyPI, and get people to use and test it, (all of which require > no approvals). Sure, but before I ?finish? the implementation, I want to reach consensus on the suitability of the proposed API. To quote PEP 1: It is better to finish the specification and rationale first and reach consensus on it before writing code. > When stable, propose it for the stdlib along with a credible promise > to maintain it for at least 3 years. My intention is to continue supporting this code after inclusion in the standard library, yes. -- \ ?Liberty, n. One of imagination's most precious possessions.? | `\ ?Ambrose Bierce, _The Devil's Dictionary_, 1906 | _o__) | Ben Finney From alan.gauld at btinternet.com Mon Jan 26 23:38:24 2009 From: alan.gauld at btinternet.com (Alan Gauld) Date: Mon, 26 Jan 2009 22:38:24 -0000 Subject: [Python-ideas] import imports References: <20090126194208.47f33aad@o> Message-ID: "spir" wrote > Here are some ideas around import. I wish to receive comments > The module attribute __all__ allows defining names to be exported. > I always use it for my own modules, and also sometimes define it in > external modules, when absent, or adapt it to my needs. I've never defined it in any of my modules. It encourages people to use from X import * which is nearly always a bad idea. > Then I'm happy with a secure "import *". I'm not sure what you mean by a secure import *. The only danger is that of name collisions and limiting the exported names only helps by a tiny amount. There is still a good chance that another module will have the same name exported. > ...unpracticle when a single change in an name export list has > to be corrected anywhere else; there can easily be a whole > chain or even a 'tree' of imports. Exactly, thats why its a bad idea IMHO. You can never predict what other modules your module might be used with and to start customising other peoples modules to your own envirohnment leads to madness in my view! > "import *" is not recommanded for good reasons, as a > blind and risky practice. Now, when this is done consciously, > knowing that the imported module defines __all__, then > the meaning is totally different, if not opposite. Not at all. Two modules could still have conflicting names. I don;t see how you can avoid it. Thats why using import foo is better. import foo also has the advantage of forcing code to explicitly flag where imported functions and classes are found. This makes debugging and maintence far easier. import * is a "bad idea" for all sorts of reasons. Namespaces are powerful tools - lets use them. > === package module lookup === > > There is a similar issue at the package/application level: > I do not know of any practice or feature that simply allows > changing a filename, a directory name, or the organisation > of the package's directory tree, without having then to > correct all dependance information inside the concerned modules. This is a more valid issue although if you always import both package and subpackages directly and especially if you use aliases to protect your code references from change import foo as f means even if foo changes name you only need to change the import statement not all the references to f.x > I would love a kind of import lookup to be based on the following > assumption: > "The main/startup module resides in the package's root dir." But most packages don;t have a startup or main module. They are libraries that the application writer uses. Thus the app writer provides the main or startup module not the package writer. > This would also encourage the good practice of never > writing absolute imports -- which is a plague. I'm not sure what you mean by absolute imports. If you mean giving the physical path to the file then You cannot really do that in Python. I don't understand the comment? > The purpose is python to know not only which modules exist, > or which modules are intended to be imported, but also > where actually they are -- or should be. I thought it did that already? It looks in sys.path for modules and when it finds them it sets the __file__ attribute if appropriate.. I got lost around here. I assume you are having problems with modules that I've never encountered? -- Alan Gauld Author of the Learn to Program web site http://www.freenetpages.co.uk/hp/alan.gauld From stephen at xemacs.org Tue Jan 27 03:13:18 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 27 Jan 2009 11:13:18 +0900 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <871vup1s62.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> <871vup1s62.fsf@benfinney.id.au> Message-ID: <871vupfqdd.fsf@xemacs.org> Ben Finney writes: > Sure, but before I "finish" the implementation, I want to reach > consensus on the suitability of the proposed API. To quote PEP 1: > > It is better to finish the specification and rationale first and > reach consensus on it before writing code. The people who are posting "show us the code" are mostly folks with long experience in the PEP process. I think you should take it as an indication that in this case the definition of "suitable API" is an "I'll know it when I've tried it" kind of thing. What you might find useful is to avoid writing any code yourself, but see how well your proposed spec fits the existing code, of which several (potential) examples have been posted. From janssen at parc.com Tue Jan 27 07:46:23 2009 From: janssen at parc.com (Bill Janssen) Date: Mon, 26 Jan 2009 22:46:23 PST Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <8763k11shn.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> Message-ID: <21108.1233038783@pippin.parc.xerox.com> Ben Finney wrote: > My knowledge of programming on Windows is quite limited. AIUI, the > steps required to make a Windows program into a daemon process are > markedly different from the canonical Stevens procedure, correct? > > In that case, how would an implementation on Windows need to change > the API given in this PEP? Yes, that's the question. You should figure out the answer, or find some more folks who can help you figure out the answer, and update the PEP to reflect it. Bill From ben+python at benfinney.id.au Tue Jan 27 08:53:36 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Jan 2009 18:53:36 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> Message-ID: <87wschyykf.fsf@benfinney.id.au> Bill Janssen writes: > Ben Finney wrote: > > > [?] how would an implementation on Windows need to change the API > > given in this PEP? > > Yes, that's the question. You should figure out the answer, or find > some more folks who can help you figure out the answer, and update > the PEP to reflect it. Based on what I can see in the Python Cookbook (e.g. ), Windows doesn't even call it a ?daemon?, instead calling it a ?service?. The customisation options and expected behaviour are wildly different from what a Unix programmer would expect. So I think that a module for creating Unix-style daemons, with API and options geared toward making that easy, has no business also getting involved with creating Windows services. It would result in an unduly complicated API for no benefit to either Unix programmers or Windows programmers. If someone else wants to offer a separate PEP to include a ?winservice? module to the standard library, that seems a better way forward for that use case. -- \ ?I'm beginning to think that life is just one long Yoko Ono | `\ album; no rhyme or reason, just a lot of incoherent shrieks and | _o__) then it's over.? ?Ian Wolff | Ben Finney From ben+python at benfinney.id.au Tue Jan 27 09:19:01 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Jan 2009 19:19:01 +1100 Subject: [Python-ideas] Draft PEP (version 0.3): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <87r62pyxe2.fsf@benfinney.id.au> Significant changes in this version: allow all configuration options to be set either via Daemon constructor arguments or attribute assignment. Thanks to Mike Meyer. :PEP: XXX :Title: Standard daemon process library :Version: 0.3 :Last-Modified: 2009-01-27 19:13 :Author: Ben Finney :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 2009-01-26 :Python-Version: 3.1 :Post-History: ======== Abstract ======== Writing a program to become a well-behaved Unix daemon is somewhat complex and tricky to get right, yet the steps are largely similar for any daemon regardless of what else the program may need to do. This PEP introduces a module to the Python standard library that provides a simple interface to the task of becoming a daemon process. .. contents:: .. Table of Contents: Abstract Specification Example usage Interface ``Daemon`` objects ``DaemonError`` objects Motivation Rationale Correct daemon behaviour Reference Implementation Other daemon implementations References Copyright ============= Specification ============= Example usage ============= Simple example of usage:: import daemon from spam import do_main_program this_daemon = daemon.Daemon() this_daemon.start() do_main_program() More complex example usage:: import os import grp import signal import daemon from spam import ( initial_program_setup, do_main_program, program_cleanup, reload_program_config, ) initial_program_setup() important_file = open('spam.data', 'w') interesting_file = open('eggs.data', 'w') this_daemon = daemon.Daemon( working_directory='/var/lib/foo', umask=0o002, terminate_callback=program_cleanup, reload_callback=reload_program_config, reload_signals=[signal.SIGHUP, signal.SIGUSR1], ) this_daemon.files_preserve = [important_file, interesting_file] mail_gid = grp.getgrnam('mail').gr_gid this_daemon.gid = mail_gid this_daemon.start() do_main_program() Interface ========= A new module, `daemon`, is added to the standard library. The module defines a class, `Daemon`, used to represent the settings and controls for a daemon process. An exception class, `DaemonError`, is defined for exceptions raised from the module. ``Daemon`` objects ================== A `Daemon` instance represents the behaviour settings for the process when it becomes a daemon. The behaviour is customised by setting options on the instance, before calling the `start` method. Each option can be passed as a keyword argument to the `Daemon` constructor, or subsequently altered by assigning to an attribute on the instance at any time prior to calling `start`. That is, for an option named `wibble`, the following invocation:: foo = daemon.Daemon(wibble=bar) foo.start() is equivalent to:: foo = daemon.Daemon() foo.wibble = bar foo.start() The following options are defined. `files_preserve` :Default: ``None`` List of files that should *not* be closed when starting the daemon. If ``None``, all open file descriptors will be closed. Elements of the list are file descriptors (as returned by a file object's `fileno()` method) or Python `file` objects. Each specifies a file that is not to be closed during daemon start. `chroot_directory` :Default: ``None`` Full path to a directory to set as the effective root directory of the process. If ``None``, specifies that the root directory is not to be changed. `working_directory` :Default: ``'/'`` Full path of the working directory to which the process should change on daemon start. Since a filesystem cannot be unmounted if a process has its current working directory on that filesystem, this should either be left at default or set to a directory that is a sensible ?home directory? for the daemon while it is running. `lockfile_directory` :Default: ``'/var/run'`` Absolute directory path to contain the daemon's lockfile. If ``None``, the lockfile behaviour for this daemon is skipped. `lockfile_name` :Default: ``None`` Base name of the lockfile for this daemon, without directory or suffix. If ``None``, the name is derived from the process command line. `umask` :Default: ``0`` File access creation mask (?umask?) to set for the process on daemon start. Since a process inherits its umask from its parent process, starting the daemon will reset the umask to this value so that files are created by the daemon with access modes as it expects. `ignore_signals` :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]`` List of signals that the process should ignore (by setting the signal action to ``signal.SIG_IGN``) on daemon start. `terminate_signals` :Default: ``[signal.SIGTERM]`` List of signals that the process should interpret as a request to terminate cleanly. `terminate_callback` :Default: ``None`` Callable to invoke when the process receives any of the `terminate_signals` signals, before then terminating the process. `reload_signals` :Default: ``[signal.SIGHUP]`` List of signals that the process should interpret as a request to reload runtime configuration. `reload_callback` :Default: ``None`` Callable to invoke when the process receives any of the `reload_signals` signals. `uid` :Default: ``None`` The user ID (?uid?) value to switch the process to on daemon start. `gid` :Default: ``None`` The group ID (?gid?) value to switch the process to on daemon start. `prevent_core` :Default: ``True`` If true, prevents the generation of core files, in order to avoid leaking sensitive information from daemons run as `root`. `stdout` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stdout`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stdout` is not re-bound. `stderr` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stderr`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stderr` is not re-bound. The following methods are defined. `start()` :Return: ``None`` Start the daemon. This performs the following steps: * If the `chroot_directory` attribute is not ``None``: * Set the effective root directory of the process to that directory (via `os.chroot`). This allows running the daemon process inside a ?chroot gaol? as a means of limiting the system's exposure to rogue behaviour by the process. * If the `lockfile_directory` attribute is not ``None``: * Look in that directory for a file named '`lockfile_name`.pid'; if it exists, raise a `DaemonError` to prevent multiple instances of the daemon process. * Close all open file descriptors, excluding those listed in the `files_preserve` attribute. * Change current working directory to the path specified by the `working_directory` attribute. * Reset the file access creation mask to the value specified by the `umask` attribute. * Detach the current process into its own process group, and disassociate from any controlling terminal. This step is skipped if it is determined to be redundant: if the process was started by `init`, by `initd`, or by `inetd`. * Set signal handlers as specified by the `ignore_signals`, `terminate_signals`, `reload_signals` attributes. * If the `prevent_core` attribute is true: * Set the resource limits for the process to prevent any core dump from the process. * Set the process uid and gid to the true uid and gid of the process, to relinquish any elevated privilege. * If the `lockfile_directory` attribute is not ``None``: * Create the lockfile for this daemon in that directory, by writing a text line containing the current process ID (?pid?) to a file named '`lockfile_name`.pid'. * If either of the attributes `uid` or `gid` are not ``None``: * Set the process uid and/or gid to the specified values. * If either of the attributes `stdout` or `stderr` are not ``None``: * Bind the names `sys.stdout` and/or `sys.stderr` to the corresponding objects. `reload()` :Return: ``None`` Reload the daemon configuration. The meaning of this is entirely defined by the customisation of this daemon: if the `reload_callback` attribute is not ``None``, call that object. The return value is discarded. `stop()` :Return: ``None`` Stop the daemon. This performs the following steps: * If the `terminate_callback` attribute is not ``None``: * Call that object. The return value is discarded. * If the `lockfile_directory` attribute is not ``None``: * Delete the lockfile for this daemon. * Raise a `SystemExit` exception. ``DaemonError`` objects ======================= The `DaemonError` class inherits from `Exception`. The module implementation will raise an instance of `DaemonError` when an error occurs in processing daemon behaviour. ========== Motivation ========== The majority of programs written to be Unix daemons either implement behaviour very similar to that in the `specification`_, or are poorly-behaved daemons by the `correct daemon behaviour`_. Since these steps should be much the same in most implementations but are very particular and easy to omit or implement incorrectly, they are a prime target for a standard well-tested implementation in the standard library. ========= Rationale ========= Correct daemon behaviour ======================== According to Stevens in [stevens]_ ?2.6, a program should perform the following steps to become a Unix daemon process. * Close all open file descriptors. * Change current working directory. * Reset the file access creation mask. * Run in the background. * Disassociate from process group. * Ignore terminal I/O signals. * Disassociate from control terminal. * Don't reacquire a control terminal. * Correctly handle the following circumstances: * Started by System V `init` process. * Daemon termination by ``SIGTERM`` signal. * Children generate ``SIGCLD`` signal. The `daemon` tool [slack-daemon]_ lists (in its summary of features) behaviour that should be performed when turning a program into a well-behaved Unix daemon process. It differs from this PEP's intent in that it invokes a *separate* program as a daemon process. The following features are appropriate for a daemon that starts itself once the program is already running: * Sets up the correct process context for a daemon. * Behaves sensibly when started by `initd(8)` or `inetd(8)`. * Revokes any suid or sgid privileges to reduce security risks in case daemon is incorrectly installed with special privileges. * Prevents the generation of core files to prevent leaking sensitive information from daemons run as root (optional). * Names the daemon by creating and locking a PID file to guarantee that only one daemon with the given name can execute at any given time (optional). * Sets the user and group under which to run the daemon (optional, root only). * Creates a chroot gaol (optional, root only). * Captures the daemon's stdout and stderr and directs them to syslog (optional). ======================== Reference Implementation ======================== The `python-daemon` package [python-daemon]_. As of 2009-01-26, the package is under active development and is not yet a full implementation of this PEP. Other daemon implementations ============================ Prior to this PEP, several existing third-party Python libraries or tools implemented some of this PEP's `correct daemon behaviour`_. The `reference implementation`_ is a fairly direct successor from the following implementations: * Many good ideas were contributed by the community to Python cookbook recipe 66012 [cookbook-66012]_. * The `bda.daemon` library [bda.daemon]_ is an implementation of [cookbook-66012]_. It is the predecessor of [python-daemon]_. Other Python daemon implementations that differ from this PEP: * The `zdaemon` tool [zdaemon]_ was written for the Zope project. Like [slack-daemon]_, it differs from this specification because it is used to run another program as a daemon process. * The Python library `daemon` [clapper-daemon]_ is (according to its homepage) no longer maintained. As of version 1.0.1, it implements the basic steps from [stevens]_. * The `daemonize` library [seutter-daemonize]_ also implements the basic steps from [stevens]_. * Twisted [twisted]_ includes, perhaps unsurprisingly, an implementation of a process daemonisation API that is integrated with the rest of the Twisted framework; it differs significantly from the API in this PEP. * The Python `initd` library [dagitses-initd]_, which uses [clapper-daemon]_, implements an equivalent of Unix `initd(8)` for controlling a daemon process. ========== References ========== .. [stevens] `Unix Network Programming`, W. Richard Stevens, 1994 Prentice Hall. .. [slack-daemon] The (non-Python) ?libslack? implementation of a `daemon` tool ``_ by ?raf? . .. [python-daemon] The `python-daemon` library ``_ by Ben Finney et al. .. [cookbook-66012] Python Cookbook recipe 66012, ?Fork a daemon process on Unix? ``_. .. [bda.daemon] The `bda.daemon` library ``_ by Robert Niederreiter et al. .. [zdaemon] The `zdaemon` tool ``_ by Guido van Rossum et al. .. [clapper-daemon] The `daemon` library ``_ by Brian Clapper. .. [seutter-daemonize] The `daemonize` library ``_ by Jerry Seutter. .. [twisted] The `Twisted` application framework ``_ by Glyph Lefkowitz et al. .. [dagitses-initd] The Python `initd` library ``_ by Michael Andreas Dagitses. ========= Copyright ========= This work is hereby placed in the public domain. To the extent that placing a work in the public domain is not legally possible, the copyright holder hereby grants to all recipients of this work all rights and freedoms that would otherwise be restricted by copyright. -- \ ?Program testing can be a very effective way to show the | `\ presence of bugs, but is hopelessly inadequate for showing | _o__) their absence.? ?Edsger W. Dijkstra | Ben Finney From phd at phd.pp.ru Tue Jan 27 10:00:59 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 27 Jan 2009 12:00:59 +0300 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87wschyykf.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> Message-ID: <20090127090058.GA302@phd.pp.ru> On Tue, Jan 27, 2009 at 06:53:36PM +1100, Ben Finney wrote: > Based on what I can see in the Python Cookbook (e.g. > ), Windows doesn't > even call it a ???daemon???, instead calling it a ???service???. The > customisation options and expected behaviour are wildly different from > what a Unix programmer would expect. Not only it's a completely different API - it requires pywin32 which is not in the stdlib. > So I think that a module for creating Unix-style daemons, with API and > options geared toward making that easy, has no business also getting > involved with creating Windows services. It would result in an unduly > complicated API for no benefit to either Unix programmers or Windows > programmers. +1 for unix-only daemonization, -1 for making a universal unix/w32 API. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From denis.spir at free.fr Tue Jan 27 10:39:05 2009 From: denis.spir at free.fr (spir) Date: Tue, 27 Jan 2009 10:39:05 +0100 Subject: [Python-ideas] import imports In-Reply-To: References: <20090126194208.47f33aad@o> Message-ID: <20090127103905.63234099@o> [Sorry, forgot to answer one of the questions in my previous reply.] Le Mon, 26 Jan 2009 22:38:24 -0000, "Alan Gauld" a ?crit : > "spir" wrote > > === package module lookup === [...] > > This would also encourage the good practice of never > > writing absolute imports -- which is a plague. > > I'm not sure what you mean by absolute imports. If you mean > giving the physical path to the file then You cannot really do > that in Python. I don't understand the comment? What I call "absolute import" happens when you give python an absolute path to look for a module (even if its your own). For sure python does not let you write an absolute path in the import instruction itself, but people do that by appending to PYTHONPATH, using 'sys.path'. Some installation instructions even explicitely tell the users to do that. More commonly, application/package authors often rely on absolute pathes for internal imports (meaning that you cannot even chose the dir's name! not to talk about the actual location...) or external ones. sys.path.append("path/to/dir") import foo Actually, for external imports, the only solutions I know are either to use environment variables, or to rely on python standard locations for packages (which are relative to python's root path, itself defined as an environment variable). For intra-package imports, there are now facilities to write relative imports directly: http://www.python.org/dev/peps/pep-0328/#rationale-for-relative-imports Denis ------ la vida e estranya From python-ideas-list at trentnelson.com Tue Jan 27 17:06:44 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Tue, 27 Jan 2009 16:06:44 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <20090127090058.GA302@phd.pp.ru> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> Message-ID: <20090127160643.GC37589@wind.teleri.net> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote: > On Tue, Jan 27, 2009 at 06:53:36PM +1100, Ben Finney wrote: > > Based on what I can see in the Python Cookbook (e.g. > > ), Windows doesn't > > even call it a ???daemon???, instead calling it a ???service???. The > > customisation options and expected behaviour are wildly different from > > what a Unix programmer would expect. > > Not only it's a completely different API - it requires pywin32 which is > not in the stdlib. > > > So I think that a module for creating Unix-style daemons, with API and > > options geared toward making that easy, has no business also getting > > involved with creating Windows services. It would result in an unduly > > complicated API for no benefit to either Unix programmers or Windows > > programmers. > > +1 for unix-only daemonization, -1 for making a universal unix/w32 API. Meh, Unix-only stuff is such a pain because most developers or projects could cobble together Unix-only stuff themselves; the same can rarely be said for Windows-equivalent stuff. It's not hard to abstract a general service interface; Qt has been doing it for a long time: http://doc.trolltech.com/solutions/qtservice/qtservice.html Trent. From phd at phd.pp.ru Tue Jan 27 17:19:33 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 27 Jan 2009 19:19:33 +0300 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <20090127160643.GC37589@wind.teleri.net> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> Message-ID: <20090127161933.GB28125@phd.pp.ru> On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote: > On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote: > > Not only it's a completely different API - it requires pywin32 which is > > not in the stdlib. [skip] > > +1 for unix-only daemonization, -1 for making a universal unix/w32 API. > > It's not hard to abstract a general service interface Do you propose to include pywin32 into python? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From mail at timgolden.me.uk Tue Jan 27 17:23:57 2009 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 27 Jan 2009 16:23:57 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <20090127161933.GB28125@phd.pp.ru> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> Message-ID: <497F351D.30700@timgolden.me.uk> Oleg Broytmann wrote: > On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote: >> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote: >>> Not only it's a completely different API - it requires pywin32 which is >>> not in the stdlib. > [skip] >>> +1 for unix-only daemonization, -1 for making a universal unix/w32 API. >> It's not hard to abstract a general service interface > > Do you propose to include pywin32 into python? Ummm.. Python already has a slew of windows-specific code in it. You could -- if it seemed suitable -- copy the pywin32 code with appropriate attribution, but you don't need the whole thing. And you don't necessarily need it at all. TJG From jnoller at gmail.com Tue Jan 27 17:30:00 2009 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 27 Jan 2009 11:30:00 -0500 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <497F351D.30700@timgolden.me.uk> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> Message-ID: <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> On Tue, Jan 27, 2009 at 11:23 AM, Tim Golden wrote: > Oleg Broytmann wrote: >> >> On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote: >>> >>> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote: >>>> >>>> Not only it's a completely different API - it requires pywin32 which >>>> is >>>> not in the stdlib. >> >> [skip] >>>> >>>> +1 for unix-only daemonization, -1 for making a universal unix/w32 >>>> API. >>> >>> It's not hard to abstract a general service interface >> >> Do you propose to include pywin32 into python? > > Ummm.. Python already has a slew of windows-specific code in > it. You could -- if it seemed suitable -- copy the pywin32 > code with appropriate attribution, but you don't need the > whole thing. And you don't necessarily need it at all. > > TJG I think adding the windows services functionality distracts from the reasonable goal of the PEP Ben is proposing. -jesse From mail at timgolden.me.uk Tue Jan 27 18:10:49 2009 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 27 Jan 2009 17:10:49 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> Message-ID: <497F4019.6080601@timgolden.me.uk> Jesse Noller wrote: > On Tue, Jan 27, 2009 at 11:23 AM, Tim Golden wrote: >> Oleg Broytmann wrote: >>> On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote: >>>> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote: >>>>> Not only it's a completely different API - it requires pywin32 which >>>>> is >>>>> not in the stdlib. >>> [skip] >>>>> +1 for unix-only daemonization, -1 for making a universal unix/w32 >>>>> API. >>>> It's not hard to abstract a general service interface >>> Do you propose to include pywin32 into python? >> Ummm.. Python already has a slew of windows-specific code in >> it. You could -- if it seemed suitable -- copy the pywin32 >> code with appropriate attribution, but you don't need the >> whole thing. And you don't necessarily need it at all. >> >> TJG > > I think adding the windows services functionality distracts from the > reasonable goal of the PEP Ben is proposing. I'm inclined to agree with you. Or at least I think a PEP proposing a cross-platform daemon/service framework would be quite different from the one Ben is authoring. I was really just commenting on Oleg's suggestion that anything making use of win32 code needed to use pywin32. TJG From phd at phd.pp.ru Tue Jan 27 18:27:17 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 27 Jan 2009 20:27:17 +0300 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <497F4019.6080601@timgolden.me.uk> References: <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <497F4019.6080601@timgolden.me.uk> Message-ID: <20090127172717.GA9501@phd.pp.ru> On Tue, Jan 27, 2009 at 05:10:49PM +0000, Tim Golden wrote: > I was really just commenting on Oleg's suggestion that anything making > use of win32 code needed to use pywin32. I haven't said that. I have said (not exactly in these words) "If you want to add an implementation of w32 services to the stdlib you must (or at least should) to add pywin32 to the stdlib." Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ben+python at benfinney.id.au Tue Jan 27 22:18:51 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2009 08:18:51 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <8637.1232941115@pippin.parc.xerox.com> <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> Message-ID: <87eiyozbus.fsf@benfinney.id.au> Jesse Noller writes: > I think adding the windows services functionality distracts from the > reasonable goal of the PEP Ben is proposing. Thanks, this is my position also. I have documented this now, and it will appear in the next version of the PEP. -- \ ?The problem with television is that the people must sit and | `\ keep their eyes glued on a screen: the average American family | _o__) hasn't time for it.? ?_The New York Times_, 1939 | Ben Finney From python-ideas-list at trentnelson.com Wed Jan 28 02:29:07 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Wed, 28 Jan 2009 01:29:07 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87eiyozbus.fsf@benfinney.id.au> References: <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> Message-ID: <20090128012906.GH57568@wind.teleri.net> On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote: > Jesse Noller writes: > > > I think adding the windows services functionality distracts from the > > reasonable goal of the PEP Ben is proposing. > > Thanks, this is my position also. I have documented this now, and it > will appear in the next version of the PEP. I disagree. Partly because I'm in a bit of a devil's advocate mood at the moment. But mostly because I know what will happen: the PEP gets approved, the daemon module gets written, and, because it's so useful, uptake is immediate, people start writing code that uses it. Sweet. Or is it? Don't get me wrong, I *heart* Unix. But I get paid to develop in Windows. Because frankly, that's where the big bucks are and it's an extremely satisfying transition to jettison all emotional reasoning and just work for whoever is willing to pay you the most. I highly recommend it. But for now, I digress. So, back to your Unix-only daemon module. It's good. Great, in fact. All sorts of projects start using it. Subversion uses it to write some funky persistent daemon that speeds up hooks by 8000% percent. Well that's just super! I'd like to use that for one of my clients, but, argh, no, wait, I can't, it's trying to use some daemon module that Google tells me is Unix-only. After poking around the source, I'm perplexed. It's not doing anything uniquely Unix-xy, just your normal, run-of-the-mill start/stop type stuff. Why on earth, I think to myself, is this great 3rd party Python utility using a Unix-only daemon module, thus requiring me to hack it to bits in order for it to run on Windows, wasting my time and my client's time. Because it was there. Trent. From jnoller at gmail.com Wed Jan 28 02:49:52 2009 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 27 Jan 2009 20:49:52 -0500 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <20090128012906.GH57568@wind.teleri.net> References: <20090126050129.GB11101@steerpike.home.puzzling.org> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> Message-ID: <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> On Tue, Jan 27, 2009 at 8:29 PM, Trent Nelson wrote: > On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote: >> Jesse Noller writes: >> >> > I think adding the windows services functionality distracts from the >> > reasonable goal of the PEP Ben is proposing. >> >> Thanks, this is my position also. I have documented this now, and it >> will appear in the next version of the PEP. > > I disagree. Partly because I'm in a bit of a devil's advocate mood > at the moment. But mostly because I know what will happen: the PEP > gets approved, the daemon module gets written, and, because it's so > useful, uptake is immediate, people start writing code that uses it. > Or is it? Don't get me wrong, I *heart* Unix. But I get paid to > develop in Windows. Because frankly, that's where the big bucks are > and it's an extremely satisfying transition to jettison all emotional > reasoning and just work for whoever is willing to pay you the most. > I highly recommend it. But for now, I digress. > Don't worry; I think most of us would gladly work for money. In fact, I would hazard most of us like money a lot. > So, back to your Unix-only daemon module. It's good. Great, in > fact. All sorts of projects start using it. Subversion uses it > to write some funky persistent daemon that speeds up hooks by > 8000% percent. Well that's just super! I'd like to use that for > one of my clients, but, argh, no, wait, I can't, it's trying to > use some daemon module that Google tells me is Unix-only. > > After poking around the source, I'm perplexed. It's not doing > anything uniquely Unix-xy, just your normal, run-of-the-mill > start/stop type stuff. Why on earth, I think to myself, is this > great 3rd party Python utility using a Unix-only daemon module, > thus requiring me to hack it to bits in order for it to run on > Windows, wasting my time and my client's time. Windows Services != Unix daemons. Admittedly, I haven't had to do it in awhile, but from what I remember, windows services are nowhere near the same type of beast as a unix daemon, and trying to make a unified daemon library that glosses over the differences might cause seizures and habitual smoking. A unix-only daemon - and then a complimenting windows-only service library - makes sense. Ideally, they would have close to the same API, but I don't think log jamming all of the vagaries of both into one beast makes sense. Then again, my windows-ability is out of date and rusty. -jesse From ben+python at benfinney.id.au Wed Jan 28 03:15:41 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2009 13:15:41 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090126050129.GB11101@steerpike.home.puzzling.org> <8763k11shn.fsf@benfinney.id.au> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> Message-ID: <87ab9cyy42.fsf@benfinney.id.au> Trent Nelson writes: > On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote: > > Jesse Noller writes: > > > > > I think adding the windows services functionality distracts from > > > the reasonable goal of the PEP Ben is proposing. > > > > Thanks, this is my position also. > > I disagree. Partly because I'm in a bit of a devil's advocate mood > at the moment. But mostly because I know what will happen: [?] I respectfully suggest that this isn't true, because your hypothetical situation continues: > After poking around the source, I'm perplexed. It's not doing > anything uniquely Unix-xy, Your disagreement seems to rest on this assertion, and I don't see how your hypothetical described can be true. Do you really see the described behaviour in the current draft PEP as ?not doing anything uniquely Unix-y?? > just your normal, run-of-the-mill start/stop type stuff. As far as I can tell, beyond trivial use cases, there is both wide discrepancy and very little overlap in ?start/stop type stuff? between a Unix daemon and an MS Windows service. At least, none of the (admittedly few) instructions on MS Windows services I've seen demonstrate otherwise. Someone who thinks otherwise is welcome to attempt a PEP that brings all that behaviour under a single banner *and* design the API to be both flexible enough to be useful in many use cases and simple enough to be worth learning instead of doing it manually. That person will not be me. -- \ ?Contentment is a pearl of great price, and whosoever procures | `\ it at the expense of ten thousand desires makes a wise and | _o__) happy purchase.? ?J. Balguy | Ben Finney From python-ideas-list at trentnelson.com Wed Jan 28 03:21:45 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Wed, 28 Jan 2009 02:21:45 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> Message-ID: <20090128022145.GC64950@wind.teleri.net> On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote: > On Tue, Jan 27, 2009 at 8:29 PM, Trent Nelson > wrote: > > On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote: > >> Jesse Noller writes: > Windows Services != Unix daemons. Admittedly, I haven't had to do it > in awhile, but from what I remember, windows services are nowhere near > the same type of beast as a unix daemon, and trying to make a unified > daemon library that glosses over the differences might cause seizures > and habitual smoking. > > A unix-only daemon - and then a complimenting windows-only service > library - makes sense. Ideally, they would have close to the same API, > but I don't think log jamming all of the vagaries of both into one > beast makes sense. Then again, my windows-ability is out of date and > rusty. See, I disagree with that as well. Take a look at what Qt did with their QtService class: http://doc.trolltech.com/solutions/qtservice/qtservice.html The Trolls are particularly good at abstracting wildly different Unix/Windows-isms into a single, usable, unified interface. The key is not thinking in terms of writing a Windows service or a Unix daemon, but in terms of providing an abstraction that makes sense to implement as a daemon on Unix and a service on Windows. I'd like to hear arguments for why that QtService interface is perceived as being deficient. If the argument is simply that the PEP author is not familiar with Windows, then I'm sure those of us that are would much rather help out with a cross-platform solution than have yet another avenue for people to write Python code that doesn't work on the most widely deployed platform in the Universe.* Trent. [*] Citation needed. From ben+python at benfinney.id.au Wed Jan 28 03:48:46 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2009 13:48:46 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <20090128022145.GC64950@wind.teleri.net> Message-ID: <8763k0ywkx.fsf@benfinney.id.au> Trent Nelson writes: > On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote: > > Windows Services != Unix daemons. Admittedly, I haven't had to do > > it in awhile, but from what I remember, windows services are > > nowhere near the same type of beast as a unix daemon, and trying > > to make a unified daemon library that glosses over the differences > > might cause seizures and habitual smoking. > > > > A unix-only daemon - and then a complimenting windows-only service > > library - makes sense. Ideally, they would have close to the same > > API, but I don't think log jamming all of the vagaries of both > > into one beast makes sense. Then again, my windows-ability is out > > of date and rusty. > > See, I disagree with that as well. Take a look at what Qt did with > their QtService class: > > http://doc.trolltech.com/solutions/qtservice/qtservice.html Thanks for mentioning that again; I saw another reference to it earlier, but didn't respond. > I'd like to hear arguments for why that QtService interface is > perceived as being deficient. Off the top of my head: The QtService model seems to be ?start an additional process and maintain it at arms length, with a bunch of communication channels into and out of it?. Contrariwise, this PEP addresses the use case of ?turn the *current* program into a daemon process, so that everything further this program does is running inside that process?. It's too big, as a result of the above difference in focus. When writing a Unix daemon program, I don't want the great majority of what that interface offers. It misses a lot, again as a result of the mismatch in purpose. After setting up a QtService, I need to continue chattering with it to find out what's going on inside it, and to do things like change its state or log a message. The current PEP, on the other hand, doesn't have the model of managing a separate process; the daemon is the *current* program, so one can just do things like output or state changes as one normally would. In short: It addresses a different problem to what I'm addressing with this PEP. -- \ ?If nature has made any one thing less susceptible than all | `\ others of exclusive property, it is the action of the thinking | _o__) power called an idea? ?Thomas Jefferson | Ben Finney From python-ideas-list at trentnelson.com Wed Jan 28 04:03:53 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Wed, 28 Jan 2009 03:03:53 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87ab9cyy42.fsf@benfinney.id.au> References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> Message-ID: <20090128030351.GD64950@wind.teleri.net> On Wed, Jan 28, 2009 at 01:15:41PM +1100, Ben Finney wrote: > Trent Nelson writes: > > > On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote: > > > Jesse Noller writes: > > > > > > > I think adding the windows services functionality distracts from > > > > the reasonable goal of the PEP Ben is proposing. > > > > > > Thanks, this is my position also. > > > > I disagree. Partly because I'm in a bit of a devil's advocate mood > > at the moment. But mostly because I know what will happen: [???] > > I respectfully suggest that this isn't true, because your hypothetical > situation continues: > > > After poking around the source, I'm perplexed. It's not doing > > anything uniquely Unix-xy, > > Your disagreement seems to rest on this assertion, and I don't see how > your hypothetical described can be true. Do you really see the > described behaviour in the current draft PEP as ???not doing anything > uniquely Unix-y???? No, the PEP is nothing but Unix-isms, that's my point. The 3rd party code I'm hypothetically wading through has been coded to that interface, but semantically, it's not doing anything unique to Unix. > `prevent_core` > :Default: ``True`` > > If true, prevents the generation of core files, in order to > avoid leaking sensitive information from daemons run as `root`. I was going to suggest service.set_secure(True); but then I wondered what is this actually achieving? Won't the resulting .core will be owned by root? >`lockfile_directory` > :Default: ``'/var/run'`` > > Absolute directory path to contain the daemon's lockfile. If > ``None``, the lockfile behaviour for this daemon is skipped. > >`lockfile_name` > :Default: ``None`` > > Base name of the lockfile for this daemon, without directory or > suffix. If ``None``, the name is derived from the process command > line. All you're doing is ensuring multiple instances of the daemon don't run. On Windows, you'd use a mutex, on Unix, sure, lockfile makes sense. A cross-platform interface wouldn't bother exposing this implementation detail. > > just your normal, run-of-the-mill start/stop type stuff. > > As far as I can tell, beyond trivial use cases, there is both wide > discrepancy and very little overlap in ???start/stop type stuff??? > between a Unix daemon and an MS Windows service. At least, none of the > (admittedly few) instructions on MS Windows services I've seen > demonstrate otherwise. Semantically the overlap is almost identical, which is why I'm interested in hearing why the QtService-type approach couldn't be leveraged here. (The implementation details differ wildly, which is what you're referring to.) > Someone who thinks otherwise is welcome to attempt a PEP that brings > all that behaviour under a single banner *and* design the API to be > both flexible enough to be useful in many use cases and simple enough > to be worth learning instead of doing it manually. That person will > not be me. class Service(object): EventType = namedtuple( Success = 1, Error = 2, Warning = 3, Information = 4, ) StartupType = namedtuple( Auto = 0, Manual = 1 ) def __init__(self, name, desc=None, startup_type=StartupType.Auto) self.name = name self.description = desc def parse_arguments(args): pass def is_running(self): pass def is_installed(self): pass def install(self): pass def uninstall(self): pass def report_event(self, msg, event_type, id, category, data): pass def exec(args): pass def terminate(): pass def request_pause(): pass def request_resume(): pass def send_command(code): pass def run(args): raise NotImplementedError() def start(): pass def stop(): pass def pause(): pass def resume(): pass }:> Trent. From ben+python at benfinney.id.au Wed Jan 28 04:24:27 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2009 14:24:27 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> Message-ID: <871vuoyuxg.fsf@benfinney.id.au> (I'm only addressing commentary on the current PEP here. Comparisons to MS Windows services have been addressed in other messages.) Trent Nelson writes: > On Wed, Jan 28, 2009 at 01:15:41PM +1100, Ben Finney wrote: > > `prevent_core` > > :Default: ``True`` > > > > If true, prevents the generation of core files, in order to > > avoid leaking sensitive information from daemons run as > > `root`. > > I was going to suggest service.set_secure(True) That name implies something emphatically not true: that a program's security is a binary toggle, and it can be turned on or off. > but then I wondered what is this actually achieving? Won't the > resulting .core will be owned by root? Yes. Which is less secure than never creating the core file at all. > >`lockfile_directory` > > :Default: ``'/var/run'`` > > > > Absolute directory path to contain the daemon's lockfile. If > > ``None``, the lockfile behaviour for this daemon is skipped. > > > >`lockfile_name` > > :Default: ``None`` > > > > Base name of the lockfile for this daemon, without directory or > > suffix. If ``None``, the name is derived from the process command > > line. > > All you're doing is ensuring multiple instances of the daemon > don't run. Also allowing the location and name of the lock file to be customised. > On Windows, you'd use a mutex, on Unix, sure, lockfile makes sense. > A cross-platform interface wouldn't bother exposing this > implementation detail. The details are exposed precisely so that they can be customised by programs that need it. This is common, for example, to allow for differing conventions or standards of filesystem layout. -- \ ?Pinky, are you pondering what I'm pondering?? ?I think so, | `\ Brain, but I find scratching just makes it worse.? ?_Pinky and | _o__) The Brain_ | Ben Finney From python-ideas-list at trentnelson.com Wed Jan 28 04:38:42 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Wed, 28 Jan 2009 03:38:42 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <871vuoyuxg.fsf@benfinney.id.au> References: <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> Message-ID: <20090128033842.GF64950@wind.teleri.net> On Wed, Jan 28, 2009 at 02:24:27PM +1100, Ben Finney wrote: > (I'm only addressing commentary on the current PEP here. Comparisons > to MS Windows services have been addressed in other messages.) (I've run out of steam trying to argue the virtues of cross- platform modules, so I'm only commenting on Ben's commentary on my commentary.) > Trent Nelson writes: > > On Wed, Jan 28, 2009 at 01:15:41PM +1100, Ben Finney wrote: > > > `prevent_core` > > > :Default: ``True`` > > > > > > If true, prevents the generation of core files, in order to > > > avoid leaking sensitive information from daemons run as > > > `root`. > > > > I was going to suggest service.set_secure(True) > > That name implies something emphatically not true: that a program's > security is a binary toggle, and it can be turned on or off. > > > but then I wondered what is this actually achieving? Won't the > > resulting .core will be owned by root? > > Yes. Which is less secure than never creating the core file at all. I must be missing something. This seems like a false sense of security. The core file is owned by root. Someone will need root access in order to read it. If someone has root access, and has malevolent intentions, it's game over, .core file or not. Trent. From python-ideas-list at trentnelson.com Wed Jan 28 04:41:05 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Wed, 28 Jan 2009 03:41:05 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library] Message-ID: <20090128034105.GG64950@wind.teleri.net> On Wed, Jan 28, 2009 at 01:48:46PM +1100, Ben Finney wrote: > Trent Nelson writes: > > > On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote: > > > Windows Services != Unix daemons. Admittedly, I haven't had to do > > > it in awhile, but from what I remember, windows services are > > > nowhere near the same type of beast as a unix daemon, and trying > > > to make a unified daemon library that glosses over the differences > > > might cause seizures and habitual smoking. > > > > > > A unix-only daemon - and then a complimenting windows-only service > > > library - makes sense. Ideally, they would have close to the same > > > API, but I don't think log jamming all of the vagaries of both > > > into one beast makes sense. Then again, my windows-ability is out > > > of date and rusty. > > > > See, I disagree with that as well. Take a look at what Qt did with > > their QtService class: > > > > http://doc.trolltech.com/solutions/qtservice/qtservice.html > > Thanks for mentioning that again; I saw another reference to it > earlier, but didn't respond. > > > I'd like to hear arguments for why that QtService interface is > > perceived as being deficient. > > Off the top of my head: > > The QtService model seems to be ???start an additional process and > maintain it at arms length, with a bunch of communication channels > into and out of it???. Contrariwise, this PEP addresses the use case of > ???turn the *current* program into a daemon process, so that everything > further this program does is running inside that process???. What are you left with at the end of the day? Something you can start, stop, query status, and log out and have it still run. How you got there (daemonising the current process) surely isn't as important as where you ended, especially if it nukes any chance of cross-platformness? > It's too big, as a result of the above difference in focus. When > writing a Unix daemon program, I don't want the great majority of what > that interface offers. So I can't persuade you from shifting focus from "When writing a Unix daemon program, I ...", to "When writing a cross-platform daemon/service, I ..."? Sidebar: this definition on the Qt page caught my attention: "A Windows service or Unix daemon (a "service"), is a program that runs regardless of whether a user is logged in or not." If you're vehemently against the notion of a cross-platform implementation that utilised Windows services; what about a Windows implementation that mimicked the Unix behaviour as best it could? That is, upon daemonisation, the Windows process would change process group or whatever such that it would keep running after you log out, until killed. Not pretty, but it avoids all the problems of Unix-only. > It misses a lot, again as a result of the mismatch in purpose. After > setting up a QtService, I need to continue chattering with it to find > out what's going on inside it, and to do things like change its state > or log a message. The current PEP, on the other hand, doesn't have the > model of managing a separate process; the daemon is the *current* > program, so one can just do things like output or state changes as one > normally would. > > In short: It addresses a different problem to what I'm addressing with > this PEP. Indeed it does, but that's 'cause I'm trying to change your focus from writing a Unix-only daemon to writing a cross-platform service that just happens to be implemented as a daemon on Unix and as a service on Windows ;-) Sure, the interface wouldn't be the same as if it were Unix-only, but that's what cross-platformness is all about, isn't it? Trent. From ben+python at benfinney.id.au Wed Jan 28 05:21:42 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2009 15:21:42 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library] References: <20090128034105.GG64950@wind.teleri.net> Message-ID: <87iqo0xdpl.fsf@benfinney.id.au> Trent Nelson writes: > What are you left with at the end of the day? Something you can > start, stop, query status, and log out and have it still run. This presupposes a point of view (?you?) that is someone *running* the program. For that point of view, this PEP is irrelevant. The audience for the current PEP, on the other hand, is a programmer *writing* such a program, who sees the API as addressing just one of the many and varied things their program needs to do. > How you got there (daemonising the current process) surely isn't > as important as where you ended I'm specifying an API, and trying to make it so people will want to use it for the purpose it's designed for. So yes, to the extent that ?how you got there? includes programming to an API, I think it is very important in this context. > So I can't persuade you from shifting focus from "When writing a > Unix daemon program, I ...", to "When writing a cross-platform > daemon/service, I ..."? I could potentially be persuaded, but the ?run an additional process as a service? model is definitely not a problem I need solved. Given that, I'm not likely to be a good champion for it. Meanwhile, the focus of this PEP *does* meet a need shared by many (?daemonise the current running program?), and I see value in keeping it focussed on that. > Sidebar: this definition on the Qt page caught my attention: > > "A Windows service or Unix daemon (a "service"), > is a program that runs regardless of whether a > user is logged in or not." I noticed that too, and it didn't strike any chord with me. It seems targeted to people who want to write an MS Windows service that also happens to run on Unix. Those people doubtless exist, and deserve to find solutions to their challenges; but I'm not one of them. > If you're vehemently against the notion of a cross-platform > implementation that utilised Windows services You are arguing against a position I've never taken. I repeat: if someone else wants what you're describing, more power to them, and I don't intend to get in their way. But it's not going to be me that writes that PEP nor the implementation for it. -- \ ?If you fell down yesterday, stand up today.? ?_The Anatomy of | `\ Frustration_, H. G. Wells, 1936 | _o__) | Ben Finney From stephen at xemacs.org Wed Jan 28 05:28:10 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Jan 2009 13:28:10 +0900 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <871vuoyuxg.fsf@benfinney.id.au> References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> Message-ID: <87k58gdpgl.fsf@xemacs.org> Ben Finney writes: > The details [of the lockfiles implementing a mutex] are exposed > precisely so that they can be customised by programs that need > it. This is common, for example, to allow for differing conventions > or standards of filesystem layout. Sure, but the question being asked is why should users of the daemon process library need to worry about that, separately from other users of lockfiles or mutexes? If they need to worry about it only because there's no satisfactory module implementing it widely available, then that could be done in a subsidiary module that handles OS-specific details, and optionally exposes OS-specific special features (a la os and os.posix). That would also make refactoring (close to) trivial when such a module did become widely available. It's obvious you don't want to go in that direction, which is IMHO good and sufficient reason for you not to do so. But as it stands IMO it's a weakness in the PEP, even if Unix-only is otherwise acceptable for the stdlib. From solipsis at pitrou.net Thu Jan 29 01:57:42 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Jan 2009 00:57:42 +0000 (UTC) Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090126050129.GB11101@steerpike.home.puzzling.org> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> Message-ID: Jesse Noller writes: > Windows Services != Unix daemons. Admittedly, I haven't had to do it > in awhile, but from what I remember, windows services are nowhere near > the same type of beast as a unix daemon, and trying to make a unified > daemon library that glosses over the differences might cause seizures > and habitual smoking. That would be like a portable library which would allow to write multiprocess programs by glossing over the differences between Unix and Windows processes... Wouldn't it? From jnoller at gmail.com Thu Jan 29 02:12:58 2009 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 28 Jan 2009 20:12:58 -0500 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: References: <20090126050129.GB11101@steerpike.home.puzzling.org> <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> Message-ID: <5B769383-4274-4EE4-A252-F101FBCB1794@gmail.com> On Jan 28, 2009, at 7:57 PM, Antoine Pitrou wrote: > Jesse Noller writes: >> Windows Services != Unix daemons. Admittedly, I haven't had to do it >> in awhile, but from what I remember, windows services are nowhere >> near >> the same type of beast as a unix daemon, and trying to make a unified >> daemon library that glosses over the differences might cause seizures >> and habitual smoking. > > That would be like a portable library which would allow to write > multiprocess > programs by glossing over the differences between Unix and Windows > processes... > Wouldn't it? > Touch? good sir, touch? From jnoller at gmail.com Thu Jan 29 02:48:50 2009 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 28 Jan 2009 20:48:50 -0500 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: References: <20090126050129.GB11101@steerpike.home.puzzling.org> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> Message-ID: <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> On Wed, Jan 28, 2009 at 7:57 PM, Antoine Pitrou wrote: > Jesse Noller writes: >> Windows Services != Unix daemons. Admittedly, I haven't had to do it >> in awhile, but from what I remember, windows services are nowhere near >> the same type of beast as a unix daemon, and trying to make a unified >> daemon library that glosses over the differences might cause seizures >> and habitual smoking. > > That would be like a portable library which would allow to write multiprocess > programs by glossing over the differences between Unix and Windows processes... > Wouldn't it? > Since Antoine called me to the mat on this one, and I've been chewing on this more today - I think the cross-platformedness (is that a word?) idea is a good one, even if the Qt design doesn't necessarily apply here completely, having a single abstraction to both (but different modules under that abstraction based on os) is attractive (as Stephen pointed out). Heck, a daemon for OS/X is going to need it's own underlying implementation as well: the proper way to install a daemon/service is via launchd [1] which I had to monkey with both for the OS/X buildbot I have, and for an application today. jesse [1] http://developer.apple.com/documentation/Darwin/Reference/ManPages/man8/launchd.8.html#//apple_ref/doc/man/8/launchd From andrew-pythonideas at puzzling.org Thu Jan 29 02:57:03 2009 From: andrew-pythonideas at puzzling.org (Andrew Bennetts) Date: Thu, 29 Jan 2009 12:57:03 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> References: <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> Message-ID: <20090129015703.GG9167@steerpike.home.puzzling.org> Jesse Noller wrote: > > Since Antoine called me to the mat on this one, and I've been chewing > on this more today - I think the cross-platformedness (is that a > word?) idea is a good one, even if the Qt design doesn't necessarily > apply here completely, having a single abstraction to both (but > different modules under that abstraction based on os) is attractive > (as Stephen pointed out). Perhaps such an abstraction should be built on top of Ben Finney's proposed unix daemon module, rather than built in? -Andrew. From jnoller at gmail.com Thu Jan 29 03:06:38 2009 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 28 Jan 2009 21:06:38 -0500 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <20090129015703.GG9167@steerpike.home.puzzling.org> References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> Message-ID: <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> On Wed, Jan 28, 2009 at 8:57 PM, Andrew Bennetts wrote: > Jesse Noller wrote: >> >> Since Antoine called me to the mat on this one, and I've been chewing >> on this more today - I think the cross-platformedness (is that a >> word?) idea is a good one, even if the Qt design doesn't necessarily >> apply here completely, having a single abstraction to both (but >> different modules under that abstraction based on os) is attractive >> (as Stephen pointed out). > > Perhaps such an abstraction should be built on top of Ben Finney's > proposed unix daemon module, rather than built in? > > -Andrew. > > Perhaps; but it makes sense to nest it under the abstraction - daemons.unixdaemon/winservice/osxdaemon From ben+python at benfinney.id.au Thu Jan 29 03:21:00 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 29 Jan 2009 13:21:00 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> Message-ID: <87zlhax377.fsf@benfinney.id.au> Andrew Bennetts writes: > Jesse Noller wrote: > > > > having a single abstraction to both Both? What is the referent here? (I couldn't find any in the context provided.) > > (but different modules under that abstraction based on os) is > > attractive (as Stephen pointed out). > > Perhaps such an abstraction should be built on top of Ben Finney's > proposed unix daemon module, rather than built in? +1. It seems Jesse and I have been simultaneously masticating. Having now chewed on this issue overnight, I can't see any reason why, given the existence of an implementation of the current PEP, that it couldn't be used to implement some *other* PEP dealing with a separate-process, service-based model. That I can't think of any reason why not is likely to mean that there are half a dozen reasons I'm unaware of :-) -- \ ?I went to the museum where they had all the heads and arms | `\ from the statues that are in all the other museums.? ?Steven | _o__) Wright | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 03:35:21 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 29 Jan 2009 13:35:21 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> Message-ID: <87vdryx2ja.fsf@benfinney.id.au> Jesse Noller writes: > Perhaps; but it makes sense to nest it under the abstraction - > daemons.unixdaemon/winservice/osxdaemon I really don't think it's helpful to conflate this ?service? model with the much simpler concept of a Unix daemon. To implement a service on Unix, it's most logical to use a daemon; but the reverse is definitely not true (which is a large part of my resistance to expanding the current PEP to conflate the two concepts). Instead, I think that people who want a ?service? would be better served by a top-level module named ?service?, to preserve that concept as separate from the concept of a daemon. In other words, my understanding is: ?import service? should get an API (so far in no PEP that I know of) for a distinct purpose, that of implementing an MS Windows-style service, which uses whatever OS-specific implementation makes sense. ?import daemon? gets a totally different API for a different purpose, that of turning the current running program into a daemon, which AFAICT only makes a whole lot of sense on Unix and is what I'm proposing in this PEP. It makes much more sense for ?daemon? to stay simple and continue to mean the Unix-specific concept of ?daemon?, without necessarily obstructing whoever wants to implement ?service?, than for ?daemon? in Python to mean something other than the Unix meaning. -- \ ?No matter how far down the wrong road you've gone, turn back.? | `\ ?Turkish proverb | _o__) | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 03:42:44 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 29 Jan 2009 13:42:44 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org> Message-ID: <87r62mx26z.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > the question being asked is why should users of the daemon process > library need to worry about that, separately from other users of > lockfiles or mutexes? Perhaps my mistake is in calling it a ?lockfile?. It's actually a rather specific Unix-native concept: the ?pidfile?, a combination sentinel, lockfile, and primitive text interface. It isn't even used much like other lockfiles (in that it wouldn't make sense, for example to wrap a context manager around it). I might return the terminology in the PEP back to ?pidfile? to be clear. To answer the question, the location of a pidfile warrants specific mention (separate from discussion of lockfiles) in at least two OS standards that I'm aware of, as well as howtos and other discussions. For that reason, it's better to have a simple way to tell a program the desired location and name of its pidfile in particular. -- \ ?[I]t is impossible for anyone to begin to learn that which he | `\ thinks he already knows.? ?Epictetus, _Discourses_ | _o__) | Ben Finney From janssen at parc.com Thu Jan 29 06:14:09 2009 From: janssen at parc.com (Bill Janssen) Date: Wed, 28 Jan 2009 21:14:09 PST Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87vdryx2ja.fsf@benfinney.id.au> References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> Message-ID: <42979.1233206049@pippin.parc.xerox.com> Ben Finney wrote: > ?import daemon? gets a totally different API for a different purpose, > that of turning the current running program into a daemon, which > AFAICT only makes a whole lot of sense on Unix and is what I'm > proposing in this PEP. I think that's an implementation detail. Unix daemons and Windows services seem more similar than different. The details about how they are established/started are different, but that's what a good API covers over. That would be worth designing. There are a number of Unix-only modules for this purpose already, aren't there? I see at least three of them on PyPI, including yours, Ben. What's the point of a new one? And what's the point of adding a new Unix-only module to the stdlib? Aren't we trying to remove platform-only modules? Didn't we remove the Mac modules in 3.x? And isn't win32all a separate distribution? Bill From ben+python at benfinney.id.au Thu Jan 29 07:00:19 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 29 Jan 2009 17:00:19 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <42979.1233206049@pippin.parc.xerox.com> Message-ID: <87mydawt1o.fsf@benfinney.id.au> Bill Janssen writes: > There are a number of Unix-only modules for this purpose already, > aren't there? I see at least three of them on PyPI, including yours, > Ben. Yes, none of which meet the criteria listed in the PEP for ?well-behaved daemon?. They're useful, to be sure; but they're lacking when judged against what a well-behaved Unix daemon should be doing. > What's the point of a new one? Perhaps I've mistakenly given the impression that I'm re-inventing the wheel; sorry for that false impression if so. To implement this PEP fully, I've taken the existing implementation that was closest to the requirements (?bda.daemon?), and based this work on altering it to conform with this PEP. I'm in active correspondence with the author of ?bda.daemon?. > And what's the point of adding a new Unix-only module to the stdlib? To have this common, fairly standardised but tricky, use case implemented once and easily available to any Python programmer, without them needing to know the fine details of what's required to make it work internally. Having multiple partial implementations in PyPI and the Cookbook leads only to the situation where a prospective user has to either know intimate details of this rather fiddly task to judge each of them, or make a choice arbitrarily. -- \ ?I don't know half of you half as well as I should like, and I | `\ like less than half of you half as well as you deserve.? ?Bilbo | _o__) Baggins | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 07:03:05 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 29 Jan 2009 17:03:05 +1100 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> Message-ID: <87iqnywsx2.fsf@benfinney.id.au> Significant changes in this version: New section distinguishing a Unix daemon from a ?service?; clarify that the API makes the *current* program into a daemon process; consistently discuss PID files (instead of lock files). :PEP: XXX :Title: Standard daemon process library :Version: 0.4 :Last-Modified: 2009-01-29 16:32 :Author: Ben Finney :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 2009-01-26 :Python-Version: 3.1 :Post-History: ======== Abstract ======== Writing a program to become a well-behaved Unix daemon is somewhat complex and tricky to get right, yet the steps are largely similar for any daemon regardless of what else the program may need to do. This PEP introduces a module to the Python standard library that provides a simple interface to the task of becoming a daemon process. .. contents:: .. Table of Contents: Abstract Specification Example usage Interface ``Daemon`` objects ``DaemonError`` objects Motivation Rationale Correct daemon behaviour A daemon is not a service Reference Implementation Other daemon implementations References Copyright ============= Specification ============= Example usage ============= Simple example of usage:: import daemon from spam import do_main_program this_daemon = daemon.Daemon() this_daemon.start() do_main_program() More complex example usage:: import os import grp import signal import daemon from spam import ( initial_program_setup, do_main_program, program_cleanup, reload_program_config, ) initial_program_setup() important_file = open('spam.data', 'w') interesting_file = open('eggs.data', 'w') this_daemon = daemon.Daemon( working_directory='/var/lib/foo', umask=0o002, terminate_callback=program_cleanup, reload_callback=reload_program_config, reload_signals=[signal.SIGHUP, signal.SIGUSR1], ) this_daemon.files_preserve = [important_file, interesting_file] mail_gid = grp.getgrnam('mail').gr_gid this_daemon.gid = mail_gid this_daemon.start() do_main_program() Interface ========= A new module, `daemon`, is added to the standard library. The module defines a class, `Daemon`, used to represent the settings and controls for a daemon process. An exception class, `DaemonError`, is defined for exceptions raised from the module. ``Daemon`` objects ================== A `Daemon` instance represents the behaviour settings for the process when it becomes a daemon. The behaviour is customised by setting options on the instance, before calling the `start` method. Each option can be passed as a keyword argument to the `Daemon` constructor, or subsequently altered by assigning to an attribute on the instance at any time prior to calling `start`. That is, for an option named `wibble`, the following invocation:: foo = daemon.Daemon(wibble=bar) foo.start() is equivalent to:: foo = daemon.Daemon() foo.wibble = bar foo.start() The following options are defined. `files_preserve` :Default: ``None`` List of files that should *not* be closed when starting the daemon. If ``None``, all open file descriptors will be closed. Elements of the list are file descriptors (as returned by a file object's `fileno()` method) or Python `file` objects. Each specifies a file that is not to be closed during daemon start. `chroot_directory` :Default: ``None`` Full path to a directory to set as the effective root directory of the process. If ``None``, specifies that the root directory is not to be changed. `working_directory` :Default: ``'/'`` Full path of the working directory to which the process should change on daemon start. Since a filesystem cannot be unmounted if a process has its current working directory on that filesystem, this should either be left at default or set to a directory that is a sensible ?home directory? for the daemon while it is running. `pidfile_directory` :Default: ``'/var/run'`` Absolute directory path to contain the daemon's PID file. If ``None``, the PID file behaviour for this daemon is skipped. `pidfile_name` :Default: ``None`` Base name of the PID file for this daemon, without directory or suffix. If ``None``, the name is derived from the process command line. `umask` :Default: ``0`` File access creation mask (?umask?) to set for the process on daemon start. Since a process inherits its umask from its parent process, starting the daemon will reset the umask to this value so that files are created by the daemon with access modes as it expects. `ignore_signals` :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]`` List of signals that the process should ignore (by setting the signal action to ``signal.SIG_IGN``) on daemon start. `terminate_signals` :Default: ``[signal.SIGTERM]`` List of signals that the process should interpret as a request to terminate cleanly. `terminate_callback` :Default: ``None`` Callable to invoke when the process receives any of the `terminate_signals` signals, before then terminating the process. `reload_signals` :Default: ``[signal.SIGHUP]`` List of signals that the process should interpret as a request to reload runtime configuration. `reload_callback` :Default: ``None`` Callable to invoke when the process receives any of the `reload_signals` signals. `uid` :Default: ``None`` The user ID (?uid?) value to switch the process to on daemon start. `gid` :Default: ``None`` The group ID (?gid?) value to switch the process to on daemon start. `prevent_core` :Default: ``True`` If true, prevents the generation of core files, in order to avoid leaking sensitive information from daemons run as `root`. `stdout` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stdout`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stdout` is not re-bound. `stderr` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stderr`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stderr` is not re-bound. The following methods are defined. `start()` :Return: ``None`` Start the daemon, turning the current program into a daemon process. This performs the following steps: * If the `chroot_directory` attribute is not ``None``: * Set the effective root directory of the process to that directory (via `os.chroot`). This allows running the daemon process inside a ?chroot gaol? as a means of limiting the system's exposure to rogue behaviour by the process. * If the `pidfile_directory` attribute is not ``None``: * Look in that directory for a file named '`pidfile_name`.pid'; if it exists, raise a `DaemonError` to prevent multiple instances of the daemon process. * Close all open file descriptors, excluding those listed in the `files_preserve` attribute. * Change current working directory to the path specified by the `working_directory` attribute. * Reset the file access creation mask to the value specified by the `umask` attribute. * Detach the current process into its own process group, and disassociate from any controlling terminal. This step is skipped if it is determined to be redundant: if the process was started by `init`, by `initd`, or by `inetd`. * Set signal handlers as specified by the `ignore_signals`, `terminate_signals`, `reload_signals` attributes. * If the `prevent_core` attribute is true: * Set the resource limits for the process to prevent any core dump from the process. * Set the process uid and gid to the true uid and gid of the process, to relinquish any elevated privilege. * If the `pidfile_directory` attribute is not ``None``: * Create the PID file for this daemon in that directory, by writing a text line containing the current process ID (?PID?) to a file named '`pidfile_name`.pid'. * If either of the attributes `uid` or `gid` are not ``None``: * Set the process uid and/or gid to the specified values. * If either of the attributes `stdout` or `stderr` are not ``None``: * Bind the names `sys.stdout` and/or `sys.stderr` to the corresponding objects. When the function returns, the running program is a daemon process. `reload()` :Return: ``None`` Reload the daemon configuration. The meaning of this is entirely defined by the customisation of this daemon: if the `reload_callback` attribute is not ``None``, call that object. The return value is discarded. `stop()` :Return: ``None`` Stop the daemon, ending the program. This performs the following steps: * If the `terminate_callback` attribute is not ``None``: * Call that object. The return value is discarded. * If the `pidfile_directory` attribute is not ``None``: * Delete the PID file for this daemon. * Raise a `SystemExit` exception. ``DaemonError`` objects ======================= The `DaemonError` class inherits from `Exception`. The module implementation will raise an instance of `DaemonError` when an error occurs in processing daemon behaviour. ========== Motivation ========== The majority of programs written to be Unix daemons either implement behaviour very similar to that in the `specification`_, or are poorly-behaved daemons by the `correct daemon behaviour`_. Since these steps should be much the same in most implementations but are very particular and easy to omit or implement incorrectly, they are a prime target for a standard well-tested implementation in the standard library. ========= Rationale ========= Correct daemon behaviour ======================== According to Stevens in [stevens]_ ?2.6, a program should perform the following steps to become a Unix daemon process. * Close all open file descriptors. * Change current working directory. * Reset the file access creation mask. * Run in the background. * Disassociate from process group. * Ignore terminal I/O signals. * Disassociate from control terminal. * Don't reacquire a control terminal. * Correctly handle the following circumstances: * Started by System V `init` process. * Daemon termination by ``SIGTERM`` signal. * Children generate ``SIGCLD`` signal. The `daemon` tool [slack-daemon]_ lists (in its summary of features) behaviour that should be performed when turning a program into a well-behaved Unix daemon process. It differs from this PEP's intent in that it invokes a *separate* program as a daemon process. The following features are appropriate for a daemon that starts itself once the program is already running: * Sets up the correct process context for a daemon. * Behaves sensibly when started by `initd(8)` or `inetd(8)`. * Revokes any suid or sgid privileges to reduce security risks in case daemon is incorrectly installed with special privileges. * Prevents the generation of core files to prevent leaking sensitive information from daemons run as root (optional). * Names the daemon by creating and locking a PID file to guarantee that only one daemon with the given name can execute at any given time (optional). * Sets the user and group under which to run the daemon (optional, root only). * Creates a chroot gaol (optional, root only). * Captures the daemon's stdout and stderr and directs them to syslog (optional). A daemon is not a service ========================= This PEP addresses only Unix-style daemons, for which the above correct behaviour is relevant, as opposed to comparable behaviours on other operating systems. There is a related concept in many systems, called a ?service?. A service differs from the model in this PEP, in that rather than having the *current* program continue to run as a daemon process, a service starts an *additional* process to run in the background, and the current process communicates with that additional process via some defined channels. The Unix-style daemon model in this PEP can be used, among other things, to implement the background-process part of a service; but this PEP does not address the other aspects of setting up and managing a service. It is the opinion of this PEP's author that combining the two disparate systems of behaviour under a single API would be contrary to the focus of this PEP. ======================== Reference Implementation ======================== The `python-daemon` package [python-daemon]_. As of 2009-01-26, the package is under active development and is not yet a full implementation of this PEP. Other daemon implementations ============================ Prior to this PEP, several existing third-party Python libraries or tools implemented some of this PEP's `correct daemon behaviour`_. The `reference implementation`_ is a fairly direct successor from the following implementations: * Many good ideas were contributed by the community to Python cookbook recipe 66012 [cookbook-66012]_. * The `bda.daemon` library [bda.daemon]_ is an implementation of [cookbook-66012]_. It is the predecessor of [python-daemon]_. Other Python daemon implementations that differ from this PEP: * The `zdaemon` tool [zdaemon]_ was written for the Zope project. Like [slack-daemon]_, it differs from this specification because it is used to run another program as a daemon process. * The Python library `daemon` [clapper-daemon]_ is (according to its homepage) no longer maintained. As of version 1.0.1, it implements the basic steps from [stevens]_. * The `daemonize` library [seutter-daemonize]_ also implements the basic steps from [stevens]_. * Twisted [twisted]_ includes, perhaps unsurprisingly, an implementation of a process daemonisation API that is integrated with the rest of the Twisted framework; it differs significantly from the API in this PEP. * The Python `initd` library [dagitses-initd]_, which uses [clapper-daemon]_, implements an equivalent of Unix `initd(8)` for controlling a daemon process. ========== References ========== .. [stevens] `Unix Network Programming`, W. Richard Stevens, 1994 Prentice Hall. .. [slack-daemon] The (non-Python) ?libslack? implementation of a `daemon` tool ``_ by ?raf? . .. [python-daemon] The `python-daemon` library ``_ by Ben Finney et al. .. [cookbook-66012] Python Cookbook recipe 66012, ?Fork a daemon process on Unix? ``_. .. [bda.daemon] The `bda.daemon` library ``_ by Robert Niederreiter et al. .. [zdaemon] The `zdaemon` tool ``_ by Guido van Rossum et al. .. [clapper-daemon] The `daemon` library ``_ by Brian Clapper. .. [seutter-daemonize] The `daemonize` library ``_ by Jerry Seutter. .. [twisted] The `Twisted` application framework ``_ by Glyph Lefkowitz et al. .. [dagitses-initd] The Python `initd` library ``_ by Michael Andreas Dagitses. ========= Copyright ========= This work is hereby placed in the public domain. To the extent that placing a work in the public domain is not legally possible, the copyright holder hereby grants to all recipients of this work all rights and freedoms that would otherwise be restricted by copyright. .. Local variables: mode: rst coding: utf-8 time-stamp-start: "^:Last-Modified:[ ]+" time-stamp-end: "$" time-stamp-line-limit: 20 time-stamp-format: "%:y-%02m-%02d %02H:%02M" End: vim: filetype=rst fileencoding=utf-8 : From phd at phd.pp.ru Thu Jan 29 13:01:49 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Thu, 29 Jan 2009 15:01:49 +0300 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87r62mx26z.fsf@benfinney.id.au> References: <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org> <87r62mx26z.fsf@benfinney.id.au> Message-ID: <20090129120149.GA29303@phd.pp.ru> On Thu, Jan 29, 2009 at 01:42:44PM +1100, Ben Finney wrote: > rather specific Unix-native concept: the "pidfile", a combination > sentinel, lockfile, and primitive text interface. IWBN to have a main() function in the module to be used from the command line: $ python -m daemon --pidfile=/var/run/mydaemon.pid status Daemon pid 1221 is running $ python -m daemon --pidfile=/var/run/mydaemon.pid stop $ python -m daemon --pidfile=/var/run/mydaemon.pid status Cannot find /var/run/mydaemon.pid A note on implementation detail: IWBN if the pidfile is removed on stop(). Please do not put the burden unto the user. > I might return the terminology in the PEP back to "pidfile" to be > clear. +1 Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Thu Jan 29 13:53:51 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Jan 2009 12:53:51 +0000 (UTC) Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> Message-ID: Ben Finney writes: > > Instead, I think that people who want a ?service? would be better > served by a top-level module named ?service?, to preserve that concept > as separate from the concept of a daemon. I may be misstaken, but I think the usual desire is to have *something* which runs in the background and doesn't get killed when you kill the parent application or terminal which launched it. Whatever it is implemented in terms of "daemon", "service" or anything else doesn't really matter in those cases. From amcnabb at mcnabbs.org Thu Jan 29 15:56:05 2009 From: amcnabb at mcnabbs.org (Andrew McNabb) Date: Thu, 29 Jan 2009 07:56:05 -0700 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library In-Reply-To: <87iqnywsx2.fsf@benfinney.id.au> References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> Message-ID: <20090129145605.GB3133@mcnabbs.org> On Thu, Jan 29, 2009 at 05:03:05PM +1100, Ben Finney wrote: > > There is a related concept in many systems, called a ?service?. A > service differs from the model in this PEP, in that rather than having > the *current* program continue to run as a daemon process, a service > starts an *additional* process to run in the background, and the > current process communicates with that additional process via some > defined channels. In my opinion, the difference between a daemon and a service is just a by-product of the difference between fork and CreateProcess. If the multiprocessing package can solve this problem, I don't see why a daemon/service package wouldn't be able to, too. > As of 2009-01-26, the package is under active development and is not > yet a full implementation of this PEP. I'm still confused why there's a PEP before there's a stable implementation and a base of users. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868 From solipsis at pitrou.net Thu Jan 29 17:54:53 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Jan 2009 16:54:53 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?= =?utf-8?q?d_daemon_process=09library?= References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> Message-ID: Hello, > `files_preserve` > :Default: ``None`` > > List of files that should *not* be closed when starting the > daemon. Why does it have to be explicit? If I have an open file lying around, it should be obvious that I don't want it closed. Otherwise I'd have closed it myself... The exception is of course the three stdio streams, which should be treated separately. > `umask` > :Default: ``0`` > > File access creation mask (?umask?) to set for the process on > daemon start. > > Since a process inherits its umask from its parent process, > starting the daemon will reset the umask to this value so that > files are created by the daemon with access modes as it expects. Why this behaviour? It's easy enough to call os.umask() manually if you want it, but there are certainly situations where you don't want to change the umask (situations where the daemon is meant to act on behalf of the user who started it). Or, perhaps, umask could be made mand (regardless of which, 0 isn't a sensible umask default!) > * If the `pidfile_directory` attribute is not ``None``: > > * Look in that directory for a file named '`pidfile_name`.pid'; > if it exists, raise a `DaemonError` to prevent multiple > instances of the daemon process. It should first check that there does exist a processus with that number, so that stale pid files don't cause too many problems. Regards Antoine. From solipsis at pitrou.net Thu Jan 29 18:02:16 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Jan 2009 17:02:16 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?= =?utf-8?q?d_daemon_process=09library?= References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> Message-ID: Antoine Pitrou writes: > Or, perhaps, umask could be made mand Finishing my sentence: Or, perhaps, umask could be made mandatory only when the uid is changed. From george.sakkis at gmail.com Thu Jan 29 19:33:39 2009 From: george.sakkis at gmail.com (George Sakkis) Date: Thu, 29 Jan 2009 13:33:39 -0500 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library In-Reply-To: <20090129145605.GB3133@mcnabbs.org> References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> <20090129145605.GB3133@mcnabbs.org> Message-ID: <91ad5bf80901291033g10f2585bt39283e2c8389c118@mail.gmail.com> On Thu, Jan 29, 2009 at 9:56 AM, Andrew McNabb wrote: >> As of 2009-01-26, the package is under active development and is not >> yet a full implementation of this PEP. > > I'm still confused why there's a PEP before there's a stable > implementation and a base of users. Let alone the *need* to be included in the stdlib, when Pypi seems perfectly appropriate for such package, especially if it's unix-only. George From python-ideas-list at trentnelson.com Thu Jan 29 20:54:01 2009 From: python-ideas-list at trentnelson.com (Trent Nelson) Date: Thu, 29 Jan 2009 19:54:01 +0000 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87vdryx2ja.fsf@benfinney.id.au> References: <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> Message-ID: <20090129195401.GB60708@wind.teleri.net> On Thu, Jan 29, 2009 at 01:35:21PM +1100, Ben Finney wrote: > It makes much more sense for ???daemon??? to stay simple and continue to > mean the Unix-specific concept of ???daemon???, without necessarily > obstructing whoever wants to implement ???service???, than for ???daemon??? > in Python to mean something other than the Unix meaning. There are two types of developers that will utilise this module. Those that are writing something for themselves, that never has to run anywhere else other than Unix. Or those that work primarily or exclusively in Unix, but understand that others might want to run their code elsewhere. If you build it, they will come. By building the latter solution, sure, you've inconvenienced the guy that wants a Unix-only daemon, but you've also enabled everyone else who wants to write stuff that has a chance of running unmodified on other platforms. And jumping on the bandwagon of a few others; I like the argument that something Unix-only like this could just as easily belong in PyPI, whereas a cross-platform solution is more viable for stdlib inclusion. We should make it harder to write non-portable code, not easier. Trent. From ben+python at benfinney.id.au Thu Jan 29 21:30:54 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 07:30:54 +1100 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> <20090129145605.GB3133@mcnabbs.org> Message-ID: <8763jxx3b5.fsf@benfinney.id.au> Andrew McNabb writes: > I'm still confused why there's a PEP before there's a stable > implementation and a base of users. Because that's exactly what PEP 1 explicitly advises. Am I wrong to be following PEP 1? -- \ ?We should strive to do things in [Gandhi's] spirit? not to use | `\ violence in fighting for our cause, but by non-participation in | _o__) what we believe is evil.? ?Albert Einstein | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 22:06:38 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 08:06:38 +1100 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> Message-ID: <871vulx1nl.fsf@benfinney.id.au> Antoine Pitrou writes: > Hello, > > > `files_preserve` > > :Default: ``None`` > > > > List of files that should *not* be closed when starting the > > daemon. > > Why does it have to be explicit? If I have an open file lying > around, it should be obvious that I don't want it closed. Otherwise > I'd have closed it myself... Is it so obvious? In Python programs, I far more often see files left open simply because the programmer expects them to be cleaned up at the appropriate time. By the references cited in this PEP, the time of starting the daemon is such an appropriate time to close all open file descriptors. > > `umask` > > :Default: ``0`` > > > > File access creation mask (?umask?) to set for the process on > > daemon start. > > > > Since a process inherits its umask from its parent process, > > starting the daemon will reset the umask to this value so that > > files are created by the daemon with access modes as it > > expects. > > Why this behaviour? Because the umask is inherited from the environment of the parent process of the current program, which is good when the current program is conceptually an extension of the parent program's environment, but not in the case where one is intentionally disassociating from that environment. > It's easy enough to call os.umask() manually if you want it, but > there are certainly situations where you don't want to change the > umask (situations where the daemon is meant to act on behalf of the > user who started it). Those situations exist, but I would argue they are not the common case. > Or, perhaps, umask could be made [mandatory when the options ?uid? > or ?gid? are set.] That diverges from the referenced ?correct behaviour?, in particular the Stevens reference. I think if this PEP is worth implementing, it's for exactly the purpose of providing a Python implementation of defined correct daemon behaviour; that definition includes re-setting the umask. > (regardless of which, 0 isn't a sensible umask default!) The default umask ?0? is chosen in the references as the umask that allows file access modes for files created by the daemon to work as expected. Quoting Stevens: This prevents any files created by the daemon from having their access bits modified. For example, if a daemon specifically created a file with a mode of 0660, so that only the user and group could read and write the file, but the ``umask`` value was 060, the group read and write permissions would be turned off. If the daemon required the group permissions to be on, so that some other process in that group could access the file, this ``umask`` value prevents it. -- \ ?I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.? ?Douglas Adams | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 22:09:16 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 08:09:16 +1100 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> Message-ID: <87wscdvmyr.fsf@benfinney.id.au> Antoine Pitrou writes: > Hello, > [?] > > * If the `pidfile_directory` attribute is not ``None``: > > > > * Look in that directory for a file named '`pidfile_name`.pid'; > > if it exists, raise a `DaemonError` to prevent multiple > > instances of the daemon process. > > It should first check that there does exist a processus with that > number, so that stale pid files don't cause too many problems. Good point, thanks. What would be appropriate behaviour in the case of a stale PID file? Abort the daemonisation attempt? Delete the stale lock file silently and continue as though it didn't exist? -- \ ?I bought some powdered water, but I don't know what to add.? | `\ ?Steven Wright | _o__) | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 22:14:07 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 08:14:07 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org> <87r62mx26z.fsf@benfinney.id.au> <20090129120149.GA29303@phd.pp.ru> Message-ID: <87pri5vmqo.fsf@benfinney.id.au> Oleg Broytmann writes: > IWBN to have a main() function in the module to be used from the > command line: > > $ python -m daemon --pidfile=/var/run/mydaemon.pid status > Daemon pid 1221 is running > > $ python -m daemon --pidfile=/var/run/mydaemon.pid stop > > $ python -m daemon --pidfile=/var/run/mydaemon.pid status > Cannot find /var/run/mydaemon.pid Interesting. Should that be defined in this PEP, or merely added to the reference implementation as a bonus? > A note on implementation detail: IWBN if the pidfile is removed on > stop(). Please do not put the burden unto the user. Yes, that's already addressed in the defined behaviour for `stop()`. -- \ ?Know what I hate most? Rhetorical questions.? ?Henry N. Camp | `\ | _o__) | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 22:18:05 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 08:18:05 +1100 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> <20090129145605.GB3133@mcnabbs.org> <91ad5bf80901291033g10f2585bt39283e2c8389c118@mail.gmail.com> Message-ID: <87ocxpvmk2.fsf@benfinney.id.au> George Sakkis writes: > Let alone the *need* to be included in the stdlib, when Pypi seems > perfectly appropriate for such package, especially if it's > unix-only. I addressed this in Message-ID: <87mydawt1o.fsf at benfinney.id.au> yesterday (a reply to Bill Janssen). -- \ ?My doctor told me to stop having intimate dinners for four. | `\ Unless there are three other people.? ?Orson Welles | _o__) | Ben Finney From phd at phd.pp.ru Thu Jan 29 22:22:15 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Fri, 30 Jan 2009 00:22:15 +0300 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87pri5vmqo.fsf@benfinney.id.au> References: <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org> <87r62mx26z.fsf@benfinney.id.au> <20090129120149.GA29303@phd.pp.ru> <87pri5vmqo.fsf@benfinney.id.au> Message-ID: <20090129212215.GA17415@phd.pp.ru> On Fri, Jan 30, 2009 at 08:14:07AM +1100, Ben Finney wrote: > Oleg Broytmann writes: > > IWBN to have a main() function in the module to be used from the > > command line: > > > > $ python -m daemon --pidfile=/var/run/mydaemon.pid status > > Daemon pid 1221 is running > > > > $ python -m daemon --pidfile=/var/run/mydaemon.pid stop > > > > $ python -m daemon --pidfile=/var/run/mydaemon.pid status > > Cannot find /var/run/mydaemon.pid > > Interesting. Should that be defined in this PEP, or merely added to > the reference implementation as a bonus? I think an implementation would be enough. > > A note on implementation detail: IWBN if the pidfile is removed on > > stop(). Please do not put the burden unto the user. > > Yes, that's already addressed in the defined behaviour for `stop()`. Thank you! Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ben+python at benfinney.id.au Thu Jan 29 22:22:45 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 08:22:45 +1100 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> <20090129145605.GB3133@mcnabbs.org> Message-ID: <87k58dvmca.fsf@benfinney.id.au> Andrew McNabb writes: > In my opinion, the difference between a daemon and a service is just > a by-product of the difference between fork and CreateProcess. This PEP addresses the case where a daemon is specifically what the programmer wants. I would like it to *also* be useful as the obvious thing to use for implementing a service on Unix, but it is designed to also allow for the case when a service is *not* what the programmer wants. > If the multiprocessing package can solve this problem, I don't see > why a daemon/service package wouldn't be able to, too. A service package would be ideal for the use case you suggest. That's not what this PEP proposes, though. -- \ ?Why, I'd horse-whip you if I had a horse.? ?Groucho Marx | `\ | _o__) | Ben Finney From ben+python at benfinney.id.au Thu Jan 29 22:28:33 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 08:28:33 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <20090129195401.GB60708@wind.teleri.net> Message-ID: <87fxj1vm2m.fsf@benfinney.id.au> Trent Nelson writes: > There are two types of developers that will utilise this module. > Those that are writing something for themselves, that never has to > run anywhere else other than Unix. Surely there is also the type that are writing *for others*, and want the program to be both well-behaved and work on any Unix-like OS? > Or those that work primarily or exclusively in Unix, but understand > that others might want to run their code elsewhere. Those people would be well served by a putative ?service? interface. I would welcome such a PEP from you, since you seem to have fairly solid ideas of what it should look like. > We should make it harder to write non-portable code, not easier. It's a bit of a stretch to go from ?designed to work on any Unix-like OS? to ?non-portable?. -- \ ?Under democracy one party always devotes its chief energies to | `\ trying to prove that the other party is unfit to rule ? and | _o__) both commonly succeed, and are right.? ?Henry L. Mencken | Ben Finney From solipsis at pitrou.net Thu Jan 29 22:53:18 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Jan 2009 21:53:18 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?= =?utf-8?q?d_daemon=09process=09library?= References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> <871vulx1nl.fsf@benfinney.id.au> Message-ID: Ben Finney writes: > > Is it so obvious? In Python programs, I far more often see files left > open simply because the programmer expects them to be cleaned up at > the appropriate time. What if I use a library which keeps its own files open in the background? Do I have to monkeypatch the library so as to get at the file descriptors and pass them by hand? > > It's easy enough to call os.umask() manually if you want it, but > > there are certainly situations where you don't want to change the > > umask (situations where the daemon is meant to act on behalf of the > > user who started it). > > Those situations exist, but I would argue they are not the common > case. Well, perhaps, but it's easier to leave the umask as is rather than ask the programmer to query the current umask and then pass it as an argument (not to mention that the current os.umask() function does not support querying without modifying...). > That diverges from the referenced ?correct behaviour?, in particular > the Stevens reference. I think if this PEP is worth implementing, it's > for exactly the purpose of providing a Python implementation of > defined correct daemon behaviour; that definition includes re-setting > the umask. Well, I think the PEP should implement a behaviour which is at the same time useful and sane. Whether it scrupulously conforms to something which isn't an official standard shouldn't be considered important. > The default umask ?0? is chosen in the references as the umask that > allows file access modes for files created by the daemon to work as > expected. And then, if I create a file using the standard Python idiom (the open() function), it will be readable and writable by anyone. It isn't sane at all. Regards, Antoine. From solipsis at pitrou.net Thu Jan 29 23:07:32 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 29 Jan 2009 22:07:32 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?= =?utf-8?q?d_daemon=09process=09library?= References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> <87wscdvmyr.fsf@benfinney.id.au> Message-ID: Ben Finney writes: > > What would be appropriate behaviour in the case of a stale PID file? > Abort the daemonisation attempt? Delete the stale lock file silently > and continue as though it didn't exist? Delete the stale lock file silently and continue as though it didn't exist. (and, of course, create another one for the current process) From ben+python at benfinney.id.au Fri Jan 30 01:09:20 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 11:09:20 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <871vup1s62.fsf@benfinney.id.au> <871vupfqdd.fsf@xemacs.org> Message-ID: <87pri5u027.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > The people who are posting "show us the code" are mostly folks with > long experience in the PEP process. I think you should take it as an > indication that in this case the definition of "suitable API" is an > "I'll know it when I've tried it" kind of thing. I am working on ?python-daemon? concurrently with this PEP now, and intend to keep publishing updates to that implementation as it approaches conformance with the PEP. > What you might find useful is to avoid writing any code yourself, but > see how well your proposed spec fits the existing code, of which > several (potential) examples have been posted. Fortunately I've based ?python-daemon? on what I determined to be the closest starting point for the well-behaved daemon behaviour. -- \ ?Any intelligent fool can make things bigger and more complex? | `\ It takes a touch of genius ? and a lot of courage ? to move in | _o__) the opposite direction.? ?Albert Einstein | Ben Finney From stephen at xemacs.org Fri Jan 30 03:12:05 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 30 Jan 2009 11:12:05 +0900 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87fxj1vm2m.fsf@benfinney.id.au> References: <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <20090129195401.GB60708@wind.teleri.net> <87fxj1vm2m.fsf@benfinney.id.au> Message-ID: <87y6wtczka.fsf@xemacs.org> Ben Finney writes: > Those people would be well served by a putative "service" interface. > I would welcome such a PEP from you, since you seem to have fairly > solid ideas of what it should look like. AFAICS it looks like yours, except with the requirement of "works on Windows" and a strong preference for "no other-OS-specific clutter with a simple 'import daemonize'". Bill added "plays nicely with Mac OS X". I don't see on what grounds you object to exploring the possibility, since you say you don't know much about Windows services. Note that nobody is suggesting that you write the code that implements daemons via Windows services (of course they'd love it if you did!), only that you open up the API design to changes that would make some programs work on all platforms without conditioning on platform, and make all programs work with minimal platform-specific code. So I suggest that you bat the ball back into their court, and announce on Python-Dev that you're willing but not able to generalize your PEP to Windows, and ask for coauthors to help. If there are no takers, then you're in a strong position to argue that Unix-like OSes are the important case anyway, nobody cares about Windows services in Python. If there are, then somebody else does the work and everybody is happy. No? > > We should make it harder to write non-portable code, not easier. > > It's a bit of a stretch to go from "designed to work on any Unix-like > OS" to "non-portable". That is not a Pythonic attitude. Here, "portable code" means "designed to work in any instance of Python conditioning only on 'this instance supports the feature'." From ben+python at benfinney.id.au Fri Jan 30 04:30:17 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 14:30:17 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <20090129195401.GB60708@wind.teleri.net> <87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org> Message-ID: <878wottqra.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > Ben Finney writes: > > > Those people would be well served by a putative "service" interface. > > I would welcome such a PEP from you, since you seem to have fairly > > solid ideas of what it should look like. > > AFAICS it looks like yours I can't see how anyone could compare the latest version of this PEP with the suggestions being made by Trent for an API, and conclude that they are similar. This might because the PEP isn't communicating very well. Does the section ?A daemon is not a service? help to explain the fundamental difference between the model suggested by Trent, versus the model in the PEP? If not, is it made clearer by the addition of nearly a dozen extra methods in Trent's suggestion, and several extra concepts, that are extraneous to the task described in the PEP? All of which seem ideally suited to a putative ?service? API. > I don't see on what grounds you object to exploring the possibility, > since you say you don't know much about Windows services. I have no objection to someone exploring a ?service? interface. Indeed, I applaud such efforts. But that's not what I'm addressing with this PEP. As for the possibility of using the daemon interface as the Unix-specific implementation of the background-process *component* of a service interface, I believe it would be very well suited for that. But I am implementing the daemon API *now* because it's a distinct task, that I've seen needed many times and implemented poorly in many places. I would be delighted if someone who knows what they want from a ?service? interface could build that, entirely different, API using a daemon as part of the implementation. > Note that nobody is suggesting that you write the code that > implements daemons via Windows services (of course they'd love it if > you did!), only that you open up the API design to changes that > would make some programs work on all platforms without conditioning > on platform, and make all programs work with minimal > platform-specific code. I maintain that it would be better to specify a platform-agnostic ?service? API, that uses the ?daemon? API as the Unix-specific implementation of the background-process component of the service. Meanwhile, I and others have a need for a simple, standardised, one-obvious-way ?daemonise the current program? library, *entirely distinct from* any of ther extra baggage that comes with a ?spawn a separate process that I can then communicate with through channels? API. I intend to satisfy that with this PEP. -- \ ?People demand freedom of speech to make up for the freedom of | `\ thought which they avoid.? ?Soren Aabye Kierkegaard (1813-1855) | _o__) | Ben Finney From ben+python at benfinney.id.au Fri Jan 30 04:44:39 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 14:44:39 +1100 Subject: [Python-ideas] =?utf-8?q?Cross-platform_file_locking=2C_PID_files?= =?utf-8?b?LCBhbmQgdGhlIOKAnGRhZW1vbuKAnSBQRVAgKHdhczogRHJhZnQgUEVQOiBT?= =?utf-8?q?tandard_daemon_process_library=29?= References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org> Message-ID: <874ozhtq3c.fsf_-_@benfinney.id.au> "Stephen J. Turnbull" writes: > If [the daemon interface deals with file locking] only because > there's no satisfactory module implementing it widely available, > then that could be done in a subsidiary module that handles > OS-specific details, and optionally exposes OS-specific special > features (a la os and os.posix). That would also make refactoring > (close to) trivial when such a module did become widely available. Skip Montanaro has a PyPI package, ?lockfile? (currently marked ?Beta?) that has such ambitions. I think the interface he proposes is elegant and I intend to submit an additional ?PID File lock? capability, to cover the behaviour expected of PID files in particular. How should I alter this PEP if I'm proposing to use functionality currently present in another package that exists only in PyPI? Should I make an even lower-level PEP to suggest inclusion of Skip's ?lockfile? package? Is that wise? Is it polite? -- \ ?Are you pondering what I'm pondering?? ?Well, I think so, | `\ Brain, but if Jimmy cracks corn, and no one cares, why does he | _o__) keep doing it?? ?_Pinky and The Brain_ | Ben Finney From stephen at xemacs.org Fri Jan 30 06:20:30 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 30 Jan 2009 14:20:30 +0900 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <878wottqra.fsf@benfinney.id.au> References: <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <20090129195401.GB60708@wind.teleri.net> <87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org> <878wottqra.fsf@benfinney.id.au> Message-ID: <87r62lcqu9.fsf@xemacs.org> Ben Finney writes: > [C]ompare the latest version of this PEP [...]. I'm assuming the one posted on 1/29 in <87iqnywsx2.fsf at benfinney.id.au>, or fairly close to that. > This might because the PEP isn't communicating very well. Does the > section "A daemon is not a service" help to explain the fundamental > difference between the model suggested by Trent, versus the model in > the PEP? That is quite clear, and you've said "daemonize-me != start-a- service" in discussion here. However, you've also said that "daemon != service" (including in the title of that section!), which clearly is mostly false when those terms are referring to the processes that are running in the background. I think that focusing on the distinction between "*becoming* a daemon" and "*starting* a service" is distracting. The point being that *becoming* a well-behaved service surely requires becoming a well- behaved background process. The real issue is not starting a separate process (just don't provide a way to do it), but rather it is that it also requires all the communication setup as well, which is out-of- scope for a daemonization PEP on any platform (apart from closing file descriptors, which is specific to the fork(2) implementation of process creation). To prevent the misinterpretation I made, I would phrase that title "A Service is Not Just a Daemon", implying "this PEP is not a full-service PEP". BTW, IMO you can delete the sentence "It is the opinion of this PEP's author that combining the two disparate systems of behaviour under a single API would be contrary to the focus of this PEP." from the PEP. It's redundant. > Meanwhile, I and others have a need for a simple, standardised, > one-obvious-way "daemonise the current program" library, *entirely > distinct from* any of ther extra baggage that comes with a "spawn a > separate process that I can then communicate with through channels" > API. I intend to satisfy that with this PEP. Well, that's what I thought was in question here. It seems to me that there are likely to be additional aspects to "daemonize me" for Mac OS X and Windows, but that the basic start-reload-stop methods should probably apply. I'm not familiar enough with the process of daemonization on Mac OS X (Bill mentioned interaction with launchd) or Windows to help at reasonable expense, but I certainly hope that those who are will review to see if a similar API might be useful to them. Unfortunately there doesn't seem to be that much common ground among the advocates, though. Steve From stephen at xemacs.org Fri Jan 30 06:39:52 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 30 Jan 2009 14:39:52 +0900 Subject: [Python-ideas] Cross-platform file locking, PID files , and the "daemon" PEP (was: Draft PEP: S tandard daemon process library) In-Reply-To: <874ozhtq3c.fsf_-_@benfinney.id.au> References: <21108.1233038783@pippin.parc.xerox.com> <87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru> <20090127160643.GC37589@wind.teleri.net> <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk> <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com> <87eiyozbus.fsf@benfinney.id.au> <20090128012906.GH57568@wind.teleri.net> <87ab9cyy42.fsf@benfinney.id.au> <20090128030351.GD64950@wind.teleri.net> <871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org> <874ozhtq3c.fsf_-_@benfinney.id.au> Message-ID: <87ocxpcpxz.fsf@xemacs.org> Ben Finney writes: > Skip Montanaro has a PyPI package, ?lockfile? (currently marked > "Beta") that has such > ambitions. > Should I make an even lower-level PEP to suggest inclusion of Skip's > ?lockfile? package? Is that wise? Is it polite? I would send your proposal for PIDlockfile to Skip. In your PEP, describe the relevant parts of the API explicitly in the PEP, with a footnote that you are deliberately extending the lockfile interface and currently intend to remain compatible with that interface. Then talk to Skip about a PEP for including his package, or perhaps extending your PEP to include it. Which is better depends on details and how aggressively Skip would want to pursue inclusion of lockfile if your daemon package doesn't make it or takes a long time. I don't know about "polite", but I think it would be unwise to try to PEP lockfile without Skip's explicit support. From ben+python at benfinney.id.au Fri Jan 30 06:46:13 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 30 Jan 2009 16:46:13 +1100 Subject: [Python-ideas] Skeletal PEP (version 0.1): Background process services Message-ID: <87pri5s5wa.fsf@benfinney.id.au> Howdy all, Many people evidently want a proposal for a ?service? interface, and saw part of what they wanted in the separate discussion of the ?Standard daemon process library?. As discussed there, a Unix daemon is a quite distinct, yet complementary, concept to a service. This is a skeleton of a PEP, presented in order that discussion of what people want from a ?service? has a concrete focus. The specification is based on work from Trent Nelson. Note that I'm *not* intending to propose, implement, or champion this PEP. I present it as is because there is a clear desire to see something like it, and in the hope that those with more enthusiasm for it can take it and improve on it to the point where it gets a champion. :PEP: XXX :Title: Background process services :Version: 0.1 :Last-Modified: 2009-01-30 16:40 :Author: Ben Finney :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 2009-01-30 :Python-Version: 3.1 :Post-History: ======== Abstract ======== This PEP introduces a package to the Python standard library that provides an interface to the task of creating a separate background process, and maintaining an ongoing service channel with that process. .. contents:: .. Table of Contents: Abstract Specification Interface ``Service`` objects Motivation Rationale Reference Implementation References Copyright ============= Specification ============= Interface ========= A new package, `service`, is added to the standard library. The package defines a class, `Service`, used to set up and maintain the service. ``Service`` objects =================== A `Service` instance represents a control channel to a separate, background process that runs concurrently with the existing program. The existing program communicates with that background process via the corresponding `Service` instance. `__init__(self, func, name, desc=None, startup_type=StartupType.Auto)` Initialises the parameters of a new service [?] The `func` argument is a callable, which will be called to run in the background process when the service is started. [?] `parse_arguments(args)` [?] `is_running()` Returns ``True`` if the background process is currently running [?] `is_installed()` Returns ``True`` if [?] `install()` [?] `uninstall()` [?] `report_event(self, msg, event_type, id, category, data)` [?] `exec()` [?] `terminate()` [?] `request_pause()` [?] `request_resume()` [?] `send_command(code)` [?] `run(args)` [?] `start()` Start a new background process running the `func` callable [?] * On Unix, this creates a new daemon process. * On MacOS, [?]. * On MS Windows, [?]. `stop()` Stop the background process [?] `pause()` [?] `resume()` [?] ========== Motivation ========== [?] ========= Rationale ========= [?] ======================== Reference Implementation ======================== [?] ========== References ========== [?] ========= Copyright ========= This work is hereby placed in the public domain. To the extent that placing a work in the public domain is not legally possible, the copyright holder hereby grants to all recipients of this work all rights and freedoms that would otherwise be restricted by copyright. .. Local variables: mode: rst coding: utf-8 time-stamp-start: "^:Last-Modified:[ ]+" time-stamp-end: "$" time-stamp-line-limit: 20 time-stamp-format: "%:y-%02m-%02d %02H:%02M" End: vim: filetype=rst fileencoding=utf-8 : From nate at binkert.org Fri Jan 30 22:38:42 2009 From: nate at binkert.org (nathan binkert) Date: Fri, 30 Jan 2009 13:38:42 -0800 Subject: [Python-ideas] Draft PEP: Standard daemon process library In-Reply-To: <87r62lcqu9.fsf@xemacs.org> References: <497F351D.30700@timgolden.me.uk> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <20090129195401.GB60708@wind.teleri.net> <87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org> <878wottqra.fsf@benfinney.id.au> <87r62lcqu9.fsf@xemacs.org> Message-ID: <217accd40901301338t65c1729cp8db16623c423346c@mail.gmail.com> It's tough to follow this whole thread as it has spread all over the place, so my apologies if this has been mentioned. Why not try to make both the unix only daemon and the generic service available? The latter could use the former for its implementation? Nate From ben+python at benfinney.id.au Fri Jan 30 23:40:20 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 31 Jan 2009 09:40:20 +1100 Subject: [Python-ideas] Draft PEP: Standard daemon process library References: <497F351D.30700@timgolden.me.uk> <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com> <20090129015703.GG9167@steerpike.home.puzzling.org> <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com> <87vdryx2ja.fsf@benfinney.id.au> <20090129195401.GB60708@wind.teleri.net> <87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org> <878wottqra.fsf@benfinney.id.au> <87r62lcqu9.fsf@xemacs.org> <217accd40901301338t65c1729cp8db16623c423346c@mail.gmail.com> Message-ID: <87ab981kq3.fsf@benfinney.id.au> nathan binkert writes: > It's tough to follow this whole thread as it has spread all over the > place, so my apologies if this has been mentioned. > > Why not try to make both the unix only daemon and the generic service > available? The latter could use the former for its implementation? Yes, this is my position. I'm glad to see that I'm not alone, thanks :-) -- \ ?Puritanism: The haunting fear that someone, somewhere, may be | `\ happy.? ?Henry L. Mencken | _o__) | Ben Finney From gagsl-py2 at yahoo.com.ar Sat Jan 31 11:36:16 2009 From: gagsl-py2 at yahoo.com.ar (Gabriel Genellina) Date: Sat, 31 Jan 2009 08:36:16 -0200 Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process library References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au> Message-ID: En Thu, 29 Jan 2009 04:03:05 -0200, Ben Finney escribi?: > :PEP: XXX > :Title: Standard daemon process library > :Version: 0.4 After reading this PEP, I see it as a big list of implementation details. A more high level view would be more suitable for a multiplatform implementation. Like when using most library modules, I trust the authors, I hope they've done their job well, that they've read their Stevens and Pietrek, and the code follows the best practices. For the most part, I don't care about the implementation - just that it don't gets in my way. I'd say that a daemon is just a background process, detached from any terminal and without any user interactivity, that keeps running independently of the logged user. *How* to get there, it's the library job. So I need a method to "become a daemon". Stopping a daemon process must be done orderly so a "stop" method is required too. It needs some way to react to external requests (signals, or control requests on Windows): they're a short list, we can use methods following a naming convention (e.g. "on_reload"). And that's all, at least conceptually; probably we need some attributes to customize the behaviour. I think it's a lot easier to implement such abstract view on different systems. Excluding Windows right from the start --because of so many details that should be emulated, and are mostly irrelevant-- would be a bad idea. -- Gabriel Genellina