From pierre.barbier at cirad.fr Wed Jun 1 14:32:54 2005 From: pierre.barbier at cirad.fr (Pierre Barbier de Reuille) Date: Wed, 01 Jun 2005 14:32:54 +0200 Subject: [py-dev] Bug with py.test when object repr contains "\n" Message-ID: <429DAAF6.1090606@cirad.fr> Here an example : class Foo( object ): def __init__( self, a,b ): self.a = a self.b = b def __repr__( self ): return "Toto(%d,\n%d)" %( self.a, self.b ) def __len__( self ): return 2 def test_foo(): bar = Foo( 3,4 ) assert len( bar ) == 3 If you try to put that in a file a run py.test, it will fail in the nice_explanation function. This can be annoying :( I did not see any former reports about that ! Pierre -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68 From jan at balster.info Wed Jun 1 16:46:20 2005 From: jan at balster.info (Jan Balster) Date: Wed, 01 Jun 2005 16:46:20 +0200 Subject: [py-dev] Bug with py.test when object repr contains "\n" In-Reply-To: <429DAAF6.1090606@cirad.fr> References: <429DAAF6.1090606@cirad.fr> Message-ID: <429DCA3C.50909@balster.info> Pierre Barbier de Reuille wrote: Hi Pierre, here is a quick patch to fix your problem. The patch doesn't break any py-lib test but perhaps some tests in pypy. I won't commit the patch to the trunk unless I am sure it won't break anything. Perhaps Holger will do ;-) Jan Patch: --- exprinfo.py (revision 12973) +++ exprinfo.py (working copy) @@ -61,11 +61,13 @@ stackcnt[-1] += 1 stackcnt.append(0) result.append(' +' + ' '*(len(stack)-1) + s + line[1:]) - else: + elif line.startswith('}'): assert line.startswith('}') stack.pop() stackcnt.pop() result[stack[-1]] += line[1:] + else: + result.append(' '* len(result[-1])+ line) assert len(stack) == 1 return '\n'.join(result) From hpk at trillke.net Wed Jun 1 16:52:39 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 1 Jun 2005 16:52:39 +0200 Subject: [py-dev] Bug with py.test when object repr contains "\n" In-Reply-To: <429DCA3C.50909@balster.info> References: <429DAAF6.1090606@cirad.fr> <429DCA3C.50909@balster.info> Message-ID: <20050601145239.GW5694@solar.trillke.net> Hi Jan, On Wed, Jun 01, 2005 at 16:46 +0200, Jan Balster wrote: > Pierre Barbier de Reuille wrote: > > Hi Pierre, > > here is a quick patch to fix your problem. > The patch doesn't break any py-lib test but perhaps some tests in pypy. > > I won't commit the patch to the trunk unless I am sure it won't break > anything. Perhaps Holger will do ;-) such types of patches should not hamper PyPy. But it would be good to have a test for the actual problem that the patch intends to fix. Otherwise feel free to commit. cheers, holger > Patch: > > --- exprinfo.py (revision 12973) > +++ exprinfo.py (working copy) > @@ -61,11 +61,13 @@ > stackcnt[-1] += 1 > stackcnt.append(0) > result.append(' +' + ' '*(len(stack)-1) + s + line[1:]) > - else: > + elif line.startswith('}'): > assert line.startswith('}') > stack.pop() > stackcnt.pop() > result[stack[-1]] += line[1:] > + else: > + result.append(' '* len(result[-1])+ line) > assert len(stack) == 1 > return '\n'.join(result) > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From grig at gheorghiu.net Mon Jun 6 18:35:27 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 6 Jun 2005 09:35:27 -0700 (PDT) Subject: [py-dev] Logging in the py library Message-ID: <20050606163527.92130.qmail@web54510.mail.yahoo.com> I think it would be useful to implement logging in the py library via the 'logging' module, which is supposed to be thread-safe. Currently there are one-off implementations of logging/tracing scattered across source files (for example the trace method in execnet/gateway.py). It would be nice to have a uniform way of capturing debug information with various levels of severity (debug, info, warning, error, critical). If you think this is a good idea, let me know. I'm willing to take a stab at the implementation. Grig From hpk at trillke.net Tue Jun 7 11:44:15 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 7 Jun 2005 11:44:15 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050606163527.92130.qmail@web54510.mail.yahoo.com> References: <20050606163527.92130.qmail@web54510.mail.yahoo.com> Message-ID: <20050607094415.GW15602@solar.trillke.net> Hi Grig! On Mon, Jun 06, 2005 at 09:35 -0700, Grig Gheorghiu wrote: > I think it would be useful to implement logging in the py library via > the 'logging' module, which is supposed to be thread-safe. Currently > there are one-off implementations of logging/tracing scattered across > source files (for example the trace method in execnet/gateway.py). It > would be nice to have a uniform way of capturing debug information with > various levels of severity (debug, info, warning, error, critical). Adding Tracing/Logging to the py lib is definitely important. I'd like to hide the logging module under some nice super-simple API, though. Or maybe not use the logging module at all or only optionally :-) I think that the severity categorization is not enough to make nice use of logging information. I don't have a fully fledged idea but i think that having a keyword-based approach to tracing would be more helpful. Basically we could associate with each logging message/object [*] a set of keywords. severity categorizations would just be a keyword in that idea. Let me try an example for e.g. tracing in py/execnet/gateway.py: import py # this import is actually problematic to do in gateway.py, # let's forget about that detail for the moment trace = py.trace.keywordtracer('execnet gateway') class Gateway(object): def __init__(self, ...): trace("initializing gateway ...") ... 'trace' may by default add a 'debug' keyword if called like this. If we want to print some warning, then we could do something like: trace.warn("remote gateway side could not be destroyed") which would add the keyword 'warning' to that particular message. By default, everything would probably go to stdout but an application should be able to redirect by keyword to specific formatting/output backends (much like the logging module). I think it's important to allow this redirection to happen _after_ the above example is already imported, so redirections-by-keyword should be re-configurable at runtime. What do you think of this basic keyword idea? (which cold still use the logging module underneath for accessing all the backends if it makes sense). And sorry, but the py lib really aims at exploring improvements over current ways of doing things :-) cheers, holger [*] one concern is how much overhead tracing adds to an application. It should be possible to trace via objects (or lambdas/functions?, whatever) that only produce a str() representation when there is a consumer for the tracing information. In particular, if i "ground" the 'debug' keyword to /dev/null then the tracing code should not spend time str()ing objects. I don't like the logging module's way of turning messages into strings too early. So e.g. trace.debug("gateway", gateway, "processes", message) would not compute all the str()s of the objects if i (somehow) set the debug keyword to 'ignore'. From grig at gheorghiu.net Tue Jun 7 17:06:35 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 7 Jun 2005 08:06:35 -0700 (PDT) Subject: Fwd: Re: [py-dev] Logging in the py library Message-ID: <20050607150636.27215.qmail@web54505.mail.yahoo.com> For some reason I only sent the message below to Holger, not to the list. Here goes: ============================================================= Hi, Holger --- holger krekel wrote: > Adding Tracing/Logging to the py lib is definitely important. > I'd like to hide the logging module under some nice > super-simple API, though. Or maybe not use the logging module > at all or only optionally :-) > > I think that the severity categorization is not enough to make > nice use of logging information. I don't have a fully fledged > idea but i think that having a keyword-based approach to > tracing would be more helpful. Basically we could associate > with each logging message/object [*] a set of keywords. > severity categorizations would just be a keyword in that idea. > > Let me try an example for e.g. tracing in py/execnet/gateway.py: > > import py # this import is actually problematic to do in > gateway.py, > # let's forget about that detail for the moment > > trace = py.trace.keywordtracer('execnet gateway') > > class Gateway(object): > def __init__(self, ...): > trace("initializing gateway ...") > ... > > 'trace' may by default add a 'debug' keyword if called like this. > If we want to print some warning, then we could do something like: > > trace.warn("remote gateway side could not be destroyed") > > which would add the keyword 'warning' to that particular message. So is this association static or dynamic? If I understand correctly, you'd want it to be dynamic, right? So for example trace.foo would add 'foo' to the message. But the severity levels need to be statically defined, at least if we want to mimic what the logging module does. In the logging module, you can set a default severity level, so that the logger will output only those messages that have greater or equal severity levels. > > By default, everything would probably go to stdout but an > application should be able to redirect by keyword to specific > formatting/output backends (much like the logging module). I > think it's important to allow this redirection to happen > _after_ the above example is already imported, so > redirections-by-keyword should be re-configurable at runtime. Yes, it would be nice to have the possibility of sending the messages to a variety of handlers (stdout, files, sockets, HTTP servers, etc.) One reservation I have about stdout being the default is that it might interfere with py.test's own catching of stdout/stderr. I think it would be better to trace to a file by default (I think that's what the twisted.log module does too, although I haven't looked at it, I only saw it mentioned somewhere). > What do you think of this basic keyword idea? (which cold > still use the logging module underneath for accessing all > the backends if it makes sense). > > And sorry, but the py lib really aims at exploring > improvements over current ways of doing things :-) > I think it's a very good idea. In my opinion, it would still be worth using the logging module underneath, since all the grunt work is already done there. I hate reinventing wheels. But polishing and simplifying the interface to the logging module is certainly something to be aimed for -- and your keyword idea goes a long way towards that goal. > > [*] one concern is how much overhead tracing adds to an application. > It should be possible to trace via objects (or > lambdas/functions?, whatever) > that only produce a str() representation when there is a consumer > for the tracing information. In particular, if i "ground" the > 'debug' > keyword to /dev/null then the tracing code should not spend time > str()ing objects. I don't like the logging module's way of > turning messages into strings too early. So e.g. > > trace.debug("gateway", gateway, "processes", message) > > would not compute all the str()s of the objects if i (somehow) > set the debug keyword to 'ignore'. Agreed. Grig From hpk at trillke.net Tue Jun 7 18:09:08 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 7 Jun 2005 18:09:08 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050607145946.84445.qmail@web54502.mail.yahoo.com> References: <20050607094415.GW15602@solar.trillke.net> <20050607145946.84445.qmail@web54502.mail.yahoo.com> Message-ID: <20050607160908.GD21207@solar.trillke.net> Hi Grig, On Tue, Jun 07, 2005 at 07:59 -0700, Grig Gheorghiu wrote: > --- holger krekel wrote: > > ... > > Basically we could associate > > with each logging message/object [*] a set of keywords. > > severity categorizations would just be a keyword in that idea. > > > > Let me try an example for e.g. tracing in py/execnet/gateway.py: > > ... > > trace("initializing gateway ...") > > ... > > 'trace' may by default add a 'debug' keyword if called like this. > > If we want to print some warning, then we could do something like: > > > > trace.warn("remote gateway side could not be destroyed") > > > > which would add the keyword 'warning' to that particular message. > > So is this association static or dynamic? If I understand correctly, > you'd want it to be dynamic, right? No, the association of keywords with messages can be static. > So for example trace.foo would add > 'foo' to the message. But the severity levels need to be statically > defined, at least if we want to mimic what the logging module does. Yes, 'trace.debug' would return a tracer that has the same keywords as 'trace' + the keyword 'debug'. Any message this tracer then produces would have all the latter keywords associated with it. > In the logging module, you can set a default severity level, > so that the logger will output only those messages that have > greater or equal severity levels. This refers to output filters and such. I think it makes sense to differentiate between a 'producing' API (basically something very simple like above - just one 'tracer' factory function) and a consuming API dealing with how to redirect messages based on keywords to specific backends/formatters. IMO the connection between producing and consuming objects should be runtime re-configurable. So it's really a small event system and may even be generalized into that at some point. The logging module would only come into play on the consuming side an internal backend and we would expose as few details about this as feasible. > > By default, everything would probably go to stdout but an > > application should be able to redirect by keyword to specific > > formatting/output backends (much like the logging module). I > > think it's important to allow this redirection to happen > > _after_ the above example is already imported, so > > redirections-by-keyword should be re-configurable at runtime. > > Yes, it would be nice to have the possibility of sending the messages > to a variety of handlers (stdout, files, sockets, HTTP servers, etc.) > One reservation I have about stdout being the default is that it might > interfere with py.test's own catching of stdout/stderr. I think it > would be better to trace to a file by default (I think that's what the > twisted.log module does too, although I haven't looked at it, I only > saw it mentioned somewhere). don't worry, py.test could reconfigure the tracing accordingly. The idea above is really about an application-independent general mechanism. I believe that the basic tracer should be as close to a mere 'print' statement as possible. Hum, to keep things really minimal we might consider: import py py.trace.debug('hello', 'world') which would by default print something like: [debug] hello world Or, to make it even more obvious: print >>py.trace.debug, "hello", "world" No API is the best API, remember? :-) (the disadvantage is that the str()s are always computed but that is a somewhat unfortunate language implementation detail that could be changed at some point :-). The nice property of the above that it is a really simple idiom which makes it obvious that the tracing statement does not mutate the program state but only has a tracing side effect. > > What do you think of this basic keyword idea? (which cold > > still use the logging module underneath for accessing all > > the backends if it makes sense). > > > > And sorry, but the py lib really aims at exploring > > improvements over current ways of doing things :-) > > > > I think it's a very good idea. In my opinion, it would still be worth > using the logging module underneath, since all the grunt work is > already done there. I hate reinventing wheels. Agreed. However, before worrying about output handlers (where the logging module could really help) i'd like to have a nice pythonic tracing model that allows to connect e.g. print-statements to tracer functions (and we can offer some default set of tracer functions which itself use the logging module). Hum. Here is a thought example of how a _consuming_ tracer function could look like: def mytracer(message): if 'error' in message.keywords: print >>somefile, "error:", str(message) elif ... py.trace[...] = mytracer ^^^ literally :-) or to associate the consuming tracer function more directly: def myerrortracer(message): print >>errorfile, str(message) py.trace['error'] = myerrortracer Wouldn't the meaning of this API be obvious? (I am not explaining it on purpose here :-) Providing adapter-consuming-tracer-functions for using the logging-module's backends should then becomes an orthogonal disconnected issue. > But polishing and simplifying the interface to the logging > module is certainly something to be aimed for -- and your > keyword idea goes a long way towards that goal. great to hear! I have to admit i have thought about the 'tracing problem' quite a bit already and i had some discussion with Vinay at the time (the author of the logging module). Your bringing up of the topic forced me to try express my train of thoughts more clearly. cheers, holger From grig at gheorghiu.net Tue Jun 7 18:46:32 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 7 Jun 2005 09:46:32 -0700 (PDT) Subject: [py-dev] Logging in the py library In-Reply-To: <20050607160908.GD21207@solar.trillke.net> Message-ID: <20050607164632.77939.qmail@web54503.mail.yahoo.com> Holger, I think it makes a lot of sense to split the functionality between a very simple 'producer' API that is exposed to end-users and a potentially more complicated 'consumer' API that interfaces with the logging module. It's pretty clear you spent time thinking about all these things :-), so I think it would probably be best if you came up with a first cut at the 'producer' API and at how things tie together. Then I'd be glad to work on the 'back-end' API that interfaces with the logging module. Would that work for you? Grig --- holger krekel wrote: > Hi Grig, > > On Tue, Jun 07, 2005 at 07:59 -0700, Grig Gheorghiu wrote: > > --- holger krekel wrote: > > > ... > > > Basically we could associate > > > with each logging message/object [*] a set of keywords. > > > severity categorizations would just be a keyword in that idea. > > > > > > Let me try an example for e.g. tracing in py/execnet/gateway.py: > > > ... > > > trace("initializing gateway ...") > > > ... > > > 'trace' may by default add a 'debug' keyword if called like this. > > > > If we want to print some warning, then we could do something > like: > > > > > > trace.warn("remote gateway side could not be destroyed") > > > > > > which would add the keyword 'warning' to that particular message. > > > > > So is this association static or dynamic? If I understand > correctly, > > you'd want it to be dynamic, right? > > No, the association of keywords with messages can be static. > > > So for example trace.foo would add > > 'foo' to the message. But the severity levels need to be statically > > defined, at least if we want to mimic what the logging module does. > > > Yes, 'trace.debug' would return a tracer that has the same keywords > as 'trace' + the keyword 'debug'. Any message this tracer then > produces would have all the latter keywords associated with it. > > > In the logging module, you can set a default severity level, > > so that the logger will output only those messages that have > > greater or equal severity levels. > > This refers to output filters and such. I think it makes > sense to differentiate between a 'producing' API (basically something > > very simple like above - just one 'tracer' factory function) > and a consuming API dealing with how to redirect messages based > on keywords to specific backends/formatters. > > IMO the connection between producing and consuming objects should > be runtime re-configurable. So it's really a small event system > and may even be generalized into that at some point. The logging > module would only come into play on the consuming side an internal > backend and we would expose as few details about this as feasible. > > > > By default, everything would probably go to stdout but an > > > application should be able to redirect by keyword to specific > > > formatting/output backends (much like the logging module). I > > > think it's important to allow this redirection to happen > > > _after_ the above example is already imported, so > > > redirections-by-keyword should be re-configurable at runtime. > > > > Yes, it would be nice to have the possibility of sending the > messages > > to a variety of handlers (stdout, files, sockets, HTTP servers, > etc.) > > One reservation I have about stdout being the default is that it > might > > interfere with py.test's own catching of stdout/stderr. I think it > > would be better to trace to a file by default (I think that's what > the > > twisted.log module does too, although I haven't looked at it, I > only > > saw it mentioned somewhere). > > don't worry, py.test could reconfigure the tracing accordingly. > The idea above is really about an application-independent > general mechanism. I believe that the basic tracer should > be as close to a mere 'print' statement as possible. > > Hum, to keep things really minimal we might consider: > > import py > py.trace.debug('hello', 'world') > > which would by default print something like: > > [debug] hello world > > Or, to make it even more obvious: > > print >>py.trace.debug, "hello", "world" > > No API is the best API, remember? :-) > > (the disadvantage is that the str()s are always computed > but that is a somewhat unfortunate language implementation > detail that could be changed at some point :-). > > The nice property of the above that it is a really simple > idiom which makes it obvious that the tracing statement > does not mutate the program state but only has a tracing > side effect. > > > > What do you think of this basic keyword idea? (which cold > > > still use the logging module underneath for accessing all > > > the backends if it makes sense). > > > > > > And sorry, but the py lib really aims at exploring > > > improvements over current ways of doing things :-) > > > > > > > I think it's a very good idea. In my opinion, it would still be > worth > > using the logging module underneath, since all the grunt work is > > already done there. I hate reinventing wheels. > > Agreed. However, before worrying about output handlers (where > the logging module could really help) i'd like to have a nice > pythonic tracing model that allows to connect e.g. print-statements > to tracer functions (and we can offer some default set of > tracer functions which itself use the logging module). > > Hum. Here is a thought example of how a _consuming_ tracer > function could look like: > > def mytracer(message): > if 'error' in message.keywords: > print >>somefile, "error:", str(message) > elif ... > > py.trace[...] = mytracer > ^^^ literally :-) > > or to associate the consuming tracer function more directly: > > def myerrortracer(message): > print >>errorfile, str(message) > > py.trace['error'] = myerrortracer > > Wouldn't the meaning of this API be obvious? > (I am not explaining it on purpose here :-) > > Providing adapter-consuming-tracer-functions for using the > logging-module's backends should then becomes an orthogonal > disconnected issue. > > > But polishing and simplifying the interface to the logging > > module is certainly something to be aimed for -- and your > > keyword idea goes a long way towards that goal. > > great to hear! > > I have to admit i have thought about the 'tracing problem' > quite a bit already and i had some discussion with Vinay at > the time (the author of the logging module). Your bringing > up of the topic forced me to try express my train of thoughts > more clearly. > > cheers, > > holger > From hpk at trillke.net Tue Jun 7 18:51:11 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 7 Jun 2005 18:51:11 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050607164632.77939.qmail@web54503.mail.yahoo.com> References: <20050607160908.GD21207@solar.trillke.net> <20050607164632.77939.qmail@web54503.mail.yahoo.com> Message-ID: <20050607165111.GH21207@solar.trillke.net> On Tue, Jun 07, 2005 at 09:46 -0700, Grig Gheorghiu wrote: > Holger, > > I think it makes a lot of sense to split the functionality between a > very simple 'producer' API that is exposed to end-users and a > potentially more complicated 'consumer' API that interfaces with the > logging module. It's pretty clear you spent time thinking about all > these things :-), so I think it would probably be best if you came up > with a first cut at the 'producer' API and at how things tie together. > Then I'd be glad to work on the 'back-end' API that interfaces with the > logging module. Would that work for you? Hehe, i guess so. Let me bet - for the fun of it - that the producer+consumer API implementation roughly described in the last mail will not exceed 70 lines of code :-) cheers, holger > --- holger krekel wrote: > > > Hi Grig, > > > > On Tue, Jun 07, 2005 at 07:59 -0700, Grig Gheorghiu wrote: > > > --- holger krekel wrote: > > > > ... > > > > Basically we could associate > > > > with each logging message/object [*] a set of keywords. > > > > severity categorizations would just be a keyword in that idea. > > > > > > > > Let me try an example for e.g. tracing in py/execnet/gateway.py: > > > > ... > > > > trace("initializing gateway ...") > > > > ... > > > > 'trace' may by default add a 'debug' keyword if called like this. > > > > > > If we want to print some warning, then we could do something > > like: > > > > > > > > trace.warn("remote gateway side could not be destroyed") > > > > > > > > which would add the keyword 'warning' to that particular message. > > > > > > > > So is this association static or dynamic? If I understand > > correctly, > > > you'd want it to be dynamic, right? > > > > No, the association of keywords with messages can be static. > > > > > So for example trace.foo would add > > > 'foo' to the message. But the severity levels need to be statically > > > defined, at least if we want to mimic what the logging module does. > > > > > > Yes, 'trace.debug' would return a tracer that has the same keywords > > as 'trace' + the keyword 'debug'. Any message this tracer then > > produces would have all the latter keywords associated with it. > > > > > In the logging module, you can set a default severity level, > > > so that the logger will output only those messages that have > > > greater or equal severity levels. > > > > This refers to output filters and such. I think it makes > > sense to differentiate between a 'producing' API (basically something > > > > very simple like above - just one 'tracer' factory function) > > and a consuming API dealing with how to redirect messages based > > on keywords to specific backends/formatters. > > > > IMO the connection between producing and consuming objects should > > be runtime re-configurable. So it's really a small event system > > and may even be generalized into that at some point. The logging > > module would only come into play on the consuming side an internal > > backend and we would expose as few details about this as feasible. > > > > > > By default, everything would probably go to stdout but an > > > > application should be able to redirect by keyword to specific > > > > formatting/output backends (much like the logging module). I > > > > think it's important to allow this redirection to happen > > > > _after_ the above example is already imported, so > > > > redirections-by-keyword should be re-configurable at runtime. > > > > > > Yes, it would be nice to have the possibility of sending the > > messages > > > to a variety of handlers (stdout, files, sockets, HTTP servers, > > etc.) > > > One reservation I have about stdout being the default is that it > > might > > > interfere with py.test's own catching of stdout/stderr. I think it > > > would be better to trace to a file by default (I think that's what > > the > > > twisted.log module does too, although I haven't looked at it, I > > only > > > saw it mentioned somewhere). > > > > don't worry, py.test could reconfigure the tracing accordingly. > > The idea above is really about an application-independent > > general mechanism. I believe that the basic tracer should > > be as close to a mere 'print' statement as possible. > > > > Hum, to keep things really minimal we might consider: > > > > import py > > py.trace.debug('hello', 'world') > > > > which would by default print something like: > > > > [debug] hello world > > > > Or, to make it even more obvious: > > > > print >>py.trace.debug, "hello", "world" > > > > No API is the best API, remember? :-) > > > > (the disadvantage is that the str()s are always computed > > but that is a somewhat unfortunate language implementation > > detail that could be changed at some point :-). > > > > The nice property of the above that it is a really simple > > idiom which makes it obvious that the tracing statement > > does not mutate the program state but only has a tracing > > side effect. > > > > > > What do you think of this basic keyword idea? (which cold > > > > still use the logging module underneath for accessing all > > > > the backends if it makes sense). > > > > > > > > And sorry, but the py lib really aims at exploring > > > > improvements over current ways of doing things :-) > > > > > > > > > > I think it's a very good idea. In my opinion, it would still be > > worth > > > using the logging module underneath, since all the grunt work is > > > already done there. I hate reinventing wheels. > > > > Agreed. However, before worrying about output handlers (where > > the logging module could really help) i'd like to have a nice > > pythonic tracing model that allows to connect e.g. print-statements > > to tracer functions (and we can offer some default set of > > tracer functions which itself use the logging module). > > > > Hum. Here is a thought example of how a _consuming_ tracer > > function could look like: > > > > def mytracer(message): > > if 'error' in message.keywords: > > print >>somefile, "error:", str(message) > > elif ... > > > > py.trace[...] = mytracer > > ^^^ literally :-) > > > > or to associate the consuming tracer function more directly: > > > > def myerrortracer(message): > > print >>errorfile, str(message) > > > > py.trace['error'] = myerrortracer > > > > Wouldn't the meaning of this API be obvious? > > (I am not explaining it on purpose here :-) > > > > Providing adapter-consuming-tracer-functions for using the > > logging-module's backends should then becomes an orthogonal > > disconnected issue. > > > > > But polishing and simplifying the interface to the logging > > > module is certainly something to be aimed for -- and your > > > keyword idea goes a long way towards that goal. > > > > great to hear! > > > > I have to admit i have thought about the 'tracing problem' > > quite a bit already and i had some discussion with Vinay at > > the time (the author of the logging module). Your bringing > > up of the topic forced me to try express my train of thoughts > > more clearly. > > > > cheers, > > > > holger > > > From dstanek at dstanek.com Tue Jun 7 19:38:25 2005 From: dstanek at dstanek.com (dstanek at dstanek.com) Date: Tue, 7 Jun 2005 13:38:25 -0400 (EDT) Subject: Fwd: Re: [py-dev] Logging in the py library In-Reply-To: <20050607150636.27215.qmail@web54505.mail.yahoo.com> References: <20050607150636.27215.qmail@web54505.mail.yahoo.com> Message-ID: <44237.207.58.192.150.1118165905.squirrel@207.58.192.150> > I think it's a very good idea. In my opinion, it would still be worth > using the logging module underneath, since all the grunt work is > already done there. I hate reinventing wheels. But polishing and > simplifying the interface to the logging module is certainly something > to be aimed for -- and your keyword idea goes a long way towards that > goal. I like the idea of having a keywork based trace module that simply wraps the logging module. Doing with would allow pylib to take advantage of all the handlers without having to write the code. David From ischenko at gmail.com Tue Jun 7 20:55:22 2005 From: ischenko at gmail.com (Max Ischenko) Date: Tue, 7 Jun 2005 21:55:22 +0300 Subject: [py-dev] running doctest-based tests Message-ID: <20050607185517.9A62327B3C@code1.codespeak.net> Hi, I have doctests in my source code which I want to have executed by py.test. >From what I understood, I need to add a conftest.py to my source package directory and specify some collector that is able to collect and run these tests. Is there any pre-built public code to do this? Looking thru py.test sources do not reveal much (except the DoctestText which seems to be not what I need). PS: I've searched thru archives and there appear quite a few threads but I didn't find any meaningful conclusions or recipes. Thanks. From ischenko at gmail.com Wed Jun 8 13:18:45 2005 From: ischenko at gmail.com (Max Ischenko) Date: Wed, 8 Jun 2005 14:18:45 +0300 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050607185517.9A62327B3C@code1.codespeak.net> Message-ID: <20050608111848.AFECE27B48@code1.codespeak.net> > I have doctests in my source code which I want to have executed by py.test. > > From what I understood, I need to add a conftest.py to my source package > directory and specify some collector that is able to collect and run these > tests. I have made some progress in implementing it myself but stuck with some mysterious error. I assume that the Item, when encounter a failure must raise a corresponding Outcome exception (which get caught by runtraced() method in session module. In my case, it got caught but then py.test breaks nevertheless and shows a traceback that point to my raise Failed() statement. Somehow it continues to think of my code as part of client code, that as library code. This is my guess, at least. Here is my conftest.py: class Directory(py.test.collect.Directory): def buildname2items(self): # just let our base class do its building d = super(Directory, self).buildname2items() # let's look for doctests ... for fn in self.fspath.listdir('*.py'): if fn.basename in d or fn.basename == 'conftest.py': continue # we have a candidate for doctests if not isdoctestmodule(fn): # if possible to implement that continue module = DoctestModule(fn, parent=self) d[fn.basename] = module print 'collected %d modules' % len(d) return d class DoctestModule(py.test.collect.Module): def funcnamefilter(self, name): return True def classnamefilter(self, name): return True def buildname2items(self): d = {} finder = doctest.DocTestFinder() doctests = finder.find(self.obj) for test in doctests: if test.examples: d[test.name] = DoctestItem(test, self) return d class DoctestItem(py.test.Item, doctest.DocTestRunner): def __init__(self, test, parent): py.test.Item.__init__(self, test.name, parent) doctest.DocTestRunner.__init__(self) self.test = test def run(self): failed, tried = doctest.DocTestRunner.run(self, self.test) if failed: raise self.Failed('ooops') When run, traceback looks like this: ____________________________________________________________________________ ___ def runtraced(self, colitem): if self.shouldclose(): raise Exit, "received external close signal" outcome = None colitem.startcapture() try: self.start(colitem) try: try: if colitem._stickyfailure: raise colitem._stickyfailure > outcome = self.run(colitem) [D:\Projects\3rd-party\py\test\session.py:75] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def run(self, colitem): if self.config.option.collectonly and isinstance(colitem, py.test.Item): return if isinstance(colitem, py.test.Item): self.skipbykeyword(colitem) > res = colitem.run() [D:\Projects\3rd-party\py\test\session.py:101] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def run(self): failed, tried = doctest.DocTestRunner.run(self, self.test) if failed: E raise self.Failed('ooops') > Failed: ooops = From dstanek at dstanek.com Wed Jun 8 14:12:23 2005 From: dstanek at dstanek.com (David Stanek) Date: Wed, 8 Jun 2005 08:12:23 -0400 (EDT) Subject: [py-dev] Future export interface Message-ID: <21801.207.58.192.150.1118232743.squirrel@207.58.192.150> I want to get my hands a little dirty by playing with the export/import changes that are in the future document. The example of the new specification looks like: from py import pkg pkg.export(__name__, pkg.Class('local', 'doc comment', extpy='./path/local.LocalPath'), pkg.Class('svnurl', 'doc comment', extpy='./path/local/svn/urlcommand.SvnUrlPath'), ), # it goes on ... ) I was thinking about how this would look a little and I am unsure of it. The keyword arguments that are currently passed to initpkg are missing from the example and adding them in to the export call may look clumsy. I was thinking of something like: from py import pkg exporter = pkg.Exporter(__name__, description, version, url, etc...) exporter.exportModule('path', 'doc comment', pkg.Class('local', 'doc comment', extpy='./path/local.Localpath'), pkg.Class('svnurl', 'doc comment', extpy='./path/local/svn/urlcommand.SvnUrlPath'), # more exported classes ) exporter.exportModule('junk', 'doc comment', pkg.Module('stuff', 'doc comment', pkg.Class('Stuff', 'comment') ) ) That is an attempt to flatten the data structure and make it more readable. Regards, David From hpk at trillke.net Wed Jun 8 15:49:07 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 8 Jun 2005 15:49:07 +0200 Subject: [py-dev] Future export interface In-Reply-To: <21801.207.58.192.150.1118232743.squirrel@207.58.192.150> References: <21801.207.58.192.150.1118232743.squirrel@207.58.192.150> Message-ID: <20050608134907.GK21207@solar.trillke.net> Hi David, On Wed, Jun 08, 2005 at 08:12 -0400, David Stanek wrote: > I want to get my hands a little dirty by playing with the > export/import changes that are in the future document. The example > of the new specification looks like: > > from py import pkg > pkg.export(__name__, > pkg.Class('local', 'doc comment', extpy='./path/local.LocalPath'), > pkg.Class('svnurl', 'doc comment', > extpy='./path/local/svn/urlcommand.SvnUrlPath'), > ), > # it goes on ... > ) > > I was thinking about how this would look a little and I am unsure of it. > > The keyword arguments that are currently passed to initpkg are missing > from the example and adding them in to the export call may look clumsy. > I was thinking of something like: > > from py import pkg > exporter = pkg.Exporter(__name__, description, version, url, etc...) > exporter.exportModule('path', 'doc comment', > pkg.Class('local', 'doc comment', extpy='./path/local.Localpath'), > pkg.Class('svnurl', 'doc comment', > extpy='./path/local/svn/urlcommand.SvnUrlPath'), > # more exported classes > ) > exporter.exportModule('junk', 'doc comment', > pkg.Module('stuff', 'doc comment', > pkg.Class('Stuff', 'comment') > ) > ) > > That is an attempt to flatten the data structure and make it more > readable. That's basically interesting but i think we shouldn't work on that before the 0.8 release is out. However, after the current import/export mechanisms seems to be be pretty stable i am more hesitant to go after a big scale change. Just going for a minimal way to add docstrings to the current dictionary-based export specification seems more worthwhile. Side note: I am still heading towards getting an issue tracker up for organizing the remaining issues before the release. cheers, holger From hpk at trillke.net Wed Jun 8 15:51:38 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 8 Jun 2005 15:51:38 +0200 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050608111848.AFECE27B48@code1.codespeak.net> References: <20050607185517.9A62327B3C@code1.codespeak.net> <20050608111848.AFECE27B48@code1.codespeak.net> Message-ID: <20050608135138.GL21207@solar.trillke.net> On Wed, Jun 08, 2005 at 14:18 +0300, Max Ischenko wrote: > > I have doctests in my source code which I want to have executed by > py.test. > > > > From what I understood, I need to add a conftest.py to my source package > > directory and specify some collector that is able to collect and run these > > tests. > > I have made some progress in implementing it myself but stuck with some > mysterious error. > > I assume that the Item, when encounter a failure must raise a corresponding > Outcome exception (which get caught by runtraced() method in session module. > > In my case, it got caught but then py.test breaks nevertheless and shows a > traceback that point to my raise Failed() statement. Somehow it continues to > think of my code as part of client code, that as library code. This is my > guess, at least. I think the code is basically fine (i had a feeling of deja-vu, did you take this code from some py-dev posting? just curious ...). However, item.run() methods currently doesn't have direct means to control output in case of a test failure. What we should do, i guess, is to say that an explicit 'py.test.skip()' doesn't print any traceback (maybe controlable by some command-line argument). Instead it prints only the given message. You could then extend your doctest approach to produce the message you want py.test to show. cheers, holger From ischenko at gmail.com Wed Jun 8 16:08:21 2005 From: ischenko at gmail.com (Max Ischenko) Date: Wed, 8 Jun 2005 17:08:21 +0300 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050608135138.GL21207@solar.trillke.net> Message-ID: <20050608140826.9173027B4B@code1.codespeak.net> > On Wed, Jun 08, 2005 at 14:18 +0300, Max Ischenko wrote: > > I have made some progress in implementing it myself but stuck with some > > mysterious error. > > > > I assume that the Item, when encounter a failure must raise a corresponding > > Outcome exception (which get caught by runtraced() method in session module. > > > > In my case, it got caught but then py.test breaks nevertheless and shows a > > traceback that point to my raise Failed() statement. Somehow it continues to > > think of my code as part of client code, that as library code. This is my > > guess, at least. > > I think the code is basically fine (i had a feeling of deja-vu, did you > take this code from some py-dev posting? just curious ...). That's it. ;) That's a sample code you wrote when replying to Ian. > However, item.run() methods currently doesn't have direct means to control > output in case of a test failure. What we should do, i guess, is > to say that an explicit 'py.test.skip()' doesn't print any traceback > (maybe controlable by some command-line argument). Instead it prints > only the given message. You could then extend your doctest approach > to produce the message you want py.test to show. Sorry, don't get it. My problem is how to communicate test failures as reported by doctest back to py.test. If all tests pass then it works ok, printing correct test names and correct number of dots (each doctest got a separate Item). But when I'm trying to report failures to py.test (using raise self.Failed) I got this awkward traceback. I looked thru the py.test sources but the exception is catched but then is magically re-thrown again. Either I'm doing something wrong (or in the wrong place) or there is a bug in py.test (I'm using the HEAD trunk). From wam at cisco.com Wed Jun 8 23:18:25 2005 From: wam at cisco.com (William McVey) Date: Wed, 08 Jun 2005 16:18:25 -0500 Subject: [py-dev] handling of 'class' attribute by py.xml.html Message-ID: <1118265505.3343.86.camel@localhost> I needed to be able to construct html elements with the 'class' attribute, in order to properly style the element with a set of pre-existing stylesheets. Ordinarily, you would think that this could be something as simple as: from py.xml import html html.div(class="somestyleclass") This obviously doesn't work because 'class' is a reserved word in python. I "solved" this using a quick hack in my local copy of the library by adding a quick hack that maps named parameter 'Class' to attribute 'class'. It's certainly not elegant, but it is effective. The diff to py/xmlobj/html.py is attached. -- William -------------- next part -------------- A non-text attachment was scrubbed... Name: html.py-Class-hack.patch Type: text/x-patch Size: 546 bytes Desc: not available URL: From hpk at trillke.net Wed Jun 8 23:59:28 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 8 Jun 2005 23:59:28 +0200 Subject: [py-dev] handling of 'class' attribute by py.xml.html In-Reply-To: <1118265505.3343.86.camel@localhost> References: <1118265505.3343.86.camel@localhost> Message-ID: <20050608215928.GN21207@solar.trillke.net> On Wed, Jun 08, 2005 at 16:18 -0500, William McVey wrote: > I needed to be able to construct html elements with the 'class' > attribute, in order to properly style the element with a set of > pre-existing stylesheets. Ordinarily, you would think that this could be > something as simple as: > > from py.xml import html > html.div(class="somestyleclass") > > This obviously doesn't work because 'class' is a reserved word in > python. However, html.div(class_="somestyleclass") does produce '
' as expected. > I "solved" this using a quick hack in my local copy of the > library by adding a quick hack that maps named parameter 'Class' to > attribute 'class'. ... > RCS file: /cvs/report/lib/python/py/xmlobj/html.py,v > retrieving revision 1.4 > diff -c -3 -r1.4 html.py > *** html.py 27 Apr 2005 19:47:34 -0000 1.4 > --- html.py 8 Jun 2005 21:17:17 -0000 > *************** > *** 20,25 **** > --- 20,26 ---- > class Style(object): > def __init__(self, **kw): > for x, y in kw.items(): > + if x == "Class": x = "class" > x = x.replace('_', '-') > setattr(self, x, y) Here you are modifying the html.Style class which does not refer to the example you gave above. However, a Style object should not contain a css 'class' anyway. Can you post a more real-life example to show the problem you are hitting? thanks & cheers, holger From wam at cisco.com Thu Jun 9 00:20:30 2005 From: wam at cisco.com (William McVey) Date: Wed, 08 Jun 2005 17:20:30 -0500 Subject: [py-dev] handling of 'class' attribute by py.xml.html In-Reply-To: <20050608215928.GN21207@solar.trillke.net> References: <1118265505.3343.86.camel@localhost> <20050608215928.GN21207@solar.trillke.net> Message-ID: <1118269230.3343.99.camel@localhost> On Wed, 2005-06-08 at 16:59, holger krekel wrote: > However, > html.div(class_="somestyleclass") > > does produce '
' as expected. Ahhh, that's perfect. > Here you are modifying the html.Style class which > does not refer to the example you gave above. > However, a Style object should not contain a css 'class' > anyway. Can you post a more real-life example to show > the problem you are hitting? Well, the example I gave pretty much was the real-world example. I needed to put 'class' attributes on html object elements. You're right though, my patch wasn't the solution I had written to solve this earlier. Your solution (trailing underscore) works just fine though so I'm not going to bother finding the hack I had originally come up with though. Thanks again. -- William From hpk at trillke.net Thu Jun 9 12:05:06 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 9 Jun 2005 12:05:06 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050607165111.GH21207@solar.trillke.net> References: <20050607160908.GD21207@solar.trillke.net> <20050607164632.77939.qmail@web54503.mail.yahoo.com> <20050607165111.GH21207@solar.trillke.net> Message-ID: <20050609100506.GO21207@solar.trillke.net> Hi Grig, hi all, On Tue, Jun 07, 2005 at 18:51 +0200, holger krekel wrote: > On Tue, Jun 07, 2005 at 09:46 -0700, Grig Gheorghiu wrote: > > Holger, > > > > I think it makes a lot of sense to split the functionality between a > > very simple 'producer' API that is exposed to end-users and a > > potentially more complicated 'consumer' API that interfaces with the > > logging module. It's pretty clear you spent time thinking about all > > these things :-), so I think it would probably be best if you came up > > with a first cut at the 'producer' API and at how things tie together. > > Then I'd be glad to work on the 'back-end' API that interfaces with the > > logging module. Would that work for you? > > Hehe, i guess so. Let me bet - for the fun of it - that the > producer+consumer API implementation roughly described in the > last mail will not exceed 70 lines of code :-) Ok, done that. It's at 54 lines of code but i feel it's already too convoluted. You'll find the code in py/misc/trace.py and py/misc/testing/test_trace.py shows some usages. The main missing piece involves thinking more about the dispatching (possibly based on partial keyword matches) and the actual API / namespace organization, for example of where to offer logging-module adapter functions. Please consider this trace.py experimental. But it hopefully gives a more concrete clue of the ideas. Feedback or suggestions very welcome (i think especially by using it a bit here and there we could gain insights/better approaches). cheers, holger From ischenko at gmail.com Thu Jun 9 12:27:18 2005 From: ischenko at gmail.com (Max Ischenko) Date: Thu, 9 Jun 2005 13:27:18 +0300 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050608135138.GL21207@solar.trillke.net> Message-ID: <20050609102723.5630527B6D@code1.codespeak.net> > However, item.run() methods currently doesn't have direct means to > control output in case of a test failure. What we should do, i guess, > is to say that an explicit 'py.test.skip()' doesn't print any > traceback (maybe controlable by some command-line argument). Instead > it prints only the given message. You could then extend your doctest > approach to produce the message you want py.test to show. After studying the code I found out that the way failures are reported is hardcoded in repr_failure method in TerminalSession. Even tkinter backend uses it. My situation requires a custom representation of a failure instead of plain traceback. A solution might be to abstract the way failures are reported by moving this responsibility from TerminalSession to the Failure class, for example. After all, this even seems logical - different kind of failures may require different representations and a Failure class is a natural place to hold this information. OTOH, the session classes *are* responsible for an actual failure's representation: be it console output, graphical window or network socket. A method to the Failure class will be added that will return a suitable string representation of a failure. Failure class itself will provide default implementation that will yield formatted traceback. This way I could subclass Failure that will yield doctest output instead of traceback. What would you say? From dstanek at dstanek.com Thu Jun 9 11:33:14 2005 From: dstanek at dstanek.com (David Stanek) Date: Thu, 9 Jun 2005 05:33:14 -0400 (EDT) Subject: [py-dev] running doctest-based tests In-Reply-To: <20050609102723.5630527B6D@code1.codespeak.net> References: <20050608135138.GL21207@solar.trillke.net> <20050609102723.5630527B6D@code1.codespeak.net> Message-ID: <35083.192.168.1.1.1118309594.squirrel@192.168.1.1> > After studying the code I found out that the way failures are reported is > hardcoded in repr_failure method in TerminalSession. Even tkinter backend > uses it. > > My situation requires a custom representation of a failure instead of > plain > traceback. > > A solution might be to abstract the way failures are reported by moving > this > responsibility from TerminalSession to the Failure class, for example. > After > all, this even seems logical - different kind of failures may require > different representations and a Failure class is a natural place to hold > this information. OTOH, the session classes *are* responsible for an > actual > failure's representation: be it console output, graphical window or > network > socket. > > A method to the Failure class will be added that will return a suitable > string representation of a failure. Failure class itself will provide > default implementation that will yield formatted traceback. This way I > could subclass Failure that will yield doctest output instead of > traceback. > > What would you say? I made a couple of changes to your conftest.py. The original one did not run under my version of Python. The classes being used were added in 2.4. Please see if that works for you. Later today I'll investigate Holger's suggestion. David ::::::conftest.py:::::: import py import doctest class Directory(py.test.collect.Directory): def buildname2items(self): # just let our base class do its building d = super(Directory, self).buildname2items() # let's look for doctests ... for fn in self.fspath.listdir('*.py'): if fn.basename in d or fn.basename == 'conftest.py': continue # we have a candidate for doctests #if not isdoctestmodule(fn): # if possible to implement that # continue module = DoctestModule(fn, parent=self) d[fn.basename] = module return d class DoctestModule(py.test.collect.Module): def funcnamefilter(self, name): return True def classnamefilter(self, name): return True def buildname2items(self): d = {} doctests = doctest._find_tests(self.obj) tester = doctest.Tester(mod=self.obj, verbose=0) for (testname, docstring, filename, lineno) in doctests: d[testname] = DoctestItem(testname, docstring, tester, self) return d class DoctestItem(py.test.Item): def __init__(self, name, test, tester, parent): py.test.Item.__init__(self, name, parent) self.test = test self.tester = tester def run(self): failed, tried = self.tester.runstring(self.test, self.name) if failed: raise self.Failed('ooops') From phil at base-art.net Thu Jun 9 13:28:48 2005 From: phil at base-art.net (Philippe Normand) Date: Thu, 09 Jun 2005 13:28:48 +0200 Subject: [py-dev] tests running order Message-ID: <1118316528.3779.80.camel@gaudi.enstb.org> Hi, I'd like my unit-tests to run in a specific order. For instance take two files test_foo.py and test_bar.py. I suppose if i run py.test, test_bar tests will be executed before test_foo (alphabetic order). Now if i want to run test_foo before test_bar ? Well the following catch may work: py.test test_foo.py py.test test_bar.py But in a more complex setup (many files), this can become quite painful.. Any ideas ? Cheers, Philippe From hpk at trillke.net Thu Jun 9 13:48:37 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 9 Jun 2005 13:48:37 +0200 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050609102723.5630527B6D@code1.codespeak.net> References: <20050608135138.GL21207@solar.trillke.net> <20050609102723.5630527B6D@code1.codespeak.net> Message-ID: <20050609114837.GP21207@solar.trillke.net> On Thu, Jun 09, 2005 at 13:27 +0300, Max Ischenko wrote: > > However, item.run() methods currently doesn't have direct means to > > control output in case of a test failure. What we should do, i guess, > > is to say that an explicit 'py.test.skip()' doesn't print any > > traceback (maybe controlable by some command-line argument). Instead > > it prints only the given message. You could then extend your doctest > > approach to produce the message you want py.test to show. > > After studying the code I found out that the way failures are reported is > hardcoded in repr_failure method in TerminalSession. Even tkinter backend > uses it. > > My situation requires a custom representation of a failure instead of plain > traceback. > > A solution might be to abstract the way failures are reported by moving this > responsibility from TerminalSession to the Failure class, for example. After > all, this even seems logical - different kind of failures may require > different representations and a Failure class is a natural place to hold > this information. OTOH, the session classes *are* responsible for an actual > failure's representation: be it console output, graphical window or network > socket. > > A method to the Failure class will be added that will return a suitable > string representation of a failure. Failure class itself will provide > default implementation that will yield formatted traceback. This way I > could subclass Failure that will yield doctest output instead of traceback. > > What would you say? your analysis is pretty correct. However, my idea was to just let py.test.skip(someobj) internally take care that str(someobj) provides the failure message representation. Doing that by an internal subclass is fine but an implementation detail that should not get exposed (Failure classes are not visible directly). cheers, holger From hpk at trillke.net Thu Jun 9 14:00:19 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 9 Jun 2005 14:00:19 +0200 Subject: [py-dev] tests running order In-Reply-To: <1118316528.3779.80.camel@gaudi.enstb.org> References: <1118316528.3779.80.camel@gaudi.enstb.org> Message-ID: <20050609120019.GQ21207@solar.trillke.net> Hi Philippe, On Thu, Jun 09, 2005 at 13:28 +0200, Philippe Normand wrote: > I'd like my unit-tests to run in a specific order. For instance take two > files test_foo.py and test_bar.py. I suppose if i run py.test, test_bar > tests will be executed before test_foo (alphabetic order). Now if i want > to run test_foo before test_bar ? > > Well the following catch may work: > > py.test test_foo.py > py.test test_bar.py > > But in a more complex setup (many files), this can become quite > painful.. Any ideas ? You will need to put a conftest.py file into your testing directory with the following example content: import py mydir = py.magic.autopath().dirpath() # directory of this conftest.py class Directory(py.test.collect.Directory): def run(self): if self.fspath != mydir: # somewhere below return super(Directory, self).run() return ['test_foo.py', 'test_bar.py'] conftest.py files are reponsible for their complete subdirectory hierarchy, therefore we need to make sure we are realling being asked (via the session calling our run() method) for our own exact directory. run() methods need to return a list of basenames. If you take over "test collection" like this then you need to explicitely list all files as py.test will obviously not collect files automatically in this directory anymore. makes all sense to you? You may want to read this to understand more: http://codespeak.net/py/current/doc/test.html#collecting-and-running-tests-implementation-remarks (sorry for the long URL). cheers, holger From phil at base-art.net Thu Jun 9 14:07:27 2005 From: phil at base-art.net (Philippe Normand) Date: Thu, 09 Jun 2005 14:07:27 +0200 Subject: [py-dev] tests running order In-Reply-To: <20050609120019.GQ21207@solar.trillke.net> References: <1118316528.3779.80.camel@gaudi.enstb.org> <20050609120019.GQ21207@solar.trillke.net> Message-ID: <1118318847.3784.83.camel@gaudi.enstb.org> Le jeudi 09 juin 2005 ? 14:00 +0200, holger krekel a ?crit : > makes all sense to you? You may want to read this to understand more: > Yes, it makes sense :) Thanks! Philippe From hpk at trillke.net Thu Jun 9 14:33:24 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 9 Jun 2005 14:33:24 +0200 Subject: [py-dev] tests running order In-Reply-To: <1118318847.3784.83.camel@gaudi.enstb.org> References: <1118316528.3779.80.camel@gaudi.enstb.org> <20050609120019.GQ21207@solar.trillke.net> <1118318847.3784.83.camel@gaudi.enstb.org> Message-ID: <20050609123324.GT21207@solar.trillke.net> On Thu, Jun 09, 2005 at 14:07 +0200, Philippe Normand wrote: > Le jeudi 09 juin 2005 ? 14:00 +0200, holger krekel a ?crit : > > > makes all sense to you? You may want to read this to understand more: > > > > Yes, it makes sense :) Thanks! however, i should have mentioned that simply naming your files test_0_foo.py test_1_bar.py would be another maybe better possibility. It expresses the intent more explicitely on the filesystem level so that people will maybe first look into test_0_*.py stuff when looking how something works ... cheers, holger From grig at gheorghiu.net Thu Jun 9 17:01:26 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Thu, 9 Jun 2005 08:01:26 -0700 (PDT) Subject: [py-dev] Logging in the py library In-Reply-To: <20050609100506.GO21207@solar.trillke.net> Message-ID: <20050609150126.85181.qmail@web54502.mail.yahoo.com> Hi, Holger --- holger krekel wrote: > Hi Grig, hi all, > > Ok, done that. It's at 54 lines of code but i feel it's already > too convoluted. You'll find the code in py/misc/trace.py and > py/misc/testing/test_trace.py shows some usages. > > The main missing piece involves thinking more about the dispatching > (possibly based on partial keyword matches) and the actual > API / namespace organization, for example of where to offer > logging-module adapter functions. > > Please consider this trace.py experimental. But it hopefully > gives a more concrete clue of the ideas. Feedback or suggestions > very welcome (i think especially by using it a bit here and there > we could gain insights/better approaches). I'm currently trying to wrap my tired brain around the trace module :-) It's deceptively simple if you look just at the number of LOC. I'll play with it and I'll try to use it in the py library, so that we can eat our own dog food. This will be a good way of seeing what's still needed etc. Anyway, thanks for putting this together, for me personally it's a tutorial in using less obvious features of Python.... Grig From ischenko at gmail.com Fri Jun 10 12:07:06 2005 From: ischenko at gmail.com (Max Ischenko) Date: Fri, 10 Jun 2005 13:07:06 +0300 Subject: [py-dev] running doctest-based tests In-Reply-To: <35083.192.168.1.1.1118309594.squirrel@192.168.1.1> Message-ID: <20050610100709.E442827B5E@code1.codespeak.net> David, > I made a couple of changes to your conftest.py. The original one did not > run under my version of Python. The classes being used were added in 2.4. > Please see if that works for you. Later today I'll investigate Holger's > suggestion. Besides DoctestFinder (that you have replaced with Tester) you need a way to capture doctest output as well. DocTest in 2.4 provides nice report_* methods plus an optional 'writer' argument to aid this task. Have no idea how it'd translate for pre-2.4 version. Python doctest module has been improved considerably for 2.4 release and I think the simplest and reasonable way to make it work for version < 2.4 is to use standalone doctest24 module (can be found on the web). If my suggestion (to use doctest24.py) is not viable for some reason in your situation then you have to rework DoctestItem.run() as well. Btw, the code you posted doesn't work with Python 2.4 because doctest has no function _find_tests() anymore. Here is my current code: class DoctestItem(py.test.Item): def __init__(self, test, parent): py.test.Item.__init__(self, test.name, parent) self.test = test def run(self): runner = doctest.DocTestRunner() out = StringIO() writer = lambda msg: out.write(msg) failed, tried = runner.run(self.test, out=writer) if failed: py.test.skip(out.getvalue()) class DoctestModule(py.test.collect.Module): def funcnamefilter(self, name): return True def classnamefilter(self, name): return True def buildname2items(self): d = {} finder = doctest.DocTestFinder() doctests = finder.find(self.obj) for test in doctests: if test.examples: d[test.name] = DoctestItem(test, self) return d From ischenko at gmail.com Fri Jun 10 12:07:06 2005 From: ischenko at gmail.com (Max Ischenko) Date: Fri, 10 Jun 2005 13:07:06 +0300 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050609114837.GP21207@solar.trillke.net> Message-ID: <20050610100711.037E527B5E@code1.codespeak.net> Holger, > > My situation requires a custom representation of a failure instead of plain > > traceback. > > > > A solution might be to abstract the way failures are reported by moving this > > responsibility from TerminalSession to the Failure class, for example. After > > all, this even seems logical - different kind of failures may require > > different representations and a Failure class is a natural place to hold > > this information. OTOH, the session classes *are* responsible for an actual > > failure's representation: be it console output, graphical window or network > > socket. > > > > A method to the Failure class will be added that will return a suitable > > string representation of a failure. Failure class itself will provide > > default implementation that will yield formatted traceback. This way I > > could subclass Failure that will yield doctest output instead of traceback. > > > > What would you say? > > your analysis is pretty correct. However, my idea was to just let > py.test.skip(someobj) internally take care that str(someobj) provides > the failure message representation. Doing that by an internal > subclass is fine but an implementation detail that should not get > exposed (Failure classes are not visible directly). Ah, now I got you, thanks for patience. Yes, the trick with skip() works fine but, IMO, is no more then a temp hack. Errors in doctest-based tests are failures and should be reported as such. I can try to work out a patch by implementing solution I have described above. Alternatively, I'm willing to discuss other approaches you or anyone else has at hand. I'd really like to have doctest support to work smoothly and (if possible) be incorporated as part of py.test library. Here is my conftest.py verbatim, in case anyone interested. import os.path import py import sys from cStringIO import StringIO import doctest class Directory(py.test.collect.Directory): def buildname2items(self): d = super(Directory, self).buildname2items() for fn in self.fspath.listdir('*.py'): if fn.basename in d or fn.basename == 'conftest.py': continue module = None try: module = fn.pyimport() except ImportError, e: continue finder = doctest.DocTestFinder() doctests = finder.find(fn.pyimport()) doctests = [t for t in doctests if t.examples] if doctests: module = DoctestModule(fn, parent=self) d[fn.basename] = module return d class DoctestModule(py.test.collect.Module): def funcnamefilter(self, name): return True def classnamefilter(self, name): return True def buildname2items(self): d = {} finder = doctest.DocTestFinder() doctests = finder.find(self.obj) for test in doctests: if test.examples: d[test.name] = DoctestItem(test, self) return d class DoctestItem(py.test.Item): def __init__(self, test, parent): py.test.Item.__init__(self, test.name, parent) self.test = test def run(self): runner = doctest.DocTestRunner() out = StringIO() writer = lambda msg: out.write(msg) failed, tried = runner.run(self.test, out=writer) if failed: py.test.skip(out.getvalue()) #raise self.Failed() From hpk at trillke.net Fri Jun 10 13:22:00 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 10 Jun 2005 13:22:00 +0200 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050610100711.037E527B5E@code1.codespeak.net> References: <20050609114837.GP21207@solar.trillke.net> <20050610100711.037E527B5E@code1.codespeak.net> Message-ID: <20050610112200.GB21207@solar.trillke.net> Hi Max, hi David, hi all, On Fri, Jun 10, 2005 at 13:07 +0300, Max Ischenko wrote: > Holger, > > > My situation requires a custom representation of a failure instead of > plain > > > traceback. > > > > > > A solution might be to abstract the way failures are reported by moving > this > > > responsibility from TerminalSession to the Failure class, for example. > After > > > all, this even seems logical - different kind of failures may require > > > different representations and a Failure class is a natural place to hold > > > this information. OTOH, the session classes *are* responsible for an > actual > > > failure's representation: be it console output, graphical window or > network > > > socket. > > > > > > A method to the Failure class will be added that will return a suitable > > > string representation of a failure. Failure class itself will provide > > > default implementation that will yield formatted traceback. This way I > > > could subclass Failure that will yield doctest output instead of > traceback. > > > > > > What would you say? > > > > your analysis is pretty correct. However, my idea was to just let > > py.test.skip(someobj) internally take care that str(someobj) provides > > the failure message representation. Doing that by an internal > > subclass is fine but an implementation detail that should not get > > exposed (Failure classes are not visible directly). > > Ah, now I got you, thanks for patience. > > Yes, the trick with skip() works fine but, IMO, is no more then a temp hack. > Errors in doctest-based tests are failures and should be reported as such. Oh sure, i am sorry. Of course, i should have talked about py.test.fail() all along! skip() doesn't make any sense here. I also think that your idea of just copying and maybe slightly modifying the 2.4 doctest module into the py lib makes sense. The general idea is that py.test and the py lib should work sanely on python 2.2, 2.3, 2.4 and 2.5/cvs. If we integrate doctest support natively, then i now think that we should aim for offering simple conftest.py configuration items like: test_files = ['test_foo.py', 'test_bar.py'] extra_doctest_files = ['whatever.py', ...] and test files should also be searched for doctests, btw. There would then be no need to write custom Collectors/Items for changing the order or the list of to-be-examined doctest modules. The above example file lists would only be associated with the directory containing the conftest.py file and not any subdirectories. On the implementation side, the given files should always be imported via the path.pyimport() method which imports modules with their fully qualified name. (http://codespeak.net/py/current/doc/test.html#package-name) David, are you interested in taking a stab at this? It's a bit involved, i am afraid, also because it involves changing some reporting functionality. If you or anyone else would like to do it, please make sure to write proper tests for any new features even though it's sometimes a bit involved to do that. I sometimes spend half the time for writing tests for new py lib features but it really pays off. cheers, holger From dstanek at dstanek.com Fri Jun 10 14:54:44 2005 From: dstanek at dstanek.com (David Stanek) Date: Fri, 10 Jun 2005 08:54:44 -0400 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050610112200.GB21207@solar.trillke.net> References: <20050609114837.GP21207@solar.trillke.net> <20050610100711.037E527B5E@code1.codespeak.net> <20050610112200.GB21207@solar.trillke.net> Message-ID: <20050610125444.GA8094@goliath.hsd1.oh.comcast.net> On Fri, Jun 10, 2005 at 01:22:00PM +0200, holger krekel wrote: > Hi Max, hi David, hi all, > > Oh sure, i am sorry. Of course, i should have talked about > py.test.fail() all along! skip() doesn't make any sense here. > > I also think that your idea of just copying and maybe slightly modifying > the 2.4 doctest module into the py lib makes sense. The general idea is > that py.test and the py lib should work sanely on python 2.2, 2.3, 2.4 > and 2.5/cvs. > > If we integrate doctest support natively, then i now think that we > should aim for offering simple conftest.py configuration items like: > > test_files = ['test_foo.py', 'test_bar.py'] > extra_doctest_files = ['whatever.py', ...] > > and test files should also be searched for doctests, btw. There would > then be no need to write custom Collectors/Items for changing the order > or the list of to-be-examined doctest modules. The above example > file lists would only be associated with the directory containing > the conftest.py file and not any subdirectories. On the implementation > side, the given files should always be imported via the path.pyimport() > method which imports modules with their fully qualified name. > (http://codespeak.net/py/current/doc/test.html#package-name) > > David, are you interested in taking a stab at this? It's a bit Sure. > involved, i am afraid, also because it involves changing some reporting > functionality. If you or anyone else would like to do it, please make > sure to write proper tests for any new features even though it's sometimes > a bit involved to do that. I sometimes spend half the time for writing > tests for new py lib features but it really pays off. Holger, After we talked last night I was thinking about making the change to both py.test.skip() and py.test.fail(). I agree with Max that this is a hack and probably not the best way to do it. So I started to work in a slightly different direction. I was thinking a more general approach would be to factor out the view logic into it's own class heirarchy. Here is what I was thinking: * added an additional param to fail() and skip(), defaulted to None * factored out the py.test.TerminalSession's repr_* functions into py.test.view.(BaseOutcomeView & DefaultOutcomeView) classes Then the conftest.py can select a custom view class and attach it to the outcome. -- David Stanek www.roninds.net GPG keyID #6272EDAF on http://pgp.mit.edu Key fingerprint = 8BAA 7E11 8856 E148 6833 655A 92E2 3E00 6272 EDAF -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From dstanek at dstanek.com Fri Jun 10 15:01:21 2005 From: dstanek at dstanek.com (David Stanek) Date: Fri, 10 Jun 2005 09:01:21 -0400 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050610100709.E442827B5E@code1.codespeak.net> References: <35083.192.168.1.1.1118309594.squirrel@192.168.1.1> <20050610100709.E442827B5E@code1.codespeak.net> Message-ID: <20050610130121.GB8094@goliath.hsd1.oh.comcast.net> On Fri, Jun 10, 2005 at 01:07:06PM +0300, Max Ischenko wrote: > David, > > > I made a couple of changes to your conftest.py. The original one did not > > run under my version of Python. The classes being used were added in 2.4. > > Please see if that works for you. Later today I'll investigate Holger's > > suggestion. > > Besides DoctestFinder (that you have replaced with Tester) you need a way to > capture doctest output as well. DocTest in 2.4 provides nice report_* > methods plus an optional 'writer' argument to aid this task. Have no idea > how it'd translate for pre-2.4 version. > > Python doctest module has been improved considerably for 2.4 release and I > think the simplest and reasonable way to make it work for version < 2.4 is > to use standalone doctest24 module (can be found on the web). I did not think of including a doctest module, but that definitely is a good idea. I just posted some ideas on how to separate the view from the session module. I was gonna use this concept to provide a view of the test failure (instead of the worthless traceback) using the report_* functionality. If that did not exists, as in 2.2 and 2.3 I was going to fall back to just printing the stderr from doctest. I was also going to provide a DoctestCollection class that encapulated the use of DoctestFinder or Tester, it would have just picked the best one. However, I'd like to follow through on putting in a 2.4 doctest. That way all users will have the same capablilites/experience. I just had to modify the conftest.py so that I could get it to run in my environment :) -- David Stanek www.roninds.net GPG keyID #6272EDAF on http://pgp.mit.edu Key fingerprint = 8BAA 7E11 8856 E148 6833 655A 92E2 3E00 6272 EDAF -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From hpk at trillke.net Fri Jun 10 15:52:17 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 10 Jun 2005 15:52:17 +0200 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050610125444.GA8094@goliath.hsd1.oh.comcast.net> References: <20050609114837.GP21207@solar.trillke.net> <20050610100711.037E527B5E@code1.codespeak.net> <20050610112200.GB21207@solar.trillke.net> <20050610125444.GA8094@goliath.hsd1.oh.comcast.net> Message-ID: <20050610135217.GE21207@solar.trillke.net> Hi David, hi all, On Fri, Jun 10, 2005 at 08:54 -0400, David Stanek wrote: > On Fri, Jun 10, 2005 at 01:22:00PM +0200, holger krekel wrote: > > Hi Max, hi David, hi all, > > > > Oh sure, i am sorry. Of course, i should have talked about > > py.test.fail() all along! skip() doesn't make any sense here. > > > > I also think that your idea of just copying and maybe slightly modifying > > the 2.4 doctest module into the py lib makes sense. The general idea is > > that py.test and the py lib should work sanely on python 2.2, 2.3, 2.4 > > and 2.5/cvs. > > > > If we integrate doctest support natively, then i now think that we > > should aim for offering simple conftest.py configuration items like: > > > > test_files = ['test_foo.py', 'test_bar.py'] > > extra_doctest_files = ['whatever.py', ...] > > > > and test files should also be searched for doctests, btw. There would > > then be no need to write custom Collectors/Items for changing the order > > or the list of to-be-examined doctest modules. > Holger, > After we talked last night I was thinking about making the change to > both py.test.skip() and py.test.fail(). I agree with Max that this > is a hack and probably not the best way to do it. It really depends on the purpose. For some purposes like the custom doctest-link checking support, having py.test.fail() cause the sesssion to display brief information without a traceback seems to make perfect sense. For doctests we need more support than just this, though. > So I started to work in a slightly different direction. I > was thinking a more general approach would be to factor out > the view logic into it's own class heirarchy. Just so you know: when i hear the word "class hierarchy" i usually get afraid :-) > Here is what I was thinking: > * added an additional param to fail() and skip(), defaulted to None > * factored out the py.test.TerminalSession's repr_* functions into > py.test.view.(BaseOutcomeView & DefaultOutcomeView) classes If you mean that py.test should grow a new 'view' sub-namespace then i am not convinced yet. I do think that separating out representation/view logic some more from the sessions makes sense, though. > Then the conftest.py can select a custom view class and attach it to > the outcome. Where exactly is the need for that? For adding proper doctest support we don't need to expose new names. py.test should just internally make sure it does nicely collect, run and report doctests. David, to allow you and others to just head off into the direction you are imagining, just do a svn cp http://codespeak.net/svn/py/dist \ http://codespeak.net/svn/py/branch/dist-doctest and work in that branch. I have a feeling that you could otherwise not work incrementally and get early feedback (until py is released the 'dist' branch should remain pretty stable). Tested Code sometimes says more than words on mailing lists :-) cheers, holger From dstanek at dstanek.com Fri Jun 10 16:50:29 2005 From: dstanek at dstanek.com (David Stanek) Date: Fri, 10 Jun 2005 10:50:29 -0400 (EDT) Subject: [py-dev] running doctest-based tests In-Reply-To: <20050610135217.GE21207@solar.trillke.net> References: <20050609114837.GP21207@solar.trillke.net> <20050610100711.037E527B5E@code1.codespeak.net> <20050610112200.GB21207@solar.trillke.net> <20050610125444.GA8094@goliath.hsd1.oh.comcast.net> <20050610135217.GE21207@solar.trillke.net> Message-ID: <13662.207.58.192.150.1118415029.squirrel@207.58.192.150> On Fri, Jun 10, 2005 at 03:52:17PM +0200, holger krekel wrote: > Hi David, hi all, > > On Fri, Jun 10, 2005 at 08:54 -0400, David Stanek wrote: > > Holger, > > After we talked last night I was thinking about making the change to > > both py.test.skip() and py.test.fail(). I agree with Max that this > > is a hack and probably not the best way to do it. > > It really depends on the purpose. For some purposes like the > custom doctest-link checking support, having py.test.fail() > cause the sesssion to display brief information without > a traceback seems to make perfect sense. For doctests > we need more support than just this, though. I thought about different ways to do this, but couldn't come up with something that I really liked. The simplest way would be to add an additional parameter to py.test.skip() and py.test.fail(), that would simply set a parameter in the Outcome class. The Outcome.showtb(?) would then be evaluated before printing the traceback. > > So I started to work in a slightly different direction. I > > was thinking a more general approach would be to factor out > > the view logic into it's own class heirarchy. > > Just so you know: when i hear the word "class hierarchy" i usually > get afraid :-) I do know the feeling. In this case it's really a single class, with one parent. Not much of a heirarchy. > > Here is what I was thinking: > > * added an additional param to fail() and skip(), defaulted to None > > * factored out the py.test.TerminalSession's repr_* functions into > > py.test.view.(BaseOutcomeView & DefaultOutcomeView) classes > > If you mean that py.test should grow a new 'view' sub-namespace > then i am not convinced yet. I do think that separating out > representation/view logic some more from the sessions makes > sense, though. Actually the way I started to implement it was without the view namespace. I actually put the *OutcomeView classes into a view.py but expose them as py.test.*OutcomeView. Not sure if that is better or not. When I wrote the example I think I just munged the two ideas together in my head :-) > > Then the conftest.py can select a custom view class and attach it to > > the outcome. > > Where exactly is the need for that? For adding proper doctest > support we don't need to expose new names. py.test should > just internally make sure it does nicely collect, run and > report doctests. There probably is not too much of a need for this. When thinking about this solution I kept thinking about allowing the codetest.py to affect the output. Max may not have had a problem if something like this already existed. For instance, he would have been able to very easily override the default behavour of printing a traceback by providing a new view. > > David, to allow you and others to just head off into the > direction you are imagining, just do a > > svn cp http://codespeak.net/svn/py/dist \ > http://codespeak.net/svn/py/branch/dist-doctest > > and work in that branch. I have a feeling that you > could otherwise not work incrementally and get early > feedback (until py is released the 'dist' branch should > remain pretty stable). Tested Code sometimes says > more than words on mailing lists :-) Will do. Thanks. I don't want to work in a direction that is too far off of the vision. So let me know when you think I am crazy. David From grig at gheorghiu.net Sat Jun 11 01:09:32 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Fri, 10 Jun 2005 16:09:32 -0700 (PDT) Subject: [py-dev] Logging in the py library In-Reply-To: <20050609100506.GO21207@solar.trillke.net> Message-ID: <20050610230932.68432.qmail@web54510.mail.yahoo.com> I just svn commit-ed 2 files: misc/log.py and misc/testing/test_log.py. I tried to keep things as simple as possible and as close to the spirit of py.trace as possible, while using the logging module as the back-end. Please consider this experimental code. I'll be working on it throughout the weekend. Comments/suggestions are very welcome, vital actually :-) Grig --- holger krekel wrote: > Hi Grig, hi all, > > On Tue, Jun 07, 2005 at 18:51 +0200, holger krekel wrote: > > On Tue, Jun 07, 2005 at 09:46 -0700, Grig Gheorghiu wrote: > > > Holger, > > > > > > I think it makes a lot of sense to split the functionality > between a > > > very simple 'producer' API that is exposed to end-users and a > > > potentially more complicated 'consumer' API that interfaces with > the > > > logging module. It's pretty clear you spent time thinking about > all > > > these things :-), so I think it would probably be best if you > came up > > > with a first cut at the 'producer' API and at how things tie > together. > > > Then I'd be glad to work on the 'back-end' API that interfaces > with the > > > logging module. Would that work for you? > > > > Hehe, i guess so. Let me bet - for the fun of it - that the > > producer+consumer API implementation roughly described in the > > last mail will not exceed 70 lines of code :-) > > Ok, done that. It's at 54 lines of code but i feel it's already > too convoluted. You'll find the code in py/misc/trace.py and > py/misc/testing/test_trace.py shows some usages. > > The main missing piece involves thinking more about the dispatching > (possibly based on partial keyword matches) and the actual > API / namespace organization, for example of where to offer > logging-module adapter functions. > > Please consider this trace.py experimental. But it hopefully > gives a more concrete clue of the ideas. Feedback or suggestions > very welcome (i think especially by using it a bit here and there > we could gain insights/better approaches). > > cheers, > > holger > From ischenko at gmail.com Sat Jun 11 13:02:19 2005 From: ischenko at gmail.com (Max Ischenko) Date: Sat, 11 Jun 2005 14:02:19 +0300 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050610112200.GB21207@solar.trillke.net> Message-ID: <20050611110220.A2ECE27B45@code1.codespeak.net> Holger, > I also think that your idea of just copying and maybe slightly modifying > the 2.4 doctest module into the py lib makes sense. The general idea is > that py.test and the py lib should work sanely on python 2.2, 2.3, 2.4 > and 2.5/cvs. Well, I didn't mean py lib specifically, just a casual user who wants to use a doctest module. But including it in a py lib may be a sane idea. Module is in the public domain by the way. > If we integrate doctest support natively, then i now think that we > should aim for offering simple conftest.py configuration items like: If we integrate doctest support natively do we need conftest.py in the first place? Currently you need no conftest.py to collect and run test code. Why don't integrate doctests in the same way? Guess a performance hit may be an issue but then a switch can be added that will disable (or enable) doctests support. Just an idea. > test_files = ['test_foo.py', 'test_bar.py'] > extra_doctest_files = ['whatever.py', ...] > > and test files should also be searched for doctests, btw. There would > then be no need to write custom Collectors/Items for changing the order > or the list of to-be-examined doctest modules. The above example > file lists would only be associated with the directory containing > the conftest.py file and not any subdirectories. On the implementation > side, the given files should always be imported via the path.pyimport() > method which imports modules with their fully qualified name. > (http://codespeak.net/py/current/doc/test.html#package-name) The situation is slightly complicated by the fact that there are two options to specify doctests (inline or in a separate file) which will require different treatment by the library. At a bare minimum, I'd be happy if py.test just provided this boring code (DoctestItem, DoctestModule, etc) that is needed to collect and execute tests. If this is available, I can create short and sweet conftest.py to build correct configuration for my exact situation. At the higher level, py.test may support some syntax sugar in conftest.py (like the one you described) that will aid writing this configuration file. Going even further, py.test may be able to collect and run doctests directly, without the need for user to write any arcane configuration files. Of course, for a typical scenario, just like it does with normal tests. If user has custom requirements and non-standard file hierarchy s/he will need to write a conftest.py. IOW, I'd like that "native doctests support" will mean that doctests become first-class citizens, just like normal tests are. That's my opinion anyway. For a start I just want doctest failures be reported properly. ;) > David, are you interested in taking a stab at this? It's a bit > involved, i am afraid, also because it involves changing some reporting > functionality. If you or anyone else would like to do it, please make > sure to write proper tests for any new features even though it's sometimes > a bit involved to do that. I sometimes spend half the time for writing > tests for new py lib features but it really pays off. Guess I should wait a bit and see what David will produce. David, if you need my help - just ask. From hpk at trillke.net Sat Jun 11 14:03:25 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 11 Jun 2005 14:03:25 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050610230932.68432.qmail@web54510.mail.yahoo.com> References: <20050609100506.GO21207@solar.trillke.net> <20050610230932.68432.qmail@web54510.mail.yahoo.com> Message-ID: <20050611120325.GK21207@solar.trillke.net> Hey Grig, On Fri, Jun 10, 2005 at 16:09 -0700, Grig Gheorghiu wrote: > I just svn commit-ed 2 files: misc/log.py and misc/testing/test_log.py. > I tried to keep things as simple as possible and as close to the spirit > of py.trace as possible, while using the logging module as the > back-end. Please consider this experimental code. I'll be working on it > throughout the weekend. > > Comments/suggestions are very welcome, vital actually :-) first of all, thanks for helping! It looks nice so far yet i am giving you detailed feedback and criticism as requested. And please, you don't need to excuse yourself for anything. This is a technical discussion :-) - we should merge the py.trace and py.log namespaces e.g. by saying that all non-underscore and/or all lowercase names are keywords and the rest are functions/classes, e.g. py.log.debug("hello world") # as current py.trace.* py.log.set_logger('debug', py.log.FileLogger(path, append=True)) and maybe forget about the [...] and getitem-trick alltogether. Keyword arguments provide nice flexibility for future features which __getitem__ can't. Note that 'set_logger' and 'FileLogger' would not be regarded as a keyword because they contain underscores and are not all lowercase. We may want to consider to keep the FileLogger() and similar names very close to the logging module. But please don't expose the Logging module's classes directly but just delegate to them underlyingly. - it seems you prefer the 'py.log' naming. I am fine with that. - make sure that all your files have the 'svn:eol-style' property set to 'native' with e.g. svn propset 'svn:eol-style' native FILE1 FILE2" - it should always be easy for an application to completely take over logging (as the current py.trace allows) and not care about custom loggers or even hierachies. So something like def myfunction(message): ... py.log.set_logger("", myfunction) should be enough to have all messages routed through my function. - for writing tests, please don't just create random files here and there. Use something like: dir = py.test.ensuretemp("logtest") p = dir.join('py_stdout') and make sure that such paths can always be passed to py.log functions and classes, e.g. by str()ing path-arguments them early. - please keep the basic py.trace functionality (renamed to py.log to your likening) self-contained and put the extra stuff (like FileLogger etc.pp) into a log_support.py file. I'd really like to use the basic mechanism from py.execnet very early on. - i reserve the right to tweak the final exact names because i really want the py lib to have a tight concise namespace organization. So for example. py.log.File(...) might be preferable to the duplication in py.log.FileLogger(...) from my anal naming point of view :-) cheers, holger From dstanek at dstanek.com Sat Jun 11 15:02:33 2005 From: dstanek at dstanek.com (David Stanek) Date: Sat, 11 Jun 2005 09:02:33 -0400 Subject: [py-dev] running doctest-based tests In-Reply-To: <20050611110220.A2ECE27B45@code1.codespeak.net> References: <20050610112200.GB21207@solar.trillke.net> <20050611110220.A2ECE27B45@code1.codespeak.net> Message-ID: <20050611130233.GA7814@goliath.hsd1.oh.comcast.net> On Sat, Jun 11, 2005 at 02:02:19PM +0300, Max Ischenko wrote: > Holger, > > > > I also think that your idea of just copying and maybe slightly modifying > > the 2.4 doctest module into the py lib makes sense. The general idea is > > that py.test and the py lib should work sanely on python 2.2, 2.3, 2.4 > > and 2.5/cvs. > > Well, I didn't mean py lib specifically, just a casual user who wants to use > a doctest module. But including it in a py lib may be a sane idea. Module is > in the public domain by the way. > > > If we integrate doctest support natively, then i now think that we > > should aim for offering simple conftest.py configuration items like: > > If we integrate doctest support natively do we need conftest.py in the first > place? > > Currently you need no conftest.py to collect and run test code. > > Why don't integrate doctests in the same way? Guess a performance hit may be > an issue but then a switch can be added that will disable (or enable) > doctests support. > > Just an idea. > > > test_files = ['test_foo.py', 'test_bar.py'] > > extra_doctest_files = ['whatever.py', ...] > > > > and test files should also be searched for doctests, btw. There would > > then be no need to write custom Collectors/Items for changing the order > > or the list of to-be-examined doctest modules. The above example > > file lists would only be associated with the directory containing > > the conftest.py file and not any subdirectories. On the implementation > > side, the given files should always be imported via the path.pyimport() > > method which imports modules with their fully qualified name. > > (http://codespeak.net/py/current/doc/test.html#package-name) > > The situation is slightly complicated by the fact that there are two options > to specify doctests (inline or in a separate file) which will require > different treatment by the library. > > At a bare minimum, I'd be happy if py.test just provided this boring code > (DoctestItem, DoctestModule, etc) that is needed to collect and execute > tests. If this is available, I can create short and sweet conftest.py to > build correct configuration for my exact situation. > > At the higher level, py.test may support some syntax sugar in conftest.py > (like the one you described) that will aid writing this configuration file. > > Going even further, py.test may be able to collect and run doctests > directly, without the need for user to write any arcane configuration files. > Of course, for a typical scenario, just like it does with normal tests. If > user has custom requirements and non-standard file hierarchy s/he will need > to write a conftest.py. > > IOW, I'd like that "native doctests support" will mean that doctests become > first-class citizens, just like normal tests are. > > That's my opinion anyway. For a start I just want doctest failures be > reported properly. ;) > > > David, are you interested in taking a stab at this? It's a bit > > involved, i am afraid, also because it involves changing some reporting > > functionality. If you or anyone else would like to do it, please make > > sure to write proper tests for any new features even though it's sometimes > > a bit involved to do that. I sometimes spend half the time for writing > > tests for new py lib features but it really pays off. > > Guess I should wait a bit and see what David will produce. > > David, if you need my help - just ask. I am working integrating doctests into the library so that a conttest.py will not bee needed. I had talked to Holger a couple of days ago about adding a --doctests option to include collecting doctests and executing them. I have a bunch of patches that are a starting point. I will be putting all of the changes in their own branch, so that I will not affect the release schedule. Changes will start showing up here: http://codespeak.net/svn/py/branch/dist-doctest Max, Once I do start commiting them, I'll post a request for comments. I think I will take you up on your offer. Once I commit existing changes we can chat a little about direction. Do you use IRC, AIM, or some other type of live medium? -- David Stanek www.roninds.net GPG keyID #6272EDAF on http://pgp.mit.edu Key fingerprint = 8BAA 7E11 8856 E148 6833 655A 92E2 3E00 6272 EDAF -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From grig at gheorghiu.net Sat Jun 11 17:54:04 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Sat, 11 Jun 2005 08:54:04 -0700 (PDT) Subject: [py-dev] Logging in the py library In-Reply-To: <20050611120325.GK21207@solar.trillke.net> Message-ID: <20050611155404.15852.qmail@web54507.mail.yahoo.com> Hi, Holger --- holger krekel wrote: > Hey Grig, > > On Fri, Jun 10, 2005 at 16:09 -0700, Grig Gheorghiu wrote: > > I just svn commit-ed 2 files: misc/log.py and > misc/testing/test_log.py. > > I tried to keep things as simple as possible and as close to the > spirit > > of py.trace as possible, while using the logging module as the > > back-end. Please consider this experimental code. I'll be working > on it > > throughout the weekend. > > > > Comments/suggestions are very welcome, vital actually :-) > > first of all, thanks for helping! It looks nice so far > yet i am giving you detailed feedback and criticism > as requested. And please, you don't need to excuse > yourself for anything. This is a technical discussion :-) > > - we should merge the py.trace and py.log namespaces > e.g. by saying that all non-underscore and/or all lowercase > names are keywords and the rest are functions/classes, e.g. > > py.log.debug("hello world") # as current py.trace.* > py.log.set_logger('debug', py.log.FileLogger(path, append=True)) > > and maybe forget about the [...] and getitem-trick alltogether. > Keyword arguments provide nice flexibility for future features > which __getitem__ can't. > > Note that 'set_logger' and 'FileLogger' would not be regarded as > a keyword because they contain underscores and are not all > lowercase. > > We may want to consider to keep the FileLogger() and similar > names very close to the logging module. But please don't > expose the Logging module's classes directly but just > delegate to them underlyingly. > > - it seems you prefer the 'py.log' naming. I am fine with that. I didn't intend to replace py.trace with py.log. I was thinking that there would be 2 orthogonal name spaces: py.trace for specifying the messages/keywords to be logged, and py.log for choosing the loggers/filters/etc. One could specify a py.log logger at run-time via a configuration file, whereas the py.trace stuff would be pretty much self-contained and 'set in stone'. But I have no problem with merging the 2 namespaces. I'll give that a shot. > > - make sure that all your files have the 'svn:eol-style' > property set to 'native' with e.g. > > svn propset 'svn:eol-style' native FILE1 FILE2" > No problem. I wanted to do a 'dos2unix' on the files before I checked them in, but I forgot... > - it should always be easy for an application to completely > take over logging (as the current py.trace allows) and not > care about custom loggers or even hierachies. So something > like > > def myfunction(message): > ... > py.log.set_logger("", myfunction) > > should be enough to have all messages routed through > my function. > > - for writing tests, please don't just create random > files here and there. Use something like: > > dir = py.test.ensuretemp("logtest") > p = dir.join('py_stdout') > > and make sure that such paths can always be passed to > py.log functions and classes, e.g. by str()ing path-arguments > them early. Got it. > > - please keep the basic py.trace functionality (renamed to > py.log to your likening) self-contained and put the extra > stuff (like FileLogger etc.pp) into a log_support.py file. > I'd really like to use the basic mechanism from py.execnet > very early on. OK, will do, if you want to merge py.trace and py.log. > > - i reserve the right to tweak the final exact names > because i really want the py lib to have a tight concise > namespace organization. So for example. > > py.log.File(...) might be preferable to the duplication in > py.log.FileLogger(...) > > from my anal naming point of view :-) No problem. You're the keeper of the keys :-) Thanks for the comments/suggestions. As you see, I have much to learn, and working on the py lib provides the perfect opportunity. Grig From hpk at trillke.net Sat Jun 11 18:14:27 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 11 Jun 2005 18:14:27 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050611155404.15852.qmail@web54507.mail.yahoo.com> References: <20050611120325.GK21207@solar.trillke.net> <20050611155404.15852.qmail@web54507.mail.yahoo.com> Message-ID: <20050611161427.GN21207@solar.trillke.net> Hi Grig, On Sat, Jun 11, 2005 at 08:54 -0700, Grig Gheorghiu wrote: > --- holger krekel wrote: > > - we should merge the py.trace and py.log namespaces > > e.g. by saying that all non-underscore and/or all lowercase > > names are keywords and the rest are functions/classes, e.g. > > > > py.log.debug("hello world") # as current py.trace.* > > py.log.set_logger('debug', py.log.FileLogger(path, append=True)) > > > > and maybe forget about the [...] and getitem-trick alltogether. > > Keyword arguments provide nice flexibility for future features > > which __getitem__ can't. > > ... > > - it seems you prefer the 'py.log' naming. I am fine with that. > > I didn't intend to replace py.trace with py.log. I was thinking that > there would be 2 orthogonal name spaces: py.trace for specifying the > messages/keywords to be logged, and py.log for choosing the > loggers/filters/etc. I think that having two root-level namespaces is too much. Let's merge the two, especially since the py/__init__.py mechanism allows to keep things in separate files nevertheless. Hum, coming to think of it you may need to tweak the __init__.py file definitions for the time being to something like: 'py.log.debug': ('./misc/log.py', 'debug'), 'py.log.warn': ('./misc/log.py', 'warn'), 'py.log.set_logger': ('./misc/log.py', 'set_logger'), 'py.log.File': ('./misc/log_support.py', 'File'), I know that this means that py.log (py.trace currently) would loose the ability to dynamically add keywords but that may actually be a good thing (e.g. to prevent typos) and it's only true at "root-level". I.e. one can still do: debug = py.log.debug.execnet and actually, if we need it, we can just add a clean helper like: py.log.Tracer("debug execnet gateway") which would return a py.log Tracer with exactly the given keywords etc.pp. Btw, we might want to speak about "Tracers" and "Loggers" to distinguish the producing from the consuming part. If you don't have some better terminology in mind. Did i mention that the other half of my programming time that i don't spend with testing, i spent with shuffling names? :-) cheers, holger From grig at gheorghiu.net Sat Jun 11 18:27:48 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Sat, 11 Jun 2005 09:27:48 -0700 (PDT) Subject: [py-dev] Logging in the py library In-Reply-To: <20050611161427.GN21207@solar.trillke.net> Message-ID: <20050611162748.79512.qmail@web54502.mail.yahoo.com> Hi, Holger --- holger krekel wrote: > Hi Grig, > > On Sat, Jun 11, 2005 at 08:54 -0700, Grig Gheorghiu wrote: > > --- holger krekel wrote: > > > - we should merge the py.trace and py.log namespaces > > > e.g. by saying that all non-underscore and/or all lowercase > > > names are keywords and the rest are functions/classes, e.g. > > > > > > py.log.debug("hello world") # as current py.trace.* > > > py.log.set_logger('debug', py.log.FileLogger(path, > append=True)) > > > > > > and maybe forget about the [...] and getitem-trick alltogether. > > > > Keyword arguments provide nice flexibility for future features > > > which __getitem__ can't. > > > ... > > > - it seems you prefer the 'py.log' naming. I am fine with that. > > > > I didn't intend to replace py.trace with py.log. I was thinking > that > > there would be 2 orthogonal name spaces: py.trace for specifying > the > > messages/keywords to be logged, and py.log for choosing the > > loggers/filters/etc. > > I think that having two root-level namespaces is too much. > Let's merge the two, especially since the py/__init__.py > mechanism allows to keep things in separate files nevertheless. > > Hum, coming to think of it you may need to tweak the __init__.py > file definitions for the time being to something like: > > 'py.log.debug': ('./misc/log.py', 'debug'), > 'py.log.warn': ('./misc/log.py', 'warn'), > > 'py.log.set_logger': ('./misc/log.py', 'set_logger'), > 'py.log.File': ('./misc/log_support.py', 'File'), > > I know that this means that py.log (py.trace currently) would > loose the ability to dynamically add keywords but that may > actually be a good thing (e.g. to prevent typos) and it's > only true at "root-level". I.e. one can still do: > > debug = py.log.debug.execnet > > and actually, if we need it, we can just add a clean helper like: > > py.log.Tracer("debug execnet gateway") > > which would return a py.log Tracer with exactly the given > keywords etc.pp. Btw, we might want to speak about "Tracers" > and "Loggers" to distinguish the producing from the consuming > part. If you don't have some better terminology in mind. > Did i mention that the other half of my programming time that > i don't spend with testing, i spent with shuffling names? :-) > OK, I'll try to implement as many of the suggestions above as possible, but it may very well be the case that I'm not grasping everything you aim to accomplish. So please feel free to jump in and change stuff around. My goal is to work on this over the weekend and check something in early Monday. Grig From hpk at trillke.net Sat Jun 11 18:37:50 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 11 Jun 2005 18:37:50 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050611162748.79512.qmail@web54502.mail.yahoo.com> References: <20050611161427.GN21207@solar.trillke.net> <20050611162748.79512.qmail@web54502.mail.yahoo.com> Message-ID: <20050611163750.GO21207@solar.trillke.net> Hi Grig, On Sat, Jun 11, 2005 at 09:27 -0700, Grig Gheorghiu wrote: > > ... > > OK, I'll try to implement as many of the suggestions above as possible, > but it may very well be the case that I'm not grasping everything you > aim to accomplish. So please feel free to jump in and change stuff > around. My goal is to work on this over the weekend and check something > in early Monday. no problem. Also feel free to checkin things in between. Otherwise i won't jump in. Nobody is relying on any of this so far (just make sure that the tests don't break). cheers & have fun, holger From hpk at trillke.net Sat Jun 11 22:08:37 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 11 Jun 2005 22:08:37 +0200 Subject: [py-dev] new py lib website Message-ID: <20050611200837.GP21207@solar.trillke.net> Hi py-dev, there is a update to the py lib's website: http://codespeak.net/py/current/doc/home.html You'll basically find the same information as before but it should be more structured. Additionally, there now is an issue tracker here: https://codespeak.net/issue/py-dev/ whre the 12 people who ever committed something to the py lib can all use their codespeak usernames+logins. The issue tracker will be the main driving point for working on the long awaited first py release, 0.8. Help is welcome. If you svn-commit a change relating to an issue in the tracker you can consider to put at the top of the commit message: issueN in-progress here goes your normal commit log message this will connect your checkin to the issue tracker. the 'N' in issueN is the issue-number you'll find in the tracker. If you issue 'py.test' in the py/documentation directory then you basically get a mirror of the original web page (modulo the tracker which is only living at codespeak). This allows to test out modifications to documentation or even to layout on your local checkout. So feel free to fix typos or add documentation to your likening. have fun, holger From jan at balster.info Sat Jun 11 22:54:05 2005 From: jan at balster.info (Jan Balster) Date: Sat, 11 Jun 2005 22:54:05 +0200 Subject: [py-dev] new py lib website In-Reply-To: <20050611200837.GP21207@solar.trillke.net> References: <20050611200837.GP21207@solar.trillke.net> Message-ID: <42AB4F6D.3070305@balster.info> holger krekel wrote: > Hi py-dev, > > there is a update to the py lib's website: > > http://codespeak.net/py/current/doc/home.html > > You'll basically find the same information as before > but it should be more structured. Additionally, there > now is an issue tracker here: > > https://codespeak.net/issue/py-dev/ > Hi Holger, Great news! Can you add a "[py-dev]" (or something like that) prefix for notification emails from roundup? When I click on "Your Issues" I get an error message "exceptions.KeyError: 'milestone'". It's an error in "issue.index.html". "Show My issues" works as expected. Jan Debugging information follows 1. While evaluating the standard:'request/batch' expression on line 13 Current variables: templates repeat false 0 context utils db nothing None i18n true 1 default request tracker template config options {'ok_message': [], 'error_message': []} loop attrs {'tal:define': 'batch request/batch', 'tal:condition': 'context/is_view_ok'} 2. A problem occurred in your template "issue.index.html". From hpk at trillke.net Sat Jun 11 22:58:13 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 11 Jun 2005 22:58:13 +0200 Subject: [py-dev] new py lib website In-Reply-To: <42AB4F6D.3070305@balster.info> References: <20050611200837.GP21207@solar.trillke.net> <42AB4F6D.3070305@balster.info> Message-ID: <20050611205813.GQ21207@solar.trillke.net> Hi Jan, On Sat, Jun 11, 2005 at 22:54 +0200, Jan Balster wrote: > holger krekel wrote: > > Hi py-dev, > > > > there is a update to the py lib's website: > > > > http://codespeak.net/py/current/doc/home.html > > > > You'll basically find the same information as before > > but it should be more structured. Additionally, there > > now is an issue tracker here: > > > > https://codespeak.net/issue/py-dev/ > > > > Great news! > > Can you add a "[py-dev]" (or something like that) prefix for notification emails > from roundup? unfortunately, that is not easily possible because roundup interprets the name inside the [] when you mail in things (you can mail-reply to an issue and it will show up). > When I click on "Your Issues" I get an error message > "exceptions.KeyError: 'milestone'". It's an error in "issue.index.html". > "Show My issues" works as expected. that should be fixed now. thanks for the feedback! holger From jan at balster.info Sat Jun 11 23:17:36 2005 From: jan at balster.info (Jan Balster) Date: Sat, 11 Jun 2005 23:17:36 +0200 Subject: [py-dev] new py lib website In-Reply-To: <20050611205813.GQ21207@solar.trillke.net> References: <20050611200837.GP21207@solar.trillke.net> <42AB4F6D.3070305@balster.info> <20050611205813.GQ21207@solar.trillke.net> Message-ID: <42AB54F0.10602@balster.info> holger krekel wrote: > Hi Jan, > > On Sat, Jun 11, 2005 at 22:54 +0200, Jan Balster wrote: > >>holger krekel wrote: >> >>>Hi py-dev, >>> >>>there is a update to the py lib's website: >>> >>> http://codespeak.net/py/current/doc/home.html >>> >>>You'll basically find the same information as before >>>but it should be more structured. Additionally, there >>>now is an issue tracker here: >>> >>> https://codespeak.net/issue/py-dev/ >>> >> >>Great news! >> >>Can you add a "[py-dev]" (or something like that) prefix for notification emails >>from roundup? > > > unfortunately, that is not easily possible because roundup interprets > the name inside the [] when you mail in things (you can mail-reply to > an issue and it will show up). Ok. (reading description of the e-mail gateway). Now I understand why roundup is the new py-dev issue tracker! > >>When I click on "Your Issues" I get an error message >>"exceptions.KeyError: 'milestone'". It's an error in "issue.index.html". >>"Show My issues" works as expected. > > > that should be fixed now. Yes, it is. Works as expected. Jan From grig at gheorghiu.net Sun Jun 12 00:23:03 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Sat, 11 Jun 2005 15:23:03 -0700 (PDT) Subject: [py-dev] Logging in the py library In-Reply-To: <20050611161427.GN21207@solar.trillke.net> Message-ID: <20050611222303.68460.qmail@web54506.mail.yahoo.com> --- holger krekel wrote: > Hi Grig, > > On Sat, Jun 11, 2005 at 08:54 -0700, Grig Gheorghiu wrote: > > --- holger krekel wrote: > > > - we should merge the py.trace and py.log namespaces > > > e.g. by saying that all non-underscore and/or all lowercase > > > names are keywords and the rest are functions/classes, e.g. > > > > > > py.log.debug("hello world") # as current py.trace.* > > > py.log.set_logger('debug', py.log.FileLogger(path, > append=True)) > > > > > > and maybe forget about the [...] and getitem-trick alltogether. > > > > Keyword arguments provide nice flexibility for future features > > > which __getitem__ can't. > > > ... > > > - it seems you prefer the 'py.log' naming. I am fine with that. > > > > I didn't intend to replace py.trace with py.log. I was thinking > that > > there would be 2 orthogonal name spaces: py.trace for specifying > the > > messages/keywords to be logged, and py.log for choosing the > > loggers/filters/etc. > > I think that having two root-level namespaces is too much. > Let's merge the two, especially since the py/__init__.py > mechanism allows to keep things in separate files nevertheless. > > Hum, coming to think of it you may need to tweak the __init__.py > file definitions for the time being to something like: > > 'py.log.debug': ('./misc/log.py', 'debug'), > 'py.log.warn': ('./misc/log.py', 'warn'), > > 'py.log.set_logger': ('./misc/log.py', 'set_logger'), > 'py.log.File': ('./misc/log_support.py', 'File'), > > I know that this means that py.log (py.trace currently) would > loose the ability to dynamically add keywords but that may > actually be a good thing (e.g. to prevent typos) and it's > only true at "root-level". I.e. one can still do: > > debug = py.log.debug.execnet > > and actually, if we need it, we can just add a clean helper like: > > py.log.Tracer("debug execnet gateway") > > which would return a py.log Tracer with exactly the given > keywords etc.pp. Btw, we might want to speak about "Tracers" > and "Loggers" to distinguish the producing from the consuming > part. If you don't have some better terminology in mind. > Did i mention that the other half of my programming time that > i don't spend with testing, i spent with shuffling names? :-) Hi, Holger I've been trying to unify the log and trace namespaces, but I'm kind of lost. When you have 'py.log.debug': ('./misc/log.py', 'debug'), in __init__.py, that means there needs to be an object called 'debug' in misc/log.py. How do you associate that object with a method of the class Tracer? I looked in other files, but all the objects I've seen that are exposed in __init__.py are either classes or instances of classes. Also, do you want to have both a Tracer class and a Logger class in log.py? It becomes kind of confusing if you're associating py.log.debug with a debug method of the class Tracer. Generally speaking, the distinction between Tracer and Logger will be kind of hard to grasp for somebody new to the py library -- at least that's my opinion... If you want to change the code and refactor the trace.py module by renaming it log.py and properly exposing the debug, warn, etc. objects in __init__.py, that would help me in understanding more accurately what you have in mind. Grig From hpk at trillke.net Sun Jun 12 01:10:08 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 12 Jun 2005 01:10:08 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050611222303.68460.qmail@web54506.mail.yahoo.com> References: <20050611161427.GN21207@solar.trillke.net> <20050611222303.68460.qmail@web54506.mail.yahoo.com> Message-ID: <20050611231007.GR21207@solar.trillke.net> On Sat, Jun 11, 2005 at 15:23 -0700, Grig Gheorghiu wrote: > --- holger krekel wrote: > > On Sat, Jun 11, 2005 at 08:54 -0700, Grig Gheorghiu wrote: > > > --- holger krekel wrote: > > Hum, coming to think of it you may need to tweak the __init__.py > > file definitions for the time being to something like: > > > > 'py.log.debug': ('./misc/log.py', 'debug'), > > 'py.log.warn': ('./misc/log.py', 'warn'), > > > > 'py.log.set_logger': ('./misc/log.py', 'set_logger'), > > 'py.log.File': ('./misc/log_support.py', 'File'), > > > I've been trying to unify the log and trace namespaces, but I'm kind of > lost. When you have > > 'py.log.debug': ('./misc/log.py', 'debug'), > > in __init__.py, that means there needs to be an object called 'debug' > in misc/log.py. How do you associate that object with a method of the > class Tracer? I looked in other files, but all the objects I've seen > that are exposed in __init__.py are either classes or instances of > classes. Sorry, if __init__.py's logic is a bit opaque. Basically you could do e.g. in trace.py at the end tracer = Tracer() debug = tracer.debug warn = tracer.warn ... and then 'py.trace.debug': ('./misc/trace.py', 'debug'), should work. The rule is simply that the 'debug' name is picked from the module which needs to provide a name-object binding. There is no restriction on what type of object that could be. > Also, do you want to have both a Tracer class and a Logger class in > log.py? It becomes kind of confusing if you're associating py.log.debug > with a debug method of the class Tracer. Generally speaking, the > distinction between Tracer and Logger will be kind of hard to grasp for > somebody new to the py library -- at least that's my opinion... You are probably right. Maybe indeed use your suggested names py.log.Consumer and py.log.Producer? So py.log.debug would be a Producer with the keyword 'debug'. > If you want to change the code and refactor the trace.py module by > renaming it log.py and properly exposing the debug, warn, etc. objects > in __init__.py, that would help me in understanding more accurately > what you have in mind. I can do that if you get stuck again (and don't consider this your fault, btw). cheers, holger From hpk at trillke.net Sun Jun 12 01:17:13 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 12 Jun 2005 01:17:13 +0200 Subject: [py-dev] Logging in the py library In-Reply-To: <20050611231007.GR21207@solar.trillke.net> References: <20050611161427.GN21207@solar.trillke.net> <20050611222303.68460.qmail@web54506.mail.yahoo.com> <20050611231007.GR21207@solar.trillke.net> Message-ID: <20050611231713.GT21207@solar.trillke.net> Hi Grig, On Sun, Jun 12, 2005 at 01:10 +0200, holger krekel wrote: > and then > > 'py.trace.debug': ('./misc/trace.py', 'debug'), one little note. You cannot mix different levels of namespace definition. 'py.log': ..., and 'py.log.something': ... i.e. one definition is not allowed to "shadow" another one. cheers, holger From ischenko at gmail.com Sun Jun 12 08:23:11 2005 From: ischenko at gmail.com (Max Ischenko) Date: Sun, 12 Jun 2005 09:23:11 +0300 Subject: [py-dev] new py lib website In-Reply-To: <20050611200837.GP21207@solar.trillke.net> Message-ID: <20050612062313.59F1427B5E@code1.codespeak.net> Hi all, > there is a update to the py lib's website: > > http://codespeak.net/py/current/doc/home.html > > You'll basically find the same information as before > but it should be more structured. Additionally, there > now is an issue tracker here: > > https://codespeak.net/issue/py-dev/ > > whre the 12 people who ever committed something to the py lib > can all use their codespeak usernames+logins. The issue > tracker will be the main driving point for working > on the long awaited first py release, 0.8. > Help is welcome. This is great news! On the site you wrote: "The main missing feature is proper testing and integration with win32 platforms and improved documentation." What kind of kind of problems do you have with win32? I'm currently working on win32 (XP) and may be able to help. From ischenko at gmail.com Sun Jun 12 08:26:56 2005 From: ischenko at gmail.com (Max Ischenko) Date: Sun, 12 Jun 2005 09:26:56 +0300 Subject: [py-dev] win32 issues In-Reply-To: <20050612062313.59F1427B5E@code1.codespeak.net> Message-ID: <20050612062656.02D2027B5E@code1.codespeak.net> Me wrote: > On the site you wrote: "The main missing feature is proper testing and > integration with win32 platforms and improved documentation." > > What kind of kind of problems do you have with win32? > > I'm currently working on win32 (XP) and may be able to help. Guess you meant the issues list on a web site. For a one, py.countloc is missing "from _findpy import py" line. From grig at gheorghiu.net Sun Jun 12 17:53:54 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Sun, 12 Jun 2005 08:53:54 -0700 (PDT) Subject: [py-dev] win32 issues In-Reply-To: <20050612062656.02D2027B5E@code1.codespeak.net> Message-ID: <20050612155354.98401.qmail@web54505.mail.yahoo.com> --- Max Ischenko wrote: > Me wrote: > > On the site you wrote: "The main missing feature is proper testing > and > > integration with win32 platforms and improved documentation." > > > > What kind of kind of problems do you have with win32? > > > > I'm currently working on win32 (XP) and may be able to help. > One thing that doesn't work for me is the full functionality of the tkinter GUI (when you run py.test --tkinter). I can't see any tree of tests in the GUI panes. The last time I ran py.test in the top-level py directory on a Win XP box, no test failed. But on Win2k Pro a lot of them fail. I'll run them again and I'll post my findings in the open issue. Grig From jan at balster.info Sun Jun 12 18:20:29 2005 From: jan at balster.info (Jan Balster) Date: Sun, 12 Jun 2005 18:20:29 +0200 Subject: [py-dev] win32 issues In-Reply-To: <20050612155354.98401.qmail@web54505.mail.yahoo.com> References: <20050612155354.98401.qmail@web54505.mail.yahoo.com> Message-ID: <42AC60CD.5010400@balster.info> Grig Gheorghiu wrote: > --- Max Ischenko wrote: > > >>Me wrote: >> >>>On the site you wrote: "The main missing feature is proper testing >> >>and >> >>>integration with win32 platforms and improved documentation." >>> >>>What kind of kind of problems do you have with win32? >>> >>>I'm currently working on win32 (XP) and may be able to help. >> > > One thing that doesn't work for me is the full functionality of the > tkinter GUI (when you run py.test --tkinter). I can't see any tree of > tests in the GUI panes. Hi Grig, because of a problem with Tix on some platforms I disabled the tree view. Do you see the list of failed/skipped/passed tests? Jan From grig at gheorghiu.net Mon Jun 13 04:16:25 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Sun, 12 Jun 2005 19:16:25 -0700 (PDT) Subject: [py-dev] Logging in the py library Message-ID: <20050613021626.53891.qmail@web54506.mail.yahoo.com> --- Grig Gheorghiu wrote: > --- holger krekel wrote: > > > Hi Grig, > > > > On Sun, Jun 12, 2005 at 01:10 +0200, holger krekel wrote: > > > and then > > > > > > 'py.trace.debug': ('./misc/trace.py', 'debug'), > > > > one little note. You cannot mix different levels of > > namespace definition. > > > > 'py.log': ..., > > and > > 'py.log.something': ... > > > > i.e. one definition is not allowed to "shadow" another one. > > > > Hi, Holger > > I just committed my changes to svn. Please take a look and see if > what > I have is a bit closer to what you had in mind. I fervently hope it > is > :-) > > Grig > Holger, Also, I haven't done the keyword implementation yet (all lowercase, no underscores). I'm waiting to see what you think of the code so far and then I'll add more stuff, such as other loggers and many more tests. Grig From grig at gheorghiu.net Mon Jun 13 05:24:25 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Sun, 12 Jun 2005 20:24:25 -0700 (PDT) Subject: [py-dev] win32 issues In-Reply-To: <42AC60CD.5010400@balster.info> Message-ID: <20050613032426.70609.qmail@web54505.mail.yahoo.com> --- Jan Balster wrote: > Grig Gheorghiu wrote: > > --- Max Ischenko wrote: > > > > > >>Me wrote: > >> > >>>On the site you wrote: "The main missing feature is proper testing > >> > >>and > >> > >>>integration with win32 platforms and improved documentation." > >>> > >>>What kind of kind of problems do you have with win32? > >>> > >>>I'm currently working on win32 (XP) and may be able to help. > >> > > > > One thing that doesn't work for me is the full functionality of the > > tkinter GUI (when you run py.test --tkinter). I can't see any tree > of > > tests in the GUI panes. > > Hi Grig, > because of a problem with Tix on some platforms I disabled the tree > view. > Do you see the list of failed/skipped/passed tests? > > Jan Hi, Jan Yes, I see the list of failed/skipped tests, as well as the count of Failed/Skipped/Passed. Grig From jan at balster.info Mon Jun 13 06:59:41 2005 From: jan at balster.info (Jan Balster) Date: Mon, 13 Jun 2005 06:59:41 +0200 Subject: [py-dev] win32 issues In-Reply-To: <20050613032426.70609.qmail@web54505.mail.yahoo.com> References: <20050613032426.70609.qmail@web54505.mail.yahoo.com> Message-ID: <42AD12BD.8020705@balster.info> Grig Gheorghiu wrote: > --- Jan Balster wrote: > > >>Grig Gheorghiu wrote: >> >>>--- Max Ischenko wrote: >>> >>> >>> >>>>Me wrote: >>>> >>>> >>>>>On the site you wrote: "The main missing feature is proper testing >>>> >>>>and >>>> >>>> >>>>>integration with win32 platforms and improved documentation." >>>>> >>>>>What kind of kind of problems do you have with win32? >>>>> >>>>>I'm currently working on win32 (XP) and may be able to help. >>>> >>>One thing that doesn't work for me is the full functionality of the >>>tkinter GUI (when you run py.test --tkinter). I can't see any tree >> >>of >> >>>tests in the GUI panes. >> >>Hi Grig, >>because of a problem with Tix on some platforms I disabled the tree >>view. >>Do you see the list of failed/skipped/passed tests? >> >>Jan > > > Hi, Jan > > Yes, I see the list of failed/skipped tests, as well as the count of > Failed/Skipped/Passed. > > Grig > Fine. You can enable/disable the failed/skipped/passed tests in the list by clicking on the count buttons :-) You should be able to see the passed tests as well. Jan From florian.schulze at gmx.net Mon Jun 13 12:53:50 2005 From: florian.schulze at gmx.net (Florian Schulze) Date: Mon, 13 Jun 2005 12:53:50 +0200 Subject: [py-dev] win32 issues In-Reply-To: <20050612155354.98401.qmail@web54505.mail.yahoo.com> References: <20050612155354.98401.qmail@web54505.mail.yahoo.com> Message-ID: On Sun, 12 Jun 2005 08:53:54 -0700 (PDT), Grig Gheorghiu wrote: > > --- Max Ischenko wrote: > >> Me wrote: >> > On the site you wrote: "The main missing feature is proper testing >> and >> > integration with win32 platforms and improved documentation." >> > >> > What kind of kind of problems do you have with win32? >> > >> > I'm currently working on win32 (XP) and may be able to help. >> > > One thing that doesn't work for me is the full functionality of the > tkinter GUI (when you run py.test --tkinter). I can't see any tree of > tests in the GUI panes. > > The last time I ran py.test in the top-level py directory on a Win XP > box, no test failed. But on Win2k Pro a lot of them fail. I'll run them > again and I'll post my findings in the open issue. > > Grig Also svn doesn't work when a locale different than en (english) is used. Either pysvn should be used when available (fallback to svn commands like now if not), or the original svn wrappers. Also executable searching in the environment path and registry isn't done yet, would be nice to have. Regards, Florian Schulze From hpk at trillke.net Mon Jun 13 12:56:35 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 13 Jun 2005 12:56:35 +0200 Subject: [py-dev] win32 issues In-Reply-To: References: <20050612155354.98401.qmail@web54505.mail.yahoo.com> Message-ID: <20050613105635.GF21207@solar.trillke.net> On Mon, Jun 13, 2005 at 12:53 +0200, Florian Schulze wrote: > On Sun, 12 Jun 2005 08:53:54 -0700 (PDT), Grig Gheorghiu > wrote: > >--- Max Ischenko wrote: > > > >>Me wrote: > >>> On the site you wrote: "The main missing feature is proper testing > >>and > >>> integration with win32 platforms and improved documentation." > >>> > >>> What kind of kind of problems do you have with win32? > >>> > >>> I'm currently working on win32 (XP) and may be able to help. > >> > > > >One thing that doesn't work for me is the full functionality of the > >tkinter GUI (when you run py.test --tkinter). I can't see any tree of > >tests in the GUI panes. > > > >The last time I ran py.test in the top-level py directory on a Win XP > >box, no test failed. But on Win2k Pro a lot of them fail. I'll run them > >again and I'll post my findings in the open issue. > > > >Grig > > Also svn doesn't work when a locale different than en (english) is used. > Either pysvn should be used when available (fallback to svn commands like > now if not), or the original svn wrappers. > Also executable searching in the environment path and registry isn't done > yet, would be nice to have. you mean py.path.local.sysfind() on win32? do you care to file an issue (possibly after registering with the tracker first)? Actually two issues, the locale problem is another one. For Unix there used to be a hack which is currently disabled i think. For Win32 i am not sure about the best way to use a specific locale for executing an svn command. cheers, holger From grig at gheorghiu.net Mon Jun 13 15:24:10 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 13 Jun 2005 06:24:10 -0700 (PDT) Subject: [py-dev] win32 issues In-Reply-To: Message-ID: <20050613132410.46747.qmail@web54509.mail.yahoo.com> --- Florian Schulze wrote: > Also executable searching in the environment path and registry isn't > done > yet, would be nice to have. If you refer to the py.test command-line tool, then if you run setup.py on win32, it will get added to the PATH via the registry. Grig From grig at gheorghiu.net Mon Jun 13 15:58:46 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 13 Jun 2005 06:58:46 -0700 (PDT) Subject: [py-dev] Latest py.test results on WinXP Message-ID: <20050613135846.32708.qmail@web54503.mail.yahoo.com> I just did an svn update and ran py.test in the top-level py directory on a WinXP box. Here's the output: C:\py\py>py.test inserting into sys.path: C:\py ============================= test process starts ============================= testing-mode: inprocess executable: C:\Python24\python.exe (2.4.1-final-0) using py lib: C:\py\py builtin\testing\test_enumerate.py[1] . code\testing\test_code.py[4] .... code\testing\test_cpython_features.py[1] . code\testing\test_excinfo.py[8] ........ code\testing\test_frame.py[2] .. code\testing\test_source.py[25] ......................... documentation\example\pytest\test_failures.py[1] . documentation\example\pytest\test_setup_flow_example.py[2] .. documentation\test_conftest.py[1] F documentation\TODO.txt[2] s. documentation\api.txt[3] s.. documentation\coding-style.txt[3] s.. documentation\contact.txt[2] s. documentation\execnet.txt[3] s.. documentation\future.txt[7] s...... documentation\getting-started.txt[7] s...... documentation\greenlet.txt[2] s. documentation\home.txt[2] s. documentation\index.txt[11] s.......... documentation\misc.txt[5] s.... documentation\releasescheme.txt[3] s.. documentation\test.txt[3] s.. documentation\why_py.txt[8] s....... documentation\xml.txt[4] s... execnet\testing\test_gateway.py[50] ................................ssssssssssss ssssss execnet\testing\test_pickle.py[2] .. magic\testing\test_assertion.py[7] ....... magic\testing\test_autopath.py[5] ..... magic\testing\test_exprinfo.py[7] ....... magic\testing\test_invoke.py[2] .. magic\testing\test_patch.py[3] ... magic\testing\test_viewtype.py[2] .. misc\testing\test_api.py[3] ... misc\testing\test_cache.py[11] ........... misc\testing\test_error.py[1] . misc\testing\test_initpkg.py[160] ...........................F.................. .F.F............................................................................ .................................s misc\testing\test_log.py[7] ....... misc\testing\test_simplecapture.py[5] ..... misc\testing\test_std.py[3] ... path\extpy\testing\test_extpy.py[52] ........................................... ......... path\local\popen5\testing\test_subprocess.py[0] path\local\testing\test_local.py[78] ........................................... ................................... path\local\testing\test_posix.py[0] path\svn\testing\test_urlcommand.py[73] ssssssssssssssssssssssssssssssssssssssss sssssssssssssssssssssssssssssssss path\svn\testing\test_wccommand.py[85] sssssssssssssssssssssssssssssssssssssssss ssssssssssssssssssssssssssssssssssssssssssss path\testing\test_api.py[4] .... process\testing\test_cmdexec.py[4] .... test\testing\import_test\package\test_import.py[1] . test\testing\test_api.py[2] .. test\testing\test_collect.py[9] ......... test\testing\test_compat.py[18] .................. test\testing\test_config.py[2] .. test\testing\test_raises.py[4] .... test\testing\test_session.py[20] .................... test\testing\test_setup_nested.py[10] .......... test\tkinter\testing\test_backend.py[18] .................. test\tkinter\testing\test_capture_out_err.py[1] . test\tkinter\testing\test_reportsession.py[2] .. test\tkinter\testing\test_repository.py[20] .................... test\tkinter\testing\test_util.py[17] ................. thread\testing\test_io.py[4] .... thread\testing\test_pool.py[7] ....... tool\testing\test_utestconvert.py[1] . xmlobj\testing\test_html.py[4] .... xmlobj\testing\test_xml.py[6] ...... __________________________ reasons for skipped tests __________________________ Skipped in C:\py\py\c-extension\conftest.py:13 reason: Skipped: c-extension testing needs platform selection Skipped in C:\py\py\path\svn\testing\svntestbase.py:23 reason: Skipped: could not create temporary svn test repository Skipped in C:\py\py\misc\testing\test_initpkg.py:79 reason: Skipped: cannot test svnversion, 'svnversion' binary not found Skipped in C:\py\py\execnet\testing\test_gateway.py:286 reason: Skipped: no known ssh target, use -S to set one Skipped in C:\py\py\documentation\conftest.py:17 reason: Skipped: docutils not importable _______________________________________________________________________________ def test_doctest_basic(): # XXX get rid of the next line: py.magic.autopath().dirpath('conftest.py').copy(tmpdir.join('conftest.py ')) tmpdir.ensure('__init__.py') xtxt = tmpdir.join('x.txt') xtxt.write(py.code.Source(""" .. >>> from os.path import abspath hello world >>> assert abspath >>> i=3 >>> print i 3 yes yes >>> i 3 end """)) config, args = py.test.Config.parse([str(xtxt)]) session = config.getsessionclass()(config, py.std.sys.stdout) session.main([xtxt]) l = session.getitemoutcomepairs(py.test.Item.Failed) assert len(l) == 0 l = session.getitemoutcomepairs(py.test.Item.Passed) E assert len(l) == 2 > assert 1 == 2 + where 1 = len([(, )]) [C:\py\py\documentation\test_conftest.py:37] [modulepath: test_doctest_basic] - - - - - - - - - - test_doctest_basic: recorded stdout - - - - - - - - - - ============================= test process starts ============================= testing-mode: inprocess executable: C:\Python24\python.exe (2.4.1-final-0) using py lib: C:\py\py None[2] s. __________________________ reasons for skipped tests __________________________ Skipped in c:\docume~1\ggheorg\locals~1\temp\pytest-140\docdoctest\conftest.py:1 7 reason: Skipped: docutils not importable ============= tests finished: 1 passed, 1 skipped in 0.08 seconds ============= _______________________________________________________________________________ def check_import(modpath): print "checking import", modpath E assert __import__(modpath) > ImportError: No module named compat.doctest [C:\py\py\misc\testing\test_initpkg.py:65] [modulepath: test_importall[8]] - - - - - - - - - - - - - - [8]: recorded stdout - - - - - - - - - - - - - - checking import py.__.compat.doctest _______________________________________________________________________________ def check_import(modpath): print "checking import", modpath E assert __import__(modpath) > ImportError: No module named compat.optparse [C:\py\py\misc\testing\test_initpkg.py:65] [modulepath: test_importall[9]] - - - - - - - - - - - - - - [9]: recorded stdout - - - - - - - - - - - - - - checking import py.__.compat.optparse _______________________________________________________________________________ def check_import(modpath): print "checking import", modpath E assert __import__(modpath) > ImportError: No module named compat.textwrap [C:\py\py\misc\testing\test_initpkg.py:65] [modulepath: test_importall[10]] - - - - - - - - - - - - - - [10]: recorded stdout - - - - - - - - - - - - - - checking import py.__.compat.textwrap _______________________________________________________________________________ ===== tests finished: 624 passed, 4 failed, 193 skipped in 39.92 seconds ===== From hpk at trillke.net Mon Jun 13 16:28:23 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 13 Jun 2005 16:28:23 +0200 Subject: [py-dev] Latest py.test results on WinXP In-Reply-To: <20050613135846.32708.qmail@web54503.mail.yahoo.com> References: <20050613135846.32708.qmail@web54503.mail.yahoo.com> Message-ID: <20050613142823.GS21207@solar.trillke.net> On Mon, Jun 13, 2005 at 06:58 -0700, Grig Gheorghiu wrote: > I just did an svn update and ran py.test in the top-level py directory > on a WinXP box. Here's the output: you don't happen to have switched to the branch in-between? I see py/compat path parts which should only be present in David's dist-doctest branch ... holger > C:\py\py>py.test > inserting into sys.path: C:\py > ============================= test process starts > ============================= > testing-mode: inprocess > executable: C:\Python24\python.exe (2.4.1-final-0) > using py lib: C:\py\py > > builtin\testing\test_enumerate.py[1] . > code\testing\test_code.py[4] .... > code\testing\test_cpython_features.py[1] . > code\testing\test_excinfo.py[8] ........ > code\testing\test_frame.py[2] .. > code\testing\test_source.py[25] ......................... > documentation\example\pytest\test_failures.py[1] . > documentation\example\pytest\test_setup_flow_example.py[2] .. > documentation\test_conftest.py[1] F > documentation\TODO.txt[2] s. > documentation\api.txt[3] s.. > documentation\coding-style.txt[3] s.. > documentation\contact.txt[2] s. > documentation\execnet.txt[3] s.. > documentation\future.txt[7] s...... > documentation\getting-started.txt[7] s...... > documentation\greenlet.txt[2] s. > documentation\home.txt[2] s. > documentation\index.txt[11] s.......... > documentation\misc.txt[5] s.... > documentation\releasescheme.txt[3] s.. > documentation\test.txt[3] s.. > documentation\why_py.txt[8] s....... > documentation\xml.txt[4] s... > execnet\testing\test_gateway.py[50] > ................................ssssssssssss > ssssss > execnet\testing\test_pickle.py[2] .. > magic\testing\test_assertion.py[7] ....... > magic\testing\test_autopath.py[5] ..... > magic\testing\test_exprinfo.py[7] ....... > magic\testing\test_invoke.py[2] .. > magic\testing\test_patch.py[3] ... > magic\testing\test_viewtype.py[2] .. > misc\testing\test_api.py[3] ... > misc\testing\test_cache.py[11] ........... > misc\testing\test_error.py[1] . > misc\testing\test_initpkg.py[160] > ...........................F.................. > .F.F............................................................................ > .................................s > misc\testing\test_log.py[7] ....... > misc\testing\test_simplecapture.py[5] ..... > misc\testing\test_std.py[3] ... > path\extpy\testing\test_extpy.py[52] > ........................................... > ......... > path\local\popen5\testing\test_subprocess.py[0] > path\local\testing\test_local.py[78] > ........................................... > ................................... > path\local\testing\test_posix.py[0] > path\svn\testing\test_urlcommand.py[73] > ssssssssssssssssssssssssssssssssssssssss > sssssssssssssssssssssssssssssssss > path\svn\testing\test_wccommand.py[85] > sssssssssssssssssssssssssssssssssssssssss > ssssssssssssssssssssssssssssssssssssssssssss > path\testing\test_api.py[4] .... > process\testing\test_cmdexec.py[4] .... > test\testing\import_test\package\test_import.py[1] . > test\testing\test_api.py[2] .. > test\testing\test_collect.py[9] ......... > test\testing\test_compat.py[18] .................. > test\testing\test_config.py[2] .. > test\testing\test_raises.py[4] .... > test\testing\test_session.py[20] .................... > test\testing\test_setup_nested.py[10] .......... > test\tkinter\testing\test_backend.py[18] .................. > test\tkinter\testing\test_capture_out_err.py[1] . > test\tkinter\testing\test_reportsession.py[2] .. > test\tkinter\testing\test_repository.py[20] .................... > test\tkinter\testing\test_util.py[17] ................. > thread\testing\test_io.py[4] .... > thread\testing\test_pool.py[7] ....... > tool\testing\test_utestconvert.py[1] . > xmlobj\testing\test_html.py[4] .... > xmlobj\testing\test_xml.py[6] ...... > > > __________________________ reasons for skipped tests > __________________________ > Skipped in C:\py\py\c-extension\conftest.py:13 > reason: Skipped: c-extension testing needs platform selection > > Skipped in C:\py\py\path\svn\testing\svntestbase.py:23 > reason: Skipped: could not create temporary svn test repository > > Skipped in C:\py\py\misc\testing\test_initpkg.py:79 > reason: Skipped: cannot test svnversion, 'svnversion' binary not found > > Skipped in C:\py\py\execnet\testing\test_gateway.py:286 > reason: Skipped: no known ssh target, use -S to set one > > Skipped in C:\py\py\documentation\conftest.py:17 > reason: Skipped: docutils not importable > > _______________________________________________________________________________ > > def test_doctest_basic(): > # XXX get rid of the next line: > > py.magic.autopath().dirpath('conftest.py').copy(tmpdir.join('conftest.py > ')) > tmpdir.ensure('__init__.py') > > xtxt = tmpdir.join('x.txt') > xtxt.write(py.code.Source(""" > .. > >>> from os.path import abspath > > hello world > > >>> assert abspath > >>> i=3 > >>> print i > 3 > > yes yes > > >>> i > 3 > > end > """)) > config, args = py.test.Config.parse([str(xtxt)]) > session = config.getsessionclass()(config, py.std.sys.stdout) > session.main([xtxt]) > l = session.getitemoutcomepairs(py.test.Item.Failed) > assert len(l) == 0 > l = session.getitemoutcomepairs(py.test.Item.Passed) > E assert len(l) == 2 > > assert 1 == 2 > + where 1 = len([(, instance>)]) > > [C:\py\py\documentation\test_conftest.py:37] > [modulepath: test_doctest_basic] > - - - - - - - - - - test_doctest_basic: recorded stdout - - - - - - - > - - - > ============================= test process starts > ============================= > testing-mode: inprocess > executable: C:\Python24\python.exe (2.4.1-final-0) > using py lib: C:\py\py > > None[2] s. > > > __________________________ reasons for skipped tests > __________________________ > Skipped in > c:\docume~1\ggheorg\locals~1\temp\pytest-140\docdoctest\conftest.py:1 > 7 > reason: Skipped: docutils not importable > > ============= tests finished: 1 passed, 1 skipped in 0.08 seconds > ============= > > _______________________________________________________________________________ > > def check_import(modpath): > print "checking import", modpath > E assert __import__(modpath) > > ImportError: No module named compat.doctest > > [C:\py\py\misc\testing\test_initpkg.py:65] > [modulepath: test_importall[8]] > - - - - - - - - - - - - - - [8]: recorded stdout - - - - - - - - - - - > - - - > checking import py.__.compat.doctest > > _______________________________________________________________________________ > > def check_import(modpath): > print "checking import", modpath > E assert __import__(modpath) > > ImportError: No module named compat.optparse > > [C:\py\py\misc\testing\test_initpkg.py:65] > [modulepath: test_importall[9]] > - - - - - - - - - - - - - - [9]: recorded stdout - - - - - - - - - - - > - - - > checking import py.__.compat.optparse > > _______________________________________________________________________________ > > def check_import(modpath): > print "checking import", modpath > E assert __import__(modpath) > > ImportError: No module named compat.textwrap > > [C:\py\py\misc\testing\test_initpkg.py:65] > [modulepath: test_importall[10]] > - - - - - - - - - - - - - - [10]: recorded stdout - - - - - - - - - - > - - - - > checking import py.__.compat.textwrap > > _______________________________________________________________________________ > ===== tests finished: 624 passed, 4 failed, 193 skipped in 39.92 > seconds ===== > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From grig at gheorghiu.net Mon Jun 13 16:34:50 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 13 Jun 2005 07:34:50 -0700 (PDT) Subject: [py-dev] Latest py.test results on WinXP In-Reply-To: <20050613142823.GS21207@solar.trillke.net> Message-ID: <20050613143450.47117.qmail@web54503.mail.yahoo.com> --- holger krekel wrote: > On Mon, Jun 13, 2005 at 06:58 -0700, Grig Gheorghiu wrote: > > I just did an svn update and ran py.test in the top-level py > directory > > on a WinXP box. Here's the output: > > you don't happen to have switched to the branch in-between? > I see py/compat path parts which should only be present > in David's dist-doctest branch ... > > holger > Nope, I've updated the code from http://codespeak.net/svn/py/dist Grig > > > C:\py\py>py.test > > inserting into sys.path: C:\py > > ============================= test process starts > > ============================= > > testing-mode: inprocess > > executable: C:\Python24\python.exe (2.4.1-final-0) > > using py lib: C:\py\py > > > > builtin\testing\test_enumerate.py[1] . > > code\testing\test_code.py[4] .... > > code\testing\test_cpython_features.py[1] . > > code\testing\test_excinfo.py[8] ........ > > code\testing\test_frame.py[2] .. > > code\testing\test_source.py[25] ......................... > > documentation\example\pytest\test_failures.py[1] . > > documentation\example\pytest\test_setup_flow_example.py[2] .. > > documentation\test_conftest.py[1] F > > documentation\TODO.txt[2] s. > > documentation\api.txt[3] s.. > > documentation\coding-style.txt[3] s.. > > documentation\contact.txt[2] s. > > documentation\execnet.txt[3] s.. > > documentation\future.txt[7] s...... > > documentation\getting-started.txt[7] s...... > > documentation\greenlet.txt[2] s. > > documentation\home.txt[2] s. > > documentation\index.txt[11] s.......... > > documentation\misc.txt[5] s.... > > documentation\releasescheme.txt[3] s.. > > documentation\test.txt[3] s.. > > documentation\why_py.txt[8] s....... > > documentation\xml.txt[4] s... > > execnet\testing\test_gateway.py[50] > > ................................ssssssssssss > > ssssss > > execnet\testing\test_pickle.py[2] .. > > magic\testing\test_assertion.py[7] ....... > > magic\testing\test_autopath.py[5] ..... > > magic\testing\test_exprinfo.py[7] ....... > > magic\testing\test_invoke.py[2] .. > > magic\testing\test_patch.py[3] ... > > magic\testing\test_viewtype.py[2] .. > > misc\testing\test_api.py[3] ... > > misc\testing\test_cache.py[11] ........... > > misc\testing\test_error.py[1] . > > misc\testing\test_initpkg.py[160] > > ...........................F.................. > > > .F.F............................................................................ > > .................................s > > misc\testing\test_log.py[7] ....... > > misc\testing\test_simplecapture.py[5] ..... > > misc\testing\test_std.py[3] ... > > path\extpy\testing\test_extpy.py[52] > > ........................................... > > ......... > > path\local\popen5\testing\test_subprocess.py[0] > > path\local\testing\test_local.py[78] > > ........................................... > > ................................... > > path\local\testing\test_posix.py[0] > > path\svn\testing\test_urlcommand.py[73] > > ssssssssssssssssssssssssssssssssssssssss > > sssssssssssssssssssssssssssssssss > > path\svn\testing\test_wccommand.py[85] > > sssssssssssssssssssssssssssssssssssssssss > > ssssssssssssssssssssssssssssssssssssssssssss > > path\testing\test_api.py[4] .... > > process\testing\test_cmdexec.py[4] .... > > test\testing\import_test\package\test_import.py[1] . > > test\testing\test_api.py[2] .. > > test\testing\test_collect.py[9] ......... > > test\testing\test_compat.py[18] .................. > > test\testing\test_config.py[2] .. > > test\testing\test_raises.py[4] .... > > test\testing\test_session.py[20] .................... > > test\testing\test_setup_nested.py[10] .......... > > test\tkinter\testing\test_backend.py[18] .................. > > test\tkinter\testing\test_capture_out_err.py[1] . > > test\tkinter\testing\test_reportsession.py[2] .. > > test\tkinter\testing\test_repository.py[20] .................... > > test\tkinter\testing\test_util.py[17] ................. > > thread\testing\test_io.py[4] .... > > thread\testing\test_pool.py[7] ....... > > tool\testing\test_utestconvert.py[1] . > > xmlobj\testing\test_html.py[4] .... > > xmlobj\testing\test_xml.py[6] ...... > > > > > > __________________________ reasons for skipped tests > > __________________________ > > Skipped in C:\py\py\c-extension\conftest.py:13 > > reason: Skipped: c-extension testing needs platform selection > > > > Skipped in C:\py\py\path\svn\testing\svntestbase.py:23 > > reason: Skipped: could not create temporary svn test repository > > > > Skipped in C:\py\py\misc\testing\test_initpkg.py:79 > > reason: Skipped: cannot test svnversion, 'svnversion' binary not > found > > > > Skipped in C:\py\py\execnet\testing\test_gateway.py:286 > > reason: Skipped: no known ssh target, use -S to set one > > > > Skipped in C:\py\py\documentation\conftest.py:17 > > reason: Skipped: docutils not importable > > > > > _______________________________________________________________________________ > > > > def test_doctest_basic(): > > # XXX get rid of the next line: > > > > > py.magic.autopath().dirpath('conftest.py').copy(tmpdir.join('conftest.py > > ')) > > tmpdir.ensure('__init__.py') > > > > xtxt = tmpdir.join('x.txt') > > xtxt.write(py.code.Source(""" > > .. > > >>> from os.path import abspath > > > > hello world > > > > >>> assert abspath > > >>> i=3 > > >>> print i > > 3 > > > > yes yes > > > > >>> i > > 3 > > > > end > > """)) > > config, args = py.test.Config.parse([str(xtxt)]) > > session = config.getsessionclass()(config, > py.std.sys.stdout) > > session.main([xtxt]) > > l = session.getitemoutcomepairs(py.test.Item.Failed) > > assert len(l) == 0 > > l = session.getitemoutcomepairs(py.test.Item.Passed) > > E assert len(l) == 2 > > > assert 1 == 2 > > + where 1 = len([(, > instance>)]) > > > > [C:\py\py\documentation\test_conftest.py:37] > > [modulepath: test_doctest_basic] > > - - - - - - - - - - test_doctest_basic: recorded stdout - - - - - > - - > > - - - > > ============================= test process starts > > ============================= > > testing-mode: inprocess > > executable: C:\Python24\python.exe (2.4.1-final-0) > > using py lib: C:\py\py > > > > None[2] s. > > > > > > __________________________ reasons for skipped tests > > __________________________ > > Skipped in > > > c:\docume~1\ggheorg\locals~1\temp\pytest-140\docdoctest\conftest.py:1 > > 7 > > reason: Skipped: docutils not importable > > > > ============= tests finished: 1 passed, 1 skipped in 0.08 seconds > > ============= > > > > > _______________________________________________________________________________ > > > > def check_import(modpath): > > print "checking import", modpath > > E assert __import__(modpath) > > > ImportError: No module named compat.doctest > > > > [C:\py\py\misc\testing\test_initpkg.py:65] > > [modulepath: test_importall[8]] > > - - - - - - - - - - - - - - [8]: recorded stdout - - - - - - - - - > - - > > - - - > > checking import py.__.compat.doctest > > > > > _______________________________________________________________________________ > > > > def check_import(modpath): > > print "checking import", modpath > > E assert __import__(modpath) > > > ImportError: No module named compat.optparse > > > > [C:\py\py\misc\testing\test_initpkg.py:65] > > [modulepath: test_importall[9]] > > - - - - - - - - - - - - - - [9]: recorded stdout - - - - - - - - - > - - > > - - - > > checking import py.__.compat.optparse > > > > > _______________________________________________________________________________ > > > > def check_import(modpath): > > print "checking import", modpath > > E assert __import__(modpath) > > > ImportError: No module named compat.textwrap > > > > [C:\py\py\misc\testing\test_initpkg.py:65] > > [modulepath: test_importall[10]] > > - - - - - - - - - - - - - - [10]: recorded stdout - - - - - - - - > - - > > - - - - > > checking import py.__.compat.textwrap > > > > > _______________________________________________________________________________ > > ===== tests finished: 624 passed, 4 failed, 193 skipped in 39.92 > > seconds ===== > > > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > > > From grig at gheorghiu.net Mon Jun 13 16:43:29 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 13 Jun 2005 07:43:29 -0700 (PDT) Subject: [py-dev] Re: [py-svn] r13350 - py/dist/py/compat In-Reply-To: <20050613143743.C179927B7C@code1.codespeak.net> Message-ID: <20050613144329.51953.qmail@web54505.mail.yahoo.com> --- hpk at codespeak.net wrote: > Author: hpk > Date: Mon Jun 13 16:37:43 2005 > New Revision: 13350 > > Added: > py/dist/py/compat/__init__.py > Log: > add missing __init__.py file > > > Added: py/dist/py/compat/__init__.py > ============================================================================== > --- (empty file) > +++ py/dist/py/compat/__init__.py Mon Jun 13 16:37:43 2005 > @@ -0,0 +1 @@ > +# OK, this got rid of 3 out of the 4 failed tests. Now only documentation\test_conftest.py fails with: _______________________________________________________________________________ def test_doctest_basic(): # XXX get rid of the next line: py.magic.autopath().dirpath('conftest.py').copy(tmpdir.join('conftest.py ')) tmpdir.ensure('__init__.py') xtxt = tmpdir.join('x.txt') xtxt.write(py.code.Source(""" .. >>> from os.path import abspath hello world >>> assert abspath >>> i=3 >>> print i 3 yes yes >>> i 3 end """)) config, args = py.test.Config.parse([str(xtxt)]) session = config.getsessionclass()(config, py.std.sys.stdout) session.main([xtxt]) l = session.getitemoutcomepairs(py.test.Item.Failed) assert len(l) == 0 l = session.getitemoutcomepairs(py.test.Item.Passed) E assert len(l) == 2 > assert 1 == 2 + where 1 = len([(, )]) [C:\py\py\documentation\test_conftest.py:37] From dstanek at dstanek.com Mon Jun 13 20:29:38 2005 From: dstanek at dstanek.com (David Stanek) Date: Mon, 13 Jun 2005 14:29:38 -0400 (EDT) Subject: [py-dev] Latest py.test results on WinXP In-Reply-To: <20050613142823.GS21207@solar.trillke.net> References: <20050613135846.32708.qmail@web54503.mail.yahoo.com> <20050613142823.GS21207@solar.trillke.net> Message-ID: <28310.207.58.192.150.1118687378.squirrel@207.58.192.150> > On Mon, Jun 13, 2005 at 06:58 -0700, Grig Gheorghiu wrote: >> I just did an svn update and ran py.test in the top-level py directory >> on a WinXP box. Here's the output: > > you don't happen to have switched to the branch in-between? > I see py/compat path parts which should only be present > in David's dist-doctest branch ... > I actually added them to the trunk. I read up on issue 8 and saw that it is listed for release 0.8.0. With the __init__.py in place it doesn't look like the inclusion of the py/compat directory causes any issues. I can roll it out if need be. Sorry about not including __init__.py, I am not sure where my head was. David From grig at gheorghiu.net Mon Jun 13 20:31:57 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 13 Jun 2005 11:31:57 -0700 (PDT) Subject: [py-dev] Latest py.test results on WinXP In-Reply-To: <28310.207.58.192.150.1118687378.squirrel@207.58.192.150> Message-ID: <20050613183158.20511.qmail@web54502.mail.yahoo.com> --- David Stanek wrote: > > On Mon, Jun 13, 2005 at 06:58 -0700, Grig Gheorghiu wrote: > >> I just did an svn update and ran py.test in the top-level py > directory > >> on a WinXP box. Here's the output: > > > > you don't happen to have switched to the branch in-between? > > I see py/compat path parts which should only be present > > in David's dist-doctest branch ... > > > > I actually added them to the trunk. I read up on issue 8 and saw that > it > is listed for release 0.8.0. With the __init__.py in place it doesn't > look > like the inclusion of the py/compat directory causes any issues. I > can > roll it out if need be. Sorry about not including __init__.py, I am > not > sure where my head was. > Yes, with the addition of __init__py the tests pass now, so I don't think there are any problems with the compat directory at this point. Grig From hpk at trillke.net Tue Jun 14 12:26:39 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 14 Jun 2005 12:26:39 +0200 Subject: [py-dev] autonosying on the tracker Message-ID: <20050614102639.GA21207@solar.trillke.net> hi, i forgot to mention that there is a possibility to become "autonosy" on new issues in the tracker. This means that upon issue creation you get added into the nosy list automatically. Currently, i have only hard-coded this in a roundup detector plugin. There is no web interface for this feature so you have to mail me personally if you want that. cheers, holger From grig at gheorghiu.net Tue Jun 14 19:45:16 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 14 Jun 2005 10:45:16 -0700 (PDT) Subject: [py-dev] Adding logdebug option to defaultconftest.py Message-ID: <20050614174517.62319.qmail@web54510.mail.yahoo.com> I'm thinking about adding a new option to the defaultconftest.py file, something like: Option('', '--logdebug', action="store_true", dest="logdebug", default=False, help="set logging level to 'debug' (default is 'info'). ") This will have the effect of sending py.log.debug messages to the logger corresponding to the 'debug' keyword. Right now, 'debug' is the default logging level (set in misc/log_support.py), but we should really have 'info' as the default. Let me know what you think. Grig From grig at gheorghiu.net Wed Jun 15 00:12:48 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 14 Jun 2005 15:12:48 -0700 (PDT) Subject: [py-dev] Adding logdebug option to defaultconftest.py In-Reply-To: <20050614174517.62319.qmail@web54510.mail.yahoo.com> Message-ID: <20050614221248.84348.qmail@web54508.mail.yahoo.com> On second thoughts, it makes more sense for the logging level to be set per logger, so maybe we should make it a parameter of set_logger. Something like: py.log.set_logger('default', StdoutLogger(), logging_level='debug') We could also have a 'global' logging level if we wanted to, but the standard logging module deals with log levels at the logger level anyway. Grig --- Grig Gheorghiu wrote: > I'm thinking about adding a new option to the defaultconftest.py > file, > something like: > > Option('', '--logdebug', > action="store_true", dest="logdebug", default=False, > help="set logging level to 'debug' (default is 'info'). ") > > This will have the effect of sending py.log.debug messages to the > logger corresponding to the 'debug' keyword. > > Right now, 'debug' is the default logging level (set in > misc/log_support.py), but we should really have 'info' as the > default. > > Let me know what you think. > > Grig > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From hpk at trillke.net Wed Jun 15 00:31:31 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 15 Jun 2005 00:31:31 +0200 Subject: [py-dev] Adding logdebug option to defaultconftest.py In-Reply-To: <20050614221248.84348.qmail@web54508.mail.yahoo.com> References: <20050614174517.62319.qmail@web54510.mail.yahoo.com> <20050614221248.84348.qmail@web54508.mail.yahoo.com> Message-ID: <20050614223131.GG21207@solar.trillke.net> On Tue, Jun 14, 2005 at 15:12 -0700, Grig Gheorghiu wrote: > On second thoughts, it makes more sense for the logging level to be set > per logger, so maybe we should make it a parameter of set_logger. > Something like: > > py.log.set_logger('default', StdoutLogger(), logging_level='debug') > > We could also have a 'global' logging level if we wanted to, but the > standard logging module deals with log levels at the logger level > anyway. For the time being, i'd like to avoid bringing logging module details to py.log. I am starting to think about using py.log here and there and it seems like i am actually going for something like: class Something: log = py.log.Producer('something') def method(self): ... self.log.adhoccategory("processing", x) this shows by default on stdout unless you set a catch-all consumer or a specific one for e.g. 'something'. the "debugging" aspect then becomes a matter of defining which logging information i care for (which could be changed interactively or per command line switches as you suggested) Do that make sense to you? cheers, holger From grig at gheorghiu.net Wed Jun 15 00:56:15 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 14 Jun 2005 15:56:15 -0700 (PDT) Subject: [py-dev] Adding logdebug option to defaultconftest.py In-Reply-To: <20050614223131.GG21207@solar.trillke.net> Message-ID: <20050614225615.97869.qmail@web54508.mail.yahoo.com> --- holger krekel wrote: > On Tue, Jun 14, 2005 at 15:12 -0700, Grig Gheorghiu wrote: > > On second thoughts, it makes more sense for the logging level to be > set > > per logger, so maybe we should make it a parameter of set_logger. > > Something like: > > > > py.log.set_logger('default', StdoutLogger(), logging_level='debug') > > > > We could also have a 'global' logging level if we wanted to, but > the > > standard logging module deals with log levels at the logger level > > anyway. > > For the time being, i'd like to avoid bringing logging module > details to py.log. I am starting to think about using > py.log here and there and it seems like i am actually going > for something like: > > class Something: > log = py.log.Producer('something') > > def method(self): > ... > self.log.adhoccategory("processing", x) > > this shows by default on stdout unless you set a catch-all > consumer or a specific one for e.g. 'something'. > > the "debugging" aspect then becomes a matter of defining > which logging information i care for (which could be changed > interactively or per command line switches as you suggested) > Do that make sense to you? Yes, it makes sense to me. I guess I was thinking more in terms of 'consumers' (which interface more heavily with the logging module), whereas you're approaching it in terms of 'agnostic' producers. So I think you're right, it doesn't make sense to burden the set_logger function with yet another parameter. Maybe the logging level should only be a parameter to the specific loggers defined in consumer.py: File, Stdout, Stderr, Email, etc. Right now it's arbitrarily set to logging.DEBUG. Grig From hpk at trillke.net Wed Jun 15 01:04:56 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 15 Jun 2005 01:04:56 +0200 Subject: [py-dev] Adding logdebug option to defaultconftest.py In-Reply-To: <20050614225615.97869.qmail@web54508.mail.yahoo.com> References: <20050614223131.GG21207@solar.trillke.net> <20050614225615.97869.qmail@web54508.mail.yahoo.com> Message-ID: <20050614230456.GH21207@solar.trillke.net> On Tue, Jun 14, 2005 at 15:56 -0700, Grig Gheorghiu wrote: > --- holger krekel wrote: > > I am starting to think about using > > py.log here and there and it seems like i am actually going > > for something like: > > > > class Something: > > log = py.log.Producer('something') > > > > def method(self): > > ... > > self.log.adhoccategory("processing", x) > > > > this shows by default on stdout unless you set a catch-all > > consumer or a specific one for e.g. 'something'. > > > > the "debugging" aspect then becomes a matter of defining > > which logging information i care for (which could be changed > > interactively or per command line switches as you suggested) > > Do that make sense to you? > > Yes, it makes sense to me. I guess I was thinking more in terms of > 'consumers' (which interface more heavily with the logging module), > whereas you're approaching it in terms of 'agnostic' producers. So I > think you're right, it doesn't make sense to burden the set_logger > function with yet another parameter. right, that's at least the experimental approach. We'll see how it plays out. > Maybe the logging level should only be a parameter to the specific > loggers defined in consumer.py: File, Stdout, Stderr, Email, etc. Right > now it's arbitrarily set to logging.DEBUG. Doesn't the logging module offer the underlying output/formatting logic in a more raw form, maybe? Or else, isn't the logic needed for File, Stdout and Stderr trivial to implement? We then probably don't need to bother with the logging module at all. Btw, I don't know how the logging's email backend works but there probably is some "commit()" like function that actually triggers sending a mail, or not? I probably don't want an email message on each logging entry :-) holger From grig at gheorghiu.net Wed Jun 15 01:12:18 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 14 Jun 2005 16:12:18 -0700 (PDT) Subject: [py-dev] Adding logdebug option to defaultconftest.py Message-ID: <20050614231219.67488.qmail@web54507.mail.yahoo.com> Forgot to copy the py-dev list.... --- holger krekel wrote: > On Tue, Jun 14, 2005 at 15:56 -0700, Grig Gheorghiu wrote: > > --- holger krekel wrote: > > > I am starting to think about using > > > py.log here and there and it seems like i am actually going > > > for something like: > > > > > > class Something: > > > log = py.log.Producer('something') > > > > > > def method(self): > > > ... > > > self.log.adhoccategory("processing", x) > > > > > > this shows by default on stdout unless you set a catch-all > > > consumer or a specific one for e.g. 'something'. > > > > > > the "debugging" aspect then becomes a matter of defining > > > which logging information i care for (which could be changed > > > interactively or per command line switches as you suggested) > > > Do that make sense to you? > > > > Yes, it makes sense to me. I guess I was thinking more in terms of > > 'consumers' (which interface more heavily with the logging module), > > whereas you're approaching it in terms of 'agnostic' producers. So > I > > think you're right, it doesn't make sense to burden the set_logger > > function with yet another parameter. > > right, that's at least the experimental approach. We'll see how > it plays out. > > > Maybe the logging level should only be a parameter to the specific > > loggers defined in consumer.py: File, Stdout, Stderr, Email, etc. > Right > > now it's arbitrarily set to logging.DEBUG. > > Doesn't the logging module offer the underlying output/formatting > logic in a more raw form, maybe? Or else, isn't the logic needed for > File, Stdout and Stderr trivial to implement? We then probably > don't need to bother with the logging module at all. I'll look into using more low-level functions/classes from the logging module. As far as implementing its functionality from scratch: at least for File/Stdout/Stderr it wouldn't be very hard to do, but then we'd need to worry about thread safety etc., which are issues that the logging module takes care for you. Plus the whole mechanism of registering various handlers for various severity levels is pretty nice, although again it wouldn't be that hard to mimic. > Btw, I don't know how the logging's email backend works but there > probably is some "commit()" like function that actually > triggers sending a mail, or not? I probably don't want an email > message on each logging entry :-) My understanding is that it DOES send an email per log entry :-(, but I haven't tinkered with the SMTP logging backend that much yet. I'll be playing with it, as well as the Syslog and WinEvent stuff, and I'll let you know what I find. Grig From dstanek at dstanek.com Wed Jun 15 19:36:45 2005 From: dstanek at dstanek.com (David Stanek) Date: Wed, 15 Jun 2005 13:36:45 -0400 (EDT) Subject: [py-dev] Odd import issue with tests for compatibility modules. Message-ID: <20027.207.58.192.150.1118857005.squirrel@207.58.192.150> The support for Python compatibility modules has been merged into trunk. I pulled the modules (doctest, optparse and testwrap) along with their tests from Python-r241. I was able to modify almost all of the tests to make sure they are testing the compat module and not the library module However, I am having an issue with py/compat/testing/doctest2.py. The original has a testing function that imports itself and then passtest that to a test.test_support.run_doctest method. To mimic that I tried passing the run_doctests method the following: * sys.modules['__main__'] * test_doctest2 after a from py.__.compat.testing import test_doctest2 These do not work...well, all test are successful, but 2 are not being run. And they just happen to be the two that contain unicode characters. If I remove the characters the test will execute successfully. The test is not failing it is just not being run. To debug I added a line to print out the __doc__ of the module, which is one of the tests that is not being executed. The string did NOT include the doctest code. So this seems to be an import thing and not something to do with unicode and/or doctest. Any ideas? I have to take a break from this so that I can get a fresh perspective. David From ianb at colorstudy.com Wed Jun 15 19:40:33 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 15 Jun 2005 12:40:33 -0500 Subject: [py-dev] Odd import issue with tests for compatibility modules. In-Reply-To: <20027.207.58.192.150.1118857005.squirrel@207.58.192.150> References: <20027.207.58.192.150.1118857005.squirrel@207.58.192.150> Message-ID: <42B06811.6030009@colorstudy.com> David Stanek wrote: > The support for Python compatibility modules has been merged into trunk. I > pulled the modules (doctest, optparse and testwrap) along with their tests > from Python-r241. I was able to modify almost all of the tests to make > sure they are testing the compat module and not the library module > > However, I am having an issue with py/compat/testing/doctest2.py. The > original has a testing function that imports itself and then passtest > that to a test.test_support.run_doctest method. To mimic that I tried > passing the run_doctests method the following: > * sys.modules['__main__'] > * test_doctest2 after a from py.__.compat.testing import test_doctest2 > > These do not work...well, all test are successful, but 2 are not being > run. And they just happen to be the two that contain unicode characters. > If I remove the characters the test will execute successfully. The test is > not failing it is just not being run. > > To debug I added a line to print out the __doc__ of the module, which is > one of the tests that is not being executed. The string did NOT include > the doctest code. So this seems to be an import thing and not something to > do with unicode and/or doctest. Would it make sense to force the import, like: from py.compat import doctest sys.modules['doctest'] = doctest ? Maybe you're already doing this, but I wasn't sure. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From dstanek at dstanek.com Thu Jun 16 12:43:47 2005 From: dstanek at dstanek.com (David Stanek) Date: Thu, 16 Jun 2005 06:43:47 -0400 Subject: [py-dev] Odd import issue with tests for compatibility modules. In-Reply-To: <42B06811.6030009@colorstudy.com> References: <20027.207.58.192.150.1118857005.squirrel@207.58.192.150> <42B06811.6030009@colorstudy.com> Message-ID: <20050616104347.GA8073@goliath.hsd1.oh.comcast.net> On Wed, Jun 15, 2005 at 12:40:33PM -0500, Ian Bicking wrote: > Would it make sense to force the import, like: > > from py.compat import doctest > sys.modules['doctest'] = doctest > > ? Maybe you're already doing this, but I wasn't sure. > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org I am already doing that in py/compat/testing/test_doctest.py. The problem is that py/compat/testing/test_doctest2.py had a function named test_mail() that currently does a 'from test import test_doctest2', which essentially is importing itself from the Python tests. That is the import that seems to be giving me trouble. -- David Stanek www.roninds.net GPG keyID #6272EDAF on http://pgp.mit.edu Key fingerprint = 8BAA 7E11 8856 E148 6833 655A 92E2 3E00 6272 EDAF -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From hpk at trillke.net Thu Jun 16 14:12:28 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 16 Jun 2005 14:12:28 +0200 Subject: [py-dev] Odd import issue with tests for compatibility modules. In-Reply-To: <20050616104347.GA8073@goliath.hsd1.oh.comcast.net> References: <20027.207.58.192.150.1118857005.squirrel@207.58.192.150> <42B06811.6030009@colorstudy.com> <20050616104347.GA8073@goliath.hsd1.oh.comcast.net> Message-ID: <20050616121228.GQ21207@solar.trillke.net> On Thu, Jun 16, 2005 at 06:43 -0400, David Stanek wrote: > On Wed, Jun 15, 2005 at 12:40:33PM -0500, Ian Bicking wrote: > > Would it make sense to force the import, like: > > > > from py.compat import doctest > > sys.modules['doctest'] = doctest > > > > ? Maybe you're already doing this, but I wasn't sure. > > > > -- > > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org > > I am already doing that in py/compat/testing/test_doctest.py. The > problem is that py/compat/testing/test_doctest2.py had a function > named test_mail() that currently does a 'from test import > test_doctest2', which essentially is importing itself from the > Python tests. That is the import that seems to be giving me trouble. I have checked in some help for getting this to work (see the commit message). Generally, it's a bit messy because and we may want to modify the tests some more to not use test_support for example but i don't know yet. The immediate problem is a different one, as one can't currently do from py.compat.doctest import * which - i think - you pointed out earlier already. For this the initpkg() function in initpkg.py needs to be enhanced or special-cased, i guess. holger From hpk at trillke.net Thu Jun 16 21:47:14 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 16 Jun 2005 21:47:14 +0200 Subject: [py-dev] refactoring py.log ... In-Reply-To: <20050614231057.79463.qmail@web54509.mail.yahoo.com> References: <20050614230456.GH21207@solar.trillke.net> <20050614231057.79463.qmail@web54509.mail.yahoo.com> Message-ID: <20050616194714.GY21207@solar.trillke.net> Hi Grig, On Tue, Jun 14, 2005 at 16:10 -0700, Grig Gheorghiu wrote: > --- holger krekel wrote: > > Doesn't the logging module offer the underlying output/formatting > > logic in a more raw form, maybe? Or else, isn't the logic needed for > > File, Stdout and Stderr trivial to implement? We then probably > > don't need to bother with the logging module at all. > > I'll look into using more low-level functions/classes from the logging > module. As far as implementing its functionality from scratch: at least > for File/Stdout/Stderr it wouldn't be very hard to do, but then we'd > need to worry about thread safety etc., which are issues that the > logging module takes care for you. Plus the whole mechanism of > registering various handlers for various severity levels is pretty > nice, although again it wouldn't be that hard to mimic. Yes, i now even think that reusing the logging module at API level might not be such a good idea. Reusing some code and ideas by copy-and-paste may just be enough. It's no dark magic that is changing everyday. Moreover, I am really beginning to come from the usage side and i feel that i don't really want severity-distinctions statically everywhere in the source code although it may seem natural at first. (see my example in an earlier mail). I guess i am at the verge of wanting to follow a more radical keyword/tag based approach. Would you mind if i refactor and simplify things accordingly? It's possible that we introduce some default severity levels but i'd like to postpone it until there is a real need. > > Btw, I don't know how the logging's email backend works but there > > probably is some "commit()" like function that actually > > triggers sending a mail, or not? I probably don't want an email > > message on each logging entry :-) > > My understanding is that it DOES send an email per log entry :-(, but I > haven't tinkered with the SMTP logging backend that much yet. This may point to an interesting problem. Logging backends/consumers are not really stateless. For example, traditionally in Unix the sysloggers have two modes of writing to files, one that commits ('fsync()') and one that doesn't. The latter is faster but log entries may get lost if there is a crash (even a minute old entries). Or does the logging module offer this distinction? Anyway, it's just a few lines of code and it's ok if we do it ourselves (and test it accordingly). Sending batched log entries is a use case where we probably could do something simple, e.g. l = [] py.log.set_logger('default', l.append) call_program() # logs various stuff for msg in l: if 'error' in msg.keywords: # send an email with a selection of messages IOW, we don't even neccessarily need a custom Email logger and introduce names and API design for it. > I'll be playing with it, as well as the Syslog and WinEvent > stuff, and I'll let you know what I find. that would be useful. cheers, holger From grig at gheorghiu.net Thu Jun 16 21:52:42 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Thu, 16 Jun 2005 12:52:42 -0700 (PDT) Subject: [py-dev] refactoring py.log ... Message-ID: <20050616195242.56013.qmail@web54502.mail.yahoo.com> Forgot to CC the list again... Hi, Holger --- holger krekel wrote: > Hi Grig, > > On Tue, Jun 14, 2005 at 16:10 -0700, Grig Gheorghiu wrote: > > --- holger krekel wrote: > > > Doesn't the logging module offer the underlying output/formatting > > > logic in a more raw form, maybe? Or else, isn't the logic needed > for > > > File, Stdout and Stderr trivial to implement? We then probably > > > don't need to bother with the logging module at all. > > > > I'll look into using more low-level functions/classes from the > logging > > module. As far as implementing its functionality from scratch: at > least > > for File/Stdout/Stderr it wouldn't be very hard to do, but then > we'd > > need to worry about thread safety etc., which are issues that the > > logging module takes care for you. Plus the whole mechanism of > > registering various handlers for various severity levels is pretty > > nice, although again it wouldn't be that hard to mimic. > > Yes, i now even think that reusing the logging module at API level > might not be such a good idea. Reusing some code and ideas by > copy-and-paste may just be enough. It's no dark magic that is > changing everyday. > > Moreover, I am really beginning to come from the usage > side and i feel that i don't really want severity-distinctions > statically everywhere in the source code although it may seem > natural at first. (see my example in an earlier mail). > > I guess i am at the verge of wanting to follow a more radical > keyword/tag based approach. Would you mind if i refactor > and simplify things accordingly? It's possible that we > introduce some default severity levels but i'd like to > postpone it until there is a real need. I don't mind at all if you refactor and simplify things. I haven't done much work on it since my last commit. You're probably right about not needing severity levels right away. I agree with you that we should look at things from an 'end-user' point of view and thus make the calls to the py.log functionality as simple and natural as possible. > > > Btw, I don't know how the logging's email backend works but there > > > probably is some "commit()" like function that actually > > > triggers sending a mail, or not? I probably don't want an email > > > message on each logging entry :-) > > > > My understanding is that it DOES send an email per log entry :-(, > but I > > haven't tinkered with the SMTP logging backend that much yet. > > This may point to an interesting problem. Logging backends/consumers > are not really stateless. For example, traditionally in Unix > the sysloggers have two modes of writing to files, one that > commits ('fsync()') and one that doesn't. The latter is > faster but log entries may get lost if there is a crash (even > a minute old entries). Or does the logging module offer this > distinction? Anyway, it's just a few lines of code and it's > ok if we do it ourselves (and test it accordingly). > > Sending batched log entries is a use case where we > probably could do something simple, e.g. > > l = [] > py.log.set_logger('default', l.append) > call_program() # logs various stuff > for msg in l: > if 'error' in msg.keywords: > # send an email with a selection of messages > > IOW, we don't even neccessarily need a custom Email logger > and introduce names and API design for it. Yeah, it would be nice to have the 'batch email' functionality. I don't think the logging module offers any distinction between single email messages and batch mode. > > I'll be playing with it, as well as the Syslog and WinEvent > > stuff, and I'll let you know what I find. > > that would be useful. Haven't had time to do this yet.... Grig From dstanek at dstanek.com Fri Jun 17 14:23:08 2005 From: dstanek at dstanek.com (David Stanek) Date: Fri, 17 Jun 2005 08:23:08 -0400 (EDT) Subject: [py-dev] Odd import issue with tests for compatibility modules. In-Reply-To: <20050616121228.GQ21207@solar.trillke.net> References: <20027.207.58.192.150.1118857005.squirrel@207.58.192.150> <42B06811.6030009@colorstudy.com> <20050616104347.GA8073@goliath.hsd1.oh.comcast.net> <20050616121228.GQ21207@solar.trillke.net> Message-ID: <43375.192.168.1.1.1119010988.squirrel@192.168.1.1> > > I have checked in some help for getting this to work (see the > commit message). Generally, it's a bit messy because and we > may want to modify the tests some more to not use test_support for > example but i don't know yet. Thanks! I'll check it out today. Actually test_support uses unittest and doctest to execute the module's tests. Once doctest support is included in py.test I won't have to use test_support anymore for the doctests. Further down the line, once unittest support is included, well you can see where I am going. The test_support will no longer be needed... > > The immediate problem is a different one, as one can't > currently do > > from py.compat.doctest import * > > which - i think - you pointed out earlier already. For this > the initpkg() function in initpkg.py needs to be enhanced > or special-cased, i guess. > I have a patch for this that I am currently working on. I have to write a couple more tests to cover all the various ways you can import things into a namespace. David From grig at gheorghiu.net Fri Jun 17 16:37:25 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Fri, 17 Jun 2005 07:37:25 -0700 (PDT) Subject: [py-dev] Re: [py-svn] r13515 - in py/dist/py: . log log/testing In-Reply-To: <20050617054437.EE33027B7D@code1.codespeak.net> Message-ID: <20050617143725.41135.qmail@web54509.mail.yahoo.com> --- hpk at codespeak.net wrote: > Author: hpk > Date: Fri Jun 17 07:44:36 2005 > New Revision: 13515 > > Modified: > py/dist/py/__init__.py > py/dist/py/log/consumer.py > py/dist/py/log/producer.py > py/dist/py/log/testing/test_log.py > Log: > issue3 in-progress > > refactored logging API > > - no severity levels offered by default anymore > > - no logging module used at the moment > > - simplified API > > - there now is a 'py.log.default' defaultlogger > > Hi, Holger I've looked at the refactored version of the log functionality and for some reason it doesn't seem as intuitive to use as the old one. Now, as an end user, you need to know the distinction between a 'producer' and a 'consumer', which might be hard to grasp, at least at first sight. You also need to use the Producer class directly, as it is not hidden anymore. My preference would be to hide names such as producers and consumers and expose more user-friendly names such as logs and loggers. Other than that, I like the fact that the keyword-based matching offers a lot of flexibility. I can do for example: py.log.db = py.log.Producer('db') py.log.setconsumer('db', function_to_insert_in_db) py.log.db("hello world") Have you have any use for multiple keywords? Just curious, as I haven't seen the need for it yet. About severity levels: I guess in the current version you can just set the consumer for that particular keyword to None in a configuration file, and that would disable logging for that keyword. That's how you would turn on and off debugging messages for example, am I right? Grig From grig at gheorghiu.net Fri Jun 17 18:15:21 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Fri, 17 Jun 2005 09:15:21 -0700 (PDT) Subject: [py-dev] py.log and multiple keywords Message-ID: <20050617161521.25843.qmail@web54504.mail.yahoo.com> Holger and others interested in this topic, I was thinking that in many situations it would be helpful to be able to specify multiple keywords that would each point to a different consumer and that would be all 'kick in' on a call to the producer. Basically, I'd want this test to pass: def test_multiple_keywords(self): log = py.log.Producer('console logfile db') py.log.setconsumer('console', py.log.STDOUT) logfile = tempdir.join('mylogfile.out') py.log.setconsumer('logfile', open(str(logfile), 'w', buffering=1)) dbfile = tempdir.join('mydbfile.out') py.log.setconsumer('db', open(str(dbfile), 'w', buffering=1)) res, out, err = callcapture(log, "hello") assert out.strip() == "[console:logfile:db] hello" assert logfile.readlines() == ['[console:logfile:db] hello\n'] assert dbfile.readlines() == ['[console:logfile:db] hello\n'] Right now the last 2 assert statements fail, because _getconsumer returns as soon as it finds a match for the first keyword, so only the 'console' consumer gets a chance to consume the message. I'm thinking that the _getconsumer function could be a generator that yields a message-processing function for each keyword it finds in its registry. What do you think? Grig From grig at gheorghiu.net Fri Jun 17 19:00:21 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Fri, 17 Jun 2005 10:00:21 -0700 (PDT) Subject: [py-dev] py.log and multiple keywords In-Reply-To: <20050617161521.25843.qmail@web54504.mail.yahoo.com> Message-ID: <20050617170021.39132.qmail@web54503.mail.yahoo.com> --- Grig Gheorghiu wrote: > Holger and others interested in this topic, > > I was thinking that in many situations it would be helpful to be able > to specify multiple keywords that would each point to a different > consumer and that would be all 'kick in' on a call to the producer. > Basically, I'd want this test to pass: > > def test_multiple_keywords(self): > log = py.log.Producer('console logfile db') > py.log.setconsumer('console', py.log.STDOUT) > logfile = tempdir.join('mylogfile.out') > py.log.setconsumer('logfile', open(str(logfile), 'w', > buffering=1)) > dbfile = tempdir.join('mydbfile.out') > py.log.setconsumer('db', open(str(dbfile), 'w', buffering=1)) > > res, out, err = callcapture(log, "hello") > assert out.strip() == "[console:logfile:db] hello" > > assert logfile.readlines() == ['[console:logfile:db] > hello\n'] > assert dbfile.readlines() == ['[console:logfile:db] hello\n'] > > Right now the last 2 assert statements fail, because _getconsumer > returns as soon as it finds a match for the first keyword, so only > the > 'console' consumer gets a chance to consume the message. > > I'm thinking that the _getconsumer function could be a generator that > yields a message-processing function for each keyword it finds in its > registry. What do you think? > > Grig Here's how I modified __call__ and _getconsumer in producer.py in order to achieve the goal stated above. Of course, the old semantics doesn't apply at this point, but all the current tests still pass. def __call__(self, *args): for func in self._getconsumer(self.keywords): if func is not None: func(self.Message(self.keywords, args)) def _getconsumer(self, keywords): found_consumer = False for keyword in keywords: consumer = self.keywords2consumer.get((keyword,)) if consumer: found_consumer = True yield consumer if not found_consumer: yield self.keywords2consumer.get('default', default_consumer) Now this test passes (on Unix, because on Windows there's a problem with opening files in buffering mode): def test_multiple_keywords(self): log = py.log.Producer('console logfile db email') py.log.setconsumer('console', py.log.STDOUT) logfile = tempdir.join('mylogfile.out') py.log.setconsumer('logfile', open(str(logfile), 'w', buffering=1)) dbfile = tempdir.join('mydbfile.out') py.log.setconsumer('db', open(str(dbfile), 'w', buffering=1)) res, out, err = callcapture(log, "hello") assert out.strip() == "[console:logfile:db:email] hello" assert logfile.readlines() == ['[console:logfile:db:email] hello\n'] assert dbfile.readlines() == ['[console:logfile:db:email] hello\n'] From grig at gheorghiu.net Fri Jun 17 19:59:04 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Fri, 17 Jun 2005 10:59:04 -0700 (PDT) Subject: [py-dev] py.log and multiple keywords In-Reply-To: <20050617161521.25843.qmail@web54504.mail.yahoo.com> Message-ID: <20050617175904.22722.qmail@web54510.mail.yahoo.com> --- Grig Gheorghiu wrote: > Holger and others interested in this topic, > > I was thinking that in many situations it would be helpful to be able > to specify multiple keywords that would each point to a different > consumer and that would be all 'kick in' on a call to the producer. > Basically, I'd want this test to pass: > > def test_multiple_keywords(self): > log = py.log.Producer('console logfile db') > py.log.setconsumer('console', py.log.STDOUT) > logfile = tempdir.join('mylogfile.out') > py.log.setconsumer('logfile', open(str(logfile), 'w', > buffering=1)) > dbfile = tempdir.join('mydbfile.out') > py.log.setconsumer('db', open(str(dbfile), 'w', buffering=1)) > > res, out, err = callcapture(log, "hello") > assert out.strip() == "[console:logfile:db] hello" > > assert logfile.readlines() == ['[console:logfile:db] > hello\n'] > assert dbfile.readlines() == ['[console:logfile:db] hello\n'] > > Right now the last 2 assert statements fail, because _getconsumer > returns as soon as it finds a match for the first keyword, so only > the > 'console' consumer gets a chance to consume the message. > > I'm thinking that the _getconsumer function could be a generator that > yields a message-processing function for each keyword it finds in its > registry. What do you think? > > Grig Sorry if I get boring by pounding on this topic, but I've been playing with the py.log stuff by incorporating it in a framework that I'm using at my work, and Holger's latest refactoring + my multiple keyword idea above works really well. Here's a section of a configuration file I have (I'm not using the py library directly yet, instead I just concatenated the consumer and producer modules in a module called log.py): ############### import os import log as qalog workingdir = "/data01/qa/work" restoredir = "/data01/qa/restore" # logging stuff run_id = qautils.get_run_id(workingdir) logfile = os.path.join(workingdir, "%s.out" % run_id) print "Log file for debug messages is %s" % logfile qalog.setconsumer('console', qalog.STDOUT) qalog.setconsumer('logfile', open(str(logfile), 'w', buffering=1)) log = qalog.Producer('console logfile') log.debug = qalog.Producer('logfile') log.info = qalog.Producer('console logfile') log.warn = qalog.Producer('console logfile') log.error = qalog.Producer('console logfile') log.critical = qalog.Producer('console logfile') ############### And here's how I use it in my test code: log.debug("Test duration:", self.result.duration) log.info("COMMAND: " + command) log.error("Cannot log data to DB when result is not defined") Here's how messages are printed on the console: [console:logfile] COMMAND: tar -xv /data01/qa/work.tar Here's how messages are printed in the log file: [logfile] add_memo for result_id 35415 returned: 'Memo added successfully' So I think that the py.log stuff is on the right track. Holger, you da man :-) Grig From hpk at trillke.net Fri Jun 17 23:04:58 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 17 Jun 2005 23:04:58 +0200 Subject: [py-dev] py.log and multiple keywords In-Reply-To: <20050617161521.25843.qmail@web54504.mail.yahoo.com> References: <20050617161521.25843.qmail@web54504.mail.yahoo.com> Message-ID: <20050617210458.GK21207@solar.trillke.net> Hi Grig, On Fri, Jun 17, 2005 at 09:15 -0700, Grig Gheorghiu wrote: > Holger and others interested in this topic, > > I was thinking that in many situations it would be helpful to be able > to specify multiple keywords that would each point to a different > consumer and that would be all 'kick in' on a call to the producer. > Basically, I'd want this test to pass: > > def test_multiple_keywords(self): > log = py.log.Producer('console logfile db') > py.log.setconsumer('console', py.log.STDOUT) > logfile = tempdir.join('mylogfile.out') > py.log.setconsumer('logfile', open(str(logfile), 'w', > buffering=1)) > dbfile = tempdir.join('mydbfile.out') > py.log.setconsumer('db', open(str(dbfile), 'w', buffering=1)) > > res, out, err = callcapture(log, "hello") > assert out.strip() == "[console:logfile:db] hello" > > assert logfile.readlines() == ['[console:logfile:db] hello\n'] > assert dbfile.readlines() == ['[console:logfile:db] hello\n'] > > Right now the last 2 assert statements fail, because _getconsumer > returns as soon as it finds a match for the first keyword, so only the > 'console' consumer gets a chance to consume the message. Yes, however with the current svn version you can simply do: def test_multiple_keywords(self): log = py.log.Producer('console logfile db') logfile = open(tempdir.join('mylogfile.out'), 'a', 1) dbfile = open(tempdir.join('mydbfile.out'), 'a', 1) def multilogger(msg): if 'logfile' in msg.keywords: print >>logfile, str(msg) if 'db' in msg.keywords: print >>dbfile, str(msg) py.log.setconsumer('console', multilogger) res, out, err = callcapture(log, "hello") assert out.strip() == "[console:logfile:db] hello" assert logfile.readlines() == ['[console:logfile:db] hello\n'] assert dbfile.readlines() == ['[console:logfile:db] hello\n'] So it seems that with the simple ability to associate functions to keywords you can basically do a lot of stuff without knowing any big API or intricate details. cheers, holger From grig at gheorghiu.net Mon Jun 20 15:56:00 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 20 Jun 2005 06:56:00 -0700 (PDT) Subject: [py-dev] py.log and multiple keywords In-Reply-To: <20050617210458.GK21207@solar.trillke.net> Message-ID: <20050620135600.76210.qmail@web54501.mail.yahoo.com> --- holger krekel wrote: > Hi Grig, > > On Fri, Jun 17, 2005 at 09:15 -0700, Grig Gheorghiu wrote: > > Holger and others interested in this topic, > > > > I was thinking that in many situations it would be helpful to be > able > > to specify multiple keywords that would each point to a different > > consumer and that would be all 'kick in' on a call to the producer. > > Basically, I'd want this test to pass: > > > > def test_multiple_keywords(self): > > log = py.log.Producer('console logfile db') > > py.log.setconsumer('console', py.log.STDOUT) > > logfile = tempdir.join('mylogfile.out') > > py.log.setconsumer('logfile', open(str(logfile), 'w', > > buffering=1)) > > dbfile = tempdir.join('mydbfile.out') > > py.log.setconsumer('db', open(str(dbfile), 'w', > buffering=1)) > > > > res, out, err = callcapture(log, "hello") > > assert out.strip() == "[console:logfile:db] hello" > > > > assert logfile.readlines() == ['[console:logfile:db] > hello\n'] > > assert dbfile.readlines() == ['[console:logfile:db] > hello\n'] > > > > Right now the last 2 assert statements fail, because _getconsumer > > returns as soon as it finds a match for the first keyword, so only > the > > 'console' consumer gets a chance to consume the message. > > Yes, however with the current svn version you can simply do: > > def test_multiple_keywords(self): > log = py.log.Producer('console logfile db') > > logfile = open(tempdir.join('mylogfile.out'), 'a', 1) > dbfile = open(tempdir.join('mydbfile.out'), 'a', 1) > > def multilogger(msg): > if 'logfile' in msg.keywords: > print >>logfile, str(msg) > if 'db' in msg.keywords: > print >>dbfile, str(msg) > > py.log.setconsumer('console', multilogger) > > res, out, err = callcapture(log, "hello") > assert out.strip() == "[console:logfile:db] hello" > > assert logfile.readlines() == ['[console:logfile:db] > hello\n'] > assert dbfile.readlines() == ['[console:logfile:db] > hello\n'] > > So it seems that with the simple ability to associate functions > to keywords you can basically do a lot of stuff without knowing > any big API or intricate details. > Hi, Holger I see your point. It's still somehow convoluted to achieve my goal using the solution you give, but on the other hand I understand that you want to keep the API 'lean and mean'. I still haven't figured out what a practical application of the current semantics for multiple keywords would be -- can you give an example? I noticed you changed the _getconsumer function to first match on the whole multiple keyword tuple, and then on prefixes of the tuple. Grig From hpk at trillke.net Mon Jun 20 17:32:03 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 20 Jun 2005 17:32:03 +0200 Subject: [py-dev] py.log and multiple keywords In-Reply-To: <20050620135600.76210.qmail@web54501.mail.yahoo.com> References: <20050617210458.GK21207@solar.trillke.net> <20050620135600.76210.qmail@web54501.mail.yahoo.com> Message-ID: <20050620153203.GA21207@solar.trillke.net> Hi Grig! [Grig] > [holger] > > So it seems that with the simple ability to associate functions > > to keywords you can basically do a lot of stuff without knowing > > any big API or intricate details. > > I see your point. It's still somehow convoluted to achieve my goal > using the solution you give, but on the other hand I understand that > you want to keep the API 'lean and mean'. I basically think that application should be left free to define it's logging structure and semantics. The 'multilogger' example i gave makes it more obvious user what the semantics of closing/opening files etc.pp. are. It's really simple code and i bet faster to write than understanding and tweaking the logging module's structure. > I still haven't figured out what a practical application of the current > semantics for multiple keywords would be -- can you give an example? I > noticed you changed the _getconsumer function to first match on the > whole multiple keyword tuple, and then on prefixes of the tuple. I have used it the following way so far: # -- 'somefile.py': import py log = py.log.Producer("somefile") py.log.setconsumer("somefile", None) # route all logging to /dev/null def dispatch(x): log.dispatch(x) def send(host, arg): log.send(arg, "to", host) # -- 'test_somefile.py' import py py.log.setconsumer("somefile", py.log.STDOUT) # show more info for failing tests py.log.setconsumer("somefile send", None) # don't show dispatching, though ... # -- end files So not only can you come up with new ad-hoc categories but you can have fine-grained view control from remote places. You can for example, have a different logging view from a main cmdline entrypoint than from the tests etc.pp. In this view, the distinction between 'debug' and 'info' etc.pp. depends on the perspective and is not a static property of the source code. IMO this lets one use py.log.Producer's much like print statements but the effort of writing down those "print" info is then not lost but put to productive use. cheers, holger From grig at gheorghiu.net Mon Jun 20 17:59:12 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Mon, 20 Jun 2005 08:59:12 -0700 (PDT) Subject: [py-dev] py.log and multiple keywords In-Reply-To: <20050620153203.GA21207@solar.trillke.net> Message-ID: <20050620155912.55703.qmail@web54507.mail.yahoo.com> --- holger krekel wrote: > Hi Grig! > >> [Grig] >> I still haven't figured out what a practical application of the >> current >> semantics for multiple keywords would be -- can you give an >> example? > [holger] > I have used it the following way so far: > > # -- 'somefile.py': > > import py > > log = py.log.Producer("somefile") > py.log.setconsumer("somefile", None) # route all logging to > /dev/null > > def dispatch(x): > log.dispatch(x) > > def send(host, arg): > log.send(arg, "to", host) > > # -- 'test_somefile.py' > import py > > py.log.setconsumer("somefile", py.log.STDOUT) # show more info > for failing tests > py.log.setconsumer("somefile send", None) # don't show > dispatching, though > > ... > # -- end files > > So not only can you come up with new ad-hoc categories but you > can have fine-grained view control from remote places. You can for > example, have a different logging view from a main cmdline entrypoint > than from the tests etc.pp. In this view, the distinction between > 'debug' and 'info' etc.pp. depends on the perspective and is > not a static property of the source code. > OK, this makes sense now. I was already using the various severity levels dynamically, via a configuration file, as I noted in a previous post. But I wasn't sure how the multiple keywords come into play. Now I get it. I'll chew on all this some more and I'll probably post a message with how I ended up using py.log in my code. Thanks, Grig From cfbolz at gmx.de Tue Jun 21 00:40:35 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 21 Jun 2005 00:40:35 +0200 Subject: [py-dev] distributed computing with py.execnet Message-ID: <42B745E3.5040703@gmx.de> Hi Py-Dev! I experimented a bit with py.execnet today and just want to share the result of the experiment. I wrote a little proof of concept for distributing a big computational task between several computers (think cluster) without having to install anything on them. There are several processes involved: a server that establishes connections via execnet gateways and sends them their task via the channels. Then the server listens at a socket. If a node has finished its task, it sends the server a notification via the socket and then the server fetches the result via the appropriate channel. Then the node gets a new task via the channel. I used the search for Mersenne primes with the Lucas-Lehmer-test as the computational task (which one wouldn't implement in Python, of course). The whole implementation, server and primality test is less than 130 lines of code (see below). There are still some things missing (like a clean shutdown for example) but the basic concept works very well. execnet rocks! Regards, Carl Friedrich #-----------server.py-------------- import py import SocketServer ip = "192.168.10.1" # ip of the server nodes = ["cfbolz at linux", "jana at albert"] port = 50007 chunk_size = 50 source = py.code.Source(py.path.local().join("task.py").read()) log = py.log.Producer("server") py.log.setconsumer("server", None) class TaskDistributor(object): def __init__(self, nodes, chunk_size=1): self.i = 1 self.nodes = nodes self.chunk_size = chunk_size self.channels = [] for node in nodes: print node channel = py.execnet.SshGateway(node).remote_exec(source) self.channels.append(channel) assert channel.receive() == "online" channel.send(len(self.channels) - 1) channel.send((ip, port)) channel.send(range(self.i, self.i + self.chunk_size)) channel.send(len(nodes) * 3) assert channel.receive() == "ok" log.init("sent task %s" % (self.i,)) self.i += self.chunk_size def new_task(self, number): channel = self.channels[number] results = channel.receive() channel.send("ok") channel.send(range(self.i, self.i + self.chunk_size)) log.new_job(self.i) self.i += self.chunk_size assert channel.receive() == "ok" for result in results: if result[1]: print "%s = 2**%s - 1 is a prime! result from node %s" % (2 ** result[0] - 1, result[0], number) if not result[1]: print "2**%s - 1 is not prime" % (self.i, ) class RequestHandler(SocketServer.BaseRequestHandler): def handle(self): try: number = int(self.request.recv(10)) log.request("from", number) self.request.send("ok") RequestHandler.distributor.new_task(number) except ValueError: self.request.send("not a number") log.request("closing socket") self.request.close() if __name__ == '__main__': RequestHandler.distributor = TaskDistributor(nodes, 100) server = SocketServer.TCPServer((ip, port), RequestHandler) print server server.serve_forever() #-----------task.py-------------- import operator import random import socket import time import sys try: import psyco psyco.full() except ImportError: pass def fermattest(p, tests=30): if p == 1: return False for i in xrange(tests): a = long(random.random() * (p - 1) + 1) if not pow(a, p, p) == a: return False return True def M(p): return 2 ** p - 1 def lucastest(p): curr = 4 m = M(p) for i in range(p - 2): curr = (curr ** 2 - 2) % m if curr == 0: return True return False def process_task(task_data): results = [] for p in task_data: print p if not fermattest(p): results.append((p, False)) else: results.append((p, lucastest(p))) return results channel.send("online") mynumber = str(channel.receive()) socketaddress = channel.receive() task_data = channel.receive() sleep = channel.receive() channel.send("ok") f = file("/tmp/out.txt", "w", 0) print >> f, "sleeping" time.sleep(sleep) # give the server time to establish all connections print >> f, "starting to work" while 1: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) result = process_task(task_data) print >>f, result, socketaddress print >>f, "connecting", s try: s.connect(socketaddress) except Exception, e: print >>f, "error!", e f.close() raise print >>f, "connected2" s.send(mynumber) print >>f, "sent number" assert s.recv(2) == "ok" s.close() channel.send(result) print >>f, "sent result" assert channel.receive() == "ok" task_data = channel.receive() print >>f, "new task data %s" % task_data channel.send("ok") From hpk at trillke.net Tue Jun 21 01:18:26 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 21 Jun 2005 01:18:26 +0200 Subject: [py-dev] distributed computing with py.execnet In-Reply-To: <42B745E3.5040703@gmx.de> References: <42B745E3.5040703@gmx.de> Message-ID: <20050620231826.GF21207@solar.trillke.net> Hi Carl-Friedrich! thanks for the feedback! On Tue, Jun 21, 2005 at 00:40 +0200, Carl Friedrich Bolz wrote: > I experimented a bit with py.execnet today and just want to share the > result of the experiment. I wrote a little proof of concept for > distributing a big computational task between several computers (think > cluster) without having to install anything on them. > > There are several processes involved: a server that establishes > connections via execnet gateways and sends them their task via the > channels. Then the server listens at a socket. If a node has finished > its task, it sends the server a notification via the socket and then the > server fetches the result via the appropriate channel. Then the node > gets a new task via the channel. In theory and probably even in practice you should not have to use a separate socket. What keeps you from (re)using a channel there as well? For example, you may do: # client (which manages the cluster) def receivernotificationcallback((hostid, result)): # process result coming from hostid (be careful because # this callback executes directly in the IO receive # thread, you may want to just put things in a queue # and process it in some collecting-result thread) channel = gw.newchannel(receivernotificationcallback) for hostid in ...: gw.remote_exec(..., channel=channel) # use preconstructed channel # and the callback resp. so the other side's stuff will go the callback function directly. If you want to have a channel for control-messages and one for results you could do: for hostid in ...: notify_channel = gw.newchannel(receivernotificationcallback) channel = gw.remote_exec(...) channel.send(notify_channel) the other side can then receive the result channel and sent the result when it is ready. This is probably a cleaner solution than the former. Note that the above usages (and especially the first one) are a bit on the edge and probably contradict some refactoring ideas i discussed with Armin. Mainly, we want to have channels arbitrarily sendable through a chain of gateways but then setup/teardown mechanics can become very involved if we allow a channel to be connected to multiple end-points, i.e. a 1:N relationship. Instead we think about restricting channels to always come in 1:1 relationships where each end could be send arbitrarily across networks and the end points would stay connected to each other (and routed through the chain of gateways) all the time. cheers, holger From cfbolz at gmx.de Tue Jun 21 02:16:19 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 21 Jun 2005 02:16:19 +0200 Subject: [py-dev] distributed computing with py.execnet In-Reply-To: <20050620231826.GF21207@solar.trillke.net> References: <42B745E3.5040703@gmx.de> <20050620231826.GF21207@solar.trillke.net> Message-ID: <42B75C53.4050701@gmx.de> Hi Holger! holger krekel wrote: > In theory and probably even in practice you should not > have to use a separate socket. What keeps you from (re)using > a channel there as well? Hum. I confess that I only read the docs not the source until now. The docs mention no channels with callbacks, only the blocking channel.receive (which isn't enough for my use case). I even thought that there had to be an easier way, but well... > > For example, you may do: > > # client (which manages the cluster) > > def receivernotificationcallback((hostid, result)): > # process result coming from hostid (be careful because > # this callback executes directly in the IO receive > # thread, you may want to just put things in a queue > # and process it in some collecting-result thread) > > channel = gw.newchannel(receivernotificationcallback) > > for hostid in ...: > gw.remote_exec(..., channel=channel) # use preconstructed channel > # and the callback resp. > so the other side's stuff will go the callback function directly. > Ah. Everything becomes clearer now! With a callback this my toy example is even shorter. See below. > If you want to have a channel for control-messages and > one for results you could do: > > for hostid in ...: > notify_channel = gw.newchannel(receivernotificationcallback) > channel = gw.remote_exec(...) > channel.send(notify_channel) > > the other side can then receive the result channel and sent > the result when it is ready. This is probably a cleaner > solution than the former. I didn't even know that sending channels is possible. Thanks a lot for the explanations! Carl Friedrich P.S.: did you know that 285542542228279613901563566102164008326164238644702889199247456602284400390600653875954571505539843239754513915896150297878399377056071435169747221107988791198200988477531339214282772016059009904586686254989084815735422480409022344297588352526004383890632616124076317387416881148592486188361873904175783145696016919574390765598280188599035578448591077683677175520434074287726578006266759615970759521327828555662781678385691581844436444812511562428136742490459363212810180276096088111401003377570363545725120924073646921576797146199387619296560302680261790118132925012323046444438622308877924609373773012481681672424493674474488537770155783006880852648161513067144814790288366664062257274665275787127374649231096375001170901890786263324619578795731425693805073056119677580338084333381987500902968831935913095269821311141322393356490178488728982288156282600813831296143663845945431144043753821542871277745606447858564159213328443580206422714694913091762716447041689678070096773590429808909616 750452927258000843500344831628297089902728649981994387647234574276263729694848304750917174186181130688518792748622612293341368928056634384466646326572476167275660839105650528975713899320211121495795311427946254553305387067821067601768750977866100460014602138408448021225053689054793742003095722096732954750721718115531871310231057902608580607L is a prime? #-------------server.py--------------- import py import sys chunk_size = 50 source = py.code.Source(py.path.local().join("task.py").read()) nodes = ["cfbolz at linux", "jana at albert"] class TaskDistributor(object): def __init__(self, nodes, chunk_size=1): self.i = 1 self.nodes = nodes self.chunk_size = chunk_size self.channels = [] for node in nodes: print node gw = py.execnet.SshGateway(node) channel = gw.newchannel(self.new_task) gw.remote_exec(source, channel=channel) self.channels.append(channel) channel.send(len(self.channels) - 1) channel.send(range(self.i, self.i + self.chunk_size)) self.i += self.chunk_size def new_task(self, message): number, results = message channel = self.channels[number] channel.send("ok") channel.send(range(self.i, self.i + self.chunk_size)) self.i += self.chunk_size for result in results: if result[1]: print "%s = 2**%s - 1 is a prime! result from node %s" % (2 ** result[0] - 1, result[0], number) if not result[1]: print "2**%s - 1 is not prime" % (self.i, ) if __name__ == '__main__': TaskDistributor(nodes, chunk_size) while 1: pass #-------------task.py--------------- import random def fermattest(p, tests=30): if p == 1: return False for i in xrange(tests): a = long(random.random() * (p - 1) + 1) if not pow(a, p, p) == a: return False return True def M(p): return 2 ** p - 1 def lucastest(p): curr = 4 m = M(p) for i in range(p - 2): curr = (curr ** 2 - 2) % m if curr == 0: return True return False def process_task(task_data): results = [] for p in task_data: if not fermattest(p): results.append((p, False)) else: results.append((p, lucastest(p))) return results mynumber = channel.receive() task_data = channel.receive() while 1: result = process_task(task_data) channel.send((mynumber, result)) assert channel.receive() == "ok" task_data = channel.receive() From hpk at trillke.net Tue Jun 21 11:56:06 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 21 Jun 2005 11:56:06 +0200 Subject: [py-dev] distributed computing with py.execnet In-Reply-To: <42B75C53.4050701@gmx.de> References: <42B745E3.5040703@gmx.de> <20050620231826.GF21207@solar.trillke.net> <42B75C53.4050701@gmx.de> Message-ID: <20050621095606.GN21207@solar.trillke.net> Hi Carl-Friedrich, On Tue, Jun 21, 2005 at 02:16 +0200, Carl Friedrich Bolz wrote: > holger krekel wrote: > >In theory and probably even in practice you should not > >have to use a separate socket. What keeps you from (re)using > >a channel there as well? > > Hum. I confess that I only read the docs not the source until now. The > docs mention no channels with callbacks, only the blocking > channel.receive (which isn't enough for my use case). I even thought > that there had to be an easier way, but well... Yeah, the thing is that semantics especially of passing around channels are still under experimentation and reconsideration so it's not advertised yet (see end of my last mail). cheers, holger From grig at gheorghiu.net Tue Jun 21 16:54:37 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 21 Jun 2005 07:54:37 -0700 (PDT) Subject: [py-dev] What's next for py.log? Message-ID: <20050621145438.18031.qmail@web54509.mail.yahoo.com> Hi, Holger Do you think the core consumer and producer code will remain as is for py.log? It seems to work pretty well. I might still sub-class the Producer class for my multiple-keyword needs -- but that can easily be done in my application code, without changing py.log itself. Are you thinking about adding more consumer objects such as syslog, email, Win Events? If so, I can give some of them a shot. I'm also thinking about advertising py.log a little on my blog. Or should I refrain until the py release? Let me know what you think. Grig From hpk at trillke.net Tue Jun 21 19:22:30 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 21 Jun 2005 19:22:30 +0200 Subject: [py-dev] What's next for py.log? In-Reply-To: <20050621145438.18031.qmail@web54509.mail.yahoo.com> References: <20050621145438.18031.qmail@web54509.mail.yahoo.com> Message-ID: <20050621172230.GR21207@solar.trillke.net> Hi Grig, On Tue, Jun 21, 2005 at 07:54 -0700, Grig Gheorghiu wrote: > Do you think the core consumer and producer code will remain as is for > py.log? I currently guess so, yes. I still feel there will be some re-thinking after more people have used it more. Especially i think that the dispatching rules (how keywords are connected to consumers) will have to become more flexible at some point. Hum, coming to think of it we might want to change 'setconsumer' into 'subscribe' and offer a way to specify if a subscriber will consume a message or just look at it. This may also help with your multilogger use case. (Also see below for an example). > It seems to work pretty well. I might still sub-class the > Producer class for my multiple-keyword needs -- but that can easily be > done in my application code, without changing py.log itself. Right. > Are you thinking about adding more consumer objects such as syslog, > email, Win Events? If so, I can give some of them a shot. syslog and win-events would probably be interesting whereas email has IMO too custom needs to warrant caring for it in the py lib at the moment. For syslog, i think it is easiest and most flexibel to just offer a py.log.Syslogger class: class Syslog(...): def __init__(self, facility, severity, prefix=""): ... def __call__(self, msg): # send to syslog daemon, take minimal code from # logging module and then in application code one can do: py.log.subscribe("* error", Syslog("user", "error", "myprog: "), consume=False) so that all messages with an 'error' keyword in it would show up in syslog. This is an application of the 'subscribe' extension idea above in that you can a) have '*' meaning an arbitrary number of keywords b) consume=False means that this subscription will not consume the message and other possible subscribers should still get invoked for the same message > I'm also thinking about advertising py.log a little on my blog. Or > should I refrain until the py release? Let me know what you think. That would be great. Personally, i feel that the exact API and especially the dispatching mechanisms will change a bit, though, considering the open questions from the above 'subscribe' idea. So it depends a bit. If you plan to introduce py.log then it is probably too early. If you want to discuss and mention the ideas including examples a bit, it's a perfect time. cheers, holger From grig at gheorghiu.net Tue Jun 21 19:46:28 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 21 Jun 2005 10:46:28 -0700 (PDT) Subject: [py-dev] What's next for py.log? In-Reply-To: <20050621172230.GR21207@solar.trillke.net> Message-ID: <20050621174628.80183.qmail@web54510.mail.yahoo.com> --- holger krekel wrote: > Hi Grig, > > On Tue, Jun 21, 2005 at 07:54 -0700, Grig Gheorghiu wrote: > > Do you think the core consumer and producer code will remain as is > for > > py.log? > > I currently guess so, yes. I still feel there will be some > re-thinking > after more people have used it more. Especially i think that the > dispatching rules (how keywords are connected to consumers) will > have to become more flexible at some point. Hum, coming to think > of it we might want to change 'setconsumer' into 'subscribe' > and offer a way to specify if a subscriber will consume a message > or just look at it. This may also help with your multilogger > use case. (Also see below for an example). Maybe set_subscriber? Because the method belongs to the Producer class, and the consumers are the ones that subscribe, right? > > Are you thinking about adding more consumer objects such as syslog, > > email, Win Events? If so, I can give some of them a shot. > > syslog and win-events would probably be interesting whereas > email has IMO too custom needs to warrant caring for it in the > py lib at the moment. For syslog, i think it is easiest and most > flexibel to just offer a py.log.Syslogger class: > > class Syslog(...): > def __init__(self, facility, severity, prefix=""): > ... > def __call__(self, msg): > # send to syslog daemon, take minimal code from > # logging module > > and then in application code one can do: > > py.log.subscribe("* error", > Syslog("user", "error", "myprog: "), > consume=False) > > so that all messages with an 'error' keyword in it would show up > in syslog. This is an application of the 'subscribe' extension idea > above in that you can > > a) have '*' meaning an arbitrary number of keywords > > b) consume=False means that this subscription will not consume > the message and other possible subscribers should still get > invoked for the same message I like the idea of having a consume flag. This will indeed make life easier for me in regards to my multi-logger use case. > > > I'm also thinking about advertising py.log a little on my blog. Or > > should I refrain until the py release? Let me know what you think. > > That would be great. Personally, i feel that the exact API and > especially the dispatching mechanisms will change a bit, though, > considering the open questions from the above 'subscribe' idea. > So it depends a bit. If you plan to introduce py.log then it is > probably too early. If you want to discuss and mention the ideas > including examples a bit, it's a perfect time. OK, will do. I'll talk about keyword-based logging and give some examples, with strong caveats that the syntax might change. Thanks, Grig From grig at gheorghiu.net Tue Jun 21 19:57:55 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 21 Jun 2005 10:57:55 -0700 (PDT) Subject: [py-dev] What's next for py.log? In-Reply-To: <20050621172230.GR21207@solar.trillke.net> Message-ID: <20050621175755.94896.qmail@web54501.mail.yahoo.com> --- holger krekel wrote: > Hi Grig, > > On Tue, Jun 21, 2005 at 07:54 -0700, Grig Gheorghiu wrote: > > Do you think the core consumer and producer code will remain as is > for > > py.log? > > I currently guess so, yes. I still feel there will be some > re-thinking > after more people have used it more. Especially i think that the > dispatching rules (how keywords are connected to consumers) will > have to become more flexible at some point. Hum, coming to think > of it we might want to change 'setconsumer' into 'subscribe' > and offer a way to specify if a subscriber will consume a message > or just look at it. This may also help with your multilogger > use case. (Also see below for an example). > > > It seems to work pretty well. I might still sub-class the > > Producer class for my multiple-keyword needs -- but that can easily > be > > done in my application code, without changing py.log itself. > > Right. > > > Are you thinking about adding more consumer objects such as syslog, > > email, Win Events? If so, I can give some of them a shot. > > syslog and win-events would probably be interesting whereas > email has IMO too custom needs to warrant caring for it in the > py lib at the moment. For syslog, i think it is easiest and most > flexibel to just offer a py.log.Syslogger class: > > class Syslog(...): > def __init__(self, facility, severity, prefix=""): > ... > def __call__(self, msg): > # send to syslog daemon, take minimal code from > # logging module > > and then in application code one can do: > > py.log.subscribe("* error", > Syslog("user", "error", "myprog: "), > consume=False) > > so that all messages with an 'error' keyword in it would show up > in syslog. This is an application of the 'subscribe' extension idea > above in that you can > > a) have '*' meaning an arbitrary number of keywords > > b) consume=False means that this subscription will not consume > the message and other possible subscribers should still get > invoked for the same message > One more thing: I found it necessary in some of my application code to set a consumer function that takes more than one argument. So I ended up doing something like this: def __call__(self, *args, **kwargs): func = self._getconsumer(self.keywords): if func is not None: return func(self.Message(self.keywords, args), **kwargs) Maybe we can add this to the current svn version. In my case, the function was a http POST function that took the message as the first argument and the CGI script to post to as the second argument. Grig From grig at gheorghiu.net Wed Jun 22 01:20:55 2005 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Tue, 21 Jun 2005 16:20:55 -0700 (PDT) Subject: [py-dev] What's next for py.log? In-Reply-To: <20050621172230.GR21207@solar.trillke.net> Message-ID: <20050621232055.30703.qmail@web54502.mail.yahoo.com> --- holger krekel wrote: > Hi Grig, > > On Tue, Jun 21, 2005 at 07:54 -0700, Grig Gheorghiu wrote: > > I'm also thinking about advertising py.log a little on my blog. Or > > should I refrain until the py release? Let me know what you think. > > That would be great. Personally, i feel that the exact API and > especially the dispatching mechanisms will change a bit, though, > considering the open questions from the above 'subscribe' idea. > So it depends a bit. If you plan to introduce py.log then it is > probably too early. If you want to discuss and mention the ideas > including examples a bit, it's a perfect time. > > cheers, > > holger See Grig From rasky at develer.com Fri Jun 24 17:32:55 2005 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 24 Jun 2005 15:32:55 -0000 Subject: [py-dev] py.test and expected failures Message-ID: <023c01c578d1$f9f04be0$bf03030a@trilan> Hello, Is there any plan to add "expected failure" support to py.test? Something like: @py.test.xfail def test_foo(): .... py.test would then silently skip over the failure, and will instead cry as soon as the test starts working again ("unexpected success"). Thanks! -- Giovanni Bajo