From SridharR at activestate.com  Tue Jul  7 01:20:24 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Mon, 06 Jul 2009 16:20:24 -0700
Subject: [py-dev] stdout encoding is set to None
Message-ID: <op.uwn5sad6brrvlq@double>

I noticed that from within py.test .. `sys.stdout.encoding` is set to  
None. This makes it impossible to print non-ascii unicode strings. To  
workaround this error, I wrapped the stdout stream using codecs.getwriter  
as explained in the PrintFails[1] python wiki page. Here's my snippet:

	def _message(self, msg, *args):
	    with _good_terminal_stream() as stream:
	        msg = u'[ui] {0}'.format(msg)
	        print >>stream, msg % args

	@contextmanager
	def _good_terminal_stream():
	    """Return a proper terminal stream"""
	    preferredencoding = locale.getpreferredencoding()
	    assert preferredencoding not in (None, 'ascii'), \
		'improper locale encoding: {0}'.format(
		preferredencoding)
	    stream = codecs.getwriter(preferredencoding)(sys.stdout)
	    try:
		yield stream
	    finally:
		stream.close()

However, this lead to py.test internal error:

	    def done(self):
		""" unpatch and clean up, returns the self.tmpfile (file object)
		    """
		os.dup2(self._savefd, self.targetfd)
		self.unsetfiles()
		os.close(self._savefd)
	>       self.tmpfile.seek(0)
	E       ValueError: I/O operation on closed file

	/home/sridharr/as/pypm-txtui/eggs/py-1.0.0b7-py2.6.egg/py/io/fdcapture.py:39:  
ValueError

I've made available the entire log file here:  
http://files.getdropbox.com/u/87045/tmp/pytesterror

-srid

*****
[1] http://wiki.python.org/moin/PrintFails


From fdg001 at gmx.net  Tue Jul  7 11:30:17 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Tue, 07 Jul 2009 10:30:17 +0100
Subject: [py-dev] using docstrings for test descriptions
Message-ID: <4A5315A9.5030401@gmx.net>

Hello all,

Holger's presentations at EuroPython last week convinced me that I 
should finally start migrating to py.test.

There's only one thing I'm missing: I'm used to providing fairly 
detailed descriptions for individual tests ("foo returns bar if baz").
While one could use the assert statement's second argument for this, 
that seems cumbersome and has some undesired side-effects (e.g. 
reporting "E    AssertionError: <description>" instead of displaying the 
respective values).
Docstrings appear to be the obvious (and pythonic) solution.

So I imagine a plugin could add docstring support, which might result in 
output as described here:
     http://gist.github.com/141977
(Being new to py.test, it is possible my expectations are misguided - 
don't hesitate to point out where that's the case.)

I'd be willing to look into how this might be implemented; it shouldn't 
be too hard (e.g. using a decorator), but some pointers on how to get 
started would be appreciated.


Thanks,

Fred


From holger at merlinux.eu  Tue Jul  7 12:11:39 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 7 Jul 2009 12:11:39 +0200
Subject: [py-dev] stdout encoding is set to None
In-Reply-To: <op.uwn5sad6brrvlq@double>
References: <op.uwn5sad6brrvlq@double>
Message-ID: <20090707101138.GJ7437@trillke.net>

Hi Sridhar, 

indeed, py.test by default redirects the stdout/stderr file
descriptors to a file, see py/io/fdcapture.py.  This leads to
the encoding error.  I guess the redirection file should
better be opened with the "preferredencoding" similar to what
you do.  Can you imagine submitting an according test and patch? 

Your context manager causes problems, btw, because it closes 
the redirection-stream which it shouldn't do.  

best,

holger

On Mon, Jul 06, 2009 at 16:20 -0700, Sridhar Ratnakumar wrote:
> I noticed that from within py.test .. `sys.stdout.encoding` is set to  
> None. This makes it impossible to print non-ascii unicode strings. To  
> workaround this error, I wrapped the stdout stream using codecs.getwriter  
> as explained in the PrintFails[1] python wiki page. Here's my snippet:
> 
> 	def _message(self, msg, *args):
> 	    with _good_terminal_stream() as stream:
> 	        msg = u'[ui] {0}'.format(msg)
> 	        print >>stream, msg % args
> 
> 	@contextmanager
> 	def _good_terminal_stream():
> 	    """Return a proper terminal stream"""
> 	    preferredencoding = locale.getpreferredencoding()
> 	    assert preferredencoding not in (None, 'ascii'), \
> 		'improper locale encoding: {0}'.format(
> 		preferredencoding)
> 	    stream = codecs.getwriter(preferredencoding)(sys.stdout)
> 	    try:
> 		yield stream
> 	    finally:
> 		stream.close()
> 
> However, this lead to py.test internal error:
> 
> 	    def done(self):
> 		""" unpatch and clean up, returns the self.tmpfile (file object)
> 		    """
> 		os.dup2(self._savefd, self.targetfd)
> 		self.unsetfiles()
> 		os.close(self._savefd)
> 	>       self.tmpfile.seek(0)
> 	E       ValueError: I/O operation on closed file
> 
> 	/home/sridharr/as/pypm-txtui/eggs/py-1.0.0b7-py2.6.egg/py/io/fdcapture.py:39:  
> ValueError
> 
> I've made available the entire log file here:  
> http://files.getdropbox.com/u/87045/tmp/pytesterror
> 
> -srid
> 
> *****
> [1] http://wiki.python.org/moin/PrintFails
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From holger at merlinux.eu  Tue Jul  7 12:28:53 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 7 Jul 2009 12:28:53 +0200
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A5315A9.5030401@gmx.net>
References: <4A5315A9.5030401@gmx.net>
Message-ID: <20090707102853.GL7437@trillke.net>

Hi Frederik, 

On Tue, Jul 07, 2009 at 10:30 +0100, Frederik Dohr wrote:
> Hello all,
> 
> Holger's presentations at EuroPython last week convinced me that I 
> should finally start migrating to py.test.

nice to hear :) 
 
> There's only one thing I'm missing: I'm used to providing fairly 
> detailed descriptions for individual tests ("foo returns bar if baz").
> While one could use the assert statement's second argument for this, 
> that seems cumbersome and has some undesired side-effects (e.g. 
> reporting "E    AssertionError: <description>" instead of displaying the 
> respective values).
> Docstrings appear to be the obvious (and pythonic) solution.

makes sense i think. 

> So I imagine a plugin could add docstring support, which might result in 
> output as described here:
>      http://gist.github.com/141977
> (Being new to py.test, it is possible my expectations are misguided - 
> don't hesitate to point out where that's the case.)

Python's unittest.py uses the docstrings to substitute the test
function name with the docstring.  Imitating this "py.test --verbose" 
could look like this: 

    lorem ipsum: PASS 
    dolor sit amet: PASS 
    consectetur adipisic...liqua: FAIL 

The failure tracebacks probably don't need to do anything special
as they anyway report the test function source code including
the docstring usually. 

makes senses?  Anybody else has opinions? 

> I'd be willing to look into how this might be implemented; it shouldn't 
> be too hard (e.g. using a decorator), but some pointers on how to get 
> started would be appreciated.

I think you can just patch py.test proper, more specifically 
py/test/plugin/pytest_terminal.py which is the single file
containing all the terminal reporting and also includes tests.
Please feel free to submit a patch.  

best,
holger


From fdg001 at gmx.net  Wed Jul  8 21:12:17 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Wed, 08 Jul 2009 20:12:17 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <20090707102853.GL7437@trillke.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
Message-ID: <4A54EF91.9060004@gmx.net>

Thanks for the kind response!

>>      http://gist.github.com/141977
> 
> Python's unittest.py uses the docstrings to substitute the test
> function name with the docstring.  Imitating this "py.test --verbose" 
> could look like this: 
>     lorem ipsum: PASS 

I had intentionally avoided that in my proposal, as I didn't want to 
break the standard (machine-readable) "file:line:message" format.
Not entirely sure whether that's a valid concern though.

> I think you can just patch py.test proper, more specifically 
> py/test/plugin/pytest_terminal.py

Thanks, that looks promising.
Might take me a week or two to come up with something presentable 
(things are a little busy, in part due to the post-conference backlog), 
but I'll definitely give it a shot and report back here.


-- F.


From fdg001 at gmx.net  Wed Jul  8 23:05:30 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Wed, 08 Jul 2009 22:05:30 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A54EF91.9060004@gmx.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
	<4A54EF91.9060004@gmx.net>
Message-ID: <4A550A1A.8080803@gmx.net>

> Thanks, that looks promising.
> Might take me a week or two to come up with something presentable

I might have spoken too soon.
After some naive hacking, I now have a simple proof of concept. In the 
spirit of "release early, release often", here's the patch:
     http://gist.github.com/141977#LID1
(also attached below)
Plus some sample output:
     http://gist.github.com/141977#LID49

While this is certainly not release-ready, I'd appreciate feedback on 
whether it's the right approach.


Thanks,

Fred


Index: py/test/plugin/pytest_terminal.py
===================================================================
--- py/test/plugin/pytest_terminal.py	(revision 66152)
+++ py/test/plugin/pytest_terminal.py	(working copy)
@@ -284,6 +284,11 @@
          except AttributeError:
              pass
          reportinfo = item.config.hook.pytest_report_iteminfo(item=item)
+        # add docstring to message
+        docstring = item.obj.__doc__ # TODO: shorten
+        reportinfo = list(reportinfo) # mutable type req'd -- XXX: hacky
+        reportinfo[-1] = "%s: %s" % (reportinfo[-1], docstring)
+        reportinfo = tuple(reportinfo)
          # cache on item
          item.__reportinfo = reportinfo
          return reportinfo


From SridharR at activestate.com  Thu Jul  9 01:19:52 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 08 Jul 2009 16:19:52 -0700
Subject: [py-dev] py.test and logging issue: ValueError: I/O operation on
	closed file
Message-ID: <op.uwru3etnbrrvlq@double>

holger and others,

do you have any comments to add to this bug?

   http://bugs.python.org/issue6333

-srid


From holger at merlinux.eu  Thu Jul  9 13:20:57 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 9 Jul 2009 13:20:57 +0200
Subject: [py-dev] py.test and logging issue: ValueError: I/O
	operation	on closed file
In-Reply-To: <op.uwru3etnbrrvlq@double>
References: <op.uwru3etnbrrvlq@double>
Message-ID: <20090709112056.GO7437@trillke.net>

On Wed, Jul 08, 2009 at 16:19 -0700, Sridhar Ratnakumar wrote:
> holger and others,
> 
> do you have any comments to add to this bug?
> 
>    http://bugs.python.org/issue6333

i added a comment there, basically suggesting to follow your suggestion.  

As py.test should nicely work on many CPython versions i also
did a general workaround in py.test proper, see here:

    http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/

I intend to release a 1.0.0b8 tomorrow with this 
and some other fixes.  

best & thanks,
holger


From holger at merlinux.eu  Thu Jul  9 13:47:09 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 9 Jul 2009 13:47:09 +0200
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <4A550A1A.8080803@gmx.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
	<4A54EF91.9060004@gmx.net> <4A550A1A.8080803@gmx.net>
Message-ID: <20090709114709.GP7437@trillke.net>

On Wed, Jul 08, 2009 at 22:05 +0100, Frederik Dohr wrote:
>> Thanks, that looks promising.
>> Might take me a week or two to come up with something presentable
>
> I might have spoken too soon.
> After some naive hacking, I now have a simple proof of concept. In the  
> spirit of "release early, release often", here's the patch:

good idea :) 

>     http://gist.github.com/141977#LID1
> (also attached below)
> Plus some sample output:
>     http://gist.github.com/141977#LID49
>
> While this is certainly not release-ready, I'd appreciate feedback on  
> whether it's the right approach.

sure, here is some quick feedback.

a) Your change presumes that all test items are python test
   functions which is not true.  With 

      if isinstance(item, py.test.collect.Function) 

   you can check that you are dealing with python test
   functions. 

b) it's probably better to do the reporting tweak in 
   "def _reportinfoline" which deals with
   exactly presenting the verbose line. 

c) please add a test to the pytest_terminal.py file 
   and i recommend to rather copy+modify the "test_verbose" 
   one from py/test/testing/acceptance_test.py 
   you can simplify it of course to just test output
   for a single test function.  

best,
holger

>
>
> Index: py/test/plugin/pytest_terminal.py
> ===================================================================
> --- py/test/plugin/pytest_terminal.py	(revision 66152)
> +++ py/test/plugin/pytest_terminal.py	(working copy)
> @@ -284,6 +284,11 @@
>          except AttributeError:
>              pass
>          reportinfo = item.config.hook.pytest_report_iteminfo(item=item)
> +        # add docstring to message
> +        docstring = item.obj.__doc__ # TODO: shorten
> +        reportinfo = list(reportinfo) # mutable type req'd -- XXX: hacky
> +        reportinfo[-1] = "%s: %s" % (reportinfo[-1], docstring)
> +        reportinfo = tuple(reportinfo)
>          # cache on item
>          item.__reportinfo = reportinfo
>          return reportinfo
>

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From SridharR at activestate.com  Thu Jul  9 22:03:01 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Thu, 09 Jul 2009 13:03:01 -0700
Subject: [py-dev] py.test and logging issue: ValueError: I/O operation
 on closed file
In-Reply-To: <20090709112056.GO7437@trillke.net>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
Message-ID: <op.uwtgnbnhbrrvlq@double>

On Thu, 09 Jul 2009 04:20:57 -0700, holger krekel <holger at merlinux.eu>  
wrote:

> As py.test should nicely work on many CPython versions i also
> did a general workaround in py.test proper, see here:
>    http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/

This will suppress *all* exceptions thrown by the logging module, no?

-srid


From py-dev at tolomea.com  Fri Jul 10 07:44:53 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 15:44:53 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote threads
Message-ID: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>

I have execnet's stdout and stderr redirecting working and it does a
fine job for the initial thread, but I'm also creating a bunch of
helper threads on the remote end of the connection and I can't figure
out how to get the stdout and stderr for them redirected.

Below is an example of what I thought might work but doesn't.

import py
import sys

code = """
def func():
   print 'hello world'

import threading
t=threading.Thread(target=func)
t.start()
t.join()
"""

gateway = py.execnet.SshGateway("localhost")
channel = gateway.remote_exec(code, sys.stdout.write)
channel.waitclose()


Regards Gordon


From holger at merlinux.eu  Fri Jul 10 09:36:54 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 09:36:54 +0200
Subject: [py-dev] py.test and logging issue: ValueError: I/O
	operation	on closed file
In-Reply-To: <op.uwtgnbnhbrrvlq@double>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
	<op.uwtgnbnhbrrvlq@double>
Message-ID: <20090710073654.GQ7437@trillke.net>

On Thu, Jul 09, 2009 at 13:03 -0700, Sridhar Ratnakumar wrote:
> On Thu, 09 Jul 2009 04:20:57 -0700, holger krekel <holger at merlinux.eu>  
> wrote:
>
>> As py.test should nicely work on many CPython versions i also
>> did a general workaround in py.test proper, see here:
>>    http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/
>
> This will suppress *all* exceptions thrown by the logging module, no?

no.  It is only surpressed when the whole testing process is
about to end - so should not affect any test or teardown
functions, just the atexit-handling of logging basically. 

holger


From holger at merlinux.eu  Fri Jul 10 09:58:25 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 09:58:25 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
Message-ID: <20090710075825.GR7437@trillke.net>

On Fri, Jul 10, 2009 at 15:44 +1000, Gordon Wrigley wrote:
> I have execnet's stdout and stderr redirecting working and it does a
> fine job for the initial thread, but I'm also creating a bunch of
> helper threads on the remote end of the connection and I can't figure
> out how to get the stdout and stderr for them redirected.
> 
> Below is an example of what I thought might work but doesn't.
> 
> import py
> import sys
> 
> code = """
> def func():
>    print 'hello world'
> 
> import threading
> t=threading.Thread(target=func)
> t.start()
> t.join()
> """
> 
> gateway = py.execnet.SshGateway("localhost")
> channel = gateway.remote_exec(code, sys.stdout.write)
> channel.waitclose()

stdout redirection is thread-aware.  This is because 
when calling 

    gateway.remote_init_threads(num=3)

you would have three execution threads on the remote
side and stdout redirection would separate the three
execution's stdout redirections.   Can you use
maybe make use of this threading handling?  If not
then maybe we can think of introducing a way 
to configure the exact behaviour but it would complicate
the API a bit which i'd like to avoid. 

best,
holger


From py-dev at tolomea.com  Fri Jul 10 12:21:27 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 20:21:27 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote
	threads
In-Reply-To: <20090710075825.GR7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
Message-ID: <f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>

>> I have execnet's stdout and stderr redirecting working and it does a
>> fine job for the initial thread, but I'm also creating a bunch of
>> helper threads on the remote end of the connection and I can't figure
>> out how to get the stdout and stderr for them redirected.

> stdout redirection is thread-aware. ?This is because
> when calling
>
> ? ?gateway.remote_init_threads(num=3)
>
> you would have three execution threads on the remote
> side and stdout redirection would separate the three
> execution's stdout redirections. ? Can you use
> maybe make use of this threading handling? ?If not
> then maybe we can think of introducing a way
> to configure the exact behaviour but it would complicate
> the API a bit which i'd like to avoid.

I need to create the threads at the remote end and on demand so I
don't think I'm going to be able to use execnets threading support.
My usage pattern is rather bizarre, maybe a better way to approach
this would be to build my own stdout / stderr redirection on top of
execnet sockets.

Gordon


From holger at merlinux.eu  Fri Jul 10 12:53:45 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 12:53:45 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
Message-ID: <20090710105345.GS7437@trillke.net>

On Fri, Jul 10, 2009 at 20:21 +1000, Gordon Wrigley wrote:
> >> I have execnet's stdout and stderr redirecting working and it does a
> >> fine job for the initial thread, but I'm also creating a bunch of
> >> helper threads on the remote end of the connection and I can't figure
> >> out how to get the stdout and stderr for them redirected.
> 
> > stdout redirection is thread-aware. ?This is because
> > when calling
> >
> > ? ?gateway.remote_init_threads(num=3)
> >
> > you would have three execution threads on the remote
> > side and stdout redirection would separate the three
> > execution's stdout redirections. ? Can you use
> > maybe make use of this threading handling? ?If not
> > then maybe we can think of introducing a way
> > to configure the exact behaviour but it would complicate
> > the API a bit which i'd like to avoid.
> 
> I need to create the threads at the remote end and on demand so I
> don't think I'm going to be able to use execnets threading support.
> My usage pattern is rather bizarre, maybe a better way to approach
> this would be to build my own stdout / stderr redirection on top of
> execnet sockets.

yes, might make sense then.  I attached a working example
how to do general output redirection, maybe that helps you. 

Maybe also simplifying/separating the API makes sense, i.e. having a 

    remote_redirect_output(..., perthread=False)

and a simple 

    remote_exec(source) 

call.  That would cater your use case as well, i guess. 

best,
holger
 _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From holger at merlinux.eu  Fri Jul 10 13:05:01 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 13:05:01 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <20090710105345.GS7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
Message-ID: <20090710110501.GT7437@trillke.net>

On Fri, Jul 10, 2009 at 12:53 +0200, holger krekel wrote:
> On Fri, Jul 10, 2009 at 20:21 +1000, Gordon Wrigley wrote:
> > >> I have execnet's stdout and stderr redirecting working and it does a
> > >> fine job for the initial thread, but I'm also creating a bunch of
> > >> helper threads on the remote end of the connection and I can't figure
> > >> out how to get the stdout and stderr for them redirected.
> > 
> > > stdout redirection is thread-aware. ?This is because
> > > when calling
> > >
> > > ? ?gateway.remote_init_threads(num=3)
> > >
> > > you would have three execution threads on the remote
> > > side and stdout redirection would separate the three
> > > execution's stdout redirections. ? Can you use
> > > maybe make use of this threading handling? ?If not
> > > then maybe we can think of introducing a way
> > > to configure the exact behaviour but it would complicate
> > > the API a bit which i'd like to avoid.
> > 
> > I need to create the threads at the remote end and on demand so I
> > don't think I'm going to be able to use execnets threading support.
> > My usage pattern is rather bizarre, maybe a better way to approach
> > this would be to build my own stdout / stderr redirection on top of
> > execnet sockets.
> 
> yes, might make sense then.  I attached a working example
> how to do general output redirection, maybe that helps you. 

now attached for real - let me know if there are problems with it.

holger
-------------- next part --------------
A non-text attachment was scrubbed...
Name: redirect_remote_output.py
Type: text/x-python
Size: 660 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20090710/a257fb1a/attachment.py>

From py-dev at tolomea.com  Fri Jul 10 13:07:50 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 21:07:50 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote
	threads
In-Reply-To: <20090710110501.GT7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
	<20090710110501.GT7437@trillke.net>
Message-ID: <f2d961430907100407x1c7a7372r46b935294ea7d5bd@mail.gmail.com>

btw, that whole sending a channel over a channel thing is beautiful
and has proved very helpful

G

On Fri, Jul 10, 2009 at 9:05 PM, holger krekel<holger at merlinux.eu> wrote:
> On Fri, Jul 10, 2009 at 12:53 +0200, holger krekel wrote:
>> On Fri, Jul 10, 2009 at 20:21 +1000, Gordon Wrigley wrote:
>> > >> I have execnet's stdout and stderr redirecting working and it does a
>> > >> fine job for the initial thread, but I'm also creating a bunch of
>> > >> helper threads on the remote end of the connection and I can't figure
>> > >> out how to get the stdout and stderr for them redirected.
>> >
>> > > stdout redirection is thread-aware. ?This is because
>> > > when calling
>> > >
>> > > ? ?gateway.remote_init_threads(num=3)
>> > >
>> > > you would have three execution threads on the remote
>> > > side and stdout redirection would separate the three
>> > > execution's stdout redirections. ? Can you use
>> > > maybe make use of this threading handling? ?If not
>> > > then maybe we can think of introducing a way
>> > > to configure the exact behaviour but it would complicate
>> > > the API a bit which i'd like to avoid.
>> >
>> > I need to create the threads at the remote end and on demand so I
>> > don't think I'm going to be able to use execnets threading support.
>> > My usage pattern is rather bizarre, maybe a better way to approach
>> > this would be to build my own stdout / stderr redirection on top of
>> > execnet sockets.
>>
>> yes, might make sense then. ?I attached a working example
>> how to do general output redirection, maybe that helps you.
>
> now attached for real - let me know if there are problems with it.
>
> holger
>


From py-dev at tolomea.com  Fri Jul 10 13:16:30 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Fri, 10 Jul 2009 21:16:30 +1000
Subject: [py-dev] execnet: redirecting stdout and stderr from remote
	threads
In-Reply-To: <20090710105345.GS7437@trillke.net>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
Message-ID: <f2d961430907100416g4e199446u63910500ef635a04@mail.gmail.com>

> yes, might make sense then. ?I attached a working example
> how to do general output redirection, maybe that helps you.

I gave the example a look over and it all makes sense and is nice and
straight forward. So far I've maintained a 1 to 1 relation ship
between threads and channels, in the context of the example are there
any threading implications to having multiple threads calling into the
channel file that has been assigned to sys.stdout?


From holger at merlinux.eu  Fri Jul 10 13:23:52 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Jul 2009 13:23:52 +0200
Subject: [py-dev] execnet: redirecting stdout and stderr from
	remote	threads
In-Reply-To: <f2d961430907100416g4e199446u63910500ef635a04@mail.gmail.com>
References: <f2d961430907092244m2565af5eo68ee2f20d9356926@mail.gmail.com>
	<20090710075825.GR7437@trillke.net>
	<f2d961430907100321l5eef744dl9abb6a488f9c98c1@mail.gmail.com>
	<20090710105345.GS7437@trillke.net>
	<f2d961430907100416g4e199446u63910500ef635a04@mail.gmail.com>
Message-ID: <20090710112352.GU7437@trillke.net>

On Fri, Jul 10, 2009 at 21:16 +1000, Gordon Wrigley wrote:
> > yes, might make sense then. ?I attached a working example
> > how to do general output redirection, maybe that helps you.
> 
> I gave the example a look over and it all makes sense and is nice and
> straight forward. So far I've maintained a 1 to 1 relation ship
> between threads and channels, in the context of the example are there
> any threading implications to having multiple threads calling into the
> channel file that has been assigned to sys.stdout?

no problems i am aware off.  It *should* be safe to have
multiple threads using channel.send() concurrently. 
It depends on write()s to pipes (or sockets) being 
atomic eventually which i think they are.  

One issue that can pop up is tearing down the whole
machinery cleanly - that can sometimes be more
involved and i believe execnet still might need 
some more work there. 

holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From fdg001 at gmx.net  Sat Jul 11 09:42:02 2009
From: fdg001 at gmx.net (Frederik Dohr)
Date: Sat, 11 Jul 2009 08:42:02 +0100
Subject: [py-dev] using docstrings for test descriptions
In-Reply-To: <20090709114709.GP7437@trillke.net>
References: <4A5315A9.5030401@gmx.net> <20090707102853.GL7437@trillke.net>
	<4A54EF91.9060004@gmx.net> <4A550A1A.8080803@gmx.net>
	<20090709114709.GP7437@trillke.net>
Message-ID: <4A58424A.3050400@gmx.net>

> sure, here is some quick feedback [...]

Thanks Holger, much appreciated.
Those suggestions are very helpful; I'll see about incorporating them soon.


-- F.


From py-dev at tolomea.com  Mon Jul 13 03:40:10 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Mon, 13 Jul 2009 11:40:10 +1000
Subject: [py-dev] execnet: shutdown issues in a heavily threaded environment
Message-ID: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>

> One issue that can pop up is tearing down the whole
> machinery cleanly - that can sometimes be more
> involved and i believe execnet still might need
> some more work there.

On that note can you shed any light on this message:

Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
open>> ignored

Currently I'm only seeing them on my test machine, but they are
occurring very frequently, they drop out at the end of the py.test
invocation, I will paste the tail end of the py.test output below.

I'm not sure exactly what is causing this, naively it seems to
indicate a circular chain of references involving one or more
channels. However immediately prior to the conclusion of each test
exit gc.garbage is empty and there are none of my objects in
gc.get_objects.

Also this only started when I reworked our remote invocation system
recently, the most notable change in the new system is that it makes
more use of threads and particularly it has daemon threads waiting on
channel receive calls.


lab at okum:~$ uname -a
Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
2009 i686 GNU/Linux
lab at okum:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b1'


============================================== 35 passed in 231.71
seconds ===============================================
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
method Thread.__bootstrap of <Thread(receiver, stopped daemon
1177635728)>>
<bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
38, in apport_excepthook
    from apport.packaging_impl import impl as packaging
ImportError: No module named apport.packaging_impl

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
    (self.name, _format_exc()))
TypeError: 'NoneType' object is not callable
 ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
open>> ignored


From pedronis at openend.se  Mon Jul 13 19:39:06 2009
From: pedronis at openend.se (Samuele Pedroni)
Date: Mon, 13 Jul 2009 19:39:06 +0200
Subject: [py-dev] ANN: oejskit 0.8.5 JavaScript in-browser testing and
 utility kit with py.test plugin
Message-ID: <4A5B713A.9010704@openend.se>

jskit contains infrastructure and in particular a py.test plugin to
enable running unit tests for JavaScript code inside browsers. The
plugin requires py.test 1.0

The approach also enables to write integration tests such that the
JavaScript code is tested against server-side Python code mocked as
necessary. Any server-side framework that can already be exposed through
WSGI can play.

More information and downloading at:

http://pypi.python.org/pypi/oejskit

including documentation and the talk I gave at Europython.

jskit was initially developed by Open End AB and is released under the
MIT license.

In various incarnations it has been in use and useful at Open End for
more than a year, we are quite happy to share it.

Samuele Pedroni for Open End




From holger at merlinux.eu  Tue Jul 14 11:10:04 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 14 Jul 2009 11:10:04 +0200
Subject: [py-dev] execnet: shutdown issues in a heavily
	threaded	environment
In-Reply-To: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
References: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
Message-ID: <20090714091004.GY7437@trillke.net>

Hi Gordon,

On Mon, Jul 13, 2009 at 11:40 +1000, Gordon Wrigley wrote:
> > One issue that can pop up is tearing down the whole
> > machinery cleanly - that can sometimes be more
> > involved and i believe execnet still might need
> > some more work there.
> 
> On that note can you shed any light on this message:
> 
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
> open>> ignored
>
> Currently I'm only seeing them on my test machine, but they are
> occurring very frequently, they drop out at the end of the py.test
> invocation, I will paste the tail end of the py.test output below.

> I'm not sure exactly what is causing this, naively it seems to
> indicate a circular chain of references involving one or more
> channels. However immediately prior to the conclusion of each test
> exit gc.garbage is empty and there are none of my objects in
> gc.get_objects.

Yes, some cycle is probably there.  Your data point hints that
there is some cycle in the execnet code itself.  

> Also this only started when I reworked our remote invocation system
> recently, the most notable change in the new system is that it makes
> more use of threads and particularly it has daemon threads waiting on
> channel receive calls.

Hum, that might be related - GC-finalization issues in
MT-environments are not easy to debug :( 
Might also be python-version dependent.  What are 
the differences between your test and production system? 

I am a bit at a loss on how to best proceed at the moment. 
I think that adding more debugging information
to execnet and systematically implementing and
checking scenarios is due - but quite a bit of effort.  
Could you maybe try coming up with an self-contained 
example test-script leading to these messages? 

On a sidenote, probably around 60% of the execnet core
programming effort revolve around teardown/finalization issues
- i wonder if it would be better to avoid usage of __del__
alltogether, only have a process-wide atexit handler and
otherwise recommend explicit calling of
gateway.exit()/channel.close methods for proper resource
handling.  

best,

holger
 
> lab at okum:~$ uname -a
> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
> 2009 i686 GNU/Linux
> lab at okum:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import py
> >>> py.version
> '1.0.0b1'
> 
> 
> ============================================== 35 passed in 231.71
> seconds ===============================================
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
> method Thread.__bootstrap of <Thread(receiver, stopped daemon
> 1177635728)>>
> <bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
> 38, in apport_excepthook
>     from apport.packaging_impl import impl as packaging
> ImportError: No module named apport.packaging_impl
> 
> Original exception was:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
>     self.__bootstrap_inner()
>   File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
>     (self.name, _format_exc()))
> TypeError: 'NoneType' object is not callable
>  ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> Exception AttributeError: "'NoneType' object has no attribute
> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
> open>> ignored
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From py-dev at tolomea.com  Tue Jul 14 11:56:37 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Tue, 14 Jul 2009 19:56:37 +1000
Subject: [py-dev] execnet: shutdown issues in a heavily threaded
	environment
In-Reply-To: <20090714091004.GY7437@trillke.net>
References: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
	<20090714091004.GY7437@trillke.net>
Message-ID: <f2d961430907140256l484790d5i5a6500a23a432ec@mail.gmail.com>

Hi Holger

>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
>> open>> ignored

> Yes, some cycle is probably there. ?Your data point hints that
> there is some cycle in the execnet code itself.

A bit of debugging revealed that I didn't really understand how my own
teardown setup was supposed to work and subsequently that it didn't
actually work. I have subsequently improved my teardown process and
this have greatly improved the situation, where I was previously
getting nearly 2 dozen of these per test run I now see only one, I
haven't dug into that one yet, but I'd be willing to bet that it is
also my problem.

>> and particularly it has daemon threads waiting on channel receive calls.

> Hum, that might be related -

It was.

One thing I did notice is that under the ipython shell having a daemon
thread waiting on a channel receive is enough to keep the both the
thread and execnet live and subsequently prevent the VM from exiting.
Strangely this isn't the case under the normal python shell, but still
I think there might be something worth investigating there.

import py
import threading
g=py.execnet.SshGateway("localhost")
c=g.remote_exec("while True:\n pass")
def bob():
    c.receive()
t=threading.Thread(target=bob)
t.daemon=True
t.start()
exit()

Obviously this is a retarded test case, but the only "application"
thread in the system is a daemon so I would naively expect it to shut
down.

> GC-finalization issues in MT-environments are not easy to debug :(

>From recent experience I can confirm that  that is very true.

> Might also be python-version dependent. ?What are
> the differences between your test and production system?

I will check this in the morning.

> I am a bit at a loss on how to best proceed at the moment.
> I think that adding more debugging information
> to execnet and systematically implementing and
> checking scenarios is due - but quite a bit of effort.
> Could you maybe try coming up with an self-contained
> example test-script leading to these messages?

For now I'm happy to consider this my problem and keep picking at it
on my end, although since the VM does come down on both ends of the
ssh link (albeit not always cleanly) this isn't a high priority for
me, just a nuisance. If I do find anything that is definitely execnet
misbehaving I will post it here.

> On a sidenote, probably around 60% of the execnet core
> programming effort revolve around teardown/finalization issues
> - i wonder if it would be better to avoid usage of __del__
> alltogether, only have a process-wide atexit handler and
> otherwise recommend explicit calling of
> gateway.exit()/channel.close methods for proper resource
> handling.

I don't mind either way as long as there is some relatively straight
forward way of getting the whole thing shut down, so that a longer
running system that opens and closes multiple execnet connections can
be written in a way that doesn't accumulate cruft over time.

Regards Gordon


From py-dev at tolomea.com  Wed Jul 15 08:05:13 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Wed, 15 Jul 2009 16:05:13 +1000
Subject: [py-dev] execnet: shutdown issues in a heavily threaded
	environment
In-Reply-To: <20090714091004.GY7437@trillke.net>
References: <f2d961430907121840vad24961r6d398f482cedcced@mail.gmail.com>
	<20090714091004.GY7437@trillke.net>
Message-ID: <f2d961430907142305h33c8c2aeq7877cb0368dfc1be@mail.gmail.com>

>> Also this only started when I reworked our remote invocation system
>> recently, the most notable change in the new system is that it makes
>> more use of threads and particularly it has daemon threads waiting on
>> channel receive calls.

> Might also be python-version dependent.  What are
> the differences between your test and production system?

development machine:

gordonw at gohma:~$ uname -a
Linux gohma 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC
2009 i686 GNU/Linux
gordonw at gohma:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b7'

test machine:

lab at okum:~$ uname -a
Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
2009 i686 GNU/Linux
lab at okum:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b1'

On Tue, Jul 14, 2009 at 7:10 PM, holger krekel<holger at merlinux.eu> wrote:
> Hi Gordon,
>
> On Mon, Jul 13, 2009 at 11:40 +1000, Gordon Wrigley wrote:
>> > One issue that can pop up is tearing down the whole
>> > machinery cleanly - that can sometimes be more
>> > involved and i believe execnet still might need
>> > some more work there.
>>
>> On that note can you shed any light on this message:
>>
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
>> open>> ignored
>>
>> Currently I'm only seeing them on my test machine, but they are
>> occurring very frequently, they drop out at the end of the py.test
>> invocation, I will paste the tail end of the py.test output below.
>
>> I'm not sure exactly what is causing this, naively it seems to
>> indicate a circular chain of references involving one or more
>> channels. However immediately prior to the conclusion of each test
>> exit gc.garbage is empty and there are none of my objects in
>> gc.get_objects.
>
> Yes, some cycle is probably there. ?Your data point hints that
> there is some cycle in the execnet code itself.
>
>> Also this only started when I reworked our remote invocation system
>> recently, the most notable change in the new system is that it makes
>> more use of threads and particularly it has daemon threads waiting on
>> channel receive calls.
>
> Hum, that might be related - GC-finalization issues in
> MT-environments are not easy to debug :(
> Might also be python-version dependent. ?What are
> the differences between your test and production system?
>
> I am a bit at a loss on how to best proceed at the moment.
> I think that adding more debugging information
> to execnet and systematically implementing and
> checking scenarios is due - but quite a bit of effort.
> Could you maybe try coming up with an self-contained
> example test-script leading to these messages?
>
> On a sidenote, probably around 60% of the execnet core
> programming effort revolve around teardown/finalization issues
> - i wonder if it would be better to avoid usage of __del__
> alltogether, only have a process-wide atexit handler and
> otherwise recommend explicit calling of
> gateway.exit()/channel.close methods for proper resource
> handling.
>
> best,
>
> holger
>
>> lab at okum:~$ uname -a
>> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
>> 2009 i686 GNU/Linux
>> lab at okum:~$ python
>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
>> [GCC 4.3.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import py
>> >>> py.version
>> '1.0.0b1'
>>
>>
>> ============================================== 35 passed in 231.71
>> seconds ===============================================
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
>> method Thread.__bootstrap of <Thread(receiver, stopped daemon
>> 1177635728)>>
>> <bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
>> Traceback (most recent call last):
>> ? File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
>> 38, in apport_excepthook
>> ? ? from apport.packaging_impl import impl as packaging
>> ImportError: No module named apport.packaging_impl
>>
>> Original exception was:
>> Traceback (most recent call last):
>> ? File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
>> ? ? self.__bootstrap_inner()
>> ? File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
>> ? ? (self.name, _format_exc()))
>> TypeError: 'NoneType' object is not callable
>> ?ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
>


From py-dev at tolomea.com  Wed Jul 15 11:05:28 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Wed, 15 Jul 2009 19:05:28 +1000
Subject: [py-dev] execnet py.test teardown issue
Message-ID: <f2d961430907150205x7f44f9d4w6d22fb3e9db9d7e9@mail.gmail.com>

Hi Holger

I have produced a minimal reproduction for my one remaining issue.

To reproduce put the following into a file and run it with py.test

import py
a = None
def test_function():
    global a
    a = py.execnet.SshGateway("localhost").remote_exec("pass")

As before the problem only reproduces on my test machine not on my
development machine. On my test machine it produces:
Exception AttributeError: "'NoneType' object has no attribute
'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=1
open>> ignored

Additionally it only reproduces inside py.test.

Now that I know what it is it can be trivially fixed by del'ing the
global variable at the end of the test. I am however curious about the
difference in behavior on the two machines and I would also like to
know if you consider this a problem or if it is just that pushing a
channel into the global namespace is a bad idea. It is probably worth
noting that in the full system what I pushed into the global namespace
wasn't a channel but a utility class that happened to have a reference
to a channel, so it wasn't quite as braindead as it looks.

Gordon

> development machine:
>
> gordonw at gohma:~$ uname -a
> Linux gohma 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC
> 2009 i686 GNU/Linux
> gordonw at gohma:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import py
>>>> py.version
> '1.0.0b7'

> test machine:
>
> lab at okum:~$ uname -a
> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
> 2009 i686 GNU/Linux
> lab at okum:~$ python
> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
> [GCC 4.3.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import py
>>>> py.version
> '1.0.0b1'


From vinjvinj at gmail.com  Wed Jul 22 05:31:25 2009
From: vinjvinj at gmail.com (Vineet Jain)
Date: Tue, 21 Jul 2009 23:31:25 -0400
Subject: [py-dev] Problem with py.lib and heapy
Message-ID: <70a7f84f0907212031k39bb085eg53f6420b24b3153c@mail.gmail.com>

I was trying to use heapy
(http://guppy-pe.sourceforge.net/)<http://guppy-pe.sourceforge.net/>
on
my application (which uses py.log and py.test)  to try and find some memory
leaks. I can't seem to get it to work with py.lib (see exception trace at
the end of the email). Can someone please guide me on how to just install
py.log (and it's minimum requirements)? I tried downloading the tar and
removed all the extra packages in setup.py, however, that failed. Currently
I have py.log commented out while I'm debugging my memory leaks.

hp = hpy()

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/__init__.py",
line 37, in hpy

return r.guppy.heapy.Use

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 45, in __getattr__

return self._share.getattr(self, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 195, in getattr

d = self.getattr2(inter, cache, owner, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 213, in getattr2

x = self.getattr_package(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 261, in getattr_package

x = self.makeModule(x, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 321, in makeModule

return Share(module, self, module.__name__, Clamp)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 184, in __init__

getattr(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 45, in __getattr__

return self._share.getattr(self, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 195, in getattr

d = self.getattr2(inter, cache, owner, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 215, in getattr2

x = self.getattr3(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 283, in getattr3

pa = getattr(pa, at)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 45, in __getattr__

return self._share.getattr(self, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 195, in getattr

d = self.getattr2(inter, cache, owner, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 213, in getattr2

x = self.getattr_package(inter, name)

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
line 250, in getattr_package

x = __import__(self.makeName(name), globals(), locals())

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
line 555, in <module>

prime_builtin_types()

File
"/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
line 546, in prime_builtin_types

for t in mod.__dict__.values():

File "/usr/lib/python2.5/site-packages/py-0.9.1-py2.5.egg/py/initpkg.py",
line 216, in getdict

assert not self.__map__, "%r not empty" % self.__map__

AssertionError: {'greenlet': ('./magic/greenlet.py', 'greenlet')} not empty
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20090721/9da9e671/attachment.html>

From py-dev at tolomea.com  Wed Jul 22 08:16:48 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Wed, 22 Jul 2009 16:16:48 +1000
Subject: [py-dev] py.test and logging issue: ValueError: I/O operation
	on closed file
In-Reply-To: <20090709112056.GO7437@trillke.net>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
Message-ID: <f2d961430907212316l434c36cegaef6da2d92c7aa1d@mail.gmail.com>

When do you expect 1.0.0b8 to be available on easy-install?

I ask because I am also experiencing this logging issue and it is
starting to get quite tedious.

Regards Gordon

On Thu, Jul 9, 2009 at 9:20 PM, holger krekel<holger at merlinux.eu> wrote:
> On Wed, Jul 08, 2009 at 16:19 -0700, Sridhar Ratnakumar wrote:
>> holger and others,
>>
>> do you have any comments to add to this bug?
>>
>> ? ?http://bugs.python.org/issue6333
>
> i added a comment there, basically suggesting to follow your suggestion.
>
> As py.test should nicely work on many CPython versions i also
> did a general workaround in py.test proper, see here:
>
> ? ?http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/
>
> I intend to release a 1.0.0b8 tomorrow with this
> and some other fixes.
>
> best & thanks,
> holger
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
>


From holger at merlinux.eu  Wed Jul 22 08:47:29 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 08:47:29 +0200
Subject: [py-dev] py.test and logging issue: ValueError: I/O
	operation	on closed file
In-Reply-To: <f2d961430907212316l434c36cegaef6da2d92c7aa1d@mail.gmail.com>
References: <op.uwru3etnbrrvlq@double> <20090709112056.GO7437@trillke.net>
	<f2d961430907212316l434c36cegaef6da2d92c7aa1d@mail.gmail.com>
Message-ID: <20090722064729.GV7437@trillke.net>

On Wed, Jul 22, 2009 at 16:16 +1000, Gordon Wrigley wrote:
> When do you expect 1.0.0b8 to be available on easy-install?
> 
> I ask because I am also experiencing this logging issue and it is
> starting to get quite tedious.

will see to get it out today. 

holger
 
> On Thu, Jul 9, 2009 at 9:20 PM, holger krekel<holger at merlinux.eu> wrote:
> > On Wed, Jul 08, 2009 at 16:19 -0700, Sridhar Ratnakumar wrote:
> >> holger and others,
> >>
> >> do you have any comments to add to this bug?
> >>
> >> ? ?http://bugs.python.org/issue6333
> >
> > i added a comment there, basically suggesting to follow your suggestion.
> >
> > As py.test should nicely work on many CPython versions i also
> > did a general workaround in py.test proper, see here:
> >
> > ? ?http://bitbucket.org/hpk42/py-trunk/changeset/f04b11d51e64/
> >
> > I intend to release a 1.0.0b8 tomorrow with this
> > and some other fixes.
> >
> > best & thanks,
> > holger
> > _______________________________________________
> > py-dev mailing list
> > py-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/py-dev
> >
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From holger at merlinux.eu  Wed Jul 22 08:51:44 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 08:51:44 +0200
Subject: [py-dev] Problem with py.lib and heapy
In-Reply-To: <70a7f84f0907212031k39bb085eg53f6420b24b3153c@mail.gmail.com>
References: <70a7f84f0907212031k39bb085eg53f6420b24b3153c@mail.gmail.com>
Message-ID: <20090722065144.GW7437@trillke.net>

Hi Vineet,

seems you are using a relatively old version (0.9.1). 

could you try to upgrade to 1.0.0b7 by doing "easy_install -U py"? 
This will probably overwrite what your system-wide package system installed
but it likely fixes the below problem as greenlets are not
part of the 1.0 series anymore. 

best,

holger

On Tue, Jul 21, 2009 at 23:31 -0400, Vineet Jain wrote:
> I was trying to use heapy
> (http://guppy-pe.sourceforge.net/)<http://guppy-pe.sourceforge.net/>
> on
> my application (which uses py.log and py.test)  to try and find some memory
> leaks. I can't seem to get it to work with py.lib (see exception trace at
> the end of the email). Can someone please guide me on how to just install
> py.log (and it's minimum requirements)? I tried downloading the tar and
> removed all the extra packages in setup.py, however, that failed. Currently
> I have py.log commented out while I'm debugging my memory leaks.
> 
> hp = hpy()
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/__init__.py",
> line 37, in hpy
> 
> return r.guppy.heapy.Use
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 45, in __getattr__
> 
> return self._share.getattr(self, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 195, in getattr
> 
> d = self.getattr2(inter, cache, owner, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 213, in getattr2
> 
> x = self.getattr_package(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 261, in getattr_package
> 
> x = self.makeModule(x, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 321, in makeModule
> 
> return Share(module, self, module.__name__, Clamp)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 184, in __init__
> 
> getattr(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 45, in __getattr__
> 
> return self._share.getattr(self, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 195, in getattr
> 
> d = self.getattr2(inter, cache, owner, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 215, in getattr2
> 
> x = self.getattr3(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 283, in getattr3
> 
> pa = getattr(pa, at)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 45, in __getattr__
> 
> return self._share.getattr(self, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 195, in getattr
> 
> d = self.getattr2(inter, cache, owner, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 213, in getattr2
> 
> x = self.getattr_package(inter, name)
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/etc/Glue.py",
> line 250, in getattr_package
> 
> x = __import__(self.makeName(name), globals(), locals())
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
> line 555, in <module>
> 
> prime_builtin_types()
> 
> File
> "/usr/lib/python2.5/site-packages/guppy-0.1.9-py2.5-linux-x86_64.egg/guppy/heapy/View.py",
> line 546, in prime_builtin_types
> 
> for t in mod.__dict__.values():
> 
> File "/usr/lib/python2.5/site-packages/py-0.9.1-py2.5.egg/py/initpkg.py",
> line 216, in getdict
> 
> assert not self.__map__, "%r not empty" % self.__map__
> 
> AssertionError: {'greenlet': ('./magic/greenlet.py', 'greenlet')} not empty

> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev


-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From Ronny.Pfannschmidt at gmx.de  Wed Jul 22 09:51:31 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Wed, 22 Jul 2009 09:51:31 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs and distribution
Message-ID: <1248249091.8033.8.camel@localhost>

Hi,

i propose 2 new hooks to deal with more intresting aspects of test
distribution and txspecs

1. pytest_generate_gateways(config/session) -> list of execnet gateways

this hook is supposed to generate the execnet gateways for test
execution

the default implementation is supposed to just do what py.test currently
does

howevers users should be free to generate own gateways to custom set up
things like virtualenvs on local or remote computers and be able to
combine txspecs with own options

2. pytest_select_gateways(test, list of gateways) -> list of gateways

this hook is supposed to select the gateways each test is supposed to
run on

the default implementation should pass a test to all of them or
distribute them reasonable

the api for this one probably needs a few more idea since i think my
initial proposal is not adequate for all uses

Regards
Ronny



From holger at merlinux.eu  Wed Jul 22 11:55:58 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 11:55:58 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs
	and	distribution
In-Reply-To: <1248249091.8033.8.camel@localhost>
References: <1248249091.8033.8.camel@localhost>
Message-ID: <20090722095558.GZ7437@trillke.net>

Hey Ronny, 

maybe it's good to also state your overall goal: running 
tests in virtualenv-environments with different packages
pre-installed. 

On Wed, Jul 22, 2009 at 09:51 +0200, Ronny Pfannschmidt wrote:
> Hi,
> 
> i propose 2 new hooks to deal with more intresting aspects of test
> distribution and txspecs
> 
> 1. pytest_generate_gateways(config/session) -> list of execnet gateways
> 
> this hook is supposed to generate the execnet gateways for test
> execution
> 
> the default implementation is supposed to just do what py.test currently
> does

probably rather a hook for setting a single gateway

    pytest_makegateway(txspec) -> execnet gateway 

because management of multiple nodes and their setup
is its own concern - some nodes might not come up etc.

In the --tx specifciation one could say e.g. 

    popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2

and a pytest_virtualenv plugin could take it, call virtualenv
and eventually start up a gateway with the venv/bin/python. 
 
> howevers users should be free to generate own gateways to custom set up
> things like virtualenvs on local or remote computers and be able to
> combine txspecs with own options

To get started, i would first try to get things working with
popen, then extend to handle remote places - would you want
to automatically install virtualenv there and/or require 
setuptools, btw?   
 
> 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> 
> this hook is supposed to select the gateways each test is supposed to
> run on

something like this could make sense.  It's a second step, i'd think, 
though. 

Can you imagine working on a branch to try implement these hooks? 
I am willing to help/review.  

> the default implementation should pass a test to all of them or
> distribute them reasonable
> the api for this one probably needs a few more idea since i think my
> initial proposal is not adequate for all uses

yes, working from concrete uses (a pytest_virtualenv
plugin) makes sense.  

best,
holger


From Ronny.Pfannschmidt at gmx.de  Wed Jul 22 12:33:27 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Wed, 22 Jul 2009 12:33:27 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs and
 distribution
In-Reply-To: <20090722095558.GZ7437@trillke.net>
References: <1248249091.8033.8.camel@localhost>
	<20090722095558.GZ7437@trillke.net>
Message-ID: <1248258807.8033.12.camel@localhost>

On Wed, 2009-07-22 at 11:55 +0200, holger krekel wrote:
> Hey Ronny, 
> 
> maybe it's good to also state your overall goal: running 
> tests in virtualenv-environments with different packages
> pre-installed. 
> 
> On Wed, Jul 22, 2009 at 09:51 +0200, Ronny Pfannschmidt wrote:
> > Hi,
> > 
> > i propose 2 new hooks to deal with more intresting aspects of test
> > distribution and txspecs
> > 
> > 1. pytest_generate_gateways(config/session) -> list of execnet gateways
> > 
> > this hook is supposed to generate the execnet gateways for test
> > execution
> > 
> > the default implementation is supposed to just do what py.test currently
> > does
> 
> probably rather a hook for setting a single gateway
> 
>     pytest_makegateway(txspec) -> execnet gateway 

indeed seperation of generating specifications and turning them into
gateways makes sense

> 
> because management of multiple nodes and their setup
> is its own concern - some nodes might not come up etc.
> 
> In the --tx specifciation one could say e.g. 
> 
>     popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2
> 
> and a pytest_virtualenv plugin could take it, call virtualenv
> and eventually start up a gateway with the venv/bin/python. 

ok, how would i generate sets of txspec declarations, 

cause i want to combine multiple arguments in order to generate the python versions and the package versions


>  
> > howevers users should be free to generate own gateways to custom set up
> > things like virtualenvs on local or remote computers and be able to
> > combine txspecs with own options
> 
> To get started, i would first try to get things working with
> popen, then extend to handle remote places - would you want
> to automatically install virtualenv there and/or require 
> setuptools, btw?   
>  
> > 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> > 
> > this hook is supposed to select the gateways each test is supposed to
> > run on
> 
> something like this could make sense.  It's a second step, i'd think, 
> though. 
i would certainly need it
for example tests for mercurial would hardly work in a virtualenv thats
only only set up for bazaar

> 
> Can you imagine working on a branch to try implement these hooks? 
> I am willing to help/review. 
sure
>  
> 
> > the default implementation should pass a test to all of them or
> > distribute them reasonable
> > the api for this one probably needs a few more idea since i think my
> > initial proposal is not adequate for all uses
> 
> yes, working from concrete uses (a pytest_virtualenv
> plugin) makes sense.  
> 




From holger at merlinux.eu  Wed Jul 22 13:03:06 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 13:03:06 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs
	and	distribution
In-Reply-To: <1248258807.8033.12.camel@localhost>
References: <1248249091.8033.8.camel@localhost>
	<20090722095558.GZ7437@trillke.net>
	<1248258807.8033.12.camel@localhost>
Message-ID: <20090722110306.GA7437@trillke.net>

On Wed, Jul 22, 2009 at 12:33 +0200, Ronny Pfannschmidt wrote:
> > In the --tx specifciation one could say e.g. 
> > 
> >     popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2
> > 
> > and a pytest_virtualenv plugin could take it, call virtualenv
> > and eventually start up a gateway with the venv/bin/python. 
> 
> ok, how would i generate sets of txspec declarations, 

> cause i want to combine multiple arguments in order to generate the python versions and the package versions
 
i guess by implementing a ``pytest_configure(config)`` hook to
add a few txspecs.  We need to extend the API to the
config object for that, currently it's readonly. Maybe: 

    def pytest_configure(config):
        # access your custom plugin options to 
        # determine which combis to generate 
        config.setvalue('tx', ['spec1', 'spec2', ...])
 
> > > howevers users should be free to generate own gateways to custom set up
> > > things like virtualenvs on local or remote computers and be able to
> > > combine txspecs with own options
> > 
> > To get started, i would first try to get things working with
> > popen, then extend to handle remote places - would you want
> > to automatically install virtualenv there and/or require 
> > setuptools, btw?   

i presume it makes sense to you to first try popen. 

> > > 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> > > 
> > > this hook is supposed to select the gateways each test is supposed to
> > > run on
> > 
> > something like this could make sense.  It's a second step, i'd think, 
> > though. 
> i would certainly need it
> for example tests for mercurial would hardly work in a virtualenv thats
> only only set up for bazaar

you could still skip the tests late-bound but i can see you want
to avoid even trying tests in unfitting environments. 
The issue here is to have a somewhat higher level API for 
what you call "list of gateways" because the raw low-level 
execnet-gateways are not fitting.  you rather want something
where you have structured access to the "test nodes", their
txspecs etc. and this requires a bit of refactoring. 

> > Can you imagine working on a branch to try implement these hooks? 
> > I am willing to help/review. 
> sure

fine.  please add a bitbucket diff-email 
service to py-svn at codespeak.net then. 

best,
holger

> >  
> > 
> > > the default implementation should pass a test to all of them or
> > > distribute them reasonable
> > > the api for this one probably needs a few more idea since i think my
> > > initial proposal is not adequate for all uses
> > 
> > yes, working from concrete uses (a pytest_virtualenv
> > plugin) makes sense.  
> > 
> 
> 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From Ronny.Pfannschmidt at gmx.de  Wed Jul 22 13:17:20 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Wed, 22 Jul 2009 13:17:20 +0200
Subject: [py-dev] hook ideas for better use/abuse of txspecs and
 distribution
In-Reply-To: <20090722110306.GA7437@trillke.net>
References: <1248249091.8033.8.camel@localhost>
	<20090722095558.GZ7437@trillke.net>
	<1248258807.8033.12.camel@localhost>
	<20090722110306.GA7437@trillke.net>
Message-ID: <1248261440.8033.20.camel@localhost>

On Wed, 2009-07-22 at 13:03 +0200, holger krekel wrote:
> On Wed, Jul 22, 2009 at 12:33 +0200, Ronny Pfannschmidt wrote:
> > > In the --tx specifciation one could say e.g. 
> > > 
> > >     popen//python=python2.5//virtualenv=docutils-0.5,mercurial-1.2
> > > 
> > > and a pytest_virtualenv plugin could take it, call virtualenv
> > > and eventually start up a gateway with the venv/bin/python. 
> > 
> > ok, how would i generate sets of txspec declarations, 
> 
> > cause i want to combine multiple arguments in order to generate the python versions and the package versions
>  
> i guess by implementing a ``pytest_configure(config)`` hook to
> add a few txspecs.  We need to extend the API to the
> config object for that, currently it's readonly. Maybe: 
> 
>     def pytest_configure(config):
>         # access your custom plugin options to 
>         # determine which combis to generate 
>         config.setvalue('tx', ['spec1', 'spec2', ...])
>  
> > > > howevers users should be free to generate own gateways to custom set up
> > > > things like virtualenvs on local or remote computers and be able to
> > > > combine txspecs with own options
> > > 
> > > To get started, i would first try to get things working with
> > > popen, then extend to handle remote places - would you want
> > > to automatically install virtualenv there and/or require 
> > > setuptools, btw?   
> 
> i presume it makes sense to you to first try popen. 
> 
> > > > 2. pytest_select_gateways(test, list of gateways) -> list of gateways
> > > > 
> > > > this hook is supposed to select the gateways each test is supposed to
> > > > run on
> > > 
> > > something like this could make sense.  It's a second step, i'd think, 
> > > though. 
> > i would certainly need it
> > for example tests for mercurial would hardly work in a virtualenv thats
> > only only set up for bazaar
> 
> you could still skip the tests late-bound but i can see you want
> to avoid even trying tests in unfitting environments. 
> The issue here is to have a somewhat higher level API for 
> what you call "list of gateways" because the raw low-level 
> execnet-gateways are not fitting.  you rather want something
> where you have structured access to the "test nodes", their
> txspecs etc. and this requires a bit of refactoring. 

ok, lets make a small resume about my tasks

1. a hook for turning a txspec to a gateway
2. a nice hook for taking options and generating a set of txspecs
  (the default being just to pass along the ones from options)
3. a virtualenv plugin

   a. support for setting up a virtualenv 
      with a custom python version + versions of packages
   b. support for executing the virtualenv setup via networked gateways
      ie the socket one and the ssh one
   

regards
Ronny




From holger at merlinux.eu  Wed Jul 22 16:23:31 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 22 Jul 2009 16:23:31 +0200
Subject: [py-dev] 1.0.0b8 out
Message-ID: <20090722142331.GC7437@trillke.net>

Hi all, 

i just put a new 1.0.0.b8 release to PyPI, find below
a list of changes.  

Documentation is also updated majorly - there now is an 
auto-generated list of pages for each plugin, just go to 

    http://pytest.org 

and look around.  And let me know of any issues. 

have fun, and thanks to all who helped.

holger


list of changes 1.0.0b7 to 1.0.0b8:

* pytest_unittest-plugin is now enabled by default

* introduced pytest_keyboardinterrupt hook and 
  refined pytest_sessionfinish hook, added tests. 

* workaround a buggy logging module interaction ("closing already closed
  files").  Thanks to Sridhar Ratnakumar for triggering. 

* if plugins use "py.test.importorskip" for importing 
  a dependency only a warning will be issued instead 
  of exiting the testing process. 

* many improvements to docs: 
  - refined funcargs doc , use the term "factory" instead of "provider"
  - added a new talk/tutorial doc page 
  - better download page
  - better plugin docstrings
  - added new plugins page and automatic doc generation script

* fixed teardown problem related to partially failing funcarg setups 
  (thanks MrTopf for reporting), "pytest_runtest_teardown" is now 
  always invoked even if the "pytest_runtest_setup" failed. 

* tweaked doctest output for docstrings in py modules, 
  thanks Radomir. 

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From py-dev at tolomea.com  Thu Jul 23 04:04:30 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 23 Jul 2009 12:04:30 +1000
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <20090722142331.GC7437@trillke.net>
References: <20090722142331.GC7437@trillke.net>
Message-ID: <f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>

Alas I'm still getting logging issues, specifically I get a big slab
of this every time a test fails

Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
    stream.write(fs % msg)
ValueError: I/O operation on closed file

Regards Gordon

On Thu, Jul 23, 2009 at 12:23 AM, holger krekel<holger at merlinux.eu> wrote:
> Hi all,
>
> i just put a new 1.0.0.b8 release to PyPI, find below
> a list of changes.
>
> Documentation is also updated majorly - there now is an
> auto-generated list of pages for each plugin, just go to
>
> ? ?http://pytest.org
>
> and look around. ?And let me know of any issues.
>
> have fun, and thanks to all who helped.
>
> holger
>
>
> list of changes 1.0.0b7 to 1.0.0b8:
>
> * pytest_unittest-plugin is now enabled by default
>
> * introduced pytest_keyboardinterrupt hook and
> ?refined pytest_sessionfinish hook, added tests.
>
> * workaround a buggy logging module interaction ("closing already closed
> ?files"). ?Thanks to Sridhar Ratnakumar for triggering.
>
> * if plugins use "py.test.importorskip" for importing
> ?a dependency only a warning will be issued instead
> ?of exiting the testing process.
>
> * many improvements to docs:
> ?- refined funcargs doc , use the term "factory" instead of "provider"
> ?- added a new talk/tutorial doc page
> ?- better download page
> ?- better plugin docstrings
> ?- added new plugins page and automatic doc generation script
>
> * fixed teardown problem related to partially failing funcarg setups
> ?(thanks MrTopf for reporting), "pytest_runtest_teardown" is now
> ?always invoked even if the "pytest_runtest_setup" failed.
>
> * tweaked doctest output for docstrings in py modules,
> ?thanks Radomir.
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
>


From py-dev at tolomea.com  Thu Jul 23 05:23:54 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Thu, 23 Jul 2009 13:23:54 +1000
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
References: <20090722142331.GC7437@trillke.net>
	<f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
Message-ID: <f2d961430907222023y1cb548aay267a55e464e7ca5c@mail.gmail.com>

Actually on further use it appears that I am getting these on test
passes as well.

Also the test set I'm using doesn't use execnet, so that isn't a
complicating factor.

Regards Gordon

On Thu, Jul 23, 2009 at 12:04 PM, Gordon Wrigley<py-dev at tolomea.com> wrote:
> Alas I'm still getting logging issues, specifically I get a big slab
> of this every time a test fails
>
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> ? ?stream.write(fs % msg)
> ValueError: I/O operation on closed file
>
> Regards Gordon
>
> On Thu, Jul 23, 2009 at 12:23 AM, holger krekel<holger at merlinux.eu> wrote:
>> Hi all,
>>
>> i just put a new 1.0.0.b8 release to PyPI, find below
>> a list of changes.
>>
>> Documentation is also updated majorly - there now is an
>> auto-generated list of pages for each plugin, just go to
>>
>> ? ?http://pytest.org
>>
>> and look around. ?And let me know of any issues.
>>
>> have fun, and thanks to all who helped.
>>
>> holger
>>
>>
>> list of changes 1.0.0b7 to 1.0.0b8:
>>
>> * pytest_unittest-plugin is now enabled by default
>>
>> * introduced pytest_keyboardinterrupt hook and
>> ?refined pytest_sessionfinish hook, added tests.
>>
>> * workaround a buggy logging module interaction ("closing already closed
>> ?files"). ?Thanks to Sridhar Ratnakumar for triggering.
>>
>> * if plugins use "py.test.importorskip" for importing
>> ?a dependency only a warning will be issued instead
>> ?of exiting the testing process.
>>
>> * many improvements to docs:
>> ?- refined funcargs doc , use the term "factory" instead of "provider"
>> ?- added a new talk/tutorial doc page
>> ?- better download page
>> ?- better plugin docstrings
>> ?- added new plugins page and automatic doc generation script
>>
>> * fixed teardown problem related to partially failing funcarg setups
>> ?(thanks MrTopf for reporting), "pytest_runtest_teardown" is now
>> ?always invoked even if the "pytest_runtest_setup" failed.
>>
>> * tweaked doctest output for docstrings in py modules,
>> ?thanks Radomir.
>>
>> --
>> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
>> Python, PyPy, pytest contracting: http://merlinux.eu
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>


From holger at merlinux.eu  Thu Jul 23 19:52:55 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 23 Jul 2009 19:52:55 +0200
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
References: <20090722142331.GC7437@trillke.net>
	<f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
Message-ID: <20090723175255.GD7437@trillke.net>

On Thu, Jul 23, 2009 at 12:04 +1000, Gordon Wrigley wrote:
> Alas I'm still getting logging issues, specifically I get a big slab
> of this every time a test fails
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
>     stream.write(fs % msg)
> ValueError: I/O operation on closed file

hum, py.test does not do anything with logging itself. 
But it captures stdout/stderr during setup, test-call
and teardown to temporary files/StringIOs.  It properly 
closes the temporary files/streams at each of the
three phases.  Do you have a single logging configuration
during setup/call/teardown, mabe? 

if you disable output capturing with "-s" you don't 
see these error message, right? 

holger


From holger at merlinux.eu  Thu Jul 23 21:23:29 2009
From: holger at merlinux.eu (holger krekel)
Date: Thu, 23 Jul 2009 21:23:29 +0200
Subject: [py-dev] 1.0.0b8 out
In-Reply-To: <20090723175255.GD7437@trillke.net>
References: <20090722142331.GC7437@trillke.net>
	<f2d961430907221904t4486633dn17ce17629faa85ed@mail.gmail.com>
	<20090723175255.GD7437@trillke.net>
Message-ID: <20090723192329.GE7437@trillke.net>

On Thu, Jul 23, 2009 at 19:52 +0200, holger krekel wrote:
> On Thu, Jul 23, 2009 at 12:04 +1000, Gordon Wrigley wrote:
> > Alas I'm still getting logging issues, specifically I get a big slab
> > of this every time a test fails
> > 
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.6/logging/__init__.py", line 769, in emit
> >     stream.write(fs % msg)
> > ValueError: I/O operation on closed file
> 
> hum, py.test does not do anything with logging itself. 
> But it captures stdout/stderr during setup, test-call
> and teardown to temporary files/StringIOs.  It properly 
> closes the temporary files/streams at each of the
> three phases.  Do you have a single logging configuration
> during setup/call/teardown, mabe? 

that seems to be the issue, here is one way to reproduce it: 

        import logging
        def setup_module(mod):
            logging.warn("hello world")

        def test_logging():
            logging.warn("hello world")
            assert 0

need to think some more about this and maybe look more
into the logging module and its assumtions. 

holger


From holger at merlinux.eu  Fri Jul 31 15:49:49 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 31 Jul 2009 15:49:49 +0200
Subject: [py-dev] pylib/py.test 1.0.0b9 out
Message-ID: <20090731134948.GL31589@trillke.net>

Hi all, 

i've just released a 1.0.0b9 to PyPI which should fix all
interactions with logging and has generally improved capturing
and reporting.  Below is the full changelog. 

If you like to help with some proof-reading, you are welcome to do
a mercurial checkout (see http://codespeak.net/py/dist/download.html) 
go to the 

    doc/  
    py/test/plugin/pytest_*.py 

directories and look through the text files and module docstrings
from which all of the plugin docs are generated at 
    
    http://codespeak.net/py/dist/test/plugin/index.html

thanks & cheers,

holger


Changes between 1.0.0b8 and 1.0.0b9
=====================================

* cleanly handle and report final teardown of test setup

* fix svn-1.6 compat issue with py.path.svnwc().versioned()
  (thanks Wouter Vanden Hove)

* setup/teardown or collection problems now show as ERRORs
  or with big "E"'s in the progress lines.  they are reported
  and counted separately. 
 
* dist-testing: properly handle test items that get locally 
  collected but cannot be collected on the remote side - often 
  due to platform/dependency reasons

* simplified py.test.mark API - see keyword plugin documentation

* integrate better with logging: capturing now by default captures
  test functions and their immediate setup/teardown in a single stream 

* capsys and capfd funcargs now have a readouterr() and a close() method
  (underlyingly py.io.StdCapture/FD objects are used which grew a 
  readouterr() method as well to return snapshots of captured out/err)

* make assert-reinterpretation work better with comparisons not 
  returning bools (reported with numpy from thanks maciej fijalkowski)

* reworked per-test output capturing into the pytest_iocapture.py plugin 
  and thus removed capturing code from config object 

* item.repr_failure(excinfo) instead of item.repr_failure(excinfo, outerr)


Changes between 1.0.0b7 and 1.0.0b8
=====================================

* pytest_unittest-plugin is now enabled by default

* introduced pytest_keyboardinterrupt hook and 
  refined pytest_sessionfinish hooked, added tests. 

* workaround a buggy logging module interaction ("closing already closed
  files").  Thanks to Sridhar Ratnakumar for triggering. 

* if plugins use "py.test.importorskip" for importing 
  a dependency only a warning will be issued instead 
  of exiting the testing process. 

* many improvements to docs: 
  - refined funcargs doc , use the term "factory" instead of "provider"
  - added a new talk/tutorial doc page 
  - better download page
  - better plugin docstrings
  - added new plugins page and automatic doc generation script

* fixed teardown problem related to partially failing funcarg setups 
  (thanks MrTopf for reporting), "pytest_runtest_teardown" is now 
  always invoked even if the "pytest_runtest_setup" failed. 

* tweaked doctest output for docstrings in py modules, 
  thanks Radomir. 

Changes between 1.0.0b3 and 1.0.0b7
=============================================

* renamed py.test.xfail back to py.test.mark.xfail to avoid 
  two ways to decorate for xfail

* re-added py.test.mark decorator for setting keywords on functions 
  (it was actually documented so removing it was not nice) 

* remove scope-argument from request.addfinalizer() because 
  request.cached_setup has the scope arg. TOOWTDI. 

* perform setup finalization before reporting failures

* apply modified patches from Andreas Kloeckner to allow 
  test functions to have no func_code (#22) and to make 
  "-k" and function keywords work  (#20) 

* apply patch from Daniel Peolzleithner (issue #23) 

* resolve issue #18, multiprocessing.Manager() and 
  redirection clash 

* make __name__ == "__channelexec__" for remote_exec code

Changes between 1.0.0b1 and 1.0.0b3
=============================================

* plugin classes are removed: one now defines 
  hooks directly in conftest.py or global pytest_*.py 
  files. 

* added new pytest_namespace(config) hook that allows 
  to inject helpers directly to the py.test.* namespace. 

* documented and refined many hooks 

* added new style of generative tests via 
  pytest_generate_tests hook that integrates 
  well with function arguments. 
 

Changes between 0.9.2 and 1.0.0b1
=============================================

* introduced new "funcarg" setup method, 
  see doc/test/funcarg.txt 

* introduced plugin architecuture and many 
  new py.test plugins, see 
  doc/test/plugins.txt

* teardown_method is now guaranteed to get 
  called after a test method has run. 
 
* new method: py.test.importorskip(mod,minversion)
  will either import or call py.test.skip()

* completely revised internal py.test architecture

* new py.process.ForkedFunc object allowing to 
  fork execution of a function to a sub process
  and getting a result back. 

XXX lots of things missing here XXX

Changes between 0.9.1 and 0.9.2
===============================

* refined installation and metadata, created new setup.py, 
  now based on setuptools/ez_setup (thanks to Ralf Schmitt 
  for his support).

* improved the way of making py.* scripts available in 
  windows environments, they are now added to the 
  Scripts directory as ".cmd" files. 

* py.path.svnwc.status() now is more complete and 
  uses xml output from the 'svn' command if available
  (Guido Wesdorp)

* fix for py.path.svn* to work with svn 1.5
  (Chris Lamb)

* fix path.relto(otherpath) method on windows to 
  use normcase for checking if a path is relative.

* py.test's traceback is better parseable from editors 
  (follows the filenames:LINENO: MSG convention)
  (thanks to Osmo Salomaa)

* fix to javascript-generation, "py.test --runbrowser" 
  should work more reliably now

* removed previously accidentally added 
  py.test.broken and py.test.notimplemented helpers. 

* there now is a py.__version__ attribute

Changes between 0.9.0 and 0.9.1
===============================

This is a fairly complete list of changes between 0.9 and 0.9.1, which can
serve as a reference for developers.

* allowing + signs in py.path.svn urls [39106]
* fixed support for Failed exceptions without excinfo in py.test [39340]
* added support for killing processes for Windows (as well as platforms that
  support os.kill) in py.misc.killproc [39655]
* added setup/teardown for generative tests to py.test [40702]
* added detection of FAILED TO LOAD MODULE to py.test [40703, 40738, 40739]
* fixed problem with calling .remove() on wcpaths of non-versioned files in
  py.path [44248]
* fixed some import and inheritance issues in py.test [41480, 44648, 44655]
* fail to run greenlet tests when pypy is available, but without stackless
  [45294]
* small fixes in rsession tests [45295]
* fixed issue with 2.5 type representations in py.test [45483, 45484]
* made that internal reporting issues displaying is done atomically in py.test
  [45518]
* made that non-existing files are igored by the py.lookup script [45519]
* improved exception name creation in py.test [45535]
* made that less threads are used in execnet [merge in 45539]
* removed lock required for atomical reporting issue displaying in py.test
  [45545]
* removed globals from execnet [45541, 45547]
* refactored cleanup mechanics, made that setDaemon is set to 1 to make atexit
  get called in 2.5 (py.execnet) [45548]
* fixed bug in joining threads in py.execnet's servemain [45549]
* refactored py.test.rsession tests to not rely on exact output format anymore
  [45646]
* using repr() on test outcome [45647]
* added 'Reason' classes for py.test.skip() [45648, 45649]
* killed some unnecessary sanity check in py.test.collect [45655]
* avoid using os.tmpfile() in py.io.fdcapture because on Windows it's only
  usable by Administrators [45901]
* added support for locking and non-recursive commits to py.path.svnwc [45994]
* locking files in py.execnet to prevent CPython from segfaulting [46010]
* added export() method to py.path.svnurl
* fixed -d -x in py.test [47277]
* fixed argument concatenation problem in py.path.svnwc [49423]
* restore py.test behaviour that it exits with code 1 when there are failures
  [49974]
* don't fail on html files that don't have an accompanying .txt file [50606]
* fixed 'utestconvert.py < input' [50645]
* small fix for code indentation in py.code.source [50755]
* fix _docgen.py documentation building [51285]
* improved checks for source representation of code blocks in py.test [51292]
* added support for passing authentication to py.path.svn* objects [52000,
  52001]
* removed sorted() call for py.apigen tests in favour of [].sort() to support
  Python 2.3 [52481]
-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu