From santoshphilip at mac.com  Sun Nov 12 06:58:06 2006
From: santoshphilip at mac.com (Santosh Philip)
Date: Sat, 11 Nov 2006 21:58:06 -0800
Subject: [py-dev] User defined exception - py.testing
Message-ID: <332851639ed9f5380744455dbf9f7090@mac.com>

I am having some trouble with testing user defined exceptions
==============
this one works
------------
def anexception(arg):
     """just raise an exception with ValueError"""
     if arg == 5:
         raise ValueError, 'Just a ValueError'

def test_anexception():
     """py.test for anexception"""
     import py.test
     py.test.raises(ValueError, anexception, 5)

===============
this one does not work
---------------
class MyError(Exception):
     """pass"""
     pass

def anotherexception(arg):
     """just raise an exception with ValueError"""
     if arg == 5:
         raise MyError, 'its a MyError'

def test_anotherexception():
     """py.test for anotherexception"""
     import py.test
     py.test.raises(ValueError, anotherexception, 5)
================

Are test for user defined exceptions working in py.test
or
Am I doing something wrong ?

when it doesn't work, the output looks like

MacBook:~/Documents/doe2python_DSU santosh$ py.test doesntwork.py
inserting into sys.path:  
/Users/santosh/Documents/coolshadowprojects/pydottest/py-dist
============================= test process starts  
==============================
testing-mode: inprocess
executable:    
/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/ 
Contents/MacOS/Python  (2.4.3-final-0)
using py lib:  
/Users/santosh/Documents/coolshadowprojects/pydottest/py-dist/py <rev  
31477>

doesntwork.py[1] F

________________________________________________________________________ 
________
______________________ entrypoint: test_anotherexception  
_______________________

     def test_anotherexception():
         """py.test for anotherexception"""
         import py.test
 >       py.test.raises(ValueError, anotherexception, 5)

[/Users/santosh/Documents/doe2python_DSU/doesntwork.py:13]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  
_ _ _ _

     def anotherexception(arg):
         """just raise an exception with ValueError"""
         if arg == 5:
E           raise MyError, 'its a MyError'
 >           MyError: its a MyError

[/Users/santosh/Documents/doe2python_DSU/doesntwork.py:8]
________________________________________________________________________ 
________
=================== tests finished: 1 failed in 0.02 seconds  
===================

thanks

Santosh



From cfbolz at gmx.de  Sun Nov 12 09:26:35 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 12 Nov 2006 09:26:35 +0100
Subject: [py-dev] User defined exception - py.testing
In-Reply-To: <332851639ed9f5380744455dbf9f7090@mac.com>
References: <332851639ed9f5380744455dbf9f7090@mac.com>
Message-ID: <4556DABB.8080603@gmx.de>

Hi Santosh!

Santosh Philip wrote:
> I am having some trouble with testing user defined exceptions
> ==============
> this one works
> ------------
> def anexception(arg):
>      """just raise an exception with ValueError"""
>      if arg == 5:
>          raise ValueError, 'Just a ValueError'
>
> def test_anexception():
>      """py.test for anexception"""
>      import py.test
>      py.test.raises(ValueError, anexception, 5)
>
> ===============
> this one does not work
> ---------------
> class MyError(Exception):
>      """pass"""
>      pass
>
> def anotherexception(arg):
>      """just raise an exception with ValueError"""
>      if arg == 5:
>          raise MyError, 'its a MyError'
>
> def test_anotherexception():
>      """py.test for anotherexception"""
>      import py.test
>      py.test.raises(ValueError, anotherexception, 5)
> ================
                      ^^^^^^^^^^
Shouldn't this be "MyError"?

Btw: you don't need to say "import py.test". Just saying "import py"
is enough (py.test will be loaded only when it is used).

Cheers,

Carl Friedrich
 _______________________________________________


From fijal at genesilico.pl  Tue Nov 14 00:49:57 2006
From: fijal at genesilico.pl (=?ISO-8859-2?Q?Maciek_Fija=C5=82kowski?=)
Date: Tue, 14 Nov 2006 00:49:57 +0100 (CET)
Subject: [py-dev] py lib API
Message-ID: <Pine.LNX.4.58.0611140049450.14648@martini.genesilico.pl>

I've created some experimental API of pylib (automatically generated).

It's very rough by now, so don't expect fancy CSS and JS (at least now),
but I'm very interested in any feedback about contents.

http://codespeak.net/~fijal/output/

Cheers,
Fijal





From prouleau001 at gmail.com  Tue Nov 14 20:52:11 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Tue, 14 Nov 2006 14:52:11 -0500
Subject: [py-dev] Speed of py.test compared to unittest
Message-ID: <5c1d522d0611141152y34d19b92o486e40c03188a904@mail.gmail.com>

Hi all!

I'v just started looking into py.test.  I really like it: it's easier
to use and requires less boiler plate code than unittest.  Thanks!

However, i found that it runs much slower than unittest (at least
under some conditions).  I was expecting a slowdown because of
py.test's nature, but I get results that are 10 to 30 times slower.

I ran a test of a simple application that converts roman numeral to
integer and vice versa (taken from Mike Pilgrim's
http://diveintopython.org/ ).
In a test that checks all 4999 possible numbers, unittest performs in
about 125 mSeconds, while py.test takes about 3.8 seconds.  That's 30
times slower.
The unittest and py.test driven scripts perform the same tests.

Is this what I should expect?

I am running Python 2.4.3 on Win32 machine with py-dist revision 34574
(the latest) from svn.

Thanks,

Pierre Rouleau


From fijal at genesilico.pl  Tue Nov 14 21:34:36 2006
From: fijal at genesilico.pl (=?ISO-8859-2?Q?Maciek_Fija=C5=82kowski?=)
Date: Tue, 14 Nov 2006 21:34:36 +0100 (CET)
Subject: [py-dev] Speed of py.test compared to unittest
Message-ID: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>


Pierre Rouleau wrote:

>Hi all!
>
>I'v just started looking into py.test.  I really like it: it's easier
>to use and requires less boiler plate code than unittest.  Thanks!
>
>However, i found that it runs much slower than unittest (at least
>under some conditions).  I was expecting a slowdown because of
>py.test's nature, but I get results that are 10 to 30 times slower.
>
>I ran a test of a simple application that converts roman numeral to
>integer and vice versa (taken from Mike Pilgrim's
>http://diveintopython.org/ ).
>In a test that checks all 4999 possible numbers, unittest performs in
>about 125 mSeconds, while py.test takes about 3.8 seconds.  That's 30
>times slower.
>The unittest and py.test driven scripts perform the same tests.
>
>Is this what I should expect?
>
>I am running Python 2.4.3 on Win32 machine with py-dist revision 34574
>(the latest) from svn.
>
>Thanks,
>
>Pierre Rouleau
>_______________________________________________
>py-dev mailing list
>py-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/py-dev
>
>
I would be happy to run some performance tests on my machine and
profiling as well. Please send to me some details (how to access the
code ie.). Please also note that a lot of time from that 3.8s is startup
time (I guess it's not too much pain to just allow it to be about 2s).




From prouleau001 at gmail.com  Tue Nov 14 22:36:20 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Tue, 14 Nov 2006 16:36:20 -0500
Subject: [py-dev] Fwd:  Speed of py.test compared to unittest
In-Reply-To: <5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>
	<5c1d522d0611141304u42ebd283j5828a720621ec5d6@mail.gmail.com>
	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>
	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>
Message-ID: <5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>

---------- Forwarded message ----------
From: Pierre Rouleau <prouleau001 at gmail.com>
Date: Nov 14, 2006 4:18 PM
Subject: Re: [py-dev] Speed of py.test compared to unittest
To: Maciek Fija??kowski <fijal at genesilico.pl>


On 11/14/06, Maciek Fija??kowski <fijal at genesilico.pl> wrote:
>
> Sure. Please do.
>
Oh, also, as far as startup time is concerned, even though the startup
time might be constant, a larger startup time will mask differences in
execution time of the code under test.  Since a unit test system
should optimally be used during development, a startup time close to 0
will show timing impact of the code being tested as it is being
modified.  And this might be useful during development (even though
you might want to measure performance of the module being developed
using other tools).

From cfbolz at gmx.de  Tue Nov 14 23:02:16 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 14 Nov 2006 23:02:16 +0100
Subject: [py-dev] Fwd:  Speed of py.test compared to unittest
In-Reply-To: <5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>	<5c1d522d0611141304u42ebd283j5828a720621ec5d6@mail.gmail.com>	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>
	<5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>
Message-ID: <455A3CE8.5000709@gmx.de>

Pierre Rouleau wrote:
> ---------- Forwarded message ----------
> From: Pierre Rouleau <prouleau001 at gmail.com>
> Date: Nov 14, 2006 4:18 PM
> Subject: Re: [py-dev] Speed of py.test compared to unittest
> To: Maciek Fija??kowski <fijal at genesilico.pl>
>
>
> On 11/14/06, Maciek Fija??kowski <fijal at genesilico.pl> wrote:
>> Sure. Please do.
>>
> Oh, also, as far as startup time is concerned, even though the startup
> time might be constant, a larger startup time will mask differences in
> execution time of the code under test.  Since a unit test system
> should optimally be used during development, a startup time close to 0
> will show timing impact of the code being tested as it is being
> modified.  And this might be useful during development (even though
> you might want to measure performance of the module being developed
> using other tools).

I am not quite sure I buy that argument since py.test tells you the time
that the test itself took (if you use the -v argument) or that all tests
together take (at the end of each test run). As far as I remember, these
times don't include any startup times. Yes, wall clock time of the whole
test run is quite a bad measure, but you don't have to use it anyway.

Cheers,

Carl Friedrich


From prouleau001 at gmail.com  Tue Nov 14 23:07:34 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Tue, 14 Nov 2006 17:07:34 -0500
Subject: [py-dev] Fwd: Speed of py.test compared to unittest
In-Reply-To: <455A3CE8.5000709@gmx.de>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>
	<5c1d522d0611141304u42ebd283j5828a720621ec5d6@mail.gmail.com>
	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>
	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>
	<5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>
	<455A3CE8.5000709@gmx.de>
Message-ID: <5c1d522d0611141407gb35968fw56f74140d69db7d@mail.gmail.com>

On 11/14/06, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Pierre Rouleau wrote:
> > ---------- Forwarded message ----------
> > From: Pierre Rouleau <prouleau001 at gmail.com>
> > Date: Nov 14, 2006 4:18 PM
> > Subject: Re: [py-dev] Speed of py.test compared to unittest
> > To: Maciek Fija??kowski <fijal at genesilico.pl>
> >
> >
> > On 11/14/06, Maciek Fija??kowski <fijal at genesilico.pl> wrote:
> >> Sure. Please do.
> >>
> > Oh, also, as far as startup time is concerned, even though the startup
> > time might be constant, a larger startup time will mask differences in
> > execution time of the code under test.  Since a unit test system
> > should optimally be used during development, a startup time close to 0
> > will show timing impact of the code being tested as it is being
> > modified.  And this might be useful during development (even though
> > you might want to measure performance of the module being developed
> > using other tools).
>
> I am not quite sure I buy that argument since py.test tells you the time
> that the test itself took (if you use the -v argument) or that all tests
> together take (at the end of each test run). As far as I remember, these
> times don't include any startup times. Yes, wall clock time of the whole
> test run is quite a bad measure, but you don't have to use it anyway.
>
> Cheers,
>
> Carl Friedrich
>
I didn't realize that the time of *each* test is shown when the -v
option is used.  Thanks for pointing that out.

P.R.

From cfbolz at gmx.de  Tue Nov 14 23:13:38 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 14 Nov 2006 23:13:38 +0100
Subject: [py-dev] Fwd: Speed of py.test compared to unittest
In-Reply-To: <5c1d522d0611141407gb35968fw56f74140d69db7d@mail.gmail.com>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>	<5c1d522d0611141304u42ebd283j5828a720621ec5d6@mail.gmail.com>	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>	<5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>	<455A3CE8.5000709@gmx.de>
	<5c1d522d0611141407gb35968fw56f74140d69db7d@mail.gmail.com>
Message-ID: <455A3F92.8060300@gmx.de>

Pierre Rouleau wrote:
> On 11/14/06, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>> I am not quite sure I buy that argument since py.test tells you the time
>> that the test itself took (if you use the -v argument) or that all tests
>> together take (at the end of each test run). As far as I remember, these
>> times don't include any startup times. Yes, wall clock time of the whole
>> test run is quite a bad measure, but you don't have to use it anyway.
>>
>> Cheers,
>>
>> Carl Friedrich
>>
> I didn't realize that the time of *each* test is shown when the -v
> option is used.  Thanks for pointing that out.

you're welcome :-)

Note that I didn't mean to imply that it isn't worthwhile to make
py.test startup time smaller. it's definitely a good thing to do, since
testing should have low overhead.

Cheers,

Carl Friedrich


From fijal at genesilico.pl  Wed Nov 15 12:30:47 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Wed, 15 Nov 2006 12:30:47 +0100
Subject: [py-dev] Fwd: Speed of py.test compared to unittest
In-Reply-To: <455A3F92.8060300@gmx.de>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>	<5c1d522d0611141304u42ebd283j5828a720621ec5d6@mail.gmail.com>	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>	<5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>	<455A3CE8.5000709@gmx.de>	<5c1d522d0611141407gb35968fw56f74140d69db7d@mail.gmail.com>
	<455A3F92.8060300@gmx.de>
Message-ID: <455AFA67.60109@genesilico.pl>

Carl Friedrich Bolz wrote:

>Pierre Rouleau wrote:
>  
>
>>On 11/14/06, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>>    
>>
>>>I am not quite sure I buy that argument since py.test tells you the time
>>>that the test itself took (if you use the -v argument) or that all tests
>>>together take (at the end of each test run). As far as I remember, these
>>>times don't include any startup times. Yes, wall clock time of the whole
>>>test run is quite a bad measure, but you don't have to use it anyway.
>>>
>>>Cheers,
>>>
>>>Carl Friedrich
>>>
>>>      
>>>
>>I didn't realize that the time of *each* test is shown when the -v
>>option is used.  Thanks for pointing that out.
>>    
>>
>
>you're welcome :-)
>
>Note that I didn't mean to imply that it isn't worthwhile to make
>py.test startup time smaller. it's definitely a good thing to do, since
>testing should have low overhead.
>
>Cheers,
>
>Carl Friedrich
>_______________________________________________
>py-dev mailing list
>py-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/py-dev
>  
>
I've profiled it a bit. Right now (for me) your tests are about 3x 
slower that unittests. It still can be improved somehow.

Btw: Tests are unfair, cause collection encounters 5k tests for py.test 
and 13 for unittests, if you rewrite it so they'll be equal it would be 
better. It's not good excuse anyway ;-)

I'll try to work a bit more on it, but I guess achieving 2x in this 
particular case would be enough. If you still encounter massive 
slowdowns in different enviroments, do not hesitate to contact us.

Cheers,
Maciej Fijalkowski



From eopadoan at altavix.com  Wed Nov 15 13:06:29 2006
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Wed, 15 Nov 2006 10:06:29 -0200
Subject: [py-dev] Speed of py.test compared to unittest
In-Reply-To: <5c1d522d0611141152y34d19b92o486e40c03188a904@mail.gmail.com>
References: <5c1d522d0611141152y34d19b92o486e40c03188a904@mail.gmail.com>
Message-ID: <dea92f560611150406t5d6c16fdn20eddb2c7157a36f@mail.gmail.com>

> I ran a test of a simple application that converts roman numeral to
> integer and vice versa (taken from Mike Pilgrim's
> http://diveintopython.org/ ).
> In a test that checks all 4999 possible numbers, unittest performs in
> about 125 mSeconds, while py.test takes about 3.8 seconds.  That's 30
> times slower.
> The unittest and py.test driven scripts perform the same tests.
I've tried to convert the Pilgrim's unittest examples to Py.test too,
and I think the performance using simple asserts inside loops was
acceptable. But using generative tests, I was tremendouly slow.

I you people think attachment is a bad idea, I can think of another
way to share the code I'm talking about.


-- 
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: romantest.py
Type: text/x-python
Size: 4536 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20061115/95613379/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: roman_test.py
Type: text/x-python
Size: 7212 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20061115/95613379/attachment-0001.py>

From prouleau001 at gmail.com  Wed Nov 15 17:39:11 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Wed, 15 Nov 2006 11:39:11 -0500
Subject: [py-dev] Fwd:  Fwd: Speed of py.test compared to unittest
In-Reply-To: <5c1d522d0611150837y12fc80d4y5cad8b16a235a4a0@mail.gmail.com>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>
	<5c1d522d0611141304u42ebd283j5828a720621ec5d6@mail.gmail.com>
	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>
	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>
	<5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>
	<455A3CE8.5000709@gmx.de>
	<5c1d522d0611141407gb35968fw56f74140d69db7d@mail.gmail.com>
	<455A3F92.8060300@gmx.de> <455AFA67.60109@genesilico.pl>
	<5c1d522d0611150837y12fc80d4y5cad8b16a235a4a0@mail.gmail.com>
Message-ID: <5c1d522d0611150839l4ecf8ff1w64fd5027b26a3470@mail.gmail.com>

On 11/15/06, Maciek Fijalkowski <fijal at genesilico.pl> wrote:
> Carl Friedrich Bolz wrote:
>
> >Pierre Rouleau wrote:
> >
> >
> >>On 11/14/06, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> >>
> >>
> >>>I am not quite sure I buy that argument since py.test tells you the time
> >>>that the test itself took (if you use the -v argument) or that all tests
> >>>together take (at the end of each test run). As far as I remember, these
> >>>times don't include any startup times. Yes, wall clock time of the whole
> >>>test run is quite a bad measure, but you don't have to use it anyway.
> >>>
> >>>Cheers,
> >>>
> >>>Carl Friedrich
> >>>
> >>>
> >>>
> >>I didn't realize that the time of *each* test is shown when the -v
> >>option is used.  Thanks for pointing that out.
> >>
> >>
> >
> >you're welcome :-)
> >
> >Note that I didn't mean to imply that it isn't worthwhile to make
> >py.test startup time smaller. it's definitely a good thing to do, since
> >testing should have low overhead.
> >
> >Cheers,
> >
> >Carl Friedrich
> >_______________________________________________
> >py-dev mailing list
> >py-dev at codespeak.net
> >http://codespeak.net/mailman/listinfo/py-dev
> >
> >
> I've profiled it a bit. Right now (for me) your tests are about 3x
> slower that unittests. It still can be improved somehow.
>
> Btw: Tests are unfair, cause collection encounters 5k tests for py.test
> and 13 for unittests, if you rewrite it so they'll be equal it would be
> better. It's not good excuse anyway ;-)
>
> I'll try to work a bit more on it, but I guess achieving 2x in this
> particular case would be enough. If you still encounter massive
> slowdowns in different enviroments, do not hesitate to contact us.
>
True, the code I sent you is not comparing the exact same tests and it
is unfair.
So, as I kept looking into unit test for Python, I found nose and its
nosetests that essentially does the same as py.test (without having
the same design goals as the Py lib).  I ran the same test on both
tools.

The results are:
- py.test tests in [4.30s, 4.17s, 4.28s, 4.42s]
- nosetests tests in [ 1.48s, 1.43s, 1.40s, 1.50s]

So it looks like achieving a 2x would make it run in the same ballpark
as nosetests (at least for that code).

I do like py.test error report better than nosetest though. The
reporting is less verbose.

py.test::

    _______________________________________________________________________________
    ________________________ entrypoint: test_bad_roman[0]
________________________

        def check(exception_cls, invalid_numeral):
            print 'Testing : ', invalid_numeral
    E       raises(exception_cls, roman.fromRoman, invalid_numeral)
    >       ExceptionFailure: DID NOT RAISE

    [D:\dev\python\test\test_roman.py:164]
    - - - - - - - - - - - - - -  [0]: recorded stdout - - - - - - - -
- - - - - - -
    Testing :  MMMM

    _______________________________________________________________________________
    ============ tests finished: 5023 passed, 1 failed in 4.17 seconds
============

and the same failure shown by nosetest::

    ======================================================================
    ERROR: test_roman.test_bad_roman:(<class
roman.InvalidRomanNumeralError at 0x00C5DDB0>, 'MMMM')
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "c:\python24\lib\site-packages\nose-0.9.1-py2.4.egg\nose\case.py",
line 52, in runTest
        self.testFunc()
      File "c:\python24\lib\site-packages\nose-0.9.1-py2.4.egg\nose\loader.py",
line 324, in run
        expr[0](*expr[1:])
      File "D:\dev\python\test\test_roman.py", line 164, in check
        raises(exception_cls, roman.fromRoman, invalid_numeral)
      File "C:\py-dist\py\test\raises.py", line 39, in raises
        expr=args, expected=ExpectedException)
    ExceptionFailure: DID NOT RAISE

    -------------------- >> begin captured stdout << ---------------------
    Testing :  MMMM

    --------------------- >> end captured stdout << ----------------------
    ----------------------------------------------------------------------
    Ran 5024 tests in 1.407s

    FAILED (errors=1)


Cheers,

P.R.


From prouleau001 at gmail.com  Wed Nov 15 18:25:02 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Wed, 15 Nov 2006 12:25:02 -0500
Subject: [py-dev] Fwd: Speed of py.test compared to unittest
In-Reply-To: <455B46E1.1090202@genesilico.pl>
References: <Pine.LNX.4.58.0611142133520.20193@martini.genesilico.pl>
	<Pine.LNX.4.58.0611142209370.20415@martini.genesilico.pl>
	<5c1d522d0611141318t6f224884p80ec1c91c4e8f327@mail.gmail.com>
	<5c1d522d0611141336v216cff28s15b9a004cb5276@mail.gmail.com>
	<455A3CE8.5000709@gmx.de>
	<5c1d522d0611141407gb35968fw56f74140d69db7d@mail.gmail.com>
	<455A3F92.8060300@gmx.de> <455AFA67.60109@genesilico.pl>
	<5c1d522d0611150837y12fc80d4y5cad8b16a235a4a0@mail.gmail.com>
	<455B46E1.1090202@genesilico.pl>
Message-ID: <5c1d522d0611150925o1bfc643alc5091ae67cd4678a@mail.gmail.com>

On 11/15/06, Maciek Fijalkowski <fijal at genesilico.pl> wrote:

> >>
> >> I've profiled it a bit. Right now (for me) your tests are about 3x
> >> slower that unittests. It still can be improved somehow.
> >>
> >> Btw: Tests are unfair, cause collection encounters 5k tests for py.test
> >> and 13 for unittests, if you rewrite it so they'll be equal it would be
> >> better. It's not good excuse anyway ;-)
> >>
> >> I'll try to work a bit more on it, but I guess achieving 2x in this
> >> particular case would be enough. If you still encounter massive
> >> slowdowns in different enviroments, do not hesitate to contact us.
> >>
> > True, the code I sent you is not comparing the exact same tests and it
> > is unfair.
> > So, as I kept looking into unit test for Python, I found nose and its
> > nosetests that essentially does the same as py.test (without having
> > the same design goals as the Py lib).  I ran the same test on both
> > tools.
> >
> > The results are:
> > - py.test tests in [4.30s, 4.17s, 4.28s, 4.42s]
> > - nosetests tests in [ 1.48s, 1.43s, 1.40s, 1.50s]
> >
> > So it looks like achieving a 2x would make it run in the same ballpark
> > as nosetests (at least for that code).
> >
> > I do like py.test error report better than nosetest though. The
> > reporting is less verbose.
> >

>
> Btw: it's updated version? (I did some improvements today/yesterday)
>
No!  I re-ran the tests with your latest version and I got faster
results! Great!

The results are now:  [ 1.94s, 1.92s, 1.92s, 1.98s, 1.92s]  which is
much closer to the nosetest results.  Note however, that when I run
nosetests, it executes the *exact* same test, which means that the
test file (test_roman.py) contains:

from py.test import raises

and it uses the raises() call to check that exceptions are raised.  I
am currently looking into nose to see how i could replace the
py.test.raises with nose equivalent.

Cheers,

P.R.


From fijal at genesilico.pl  Wed Nov 15 18:42:21 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Wed, 15 Nov 2006 18:42:21 +0100
Subject: [py-dev] Some fixes to minor typos inside strings
In-Reply-To: <5c1d522d0611150934i739621f3if57a40c683515ce2@mail.gmail.com>
References: <5c1d522d0611150934i739621f3if57a40c683515ce2@mail.gmail.com>
Message-ID: <455B517D.70502@genesilico.pl>

Pierre Rouleau wrote:

> Maciek,
>
> I noticed a couple of typos in the command line strings in
> test/cmdline.py and test/defaultconftest.py.
>
> Here's the diff for the changes I made locally.
>
> The universal diff output files are attached.
>
> Very minor changes, but ...
>
> P.R.
>
>------------------------------------------------------------------------
>
>Index: C:/py-dist/py/test/cmdline.py
>===================================================================
>--- C:/py-dist/py/test/cmdline.py	(revision 34637)
>+++ C:/py-dist/py/test/cmdline.py	(working copy)
>@@ -32,7 +32,7 @@
>         config.option.startserver = True
>     try:
>         if config.getinitialvalue('startserver'):
>-            py.std.warnings.warn("Startserver flag in config is deprecated, use commandline option istead")
>+            py.std.warnings.warn("Startserver flag in config is deprecated, use commandline option instead")
>     except ValueError:
>         pass
>     
>  
>
>------------------------------------------------------------------------
>
>Index: C:/py-dist/py/test/defaultconftest.py
>===================================================================
>--- C:/py-dist/py/test/defaultconftest.py	(revision 34637)
>+++ C:/py-dist/py/test/defaultconftest.py	(working copy)
>@@ -15,7 +15,7 @@
>     py.test.Config.addoptions('general options',
>         Option('-v', '--verbose',
>                action="count", dest="verbose", default=0,
>-               help="increase verbosity"),
>+               help="increase verbosity."),
>         Option('-x', '--exitfirst',
>                action="store_true", dest="exitfirst", default=False,
>                help="exit instantly on first error or failed test."),
>@@ -24,33 +24,33 @@
>                help="disable catching of sys.stdout/stderr output."),
>         Option('-k',
>                action="store", dest="keyword", default='',
>-               help="only run test items matching the given (google-style) keyword expression"),
>+               help="only run test items matching the given (google-style) keyword expression."),
>         Option('-l', '--showlocals',
>                action="store_true", dest="showlocals", default=False,
>-               help="show locals in tracebacks (disabled by default)"),
>+               help="show locals in tracebacks (disabled by default)."),
>         Option('', '--pdb',
>                action="store_true", dest="usepdb", default=False,
>                help="start pdb (the Python debugger) on errors."),
>         Option('', '--tb',
>                action="store", dest="tbstyle", default='long',
>                type="choice", choices=['long', 'short', 'no'],
>-               help="traceback verboseness (long/short/no)"),
>+               help="traceback verboseness (long/short/no)."),
>         Option('', '--fulltrace',
>                action="store_true", dest="fulltrace", default=False,
>-               help="don't cut any tracebacks (default is to cut)"),
>+               help="don't cut any tracebacks (default is to cut)."),
>         Option('', '--nomagic',
>                action="store_true", dest="nomagic", default=False,
>-               help="refrain from using magic as much as possible"),
>+               help="refrain from using magic as much as possible."),
>         Option('', '--collectonly',
>                action="store_true", dest="collectonly", default=False,
>-               help="only collect tests, don't execute them. "),
>+               help="only collect tests, don't execute them."),
>         Option('', '--traceconfig',
>                action="store_true", dest="traceconfig", default=False,
>-               help="trace considerations of conftest.py files. "),
>+               help="trace considerations of conftest.py files."),
>         Option('', '--apigen',
>                action="store_true", dest="apigen", default=False,
>-               help="Generated API docs out of tests. Needs to have defined"\
>-               "__package__ for module or overwritten in conftest")
>+               help="Generated API docs out of tests. Needs to have defined "\
>+               "__package__ for module or overwritten in conftest.")
>     )
>     py.test.Config.addoptions('test-session related options', 
>         Option('', '--tkinter',
>@@ -67,11 +67,11 @@
>                help="python executable to run the tests with. "),
>         Option('-w', '--startserver',
>                action="store_true", dest="startserver", default=False,
>-               help="Start HTTP server listening on localhost:8000 for test"
>+               help="Start HTTP server listening on localhost:8000 for test."
>                ),
>         Option('', '--runbrowser',
>                action="store_true", dest="runbrowser", default=False,
>-               help="Run browser to point to your freshly started web server"
>+               help="Run browser to point to your freshly started web server."
>                ),
>     )
>     
>  
>
Thx a lot. Commited. It really doesn't matter that the changes are 
minor, we appreciate any feedback.


From prouleau001 at gmail.com  Wed Nov 15 22:15:59 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Wed, 15 Nov 2006 16:15:59 -0500
Subject: [py-dev] Another very small typo in test/defaultconftest.py
Message-ID: <5c1d522d0611151315x2538d980q2584205e02d33fe2@mail.gmail.com>

I had made a mistake in my revious post and placed a perdio in a place
where a space character was needed: the help text for --apigen, just
before the '__package__'.

The diff that fixes it is attached.

Cheers,

P.R.
-------------- next part --------------
Index: C:/py-dist/py/test/defaultconftest.py
===================================================================
--- C:/py-dist/py/test/defaultconftest.py	(revision 34641)
+++ C:/py-dist/py/test/defaultconftest.py	(working copy)
@@ -49,7 +49,7 @@
                help="trace considerations of conftest.py files."),
         Option('', '--apigen',
                action="store_true", dest="apigen", default=False,
-               help="Generated API docs out of tests. Needs to have defined"\
+               help="Generated API docs out of tests. Needs to have defined "\
                "__package__ for module or overwritten in conftest.")
     )
     py.test.Config.addoptions('test-session related options', 

From prouleau001 at gmail.com  Thu Nov 16 21:54:47 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Thu, 16 Nov 2006 15:54:47 -0500
Subject: [py-dev] Failing to manage test state of a module with py.test
Message-ID: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>

Hi all,

Using the latest of py.test (svn 34683), I can't seem to be able to
control a module test setup using setup_module() as described in
section 2.15 of http://codespeak.net/py/current/doc/test.html

The document gives the following example:

def setup_module(module):
    """ setup up any state specific to the execution
        of the given module.
    """

Now, I am assuming that the the definition of setup_module() must be
written /inside/ the test script that test the module-under-test
tested, right?

Second, I am also assuming that setup_module's argument is the name of
the module-under-test.

Given the above assumptions, if I write a test script to test roman.py
 would be called test_roman.py and would include

def setup_module(roman):
    print 'SETTING up roman for testing....'

And if I ran py.test -s test_roman.py I should be able to see the
printed output. I don't.

So, what do i do wrong here?

Thanks


From fijal at genesilico.pl  Thu Nov 16 22:04:02 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Thu, 16 Nov 2006 22:04:02 +0100
Subject: [py-dev] Failing to manage test state of a module with py.test
In-Reply-To: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>
References: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>
Message-ID: <455CD242.8080507@genesilico.pl>

Pierre Rouleau wrote:

>Hi all,
>
>Using the latest of py.test (svn 34683), I can't seem to be able to
>control a module test setup using setup_module() as described in
>section 2.15 of http://codespeak.net/py/current/doc/test.html
>
>The document gives the following example:
>
>def setup_module(module):
>    """ setup up any state specific to the execution
>        of the given module.
>    """
>
>Now, I am assuming that the the definition of setup_module() must be
>written /inside/ the test script that test the module-under-test
>tested, right?
>
>Second, I am also assuming that setup_module's argument is the name of
>the module-under-test.
>
>Given the above assumptions, if I write a test script to test roman.py
> would be called test_roman.py and would include
>
>def setup_module(roman):
>    print 'SETTING up roman for testing....'
>
>And if I ran py.test -s test_roman.py I should be able to see the
>printed output. I don't.
>
>So, what do i do wrong here?
>
>Thanks
>_______________________________________________
>py-dev mailing list
>py-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/py-dev
>  
>
Hum. I cannot reproduce you problem (I can clearly see 'SETTING up 
roman...' just after [5024] and before any dots. Yes, this is supposed 
to go to main body of testing module and it receives module object 
itself (not a name)


From prouleau001 at gmail.com  Thu Nov 16 22:22:14 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Thu, 16 Nov 2006 16:22:14 -0500
Subject: [py-dev] Failing to manage test state of a module with py.test
In-Reply-To: <455CD242.8080507@genesilico.pl>
References: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>
	<455CD242.8080507@genesilico.pl>
Message-ID: <5c1d522d0611161322t3bfa2ed7od4dc838b14487261@mail.gmail.com>

On 11/16/06, Maciek Fijalkowski <fijal at genesilico.pl> wrote:
> Pierre Rouleau wrote:
>
> >Hi all,
> >
> >Using the latest of py.test (svn 34683), I can't seem to be able to
> >control a module test setup using setup_module() as described in
> >section 2.15 of http://codespeak.net/py/current/doc/test.html
> >
> >The document gives the following example:
> >
> >def setup_module(module):
> >    """ setup up any state specific to the execution
> >        of the given module.
> >    """
> >
> >Now, I am assuming that the the definition of setup_module() must be
> >written /inside/ the test script that test the module-under-test
> >tested, right?
> >
> >Second, I am also assuming that setup_module's argument is the name of
> >the module-under-test.
> >
> >Given the above assumptions, if I write a test script to test roman.py
> > would be called test_roman.py and would include
> >
> >def setup_module(roman):
> >    print 'SETTING up roman for testing....'
> >
> >And if I ran py.test -s test_roman.py I should be able to see the
> >printed output. I don't.
> >
> >So, what do i do wrong here?
> >
> >Thanks
> >_______________________________________________
> >py-dev mailing list
> >py-dev at codespeak.net
> >http://codespeak.net/mailman/listinfo/py-dev
> >
> >
> Hum. I cannot reproduce you problem (I can clearly see 'SETTING up
> roman...' just after [5024] and before any dots. Yes, this is supposed
> to go to main body of testing module and it receives module object
> itself (not a name)
>

Sorry, i also got it working if all the setup_module() does is
printing the message.

I forgot to mention that the setup_module() was setting the value of
knownValues after declaring it global.  I though that it would then be
available to all test functions (as is the case with nose), but it's
not.

The setup_module I have looks like this:

def setup_module(roman):
    print 'SETTING UP........................'

    global knownValues
    knownValues = ( (1, 'I'),
                    (2, 'II'),
                    (3, 'III'),
                    (4, 'IV'),
                    (5, 'V'),
                    (6, 'VI'),
                    (7, 'VII'),
                    ....


I wanted to know if the values set up by the setup_module function
would be available to the test functions.  The test function that
fails is failing because knownValues is not accessible to the check
inside test_known_values.

def test_known_values() :
    """Test all known values."""

    def check(number, roman_numeral):
        print 'test_known_values Testing: ', roman_numeral
        assert roman.toRoman(number) == roman_numeral
        assert roman.fromRoman(roman_numeral) == number
        # all Roman numerals should be in upper case
        assert roman_numeral.upper() == roman_numeral
        # lower case Roman numerals are not accepted
        raises(roman.InvalidRomanNumeralError, roman.fromRoman,
roman_numeral.lower())

    tested_numbers = []

    # test the numbers in the table above
    for number, roman_numeral in knownValues:
        yield check, number, roman_numeral
        tested_numbers.append(number)

    # test the others
    for number in xrange(1,5000):
        if number not in tested_numbers:
            yield check, number, roman.toRoman(number)



Note that nose is able to recover the global variable and the test
succeeds in nosetests. py.test does not seem able to do it.  It's not
that I absolutely want to use global variables, but I wanted to checks
the module setup mechanism.

--

P.R.


From fijal at genesilico.pl  Thu Nov 16 22:34:41 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Thu, 16 Nov 2006 22:34:41 +0100
Subject: [py-dev] Failing to manage test state of a module with py.test
In-Reply-To: <5c1d522d0611161322t3bfa2ed7od4dc838b14487261@mail.gmail.com>
References: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>	
	<455CD242.8080507@genesilico.pl>
	<5c1d522d0611161322t3bfa2ed7od4dc838b14487261@mail.gmail.com>
Message-ID: <455CD971.5010402@genesilico.pl>

Pierre Rouleau wrote:

> On 11/16/06, Maciek Fijalkowski <fijal at genesilico.pl> wrote:
>
>> Pierre Rouleau wrote:
>>
>> >Hi all,
>> >
>> >Using the latest of py.test (svn 34683), I can't seem to be able to
>> >control a module test setup using setup_module() as described in
>> >section 2.15 of http://codespeak.net/py/current/doc/test.html
>> >
>> >The document gives the following example:
>> >
>> >def setup_module(module):
>> >    """ setup up any state specific to the execution
>> >        of the given module.
>> >    """
>> >
>> >Now, I am assuming that the the definition of setup_module() must be
>> >written /inside/ the test script that test the module-under-test
>> >tested, right?
>> >
>> >Second, I am also assuming that setup_module's argument is the name of
>> >the module-under-test.
>> >
>> >Given the above assumptions, if I write a test script to test roman.py
>> > would be called test_roman.py and would include
>> >
>> >def setup_module(roman):
>> >    print 'SETTING up roman for testing....'
>> >
>> >And if I ran py.test -s test_roman.py I should be able to see the
>> >printed output. I don't.
>> >
>> >So, what do i do wrong here?
>> >
>> >Thanks
>> >_______________________________________________
>> >py-dev mailing list
>> >py-dev at codespeak.net
>> >http://codespeak.net/mailman/listinfo/py-dev
>> >
>> >
>> Hum. I cannot reproduce you problem (I can clearly see 'SETTING up
>> roman...' just after [5024] and before any dots. Yes, this is supposed
>> to go to main body of testing module and it receives module object
>> itself (not a name)
>>
>
> Sorry, i also got it working if all the setup_module() does is
> printing the message.
>
> I forgot to mention that the setup_module() was setting the value of
> knownValues after declaring it global.  I though that it would then be
> available to all test functions (as is the case with nose), but it's
> not.
>
> The setup_module I have looks like this:
>
> def setup_module(roman):
>    print 'SETTING UP........................'
>
>    global knownValues
>    knownValues = ( (1, 'I'),
>                    (2, 'II'),
>                    (3, 'III'),
>                    (4, 'IV'),
>                    (5, 'V'),
>                    (6, 'VI'),
>                    (7, 'VII'),
>                    ....
>
>
> I wanted to know if the values set up by the setup_module function
> would be available to the test functions.  The test function that
> fails is failing because knownValues is not accessible to the check
> inside test_known_values.
>
> def test_known_values() :
>    """Test all known values."""
>
>    def check(number, roman_numeral):
>        print 'test_known_values Testing: ', roman_numeral
>        assert roman.toRoman(number) == roman_numeral
>        assert roman.fromRoman(roman_numeral) == number
>        # all Roman numerals should be in upper case
>        assert roman_numeral.upper() == roman_numeral
>        # lower case Roman numerals are not accepted
>        raises(roman.InvalidRomanNumeralError, roman.fromRoman,
> roman_numeral.lower())
>
>    tested_numbers = []
>
>    # test the numbers in the table above
>    for number, roman_numeral in knownValues:
>        yield check, number, roman_numeral
>        tested_numbers.append(number)
>
>    # test the others
>    for number in xrange(1,5000):
>        if number not in tested_numbers:
>            yield check, number, roman.toRoman(number)
>
>
>
> Note that nose is able to recover the global variable and the test
> succeeds in nosetests. py.test does not seem able to do it.  It's not
> that I absolutely want to use global variables, but I wanted to checks
> the module setup mechanism.
>
> -- 
>
> P.R.

Of course tha you just write:

def setup_module(mod):
    mod.some_global_value = whatever_you_like

:-) It's of course better, because explicit is better than implicit (you 
avoid exporting all variables used in setup_)


From prouleau001 at gmail.com  Thu Nov 16 22:43:30 2006
From: prouleau001 at gmail.com (Pierre Rouleau)
Date: Thu, 16 Nov 2006 16:43:30 -0500
Subject: [py-dev] Failing to manage test state of a module with py.test
In-Reply-To: <455CD971.5010402@genesilico.pl>
References: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>
	<455CD242.8080507@genesilico.pl>
	<5c1d522d0611161322t3bfa2ed7od4dc838b14487261@mail.gmail.com>
	<455CD971.5010402@genesilico.pl>
Message-ID: <5c1d522d0611161343p317b142dl5a76d9d8aaffd97e@mail.gmail.com>

On 11/16/06, Maciek Fijalkowski <fijal at genesilico.pl> wrote:
> Pierre Rouleau wrote:
>

> >
> > I forgot to mention that the setup_module() was setting the value of
> > knownValues after declaring it global.  I though that it would then be
> > available to all test functions (as is the case with nose), but it's
> > not.
> >
> > The setup_module I have looks like this:
> >
> > def setup_module(roman):
> >    print 'SETTING UP........................'
> >
> >    global knownValues
> >    knownValues = ( (1, 'I'),
> >                    (2, 'II'),

> > Note that nose is able to recover the global variable and the test
> > succeeds in nosetests. py.test does not seem able to do it.  It's not
> > that I absolutely want to use global variables, but I wanted to checks
> > the module setup mechanism.
> >
>
> Of course tha you just write:
>
> def setup_module(mod):
>     mod.some_global_value = whatever_you_like
>
> :-) It's of course better, because explicit is better than implicit (you
> avoid exporting all variables used in setup_)
>

But of course!!! what was I thinking! ;)  Now I see why there is a
module argument!!

Thanks!


From hpk at trillke.net  Fri Nov 17 08:17:46 2006
From: hpk at trillke.net (holger krekel)
Date: Fri, 17 Nov 2006 08:17:46 +0100
Subject: [py-dev] Failing to manage test state of a module with py.test
In-Reply-To: <455CD971.5010402@genesilico.pl>
References: <5c1d522d0611161254n33f5c0adyf7cbc82029ce4298@mail.gmail.com>
	<455CD242.8080507@genesilico.pl>
	<5c1d522d0611161322t3bfa2ed7od4dc838b14487261@mail.gmail.com>
	<455CD971.5010402@genesilico.pl>
Message-ID: <20061117071746.GG32728@solar.trillke.net>

On Thu, Nov 16, 2006 at 22:34 +0100, Maciek Fijalkowski wrote:
> Pierre Rouleau wrote:
> > def setup_module(roman):
> >    print 'SETTING UP........................'
> >
> >    global knownValues
> >    knownValues = ( (1, 'I'),
> >                    (2, 'II'),
> >                    (3, 'III'),
> >                    (4, 'IV'),
> >                    (5, 'V'),
> >                    (6, 'VI'),
> >                    (7, 'VII'),
> >                    ....
> >
> >
> > I wanted to know if the values set up by the setup_module function
> > would be available to the test functions.  [...]
> >
> Of course tha you just write:
> 
> def setup_module(mod):
>     mod.some_global_value = whatever_you_like
> 
> :-) It's of course better, because explicit is better than implicit (you 
> avoid exporting all variables used in setup_)

yes, that is the usual way, but i am not sure why Pierre's code 
fails, there is no obvious reason, unless the setup_module 
is actually imported from somewhere else in which case
the "test_roman" module source and the setup_module 
would not share the same "globals" - this is the reason 
why it is better to use the argument, because you can then 
share "setup_module" code across various test modules. 

best, 

    holger