From holger at merlinux.de Fri Feb 1 11:58:20 2008 From: holger at merlinux.de (holger krekel) Date: Fri, 1 Feb 2008 11:58:20 +0100 Subject: [py-dev] "event" branch and architecture Message-ID: <20080201105820.GC3020@solar.trillke> Hi py-dev! i am currently hammering on the svn/py/branch/event branch [*]. The main intention is to modify py.test's internal architecture towards using events and to base reporting and some internal processing on these events. Here is the basic event loop (exposed as py.event.Hub for now): class Hub(list): """ General Event Hub """ def notify(self, event): for subscriber in self: subscriber(event) This is inspired by a talk from Jim Fulton about zope3's event loop - zope.event just contains a global list and a notify function like the above. However, i'd like to keep things more local for now (going global is usually much easier than going local, to begin with and local state is also nicer for TDD) For py.test the upcoming event archicture should allow us to e.g.: * have co-existing reporters e.g. one that writes test results to an internal database and one that provides terminal reporting. * Or one terminal and one html-output reporter. * GUI based reporters (where we can easily route events to GUI loops) * Or a small plugin that allows to automatically copy failure information to e.g. http://paste.pocoo.org/. * ... I am also using the event branch to do various cleanups and refactorings, mainly i am aiming at refactoring distributed and remote testing to share more code (and Events) with local testing. I hope that in the end one can write a reporter that does not need to care (too much) how tests are/were distributed across machines. best & cheers, holger [*] In fact i am working there in py/test2 - which is a branch itself of py/test. I did this (with the help of the py/bin/_maketest2.py helper, btw) in order to be able to use py.test for refactoring py.test. Otherwise if i change files in py/test/ and try to run py.test it would affect my very own test run, making TDD hard or impossible. -- Holger Krekel - freelance manager and programmer merlinux GmbH: http://merlinux.de py.test: http://pytest.org py lib: http://pylib.org PyPy: http://codespeak.net/pypy From tav at espians.com Fri Feb 1 12:21:20 2008 From: tav at espians.com (tav) Date: Fri, 1 Feb 2008 11:21:20 +0000 Subject: [py-dev] "event" branch and architecture In-Reply-To: <20080201105820.GC3020@solar.trillke> References: <20080201105820.GC3020@solar.trillke> Message-ID: <95d8c0810802010321h429ecc74r6a7015785a99ad9c@mail.gmail.com> Hey Holger, > i am currently hammering on the svn/py/branch/event branch [*]. > The main intention is to modify py.test's internal architecture > towards using events and to base reporting and some internal > processing on these events. Whilst Zope's event system is nice in its simplicity, it suffers from many faults once you go beyond the basic use cases. The easiest one to explain is race conditions that take place with regards to nested subscriptions. That is, when calling a subscriber/listener with an event, if that action adds a new listener... and/or notifies the Hub... you have a lot of pain to work around. Now, throw in exceptions into that and *boom*, you have a rather messy minefield. Can I recommend using Trellis by any chance? * http://peak.telecommunity.com/DevCenter/Trellis It solves a lot of very difficult problems rather elegantly and the core is just 450 lines of code!! Whilst the documentation is a tad confusing, it does have a lot of it and some parts are written very nicely. -- hope-this-helps, tav founder and ceo, esp metanational llp plex:espians/tav | tav at espians.com | +44 (0) 7809 569 369 From holger at merlinux.de Fri Feb 1 12:45:53 2008 From: holger at merlinux.de (holger krekel) Date: Fri, 1 Feb 2008 12:45:53 +0100 Subject: [py-dev] "event" branch and architecture In-Reply-To: <95d8c0810802010321h429ecc74r6a7015785a99ad9c@mail.gmail.com> References: <20080201105820.GC3020@solar.trillke> <95d8c0810802010321h429ecc74r6a7015785a99ad9c@mail.gmail.com> Message-ID: <20080201114553.GE3020@solar.trillke> Hey tav! On Fri, Feb 01, 2008 at 11:21 +0000, tav wrote: > Hey Holger, > > > i am currently hammering on the svn/py/branch/event branch [*]. > > The main intention is to modify py.test's internal architecture > > towards using events and to base reporting and some internal > > processing on these events. > > Whilst Zope's event system is nice in its simplicity, it suffers from > many faults once you go beyond the basic use cases. The easiest one to > explain is race conditions that take place with regards to nested > subscriptions. That is, when calling a subscriber/listener with an > event, if that action adds a new listener... and/or notifies the > Hub... you have a lot of pain to work around. I can see that triggering events nestedly can be problematic. But I am not sure it is a problem for the upcoming py.test use cases and if so i think we could extend the hub's notify protocol to allow subscribers to return new events that are to trigger notification after the current run is through. > Now, throw in exceptions into that and *boom*, you have a rather messy > minefield. Well, subscribers need to take care for dealing with exceptions. We could do a "try-except" but then what do we do with the exception, anyway? For me, the beauty of the z3 event loop lies in the fact that every python programmer can understand easily what he is dealing with if wanting to have a subscriber (or notifier) participate. > Can I recommend using Trellis by any chance? sure, you are welcome! > * http://peak.telecommunity.com/DevCenter/Trellis > > It solves a lot of very difficult problems rather elegantly and the > core is just 450 lines of code!! Whilst the documentation is a tad > confusing, it does have a lot of it and some parts are written very > nicely. Introducing events for py.test particularly aims at *decoupling* things, also to allow for "plugin" based features. Isn't Trellis rather aiming at coupling things automatically? I haven't looked at the Trellis code (is it using other PEAK code btw?) but from reading the web page i appears to me that Trellis is a framework rather than a simple pattern - the py lib avoids to do frameworkish things like introducing subclassing requirements. best & cheers, holger -- Holger Krekel - freelance manager and programmer merlinux GmbH: http://merlinux.de py.test: http://pytest.org py lib: http://pylib.org PyPy: http://codespeak.net/pypy From holger at merlinux.de Fri Feb 1 22:57:05 2008 From: holger at merlinux.de (holger krekel) Date: Fri, 1 Feb 2008 22:57:05 +0100 Subject: [py-dev] py.execnet: hang in channel.receive() In-Reply-To: References: Message-ID: <20080201215705.GO3020@solar.trillke> Hi Bob, could you checkout http://codespeak.net/svn/pypy/branch/bugfix-0.9.0 change to the checked out directory and perform your test below again? This is a vanilla 0.9.0 with debugging info enabled. It creates debug files in /tmp/execnet-debug-PROCESS-ID, one for the client and for the spawned site. Could you forward the contents of these files? best, holger On Thu, Jan 31, 2008 at 16:29 -0500, Bob Kuehne wrote: > hi gang, > > i first saw holger present py.execnet at pycon a few years back, and > have been waiting to use this cool tool. so now i'm on it, and i've > got a problem. i'm running this on two macs, both running os x 10.5.1. > i've installed py via easy_install, so i've got a luverly egg, and > here's my script that i'm trying to just test with: > > -- > import py > gw = py.execnet.SshGateway( "localhost" ) > channel = gw.remote_exec( 'channel.send( [ [ False, True, False, > 0, ] ] )' ) > print channel.receive() > channel.close() > gw.exit() > -- > > so, on mac numero 1, running python 2.5.1, i see the output i'd expect: > > -- > Python 2.5.1 (r251:54863, Oct 5 2007, 21:08:09) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import py > >>> gw = py.execnet.SshGateway( "localhost" ) > >>> channel = gw.remote_exec( 'channel.send( [ [ False, True, False, > 0, ] ] )' ) > >>> print channel.receive() > [[False, True, False, 0]] > >>> channel.close() > >>> gw.exit() > -- > > but on another mac, also python 2.5.1, i see the same, but the > 'channel.receive()' is hung. if i break there (control-c), i get the > following: > > -- > Python 2.5.1 (r251:54863, Oct 9 2007, 20:39:46) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import py > >>> gw = py.execnet.SshGateway( "localhost" ) > >>> channel = gw.remote_exec( 'channel.send( [ [ False, True, False, > 0, ] ] )' ) > >>> print channel.receive() > ^CKilled by signal 2. > Traceback (most recent call last): > File "", line 1, in > File "/Library/Python/2.5/site-packages/py-0.9.0-py2.5.egg/py/ > execnet/channel.py", line 172, in receive > x = queue.get() > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/Queue.py", line 165, in get > self.not_empty.wait() > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/threading.py", line 214, in wait > waiter.acquire() > KeyboardInterrupt > -- > > now, the question is what's different. and it's at this point that i > wave my arms and go, wtf? i really don't know, but i'm wondering aloud > if anyone else has seen this sort of behavior, and if so, what they > might have encountered or changed to get it to be better behaved. > > thanks, > bob > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Holger Krekel - freelance manager and programmer merlinux GmbH: http://merlinux.de py.test: http://pytest.org py lib: http://pylib.org PyPy: http://codespeak.net/pypy From brian at dorseys.org Tue Feb 5 09:25:10 2008 From: brian at dorseys.org (Brian Dorsey) Date: Tue, 5 Feb 2008 00:25:10 -0800 Subject: [py-dev] Is py lib advertising 'initpkg' ? Message-ID: <66e877b70802050025s2d40b92cu9256fc1c1f08ca3e@mail.gmail.com> I ran into this comp.lang.python post today: http://groups.google.com/group/comp.lang.python/browse_thread/thread/5a9580e76484f73/ Seems like initpkg might be a good solution for the poster... however, I didn't see any reference to initpkg in any of the docs for py lib. Is it intended to be an implementation detail, or a tool for others to use? Take care, -Brian From holger at merlinux.de Tue Feb 5 11:04:26 2008 From: holger at merlinux.de (holger krekel) Date: Tue, 5 Feb 2008 11:04:26 +0100 Subject: [py-dev] Is py lib advertising 'initpkg' ? In-Reply-To: <66e877b70802050025s2d40b92cu9256fc1c1f08ca3e@mail.gmail.com> References: <66e877b70802050025s2d40b92cu9256fc1c1f08ca3e@mail.gmail.com> Message-ID: <20080205100426.GH31519@solar.trillke> Hi Brian! On Tue, Feb 05, 2008 at 00:25 -0800, Brian Dorsey wrote: > I ran into this comp.lang.python post today: > http://groups.google.com/group/comp.lang.python/browse_thread/thread/5a9580e76484f73/ > > Seems like initpkg might be a good solution for the poster... however, > I didn't see any reference to initpkg in any of the docs for py lib. > Is it intended to be an implementation detail, or a tool for others to > use? good question! i very quickly skimmed the above (huge) thread. Indeed, the goal of separating a package's namespace organisation from file locations on disk is a main motivation of py' initpkg. I am using initpkg in a couple of other projects with PKG/__init__.py containing: from py import initpkg initpkg("vadm", ... exportdefs={ ... } ) However, there indeed isn't documentation on using it. The initpkg mechanism is not using __import__ hooks but subclasses the Module Class. This also provides lazy importing facilities and greatly reduces the need for "import" lines at the header of files. It also gives good control on which names exactly are seen from package's users. IMO feel free to carefully point him to the mechanism and invite him to to try out. He may also subscribe here in order to discuss more or get support - i likely won't make it to post to c.l.py myself. best, holger From johnny at johnnydebris.net Thu Feb 7 12:41:18 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Thu, 07 Feb 2008 12:41:18 +0100 Subject: [py-dev] py.path.svn* username and password support Message-ID: <47AAEE5E.5030203@johnnydebris.net> Hi! Someone asked me if I want to implement support for passing a username and password to the py.path.svn* functionality, allowing the application to provide a username and pw both for svnurls and svnwcs. I've investigated SVN and the code a bit, and it looks like this can be done in a relatively nice way: * svn seems to support command-line arguments for username and password, and also for telling it to not store auth information - with these arguments we can make sure the username/pw we provide are used, and not stored on the client * there's a switch --non-interactive which we can then enable to make very sure applications don't hang on asking username/password or things like that (currently that can be a problem when using svnwc or svnurl in scripts) I guess this would mean that for all operations that hit the server, the username and password arguments should be added. Perhaps it would be nice to have to provide it only on instantiation or something, although I fear that that will be harder to implement and use correctly in the situation of nested working copies (externals), etc. I hope you guys like the idea of having this in the py lib, it sounds like a useful addition to me... If no-one minds, I'll start implementing in a couple of days, I'd like to have it done (and tested well, obviously) before the release, if possible... Please let me know what you think. Cheers, Guido From holger at merlinux.de Thu Feb 7 19:06:39 2008 From: holger at merlinux.de (holger krekel) Date: Thu, 7 Feb 2008 19:06:39 +0100 Subject: [py-dev] py.path.svn* username and password support In-Reply-To: <47AAEE5E.5030203@johnnydebris.net> References: <47AAEE5E.5030203@johnnydebris.net> Message-ID: <20080207180638.GL17687@solar.trillke> Hi Guido! On Thu, Feb 07, 2008 at 12:41 +0100, Guido Wesdorp wrote: > Hi! > > Someone asked me if I want to implement support for passing a username > and password to the py.path.svn* functionality, allowing the application > to provide a username and pw both for svnurls and svnwcs. I've > investigated SVN and the code a bit, and it looks like this can be done > in a relatively nice way: > > * svn seems to support command-line arguments for username and password, > and also for telling it to not store auth information - with these > arguments we can make sure the username/pw we provide are used, and not > stored on the client why do you want to make sure this is not stored on the fs? > * there's a switch --non-interactive which we can then enable to make > very sure applications don't hang on asking username/password or things > like that (currently that can be a problem when using svnwc or svnurl in > scripts) Sometimes py.path.svn* objects are used in interactively running scripts in which case it's ok that they ask for input. > I guess this would mean that for all operations that hit the server, the > username and password arguments should be added. Perhaps it would be > nice to have to provide it only on instantiation or something, although > I fear that that will be harder to implement and use correctly in the > situation of nested working copies (externals), etc. Not sure about externals but i guess it's possible to allow for roughly this to work: wc = py.path.svnwc("somepath") wc.checkout(URI, username=xxx, password=yyy) for subwcpath in wc.listdir(): assert subwcpath.username # inherited auth information assert subwcpath.password ... However, 1) inheriting auth information may not always be what one wants 2) it's simpler to rely on the client FS storing auth data > I hope you guys like the idea of having this in the py lib, it sounds > like a useful addition to me... If no-one minds, I'll start implementing > in a couple of days, I'd like to have it done (and tested well, > obviously) before the release, if possible... sure - let me note here that due to my prolonged ill-ish-ness (since last saturday now) and other surprising developments there may be no full 1.0 in Feb - in this case i'd suggest to first go for 0.9.1, with just bugfixes and a few added feaures. I hope that you and others could help with this. best, holger From schmir at gmail.com Thu Feb 7 19:27:36 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Thu, 7 Feb 2008 19:27:36 +0100 Subject: [py-dev] __package__ attribute used by python 2.6 Message-ID: <932f8baf0802071027h240d5431m8e97bd2eb5c5988b@mail.gmail.com> Hi all, __package__ is used extensively throughout the py lib. Unfortunately it will also be used by python 2.6 (see http://www.python.org/dev/peps/pep-0366/) I have replaced all occurences of __package__ with __pypackage__ to make py 0.9.0 work with python 2.6. I didn't run the testsuite but py.test seemed to work fine afterwards. Any plans to fix it? - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.de Fri Feb 8 09:48:45 2008 From: holger at merlinux.de (holger krekel) Date: Fri, 8 Feb 2008 09:48:45 +0100 Subject: [py-dev] __package__ attribute used by python 2.6 In-Reply-To: <932f8baf0802071027h240d5431m8e97bd2eb5c5988b@mail.gmail.com> References: <932f8baf0802071027h240d5431m8e97bd2eb5c5988b@mail.gmail.com> Message-ID: <20080208084845.GQ17687@solar.trillke> Hi Ralf, On Thu, Feb 07, 2008 at 19:27 +0100, Ralf Schmitt wrote: > Hi all, > > __package__ is used extensively throughout the py lib. > Unfortunately it will also be used by python 2.6 (see > http://www.python.org/dev/peps/pep-0366/) > I have replaced all occurences of __package__ with __pypackage__ to make py > 0.9.0 work with python 2.6. > I didn't run the testsuite but py.test seemed to work fine afterwards. > Any plans to fix it? i renamed it now to __pkg__ on http://codespeak.net/svn/py/trunk http://codespeak.net/svn/py/branch/bugfix-0.9.0 best & thanks, holger -- Holger Krekel - freelance manager and programmer merlinux GmbH: http://merlinux.de py.test: http://pytest.org py lib: http://pylib.org PyPy: http://codespeak.net/pypy From johnny at johnnydebris.net Sat Feb 9 15:03:24 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Sat, 09 Feb 2008 15:03:24 +0100 Subject: [py-dev] 0.9.0:HEAD revision overview for 0.9.1 release Message-ID: <47ADB2AC.3050207@johnnydebris.net> Hi! I went through all the checkins between r 38967 and the current HEAD and made lists of whether they were small bugfixes, refactorings and doc typos, or contain, or are part of, small or larger features. This hopefully helps when finding out what to port to the 0.9.1 branch for release... The document is attached. Cheers, Guido -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 091_revisions.txt URL: From johnny at johnnydebris.net Tue Feb 12 10:19:40 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Tue, 12 Feb 2008 10:19:40 +0100 Subject: [py-dev] Some ideas Message-ID: <47B164AC.5040108@johnnydebris.net> Hi! I thought a bit about the authentication feature for py.path.svn*, and wrote my ideas in a small doc. The doc is attached - please discuss if you're interested... Cheers, Guido -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: auth.txt URL: From holger at merlinux.de Tue Feb 12 12:09:15 2008 From: holger at merlinux.de (holger krekel) Date: Tue, 12 Feb 2008 12:09:15 +0100 Subject: [py-dev] Some ideas In-Reply-To: <47B164AC.5040108@johnnydebris.net> References: <47B164AC.5040108@johnnydebris.net> Message-ID: <20080212110915.GY17687@solar.trillke> Hi Guido! On Tue, Feb 12, 2008 at 10:19 +0100, Guido Wesdorp wrote: > I thought a bit about the authentication feature for py.path.svn*, and > wrote my ideas in a small doc. The doc is attached - please discuss if > you're interested... nice way of doing this, thanks! > SVN authentication support > ========================== > > This document describes authentication support to both py.path.svnwc and > py.path.svnurl (yet to be implemented). This should allow using the library in > a completely automated situation, without having to provide the credentials > interactively. > > Current ideas for implementation > -------------------------------- > > The credentials are passed to the constructor of the path objects, and are used > (transparently) for every action that accesses the server. Also, when provided, > they are passed recursively to all child objects created by methods such as > join(), ensure(), etc. Transparently inheriting AUTH info might sometimes not be desired or be tricky, not sure. I'd try this after all else is done (see below). > * Many new arguments required > > I think it's not very nice to have so many new arguments to the constructors, > and especially dislike having an argument that's only useful when other args > are provided (auth_cache). Not sure how to improve that, though... What about a py.path.SvnAuth class ... * that groks all these options * can create the proper commandline switches (and unit-testing particularly this) * then add an explicit 'svnauth' parameter to wc.checkout and wc.commit (which other svnwc methods need it?) * add an explicit 'svnauth' param to svnurl methods * -- * add an explicit 'svnauth' param to the svnurl constructor and experiment with inheriting it around I think that an "SvnAuth" instance should also be able to signal "just use defaults/whatever is stored on the FS". Eventually we might want to generalize SvnAuth slightly into some Identity or Auth class that can also be used e.g. for py.execnet.SshGateway. HTH, holger -- Holger Krekel - freelance manager and programmer merlinux GmbH: http://merlinux.de py.test: http://pytest.org py lib: http://pylib.org PyPy: http://codespeak.net/pypy From johnny at johnnydebris.net Tue Feb 12 12:56:16 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Tue, 12 Feb 2008 12:56:16 +0100 Subject: [py-dev] Some ideas In-Reply-To: <20080212110915.GY17687@solar.trillke> References: <47B164AC.5040108@johnnydebris.net> <20080212110915.GY17687@solar.trillke> Message-ID: <47B18960.8060902@johnnydebris.net> holger krekel wrote: > Transparently inheriting AUTH info might sometimes not be desired > or be tricky, not sure. I'd try this after all else is done (see below). > > Not sure about this one. If you do a checkout with authentication args, all children will be checked out with the same credentials. If you later perform an action on that child that requires authentication, I think the expected behaviour is that the credentials used for the previous checkout are used, unless other credentials are provided. >> * Many new arguments required >> >> I think it's not very nice to have so many new arguments to the constructors, >> and especially dislike having an argument that's only useful when other args >> are provided (auth_cache). Not sure how to improve that, though... >> > > What about a py.path.SvnAuth class ... > > I was thinking about something like that, but figured it would be kind of heavy for just the SVN auth... However, > Eventually we might want to generalize SvnAuth slightly > into some Identity or Auth class that can also be > used e.g. for py.execnet.SshGateway. > > in that case it may be nice to have, indeed... ;) Then... > * then add an explicit 'svnauth' parameter to wc.checkout and wc.commit > (which other svnwc methods need it?) Well, that depends on the use case. For a repository that requires authentication for all types of access, if the path objects aren't created with credentials passed to the constructor, it may be needed for all actions that hit the server (e.g. update, lock, etc.). Perhaps it makes more sense to only allow passing auth to the constructor? I don't really see a use case for using different credentials for different methods, but I may overlook something? Cheers, Guido From holger at merlinux.de Tue Feb 12 14:04:19 2008 From: holger at merlinux.de (holger krekel) Date: Tue, 12 Feb 2008 14:04:19 +0100 Subject: [py-dev] Some ideas In-Reply-To: <47B18960.8060902@johnnydebris.net> References: <47B164AC.5040108@johnnydebris.net> <20080212110915.GY17687@solar.trillke> <47B18960.8060902@johnnydebris.net> Message-ID: <20080212130418.GZ17687@solar.trillke> On Tue, Feb 12, 2008 at 12:56 +0100, Guido Wesdorp wrote: > holger krekel wrote: > > Transparently inheriting AUTH info might sometimes not be desired > > or be tricky, not sure. I'd try this after all else is done (see below). > > > > > Not sure about this one. If you do a checkout with authentication args, > all children will be checked out with the same credentials. If you later > perform an action on that child that requires authentication, I think > the expected behaviour is that the credentials used for the previous > checkout are used, unless other credentials are provided. This is what SVN cares for itself by storing credentials in the FS. As to in-process (somewhat implicit) inheritance of auth information: I hold that it might be complicated to get it right and and simple enough. IMO all methods modifying the repo need to accept an explicit auth parameter independently if there also is auth-inheritance-starting-from-constructor going on or not. Due to the many options that an SvnAuth class needs to handle i'd like to avoid cluttering the "parameter namespace" of the svn path constructors. The Constructors are anyway already a bit messy IMO. Also it's easier for testing to encapsulate things in a class and make sure it generates the right command line switches etc. when asked for it. So what about going ahead with implementing this class and using it for the few svnwc methods? The SvnAuth class and the issue of inheritance are orthogonal issues. Therefore much testing and work can be done independently. I suggest we don't discuss much more currently if that is ok with you. holger > >> * Many new arguments required > >> > >> I think it's not very nice to have so many new arguments to the constructors, > >> and especially dislike having an argument that's only useful when other args > >> are provided (auth_cache). Not sure how to improve that, though... > >> > > > > What about a py.path.SvnAuth class ... > > > > > I was thinking about something like that, but figured it would be kind > of heavy for just the SVN auth... However, It's more modular and clearer to test IMO (and will be used both from svnurl's and svnwc's). > > Eventually we might want to generalize SvnAuth slightly > > into some Identity or Auth class that can also be > > used e.g. for py.execnet.SshGateway. > > > > > in that case it may be nice to have, indeed... ;) :) > Then... > > > * then add an explicit 'svnauth' parameter to wc.checkout and wc.commit > > > (which other svnwc methods need it?) > > Well, that depends on the use case. For a repository that requires authentication for all types of access, if the path objects aren't created with credentials passed to the constructor, it may be needed for all actions that hit the server (e.g. update, lock, etc.). hum, but svnwc does not have that many more (than commit/update/checkout/lock) methods that actually go the remote repo, does it? But i haven't checked. > Perhaps it makes more sense to only allow passing auth to the constructor? I don't really see a use case for using different credentials for different methods, but I may overlook something? > > Cheers, > > Guido > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Holger Krekel - freelance manager and programmer merlinux GmbH: http://merlinux.de py.test: http://pytest.org py lib: http://pylib.org PyPy: http://codespeak.net/pypy From johnny at johnnydebris.net Wed Feb 13 21:54:22 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Wed, 13 Feb 2008 21:54:22 +0100 Subject: [py-dev] Svn auth update Message-ID: <47B358FE.8000709@johnnydebris.net> Hi! I've implemented the SvnAuth class now, as Holger suggested, and made that the svnwc constructor, and the methods that connect to the server, all accept it as an argument (and use it, of course ;). You can see the progress on the 'guido-svn-auth' branch (SvnAuth code in py/path/svn/auth.py, and an updated version of the document I attached before in py/path/svn/auth.txt). After playing with the api a bit, I think agree with Holger that passing auth to children from methods that return path objects is not necessary, and may even be confusing, so perhaps we should indeed skip that for now. What I do want to add somehow is allow to set the --non-interactive switch somehow, although I'm not sure how (it doesn't seem proper to have it be part of the authentication information, but having yet another argument to all those methods is also not nice). I guess I'll continue with adding support for the auth argument to py.path.svnurl on friday, if no-one objects... Cheers, Guido From holger at merlinux.de Thu Feb 14 08:56:50 2008 From: holger at merlinux.de (holger krekel) Date: Thu, 14 Feb 2008 08:56:50 +0100 Subject: [py-dev] Svn auth update In-Reply-To: <47B358FE.8000709@johnnydebris.net> References: <47B358FE.8000709@johnnydebris.net> Message-ID: <20080214075650.GD17687@solar.trillke> Hi Guido! On Wed, Feb 13, 2008 at 21:54 +0100, Guido Wesdorp wrote: > Hi! > > I've implemented the SvnAuth class now, as Holger suggested, and made > that the svnwc constructor, and the methods that connect to the server, > all accept it as an argument (and use it, of course ;). great! I am still not sure if passing auth to the constructor makes sense, though. > You can see the progress on the 'guido-svn-auth' branch (SvnAuth code in > py/path/svn/auth.py, and an updated version of the document I attached > before in py/path/svn/auth.txt). I've reviewed and made 51470|1|2 commits accordingly - please check. > After playing with the api a bit, I think agree with Holger that passing > auth to children from methods that return path objects is not necessary, > and may even be confusing, so perhaps we should indeed skip that for now. > > What I do want to add somehow is allow to set the --non-interactive > switch somehow, although I'm not sure how (it doesn't seem proper to > have it be part of the authentication information, but having yet > another argument to all those methods is also not nice). I'd just put it to the SvnAuth class - and modify the docstring to say that it encapsulates svn authentication aspects. > I guess I'll continue with adding support for the auth argument to > py.path.svnurl on friday, if no-one objects... Could you maybe write a large functional test that creates a repo, does checkouts and uses all the wc methods with auth? best & cheers, holger From johnny at johnnydebris.net Thu Feb 14 09:01:38 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Thu, 14 Feb 2008 09:01:38 +0100 Subject: [py-dev] Svn auth update In-Reply-To: <20080214075650.GD17687@solar.trillke> References: <47B358FE.8000709@johnnydebris.net> <20080214075650.GD17687@solar.trillke> Message-ID: <47B3F562.9060908@johnnydebris.net> holger krekel wrote: > great! I am still not sure if passing auth to the constructor makes sense, though. > > Hm, well, I'm not sure either... It does make the API nicer to use imo, though... > I've reviewed and made 51470|1|2 commits accordingly - please check. > I checked as the checkin messages came in... :) Looking good, thanks! > I'd just put it to the SvnAuth class - and modify the docstring > to say that it encapsulates svn authentication aspects. > > Okay. > Could you maybe write a large functional test that creates a > repo, does checkouts and uses all the wc methods with auth? > > Hm, yes, I'll try that... Thanks for the refactoring and feedback! Cheers, Guido From faassen at startifact.com Tue Feb 19 20:36:18 2008 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 19 Feb 2008 20:36:18 +0100 Subject: [py-dev] py.path.svnwc ensure problems Message-ID: Hi there, Imagine the following scenario: * we have a svnwc: wc * through some external mechanism, a directory 'map5' is created in it. This directory is *not* added to SVN using 'svn add' yet. * now through svnwc, we try to create a file in that directory by using .ensure(): wc.join('map5').join('content').ensure() We then get the following error: ... File "/home/faassen/buildout-eggs/py-0.9.0-py2.4.egg/py/path/svn/wccommand.py", line 160, in ensure p.add() File "/home/faassen/buildout-eggs/py-0.9.0-py2.4.egg/py/path/svn/wccommand.py", line 175, in add self._svn('add') File "/home/faassen/buildout-eggs/py-0.9.0-py2.4.egg/py/path/svn/wccommand.py", line 87, in _svn out = py.process.cmdexec(string) File "/home/faassen/buildout-eggs/py-0.9.0-py2.4.egg/py/process/cmdexec.py", line 94, in posix_exec_cmd ''.join(out), ''.join(err)) Error: ExecutionFailed: 1 LC_ALL=C svn add "/home/faassen/tmp/resynch/app2/data/map5/content" svn: '/home/faassen/tmp/resynch/app2/data/map5' is not a working copy svn: Can't open file '/home/faassen/tmp/resynch/app2/data/map5/.svn/entries': No such file or directory It would be nicer for my purposes if 'ensure()' also ensured that all directories above 'content' are added to SVN first (if indeed they haven't been added yet). Would this be good behavior in general for py.path.svnwc, or is there a good reason not to do this? Regards, Martijn From faassen at startifact.com Tue Feb 19 20:49:08 2008 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 19 Feb 2008 20:49:08 +0100 Subject: [py-dev] py.path.svnwc ensure problems In-Reply-To: References: Message-ID: Hi there, Here's an argument why .ensure() should add directories automatically. Imagine the following sequence of events: two checkouts of the same stuff, a and b In a: create a directory foo create a content object bar in foo svn add foo and bar svn commit In b: svn up get foo and its contents, bar Back to a: svn remove foo svn commit Back to b: change foo/bar svn up What happens now is that the foo directory will remain in b, and the changed 'bar' file will remain as well. Both will however be considered to be unknown. One can reproduce this sequence of events working with py.path.svnwc. In fact, I have. :) My code has an .ensure() to make sure that foo/bar exists. This ensure() fails as 'foo' isn't known to SVN anymore after the last SVN up. If ensure() instead re-added 'foo' to SVN, things would proceed properly. You would think that ensure() in the case of svnwc would need to ensure that the whole path exists in SVN instead of bailing out with an error. One question is what .ensure() should do with any other content in foo should foo be automatically added. I guess it should not automatically re-add it. Actually in my use case it should actually probably be doing so, so I have some thinking to do... Regards, Martijn From holger at merlinux.de Tue Feb 19 21:51:15 2008 From: holger at merlinux.de (holger krekel) Date: Tue, 19 Feb 2008 21:51:15 +0100 Subject: [py-dev] py.path.svnwc ensure problems In-Reply-To: References: Message-ID: <20080219205115.GO17687@solar.trillke> Hi Martijn! On Tue, Feb 19, 2008 at 20:36 +0100, Martijn Faassen wrote: > > wc.join('map5').join('content').ensure() > > We then get the following error: > > ... > > Error: ExecutionFailed: 1 LC_ALL=C svn add > "/home/faassen/tmp/resynch/app2/data/map5/content" > svn: '/home/faassen/tmp/resynch/app2/data/map5' is not a working copy > svn: Can't open file > '/home/faassen/tmp/resynch/app2/data/map5/.svn/entries': No such file or > directory > > It would be nicer for my purposes if 'ensure()' also ensured that all > directories above 'content' are added to SVN first (if indeed they > haven't been added yet). > Would this be good behavior in general for py.path.svnwc, or is there a > good reason not to do this? I can imagine that this can makes sense. Might be a bit hairy to implement, not sure. In any case I'd rather like to see things in terms of tests. You might take py/path/svn/testing/test_wccommand.py's test_ensure as a good starting point for expressing the desired behaviour. Feel free to implement it and submit a patch or just commit to py/trunk if you are sure enough - i can review. best, holger From johnny at johnnydebris.net Wed Feb 20 22:43:04 2008 From: johnny at johnnydebris.net (Guido Wesdorp) Date: Wed, 20 Feb 2008 22:43:04 +0100 Subject: [py-dev] svn auth - revisiting svnwc auth object passing Message-ID: <47BC9EE8.4050706@johnnydebris.net> Hi there! I've implemented auth support for py.path.svnurl now too, seems to work nicely. Since it requires authentication for just about all its methods, I decided (after some discussion with Holger on IRC and some playing around) to not have an 'auth' argument to all of the methods. Instead only the constructor groks an auth argument, path objects that are generated by any of the path's methods 'inherit' it, and an 'svnurl.auth' property is exposed to allow overriding it. It took a bit of playing around to get it to work properly, but I think it does now, please take a look if you're interested. If you like the way this works, I would (again ;) like to ask whether I'm allowed to change svnwc so it works the same there - at least, both for consistency's sake and because it's a bit cleaner imo - have an auth property instead of having the argument to all methods that can connect to the server would be nice... Please let me know what you think. Cheers, Guido From holger at merlinux.de Sun Feb 24 20:37:06 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 24 Feb 2008 20:37:06 +0100 Subject: [py-dev] svn auth - revisiting svnwc auth object passing In-Reply-To: <47BC9EE8.4050706@johnnydebris.net> References: <47BC9EE8.4050706@johnnydebris.net> Message-ID: <20080224193706.GN17687@solar.trillke> Hey Guido, some first feedback ... could you refactor the tests such that e.g. changing the escaping of cmdline commands does not require lots of changes to test_auth.py? For example: def test_export(self): auth = SvnAuth('foo', 'bar') u = svnurl_no_svn('http://foo.bar/svn', auth=auth) target = py.path.local('/foo') u.export(target) assert u.commands[0].endswith('svn export "http://foo.bar/svn" "/foo" ' '--username="foo" --password="bar"') Apart from that this fails on windows, IMO it would be enough to test for the presence of the username/password combo and to do this from one place (and also the creation of "auth" from place). IOW: avoid redundancy and nitpicky output checking tests. For the slower functional test: could you maybe do some clever extra setup/teardown and re-use e.g. test_urlcommand.py's TestURLCommandPath tests? Some tests might not work, but then it's maybe better to rework those. If adding another base line for Path APIs i wouldn't like to have to write tests at several places for it. IOW: integrate tests some more with existing tests. hope this makes some sense, I am probably quite busy the next days. best, holger On Wed, Feb 20, 2008 at 22:43 +0100, Guido Wesdorp wrote: > Hi there! > > I've implemented auth support for py.path.svnurl now too, seems to work > nicely. Since it requires authentication for just about all its methods, > I decided (after some discussion with Holger on IRC and some playing > around) to not have an 'auth' argument to all of the methods. Instead > only the constructor groks an auth argument, path objects that are > generated by any of the path's methods 'inherit' it, and an > 'svnurl.auth' property is exposed to allow overriding it. It took a bit > of playing around to get it to work properly, but I think it does now, > please take a look if you're interested. > > If you like the way this works, I would (again ;) like to ask whether > I'm allowed to change svnwc so it works the same there - at least, both > for consistency's sake and because it's a bit cleaner imo - have an auth > property instead of having the argument to all methods that can connect > to the server would be nice... > > Please let me know what you think. > > Cheers, > > Guido > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Holger Krekel - freelance manager and programmer pylib py.test/greenlets/svn APIs: http://pylib.org PyPy Python/Compiler tool chain: http://codespeak.net/pypy merlinux collaborative contracting: http://merlinux.de From holger at merlinux.de Sun Feb 24 20:54:16 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 24 Feb 2008 20:54:16 +0100 Subject: [py-dev] svn auth - revisiting svnwc auth object passing In-Reply-To: <20080224193706.GN17687@solar.trillke> References: <47BC9EE8.4050706@johnnydebris.net> <20080224193706.GN17687@solar.trillke> Message-ID: <20080224195416.GO17687@solar.trillke> hi again, I forgot one thing regarding: please avoid code redundancy also with the cache-handling in svnurlcommand (and everywhere else for that matter :) as well. E.g. lines such as: auth = self.auth and self.auth.makecmdoptions() or None self._lsnorevcache.delentry((self.dirpath().strpath, auth)) appear multiple times. It's probably best to rather use a helper like self._norev_delentry(self.dirpath()) and have the helper method figure things out. Same is true for the getorbuild where you might introduce a helper maybe as well. I haven't checked in detail but maybe it's even possible to have the helper disambiguate the cases of "norev" and "rev" and use caches appropriately. Btw, i have the subjective impression that caching is not working properly - I remember the tests to run much faster. thanks & cheers, holger On Sun, Feb 24, 2008 at 20:37 +0100, holger krekel wrote: > Hey Guido, > > some first feedback ... could you refactor the tests such that > e.g. changing the escaping of cmdline commands does not require > lots of changes to test_auth.py? For example: > > def test_export(self): > auth = SvnAuth('foo', 'bar') > u = svnurl_no_svn('http://foo.bar/svn', auth=auth) > target = py.path.local('/foo') > u.export(target) > assert u.commands[0].endswith('svn export "http://foo.bar/svn" "/foo" ' > '--username="foo" --password="bar"') > > Apart from that this fails on windows, IMO it would be enough > to test for the presence of the username/password combo > and to do this from one place (and also the creation of "auth" > from place). IOW: avoid redundancy and nitpicky output checking tests. > > For the slower functional test: could you maybe do some clever > extra setup/teardown and re-use e.g. test_urlcommand.py's > TestURLCommandPath tests? Some tests might not work, but then > it's maybe better to rework those. If adding another base line > for Path APIs i wouldn't like to have to write tests at several > places for it. IOW: integrate tests some more with existing > tests. > > hope this makes some sense, I am probably quite busy the > next days. > > best, > > holger > > > > > On Wed, Feb 20, 2008 at 22:43 +0100, Guido Wesdorp wrote: > > Hi there! > > > > I've implemented auth support for py.path.svnurl now too, seems to work > > nicely. Since it requires authentication for just about all its methods, > > I decided (after some discussion with Holger on IRC and some playing > > around) to not have an 'auth' argument to all of the methods. Instead > > only the constructor groks an auth argument, path objects that are > > generated by any of the path's methods 'inherit' it, and an > > 'svnurl.auth' property is exposed to allow overriding it. It took a bit > > of playing around to get it to work properly, but I think it does now, > > please take a look if you're interested. > > > > If you like the way this works, I would (again ;) like to ask whether > > I'm allowed to change svnwc so it works the same there - at least, both > > for consistency's sake and because it's a bit cleaner imo - have an auth > > property instead of having the argument to all methods that can connect > > to the server would be nice... > > > > Please let me know what you think. > > > > Cheers, > > > > Guido > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > > > > -- > Holger Krekel - freelance manager and programmer > pylib py.test/greenlets/svn APIs: http://pylib.org > PyPy Python/Compiler tool chain: http://codespeak.net/pypy > merlinux collaborative contracting: http://merlinux.de > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Holger Krekel - freelance manager and programmer pylib py.test/greenlets/svn APIs: http://pylib.org PyPy Python/Compiler tool chain: http://codespeak.net/pypy merlinux collaborative contracting: http://merlinux.de