From pje@telecommunity.com Fri Sep 6 22:07:56 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Fri, 06 Sep 2002 17:07:56 -0400 Subject: [ZODB-Dev] Re: [Persistence-sig] "Straw Man" transaction API In-Reply-To: <15689.19626.780605.301525@slothrop.zope.com> References: <200208010523.g715N3625198@localhost.localdomain> <15688.35220.331816.465900@anthem.wooz.org> <200208010523.g715N3625198@localhost.localdomain> Message-ID: <3.0.5.32.20020906170756.0160cc20@telecommunity.com> I just discovered an interesting (design?) flaw in both my "Straw Man" API and ZODB4 regarding savepoints... If you savepoint the transaction with participants p1 and p2, and then participant p3 joins the transaction *after* this, the created savepoint object has no way to know how to rollback p3, if it's rolled back. It seems some additional complexity in the join method and the savepoint aggregation are required. :( At 10:58 AM 8/1/02 -0400, Jeremy Hylton wrote: >>>>>> "AB" == Anthony Baxter writes: > > >>>> Barry A. Warsaw wrote > >> Say you had savepoint(t1), savepoint(t2), and savepoint(t3) where > >> t1 < t2 < t3. Then you rolled back savepoint(t1) and then try to > >> rollback savepoint(t3), you'd get an exception right? > >Yes. RollbackError. > > AB> If you have multiple savepoints in the same transaction, should > AB> you be allowed to roll back the one that's not the most recent? > AB> To my brain, this doesn't make sense... > >It does make sense. It's all about saving partial progress so that >you can return to it later if something goes wrong. Nested >transactions are an example of something that wants to rollback to >arbitrary earlier savepoints. If part of a subtransaction fails, you >need to rollback to th beginning of the subtransaction. You need not >abort the entire transaction, because some part of the application can >recover and continue from the last savepoint. In particular, the >state of persistent objects gets rollback but the total state of your >application (e.g. control flow, non-persistent local variables, etc.) >is not. From cr1st1@go.ro Thu Sep 12 13:49:49 2002 From: cr1st1@go.ro (Cristi Cristi) Date: 12 Sep 2002 12:49:49 -0000 Subject: [Persistence-sig] Why only ZODB ? what about 4ODS ? Message-ID: <20020912124949.5967.qmail@relay1.home.ro> Why are you talking mostly about zodb? Have you looked around to see if there are other packages that might be of interest? For example I suggest 4ODS, which is part of the 4Suite python libraries (which are already distributed in newer linux distributions). I looked at it and it seems to be pretty well writen. http://4suite.org/index.xhtml http://uche.ogbuji.net:8080/uche.ogbuji.net/tech/4Suite/4ODS-userguide.html (a little out of date I think) Please share your opinions. Regards, Cristi ---- Home, no matter how far... http://www.home.ro From pje@telecommunity.com Thu Sep 12 14:55:30 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Thu, 12 Sep 2002 09:55:30 -0400 Subject: [Persistence-sig] Why only ZODB ? what about 4ODS ? In-Reply-To: <20020912124949.5967.qmail@relay1.home.ro> Message-ID: <5.1.0.14.0.20020912094930.00ab77d0@mail.telecommunity.com> At 12:49 PM 9/12/02 +0000, Cristi Cristi wrote: >Why are you talking mostly about zodb? Mainly because it's one of the systems that's been around longer, and lots of people have experience with it. >Have you looked around >to see if there are other packages that might be of interest? Yes. >For example I suggest 4ODS, which is part of the 4Suite python >libraries (which are already distributed in newer linux >distributions). I looked at it and it seems to be pretty well >writen. http://4suite.org/index.xhtml >http://uche.ogbuji.net:8080/uche.ogbuji.net/tech/4Suite/4ODS-userguide.html >(a little out of date I think) Please share your opinions. I think that the intended relationship between the SIG and a project like 4ODS, is that the SIG wants to produce transaction, caching, and persistence API definitions that can be used by 4ODS as a basis for its larger framework, just as much as for ZODB or PEAK. That is, this isn't about picking a persistence system or mechanism and blessing it as the official Python system, but rather about having a common low-level API, similar in principle to the Python DBAPI standard. That is, in theory I should be able to write objects that could be persisted by ZODB, PEAK, or 4ODS, just by swapping out which package I install or activate. (In practice, as with the DBAPI, there are likely to be other compatibility issues that would need addressing, but at least one is not starting over with a *completely* different API.) Anyway, to accomplish this it would ideally require the participation of the 4ODS developers. Have you suggested to them that they get involved in this forum? From jeremy@alum.mit.edu Fri Sep 13 16:24:53 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Fri, 13 Sep 2002 11:24:53 -0400 Subject: [ZODB-Dev] Re: [Persistence-sig] "Straw Man" transaction API In-Reply-To: <3.0.5.32.20020906170756.0160cc20@telecommunity.com> References: <200208010523.g715N3625198@localhost.localdomain> <15688.35220.331816.465900@anthem.wooz.org> <3.0.5.32.20020906170756.0160cc20@telecommunity.com> Message-ID: <15746.837.746646.807201@slothrop.zope.com> >>>>> "PJE" == Phillip J Eby writes: PJE> I just discovered an interesting (design?) flaw in both my PJE> "Straw Man" API and ZODB4 regarding savepoints... PJE> If you savepoint the transaction with participants p1 and p2, PJE> and then participant p3 joins the transaction *after* this, the PJE> created savepoint object has no way to know how to rollback p3, PJE> if it's rolled back. PJE> It seems some additional complexity in the join method and the PJE> savepoint aggregation are required. :( Interesting problem :-). If a participant joined after a savepoint, would it be sufficient to abort that participant? Jeremy From pje@telecommunity.com Fri Sep 13 16:50:10 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Fri, 13 Sep 2002 11:50:10 -0400 Subject: [ZODB-Dev] Re: [Persistence-sig] "Straw Man" transaction API In-Reply-To: <15746.837.746646.807201@slothrop.zope.com> References: <3.0.5.32.20020906170756.0160cc20@telecommunity.com> <200208010523.g715N3625198@localhost.localdomain> <15688.35220.331816.465900@anthem.wooz.org> <3.0.5.32.20020906170756.0160cc20@telecommunity.com> Message-ID: <3.0.5.32.20020913115010.0161b100@telecommunity.com> At 11:24 AM 9/13/02 -0400, Jeremy Hylton wrote: >>>>>> "PJE" == Phillip J Eby writes: > > PJE> I just discovered an interesting (design?) flaw in both my > PJE> "Straw Man" API and ZODB4 regarding savepoints... > > PJE> If you savepoint the transaction with participants p1 and p2, > PJE> and then participant p3 joins the transaction *after* this, the > PJE> created savepoint object has no way to know how to rollback p3, > PJE> if it's rolled back. > > PJE> It seems some additional complexity in the join method and the > PJE> savepoint aggregation are required. :( > >Interesting problem :-). > >If a participant joined after a savepoint, would it be sufficient to >abort that participant? > In theory - but neither StrawMan nor the ZODB4 interface allows for a "roll back inside the transaction". The rollback message is an abort of the entire transaction. It seems to me that the "correct" solution is to request a savepoint from the new participant, and then aggregate that savepoint into all the currently-valid savepoints for the transaction. I actually can't think of any solution that doesn't require *some* kind of additional bookkeeping by the transaction to deal with this, so we might as well do it correctly. The error arose, I believe, because in my original "Straw Man" proposal I assumed persistent subscriptions, i.e. where participants didn't normally just up and join transactions, in fact I explicitly said a "TransactionInProgress" error would be raised under this circumstance. If we allow participants to join after a savepoint has occurred, we'll have to keep track of which savepoints have been issued and which ones are valid. At least we can do this somewhat straightforwardly: a savepoint stack, managed by the transaction. Aggregating savepoints will need to know their transaction, and have the ability to add an extra contained savepoint after they've been created. When rollback() is called on an aggregating savepoint, it must tell the transaction that it is being rolled back, and the transaction needs to find it in the savepoint stack and invalidate (drop its reference to) the subsequent savepoints. When a participant joins a transaction with a non-empty savepoint stack, the transaction requests a series of savepoints from the new participant (one per existing savepoint), and aggregates them into the existing savepoints. One other side effect of all this is the ability (or lack thereof) to tell whether rollback is possible at the time you obtain a savepoint, since an unrevertable participant may join following the savepoint! This suggests to me that there are really two savepoint interfaces: a simpler one that is implemented by participants, and a queryable one which is implemented by the aggregating savepoint objects that transactions return. Your thoughts? From jeremy@alum.mit.edu Fri Sep 13 17:14:33 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Fri, 13 Sep 2002 12:14:33 -0400 Subject: [ZODB-Dev] Re: [Persistence-sig] "Straw Man" transaction API In-Reply-To: <3.0.5.32.20020913115010.0161b100@telecommunity.com> References: <3.0.5.32.20020906170756.0160cc20@telecommunity.com> <200208010523.g715N3625198@localhost.localdomain> <15688.35220.331816.465900@anthem.wooz.org> <3.0.5.32.20020913115010.0161b100@telecommunity.com> Message-ID: <15746.3817.56608.171728@slothrop.zope.com> >>>>> "PJE" == Phillip J Eby writes: PJE> In theory - but neither StrawMan nor the ZODB4 interface allows PJE> for a "roll back inside the transaction". The rollback message PJE> is an abort of the entire transaction. PJE> It seems to me that the "correct" solution is to request a PJE> savepoint from the new participant, and then aggregate that PJE> savepoint into all the currently-valid savepoints for the PJE> transaction. I'm not sure what rollback inside the transaction is better or different than abort the transaction. Let's assume the following series of events, where Os are objects and Ps are transaction participants and On is managed by Pn. O1 modified P1 joins O2 modified P2 joins savepoint 1 O3 modified P3 joins savepoint 2 rollback to 1 O4 modified P4 joins commit In this case, P3 joined the transaction because O3 was modified. Then the transaction was rollback to a point before the modification to O3. In the end, the transaction never modified O3 and P3 had no reason to participate. In that case, it makes sense to me that P3 gets an abort when the rollback occurs. Jeremy From shane@zope.com Fri Sep 13 17:23:31 2002 From: shane@zope.com (Shane Hathaway) Date: Fri, 13 Sep 2002 12:23:31 -0400 Subject: [ZODB-Dev] Re: [Persistence-sig] "Straw Man" transaction API References: <200208010523.g715N3625198@localhost.localdomain> <15688.35220.331816.465900@anthem.wooz.org> <3.0.5.32.20020906170756.0160cc20@telecommunity.com> <15746.837.746646.807201@slothrop.zope.com> Message-ID: <3D821103.6020105@zope.com> Jeremy Hylton wrote: >>>>>>"PJE" == Phillip J Eby writes: >>>>> > > PJE> I just discovered an interesting (design?) flaw in both my > PJE> "Straw Man" API and ZODB4 regarding savepoints... > > PJE> If you savepoint the transaction with participants p1 and p2, > PJE> and then participant p3 joins the transaction *after* this, the > PJE> created savepoint object has no way to know how to rollback p3, > PJE> if it's rolled back. > > PJE> It seems some additional complexity in the join method and the > PJE> savepoint aggregation are required. :( > > Interesting problem :-). > > If a participant joined after a savepoint, would it be sufficient to > abort that participant? Here's another way to look at this. All IMHO. I really don't think of ZODB 3 as needing "subtransactions" or even "savepoints". The only subtransaction-related need I've ever seen in ZODB applications is the ability to move the current set of changes out of RAM. No ZODB application that I've seen uses get_transaction().abort(1). Applications use get_transaction().commit(1) mainly for the purpose of conserving RAM. In the future, ZODB applications could access this kind of functionality through some other API, perhaps "jar.swapOutChanges()" or something, making the intent clearer and removing the dependency on subtransactions. To the uninitiated, "commit(1)" does not express the intent of conserving RAM. So perhaps the initial outcome of this SIG does not need to account for subtransactions and/or savepoints. Is it a real need? Shane From pje@telecommunity.com Fri Sep 13 17:30:13 2002 From: pje@telecommunity.com (Phillip J. Eby) Date: Fri, 13 Sep 2002 12:30:13 -0400 Subject: [ZODB-Dev] Re: [Persistence-sig] "Straw Man" transaction API In-Reply-To: <3D821103.6020105@zope.com> References: <200208010523.g715N3625198@localhost.localdomain> <15688.35220.331816.465900@anthem.wooz.org> <3.0.5.32.20020906170756.0160cc20@telecommunity.com> <15746.837.746646.807201@slothrop.zope.com> Message-ID: <3.0.5.32.20020913123013.0161b790@telecommunity.com> At 12:23 PM 9/13/02 -0400, Shane Hathaway wrote: >Here's another way to look at this. All IMHO. > >I really don't think of ZODB 3 as needing "subtransactions" or even >"savepoints". The only subtransaction-related need I've ever seen in >ZODB applications is the ability to move the current set of changes out >of RAM. No ZODB application that I've seen uses >get_transaction().abort(1). I *almost* used it once. :) The only use-case I've ever had for using it (and I never actually did use it for this), was to handle a batch of operations from a single form submit, with each line item handled as a separate rollback, so that the resulting page would list errors for line items that failed, but line items that succeeded would be committed. That is, a try/except block around each line item, that set a savepoint before the item, and rolled back to the savepoint if an exception occurred, append the exception to a list, and proceeded to the next line item. >Applications use >get_transaction().commit(1) mainly for the purpose of conserving RAM. ZPatterns-based apps also do it to synchronize external state (i.e. an RDBMS) with internal state (objects), so that queries performed against the external state will reflect the current internal state. Naturally, this is a hack. :) In PEAK, "query jars" will directly request that their underlying "storage jars" 'flush' themselves to external storage before performing a query, so a global hack of this kind is not required. >In the future, ZODB applications could access this kind of functionality >through some other API, perhaps "jar.swapOutChanges()" or something, In PEAK's case, this would be available via a Rack.flush() operation, so I see your point. >making the intent clearer and removing the dependency on >subtransactions. To the uninitiated, "commit(1)" does not express the >intent of conserving RAM. > >So perhaps the initial outcome of this SIG does not need to account for >subtransactions and/or savepoints. Is it a real need? Good point. I can live without them entirely. Evidently, so can most Java developers (JTA doesn't require, nor specify how to provide, nested transactions). I don't see anything wrong with dropping this from a hypothetical PyTA; we can always come back a year or two from now and define a standard "extension" for this after there has been field experimentation/experience.