From Chris.Clark at ingres.com Tue Sep 13 00:20:18 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Mon, 12 Sep 2011 15:20:18 -0700 Subject: [DB-SIG] autocommit support in pep-249 Message-ID: <4E6E85A2.5020509@ingres.com> Hi All, I was discussing with someone today autocommit support and found an area of pep-249 where I think we could improve the wording (and the spec): http://www.python.org/dev/peps/pep-0249/ ..... .commit() Commit any pending transaction to the database. Note that if the database supports an auto-commit feature, this must be initially off. An interface method may be provided to turn it back on. ..... It is the last sentence that I'm looking at: > An interface method may be provided to turn it back on. Comments: * The "may" clearly marks this as an optional interface (and I'm happy about that). * It would be helpful to make clear what the interface method would be so module authors can implement this consistently * if autocommit can be turned on in a connection, there should have a way to turn it off again Here are some examples of the "interface method" implemented out in the wild today * It looks like mxodbc handles this as a setconnectoption() method on the connection. http://www.egenix.com/products/python/mxODBCConnect/doc/mxodbc/ * a number of other ODBC based drivers offer an option in the connect() constructor (only) * The mysql module implies it has an method on the connection object for this conn.autocommit(), but this does not appear to be available in the pep-249 compatible API ! Obviously there is plenty of scope for different implementations. I think there is value in documenting a recommendation on what this interface should look like. I like the API approach mxodbc has taken (it matches ODBC very well) but it doesn't feel Pythonic :-( To get the ball rolling, here is an initial suggestion: * needs to take a boolean flag as a parameter to enable and also disable * needs to return the auto commit state so it can be queried pep changes, marked between ** and **: Connection Objects Connection Objects should respond to the following methods: ...... .commit() Commit any pending transaction to the database. Note that if the database supports an auto-commit feature, this must be initially off. **See the option autocommit() method to change autocommit behavior.** Database modules that do not support transactions should implement this method with void functionality. ..... ** Connection Objects may optional implement and respond to the following methods: autocommit(on=None) returns True when autocommit is on, and False when autocommit is off. If the keyword parameter "on" is set to True, the connection will commit any open transactions (as if connection.commit() was issued) and all subsequent transactions will auto commit, the method returns True if successfully in autocommit mode. If the keyword parameter "on" is set to False, the connection will commit any open transactions (as if connection.commit() was issued) and all subsequent transactions will no longer auto commit, the method returns False if successfully out of autocommit mode.. ** The reason for the immediate commit is to try and avoid DBMS specific behaviors (read error conditions) with different vendors when an auto commit is requested with open transactions. Comments? Chris From vernondcole at gmail.com Tue Sep 13 04:15:07 2011 From: vernondcole at gmail.com (Vernon Cole) Date: Mon, 12 Sep 2011 20:15:07 -0600 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6E85A2.5020509@ingres.com> References: <4E6E85A2.5020509@ingres.com> Message-ID: This sounds like yet another item which should go into the dbapi 3.0 spec... as soon as someone has the courage to start one. -- Vernon On Mon, Sep 12, 2011 at 4:20 PM, Chris Clark wrote: > Hi All, > > I was discussing with someone today autocommit support and found an area of > pep-249 where I think we could improve the wording (and the spec): > > http://www.python.org/dev/peps/pep-0249/ > > ..... > > ? ? ? .commit() > ? ? ? ? ? ? ? ? ? ?Commit any pending transaction to the database. Note that > ? ? ? ? ? if the database supports an auto-commit feature, this must > ? ? ? ? ? be initially off. An interface method may be provided to > ? ? ? ? ? turn it back on. > > ..... > > It is the last sentence that I'm looking at: > >> An interface method may be provided to turn it back on. > > Comments: > > ? * The "may" clearly marks this as an optional interface (and I'm > ? ? happy about that). > ? * It would be helpful to make clear what the interface method would > ? ? be so module authors can implement this consistently > ? * if autocommit can be turned on in a connection, there should have > ? ? a way to turn it off again > > > Here are some examples of the "interface method" implemented out in the wild > today > > ? * It looks like mxodbc handles this as a setconnectoption() method > ? ? on the connection. > ? ? http://www.egenix.com/products/python/mxODBCConnect/doc/mxodbc/ > ? * a number of other ODBC based drivers offer an option in the > ? ? connect() constructor (only) > ? * The mysql module implies it > ? ? has an method on the connection object for this conn.autocommit(), > ? ? but this does not appear to be available in the pep-249 compatible > ? ? API ! > > > Obviously there is plenty of scope for different implementations. I think > there is value in documenting a recommendation on what this interface should > look like. > > I like the API approach mxodbc has taken (it matches ODBC very well) but it > doesn't feel Pythonic :-( > > To get the ball rolling, here is an initial suggestion: > > ? * needs to take a boolean flag as a parameter to enable and also disable > ? * needs to return the auto commit state so it can be queried > > pep changes, marked between ** and **: > > Connection Objects > > ? Connection Objects should respond to the following methods: > ...... > > ? ? ? .commit() > ? ? ? ? ? ? ? ? ? ?Commit any pending transaction to the database. Note that > ? ? ? ? ? if the database supports an auto-commit feature, this must > ? ? ? ? ? be initially off. **See the option autocommit() method to change > ? ? ? ? ? autocommit behavior.** > ? ? ? ? ? ? ? ? ? ? ?Database modules that do not support transactions > should > ? ? ? ? ? implement this method with void functionality. > > ..... > ** > ? Connection Objects may optional implement and respond to the following > methods: > > ? autocommit(on=None) > ? ? ?returns True when autocommit is on, and False when autocommit is off. > ? ? ?If the keyword parameter "on" is set to True, the connection will > commit any open transactions (as if connection.commit() was issued) and all > subsequent transactions will auto commit, the method returns True if > successfully in autocommit mode. > ? ? ?If the keyword parameter "on" is set to False, the connection will > commit any open transactions (as if connection.commit() was issued) and all > subsequent transactions will no longer auto commit, the method returns False > if successfully out of autocommit mode.. > ** > > The reason for the immediate commit is to try and avoid DBMS specific > behaviors (read error conditions) with different vendors when an auto commit > is requested with open transactions. > > Comments? > > > Chris > > _______________________________________________ > DB-SIG maillist ?- ?DB-SIG at python.org > http://mail.python.org/mailman/listinfo/db-sig > From daniele.varrazzo at gmail.com Tue Sep 13 12:28:52 2011 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Tue, 13 Sep 2011 11:28:52 +0100 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6E85A2.5020509@ingres.com> References: <4E6E85A2.5020509@ingres.com> Message-ID: On Mon, Sep 12, 2011 at 11:20 PM, Chris Clark wrote: > I was discussing with someone today autocommit support and found an area of > pep-249 where I think we could improve the wording (and the spec): Agreed, it would have been long due. The quick survey I did when we introduced the session in psycopg (last May) has been: - MySQLdb: conn.autocommit() - cx_Oracle: conn.autocommit (attribute, not function) - pyodbc: conn.autocommit (attribute) - sqlite3: conn.isolation_level = None - KInterbaseDB: not supported (to give context, in psycopg you had to use conn.set_isolation_level(0) before version 2.4.2, when we introduced better ways to manipulate the session - see for the discussion. Psycopg now uses an autocommit attribute - see ). > To get the ball rolling, here is an initial suggestion: > > ? * needs to take a boolean flag as a parameter to enable and also disable > ? * needs to return the auto commit state so it can be queried [thus] > ? autocommit(on=None) > ? ? ?returns True when autocommit is on, and False when autocommit is off. > ? ? ?If the keyword parameter "on" is set to True, the connection will > commit any open transactions (as if connection.commit() was issued) and all > subsequent transactions will auto commit, the method returns True if > successfully in autocommit mode. > ? ? ?If the keyword parameter "on" is set to False, the connection will > commit any open transactions (as if connection.commit() was issued) and all > subsequent transactions will no longer auto commit, the method returns False > if successfully out of autocommit mode.. Because you need a getter and a setter, I feel the right solution is to use a property. A function that behaves as a getter when called with no parameter and as a setter if it has one is not something widely used in pythonland. A more natural solution, but inferior, would be using two functions (of which choosing the name is a bikeshedding bloodbath). Properties are just the right tools for this task, and appear to be the most used solution in popular drivers. The behaviour of setting autocommit when a transaction is already open is also to be seen: there are three basic option: - have a commit - have a rollback - have an error I feel the implicit commit is the most dangerous option. An implicit rollback may be more acceptable (it's the same behaviour of closing the connection with a transaction in progress): I proposed it for psycopg but, in the discussion, raising an exception proved the most popular option. -- Daniele From vernondcole at gmail.com Tue Sep 13 17:04:08 2011 From: vernondcole at gmail.com (Vernon Cole) Date: Tue, 13 Sep 2011 09:04:08 -0600 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: References: <4E6E85A2.5020509@ingres.com> Message-ID: > Properties are just the right tools for this > task, and appear to be the most used solution in popular drivers. > I think that term "attributes" is more commonly used, rather than "properties", but I agree completely. "conn.autocommit = True" seems pythonic, as does "if not conn.autocommit". Getters and Setters feel like java. The implementing programmer has getattr and setattr for exactly that purpose, and the calling programmer has a clean interface. > The behaviour of setting autocommit when a transaction is already open > is also to be seen: there are three basic option: > > - have a commit > - have a rollback > - have an error > > I feel the implicit commit is the most dangerous option. An implicit > rollback may be more acceptable (it's the same behaviour of closing > the connection with a transaction in progress): I proposed it for > psycopg but, in the discussion, raising an exception proved the most > popular option. > I don't think I know how to even determine whether there is a transaction in progress in ADO.(*) If there is a way, then checking on it just so that I can raise an exception seems expensive. Remember that most of the time the resulting stack trace would not be seen by the programmer who goofed up, but a completely innocent user -- and to her it looks like the program just crashed for no reason. The rollback would be easy to program and user friendly. I vote rollback. -- Vernon (*) Most of you are dealing with only one underlying database, so have only that to worry about. I have to worry about all of your databases (with the exception of sqlite) and tons of others which neither of us has ever heard of. The impression that Microsoft "Active Data Objects" applies only to Microsoft SQL server databases is very wrong. There are ODBC drivers for almost everything and ADO providers for most of the rest. My user may be accessing anything from a .csv text file to an Active Directory user list. I prefer simple, thank you. From Chris.Clark at ingres.com Tue Sep 13 18:39:21 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Tue, 13 Sep 2011 09:39:21 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: References: <4E6E85A2.5020509@ingres.com> Message-ID: <4E6F8739.3010403@ingres.com> Vernon Cole wrote: >> Properties are just the right tools for this >> task, and appear to be the most used solution in popular drivers. >> >> > I think that term "attributes" is more commonly used, rather than > "properties", but I agree completely. > "conn.autocommit = True" seems pythonic, as does "if not > conn.autocommit". Getters and Setters feel like java. The > implementing programmer has getattr and setattr for exactly that > purpose, and the calling programmer has a clean interface. > Property is the correct name for this in Python, see http://docs.python.org/library/functions.html I agree using a property is a good idea, very easy to document and use. If/when I get around to it ;-) that is the route I'll take with jyjdbc. >> The behaviour of setting autocommit when a transaction is already open >> is also to be seen: there are three basic option: >> >> - have a commit >> - have a rollback >> - have an error >> >> I feel the implicit commit is the most dangerous option. An implicit >> rollback may be more acceptable (it's the same behaviour of closing >> the connection with a transaction in progress): I proposed it for >> psycopg but, in the discussion, raising an exception proved the most >> popular option. >> >> > I don't think I know how to even determine whether there is a > transaction in progress in ADO.(*) If there is a way, then checking > on it just so that I can raise an exception seems expensive. Remember > that most of the time the resulting stack trace would not be seen by > the programmer who goofed up, but a completely innocent user -- and to > her it looks like the program just crashed for no reason. The > rollback would be easy to program and user friendly. I vote rollback. > The exception route has the potential to be the easiest for most DBMS implementations, most DBMSs will raise an error, so the driver doesn't need to track state and can just propagate the DBMS error. For some backends, state tracking would be required which is why I suggested the "easy" commit option. Rollback seems semi safe BUT when setting autocommit to ON the default behavior of a rollback seems a little surprising. I'm not in love with committing when the autocommit state is changed but for the "enable auto commit" operation it seems logical, the user is requesting commits take place under the covers. Conversely when disabling autocommit (after it has previously been enabled) we are either not in a transaction (due to auto commit being on) or we could still be fetching mid select, in which case a commit would be the same as a rollback. I.e. only impact on locking, any writes to the database would have been already auto committed. So I think we have 3 people each advocating slightly different options for what to do when changing auto commit options :-) 1. exception 2. commit 3. rollback I'll be happy with what ever we as a group decide (so long as we document it), I think committing will be least surprising for end users, so whilst I'm not keen on it I feel I should champion it. Are people happy with this rough approach (we can thrash out exception/rollback/commit) where this is an optional extension, i.e. not required? Chris From jd at commandprompt.com Tue Sep 13 18:45:22 2011 From: jd at commandprompt.com (Joshua D. Drake) Date: Tue, 13 Sep 2011 09:45:22 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6F8739.3010403@ingres.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> Message-ID: <4E6F88A2.80804@commandprompt.com> On 09/13/2011 09:39 AM, Chris Clark wrote: > > Vernon Cole wrote: >>> Properties are just the right tools for this >>> task, and appear to be the most used solution in popular drivers. > So I think we have 3 people each advocating slightly different options > for what to do when changing auto commit options :-) > > 1. exception > 2. commit > 3. rollback > Just to throw some PostgreSQL two cents in here. It should not be legal to change transaction state once a transaction has begun. The "driver" should throw an error if that is tried. It shouldn't even reach the database. Consider: conn.begin() insert insert At this point we have an open transaction with two inserts pending. We are then going to: conn.autocommit()? Uh, no. The driver should throw an error and the database should remain waiting for next statement whether it be commit or whatever. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL Conference - http://www.postgresqlconference.org/ @cmdpromptinc - @postgresconf - 509-416-6579 From Chris.Clark at ingres.com Tue Sep 13 19:53:16 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Tue, 13 Sep 2011 10:53:16 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6F88A2.80804@commandprompt.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> Message-ID: <4E6F988C.4000704@ingres.com> Joshua D. Drake wrote: > > On 09/13/2011 09:39 AM, Chris Clark wrote: >> >> Vernon Cole wrote: >>>> Properties are just the right tools for this >>>> task, and appear to be the most used solution in popular drivers. > >> So I think we have 3 people each advocating slightly different options >> for what to do when changing auto commit options :-) >> >> 1. exception >> 2. commit >> 3. rollback >> > > Just to throw some PostgreSQL two cents in here. It should not be > legal to change transaction state once a transaction has begun. The > "driver" should throw an error if that is tried. It shouldn't even > reach the database. Consider: > > conn.begin() > insert > insert > > At this point we have an open transaction with two inserts pending. We > are then going to: > > conn.autocommit()? > > Uh, no. The driver should throw an error and the database should > remain waiting for next statement whether it be commit or whatever. > Thanks for the feedback Joshua, could you go in to more detail to explain your reasoning please? I think the more justifications we have the easier it is to make a decision. You made an interesting comment on the autocommit request, "It shouldn't even reach the database". I think you are saying the "set auto commit on" request should not be sent to the DBMS and be handled by the driver when there are open transactions. If I modify your example slightly: conn.begin() curs.execute('issue server non transaction based directive ') # i.e. not dml, not ddl curs.execute('issue session directive ') # for example change session language of error messages, Oracle NLS settings conn.autocommit() What should be the behavior be in this instance? No exception being raised makes the most sense to me as we are not in a transaction, I'm unclear how the driver could make this decision unless it parses and understands each curs.execute call. Some drivers like Vernon's ADO driver are DBMS independent so it is not reasonably for them to parse query text that is passed into into execute(). A good application won't fall foul of this but we have to deal with applications that are not written well so that drivers all behave predictably if they've implemented the spec. I personally prefer an exception but I think the commit is least surprising. They've asked to autocommit so an implicit commit makes sense. A few notes I should have added: * if we go the exception route, we should document that this is DBMS/driver defined, i.e. don't be specific as to which exception is raised (other than being a dbi exception) * in the unlikely event of a backend that does not support autocommit, the driver would now have the option to emulate that behavior Chris From jd at commandprompt.com Tue Sep 13 20:39:18 2011 From: jd at commandprompt.com (Joshua D. Drake) Date: Tue, 13 Sep 2011 11:39:18 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6F988C.4000704@ingres.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> Message-ID: <4E6FA356.7060800@commandprompt.com> On 09/13/2011 10:53 AM, Chris Clark wrote: >> Just to throw some PostgreSQL two cents in here. It should not be >> legal to change transaction state once a transaction has begun. The >> "driver" should throw an error if that is tried. It shouldn't even >> reach the database. Consider: >> >> conn.begin() >> insert >> insert >> >> At this point we have an open transaction with two inserts pending. We >> are then going to: >> >> conn.autocommit()? >> >> Uh, no. The driver should throw an error and the database should >> remain waiting for next statement whether it be commit or whatever. >> > > Thanks for the feedback Joshua, could you go in to more detail to > explain your reasoning please? I think the more justifications we have > the easier it is to make a decision. > > You made an interesting comment on the autocommit request, "It shouldn't > even reach the database". I think you are saying the "set auto commit > on" request should not be sent to the DBMS and be handled by the driver > when there are open transactions. If I modify your example slightly: > > conn.begin() > curs.execute('issue server non transaction based directive ') # i.e. not > dml, not ddl > curs.execute('issue session directive ') # for example change session > language of error messages, Oracle NLS settings > conn.autocommit() > > What should be the behavior be in this instance? No exception being > raised makes the most sense to me as we are not in a transaction, I'm > unclear how the driver could make this decision unless it parses and > understands each curs.execute call. Oh now that is interesting. Hmmm... perhaps conn.autocommit() is actually a synonym for conn.begin() except that it sets the autocommit property? So we would have two semantics: conn.begin() -- Starts a transaction, will not commit unless explicit conn.commit() is called. conn.autocommit() -- Works in autocommit mode, calling conn.commit() is redundant, each statement is executed as it is delivered to the database. ? Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL Conference - http://www.postgresqlconference.org/ @cmdpromptinc - @postgresconf - 509-416-6579 From eric-python at soroos.net Tue Sep 13 20:19:35 2011 From: eric-python at soroos.net (Eric Soroos) Date: Tue, 13 Sep 2011 11:19:35 -0700 Subject: [DB-SIG] pyPgSQL updates Message-ID: <4E6F9EB7.1050505@soroos.net> Hello, It appears that pyPgSQL has been abandoned and Postgresql is starting to pass it by. I've sent a note to the pyPgSQL dev list and gotten no response. So I'm doing a bit of maintenance now. I've worked up some changes to the quoting to allow it to use the new to 9.0 hex encoding for bytea fields. At the same time, I've changed the string and bytea quoting to use PQescapeStringConn and PQescapeByteaConn so that they properly quote the incoming items based on the connection settings. Code is at: https://github.com/wiredfool/PyPgSQL I'm also looking at including support for PQexecParams to reduce the memory footprint when adding large bytea items to the database. In my tests, the released version uses ~40x the binary size, my updated version uses 10-20x, and PQexecParams uses ~ 2.7x (though, the test harness starts to be a factor there). This method appears to be well outside the DB2.0 spec, so it would be driver specific. thanks eric From mike_mp at zzzcomputing.com Tue Sep 13 22:06:21 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 13 Sep 2011 16:06:21 -0400 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6FA356.7060800@commandprompt.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <4E6FA356.7060800@commandprompt.com> Message-ID: <1C86FB42-7484-47D2-A5A7-968FD5F2F466@zzzcomputing.com> On Sep 13, 2011, at 2:39 PM, Joshua D. Drake wrote: > > Oh now that is interesting. Hmmm... perhaps conn.autocommit() is actually a synonym for conn.begin() except that it sets the autocommit property? So we would have two semantics: > > conn.begin() -- Starts a transaction, will not commit unless explicit conn.commit() is called. > > conn.autocommit() -- Works in autocommit mode, calling conn.commit() is redundant, each statement is executed as it is delivered to the database. I'm -1 on conn.begin(). I've spent years explaining to users the beauty of the DBAPI in that it has no "begin" method. Providing for two different modes of doing the same thing, i.e. working with a standard DBAPI connection in a transaction, or forcing autocommit to True then using begin(), creates for a more confusing and controversial interface. Right now, pysqlite users are shielded from SQLite's aggressive file locking by the fact that pysqlite doesn't actually emit BEGIN until DML is encountered. There is an enhancement request on pysqlite's bugtrcker to allow this to be more configurable. In the meantime, I sometimes get email requests stating that SQLAlchemy should be emitting "BEGIN" itself to work around this issue. While I give these users a workaround for what's essentially a missing feature in pysqlite, SQLA never emits the BEGIN. It makes life much easier for SQLAlchemy that DBAPIs are responsible for handling transactional markers like this and I'd hate to see the interface diluted - there is sometimes functionality within "implicit begin" that may not be easy to replicate with an "explicit begin", and having both would make life more complicated for DBAPI authors as well as users. From daniele.varrazzo at gmail.com Wed Sep 14 00:01:05 2011 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Tue, 13 Sep 2011 23:01:05 +0100 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6F988C.4000704@ingres.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> Message-ID: On Tue, Sep 13, 2011 at 6:53 PM, Chris Clark wrote: > You made an interesting comment on the autocommit request, "It shouldn't > even reach the database". I think you are saying the "set auto commit on" > request should not be sent to the DBMS and be handled by the driver when > there are open transactions. If I modify your example slightly: > > conn.begin() > curs.execute('issue server non transaction based directive ') ?# i.e. not > dml, not ddl > curs.execute('issue session directive ') ?# for example change session > language of error messages, Oracle NLS settings > conn.autocommit() > > What should be the behavior be in this instance? No exception being raised > makes the most sense to me as we are not in a transaction, I'm unclear how > the driver could make this decision unless it parses and understands each > curs.execute call. I think with Postgres the situation is slightly easier, as there is no "non transaction" statement: any statement including DML is transactional except a couple (such as CREATE DATABASE) that would raise an error if run inside a transaction. Because the situation is this simple, psycopg can just run a BEGIN before the first statement executed, whatever it is, and there is no need to parse it; it then knows the state of the connection and an error such as switching to autocommit while in transaction is indeed handled by the adapter without calling the backend. What does the Oracle adapter currently do? From my interpretation of the dbapi there is no explicit conn.begin(), a transaction is implicitly started at the first statement. How does the driver implement this behaviour? Is it something it can delegate to the backend? Ditto for ODBC, where parsing the statements would be not impratical but just impossible: how does the adapter currently implement the "not autocommit" default dbapi behaviour? Is the begin handled by the adapter or by the backend? -- Daniele From mike_mp at zzzcomputing.com Wed Sep 14 00:18:29 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 13 Sep 2011 18:18:29 -0400 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> Message-ID: <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> On Sep 13, 2011, at 6:01 PM, Daniele Varrazzo wrote: > > I think with Postgres the situation is slightly easier, as there is no > "non transaction" statement: any statement including DML is > transactional except a couple (such as CREATE DATABASE) that would > raise an error if run inside a transaction. Because the situation is > this simple, psycopg can just run a BEGIN before the first statement > executed, whatever it is, and there is no need to parse it; it then > knows the state of the connection and an error such as switching to > autocommit while in transaction is indeed handled by the adapter > without calling the backend. > > What does the Oracle adapter currently do? From my interpretation of > the dbapi there is no explicit conn.begin(), a transaction is > implicitly started at the first statement. How does the driver > implement this behaviour? Is it something it can delegate to the > backend? > > Ditto for ODBC, where parsing the statements would be not impratical > but just impossible: how does the adapter currently implement the "not > autocommit" default dbapi behaviour? Is the begin handled by the > adapter or by the backend? The pysqlite adapter does actually parse incoming statements, looking for DML in an attempt to delay the BEGIN until necessary, as a means of avoiding aggressive file-locking between otherwise read-only connections to a SQLite database. But I agree in general most DBAPIs don't do anything like this. As for Oracle, like the DBAPI itself it has no "BEGIN" statement for transactions, and doesn't natively support "autocommit" in any way - there's always a transaction in progress that starts implicitly after a previous COMMIT or ROLLBACK. Therefore it works very nicely with DBAPI's scheme and I doubt cx_oracle needs to do anything here. This is one reason why i really don't want DBAPI to have a begin() - not all backends have such a concept. From jd at commandprompt.com Wed Sep 14 03:15:12 2011 From: jd at commandprompt.com (Joshua D. Drake) Date: Tue, 13 Sep 2011 18:15:12 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> Message-ID: <4E700020.8050505@commandprompt.com> Hello, So, I have been a little out of the loop on DBAPI2 for a while. I just went back and read it and would like to change my argument a bit. (For some bizarre reason I was thinking that .begin() was part of the spec). It seems to me that the default behavior as it is implicit, should not be autocommit style. It should be standard transaction style. Therefore based on the OP: """ .commit() Commit any pending transaction to the database. Note that if the database supports an auto-commit feature, this must be initially off. An interface method may be provided to turn it back on. """ At least from a PostgreSQL perspective this is invalid. If you do not issue a begin (implicitly or not), it is going to work in an autocommit style mode and you can't turn it off. Since we (python) issue an implicit begin when we create a connection, PostgreSQL automatically turns off autocommit (fun huh?). One way to handle this would be to have a connection property that explicitly turns on autocommit. If set to TRUE, PostgreSQL would just act in its default behavior, if set to FALSE it would implicitly create a transaction. Even with a long lived connection you could always send a BEGIN; explicitly and a proper transaction would start. You would just have to remember to call .commit() in order for your data to be commited after the BEGIN; If you called .commit() in autocommit mode you just get a warning that says no transaction in progress. I think we the ability to call an explicit .begin() that can be implemented properly up to the driver level. This: conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) cur = conn.cursor() cur.execute("""INSERT""") cur.execute("""SELECT * from bar""") cur.begin() cur.execute("""INSERT""") cur.commit() cur.close() Is much better than: conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) cur = conn.cursor() cur.execute("""INSERT""") cur.execute("""SELECT * from bar""") cur.execute("""BEGIN""") cur.execute("""INSERT""") cur.commit() cur.close() Sincerely, Joshua D. Drake P.S. And since we are on the topic, we really need proper prepare() -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL Conference - http://www.postgresqlconference.org/ @cmdpromptinc - @postgresconf - 509-416-6579 From mike_mp at zzzcomputing.com Wed Sep 14 06:12:09 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Wed, 14 Sep 2011 00:12:09 -0400 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E700020.8050505@commandprompt.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> <4E700020.8050505@commandprompt.com> Message-ID: <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> On Sep 13, 2011, at 9:15 PM, Joshua D. Drake wrote: > > conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) > cur = conn.cursor() > cur.execute("""INSERT""") > cur.execute("""SELECT * from bar""") > cur.begin() > cur.execute("""INSERT""") > cur.commit() > cur.close() > > Is much better than: > > conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) > cur = conn.cursor() > cur.execute("""INSERT""") > cur.execute("""SELECT * from bar""") > cur.execute("""BEGIN""") > cur.execute("""INSERT""") > cur.commit() > cur.close() > what's wrong with: conn = psycopg2.connect(....) cur = conn.cursor() cur.execute(" ... sql ...") conn.commit() ? if one wants to work with transactions, DBAPI in it's default mode provides that. If you want all statements to be ad-hoc such that they are "committed" immediately, whether that means COMMIT after every statement as it does on Oracle or just no BEGIN emitted as it does on Postgresql, turn on "autocommit". "autocommit" is a flag you should only be able to change when no transactional state has otherwise accumulated, but otherwise, switch it any time. Adding explicit begin() means the DBAPI starts to look confused as to how it should be used - DBAPI authors also have to support two different transactional styles. When I first came to use DBAPI, I was taken aback by how it has no begin(), even though I had spent many years using Oracle, but I came to be used to it. I'd be curious to know the original rationale for it to be that way. If DBAPI only allowed transactions via begin() and commit(), that would be fine also, but its doubtful the current mode of operation could ever be removed. From mal at egenix.com Wed Sep 14 12:45:49 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 Sep 2011 12:45:49 +0200 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6E85A2.5020509@ingres.com> References: <4E6E85A2.5020509@ingres.com> Message-ID: <4E7085DD.60609@egenix.com> Chris Clark wrote: > Hi All, > > I was discussing with someone today autocommit support and found an area > of pep-249 where I think we could improve the wording (and the spec): > > http://www.python.org/dev/peps/pep-0249/ > > ..... > > .commit() > Commit any pending transaction to the database. Note > that > if the database supports an auto-commit feature, this must > be initially off. An interface method may be provided to > turn it back on. > > ..... > > It is the last sentence that I'm looking at: > >> An interface method may be provided to turn it back on. > > Comments: > > * The "may" clearly marks this as an optional interface (and I'm > happy about that). > * It would be helpful to make clear what the interface method would > be so module authors can implement this consistently > * if autocommit can be turned on in a connection, there should have > a way to turn it off again > ... > Comments? mxODBC and mxODBC Connect will support a writable connection attribute connection.autocommit to turn the feature on/off. This works by assigning a boolean to the attribute. The attribute also allows for an easy way to check whether autocommit is active or not. They both already support doing this via the (non-standard) DB-API extension connection.setconnectoption() method. >From an API perspective, using an attribute is the right and user-friendly approach. My only gripe with this is the fact, that querying or setting the attribute can cause exceptions, which you'd normally only expect from function/method calls. Other than that I'm +1 on adding the attribute as standard DB-API 2.0 extension. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 14 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 20 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Sep 14 13:00:01 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 Sep 2011 13:00:01 +0200 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <1C86FB42-7484-47D2-A5A7-968FD5F2F466@zzzcomputing.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <4E6FA356.7060800@commandprompt.com> <1C86FB42-7484-47D2-A5A7-968FD5F2F466@zzzcomputing.com> Message-ID: <4E708931.6000005@egenix.com> Michael Bayer wrote: > > On Sep 13, 2011, at 2:39 PM, Joshua D. Drake wrote: > >> >> Oh now that is interesting. Hmmm... perhaps conn.autocommit() is actually a synonym for conn.begin() except that it sets the autocommit property? So we would have two semantics: >> >> conn.begin() -- Starts a transaction, will not commit unless explicit conn.commit() is called. >> >> conn.autocommit() -- Works in autocommit mode, calling conn.commit() is redundant, each statement is executed as it is delivered to the database. > > I'm -1 on conn.begin(). I've spent years explaining to users the beauty of the DBAPI in that it has no "begin" method. Providing for two different modes of doing the same thing, i.e. working with a standard DBAPI connection in a transaction, or forcing autocommit to True then using begin(), creates for a more confusing and controversial interface. -1 on connection.begin() as well. DB-API modules are free to implement this, but it should not be part of the DB-API standard, since it causes rather complicated transactional/non-transactional states on the connection. The simple implicit transaction start when opening a connection or doing a commit/rollback is much less error prone and protects the user from data corruption in error situations. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 14 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 20 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at ingres.com Wed Sep 14 18:51:35 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Wed, 14 Sep 2011 09:51:35 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E7085DD.60609@egenix.com> References: <4E6E85A2.5020509@ingres.com> <4E7085DD.60609@egenix.com> Message-ID: <4E70DB97.4060506@ingres.com> M.-A. Lemburg wrote: > Chris Clark wrote: > >> Hi All, >> >> I was discussing with someone today autocommit support and found an area >> of pep-249 where I think we could improve the wording (and the spec): >> >> http://www.python.org/dev/peps/pep-0249/ >> >> ..... >> >> .commit() >> Commit any pending transaction to the database. Note >> that >> if the database supports an auto-commit feature, this must >> be initially off. An interface method may be provided to >> turn it back on. >> >> ..... >> >> It is the last sentence that I'm looking at: >> >> >>> An interface method may be provided to turn it back on. >>> >> Comments: >> >> * The "may" clearly marks this as an optional interface (and I'm >> happy about that). >> * It would be helpful to make clear what the interface method would >> be so module authors can implement this consistently >> * if autocommit can be turned on in a connection, there should have >> a way to turn it off again >> ... >> Comments? >> > > mxODBC and mxODBC Connect will support a writable connection > attribute connection.autocommit to turn the feature on/off. > This works by assigning a boolean to the attribute. The > attribute also allows for an easy way to check whether autocommit > is active or not. > > They both already support doing this via the (non-standard) DB-API > extension connection.setconnectoption() method. > > From an API perspective, using an attribute is the right > and user-friendly approach. My only gripe with this is the fact, > that querying or setting the attribute can cause exceptions, > which you'd normally only expect from function/method calls. > > Other than that I'm +1 on adding the attribute as standard > DB-API 2.0 extension. > > Thanks M.A.. It looks like we are heading towards a consensus on an exception being used (when mid transaction and autocommit on is requested). So we should document that, I'm inclined to be a little whishy-washy and have a short line along the lines of: If an attempt is made to enable autocommit mid transactions this is likely to cause a DBI exception, the exception raised may vary between drivers/backends but will be a sub-class of driver.Error. I'd like to try and cover the case mentioned above, where "*querying* or setting the attribute can cause exceptions". Is this likely? If so we should document that with a one liner too. I'd like to steal your wording for documentation, I've only changed the first few words: --------- A connection may optionally support a writable connection attribute connection.autocommit to turn the feature on/off. This works by assigning a boolean to the attribute. The attribute also allows for an easy way to check whether autocommit is active or not. If an attempt is made to enable autocommit mid transactions this is likely to cause a DBI exception, the exception raised may vary between drivers/backends but will be a sub-class of driver.Error. A driver may choose to implement autocommit manually (that is, manually issue commits at the end of each cursor operation), to raise an exception if mid transaction, or pass the autocommit request to the backend and raise an exception for errors that the backend raises. --------- It may be that we have more discussion on the commit/rollback options but I'm not planning on raising those options again unless there is interest from other posters. RE the conn.begin() that is a -1 from me, this seems like a separate item to autocommit and may deserve a new thread. This doesn't prevent implementers from adding a non-standard exception though if it will prove to be useful for some users. Chris From jd at commandprompt.com Wed Sep 14 18:56:32 2011 From: jd at commandprompt.com (Joshua D. Drake) Date: Wed, 14 Sep 2011 09:56:32 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> <4E700020.8050505@commandprompt.com> <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> Message-ID: <4E70DCC0.7070101@commandprompt.com> On 09/13/2011 09:12 PM, Michael Bayer wrote: > > > On Sep 13, 2011, at 9:15 PM, Joshua D. Drake wrote: > >> >> conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) >> cur = conn.cursor() >> cur.execute("""INSERT""") >> cur.execute("""SELECT * from bar""") >> cur.begin() >> cur.execute("""INSERT""") >> cur.commit() >> cur.close() >> >> Is much better than: >> >> conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) >> cur = conn.cursor() >> cur.execute("""INSERT""") >> cur.execute("""SELECT * from bar""") >> cur.execute("""BEGIN""") >> cur.execute("""INSERT""") >> cur.commit() >> cur.close() >> > > what's wrong with: > > conn = psycopg2.connect(....) > cur = conn.cursor() > cur.execute(" ... sql ...") > conn.commit() Nothing, that doesn't actually do anything I didn't suggest above. > > ? if one wants to work with transactions, DBAPI in it's default mode provides that. Correct. > If you want all statements to be ad-hoc such that they are "committed" immediately, whether that means COMMIT after every statement as it does on Oracle or just no BEGIN emitted as it does on Postgresql, turn on "autocommit". "autocommit" is a flag you should only be able to change when no transactional state has otherwise accumulated, but otherwise, switch it any time. Yes. > > Adding explicit begin() means the DBAPI starts to look confused as to how it should be used - DBAPI authors also have to support two different transactional styles. No. What I am saying is, the begin() is optional. If you look at my example, if PostgreSQL is in autocommit mode, the only way to start a transaction is to execute a query to start the transaction (BEGIN or START TRANSACTION). From a, "it would be nice if" perspective, I would like to have the option of using .begin() to start a transaction. That seems to jive better with proper coding. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL Conference - http://www.postgresqlconference.org/ @cmdpromptinc - @postgresconf - 509-416-6579 From Chris.Clark at ingres.com Wed Sep 14 19:11:24 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Wed, 14 Sep 2011 10:11:24 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E70DCC0.7070101@commandprompt.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> <4E700020.8050505@commandprompt.com> <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> <4E70DCC0.7070101@commandprompt.com> Message-ID: <4E70E03C.5040408@ingres.com> Joshua D. Drake wrote: > > On 09/13/2011 09:12 PM, Michael Bayer wrote: >> >> >> On Sep 13, 2011, at 9:15 PM, Joshua D. Drake wrote: >> >>> >>> conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) >>> cur = conn.cursor() >>> cur.execute("""INSERT""") >>> cur.execute("""SELECT * from bar""") >>> cur.begin() >>> cur.execute("""INSERT""") >>> cur.commit() >>> cur.close() >>> >>> Is much better than: >>> >>> conn=psycopg2.connect("dbname='foo' user='dbuser', AUTOCOMMIT=TRUE) >>> cur = conn.cursor() >>> cur.execute("""INSERT""") >>> cur.execute("""SELECT * from bar""") >>> cur.execute("""BEGIN""") >>> cur.execute("""INSERT""") >>> cur.commit() >>> cur.close() >>> >> >> what's wrong with: >> >> conn = psycopg2.connect(....) >> cur = conn.cursor() >> cur.execute(" ... sql ...") >> conn.commit() > > Nothing, that doesn't actually do anything I didn't suggest above. > >> >> ? if one wants to work with transactions, DBAPI in it's default >> mode provides that. > > Correct. > >> If you want all statements to be ad-hoc such that they are >> "committed" immediately, whether that means COMMIT after every >> statement as it does on Oracle or just no BEGIN emitted as it does on >> Postgresql, turn on "autocommit". "autocommit" is a flag you should >> only be able to change when no transactional state has otherwise >> accumulated, but otherwise, switch it any time. > > Yes. > >> >> Adding explicit begin() means the DBAPI starts to look confused as to >> how it should be used - DBAPI authors also have to support two >> different transactional styles. > > No. What I am saying is, the begin() is optional. If you look at my > example, if PostgreSQL is in autocommit mode, the only way to start a > transaction is to execute a query to start the transaction (BEGIN or > START TRANSACTION). From a, "it would be nice if" perspective, I would > like to have the option of using .begin() to start a transaction. That > seems to jive better with proper coding. I think the semantics you are describing are: 1. auto commit mode is on 2. whilst in auto commit mode the user wants to start a transaction that will not be autocommit The above is a guess I'm making, ;-) Those semantics are very different to other DBMS implementations and maybe why there is some confusion over the suggestion. Chris From mike_mp at zzzcomputing.com Wed Sep 14 19:25:46 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Wed, 14 Sep 2011 13:25:46 -0400 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E70DCC0.7070101@commandprompt.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> <4E700020.8050505@commandprompt.com> <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> <4E70DCC0.7070101@commandprompt.com> Message-ID: <0C3D3FD0-2255-4C70-AE9F-C1EFF8528866@zzzcomputing.com> On Sep 14, 2011, at 12:56 PM, Joshua D. Drake wrote: > No. What I am saying is, the begin() is optional. If you look at my example, if PostgreSQL is in autocommit mode, the only way to start a transaction is to execute a query to start the transaction (BEGIN or START TRANSACTION). From a, "it would be nice if" perspective, I would like to have the option of using .begin() to start a transaction. That seems to jive better with proper coding. > "proper" on postgresql sure. On Oracle, there is no BEGIN. I like that the DBAPI tries to simplify this disconnect with a consistent, moderately coarse-grained facade rather than confusing users and burdening DBAPI authors with two ways to do it. From jd at commandprompt.com Wed Sep 14 20:00:19 2011 From: jd at commandprompt.com (Joshua D. Drake) Date: Wed, 14 Sep 2011 11:00:19 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <0C3D3FD0-2255-4C70-AE9F-C1EFF8528866@zzzcomputing.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> <4E700020.8050505@commandprompt.com> <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> <4E70DCC0.7070101@commandprompt.com> <0C3D3FD0-2255-4C70-AE9F-C1EFF8528866@zzzcomputing.com> Message-ID: <4E70EBB3.9040405@commandprompt.com> On 09/14/2011 10:25 AM, Michael Bayer wrote: > > > On Sep 14, 2011, at 12:56 PM, Joshua D. Drake wrote: > >> No. What I am saying is, the begin() is optional. If you look at my example, if PostgreSQL is in autocommit mode, the only way to start a transaction is to execute a query to start the transaction (BEGIN or START TRANSACTION). From a, "it would be nice if" perspective, I would like to have the option of using .begin() to start a transaction. That seems to jive better with proper coding. >> > > "proper" on postgresql sure. On Oracle, there is no BEGIN. I like that the DBAPI tries to simplify this disconnect with a consistent, moderately coarse-grained facade rather than confusing users and burdening DBAPI authors with two ways to do it. > Thus the "optional". Those that can make use of it can, those that don't don't need it or it will be silently ignored. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development The PostgreSQL Conference - http://www.postgresqlconference.org/ @cmdpromptinc - @postgresconf - 509-416-6579 From mal at egenix.com Wed Sep 14 20:18:06 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 Sep 2011 20:18:06 +0200 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E70DB97.4060506@ingres.com> References: <4E6E85A2.5020509@ingres.com> <4E7085DD.60609@egenix.com> <4E70DB97.4060506@ingres.com> Message-ID: <4E70EFDE.8040101@egenix.com> Chris Clark wrote: > M.-A. Lemburg wrote: >> Chris Clark wrote: >> >>> Hi All, >>> >>> I was discussing with someone today autocommit support and found an area >>> of pep-249 where I think we could improve the wording (and the spec): >>> >>> http://www.python.org/dev/peps/pep-0249/ >>> >>> ..... >>> >>> .commit() >>> Commit any pending transaction to the database. Note >>> that >>> if the database supports an auto-commit feature, this must >>> be initially off. An interface method may be provided to >>> turn it back on. >>> >>> ..... >>> >>> It is the last sentence that I'm looking at: >>> >>> >>>> An interface method may be provided to turn it back on. >>>> >>> Comments: >>> >>> * The "may" clearly marks this as an optional interface (and I'm >>> happy about that). >>> * It would be helpful to make clear what the interface method would >>> be so module authors can implement this consistently >>> * if autocommit can be turned on in a connection, there should have >>> a way to turn it off again >>> ... >>> Comments? >>> >> >> mxODBC and mxODBC Connect will support a writable connection >> attribute connection.autocommit to turn the feature on/off. >> This works by assigning a boolean to the attribute. The >> attribute also allows for an easy way to check whether autocommit >> is active or not. >> >> They both already support doing this via the (non-standard) DB-API >> extension connection.setconnectoption() method. >> >> From an API perspective, using an attribute is the right >> and user-friendly approach. My only gripe with this is the fact, >> that querying or setting the attribute can cause exceptions, >> which you'd normally only expect from function/method calls. >> >> Other than that I'm +1 on adding the attribute as standard >> DB-API 2.0 extension. >> >> > > Thanks M.A.. > > It looks like we are heading towards a consensus on an exception being > used (when mid transaction and autocommit on is requested). So we should > document that, I'm inclined to be a little whishy-washy and have a short > line along the lines of: > > If an attempt is made to enable autocommit mid transactions this is > likely to cause a DBI exception, the exception raised may vary between > drivers/backends but will be a sub-class of driver.Error. I actually like the way ODBC handles these cases: * Turning on autocommit has the same effect as doing a .commit() and then proceeding in autocommit mode. * Turning off autocommit causes a new transaction to start. but I guess we cannot require this from database modules, since the semantics are usually defined by the database backend and not the client lib or driver. > I'd like to try and cover the case mentioned above, where "*querying* or > setting the attribute can cause exceptions". Is this likely? If so we > should document that with a one liner too. It's not likely, but can happen, e.g. if the connection to the database is lost or the database doesn't support transactions or if the backend doesn't support transactions (turning autocommit off would then raise an exception). > I'd like to steal your wording for documentation, I've only changed the > first few words: > > --------- > A connection may optionally support a writable connection > attribute connection.autocommit to turn the feature on/off. > This works by assigning a boolean to the attribute. The > attribute also allows for an easy way to check whether autocommit > is active or not. > > If an attempt is made to enable autocommit mid transactions this is > likely to cause a DBI exception, the exception raised may vary between > drivers/backends but will be a sub-class of driver.Error. A driver may > choose to implement autocommit manually (that is, manually issue commits > at the end of each cursor operation), to raise an exception if mid > transaction, or pass the autocommit request to the backend and raise an > exception for errors that the backend raises. > --------- I'm not sure about that last paragraph. The exception raising part should really just be about turning autocommit on/off, not about the way exceptions are raised as a result of the autocommit implementation during cursor operations. > It may be that we have more discussion on the commit/rollback options > but I'm not planning on raising those options again unless there is > interest from other posters. > > RE the conn.begin() that is a -1 from me, this seems like a separate > item to autocommit and may deserve a new thread. This doesn't prevent > implementers from adding a non-standard exception though if it will > prove to be useful for some users. Agreed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 14 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 20 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Chris.Clark at ingres.com Wed Sep 14 20:34:11 2011 From: Chris.Clark at ingres.com (Chris Clark) Date: Wed, 14 Sep 2011 11:34:11 -0700 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E70EFDE.8040101@egenix.com> References: <4E6E85A2.5020509@ingres.com> <4E7085DD.60609@egenix.com> <4E70DB97.4060506@ingres.com> <4E70EFDE.8040101@egenix.com> Message-ID: <4E70F3A3.6020505@ingres.com> M.-A. Lemburg wrote: > Chris Clark wrote: > >> M.-A. Lemburg wrote: >> >>> Chris Clark wrote: >>> >>> >>>> Hi All, >>>> >>>> I was discussing with someone today autocommit support and found an area >>>> of pep-249 where I think we could improve the wording (and the spec): >>>> >>>> http://www.python.org/dev/peps/pep-0249/ >>>> >>>> ..... >>>> >>>> .commit() >>>> Commit any pending transaction to the database. Note >>>> that >>>> if the database supports an auto-commit feature, this must >>>> be initially off. An interface method may be provided to >>>> turn it back on. >>>> >>>> ..... >>>> >>>> It is the last sentence that I'm looking at: >>>> >>>> >>>> >>>>> An interface method may be provided to turn it back on. >>>>> >>>>> >>>> Comments: >>>> >>>> * The "may" clearly marks this as an optional interface (and I'm >>>> happy about that). >>>> * It would be helpful to make clear what the interface method would >>>> be so module authors can implement this consistently >>>> * if autocommit can be turned on in a connection, there should have >>>> a way to turn it off again >>>> ... >>>> Comments? >>>> >>>> >>> mxODBC and mxODBC Connect will support a writable connection >>> attribute connection.autocommit to turn the feature on/off. >>> This works by assigning a boolean to the attribute. The >>> attribute also allows for an easy way to check whether autocommit >>> is active or not. >>> >>> They both already support doing this via the (non-standard) DB-API >>> extension connection.setconnectoption() method. >>> >>> From an API perspective, using an attribute is the right >>> and user-friendly approach. My only gripe with this is the fact, >>> that querying or setting the attribute can cause exceptions, >>> which you'd normally only expect from function/method calls. >>> >>> Other than that I'm +1 on adding the attribute as standard >>> DB-API 2.0 extension. >>> >>> >>> >> Thanks M.A.. >> >> It looks like we are heading towards a consensus on an exception being >> used (when mid transaction and autocommit on is requested). So we should >> document that, I'm inclined to be a little whishy-washy and have a short >> line along the lines of: >> >> If an attempt is made to enable autocommit mid transactions this is >> likely to cause a DBI exception, the exception raised may vary between >> drivers/backends but will be a sub-class of driver.Error. >> > > I actually like the way ODBC handles these cases: > > * Turning on autocommit has the same effect as doing a .commit() > and then proceeding in autocommit mode. > > * Turning off autocommit causes a new transaction to start. > > but I guess we cannot require this from database modules, since > the semantics are usually defined by the database backend and not > the client lib or driver. > Aghhhh! ;-) That's what I initial sent out in my first proposal. I think the "switching on autocommit, causes a commit" behavior is easy to document and easy to implement. The driver could easily issue the commit under the covers when autocommit is requested. I.e. it seems reasonable for the driver to over ride the backend behavior to provide a consistent interface across all backends. I'm not clear if the exception approach can be implemented consistently. DBI drivers already have to provide a consistent approach to transactions by not needing a "begin transaction" call for users. >> I'd like to try and cover the case mentioned above, where "*querying* or >> setting the attribute can cause exceptions". Is this likely? If so we >> should document that with a one liner too. >> > > It's not likely, but can happen, e.g. if the connection to the > database is lost or the database doesn't support transactions or > if the backend doesn't support transactions (turning autocommit > off would then raise an exception). > That makes sense, I have to confess I'd been envisioning the driver keeping track of the auto commit state and assuming that if the user sent a cursor.execute('set auto commit on') it was reasonble for the driver to not know about it. Obviously querying the layer below would avoid that. Thanks for explaining that, I'm glad you did but I've changed my mind about documenting it :-) >> I'd like to steal your wording for documentation, I've only changed the >> first few words: >> >> --------- >> A connection may optionally support a writable connection >> attribute connection.autocommit to turn the feature on/off. >> This works by assigning a boolean to the attribute. The >> attribute also allows for an easy way to check whether autocommit >> is active or not. >> >> If an attempt is made to enable autocommit mid transactions this is >> likely to cause a DBI exception, the exception raised may vary between >> drivers/backends but will be a sub-class of driver.Error. A driver may >> choose to implement autocommit manually (that is, manually issue commits >> at the end of each cursor operation), to raise an exception if mid >> transaction, or pass the autocommit request to the backend and raise an >> exception for errors that the backend raises. >> --------- >> > > I'm not sure about that last paragraph. The exception raising > part should really just be about turning autocommit on/off, not about > the way exceptions are raised as a result of the autocommit > implementation during cursor operations. > I not fond of it either :-( possibly for different reasons. The behavior I described above is not clearly documented, I'm not sure if all backends can/will raise an error. This is partially why I suggested implicit commits as this is much easier for a user to grok. Chris From mal at egenix.com Wed Sep 14 20:38:15 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 Sep 2011 20:38:15 +0200 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E70EBB3.9040405@commandprompt.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E6F88A2.80804@commandprompt.com> <4E6F988C.4000704@ingres.com> <2C601EA2-B2F3-4DE0-9306-12BBEFAF19E1@zzzcomputing.com> <4E700020.8050505@commandprompt.com> <2BA493DF-441A-4F9B-B25E-36F006BDD9F8@zzzcomputing.com> <4E70DCC0.7070101@commandprompt.com> <0C3D3FD0-2255-4C70-AE9F-C1EFF8528866@zzzcomputing.com> <4E70EBB3.9040405@commandprompt.com> Message-ID: <4E70F497.3020802@egenix.com> Joshua D. Drake wrote: > > On 09/14/2011 10:25 AM, Michael Bayer wrote: >> >> >> On Sep 14, 2011, at 12:56 PM, Joshua D. Drake wrote: >> >>> No. What I am saying is, the begin() is optional. If you look at my >>> example, if PostgreSQL is in autocommit mode, the only way to start a >>> transaction is to execute a query to start the transaction (BEGIN or >>> START TRANSACTION). From a, "it would be nice if" perspective, I >>> would like to have the option of using .begin() to start a >>> transaction. That seems to jive better with proper coding. >>> >> >> "proper" on postgresql sure. On Oracle, there is no BEGIN. I like >> that the DBAPI tries to simplify this disconnect with a consistent, >> moderately coarse-grained facade rather than confusing users and >> burdening DBAPI authors with two ways to do it. >> > > Thus the "optional". Those that can make use of it can, those that don't > don't need it or it will be silently ignored. I find the following is more readable (explicit better then implicit etc.): connection = connect(dbname, uid, pwd) # Regular user code: ... transactional code ... # Experts only section: try: connection.autocommit = True ... non-transactional code ... finally: connection.autocommit = False # Regular user code: ... transactional code ... connection.commit() You usually only need autocommit code in case you are dealing with databases that cannot handle database definition SQL (DDL) inside transactions (e.g. create/drop databases or tables), if you have a need for atomic operations, or need to avoid locking for some special reason. In practice, it's often better to open a separate connection in autocommit mode, than to switch modes on a single connection. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 14 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 20 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fog at dndg.it Wed Sep 14 14:16:22 2011 From: fog at dndg.it (Federico Di Gregorio) Date: Wed, 14 Sep 2011 14:16:22 +0200 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: <4E6F8739.3010403@ingres.com> References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> Message-ID: <4E709B16.2060009@dndg.it> On 13/09/11 18:39, Chris Clark wrote: [snip] >>> The behaviour of setting autocommit when a transaction is already open >>> is also to be seen: there are three basic option: >>> >>> - have a commit >>> - have a rollback >>> - have an error >>> >>> I feel the implicit commit is the most dangerous option. An implicit >>> rollback may be more acceptable (it's the same behaviour of closing >>> the connection with a transaction in progress): I proposed it for >>> psycopg but, in the discussion, raising an exception proved the most >>> popular option. >>> >>> >> I don't think I know how to even determine whether there is a >> transaction in progress in ADO.(*) If there is a way, then checking >> on it just so that I can raise an exception seems expensive. Remember >> that most of the time the resulting stack trace would not be seen by >> the programmer who goofed up, but a completely innocent user -- and to >> her it looks like the program just crashed for no reason. The >> rollback would be easy to program and user friendly. I vote rollback. >> > > The exception route has the potential to be the easiest for most DBMS > implementations, most DBMSs will raise an error, so the driver doesn't > need to track state and can just propagate the DBMS error. For some > backends, state tracking would be required which is why I suggested the > "easy" commit option. Rollback seems semi safe BUT when setting > autocommit to ON the default behavior of a rollback seems a little > surprising. > > I'm not in love with committing when the autocommit state is changed but > for the "enable auto commit" operation it seems logical, the user is > requesting commits take place under the covers. Conversely when > disabling autocommit (after it has previously been enabled) we are > either not in a transaction (due to auto commit being on) or we could > still be fetching mid select, in which case a commit would be the same > as a rollback. I.e. only impact on locking, any writes to the database > would have been already auto committed. [snap] Client code can set .autocommit (yes, I am +1 for the attribute) to true by two different code paths: the "correct" one and the "buggy" one. The correct code path is not a problem: as long as the programmer knows the behaviour she can do the right thing: 1. exception => make sure to commit() or rollback() before, depending on the wanted outcome; 2. commit => rollback() if you don't want to commit changes, do nothing in the other case; 3. rollback => as (2), reversed. What I am interested in is the behaviour of the driver when the code sets .autocommit to true following a buggy code path, i.e., when there is a pending transaction and the code is unaware of it: 1. exception => pending transaction is lost, user (and eventually also the programmer) gets a stack trace: this is good because the user is sure about the outcome (data is lost) _and_ has information about the problem; 2. commit => pending transaction is commited but it is what the code was really supposed to do? noboby will know until someone peeks at the database and _then_ a possibly difficult bug hunting session begins; 3. rollback => as (2), but data is lost. Having commit or rollbacks happen because of a bug really scares me: spurious data starts to popup into the database and finding the bugs usually isn't a piece of cake. That's why I vote for the exception. federico -- Federico Di Gregorio fog at initd.org If nobody understand you, that doesn't mean you're an artist. -- anonymous From vernondcole at gmail.com Thu Sep 15 12:26:08 2011 From: vernondcole at gmail.com (Vernon Cole) Date: Thu, 15 Sep 2011 04:26:08 -0600 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E709B16.2060009@dndg.it> Message-ID: (I missed replying to the group. --- buggy mental code path ...;) ---------- Forwarded message ---------- From: Vernon Cole Date: Thu, Sep 15, 2011 at 4:23 AM Subject: Re: [DB-SIG] autocommit support in pep-249 To: Federico Di Gregorio On Wed, Sep 14, 2011 at 6:16 AM, Federico Di Gregorio wrote: > [snap] > > Client code can set .autocommit (yes, I am +1 for the attribute) to true > by two different code paths: the "correct" one and the "buggy" one. The > correct code path is not a problem: as long as the programmer knows the > behaviour she can do the right thing: > > 1. exception => make sure to commit() or rollback() before, > depending on the wanted outcome; > 2. commit => rollback() if you don't want to commit changes, > do nothing in the other case; > 3. rollback => as (2), reversed. > > What I am interested in is the behaviour of the driver when the code > sets .autocommit to true following a buggy code path, i.e., when there > is a pending transaction and the code is unaware of it: > > 1. exception => pending transaction is lost, user (and eventually > also the programmer) gets a stack trace: this is > good because the user is sure about the outcome (data > is lost) _and_ has information about the problem; > 2. commit => pending transaction is commited but it is what the > code was really supposed to do? noboby will know until > someone peeks at the database and _then_ a possibly > difficult bug hunting session begins; > 3. rollback => as (2), but data is lost. > > Having commit or rollbacks happen because of a bug really scares me: > spurious data starts to popup into the database and finding the bugs > usually isn't a piece of cake. That's why I vote for the exception. > > federico > > It's those buggy code paths that we are discussing, and I am much more interested in doing the right thing on a good code path. We're using Python here, not Pascal. If I wanted to force all programmers to be perfect, I'ld require that all variables be declared. What we are discussing here is what to do in a situation which a wise programmer will never get in to. I don't want to waste a lot of time and a lot of code making sure that the results of poor programming are consistent between different database engines. I would advocate that the results of this questionable practice should be explicitly left as "undefined", with the cautionary note that an exception MIGHT occur, and suggest that the application programmer should either commit or rollback any in-progress transactions before changing the value of autocommit. That way no complicated parsing or state tracking is required. My __setattr__ for .autocommit would simply pass the new value on to the ADO engine to do whatever it pleased with it -- and pass back any error which the engine might decide to return. I LIKE simple -- it usually works better. -- Vernon -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Sep 15 12:52:43 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 15 Sep 2011 12:52:43 +0200 Subject: [DB-SIG] autocommit support in pep-249 In-Reply-To: References: <4E6E85A2.5020509@ingres.com> <4E6F8739.3010403@ingres.com> <4E709B16.2060009@dndg.it> Message-ID: <4E71D8FB.10502@egenix.com> Vernon Cole wrote: > (I missed replying to the group. --- buggy mental code path ...;) > > ---------- Forwarded message ---------- > From: Vernon Cole > Date: Thu, Sep 15, 2011 at 4:23 AM > Subject: Re: [DB-SIG] autocommit support in pep-249 > To: Federico Di Gregorio > > > > > On Wed, Sep 14, 2011 at 6:16 AM, Federico Di Gregorio wrote: > >> [snap] >> >> Client code can set .autocommit (yes, I am +1 for the attribute) to true >> by two different code paths: the "correct" one and the "buggy" one. The >> correct code path is not a problem: as long as the programmer knows the >> behaviour she can do the right thing: >> >> 1. exception => make sure to commit() or rollback() before, >> depending on the wanted outcome; >> 2. commit => rollback() if you don't want to commit changes, >> do nothing in the other case; >> 3. rollback => as (2), reversed. >> >> What I am interested in is the behaviour of the driver when the code >> sets .autocommit to true following a buggy code path, i.e., when there >> is a pending transaction and the code is unaware of it: >> >> 1. exception => pending transaction is lost, user (and eventually >> also the programmer) gets a stack trace: this is >> good because the user is sure about the outcome (data >> is lost) _and_ has information about the problem; >> 2. commit => pending transaction is commited but it is what the >> code was really supposed to do? noboby will know until >> someone peeks at the database and _then_ a possibly >> difficult bug hunting session begins; >> 3. rollback => as (2), but data is lost. >> >> Having commit or rollbacks happen because of a bug really scares me: >> spurious data starts to popup into the database and finding the bugs >> usually isn't a piece of cake. That's why I vote for the exception. >> >> federico >> >> It's those buggy code paths that we are discussing, and I am much more > interested in doing the right thing on a good code path. We're using Python > here, not Pascal. If I wanted to force all programmers to be perfect, I'ld > require that all variables be declared. What we are discussing here is > what to do in a situation which a wise programmer will never get in to. I > don't want to waste a lot of time and a lot of code making sure that the > results of poor programming are consistent between different database > engines. I would advocate that the results of this questionable practice > should be explicitly left as "undefined", with the cautionary note that an > exception MIGHT occur, and suggest that the application programmer should > either commit or rollback any in-progress transactions before changing the > value of autocommit. > > That way no complicated parsing or state tracking is required. My > __setattr__ for .autocommit would simply pass the new value on to the ADO > engine to do whatever it pleased with it -- and pass back any error which > the engine might decide to return. > > I LIKE simple -- it usually works better. I agree with Vernon: the DB-API cannot mandate an exception in case of a pending transaction, since the module may very well not be aware of such a pending transaction: Some databases manage transactions on the server side and it's also possible to run COMMIT and ROLLBACK using cursor.execute() (even though that is discouraged). If you then tell the user that an exception will be raised in case of a pending transaction, she will feel mistakenly secure about doing connection.autocommit = True without an explicit .commit() or .rollback(). I find the implicit .commit() of setting connection.autocommit = True very intuitive. After all, you expect the connection to take care of the .commit()s for you when doing so. Likewise, setting the attribute to False, means that the programmer has to take care of the .commit()s again. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 19 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/