From anthony.tuininga at gmail.com Wed Oct 12 04:43:45 2011 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Tue, 11 Oct 2011 20:43:45 -0600 Subject: [DB-SIG] cx_Oracle 5.1.1 Message-ID: What is cx_Oracle? cx_Oracle is a Python extension module that allows access to Oracle and conforms to the Python database API 2.0 specifications with a few exceptions. Where do I get it? http://cx-oracle.sourceforge.net What's new? 1) Simplify management of threads for callbacks performed by database change notification and eliminate a crash that occurred under high load in certain situations. Thanks to Calvin S. for noting the issue and suggesting a solution and testing the patch. 2) Force server detach on close so that the connection is completely closed and not just the session as before. 3) Force use of OCI_UTF16ID for NCLOBs as using the default character set would result in ORA-03127 with Oracle 11.2.0.2 and UTF8 character set. 4) Avoid attempting to clear temporary LOBs a second time when destroying the variable as in certain situations this results in spurious errors. 5) Added additional parameter service_name to makedsn() which can be used to use the service_name rather than the SID in the DSN string that is generated. 6) Fix cursor description in test suite to take into account the number of bytes per character. 7) Added tests for NCLOBS to the test suite. 8) Removed redundant code in setup.py for calculating the library path. From anthony.tuininga at gmail.com Sat Oct 15 22:54:02 2011 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Sat, 15 Oct 2011 14:54:02 -0600 Subject: [DB-SIG] ceODBC 2.0.1 Message-ID: What is ceODBC? ceODBC is a Python extension module that enables access to databases using the ODBC API and conforms to the Python database API 2.0 specifications with a few additions. I have tested this on Windows against SQL Server, Access, dBASE and Oracle and others have reported success on more obscure drivers. On Linux I have tested this against PostgreSQL and MySQL. Where do I get it? http://ceodbc.sourceforge.net What's new? 1) Removed memory leak that occurred when binding parameters to a cursor; thanks to Robert Ritchie and Don Reid for discovering this. 2) Remove the password from the DSN in order to eliminate potential security leaks. 3) Improve performance when logging is disabled or not at level DEBUG by avoiding the entire attempt to log bind variable values. 4) Use the size value rather than the length value when defining result set variables since the length value is for the length of the column name; thanks to Heran Quan for the patch. 5) Added support for Python 3.2. From daniele.varrazzo at gmail.com Wed Oct 19 02:53:07 2011 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Wed, 19 Oct 2011 01:53:07 +0100 Subject: [DB-SIG] conn.close() idempotence Message-ID: Hello, in a discussion on the psycopg mailing list, an user has been surprised by the fact that .close() called on a closed connection raised error. I thought this was an implementation accident in the driver and a more robust semantics should allow for close to be called on a closed object without effect. But going to code a forgiving double close, I found a test failing in the Stuart Bishop DBAPI unit test, # connection.close should raise an Error if called more than once self.assertRaises(self.driver.Error,con.close) I see no reason to require this, and it actually creates some problem. For example, if the connection is dropped for a network problem, the driver may detect the invalid connection and put the connection in closed status. But at this point any guard such as: conn = connection() try: # work finally: conn.close() risks to become a problem, with the close() that may raise an error because the connection has been implicitly closed by a communication error. Note that close() in itself often is a quite safe operation, not involving database communication, so it is not expected to fail and may not be well guarded. The DBAPI says about conn.close(): The connection will be unusable from this point forward; an Error (or subclass) exception will be raised if any operation is attempted with the connection. now, is conn.close() an "operation" on the connection? Even if it doesn't involve server communication? I don't think an idempotent close() violates this rule, so I'm really asking about an interpretation of the DBAPI, not a change. Another user in the ML pointed out that an idempotent close is also the behaviour of the Python file objects. My opinion is that the DBAPI test suite can be relaxed and conn.close() should have no effect on a closed connection. What do you think? -- Daniele From mike_mp at zzzcomputing.com Wed Oct 19 05:36:04 2011 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Tue, 18 Oct 2011 23:36:04 -0400 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: References: Message-ID: On Oct 18, 2011, at 8:53 PM, Daniele Varrazzo wrote: > > The DBAPI says about conn.close(): > > The connection will be unusable from this point > forward; an Error (or subclass) exception will be raised > if any operation is attempted with the connection. > > now, is conn.close() an "operation" on the connection? Even if it > doesn't involve server communication? I don't think an idempotent > close() violates this rule, so I'm really asking about an > interpretation of the DBAPI, not a change. > > Another user in the ML pointed out that an idempotent close is also > the behaviour of the Python file objects. > > My opinion is that the DBAPI test suite can be relaxed and > conn.close() should have no effect on a closed connection. What do you > think? I'm +1 for close() on a closed connection being harmless. rollback() already has this behavior, as the spec specifies "rolls back the start of any pending transaction", thus allowing for no activity if no transaction is present. From mal at egenix.com Wed Oct 19 09:59:52 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 19 Oct 2011 09:59:52 +0200 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: References: Message-ID: <4E9E8378.10800@egenix.com> Daniele Varrazzo wrote: > Hello, > > in a discussion on the psycopg mailing list, an user has been > surprised by the fact that .close() called on a closed connection > raised error. I thought this was an implementation accident in the > driver and a more robust semantics should allow for close to be called > on a closed object without effect. But going to code a forgiving > double close, I found a test failing in the Stuart Bishop DBAPI unit > test, > > # connection.close should raise an Error if called more than once > self.assertRaises(self.driver.Error,con.close) This is a consequence of making the connection object unusable after .close() was called for the first time. > I see no reason to require this, and it actually creates some problem. > For example, if the connection is dropped for a network problem, the > driver may detect the invalid connection and put the connection in > closed status. But at this point any guard such as: > > conn = connection() > try: > # work > finally: > conn.close() > > risks to become a problem, with the close() that may raise an error > because the connection has been implicitly closed by a communication > error. Note that close() in itself often is a quite safe operation, > not involving database communication, so it is not expected to fail > and may not be well guarded. That last sentence is not quite correct: .close() issues an implicit .rollback() and usually also tells the database backend to free up resources maintained for the connection, so it does require communication with the backend. There are a few situations which can result in an exception: * the network has gone down or is temporarily interrupted * an operation is still pending completion on the connection, e.g. a two-phase commit or an asynchronously running statement * there's not enough memory available to complete the operation Note that in the above cases, .close() will not complete and thus also not necessarily put the connection in an unusable state. The exact semantics depend on the database backend. Example: You may have a temporary network error, so .close() issues an exception, but still allows to call .close() again after the issue has been resolved. > The DBAPI says about conn.close(): > > The connection will be unusable from this point > forward; an Error (or subclass) exception will be raised > if any operation is attempted with the connection. > > now, is conn.close() an "operation" on the connection? Even if it > doesn't involve server communication? I don't think an idempotent > close() violates this rule, so I'm really asking about an > interpretation of the DBAPI, not a change. See above. .close() does imply a few operations that can fail. You are basically suggesting to silence and ignore those exceptions. That's generally not a good idea, since the application may want to report those errors to the user or take some other action. > Another user in the ML pointed out that an idempotent close is also > the behaviour of the Python file objects. > > My opinion is that the DBAPI test suite can be relaxed and > conn.close() should have no effect on a closed connection. What do you > think? In summary, I'd rather not have .close() silence exceptions. What we could do is expose an attribute connection.closed which gets set to True by a successful run of connection.close(). That way you can test for the "unusable" state on a connection, e.g. if not connection.closed: connection.close() -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 19 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fog at initd.org Wed Oct 19 10:06:40 2011 From: fog at initd.org (Federico Di Gregorio) Date: Wed, 19 Oct 2011 10:06:40 +0200 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: <4E9E8378.10800@egenix.com> References: <4E9E8378.10800@egenix.com> Message-ID: <4E9E8510.7090105@initd.org> On 19/10/11 09:59, M.-A. Lemburg wrote: [snip] >> risks to become a problem, with the close() that may raise an error >> > because the connection has been implicitly closed by a communication >> > error. Note that close() in itself often is a quite safe operation, >> > not involving database communication, so it is not expected to fail >> > and may not be well guarded. > That last sentence is not quite correct: .close() issues an implicit > .rollback() and usually also tells the database backend to free up > resources maintained for the connection, so it does require communication > with the backend. Here is the problem. Does .close() always issue an implicit .rollback()? The DBAPI says yes but, as noted previously, the driver can choose to NOT send a rollback. In fact on the second .close() the driver SHOULD NOT send a rollback because there is no transaction in progress. I'd vote for idempotent .close() interpretation too. federico -- Federico Di Gregorio fog at initd.org Una nazionale senza neanche una nazione. -- macchinavapore -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From mal at egenix.com Wed Oct 19 10:19:40 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 19 Oct 2011 10:19:40 +0200 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: <4E9E8510.7090105@initd.org> References: <4E9E8378.10800@egenix.com> <4E9E8510.7090105@initd.org> Message-ID: <4E9E881C.4030108@egenix.com> Federico Di Gregorio wrote: > On 19/10/11 09:59, M.-A. Lemburg wrote: > [snip] >>> risks to become a problem, with the close() that may raise an error >>>> because the connection has been implicitly closed by a communication >>>> error. Note that close() in itself often is a quite safe operation, >>>> not involving database communication, so it is not expected to fail >>>> and may not be well guarded. >> That last sentence is not quite correct: .close() issues an implicit >> .rollback() and usually also tells the database backend to free up >> resources maintained for the connection, so it does require communication >> with the backend. > > Here is the problem. Does .close() always issue an implicit .rollback()? Yes. > The DBAPI says yes but, as noted previously, the driver can choose to > NOT send a rollback. Not if it complies to the DB-API. > In fact on the second .close() the driver SHOULD > NOT send a rollback because there is no transaction in progress. The second .close() would in that case raise an exception as per the definition in the database, since it has become unusable :-) BTW: Regardless of whether the driver explicitly issues a rollback or not, the database backend will roll back the connection if the connection closes and there's a pending transaction. > I'd vote for idempotent .close() interpretation too. Please read my full reply: the implicit rollback is not the only operation that can fail. Silencing errors is not a good idea. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 19 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Wed Oct 19 10:47:12 2011 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Wed, 19 Oct 2011 09:47:12 +0100 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: <4E9E8378.10800@egenix.com> References: <4E9E8378.10800@egenix.com> Message-ID: On Wed, Oct 19, 2011 at 8:59 AM, M.-A. Lemburg wrote: >> risks to become a problem, with the close() that may raise an error >> because the connection has been implicitly closed by a communication >> error. Note that close() in itself often is a quite safe operation, >> not involving database communication, so it is not expected to fail >> and may not be well guarded. > > That last sentence is not quite correct: .close() issues an implicit > .rollback() and usually also tells the database backend to free up > resources maintained for the connection, so it does require communication > with the backend. If the rollback is implicit in the backend instead of in the driver, the driver can just cut the communication with the server and obtain the dbapi semantics without issuing a new command. While this is just a trick when close() is called explicitly, it is about necessary on del: not only sending a rollback is a complex operation to be executed by __del__ (just because: it can fail and del would swallow the exception, thing we don't like): in complex environments (such as nonblocking aynchronous communication) pushing the rollback to the server and reading its response implies interaction between other python code and the connection being destroyed, whose results are at best undetermined. >> now, is conn.close() an "operation" on the connection? Even if it >> doesn't involve server communication? I don't think an idempotent >> close() violates this rule, so I'm really asking about an >> interpretation of the DBAPI, not a change. > > See above. .close() does imply a few operations that can fail. > You are basically suggesting to silence and ignore those exceptions. > That's generally not a good idea, since the application may want > to report those errors to the user or take some other action. No, I'm not suggesting to silence exception: I'm suggesting to not enforce an exception throw when it is not necessary. > What we could do is expose an attribute connection.closed which > gets set to True by a successful run of connection.close(). > That way you can test for the "unusable" state on a connection, > e.g. > > if not connection.closed: > ? ? connection.close() We already have such attribute as an extension. I am basically suggesting this pattern to be implicit in the close() instead of having people to code it everywhere. -- Daniele From mal at egenix.com Wed Oct 19 10:49:14 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 19 Oct 2011 10:49:14 +0200 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: <4E9E881C.4030108@egenix.com> References: <4E9E8378.10800@egenix.com> <4E9E8510.7090105@initd.org> <4E9E881C.4030108@egenix.com> Message-ID: <4E9E8F0A.7030303@egenix.com> Thinking about this some more ... Perhaps what you're really after is the following and I was just misunderstanding the original proposal: class Connection: closed = False def close(self): if self.closed: return # close the connection ... self.closed = True In other words, you don't silence any errors, but instead use a flag to store the successful close operation and then simply ignore all future calls to the method without raising an exception. M.-A. Lemburg wrote: > Federico Di Gregorio wrote: >> On 19/10/11 09:59, M.-A. Lemburg wrote: >> [snip] >>>> risks to become a problem, with the close() that may raise an error >>>>> because the connection has been implicitly closed by a communication >>>>> error. Note that close() in itself often is a quite safe operation, >>>>> not involving database communication, so it is not expected to fail >>>>> and may not be well guarded. >>> That last sentence is not quite correct: .close() issues an implicit >>> .rollback() and usually also tells the database backend to free up >>> resources maintained for the connection, so it does require communication >>> with the backend. >> >> Here is the problem. Does .close() always issue an implicit .rollback()? > > Yes. > >> The DBAPI says yes but, as noted previously, the driver can choose to >> NOT send a rollback. > > Not if it complies to the DB-API. > >> In fact on the second .close() the driver SHOULD >> NOT send a rollback because there is no transaction in progress. > > The second .close() would in that case raise an exception > as per the definition in the database, since it has become > unusable :-) > > BTW: Regardless of whether the driver explicitly issues a rollback > or not, the database backend will roll back the connection if the > connection closes and there's a pending transaction. > >> I'd vote for idempotent .close() interpretation too. > > Please read my full reply: the implicit rollback is not > the only operation that can fail. Silencing errors is not > a good idea. > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 19 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Wed Oct 19 10:55:33 2011 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Wed, 19 Oct 2011 09:55:33 +0100 Subject: [DB-SIG] conn.close() idempotence In-Reply-To: <4E9E8F0A.7030303@egenix.com> References: <4E9E8378.10800@egenix.com> <4E9E8510.7090105@initd.org> <4E9E881C.4030108@egenix.com> <4E9E8F0A.7030303@egenix.com> Message-ID: On Wed, Oct 19, 2011 at 9:49 AM, M.-A. Lemburg wrote: > Thinking about this some more ... > > Perhaps what you're really after is the following and I was just > misunderstanding the original proposal: > > class Connection: > > ? ?closed = False > > ? ?def close(self): > ? ? ? ?if self.closed: > ? ? ? ? ? ?return > ? ? ? ?# close the connection > ? ? ? ?... > ? ? ? ?self.closed = True > > In other words, you don't silence any errors, but instead use > a flag to store the successful close operation and then simply > ignore all future calls to the method without raising an > exception. Yep, the idea is just this :) Seems a reasonable behaviour but it currently raises an exception in the DBAPI unit test, which I feel a little bit on the strict side. -- Daniele