From mal at egenix.com Fri Oct 28 09:58:36 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Fri, 28 Oct 2022 15:58:36 +0200 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) Message-ID: While discussing with Erlend about the semantics of Connection.autocommit, I was a bit surprised to find that we had obviously not added this as a DB-API 2.0 standard extension yet: https://discuss.python.org/t/introducing-a-pep-249-compliant-transaction-behaviour-to-sqlite3/16516 I'd like to change that, so here's a draft wording: """ Connection.autocommit Attribute to query and set the autocommit mode of the connection. Returns True if the connection is operating in autocommit (non- transactional) mode. Returns False is the connection is operating in manual commit (transactional) mode. Setting the attribute to True or False adjusts the connection?s mode accordingly. """ The above is what quite a few database modules already implement, so there shouldn't much to discuss. Something that's not immediately clear, though, is how to handle semantics when switching from one to the other. A. Things are clear for changing from True to False (disabling autocommit): the database leaves autocommit mode and starts a new transaction. B. Things are less clear when changing from False to True (enabling autocommit). The case for an empty transaction is straight forward: since there's nothing to commit or rollback, the database enters autocommit mode without any other effects. If there is a pending transaction, though, there are three approaches we could take: 1. The database module raises an exception, to force an explicit .commit() or .rollback() by the programmer. 2. The module automatically commits the pending transaction, since that's what autocommit is all about. 3. We leave these semantics open and up to the database module to determine. My preference would be option 2, since this makes things clear for everyone and is intuitive (well, at least for me :-)). Thoughts ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Oct 28 10:02:10 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 28 Oct 2022 10:02:10 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: Message-ID: <160dead5-3868-441c-bc3a-6683c52ad9af@app.fastmail.com> On Fri, Oct 28, 2022, at 9:58 AM, Marc-Andre Lemburg wrote: > If there is a pending transaction, though, there are three approaches > we could take: > > 1. The database module raises an exception, to force an explicit > .commit() or .rollback() by the programmer. > > 2. The module automatically commits the pending transaction, > since that's what autocommit is all about. > > 3. We leave these semantics open and up to the database module > to determine. > > My preference would be option 2, since this makes things clear for > everyone and is intuitive (well, at least for me :-)). option 3 is likely what will occur in practice. it feels a little awkward for an attribute set operation to move forward with a transactional modification to the database server. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Oct 28 10:06:52 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Fri, 28 Oct 2022 16:06:52 +0200 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <160dead5-3868-441c-bc3a-6683c52ad9af@app.fastmail.com> References: <160dead5-3868-441c-bc3a-6683c52ad9af@app.fastmail.com> Message-ID: On 28.10.2022 16:02, Mike Bayer wrote: > > > On Fri, Oct 28, 2022, at 9:58 AM, Marc-Andre Lemburg wrote: >> If there is a pending transaction, though, there are three approaches >> we could take: >> >> 1. The database module raises an exception, to force an explicit >> ??? .commit() or .rollback() by the programmer. >> >> 2. The module automatically commits the pending transaction, >> ??? since that's what autocommit is all about. >> >> 3. We leave these semantics open and up to the database module >> ??? to determine. >> >> My preference would be option 2, since this makes things clear for >> everyone and is intuitive (well, at least for me :-)). > > option 3 is likely what will occur in practice.?? it feels a little > awkward for an attribute set operation to move forward with a > transactional modification to the database server. Think of this as a property, which is how many database modules will implement it. I agree that it feels a bit awkward, especially since setting an attribute should normally not cause exceptions, but it's become a standard among modules nonetheless (I don't remember who started with it). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Oct 28 10:14:54 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 28 Oct 2022 10:14:54 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: <160dead5-3868-441c-bc3a-6683c52ad9af@app.fastmail.com> Message-ID: <995b50f8-d2f2-4163-9689-376a031333e0@app.fastmail.com> On Fri, Oct 28, 2022, at 10:06 AM, Marc-Andre Lemburg wrote: > On 28.10.2022 16:02, Mike Bayer wrote: > > > > > > On Fri, Oct 28, 2022, at 9:58 AM, Marc-Andre Lemburg wrote: > >> If there is a pending transaction, though, there are three approaches > >> we could take: > >> > >> 1. The database module raises an exception, to force an explicit > >> .commit() or .rollback() by the programmer. > >> > >> 2. The module automatically commits the pending transaction, > >> since that's what autocommit is all about. > >> > >> 3. We leave these semantics open and up to the database module > >> to determine. > >> > >> My preference would be option 2, since this makes things clear for > >> everyone and is intuitive (well, at least for me :-)). > > > > option 3 is likely what will occur in practice. it feels a little > > awkward for an attribute set operation to move forward with a > > transactional modification to the database server. > > Think of this as a property, which is how many database modules > will implement it. > > I agree that it feels a bit awkward, especially since setting > an attribute should normally not cause exceptions, but it's become > a standard among modules nonetheless (I don't remember who started > with it). well, an exception saying, "I'm not going to commit the transaction here, please call commit() or rollback() first" is of much narrower scope than one that went all the way out to the database to commit a transaction, which then failed. Or worse, some routine that sets .autocommit as a matter of course ends up causing network round trips in an unexpected place. I think the bigger issue here is that this proposes allowing network IO to occur by setting an attribute. I know this is of course *possible* but this is not considered to be a good practice, particularly in the age of asyncio (which I know, DBAPI is not asyncio, but these things still have interactions in practice). The precedent however is that DBAPIs take lots of liberties with things whether the DBAPI spec allows them to or not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Oct 28 10:20:13 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Fri, 28 Oct 2022 16:20:13 +0200 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: Message-ID: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: > If there is a pending transaction, though, there are three approaches > we could take: > > 1. The database module raises an exception, to force an explicit > ?? .commit() or .rollback() by the programmer. > > 2. The module automatically commits the pending transaction, > ?? since that's what autocommit is all about. Just checked: Option 2 is what ODBC mandates... https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function (search for SQL_ATTR_AUTOCOMMIT) > 3. We leave these semantics open and up to the database module > ?? to determine. > > My preference would be option 2, since this makes things clear for > everyone and is intuitive (well, at least for me :-)). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Oct 28 10:51:40 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 28 Oct 2022 10:51:40 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> Message-ID: <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: > On 28.10.2022 15:58, Marc-Andre Lemburg wrote: > > If there is a pending transaction, though, there are three approaches > > we could take: > > > > 1. The database module raises an exception, to force an explicit > > .commit() or .rollback() by the programmer. > > > > 2. The module automatically commits the pending transaction, > > since that's what autocommit is all about. > > Just checked: Option 2 is what ODBC mandates... > https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function > (search for SQL_ATTR_AUTOCOMMIT) That's a C API, which has different programming conventions than what Python has, and it's also referring towards a function that's being called, so while that's a datapoint to note, I dont think by itself it really indicates how a Python API should organize itself one way or the other. As I indicated before, if we were writing an asyncio DBAPI spec, doing any kind of IO on an attribute set would be a non-starter; IO under asyncio always requires an awaitable function is called. I think option 2 is probably what most DBAPIs would do anyway and they probably do now (we should maybe check what the current practice is among some of the current implementations). In SQLAlchemy, we expose "autocommit" through a method that expects that IO may occur, so we aren't affected by the decision here. We absolutely raise an error if someone is attempting to modify the "autocommit" status, as well as the isolation level, mid-transaction - this indicates a lack of clear intent on the part of the user, and if there's one thing I've learned in the past 16 years of doing Python it's that it definitely is better to raise when the user does something which does not seem to have a clear rationale - the user is likely confused and would do better to have some guardrails let them know as such, rather than guessing and doing something silently. The latter behavior is how we get angry bug reports from someone who spent hours/days/weeks trying to understand some unexpected behavior and found it based on implicit decisions that came from the user's original mis-configurations. This is absolutely not the kind of culture you see in MSFT C APIs, where a decidedly different kind of programmer will pore over every line of documents like the one you link and will invoke such methods with deep caution. In SQLAlchemy, we consider "autocommit" to be one of several "isolation level" choices, which may be seen as more of an "ACID level" setting. A block of code that raises looks like: from sqlalchemy import create_engine, text e = create_engine("postgresql://scott:tiger at localhost/test") with e.begin() as conn: conn.execute(text("insert into table (x) values ('foo')")) # raises. what legitimate reason would there be for the # user to call this in the middle of a transaction? conn.execution_options(isolation_level="AUTOCOMMIT") I think if we wanted to cover what "Pythonisms" are crossed by option 1 or 2, I can think of a bunch as follows: "In the face of ambiguity, refuse the temptation to guess." - option 1 wins "dont do IO on attribute set operations" - option 1 wins "Explicit is better than implicit." - option 1 wins This is all strictly 2 cents on my part, DBAPIs are already doing whatever they're going to do in this area and my own libraries are ready for whatever. > > > 3. We leave these semantics open and up to the database module > > to determine. > > > > My preference would be option 2, since this makes things clear for > > everyone and is intuitive (well, at least for me :-)). > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Oct 28 2022) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Oct 28 12:14:42 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Fri, 28 Oct 2022 18:14:42 +0200 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> Message-ID: <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> On 28.10.2022 16:51, Mike Bayer wrote: > > > On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: >> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: >> > If there is a pending transaction, though, there are three approaches >> > we could take: >> > >> > 1. The database module raises an exception, to force an explicit >> >? ?? .commit() or .rollback() by the programmer. >> > >> > 2. The module automatically commits the pending transaction, >> >? ?? since that's what autocommit is all about. >> >> Just checked: Option 2 is what ODBC mandates... >> https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function >> (search for SQL_ATTR_AUTOCOMMIT) > > That's a C API, which has different programming conventions than what > Python has, and it's also referring towards a function that's being > called, so while that's a datapoint to note, I dont think by itself it > really indicates how a Python API should organize itself one way or the > other. I just wanted to note that the semantics of what to do when switching from False to True are already defined in one of the major database APIs standards, so it's good practice to follow such a standard. >??? As I indicated before, if we were writing an asyncio DBAPI > spec, doing any kind of IO on an attribute set would be a non-starter; > IO under asyncio always requires an awaitable function is called. > > I think option 2 is probably what most DBAPIs would do anyway and they > probably do now (we should maybe check what the current practice is > among some of the current implementations). That would be my guess as well. I know that mxODBC used to do option 2 (simply as the result of ODBC mandating this). > [...] > > I think if we wanted to cover what "Pythonisms" are crossed by option 1 > or 2, I can think of a bunch as follows: > > "In the face of ambiguity, refuse the temptation to guess." - option 1 wins > "dont do IO on attribute set operations" - option 1 wins > "Explicit is better than implicit." - option 1 wins > > This is all strictly 2 cents on my part, DBAPIs are already doing > whatever they're going to do in this area and my own libraries are ready > for whatever. I agree that a method would be better for this, e.g. .set_autocommit(True/False) Something else we could do (*): Only define .autocommit for reading in the DB-API and add a new standard extension to adjust the setting via a method. I'd still favor the option 2 semantics for the new method. Because it's a new method, we could clearly define the behavior and avoid all guessing. Overall, I believe that important settings such as autocommit should only be set in the connection constructor, since the semantics change dramatically between autocommit and manual commit. In such a world, we'd only have an autocommit keyword argument in the Connection constructor and a read-only attribute on the object to query this after creation. Such a change could be implemented for a DB-API 3.0. For 2.0, the ship has sailed already. So how about going with the above compromise (*) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Oct 28 12:36:05 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 28 Oct 2022 12:36:05 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> Message-ID: <324bfbee-c566-44b7-a7c9-f48f25ece2f5@app.fastmail.com> On Fri, Oct 28, 2022, at 12:14 PM, Marc-Andre Lemburg wrote: > > Overall, I believe that important settings such as autocommit > should only be set in the connection constructor, since the > semantics change dramatically between autocommit and manual > commit. oh, that change would be much more intrusive. This defeats the usability of connection pools, and being able to reset session state on a connection so that it may be returned to a pool for re-use is a normal thing. Within the MS ODBC realm Ive just learned of the sp_reset_connection stored procedure, which seems to be sparsely documented, but is mentioned as essential for pooling here: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/using-integrated-authentication?view=sql-server-ver16#tracking-access-to-a-database > > In such a world, we'd only have an autocommit keyword argument > in the Connection constructor and a read-only attribute on the > object to query this after creation. > > Such a change could be implemented for a DB-API 3.0. For 2.0, > the ship has sailed already. > > So how about going with the above compromise (*) ? > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Oct 28 2022) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Oct 28 12:48:56 2022 From: mal at egenix.com (Marc-Andre Lemburg) Date: Fri, 28 Oct 2022 18:48:56 +0200 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <324bfbee-c566-44b7-a7c9-f48f25ece2f5@app.fastmail.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <324bfbee-c566-44b7-a7c9-f48f25ece2f5@app.fastmail.com> Message-ID: <76ad997d-da17-6dfd-381a-2e7fde2292c2@egenix.com> On 28.10.2022 18:36, Mike Bayer wrote: > > > On Fri, Oct 28, 2022, at 12:14 PM, Marc-Andre Lemburg wrote: >> >> Overall, I believe that important settings such as autocommit >> should only be set in the connection constructor, since the >> semantics change dramatically between autocommit and manual >> commit. > > oh, that change would be much more intrusive.?? This defeats the > usability of connection pools, and being able to reset session state on > a connection so that it may be returned to a pool for re-use is a normal > thing.? Within the MS ODBC realm Ive just learned of the > sp_reset_connection stored procedure, which seems to be sparsely > documented, but is mentioned as essential for pooling here: > https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/using-integrated-authentication?view=sql-server-ver16#tracking-access-to-a-database In my application designs, I typically have two or three pools: one for autoconnect connections, one for manual commit ones and sometimes one for read-only connections. In many respects, those types of connections are too different to safely keep them in the same pool. The optimization settings also often deep, so it would be too costly to reset them for placing them into a generic pool. But those are my 2 cents. >> In such a world, we'd only have an autocommit keyword argument >> in the Connection constructor and a read-only attribute on the >> object to query this after creation. >> >> Such a change could be implemented for a DB-API 3.0. For 2.0, >> the ship has sailed already. >> >> So how about going with the above compromise (*) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mike_mp at zzzcomputing.com Fri Oct 28 13:09:46 2022 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Fri, 28 Oct 2022 13:09:46 -0400 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <76ad997d-da17-6dfd-381a-2e7fde2292c2@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> <324bfbee-c566-44b7-a7c9-f48f25ece2f5@app.fastmail.com> <76ad997d-da17-6dfd-381a-2e7fde2292c2@egenix.com> Message-ID: <7af1f80d-dd15-46e3-8519-d24a6d014e23@app.fastmail.com> On Fri, Oct 28, 2022, at 12:48 PM, Marc-Andre Lemburg wrote: > On 28.10.2022 18:36, Mike Bayer wrote: > > > > > > On Fri, Oct 28, 2022, at 12:14 PM, Marc-Andre Lemburg wrote: > >> > >> Overall, I believe that important settings such as autocommit > >> should only be set in the connection constructor, since the > >> semantics change dramatically between autocommit and manual > >> commit. > > > > oh, that change would be much more intrusive. This defeats the > > usability of connection pools, and being able to reset session state on > > a connection so that it may be returned to a pool for re-use is a normal > > thing. Within the MS ODBC realm Ive just learned of the > > sp_reset_connection stored procedure, which seems to be sparsely > > documented, but is mentioned as essential for pooling here: > > https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/using-integrated-authentication?view=sql-server-ver16#tracking-access-to-a-database > > In my application designs, I typically have two or three pools: one for > autoconnect connections, one for manual commit ones and sometimes > one for read-only connections. > > In many respects, those types of connections are too different > to safely keep them in the same pool. The optimization settings > also often deep, so it would be too costly to reset them for > placing them into a generic pool. > > But those are my 2 cents. Agree I would do it the same way and we document it here: https://docs.sqlalchemy.org/en/14/orm/session_transaction.html#setting-isolation-for-a-sessionmaker-engine-wide however, being able to set autocommit mid-connectivity I think is still important for use cases such as some database migrations, where the connection needs to be set into autocommit to perform some activities that can't run transactionally, but then to allow other operations to proceed within transactions. It's true this can use two connections as well but IMO I dont think the DBAPI should be getting *that* opinionated. again it doesnt matter since DBAPIs already have .autocommit attributes on them :) > > >> In such a world, we'd only have an autocommit keyword argument > >> in the Connection constructor and a read-only attribute on the > >> object to query this after creation. > >> > >> Such a change could be implemented for a DB-API 3.0. For 2.0, > >> the ship has sailed already. > >> > >> So how about going with the above compromise (*) ? > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Oct 28 2022) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.varrazzo at gmail.com Sun Oct 30 12:28:52 2022 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Sun, 30 Oct 2022 17:28:52 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: References: Message-ID: On Fri, 28 Oct 2022 at 15:58, Marc-Andre Lemburg wrote: My few pence: > If there is a pending transaction, though, there are three approaches > we could take: > > 1. The database module raises an exception, to force an explicit > .commit() or .rollback() by the programmer. > > 2. The module automatically commits the pending transaction, > since that's what autocommit is all about. > > 3. We leave these semantics open and up to the database module > to determine. > > My preference would be option 2, since this makes things clear for > everyone and is intuitive (well, at least for me :-)). 2 is a surprising side effect and loaded foot-gun. Either the user has an intent, in which case 1 is an useful indication that there's a ProgrammingError there, or there is a bug, let's say in a multithreaded program, in which case 1 is an useful indication that something is broken. I believe psycopg2 has implemented 2 for the first few years. Sometimes, maybe 2011, maybe before, the behaviour switched to 1. There was some drama around the fact that some programs previously broken started erroring out because they had transactions open in moments in which nobody thought there were, but quickly it was acknowledged that fixing those programs was a good idea. Nobody has complained since, or argued that an automatic commit of the current transaction on setting autocommit is a good idea, or a desired, or intuitive, behaviour. On a related tangent (if you like a good oxymoron), should the DBAPI ever suggest an async interface, an `autocommit` setter performing I/O (in order to commit) would be impossible to implement, as Python doesn't offer async attribute access. In Psycopg 3 it turned out that, even without I/O access, an autocommit setter was impossible to implement anyway, because the lock that the connection uses to guard its state from concurrent access should be likely async too. For this reason, the async version of psycopg 3 connection has an `async def set_autocommit()` method instead (https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.AsyncConnection). Pragmatically, I doubt that the dbapi can do anything different than 3, and maybe should warn the user that setting autocommit with a transaction open has an undetermined behaviour. Cheers -- Daniele From erlenda at gmail.com Sun Oct 30 17:38:53 2022 From: erlenda at gmail.com (Erlend Egeberg Aasland) Date: Sun, 30 Oct 2022 22:38:53 +0100 Subject: [DB-SIG] Adding Connection.autocommit as standard extension to DB-API 2.0 (PEP 249) In-Reply-To: <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> References: <8ad828b9-5877-b7d7-6e86-6772aada510d@egenix.com> <9756fb96-8a64-4417-bbf8-56e946d5c0bd@app.fastmail.com> <751f84e5-28f1-d5f1-41c1-fdab10005ccc@egenix.com> Message-ID: > On 28 Oct 2022, at 18:14, Marc-Andre Lemburg wrote: > > On 28.10.2022 16:51, Mike Bayer wrote: >> On Fri, Oct 28, 2022, at 10:20 AM, Marc-Andre Lemburg wrote: >>> On 28.10.2022 15:58, Marc-Andre Lemburg wrote: >>> > If there is a pending transaction, though, there are three approaches >>> > we could take: >>> > >>> > 1. The database module raises an exception, to force an explicit >>> > .commit() or .rollback() by the programmer. >>> > >>> > 2. The module automatically commits the pending transaction, >>> > since that's what autocommit is all about. >>> >>> Just checked: Option 2 is what ODBC mandates... >>> https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlsetconnectattr-function >>> (search for SQL_ATTR_AUTOCOMMIT) >> That's a C API, which has different programming conventions than what Python has, and it's also referring towards a function that's being called, so while that's a datapoint to note, I dont think by itself it really indicates how a Python API should organize itself one way or the other. > > I just wanted to note that the semantics of what to do when switching > from False to True are already defined in one of the major database > APIs standards, so it's good practice to follow such a standard. +1 My preference would be 2. or 3. > Overall, I believe that important settings such as autocommit > should only be set in the connection constructor, since the > semantics change dramatically between autocommit and manual > commit. > > In such a world, we'd only have an autocommit keyword argument > in the Connection constructor and a read-only attribute on the > object to query this after creation. I would be ok with that. Erlend