From info at egenix.com Tue Dec 2 10:17:54 2014 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Tue, 02 Dec 2014 10:17:54 +0100 Subject: [DB-SIG] ANN: eGenix mxODBC Connect 2.1.2 - Python Database Interface Message-ID: <547D83C2.10103@egenix.com> ________________________________________________________________________ ANNOUNCING eGenix.com mxODBC Connect Python Database Interface Version 2.1.2 mxODBC Connect is our commercially supported client-server product for connecting Python applications to relational databases in a truly platform independent way. This announcement is also available on our web-site for online reading: http://www.egenix.com/company/news/eGenix-mxODBC-Connect-2.1.2-GA.html ________________________________________________________________________ INTRODUCTION The mxODBC Connect Database Interface for Python allows users to easily connect Python applications to all major databases on the market today in a highly portable, convenient and secure way. Python Database Connectivity the Easy Way ----------------------------------------- Unlike our mxODBC Python extension, mxODBC Connect is designed as client-server application, so you no longer need to find production quality ODBC drivers for all the platforms you target with your Python application. Instead you use an easy to install royalty-free Python client library which connects directly to the mxODBC Connect database server over the network. This makes mxODBC Connect a great basis for writing cross-platform multi-tier database applications and utilities in Python, especially if you run applications that need to communicate with databases such as MS SQL Server and MS Access, Oracle Database, IBM DB2 and Informix, Sybase ASE and Sybase Anywhere, MySQL, PostgreSQL, SAP MaxDB and many more, that run on Windows or Linux machines. Ideal for Database Driven Client Applications --------------------------------------------- By removing the need to install and configure ODBC drivers on the client side and dealing with complicated network setups for each set of drivers, mxODBC Connect greatly simplifies deployment of database driven client applications, while at the same time making the network communication between client and database server more efficient and more secure. For more information, please have a look at the mxODBC Connect product page, in particular, the full list of available features. For more information, please see the product page: http://www.egenix.com/products/python/mxODBCConnect/ ________________________________________________________________________ NEWS mxODBC Connect 2.1.2 is a patch level release of our successful mxODBC Connect product. In the last patch level release 2.1.1, we had put a lot of emphasis on enhancing the TLS/SSL setup of the mxODBC Connect product: https://cms.egenix.com/company/news/eGenix-mxODBC-Connect-2.1.1-GA.html In this release, we are fixing a pip installation problem, that occurred with the mxODBC Connect Client on a few platforms, as well as a some other minor issues we found: Security Enhancements --------------------- * OpenSSL cipher string list updated to explicitly disallow use of low security or export ciphers. mxODBC Connect Enhancements --------------------------- * Fixed a problem that could cause the mxODBC Connect Client to not install correctly with pip. * Successfully tested against Python 2.7.9, which will come with a new ssl module. * Fixed the package version number to show the correct release version. * Fixed OpenSSL warnings in the Unix installer and scripts. For the full set of changes, including those of the 2.1 series of mxODBC Connect, please check the mxODBC Connect change log: http://www.egenix.com/products/python/mxODBCConnect/changelog.html ________________________________________________________________________ UPGRADING You are encouraged to upgrade to this latest mxODBC Connect release. When upgrading, please always upgrade both the server and the client installations to the same version - even for patch level releases. We will give out 20% discount coupons for upgrade purchases going from mxODBC Connect Server 1.x to 2.1 and 50% coupons for upgrades from mxODBC 2.x to 2.1. Please contact the eGenix.com Sales Team (sales at egenix.com) with your existing license serials for details. Users of our stand-alone mxODBC product will have to purchase new licenses from our online shop in order to use mxODBC Connect. You can request free 30-day evaluation licenses by visiting our web-site or writing to sales at egenix.com, stating your name (or the name of the company) and the number of eval licenses that you need. http://www.egenix.com/products/python/mxODBCConnect/#Evaluation ________________________________________________________________________ DOWNLOADS The download archives as well as instructions for installation and configuration of the product can be found on the product page: http://www.egenix.com/products/python/mxODBCConnect/ If you want to try the package, jump straight to the download instructions: https://cms.egenix.com/products/python/mxODBCConnect/#Download Fully functional evaluation licenses for the mxODBC Connect Server are available free of charge: http://www.egenix.com/products/python/mxODBCConnect/#Evaluation mxODBC Connect Client is always free of charge. _______________________________________________________________________ SUPPORT Commercial support for this product is available from eGenix.com. Please see http://www.egenix.com/services/support/ for details about our support offerings. _______________________________________________________________________ INFORMATION About Python (http://www.python.org/): Python is an object-oriented Open Source programming language which runs on all modern platforms. By integrating ease-of-use, clarity in coding, enterprise application connectivity and rapid application design, Python establishes an ideal programming platform for today's IT challenges. About eGenix (http://www.egenix.com/): eGenix is a software project, consulting and product company focusing on expert project services and professional quality products for companies, Python users and developers. Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Plone/Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From phd at phdru.name Mon Dec 8 22:34:43 2014 From: phd at phdru.name (Oleg Broytman) Date: Mon, 8 Dec 2014 22:34:43 +0100 Subject: [DB-SIG] SQLObject 1.7.0 Message-ID: <20141208213443.GA12991@phdru.name> Hello! I'm pleased to announce version 1.7.0, the first stable release of branch 1.7 of SQLObject. What's new in SQLObject ======================= * Python 2.5 is no longer supported. The minimal supported version is Python 2.6. * DateTimeCol and TimeCol can read values with microseconds (created by SQLObject 2.0) but do not write microseconds back. * Upgrade ez_setup to 2.2. * Adapt duplicate error message strings for SQLite 3.8. * Allow unicode in .orderBy(u'-column'). * Fix a minor bug in MSSQLConnection: do not override callable server_version with a non-callable. Contributors for this release are Geoffrey Wossum, Neil Muller and Andrew Trusty. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/1.7.0 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From elfring at users.sourceforge.net Sun Dec 7 22:06:36 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Sun, 07 Dec 2014 22:06:36 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements Message-ID: <5484C15C.8010003@users.sourceforge.net> Hello, An interface for parameterised SQL statements (working with placeholders) is provided by the execute() method from the Cursor class at the moment. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute I assume that the "SQL Statement Object" from the SQLite C interface is reused there already. http://sqlite.org/c3ref/stmt.html I imagine that it will be more efficient occasionally to offer also a base class like "prepared_statement" so that the parameter specification does not need to be parsed for every passed command. I suggest to improve corresponding preparation and compilation possibilities. https://bugs.python.org/issue22956 Regards, Markus From mal at egenix.com Tue Dec 9 10:40:44 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 09 Dec 2014 10:40:44 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5484C15C.8010003@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> Message-ID: <5486C39C.2040608@egenix.com> On 07.12.2014 22:06, SF Markus Elfring wrote: > Hello, > > An interface for parameterised SQL statements (working with > placeholders) is provided by the execute() method from the Cursor class > at the moment. > https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute > > I assume that the "SQL Statement Object" from the SQLite C interface is > reused there already. > http://sqlite.org/c3ref/stmt.html > > I imagine that it will be more efficient occasionally to offer also a > base class like "prepared_statement" so that the parameter specification > does not need to be parsed for every passed command. > I suggest to improve corresponding preparation and compilation > possibilities. > https://bugs.python.org/issue22956 Please see our previous discussion on this subject: https://mail.python.org/pipermail/db-sig/2014-March/006090.html This was the first draft result of that discussion: https://mail.python.org/pipermail/db-sig/2014-March/006096.html It's not yet been added to the DB-API as standard extension. Perhaps we ought to continue that discussion. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 09 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Plone/Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-02: Released mxODBC Connect 2.1.2 ... http://egenix.com/go66 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From daniele.varrazzo at gmail.com Tue Dec 9 13:24:01 2014 From: daniele.varrazzo at gmail.com (Daniele Varrazzo) Date: Tue, 9 Dec 2014 12:24:01 +0000 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5486C39C.2040608@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> Message-ID: On Tue, Dec 9, 2014 at 9:40 AM, M.-A. Lemburg wrote: > On 07.12.2014 22:06, SF Markus Elfring wrote: >> Hello, >> >> An interface for parameterised SQL statements (working with >> placeholders) is provided by the execute() method from the Cursor class >> at the moment. >> https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute >> >> I assume that the "SQL Statement Object" from the SQLite C interface is >> reused there already. >> http://sqlite.org/c3ref/stmt.html >> >> I imagine that it will be more efficient occasionally to offer also a >> base class like "prepared_statement" so that the parameter specification >> does not need to be parsed for every passed command. >> I suggest to improve corresponding preparation and compilation >> possibilities. >> https://bugs.python.org/issue22956 > > Please see our previous discussion on this subject: > > https://mail.python.org/pipermail/db-sig/2014-March/006090.html > > This was the first draft result of that discussion: > > https://mail.python.org/pipermail/db-sig/2014-March/006096.html > > It's not yet been added to the DB-API as standard extension. Perhaps > we ought to continue that discussion. Sorry for not keeping up with the discussion I'd started myself. I have a few observations about your draft: The prepared operation string is retained by the cursor to allow executing the operation without having to prepare the operation again. should the prepared statement be accessible by an attribute? Many the drivers would make so, so wouldn't be better define what should it be? ``.prepared``? (a string or None if none is prepared). It could be one of the "optional" ones in case other drivers wouldn't be able to retrieve it back. In order to benefit from this caching, the .execute*() methods must be run with the same operation string that was passed to the .prepare() method. Not much of a problem with this interface. However sometimes people sticks a long string literal as execute method, e.g.:: cur.execute("""LONG SQL""", args) for record in cur: ... with this interface the sql should be stored in a variable and passed to both prepare and execute: SQL = """LONG SQL""" cur.prepare(SQL) cur.execute(SQL, args) in case of prepared statements, wouldn't it be handy to be able to pass None as the first execute parameter, which would result in the execution of the prepared statement or a ProgrammingError if no statement has been prepared? cur.prepare("""LONG SQL""") cur.execute(None, args) for record in cur: ... not a performance saving of course: just for sake of ergonomic. Of course having cur.prepared one could always refer to that with ``cur.execute(cur.prepared, args)`` The prepared operation is only available until the next call to one of the .execute*() methods or another call to the .prepare() method. This is a bit unclear to me: I believe you mean "until the next call to one of the .execute() methods with a different query", right? (otherwise it seems you can execute a prepared statement only once...). But does it mean that the well defined behaviour of the driver is to actively clear the prepared statement upon an .execute() with a different argument, or just that it is not defined whether the previous query is still prepared? ---- One issue I would like to have clarified (or clarified that it cannot be clarified...) is: who owns the prepared statement? My data points, which I don't want to overgeneralise, are: - in Postgres, the prepared statements belong to the connection, hence any of its cursors may use a statement prepared only once. - I work on a large software system implemented in Erlang and based on Postgres. The Erlang driver we use doesn't have a connection/cursor distinction and manages a connection pool. We automatically prepare almost every query we send to the backend so further calls (which in a Python transposition would be executed by a different cursor but likely by the same connection) benefit from preparation. So, one of the options I would like to provide to psycopg users is to leverage the possibility to share prepared statement across cursors: this may or may not imply automatic prepare upon execute, it would likely imply that preparing the same statements multiple time in the same connection would be no-op etc. What I am wondering is: are Postgres features too much specific to drive the API design? E.g. are there around db/drivers pairs for which the prepared statement would belong to the cursor? Or is the ownership of the prepared statements to the connection so common that the API design may benefit of it, e.g. making prepared statements shared across cursors, allowing more than one prepared statement per connection etc.? ---- If the prepare() extension to the API is released as proposed above by M-A, my psycopg implementation would be something along this line: - implement the cursor.prepare() method, a cursor.prepared attribute; - implement the behaviour as defined by the DBAPI; - the prepared statements would actually be registered inside the connection, so a cursor.execute*() call would automatically use a statement if found prepared by any of the connection's cursors (but preparation must be still explicit); - call to execute with unprepared queries wouldn't affect the prepared ones; - as an option (e.g. by using a certain Connection subclass declared a DBAPI extension) every executed statement would be automatically prepared (there may be a cursor.execute(..., prepared=False) maybe to blacklist specific queries). Would this implementation be compliant enough or is it a too stretched interpretation of the API? Please understand that these points are not raised to drive the API where my own project wants: as my knowledge of other DBMSs is limited I'm trying to understand if some of the features IMO valuable I've found in other software systems can be implemented generally by other DBMS/driver pairs, or if I'd still be free to implement such features for the Postgres user benefit while maintaining the DBAPI spirit and guidelines. Thank you very much for your insights. -- Daniele From phd at phdru.name Sun Dec 14 15:52:49 2014 From: phd at phdru.name (Oleg Broytman) Date: Sun, 14 Dec 2014 15:52:49 +0100 Subject: [DB-SIG] SQLObject 1.7.2 Message-ID: <20141214145249.GA12727@phdru.name> Hello! I'm pleased to announce version 1.7.2, a bugfix release of branch 1.7 of SQLObject. What's new in SQLObject ======================= * Fix a bug: zero-pad microseconds on the right, not on the left; 0.0835 seconds means 83500 microseconds. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/1.7.2 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From mal at egenix.com Wed Dec 17 14:31:02 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 17 Dec 2014 14:31:02 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> Message-ID: <54918596.1000807@egenix.com> On 09.12.2014 13:24, Daniele Varrazzo wrote: > On Tue, Dec 9, 2014 at 9:40 AM, M.-A. Lemburg wrote: >> On 07.12.2014 22:06, SF Markus Elfring wrote: >>> Hello, >>> >>> An interface for parameterised SQL statements (working with >>> placeholders) is provided by the execute() method from the Cursor class >>> at the moment. >>> https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute >>> >>> I assume that the "SQL Statement Object" from the SQLite C interface is >>> reused there already. >>> http://sqlite.org/c3ref/stmt.html >>> >>> I imagine that it will be more efficient occasionally to offer also a >>> base class like "prepared_statement" so that the parameter specification >>> does not need to be parsed for every passed command. >>> I suggest to improve corresponding preparation and compilation >>> possibilities. >>> https://bugs.python.org/issue22956 >> >> Please see our previous discussion on this subject: >> >> https://mail.python.org/pipermail/db-sig/2014-March/006090.html >> >> This was the first draft result of that discussion: >> >> https://mail.python.org/pipermail/db-sig/2014-March/006096.html >> >> It's not yet been added to the DB-API as standard extension. Perhaps >> we ought to continue that discussion. Just for better understanding, here's the proposal copied inline: """ .prepare(operation) Prepare a database operation (query or command) without executing it, e.g. to check for syntax errors, determine the parameter count or initialize the cursor for subsequent calls to the .execute*() methods. The prepared operation string is retained by the cursor to allow executing the operation without having to prepare the operation again. In order to benefit from this caching, the .execute*() methods must be run with the same operation string that was passed to the .prepare() method. The call to .prepare() closes any pending result sets on the cursor. The prepared operation is only available until the next call to one of the .execute*() methods or another call to the .prepare() method. Return values are not defined. """ > Sorry for not keeping up with the discussion I'd started myself. I > have a few observations about your draft: > > The prepared operation string is retained by the cursor to allow > executing the operation without having to prepare the operation again. > > should the prepared statement be accessible by an attribute? Many the > drivers would make so, so wouldn't be better define what should it be? > ``.prepared``? (a string or None if none is prepared). It could be one > of the "optional" ones in case other drivers wouldn't be able to > retrieve it back. mxODBC has for a long time used cursor.command to expose the last executed or prepared statement, mostly to make it easy to rerun an .execute*() method with the same command string in order to have the driver reuse the cached access plan for the already prepared statement. Here's an example: cursor.prepare('select * from mytable') cursor.execute(cursor.command) We could make this or your .prepared attribute an optional extension. > In order to benefit from this caching, the .execute*() methods must > be run with the same operation string that was passed to the .prepare() > method. > > Not much of a problem with this interface. However sometimes people > sticks a long string literal as execute method, e.g.:: > > cur.execute("""LONG SQL""", args) > for record in cur: ... > > with this interface the sql should be stored in a variable and passed > to both prepare and execute: > > SQL = """LONG SQL""" > cur.prepare(SQL) > cur.execute(SQL, args) > > in case of prepared statements, wouldn't it be handy to be able to > pass None as the first execute parameter, which would result in the > execution of the prepared statement or a ProgrammingError if no > statement has been prepared? I'm not sure I like this kind of implicit definition of the SQL command. With cursor.command or cursor.prepared it's clear what will get executed. Also note that passing in None could mask programming errors, e.g. in case you pass in cursor.command, but no prepared command is available (the attribute would then return None). You'd still get an error, but not the obvious one of having None as SQL command parameter being invalid. > cur.prepare("""LONG SQL""") > cur.execute(None, args) > for record in cur: ... > > not a performance saving of course: just for sake of ergonomic. Of > course having cur.prepared one could always refer to that with > ``cur.execute(cur.prepared, args)`` > > The prepared operation is only available until the next call to > one of the .execute*() methods or another call to the .prepare() > method. > > This is a bit unclear to me: I believe you mean "until the next call > to one of the .execute() methods with a different query", right? Yes. Good point. How about: """ The prepared operation is only available until the next call to one of the .execute*() methods or another call to the .prepare() method using a different command string than the one used for preparing the previous operation. """ > (otherwise it seems you can execute a prepared statement only > once...). But does it mean that the well defined behaviour of the > driver is to actively clear the prepared statement upon an .execute() > with a different argument, or just that it is not defined whether the > previous query is still prepared? No, I only wanted to say that the driver will forgot the prepared statement in case a new statement is used. > ---- > > One issue I would like to have clarified (or clarified that it cannot > be clarified...) is: who owns the prepared statement? My data points, > which I don't want to overgeneralise, are: > > - in Postgres, the prepared statements belong to the connection, hence > any of its cursors may use a statement prepared only once. > - I work on a large software system implemented in Erlang and based on > Postgres. The Erlang driver we use doesn't have a connection/cursor > distinction and manages a connection pool. We automatically prepare > almost every query we send to the backend so further calls (which in a > Python transposition would be executed by a different cursor but > likely by the same connection) benefit from preparation. > > So, one of the options I would like to provide to psycopg users is to > leverage the possibility to share prepared statement across cursors: > this may or may not imply automatic prepare upon execute, it would > likely imply that preparing the same statements multiple time in the > same connection would be no-op etc. > > What I am wondering is: are Postgres features too much specific to > drive the API design? E.g. are there around db/drivers pairs for which > the prepared statement would belong to the cursor? Or is the ownership > of the prepared statements to the connection so common that the API > design may benefit of it, e.g. making prepared statements shared > across cursors, allowing more than one prepared statement per > connection etc.? I think this is too specific to a particular driver/backend. E.g. in ODBC the prepared statement is owned by the cursor and there may only be one such statement per cursor at any time. It is possible to open a few cursors, prepare frequently used statements on them and them make them accessible via a cursor pool. This works well (I know since I've implemented such a logic some 14-15 years ago in a large application). > ---- > > If the prepare() extension to the API is released as proposed above by > M-A, my psycopg implementation would be something along this line: > > - implement the cursor.prepare() method, a cursor.prepared attribute; > - implement the behaviour as defined by the DBAPI; > - the prepared statements would actually be registered inside the > connection, so a cursor.execute*() call would automatically use a > statement if found prepared by any of the connection's cursors (but > preparation must be still explicit); > - call to execute with unprepared queries wouldn't affect the prepared ones; > - as an option (e.g. by using a certain Connection subclass declared a > DBAPI extension) every executed statement would be automatically > prepared (there may be a cursor.execute(..., prepared=False) maybe to > blacklist specific queries). > > Would this implementation be compliant enough or is it a too stretched > interpretation of the API? This would work out, I guess. You just need to be careful with having too many prepared cursors in the pool: * the cursors require memory on the client side and often also on the server side * cursors can keep open result sets and both block connections and significant resources on both client and server side To address part of the latter problem, we added a cursor.flush() method in mxODBC. > Please understand that these points are not raised to drive the API > where my own project wants: as my knowledge of other DBMSs is limited > I'm trying to understand if some of the features IMO valuable I've > found in other software systems can be implemented generally by other > DBMS/driver pairs, or if I'd still be free to implement such features > for the Postgres user benefit while maintaining the DBAPI spirit and > guidelines. Sure, that's the purpose of the DB-API - it is extensible by design. The DB-API standard extensions were added to address proliferation of such additions. If possible, the database modules should try to standardize on these extensions, which is why discussions such as these are useful. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 17 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Dec 17 15:21:43 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 17 Dec 2014 15:21:43 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54918A37.2040001@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> Message-ID: <54919177.70205@egenix.com> On 17.12.2014 14:50, SF Markus Elfring wrote: >> How about: >> >> """ >> The prepared operation is only available until the next call to >> one of the .execute*() methods or another call to the .prepare() >> method using a different command string than the one used for >> preparing the previous operation. >> """ > > Can the prepare() method return a handle for a SQL statement? The proposal says: """ Return values are not defined. """ which we traditionally use on the DB-API to mean: the method may return something, but whatever it returns is database module specific and not something to rely on on general. > Will prepared SQL statements be mapped to a specific class hierarchy > in Python? Not sure what you mean. Having prepared a statement only means that the cursor has run it by the database to build an access plan and, as a side-effect, has most likely run the parser on it. In practice, a database module may implement this method as mostly no-op operation. Some databases don't support prepared statements or don't work well in this mode, other may not run the parser and thus not necessarily raise errors during the call (only in the subsequent .execute()). The most likely case, however, is that you can use .prepare() to check for errors (and hopefully, without causing the transaction to be canceled, in case there are errors). >> It is possible to open a few cursors, prepare frequently >> used statements on them and them make them accessible via >> a cursor pool. This works well (I know since I've implemented >> such a logic some 14-15 years ago in a large application). > > How should the application programming interface look like > in Python for this use case? Depends on the application and what you want to achieve. There's no general answer to this. >> The DB-API standard extensions were added to address proliferation >> of such additions. If possible, the database modules should try >> to standardize on these extensions, which is why discussions such >> as these are useful. > > Are there more software improvements in a waiting queue? There are a few extensions that we've been discussing off and on. The cursor.prepare() method is one of the more recent ones. One of these days, I'll have to go through the archives and collect a list :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 17 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Dec 17 16:03:09 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 17 Dec 2014 16:03:09 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5491956D.9000503@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> Message-ID: <54919B2D.7010306@egenix.com> On 17.12.2014 15:38, SF Markus Elfring wrote: >>> Can the prepare() method return a handle for a SQL statement? >> >> The proposal says: >> >> """ >> Return values are not defined. >> """ > > How do you think about to change the application programming > interface here? > > Would a class like "prepared_statement" be more useful at this place? The DB-API tries to make higher level abstractions possible, but doesn't mandate any of them in order to make writing database modules easier and to get more database backends supported. I think you're looking for SQLAlchemy or SQLObject :-) >>> Will prepared SQL statements be mapped to a specific class hierarchy >>> in Python? >> >> Not sure what you mean. > > The structured query language supports a few standard statements like > insert, update and delete. I imagine that specific classes will be > useful for their parameterisation, won't they? Please see above. >>>> It is possible to open a few cursors, prepare frequently >>>> used statements on them and them make them accessible via >>>> a cursor pool. This works well (I know since I've implemented >>>> such a logic some 14-15 years ago in a large application). >>> >>> How should the application programming interface look like >>> in Python for this use case? >> >> Depends on the application and what you want to achieve. >> There's no general answer to this. > > I find that view strange. > > I would appreciate if the corresponding software abstractions > can be improved in reasonable ways. Please explain what you want in more detail, so we can check whether there's room for improvement :-) Note that the DB-API doesn't require implementing connection or cursor pools. It only provides ways of making these possible at higher abstraction levels. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 17 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Dec 17 16:41:12 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 17 Dec 2014 16:41:12 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54919EC2.7040007@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> Message-ID: <5491A418.4030300@egenix.com> On 17.12.2014 16:18, SF Markus Elfring wrote: >> Please explain what you want in more detail, so we can check >> whether there's room for improvement :-) > > Can the SQL update statement be encapsulated by a dedicated Python > class for example? Sure. You could write a class that only supports update statements and has methods for running/customizing them. This could then use the DB-API to talk to the database, abstracting away the details you don't want to deal with in the application. Another possibility is to write a layer on top of the DB-API to abstract the underlying queries away from the application and only have the layer provide dedicated methods for the things the application needs, such as query method for specific details or inserting application objects into the database. This is the approach we usually take in our projects, since it provides better separation of the application logic from the database logic than using ORMs usually provides. It's also possible to use such an abstraction layer on top of an ORM, if you want to the ORM to deal with abstracting away database backend details. >> Note that the DB-API doesn't require implementing connection >> or cursor pools. It only provides ways of making these possible >> at higher abstraction levels. > > I am looking for higher service levels without following the software > design directions from object-relational managers like SQLAlchemy > and SQLObject. There were a few attempts in that direction: https://wiki.python.org/moin/HigherLevelDatabaseProgramming but most of them are no longer available. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 17 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Wed Dec 17 18:54:01 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Wed, 17 Dec 2014 12:54:01 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5491A418.4030300@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: > On Dec 17, 2014, at 10:41 AM, M.-A. Lemburg wrote: > > > Another possibility is to write a layer on top of the DB-API > to abstract the underlying queries away from the application > and only have the layer provide dedicated methods for the > things the application needs, such as query method for specific > details or inserting application objects into the database. > > This is the approach we usually take in our projects, since > it provides better separation of the application logic from the > database logic than using ORMs usually provides. It's also possible > to use such an abstraction layer on top of an ORM, if you want > to the ORM to deal with abstracting away database backend details. When I read this quickly, it seems to make sense, but when I really try to imagine what this means, I come out at the same place every time: if you?ve written a ?method for inserting application objects into the database?, you?ve written an ORM. Databases don?t store ?application objects?, they store rows. So there has to be ?ORM? in there. >> >> I am looking for higher service levels without following the software >> design directions from object-relational managers like SQLAlchemy >> and SQLObject. replying to Markus - SQLAlchemy?s ORM is only an optional component of the SQLAlchemy database toolkit overall. If you would like a very mature and well proven SQL abstraction layer that does not provide any design directions whatsoever (not to mention much better performance than the ORM), please consider SQLAlchemy Core: http://docs.sqlalchemy.org/en/rel_0_9/core/index.html. From mal at egenix.com Wed Dec 17 22:02:42 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 17 Dec 2014 22:02:42 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: <5491EF72.1090201@egenix.com> On 17.12.2014 18:54, Michael Bayer wrote: > >> On Dec 17, 2014, at 10:41 AM, M.-A. Lemburg wrote: >> >> >> Another possibility is to write a layer on top of the DB-API >> to abstract the underlying queries away from the application >> and only have the layer provide dedicated methods for the >> things the application needs, such as query method for specific >> details or inserting application objects into the database. >> >> This is the approach we usually take in our projects, since >> it provides better separation of the application logic from the >> database logic than using ORMs usually provides. It's also possible >> to use such an abstraction layer on top of an ORM, if you want >> to the ORM to deal with abstracting away database backend details. > > When I read this quickly, it seems to make sense, but when I really try to imagine what this means, I come out at the same place every time: if you?ve written a ?method for inserting application objects into the database?, you?ve written an ORM. Databases don?t store ?application objects?, they store rows. So there has to be ?ORM? in there. You're right: it depends on how you read "ORM" :-) If you mean: mapping Python objects to relational database, then yes, we always do that, since even an integer is a Python object. What I usually refer to as "ORM", and meant in the above paragraph, is the approach of using methods and (special) attributes on objects to build and run SQL queries behind the scenes, with the main intent of abstracting away the SQL. >>> I am looking for higher service levels without following the software >>> design directions from object-relational managers like SQLAlchemy >>> and SQLObject. > > replying to Markus - > > SQLAlchemy?s ORM is only an optional component of the SQLAlchemy database toolkit overall. If you would like a very mature and well proven SQL abstraction layer that does not provide any design directions whatsoever (not to mention much better performance than the ORM), please consider SQLAlchemy Core: http://docs.sqlalchemy.org/en/rel_0_9/core/index.html. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 17 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Dec 17 22:25:20 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 17 Dec 2014 22:25:20 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: <5491F4C0.2090905@egenix.com> On 17.12.2014 20:37, Tony Locke wrote: > On the question of extending the DB-API to accommodate prepared > statements, from my narrow point of view as a maintainer of pg8000 > it's unnecessary, as pg8000 uses prepared statements for everything > anyway. In later versions of Postgres, the query plan is potentially > updated every time a prepared statement is executed, as opposed to the > old days when the query plan for a prepared statement was fixed. > > So (again from my limited point of view) it seems there isn't any > reason ever to use anything other than a prepared statement. :-) Just clarify: some databases don't need the prepare step on the client side, since they do (mostly) everything on the server side. Doing a prepare step will either not change anything, or worse, add an unnecessary network roundtrip. With others you always have to run a prepare step, since the clients needs the parsed data to convert/handle parameters. For this reason, ODBC provides both modes of operation. The optional .prepare() method we're discussing here is only be useful for the latter class of databases, since the DB-API currently doesn't provide a way to say "prepare this SQL for execution, but don't execute it just yet". > Regards, > > Tony. > > On 17 December 2014 at 17:54, Michael Bayer wrote: >> >>> On Dec 17, 2014, at 10:41 AM, M.-A. Lemburg wrote: >>> >>> >>> Another possibility is to write a layer on top of the DB-API >>> to abstract the underlying queries away from the application >>> and only have the layer provide dedicated methods for the >>> things the application needs, such as query method for specific >>> details or inserting application objects into the database. >>> >>> This is the approach we usually take in our projects, since >>> it provides better separation of the application logic from the >>> database logic than using ORMs usually provides. It's also possible >>> to use such an abstraction layer on top of an ORM, if you want >>> to the ORM to deal with abstracting away database backend details. >> >> When I read this quickly, it seems to make sense, but when I really try to imagine what this means, I come out at the same place every time: if you?ve written a ?method for inserting application objects into the database?, you?ve written an ORM. Databases don?t store ?application objects?, they store rows. So there has to be ?ORM? in there. >> >>>> >>>> I am looking for higher service levels without following the software >>>> design directions from object-relational managers like SQLAlchemy >>>> and SQLObject. >> >> replying to Markus - >> >> SQLAlchemy?s ORM is only an optional component of the SQLAlchemy database toolkit overall. If you would like a very mature and well proven SQL abstraction layer that does not provide any design directions whatsoever (not to mention much better performance than the ORM), please consider SQLAlchemy Core: http://docs.sqlalchemy.org/en/rel_0_9/core/index.html. >> >> >> >> _______________________________________________ >> DB-SIG maillist - DB-SIG at python.org >> https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 17 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From elfring at users.sourceforge.net Wed Dec 17 14:50:47 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Wed, 17 Dec 2014 14:50:47 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54918596.1000807@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> Message-ID: <54918A37.2040001@users.sourceforge.net> > How about: > > """ > The prepared operation is only available until the next call to > one of the .execute*() methods or another call to the .prepare() > method using a different command string than the one used for > preparing the previous operation. > """ Can the prepare() method return a handle for a SQL statement? Will prepared SQL statements be mapped to a specific class hierarchy in Python? > It is possible to open a few cursors, prepare frequently > used statements on them and them make them accessible via > a cursor pool. This works well (I know since I've implemented > such a logic some 14-15 years ago in a large application). How should the application programming interface look like in Python for this use case? > The DB-API standard extensions were added to address proliferation > of such additions. If possible, the database modules should try > to standardize on these extensions, which is why discussions such > as these are useful. Are there more software improvements in a waiting queue? Regards, Markus From elfring at users.sourceforge.net Wed Dec 17 15:38:37 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Wed, 17 Dec 2014 15:38:37 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54919177.70205@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> Message-ID: <5491956D.9000503@users.sourceforge.net> >> Can the prepare() method return a handle for a SQL statement? > > The proposal says: > > """ > Return values are not defined. > """ How do you think about to change the application programming interface here? Would a class like "prepared_statement" be more useful at this place? >> Will prepared SQL statements be mapped to a specific class hierarchy >> in Python? > > Not sure what you mean. The structured query language supports a few standard statements like insert, update and delete. I imagine that specific classes will be useful for their parameterisation, won't they? >>> It is possible to open a few cursors, prepare frequently >>> used statements on them and them make them accessible via >>> a cursor pool. This works well (I know since I've implemented >>> such a logic some 14-15 years ago in a large application). >> >> How should the application programming interface look like >> in Python for this use case? > > Depends on the application and what you want to achieve. > There's no general answer to this. I find that view strange. I would appreciate if the corresponding software abstractions can be improved in reasonable ways. Regards, Markus From elfring at users.sourceforge.net Wed Dec 17 16:18:26 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Wed, 17 Dec 2014 16:18:26 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54919B2D.7010306@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> Message-ID: <54919EC2.7040007@users.sourceforge.net> > Please explain what you want in more detail, so we can check > whether there's room for improvement :-) Can the SQL update statement be encapsulated by a dedicated Python class for example? > Note that the DB-API doesn't require implementing connection > or cursor pools. It only provides ways of making these possible > at higher abstraction levels. I am looking for higher service levels without following the software design directions from object-relational managers like SQLAlchemy and SQLObject. Regards, Markus From songofacandy at gmail.com Wed Dec 17 19:13:44 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 18 Dec 2014 03:13:44 +0900 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: As I said before, prepared statement is normally bound to connection. So `.prepare()` method should be connection's method, not cursor's. prepared = conn.prepare("SELECT ?+?") ... cur = conn.cursor() cur.execute(prepared, (1, 2)) cur.fetchone() # returns (3,) On Thu, Dec 18, 2014 at 2:54 AM, Michael Bayer wrote: > >> On Dec 17, 2014, at 10:41 AM, M.-A. Lemburg wrote: >> >> >> Another possibility is to write a layer on top of the DB-API >> to abstract the underlying queries away from the application >> and only have the layer provide dedicated methods for the >> things the application needs, such as query method for specific >> details or inserting application objects into the database. >> >> This is the approach we usually take in our projects, since >> it provides better separation of the application logic from the >> database logic than using ORMs usually provides. It's also possible >> to use such an abstraction layer on top of an ORM, if you want >> to the ORM to deal with abstracting away database backend details. > > When I read this quickly, it seems to make sense, but when I really try to imagine what this means, I come out at the same place every time: if you?ve written a ?method for inserting application objects into the database?, you?ve written an ORM. Databases don?t store ?application objects?, they store rows. So there has to be ?ORM? in there. > >>> >>> I am looking for higher service levels without following the software >>> design directions from object-relational managers like SQLAlchemy >>> and SQLObject. > > replying to Markus - > > SQLAlchemy?s ORM is only an optional component of the SQLAlchemy database toolkit overall. If you would like a very mature and well proven SQL abstraction layer that does not provide any design directions whatsoever (not to mention much better performance than the ORM), please consider SQLAlchemy Core: http://docs.sqlalchemy.org/en/rel_0_9/core/index.html. > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig -- INADA Naoki From tlocke at tlocke.org.uk Wed Dec 17 20:37:57 2014 From: tlocke at tlocke.org.uk (Tony Locke) Date: Wed, 17 Dec 2014 19:37:57 +0000 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: On the question of extending the DB-API to accommodate prepared statements, from my narrow point of view as a maintainer of pg8000 it's unnecessary, as pg8000 uses prepared statements for everything anyway. In later versions of Postgres, the query plan is potentially updated every time a prepared statement is executed, as opposed to the old days when the query plan for a prepared statement was fixed. So (again from my limited point of view) it seems there isn't any reason ever to use anything other than a prepared statement. Regards, Tony. On 17 December 2014 at 17:54, Michael Bayer wrote: > >> On Dec 17, 2014, at 10:41 AM, M.-A. Lemburg wrote: >> >> >> Another possibility is to write a layer on top of the DB-API >> to abstract the underlying queries away from the application >> and only have the layer provide dedicated methods for the >> things the application needs, such as query method for specific >> details or inserting application objects into the database. >> >> This is the approach we usually take in our projects, since >> it provides better separation of the application logic from the >> database logic than using ORMs usually provides. It's also possible >> to use such an abstraction layer on top of an ORM, if you want >> to the ORM to deal with abstracting away database backend details. > > When I read this quickly, it seems to make sense, but when I really try to imagine what this means, I come out at the same place every time: if you?ve written a ?method for inserting application objects into the database?, you?ve written an ORM. Databases don?t store ?application objects?, they store rows. So there has to be ?ORM? in there. > >>> >>> I am looking for higher service levels without following the software >>> design directions from object-relational managers like SQLAlchemy >>> and SQLObject. > > replying to Markus - > > SQLAlchemy?s ORM is only an optional component of the SQLAlchemy database toolkit overall. If you would like a very mature and well proven SQL abstraction layer that does not provide any design directions whatsoever (not to mention much better performance than the ORM), please consider SQLAlchemy Core: http://docs.sqlalchemy.org/en/rel_0_9/core/index.html. > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig From elfring at users.sourceforge.net Thu Dec 18 08:10:09 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Thu, 18 Dec 2014 08:10:09 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: <54927DD1.60306@users.sourceforge.net> >>> I am looking for higher service levels without following the software >>> design directions from object-relational managers like SQLAlchemy >>> and SQLObject. > > If you would like a very mature and well proven SQL abstraction layer > that does not provide any design directions whatsoever (not to mention > much better performance than the ORM), please consider SQLAlchemy Core Thanks for your suggestion. Where and how does this application programming interface deal with prepared SQL statements? http://docs.sqlalchemy.org/en/rel_0_9/core/dml.html Regards, Markus From mal at egenix.com Thu Dec 18 11:39:24 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Dec 2014 11:39:24 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> Message-ID: <5492AEDC.6060400@egenix.com> On 17.12.2014 19:13, INADA Naoki wrote: > As I said before, prepared statement is normally bound to connection. > So `.prepare()` method should be connection's method, not cursor's. > > prepared = conn.prepare("SELECT ?+?") > ... > cur = conn.cursor() > cur.execute(prepared, (1, 2)) > cur.fetchone() # returns (3,) I'm not sure which database you are talking about, but in terms of concepts, statements are run on cursors, so adding the method to connections doesn't look right (we dropped this logic when moving from DB-API 1.0 to 2.0 a long time ago). Also note that the prepare step may need access to the cursor configuration settings to be correctly interpreted by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 18 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Dec 18 13:33:02 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Dec 2014 13:33:02 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> Message-ID: <5492C97E.3030303@egenix.com> On 18.12.2014 12:27, INADA Naoki wrote: > On Thu, Dec 18, 2014 at 7:39 PM, M.-A. Lemburg wrote: >> On 17.12.2014 19:13, INADA Naoki wrote: >>> As I said before, prepared statement is normally bound to connection. >>> So `.prepare()` method should be connection's method, not cursor's. >>> >>> prepared = conn.prepare("SELECT ?+?") >>> ... >>> cur = conn.cursor() >>> cur.execute(prepared, (1, 2)) >>> cur.fetchone() # returns (3,) >> >> I'm not sure which database you are talking about, > > I'm a developer of MySQL drivers (PyMySQL and mysqlclient). > prepared statement is alive as long as connection is alive. > We can use one prepared statement multiple times. So MySQL separates the concepts of a prepared statement and a statement which is used to execute a query ? >> but in terms >> of concepts, statements are run on cursors, so adding the method >> to connections doesn't look right (we dropped this logic when moving >> from DB-API 1.0 to 2.0 a long time ago). > > PEP 249 says: > >> Cursor Objects >> >> These objects represent a database cursor, which is used to manage the context of a fetch operation. > > Preparing statement is not fetching query result. The DB-API is a bit terse in this respect. The two central concepts in the DB-API are connections and cursors: Connections provide a connection interface to the database and encapsulate a transactional view on database operations. Cursors provide a way to execute SQL statements and fetch the corresponding data. Cursors are created on connections and bound to these. Now instead of creating a third concept, that of a prepared statement, I think it's better to stick to the above two concepts and simply add a method to access the intermediate step of preparing a statement on the cursor, which will then get executed on the cursor. Since the DB-API tries to provide an API which works for many databases, the small glitch with having MySQL use a different concept is acceptable, IMO. >> Also note that the prepare step may need access to the >> cursor configuration settings to be correctly interpreted >> by the database. > > I'm not sure which database you are talking about. > MySQL has configuration per connection, not per cursor. My experience is from working with ODBC and ODBC drivers. IBM DB2 and MS SQL Server use ODBC as their native database interface API. In ODBC, cursors are called "statements" and you have two modes of operation: a) direct execution, which sends the SQL straight to the database b) prepare + execute, which separates the prepare from the execute step You can configure both connections and cursors in ODBC, e.g. the cursor type setting is configured on a per cursor basis and this has direct influence on what locking mechanisms need to be enabled in the database to run the SQL statement: http://msdn.microsoft.com/en-us/library/ms712631%28v=vs.85%29.aspx Note that the DB-API is modeled in many aspects after the ODBC API and its concepts, since ODBC is an industry standard and provides a good common denominator for how database APIs work and which concepts they can support. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 18 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Thu Dec 18 16:39:00 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 10:39:00 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54927DD1.60306@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <54927DD1.60306@users.sourceforge.net> Message-ID: > On Dec 18, 2014, at 2:10 AM, SF Markus Elfring wrote: > >>>> I am looking for higher service levels without following the software >>>> design directions from object-relational managers like SQLAlchemy >>>> and SQLObject. >> >> If you would like a very mature and well proven SQL abstraction layer >> that does not provide any design directions whatsoever (not to mention >> much better performance than the ORM), please consider SQLAlchemy Core > > Thanks for your suggestion. > > Where and how does this application programming interface deal with prepared SQL statements? > http://docs.sqlalchemy.org/en/rel_0_9/core/dml.html because the DBAPI has no prepared statement API (as of yet), the only access we are given to that functionality is if the DBAPI in use already does this internally. The most common way to take advantage of prepared statements, if they in fact provide a performance improvement on the target backend, is to use the DBAPI executemany() method; if the backing DBAPI uses prepared statements in order to achieve a performance gain here, then you?ll get access to that. executemany() performs dramatically better than execute() in any case because it?s Python function call overhead that really is the source of performance issues. SQLAlchemy offers the executemany() API just by passing a list of parameters to the execute() method: http://docs.sqlalchemy.org/en/rel_0_9/core/tutorial.html#executing-multiple-statements, but this only applies to CRUD (insert, update, delete) statements. In my experience, caching of prepared statements in modern use is not a major source of performance gain unless your application overall is already blindingly fast, more than Python typically provides. Consider that a site like reddit.com serves six billion page views a month from Postgresql databases with no use of prepared statements. From mike_mp at zzzcomputing.com Thu Dec 18 16:46:35 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 10:46:35 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5484C15C.8010003@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> Message-ID: <1CAAB7B0-9C35-4D6C-B2C7-EEE9F24D67C6@zzzcomputing.com> > On Dec 7, 2014, at 4:06 PM, SF Markus Elfring wrote: > > I imagine that it will be more efficient occasionally to offer also a > base class like "prepared_statement" so that the parameter specification > does not need to be parsed for every passed command. > I suggest to improve corresponding preparation and compilation > possibilities. > https://bugs.python.org/issue22956 IMHO, before we add a complex new interface to the DBAPI, which will greatly add to the workload both of DBAPI authors now asked to support this as well as downstream abstraction layers who will have the same issue, I?d like to request that we have a formal rationale for the addition of this feature as an explicit API. Looking through this thread, I?ve searched for one, and it appears that this is the only one: "I **imagine** that it will be more efficient?. As it happens, I ?imagine? that it will make hardly any difference at all, if not perform worse due to the complexity of prepared statements, as well as that it will cause downstream users to write more complicated code in order to deal with attempting to cache these statements. We use Python because it is a high level scripting language, that foregoes the exposure of low-level details in favor of simplicity of code. Might we add features to the DBAPI based on concrete benchmarks and observations rather than our imaginations? Can someone please take up this task and prove to me with benchmarks that this effort will be worthwhile (or at least show me what I?m going to have to put into my documentation as to when these new APIs are appropriate) ? From mike_mp at zzzcomputing.com Thu Dec 18 16:59:39 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 10:59:39 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5492AEDC.6060400@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> Message-ID: > On Dec 18, 2014, at 5:39 AM, M.-A. Lemburg wrote: > > On 17.12.2014 19:13, INADA Naoki wrote: >> As I said before, prepared statement is normally bound to connection. >> So `.prepare()` method should be connection's method, not cursor's. >> >> prepared = conn.prepare("SELECT ?+?") >> ... >> cur = conn.cursor() >> cur.execute(prepared, (1, 2)) >> cur.fetchone() # returns (3,) > > I'm not sure which database you are talking about, but in terms > of concepts, statements are run on cursors, so adding the method > to connections doesn't look right (we dropped this logic when moving > from DB-API 1.0 to 2.0 a long time ago). > > Also note that the prepare step may need access to the > cursor configuration settings to be correctly interpreted > by the database. oh, so are you saying that if one produces a prepared statement from a cursor, and then that cursor is closed, I have no option to re-use that prepared statement anywhere else, is that right? That would make the entire feature a non-starter for me. SQLAlchemy doesn?t hold cursors open beyond a single statement. My users would very much want a prepared-statement-per-transaction object. JDBC provides prepared statements as a service of the Connection. I think DBAPI should be doing the same. From phd at phdru.name Thu Dec 18 18:23:17 2014 From: phd at phdru.name (Oleg Broytman) Date: Thu, 18 Dec 2014 18:23:17 +0100 Subject: [DB-SIG] SQLObject 1.7.3 Message-ID: <20141218172317.GA824@phdru.name> Hello! I'm pleased to announce version 1.7.3, a release with minor documentation update of branch 1.7 of SQLObject. What's new in SQLObject ======================= * Extend setup.py: include docs and tests into the egg. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Python 2.6 or 2.7 is required. Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/1.7.3 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From vernondcole at gmail.com Thu Dec 18 20:37:10 2014 From: vernondcole at gmail.com (Vernon D. Cole) Date: Thu, 18 Dec 2014 12:37:10 -0700 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> Message-ID: 1) There is already an established usual practice (or quasi-standard) for prepared statements: >>> sql_stmt = "some sql string" >>> cursr.prepare(sql_stmt) >>> cursr.execute(sql_stmt, args) # note: the statement string must be identical Any change in sql_stmt undoes the prepared status and the "execute" method continues as if the "prepare" method had not been executed. 2) At least two implementations enhance this by having the "prepare" method store a reference to the sql statement as the read-only cursor attribute "cursr.command". The api must store this reference, in order to see whether the sql statement is changed, so making it available is inexpensive. The calling sequence then becomes: >>> cursr.prepare("some sql string") >>> cursr.execute(cursr.command, args) How can you get simpler, or easier to understand, than that? It is also easy to implement, at least, it was on the platform I use (i.e. Microsoft ADO). 3) If a different way of handling prepared statements is invented and standardized, then those of us who have already implemented this method will have to support both -- since we have existing customers already using this one. 4) How much will prepared statements really help? Published Microsoft sources indicate that it will make no significant difference on their databases. [One wonders whether they are already so slow that they cannot get any worse?] In my case, it saves some work in Python, such as converting between format types -- but the actual database query is just as inefficient both ways. From mike_mp at zzzcomputing.com Thu Dec 18 21:23:02 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 15:23:02 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54932456.3080402@users.sourceforge.net> References: <5484C15C.8010003@users.sourceforge.net> <1CAAB7B0-9C35-4D6C-B2C7-EEE9F24D67C6@zzzcomputing.com> <54932456.3080402@users.sourceforge.net> Message-ID: <36A7B584-1A63-4442-AE33-1D0B3007F195@zzzcomputing.com> > On Dec 18, 2014, at 2:00 PM, SF Markus Elfring wrote: > >> Can someone please take up this task and prove to me with benchmarks that this effort will be worthwhile >> (or at least show me what I?m going to have to put into my documentation as to when these new APIs are appropriate) ? > > Some database software supports the processing of prepared SQL statements > by well-known application programming interfaces for decades, doesn't it? > > Would you like to reuse any more experiences about execution speed and program > safety from software development history? Below is a benchmark script, psycopg2 vs. pg8000. The former uses no prepared statements whatsoever, the latter uses prepared statements for all executions. To even make it more ?fair? I will illustrate an executemany(), so that pg8000 can definitely take advantage of ?caching? this statement at the prepared statement level. The output times, for INSERTing 50000 rows, using five batches of 10000 (so that pg8000 can re-use the same prepared statement 10000 times), is: Total time for psycopg2: 4.992107 Total time for pg8000: 10.770641 Now as it turns out, this test is entirely unfair. psycopg2 is written in pure C code and talks to libpq directly, whereas pg8000 is pure Python and invokes postgresql?s protocol directly. There is no chance of pg8000 ever approaching the speed of psycopg2. In my experiences about execution speed in Python, enhancements like being able to cache prepared statements are dwarfed by the key issues in Python - that of function call overhead and code complexity. In this case, choosing the DBAPI because one uses prepared statements and the other does not would not be the right approach. It doesn?t matter that prepared statements are traditionally useful. The complexity they would introduce, where programmers take a guess that using prepared statements in conjunction with the necessary statement-caching architecture surrounding their use will make their program faster, will more often than not have a net negative effect. Working with Openstack developers, I regularly have to shoot down all kinds of changes that are made in the name of ?will (theoretically) perform better?, yet in reality, when benchmarked, they perform worse, or hardly better at all, at the expense of greater complexity. The theories are supported by all kinds of blanket statements such as ?iterators are faster?, ?using Core statements are faster than the ORM?, ?asynchronous code is faster than synchronous?, which at face value are true in some circumstances, but in practice in many specific situations are not at all better from a performance standpoint and only complicate the code. Basically I want there to be very good reason for this feature to be an explicit part of the DBAPI because it is going to give me personally a lot of extra headaches when people start asking for it in the name of ?performance?, especially since that in practice, psycopg2, pg8000 and perhaps mxODBC will be the only DBAPIs to add this feature, the MySQL DBAPIs are unlikely to be interested in this, and the pysqlite API almost certainly won?t as they have been stalling for years just on very rudimental transactional issues that remain open. A DBAPI can already choose to make use of cached prepared statements as an internal optimization, using an LRU cache that could be configurable via the connect() function, or just via the executemany() API as some do right now. In my work with consumers of the DBAPI, prepared statement support is not high (or even at all) on the list of needs. DBAPIs that can do unicode very well, support asynchronous APIs such as asyncio as well as implicit environments such as gevent, have lots of robust datatype support as well as very robust failure mode handling, that?s what people are looking for (also areas that psycopg2 does extremely well), and these are all areas in which the actual pep-249 is IMO sorely lacking in guidance and could use more immediate attention first. import random import timeit import psycopg2 import pg8000 def setup(conn): cursor = conn.cursor() cursor.execute("drop table if exists data") cursor.execute( "create table data (id SERIAL primary key, data VARCHAR(100))") cursor.close() def run_test(conn): cursor = conn.cursor() cursor.executemany( "insert into data (data) values (%s)", [ ("some value: %d" % random.randint(0, 10000),) for counter in xrange(10000) ] ) cursor.close() def do_time(dbapi): global conn conn = dbapi.connect( user='scott', password='tiger', database='test', host='localhost') time = timeit.timeit( "run_test(conn)", "from __main__ import run_test, setup, conn; setup(conn)", number=5 ) conn.close() return time psycopg2_time = do_time(psycopg2) pg8000_time = do_time(pg8000) print("Total time for psycopg2: %f" % psycopg2_time) print("Total time for pg8000: %f" % pg8000_time) From mal at egenix.com Thu Dec 18 21:39:45 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Dec 2014 21:39:45 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> Message-ID: <54933B91.9030408@egenix.com> On 18.12.2014 16:59, Michael Bayer wrote: > >> On Dec 18, 2014, at 5:39 AM, M.-A. Lemburg wrote: >> >> On 17.12.2014 19:13, INADA Naoki wrote: >>> As I said before, prepared statement is normally bound to connection. >>> So `.prepare()` method should be connection's method, not cursor's. >>> >>> prepared = conn.prepare("SELECT ?+?") >>> ... >>> cur = conn.cursor() >>> cur.execute(prepared, (1, 2)) >>> cur.fetchone() # returns (3,) >> >> I'm not sure which database you are talking about, but in terms >> of concepts, statements are run on cursors, so adding the method >> to connections doesn't look right (we dropped this logic when moving >> from DB-API 1.0 to 2.0 a long time ago). >> >> Also note that the prepare step may need access to the >> cursor configuration settings to be correctly interpreted >> by the database. > > oh, so are you saying that if one produces a prepared statement from a cursor, and then that cursor is closed, I have no option to re-use that prepared statement anywhere else, is that right? Yes, that's what happens if you close a cursor. Note that cursors are normally kept open to run multiple statements or multiple runs of the same statement. > That would make the entire feature a non-starter for me. SQLAlchemy doesn?t hold cursors open beyond a single statement. My users would very much want a prepared-statement-per-transaction object. Perhaps you ought to reconsider this approach. Creating and closing cursors all the time does involve somewhat of an overhead. > JDBC provides prepared statements as a service of the Connection. I think DBAPI should be doing the same. In order to make use of those prepared statements you will have to create a cursor one way or another anyway, so I don't see an advantage of having the method on connections. It's possible for database modules to provide a .prepare() method on connections, but since fewer databases provide this functionality than having one on cursors, the DB-API should go with what's easy for module authors to implement. Note that the use of cached prepared cursors for performance reasons is only one use of having the prepare step available on cursors. In practice, the more important one is to be able check SQL statements for errors without executing them and possibly causing a rollback on the transaction. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 18 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Dec 18 21:56:27 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Dec 2014 21:56:27 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <36A7B584-1A63-4442-AE33-1D0B3007F195@zzzcomputing.com> References: <5484C15C.8010003@users.sourceforge.net> <1CAAB7B0-9C35-4D6C-B2C7-EEE9F24D67C6@zzzcomputing.com> <54932456.3080402@users.sourceforge.net> <36A7B584-1A63-4442-AE33-1D0B3007F195@zzzcomputing.com> Message-ID: <54933F7B.9090204@egenix.com> On 18.12.2014 21:23, Michael Bayer wrote: > Basically I want there to be very good reason for this feature to be an explicit part of the DBAPI because it is going to give me personally a lot of extra headaches when people start asking for it in the name of ?performance?, especially since that in practice, psycopg2, pg8000 and perhaps mxODBC will be the only DBAPIs to add this feature, the MySQL DBAPIs are unlikely to be interested in this, and the pysqlite API almost certainly won?t as they have been stalling for years just on very rudimental transactional issues that remain open. A DBAPI can already choose to make use of cached prepared statements as an internal optimization, using an LRU cache that could be configurable via the connect() function, or just via the executemany() API as some do right now. We are discussing a standard extension to the DB-API, not a mandatory feature. The point of the discussion is to come up with a design that the database authors, who want to implement this or have already implemented it, can use the same semantics for the extension. As mentioned in my other reply, caching or reusing prepared cursors is not necessarily the main reason for having a .prepare() method. You can have prepared cursors without this method by running the queries on a cursor and then caching the used cursors for reuse in a pool. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 18 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Thu Dec 18 21:57:01 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 15:57:01 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54933B91.9030408@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> Message-ID: <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> > On Dec 18, 2014, at 3:39 PM, M.-A. Lemburg wrote: > > >> That would make the entire feature a non-starter for me. SQLAlchemy doesn?t hold cursors open beyond a single statement. My users would very much want a prepared-statement-per-transaction object. > > Perhaps you ought to reconsider this approach. Creating and closing > cursors all the time does involve somewhat of an overhead. I will attempt to try this, though I am anticipating that DBAPIs are going to be problematic with this approach. One concrete example is the case where on psycopg2, we offer the option to use a ?named? cursor, which on psycopg2 has the effect of maintaining the state of this cursor on the server side. However psycopg2 throws an error if such a cursor is used for anything other than a SELECT statement. So right there, we need more than one cursor based on the contents of the SQL. This is kind of a very specific situation though, I?ll see if the approach in general produces issues. > Note that the use of cached prepared cursors for performance > reasons is only one use of having the prepare step available > on cursors. In practice, the more important one is to be able > check SQL statements for errors without executing them and > possibly causing a rollback on the transaction. Which kinds of errors are you referring to, if the statement has not been invoked, I would imagine this refers only to syntactical errors? What kind of application contains SQL that may have syntactical errors that only become apparent at runtime and can?t be eliminated during testing? > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Dec 18 2014) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mal at egenix.com Thu Dec 18 22:36:41 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Dec 2014 22:36:41 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> Message-ID: <549348E9.7080106@egenix.com> On 18.12.2014 21:57, Michael Bayer wrote: > >> On Dec 18, 2014, at 3:39 PM, M.-A. Lemburg wrote: >> >> >>> That would make the entire feature a non-starter for me. SQLAlchemy doesn?t hold cursors open beyond a single statement. My users would very much want a prepared-statement-per-transaction object. >> >> Perhaps you ought to reconsider this approach. Creating and closing >> cursors all the time does involve somewhat of an overhead. > > I will attempt to try this, though I am anticipating that DBAPIs are going to be problematic with this approach. One concrete example is the case where on psycopg2, we offer the option to use a ?named? cursor, which on psycopg2 has the effect of maintaining the state of this cursor on the server side. However psycopg2 throws an error if such a cursor is used for anything other than a SELECT statement. So right there, we need more than one cursor based on the contents of the SQL. This is kind of a very specific situation though, I?ll see if the approach in general produces issues. >> Note that the use of cached prepared cursors for performance >> reasons is only one use of having the prepare step available >> on cursors. In practice, the more important one is to be able >> check SQL statements for errors without executing them and >> possibly causing a rollback on the transaction. > > Which kinds of errors are you referring to, if the statement has not been invoked, I would imagine this refers only to syntactical errors? What kind of application contains SQL that may have syntactical errors that only become apparent at runtime and can?t be eliminated during testing? PostgreSQL is an example where the transaction gets canceled if you e.g. query a non-existing table in the SQL statement or have use some other non-existing entity in your SQL. Since the transaction runs on the connection, all cursors currently open are affected by this, and this can be a hassle to clean up, depending on what you're doing :-) Another use case is having the database tell you more about the result columns or the data types of the parameters used in the SQL statement. The latter information is not available through standard DB-API interfaces, though. Perhaps something to consider for another optional extension :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 18 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Thu Dec 18 22:42:47 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 16:42:47 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <549348E9.7080106@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> <549348E9.7080106@egenix.com> Message-ID: > On Dec 18, 2014, at 4:36 PM, M.-A. Lemburg wrote: > >> >> Which kinds of errors are you referring to, if the statement has not been invoked, I would imagine this refers only to syntactical errors? What kind of application contains SQL that may have syntactical errors that only become apparent at runtime and can?t be eliminated during testing? > > PostgreSQL is an example where the transaction gets canceled if > you e.g. query a non-existing table in the SQL statement or have > use some other non-existing entity in your SQL. right, but in practice, those errors are caught in testing, as they relate to the fixed structure of the SQL statement and are therefore part of the application?s code; there is no need for such errors to be recoverable. Only if your application is a open-ended query builder that exposes the database directly, or it?s some kind of live upgrade of database schemas, would this be an issue; in the latter case you?d be rolling back. > > Since the transaction runs on the connection, all cursors currently > open are affected by this, and this can be a hassle to clean up, > depending on what you're doing :-) > Another use case is having the database tell you more about the > result columns or the data types of the parameters used in the > SQL statement. The latter information is not available through > standard DB-API interfaces, though. Perhaps something to consider > for another optional extension :-) Well that doesn?t need to expose a prepared statement to be possible. The DBAPI spec as is has a lot of problems with this, such as setinputsizes/setoutputsizes, because the typing model in pep-249 is so extremely minimal. DBAPIs generally don?t try to expand upon that model because there are no guidelines for doing such (e.g., FLOAT / INTEGER subclasses of NUMBER, for example). > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Dec 18 2014) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Thu Dec 18 23:19:56 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 17:19:56 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> Message-ID: > On Dec 18, 2014, at 3:57 PM, Michael Bayer wrote: > >> >> On Dec 18, 2014, at 3:39 PM, M.-A. Lemburg wrote: >> >> >>> That would make the entire feature a non-starter for me. SQLAlchemy doesn?t hold cursors open beyond a single statement. My users would very much want a prepared-statement-per-transaction object. >> >> Perhaps you ought to reconsider this approach. Creating and closing >> cursors all the time does involve somewhat of an overhead. > > I will attempt to try this, though I am anticipating that DBAPIs are going to be problematic with this approach. One concrete example is the case where on psycopg2, we offer the option to use a ?named? cursor, which on psycopg2 has the effect of maintaining the state of this cursor on the server side. However psycopg2 throws an error if such a cursor is used for anything other than a SELECT statement. So right there, we need more than one cursor based on the contents of the SQL. This is kind of a very specific situation though, I?ll see if the approach in general produces issues. So I tried this, and pleasantly, there?s not *too* much side effect, meaning a quick test against a few databases didn?t lead to many issues. Where there were issues are in the tests relating to connection invalidation within a 2pc context; I didn?t dig in to what the issues are but its possible that the MySQL and psycopg2 DBAPIs have some more quirks with cursors when 2pc is used (or my changes just were missing some edge cases). However, if I were to change this for real, it means that small bump in stability now gets sent out to everyone, working on databases I don?t even have regular access to such as sybase and DB2, and whatever quirks of reusing cursors might exist that I?ve not been able to test; many years of effort and user-feedback has gone into getting our Connection class to be stable and predictable in an extremely wide variety of situations (where we?re talking here about failure modes: disconnects, deadlocks, timeouts, intercepting these conditions perfectly and getting the system back into a stable state as efficiently and crash-free as possible), and here we?re presented with the potential of overhead from opening and closing many cursors, rather than keeping one around for?I would presume the transaction scope. This is exactly what I was getting at in my other email. We are considering a significant change in a key area of stability in the name of ?reducing overhead?, so is it really worth it? For the drivers that the vast majority of my users care about at least, the effect would appear to be negligible, hitting barely a 1% difference with the pure Python drivers that have much bigger performance problems just by being in pure Python: psycopg2 single cursor: 6.159881 (10000 executions) psycopg2 multi cursor: 6.173749 (10000 executions) pg8000 single cursor: 28.213494 (1000 executions) pg8000 multi cursor: 28.620359 (1000 executions) mysqldb single cursor (10000 executions): 11.702930 mysqldb multi cursor (10000 executions): 11.809935 mysql connector single cursor (1000 executions): 25.707400 mysql connector multi cursor (1000 executions): 26.096313 I also had the idea that maybe the above cases don?t show much because these drivers aren?t using pure ?server side? cursors in the first place; I know in ODBC, we have more of a formal ?cursor? construct in the protocol and that is probably what you?re referring to (though I googled that just now, and per http://msdn.microsoft.com/en-us/library/ms130794.aspx it says explicit cursors are rarely used and ODBC automatically opens a cursor for individual result sets). So I decided to try psycopg2 with a ?named? cursor, and got this: psycopg2.ProgrammingError: can't call .execute() on named cursors more than once Wow! So that?s definitely that :). If my DB2 users upgrade to SQLAlchemy 1.0 and start experiencing less stable behavior with connections, they?d be fairly upset if I told them it was in the name of a 0.3% overhead improvement. Kind of like in the case of prepared statements, if the server-side cursors can truly be safely recycled, this is highly dependent on the database and the DBAPI methodology in use, and again the DBAPI could offer this as a configurable feature not exposed on the outside (the way ODBC can offer transparent connection pooling, if you will). import random import timeit import psycopg2 import pg8000 import MySQLdb from mysql import connector as myconnpy def setup(conn): cursor = conn.cursor() cursor.execute("drop table if exists data") cursor.execute( "create table data (id integer primary key, data VARCHAR(100))") cursor.executemany( "insert into data (id, data) values (%s, %s)", [ (counter + 1, "some value: %d" % random.randint(0, 10000),) for counter in xrange(1000) ] ) cursor.close() def run_test_multi_cursor(conn): cursor = conn.cursor() cursor.execute("select * from data") cursor.fetchall() cursor.close() def run_test_single_cursor(conn, cursor): cursor.execute("select * from data") cursor.fetchall() def do_time(dbapi, single_cursor, style, number): global conn if style == 'postgresql': conn = dbapi.connect( user='scott', password='tiger', database='test', host='localhost') elif style == 'mysql': conn = dbapi.connect( user='scott', passwd='tiger', db='test', host='localhost') if single_cursor: global cursor cursor = conn.cursor() time = timeit.timeit( "run_test_single_cursor(conn, cursor)", "from __main__ import run_test_single_cursor, setup, " "conn, cursor; setup(conn)", number=number ) cursor.close() else: time = timeit.timeit( "run_test_multi_cursor(conn)", "from __main__ import run_test_multi_cursor, " "setup, conn; setup(conn)", number=number ) conn.close() return time psycopg2_cursor_time = do_time(psycopg2, True, 'postgresql', 10000) psycopg2_non_cursor_time = do_time(psycopg2, False, 'postgresql', 10000) pg8000_cursor_time = do_time(pg8000, True, 'postgresql', 1000) pg8000_non_cursor_time = do_time(pg8000, False, 'postgresql', 1000) mysqldb_cursor_time = do_time(MySQLdb, True, 'mysql', 10000) mysqldb_non_cursor_time = do_time(MySQLdb, False, 'mysql', 10000) mysqlconn_cursor_time = do_time(myconnpy, True, 'mysql', 1000) mysqlconn_non_cursor_time = do_time(myconnpy, False, 'mysql', 1000) print("psycopg2 single cursor: %f (10000 executions)" % psycopg2_cursor_time) print("psycopg2 multi cursor: %f (10000 executions)" % psycopg2_non_cursor_time) print("pg8000 single cursor: %f (1000 executions)" % pg8000_cursor_time) print("pg8000 multi cursor: %f (1000 executions)" % pg8000_non_cursor_time) print("mysqldb single cursor (10000 executions): %f" % mysqldb_cursor_time) print("mysqldb multi cursor (10000 executions): %f" % mysqldb_non_cursor_time) print("mysql connector single cursor (1000 executions): %f" % mysqlconn_cursor_time) print("mysql connector multi cursor (1000 executions): %f" % mysqlconn_non_cursor_time) > >> Note that the use of cached prepared cursors for performance >> reasons is only one use of having the prepare step available >> on cursors. In practice, the more important one is to be able >> check SQL statements for errors without executing them and >> possibly causing a rollback on the transaction. > > Which kinds of errors are you referring to, if the statement has not been invoked, I would imagine this refers only to syntactical errors? What kind of application contains SQL that may have syntactical errors that only become apparent at runtime and can?t be eliminated during testing? > > > > >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Dec 18 2014) >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig From mal at egenix.com Thu Dec 18 23:55:50 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Dec 2014 23:55:50 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> Message-ID: <54935B76.5070805@egenix.com> On 18.12.2014 23:19, Michael Bayer wrote: > >> On Dec 18, 2014, at 3:57 PM, Michael Bayer wrote: >> >>> >>> On Dec 18, 2014, at 3:39 PM, M.-A. Lemburg wrote: >>> >>> >>>> That would make the entire feature a non-starter for me. SQLAlchemy doesn?t hold cursors open beyond a single statement. My users would very much want a prepared-statement-per-transaction object. >>> >>> Perhaps you ought to reconsider this approach. Creating and closing >>> cursors all the time does involve somewhat of an overhead. >> >> I will attempt to try this, though I am anticipating that DBAPIs are going to be problematic with this approach. One concrete example is the case where on psycopg2, we offer the option to use a ?named? cursor, which on psycopg2 has the effect of maintaining the state of this cursor on the server side. However psycopg2 throws an error if such a cursor is used for anything other than a SELECT statement. So right there, we need more than one cursor based on the contents of the SQL. This is kind of a very specific situation though, I?ll see if the approach in general produces issues. > > So I tried this, and pleasantly, there?s not *too* much side effect, meaning a quick test against a few databases didn?t lead to many issues. Where there were issues are in the tests relating to connection invalidation within a 2pc context; I didn?t dig in to what the issues are but its possible that the MySQL and psycopg2 DBAPIs have some more quirks with cursors when 2pc is used (or my changes just were missing some edge cases). > > However, if I were to change this for real, it means that small bump in stability now gets sent out to everyone, working on databases I don?t even have regular access to such as sybase and DB2, and whatever quirks of reusing cursors might exist that I?ve not been able to test; many years of effort and user-feedback has gone into getting our Connection class to be stable and predictable in an extremely wide variety of situations (where we?re talking here about failure modes: disconnects, deadlocks, timeouts, intercepting these conditions perfectly and getting the system back into a stable state as efficiently and crash-free as possible), and here we?re presented with the potential of overhead from opening and closing many cursors, rather than keeping one around for?I would presume the transaction scope. > > This is exactly what I was getting at in my other email. We are considering a significant change in a key area of stability in the name of ?reducing overhead?, so is it really worth it? For the drivers that the vast majority of my users care about at least, the effect would appear to be negligible, hitting barely a 1% difference with the pure Python drivers that have much bigger performance problems just by being in pure Python: > > psycopg2 single cursor: 6.159881 (10000 executions) > psycopg2 multi cursor: 6.173749 (10000 executions) > > pg8000 single cursor: 28.213494 (1000 executions) > pg8000 multi cursor: 28.620359 (1000 executions) > > mysqldb single cursor (10000 executions): 11.702930 > mysqldb multi cursor (10000 executions): 11.809935 > > mysql connector single cursor (1000 executions): 25.707400 > mysql connector multi cursor (1000 executions): 26.096313 The results are somewhat biased, since your test spends most of the time with fetching data from the database, not with the creation and deallocation of the cursor. You should try this with a table that only has e.g. 10-100 entries or a query which only yields a few result rows for comparison. In e.g. user interface applications using databases, most queries only return 1-100 rows, and of those, the number of queries which return just a single row are the most common cases. For data processing applications, the situation is different and you do work with large result sets. The cursor creation overhead doesn't really matter much for those. > I also had the idea that maybe the above cases don?t show much because these drivers aren?t using pure ?server side? cursors in the first place; I know in ODBC, we have more of a formal ?cursor? construct in the protocol and that is probably what you?re referring to (though I googled that just now, and per http://msdn.microsoft.com/en-us/library/ms130794.aspx it says explicit cursors are rarely used and ODBC automatically opens a cursor for individual result sets). So I decided to try psycopg2 with a ?named? cursor, and got this: > > psycopg2.ProgrammingError: can't call .execute() on named cursors more than once > > Wow! So that?s definitely that :). I think there's a misunderstanding here. A named cursor refers to a database cursor pointing into a result set. Once that result set has been created and the cursor points to it, you cannot run another .execute() against it, unless you first close the named cursor - simply because it is already in use. > If my DB2 users upgrade to SQLAlchemy 1.0 and start experiencing less stable behavior with connections, they?d be fairly upset if I told them it was in the name of a 0.3% overhead improvement. Kind of like in the case of prepared statements, if the server-side cursors can truly be safely recycled, this is highly dependent on the database and the DBAPI methodology in use, and again the DBAPI could offer this as a configurable feature not exposed on the outside (the way ODBC can offer transparent connection pooling, if you will). > > > > > import random > import timeit > import psycopg2 > import pg8000 > import MySQLdb > from mysql import connector as myconnpy > > > def setup(conn): > cursor = conn.cursor() > cursor.execute("drop table if exists data") > cursor.execute( > "create table data (id integer primary key, data VARCHAR(100))") > cursor.executemany( > "insert into data (id, data) values (%s, %s)", > [ > (counter + 1, "some value: %d" % random.randint(0, 10000),) > for counter in xrange(1000) > ] > ) > cursor.close() > > > def run_test_multi_cursor(conn): > cursor = conn.cursor() > cursor.execute("select * from data") > cursor.fetchall() > cursor.close() > > > def run_test_single_cursor(conn, cursor): > cursor.execute("select * from data") > cursor.fetchall() > > > def do_time(dbapi, single_cursor, style, number): > global conn > > if style == 'postgresql': > conn = dbapi.connect( > user='scott', password='tiger', > database='test', host='localhost') > elif style == 'mysql': > conn = dbapi.connect( > user='scott', passwd='tiger', > db='test', host='localhost') > > if single_cursor: > global cursor > cursor = conn.cursor() > time = timeit.timeit( > "run_test_single_cursor(conn, cursor)", > "from __main__ import run_test_single_cursor, setup, " > "conn, cursor; setup(conn)", > number=number > ) > cursor.close() > else: > time = timeit.timeit( > "run_test_multi_cursor(conn)", > "from __main__ import run_test_multi_cursor, " > "setup, conn; setup(conn)", > number=number > ) > > conn.close() > return time > > psycopg2_cursor_time = do_time(psycopg2, True, 'postgresql', 10000) > psycopg2_non_cursor_time = do_time(psycopg2, False, 'postgresql', 10000) > pg8000_cursor_time = do_time(pg8000, True, 'postgresql', 1000) > pg8000_non_cursor_time = do_time(pg8000, False, 'postgresql', 1000) > mysqldb_cursor_time = do_time(MySQLdb, True, 'mysql', 10000) > mysqldb_non_cursor_time = do_time(MySQLdb, False, 'mysql', 10000) > mysqlconn_cursor_time = do_time(myconnpy, True, 'mysql', 1000) > mysqlconn_non_cursor_time = do_time(myconnpy, False, 'mysql', 1000) > > > print("psycopg2 single cursor: %f (10000 executions)" % psycopg2_cursor_time) > print("psycopg2 multi cursor: %f (10000 executions)" % psycopg2_non_cursor_time) > print("pg8000 single cursor: %f (1000 executions)" % pg8000_cursor_time) > print("pg8000 multi cursor: %f (1000 executions)" % pg8000_non_cursor_time) > print("mysqldb single cursor (10000 executions): %f" % mysqldb_cursor_time) > print("mysqldb multi cursor (10000 executions): %f" % mysqldb_non_cursor_time) > print("mysql connector single cursor (1000 executions): %f" % mysqlconn_cursor_time) > print("mysql connector multi cursor (1000 executions): %f" % mysqlconn_non_cursor_time) > > > >> >>> Note that the use of cached prepared cursors for performance >>> reasons is only one use of having the prepare step available >>> on cursors. In practice, the more important one is to be able >>> check SQL statements for errors without executing them and >>> possibly causing a rollback on the transaction. >> >> Which kinds of errors are you referring to, if the statement has not been invoked, I would imagine this refers only to syntactical errors? What kind of application contains SQL that may have syntactical errors that only become apparent at runtime and can?t be eliminated during testing? >> >> >> >> >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, Dec 18 2014) >>>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >> >> _______________________________________________ >> DB-SIG maillist - DB-SIG at python.org >> https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 18 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Fri Dec 19 00:12:36 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 18 Dec 2014 18:12:36 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54935B76.5070805@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> <54935B76.5070805@egenix.com> Message-ID: M.-A. Lemburg wrote: > On 18.12.2014 23:19, Michael Bayer wrote: >>> On Dec 18, 2014, at 3:57 PM, Michael Bayer wrote: >>> >>>> On Dec 18, 2014, at 3:39 PM, M.-A. Lemburg wrote: >>>> >>>> >>>>> That would make the entire feature a non-starter for me. SQLAlchemy >>>>> doesn?t hold cursors open beyond a single statement. My users would >>>>> very much want a prepared-statement-per-transaction object. >>>> >>>> Perhaps you ought to reconsider this approach. Creating and closing >>>> cursors all the time does involve somewhat of an overhead. >>> >>> I will attempt to try this, though I am anticipating that DBAPIs are >>> going to be problematic with this approach. One concrete example is the >>> case where on psycopg2, we offer the option to use a ?named? cursor, >>> which on psycopg2 has the effect of maintaining the state of this cursor >>> on the server side. However psycopg2 throws an error if such a cursor is >>> used for anything other than a SELECT statement. So right there, we need >>> more than one cursor based on the contents of the SQL. This is kind of a >>> very specific situation though, I?ll see if the approach in general >>> produces issues. >> >> So I tried this, and pleasantly, there?s not *too* much side effect, >> meaning a quick test against a few databases didn?t lead to many issues. >> Where there were issues are in the tests relating to connection >> invalidation within a 2pc context; I didn?t dig in to what the issues are >> but its possible that the MySQL and psycopg2 DBAPIs have some more quirks >> with cursors when 2pc is used (or my changes just were missing some edge >> cases). >> >> However, if I were to change this for real, it means that small bump in >> stability now gets sent out to everyone, working on databases I don?t >> even have regular access to such as sybase and DB2, and whatever quirks >> of reusing cursors might exist that I?ve not been able to test; many >> years of effort and user-feedback has gone into getting our Connection >> class to be stable and predictable in an extremely wide variety of >> situations (where we?re talking here about failure modes: disconnects, >> deadlocks, timeouts, intercepting these conditions perfectly and getting >> the system back into a stable state as efficiently and crash-free as >> possible), and here we?re presented with the potential of overhead from >> opening and closing many cursors, rather than keeping one around for?I >> would presume the transaction scope. >> >> This is exactly what I was getting at in my other email. We are >> considering a significant change in a key area of stability in the name >> of ?reducing overhead?, so is it really worth it? For the drivers that >> the vast majority of my users care about at least, the effect would >> appear to be negligible, hitting barely a 1% difference with the pure >> Python drivers that have much bigger performance problems just by being >> in pure Python: >> >> psycopg2 single cursor: 6.159881 (10000 executions) >> psycopg2 multi cursor: 6.173749 (10000 executions) >> >> pg8000 single cursor: 28.213494 (1000 executions) >> pg8000 multi cursor: 28.620359 (1000 executions) >> >> mysqldb single cursor (10000 executions): 11.702930 >> mysqldb multi cursor (10000 executions): 11.809935 >> >> mysql connector single cursor (1000 executions): 25.707400 >> mysql connector multi cursor (1000 executions): 26.096313 > > The results are somewhat biased, since your test spends most of > the time with fetching data from the database, not with > the creation and deallocation of the cursor. > > You should try this with a table that only has e.g. 10-100 entries or > a query which only yields a few result rows for comparison. Good point, here is five rows fetched and the number of runs multiplied by 10: psycopg2 single cursor: 9.307149 (100000 executions) psycopg2 multi cursor: 9.339134 (100000 executions) pg8000 single cursor: 3.406491 (10000 executions) pg8000 multi cursor: 3.391371 (10000 executions) mysqldb single cursor (100000 executions): 8.512559 mysqldb multi cursor (100000 executions): 9.355145 mysql connector single cursor (10000 executions): 3.218654 mysql connector multi cursor (10000 executions): 4.089793 Differences are still negligible for Postgresql but for MySQL they are now tangible, but less so for the C APIs which are already an order of magnitude faster than the Python API. That is, the speed hit in MySQL-connector is nothing compared to the slowness introduced by it being in pure Python. >> psycopg2.ProgrammingError: can't call .execute() on named cursors more than once >> >> Wow! So that?s definitely that :). > > I think there's a misunderstanding here. A named cursor refers to > a database cursor pointing into a result set. Once that result > set has been created and the cursor points to it, you cannot > run another .execute() against it, unless you first close the > named cursor - simply because it is already in use. This is in fact why SQLAlchemy works the way it does in the first place; cursors have always to me been something that you allocate for a single statement?s result set. In the old ASP days, microsoft?s tools always handed us a scrollable, updatable cursor, in fact. PG?s named behavior seems natural to me and it is surprising to me that you?re suggesting that cursor-per-statement is a poor practice. From phd at phdru.name Sat Dec 20 19:20:19 2014 From: phd at phdru.name (Oleg Broytman) Date: Sat, 20 Dec 2014 19:20:19 +0100 Subject: [DB-SIG] SQLObject 2.0 Message-ID: <20141220182019.GA9798@phdru.name> Hello! I'm pleased to announce version 2.0.0, the first stable release of branch 2.0 of SQLObject. What's new in SQLObject ======================= Features & Interface -------------------- * DateTimeCol and TimeCol can read and write values with microseconds. WARNING: microseconds are supported by MariaDB since version 5.3.0 and by MySQL since version 5.6.4, and even these versions require special handling: columns to store microseconds have to be declared with precision 6: TIME(6), DATETIME(6), TIMESTAMP(6). SQLObject does the right thing when creating a new database but existing databases have to be changed: run something like ``ALTER TABLE name MODIFY COLUMN col TIME(6)`` for every column that you want to store microseconds. WARNING: backward compatibility problem! Date/Time columns created with microseconds cannot be read back from SQLite databases (and perhaps other backends) with versions of SQLObject older than 1.7. Minor features -------------- * PostgresConnection, when used with fromDatabase=True, sets alternateID for unique columns. * Extend sdist: include everything (even generated html) into source distribution. * Extend setup.py: include docs and tests into the egg. Development ----------- * Development was switched from Subversion to git. Documentation ------------- * Old news were restored back to version 0.2.1. * News.txt was split into 5 small files. Contributors for this release are Andrew Trusty and Jared Jennings. For a more complete list, please see the news: http://sqlobject.org/News.html What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB). Python 2.6 or 2.7 is required. Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: https://pypi.python.org/pypi/SQLObject/2.0.0 News and changes: http://sqlobject.org/News.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From songofacandy at gmail.com Thu Dec 18 12:27:05 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 18 Dec 2014 20:27:05 +0900 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5492AEDC.6060400@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> Message-ID: On Thu, Dec 18, 2014 at 7:39 PM, M.-A. Lemburg wrote: > On 17.12.2014 19:13, INADA Naoki wrote: >> As I said before, prepared statement is normally bound to connection. >> So `.prepare()` method should be connection's method, not cursor's. >> >> prepared = conn.prepare("SELECT ?+?") >> ... >> cur = conn.cursor() >> cur.execute(prepared, (1, 2)) >> cur.fetchone() # returns (3,) > > I'm not sure which database you are talking about, I'm a developer of MySQL drivers (PyMySQL and mysqlclient). prepared statement is alive as long as connection is alive. We can use one prepared statement multiple times. > but in terms > of concepts, statements are run on cursors, so adding the method > to connections doesn't look right (we dropped this logic when moving > from DB-API 1.0 to 2.0 a long time ago). PEP 249 says: > Cursor Objects > > These objects represent a database cursor, which is used to manage the context of a fetch operation. Preparing statement is not fetching query result. > > Also note that the prepare step may need access to the > cursor configuration settings to be correctly interpreted > by the database. I'm not sure which database you are talking about. MySQL has configuration per connection, not per cursor. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Dec 18 2014) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ -- INADA Naoki From songofacandy at gmail.com Thu Dec 18 14:33:19 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 18 Dec 2014 22:33:19 +0900 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5492C97E.3030303@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <5492C97E.3030303@egenix.com> Message-ID: On Thu, Dec 18, 2014 at 9:33 PM, M.-A. Lemburg wrote: > On 18.12.2014 12:27, INADA Naoki wrote: >> On Thu, Dec 18, 2014 at 7:39 PM, M.-A. Lemburg wrote: >>> On 17.12.2014 19:13, INADA Naoki wrote: >>>> As I said before, prepared statement is normally bound to connection. >>>> So `.prepare()` method should be connection's method, not cursor's. >>>> >>>> prepared = conn.prepare("SELECT ?+?") >>>> ... >>>> cur = conn.cursor() >>>> cur.execute(prepared, (1, 2)) >>>> cur.fetchone() # returns (3,) >>> >>> I'm not sure which database you are talking about, >> >> I'm a developer of MySQL drivers (PyMySQL and mysqlclient). >> prepared statement is alive as long as connection is alive. >> We can use one prepared statement multiple times. > > So MySQL separates the concepts of a prepared statement and > a statement which is used to execute a query ? I can't catch what you mean exactly. In MySQL, COM_PREPARE returns statement id. http://dev.mysql.com/doc/internals/en/com-stmt-prepare-response.html#packet-COM_STMT_PREPARE_OK Cursor's lifetime may be as short as executing one query. Typical usage is: _query = "SELECT a, b FROM tbl WHERE id=? AND k >>> but in terms >>> of concepts, statements are run on cursors, so adding the method >>> to connections doesn't look right (we dropped this logic when moving >>> from DB-API 1.0 to 2.0 a long time ago). >> >> PEP 249 says: >> >>> Cursor Objects >>> >>> These objects represent a database cursor, which is used to manage the context of a fetch operation. >> >> Preparing statement is not fetching query result. > > The DB-API is a bit terse in this respect. The two central concepts > in the DB-API are connections and cursors: > > Connections provide a connection interface to the database and > encapsulate a transactional view on database operations. > > Cursors provide a way to execute SQL statements and fetch the > corresponding data. Cursors are created on connections and > bound to these. > > Now instead of creating a third concept, that of a prepared > statement, I think it's better to stick to the above two concepts > and simply add a method to access the intermediate step of preparing > a statement on the cursor, which will then get executed on the > cursor. > OK. In case of MySQL, cursor.prepare() can proxy to connection.prepare() to reuse prepared statement over cursor lifetime. But if prepares statement is bound to cursor, why not just adding `PreparedCusor` type? I feel there are no need for providing `.prepare()` method separately. > Since the DB-API tries to provide an API which works for many > databases, the small glitch with having MySQL use a different > concept is acceptable, IMO. > >>> Also note that the prepare step may need access to the >>> cursor configuration settings to be correctly interpreted >>> by the database. >> >> I'm not sure which database you are talking about. >> MySQL has configuration per connection, not per cursor. > > My experience is from working with ODBC and ODBC drivers. > IBM DB2 and MS SQL Server use ODBC as their native database > interface API. In ODBC, cursors are called "statements" and > you have two modes of operation: > > a) direct execution, which sends the SQL straight to the > database > > b) prepare + execute, which separates the prepare from the > execute step Doesn't it support b') prepare first, and execute it multiple times after ? > > You can configure both connections and cursors in ODBC, > e.g. the cursor type setting is configured on a per > cursor basis and this has direct influence on what locking > mechanisms need to be enabled in the database to run the > SQL statement: > > http://msdn.microsoft.com/en-us/library/ms712631%28v=vs.85%29.aspx > > Note that the DB-API is modeled in many aspects after the ODBC > API and its concepts, since ODBC is an industry standard and > provides a good common denominator for how database APIs work > and which concepts they can support. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Dec 18 2014) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ -- INADA Naoki From songofacandy at gmail.com Thu Dec 18 16:48:24 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 19 Dec 2014 00:48:24 +0900 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <54927DD1.60306@users.sourceforge.net> Message-ID: > > In my experience, caching of prepared statements in modern use is not a major source of performance gain unless your application overall is already blindingly fast, more than Python typically provides. Consider that a site like reddit.com serves six billion page views a month from Postgresql databases with no use of prepared statements. When not using prepares statement, there is one roundtrip per query. When using prepared statement without caching, there are three roundtrips per query (open, execute and close). Caching is important for avoiding performance degration, not for increasing performance. -- INADA Naoki From elfring at users.sourceforge.net Thu Dec 18 20:00:38 2014 From: elfring at users.sourceforge.net (SF Markus Elfring) Date: Thu, 18 Dec 2014 20:00:38 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <1CAAB7B0-9C35-4D6C-B2C7-EEE9F24D67C6@zzzcomputing.com> References: <5484C15C.8010003@users.sourceforge.net> <1CAAB7B0-9C35-4D6C-B2C7-EEE9F24D67C6@zzzcomputing.com> Message-ID: <54932456.3080402@users.sourceforge.net> > Can someone please take up this task and prove to me with benchmarks that this effort will be worthwhile > (or at least show me what I?m going to have to put into my documentation as to when these new APIs are appropriate) ? Some database software supports the processing of prepared SQL statements by well-known application programming interfaces for decades, doesn't it? Would you like to reuse any more experiences about execution speed and program safety from software development history? Regards, Markus From mike_mp at zzzcomputing.com Sun Dec 21 19:40:08 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sun, 21 Dec 2014 13:40:08 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <5492C97E.3030303@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <5492C97E.3030303@egenix.com> Message-ID: M.-A. Lemburg wrote: > > My experience is from working with ODBC and ODBC drivers. > IBM DB2 and MS SQL Server use ODBC as their native database > interface API. In ODBC, cursors are called "statements" and > you have two modes of operation: > > a) direct execution, which sends the SQL straight to the > database > > b) prepare + execute, which separates the prepare from the > execute step > > You can configure both connections and cursors in ODBC, > e.g. the cursor type setting is configured on a per > cursor basis and this has direct influence on what locking > mechanisms need to be enabled in the database to run the > SQL statement: > > http://msdn.microsoft.com/en-us/library/ms712631%28v=vs.85%29.aspx > > Note that the DB-API is modeled in many aspects after the ODBC > API and its concepts, since ODBC is an industry standard and > provides a good common denominator for how database APIs work > and which concepts they can support. /rant-on In general, while ODBC is a good enough standard to serve as the basis for general concepts, I hope it doesn?t remain the final arbiter for most/all design decisions in the DBAPI, which I hope to see progress at some point to even a DBAPI 3. ODBC, while an industry standard, IMHO is slowly becoming a relic of the commercial closed-source software industry, of which Python itself decidedly stands in contrast towards as a key member of open source ecosystem. ODBC is of course widely used for Microsoft SQL Server, the commercial database for which it was ultimately designed for, but for the ever-more-popular open source databases such as Postgresql and MySQL, while ODBC drivers certainly exist, their use is almost unheard of in the non-Windows community. I?ve worked heavily with Oracle with Java and Perl starting in the 1990s, and even then, ODBC use on non-windows platforms was unheard of (specifically with Java we had started with the JDBC-ODBC bridge which was terribly unstable, and long ago was trumped by the type-4 drivers for JDBC/Oracle). Even the SQL Server and DB2 shops that I work with, again because they are hosting their applications from a linux platform, don?t use ODBC either, on Python they instead favor the pymssql driver and IBM?s own non-ODBC-based IBM-DB driver (https://code.google.com/p/ibm-db/wiki/README). ODBC is definitely a standard, but in my experience it?s a stogy old standard created by and dominated by closed source solutions, and is generally the last choice of the open source developer as specific open source drivers are invariably always very difficult and buggy to deal with, even hiding behind the DBAPI. IMO it does tend to expose a little too much of underlying mechanics, as it is an API that is largely a product of the 80?s and 90?s before higher-level VM-based and/or scripting languages had achieved wide dominance. To the degree that the DBAPI has already drawn upon it is fine, but I would be concerned if the DBAPI is to continue to attempt to mold itself as an ongoing ODBC-compatibility layer going forward. /rant-off However, getting back to the issue of prepared statements, I will note that specific to MySQL, MySQL?s own DBAPI driver, MySQL-connector-Python, does in fact tie the prepared statement to the cursor: http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html, most expediently as a flag to the connection.cursor() call, ?cursor = connection.cursor(prepared=True)?, which is a shortcut for using their CursorPrepared cursor class. I actually find MySQL-connector?s API quite appealing here: "The first time you pass a statement to the cursor's execute() method, it prepares the statement. For subsequent invocations of execute(), the preparation phase is skipped if the statement is the same.? That is, the cursor establishes that it will use prepared statements in the background, and as long as you pass the same string each time, it uses the same prepared statement. The moment you send a different string, now a new prepared statement is produced which is used specific to that string. This eliminates the need for an awkward ?prepare()? step and the introduction of new and potentially more awkward APIs. Does that appeal to anyone else here? From mal at egenix.com Sun Dec 21 20:10:39 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 21 Dec 2014 20:10:39 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <5492C97E.3030303@egenix.com> Message-ID: <54971B2F.2020000@egenix.com> On 21.12.2014 19:40, Michael Bayer wrote: > > > M.-A. Lemburg wrote: > >> >> My experience is from working with ODBC and ODBC drivers. >> IBM DB2 and MS SQL Server use ODBC as their native database >> interface API. In ODBC, cursors are called "statements" and >> you have two modes of operation: >> >> a) direct execution, which sends the SQL straight to the >> database >> >> b) prepare + execute, which separates the prepare from the >> execute step >> >> You can configure both connections and cursors in ODBC, >> e.g. the cursor type setting is configured on a per >> cursor basis and this has direct influence on what locking >> mechanisms need to be enabled in the database to run the >> SQL statement: >> >> http://msdn.microsoft.com/en-us/library/ms712631%28v=vs.85%29.aspx >> >> Note that the DB-API is modeled in many aspects after the ODBC >> API and its concepts, since ODBC is an industry standard and >> provides a good common denominator for how database APIs work >> and which concepts they can support. > > /rant-on > > In general, while ODBC is a good enough standard to serve as the basis for > general concepts, I hope it doesn?t remain the final arbiter for most/all > design decisions in the DBAPI, which I hope to see progress at some point to > even a DBAPI 3. ODBC, while an industry standard, IMHO is slowly becoming a > relic of the commercial closed-source software industry, of which Python > itself decidedly stands in contrast towards as a key member of open source > ecosystem. ODBC is of course widely used for Microsoft SQL Server, the > commercial database for which it was ultimately designed for, but for the > ever-more-popular open source databases such as Postgresql and MySQL, while > ODBC drivers certainly exist, their use is almost unheard of in the > non-Windows community. I?ve worked heavily with Oracle with Java and Perl > starting in the 1990s, and even then, ODBC use on non-windows platforms was > unheard of (specifically with Java we had started with the JDBC-ODBC bridge > which was terribly unstable, and long ago was trumped by the type-4 drivers > for JDBC/Oracle). Even the SQL Server and DB2 shops that I work with, again > because they are hosting their applications from a linux platform, don?t use > ODBC either, on Python they instead favor the pymssql driver and IBM?s own > non-ODBC-based IBM-DB driver (https://code.google.com/p/ibm-db/wiki/README). > ODBC is definitely a standard, but in my experience it?s a stogy old > standard created by and dominated by closed source solutions, and is > generally the last choice of the open source developer as specific open > source drivers are invariably always very difficult and buggy to deal with, > even hiding behind the DBAPI. IMO it does tend to expose a little too much > of underlying mechanics, as it is an API that is largely a product of the > 80?s and 90?s before higher-level VM-based and/or scripting languages had > achieved wide dominance. To the degree that the DBAPI has already drawn upon > it is fine, but I would be concerned if the DBAPI is to continue to attempt > to mold itself as an ongoing ODBC-compatibility layer going forward. > > /rant-off Just to clarify: both MS SQL Server and IBM DB2 use ODBC as their native database interface C API. They have both extended it to include some special features which are not part of the ODBC standard, but in way that's compatible with ODBC. Just look at the source code. So while you may not see that you're using ODBC, you in fact are under the hood :-) The only difference is that these database modules are directly linked against the driver libraries. We did that as well in subpackages of older versions of mxODBC, but found that getting things to work in such a direct setup is much more difficult (for our users) than using an ODBC manager in between and registering the drivers with this subsystem. The ODBC manager also makes it possible to plug in different drivers easily without having to change the application, and it can provide transparent connection pooling, so you stay flexible. > However, getting back to the issue of prepared statements, I will note that > specific to MySQL, MySQL?s own DBAPI driver, MySQL-connector-Python, does in > fact tie the prepared statement to the cursor: > http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html, > most expediently as a flag to the connection.cursor() call, ?cursor = > connection.cursor(prepared=True)?, which is a shortcut for using their > CursorPrepared cursor class. > > I actually find MySQL-connector?s API quite appealing here: "The first time > you pass a statement to the cursor's execute() method, it prepares the > statement. For subsequent invocations of execute(), the preparation phase is > skipped if the statement is the same.? That's a standard DB-API optimization feature, not special to MySQL. > That is, the cursor establishes that it will use prepared statements in the > background, and as long as you pass the same string each time, it uses the > same prepared statement. The moment you send a different string, now a new > prepared statement is produced which is used specific to that string. This > eliminates the need for an awkward ?prepare()? step and the introduction of > new and potentially more awkward APIs. Does that appeal to anyone else here? Michael, I think you've misunderstood the proposal. The idea is to expose the prepare step which normally happens implicitly, as an explicit optional intermediate step. The main DB-API functionality is not affected by the proposal. We're only trying to find a standard for database modules to adapt to, which are already exposing the prepare step. Just like we've done several times in the past with other extensions which several modules implemented in sometimes slightly different ways. The aim here is to prevent different semantics from causing surprises with the database module users. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 21 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Sun Dec 21 20:43:09 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 21 Dec 2014 20:43:09 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> <54935B76.5070805@egenix.com> Message-ID: <549722CD.3090907@egenix.com> On 19.12.2014 00:12, Michael Bayer wrote: > M.-A. Lemburg wrote: >> I think there's a misunderstanding here. A named cursor refers to >> a database cursor pointing into a result set. Once that result >> set has been created and the cursor points to it, you cannot >> run another .execute() against it, unless you first close the >> named cursor - simply because it is already in use. > > This is in fact why SQLAlchemy works the way it does in the first place; > cursors have always to me been something that you allocate for a single > statement?s result set. In the old ASP days, microsoft?s tools always handed > us a scrollable, updatable cursor, in fact. PG?s named behavior seems > natural to me and it is surprising to me that you?re suggesting that > cursor-per-statement is a poor practice. I'm not saying that having a cursor per statement execution is poor practice, but it's worth considering using cursors for multiple statement execution to reduce overhead. Note that you have to differentiate between client and server side cursors. Named cursors are normally server side, whereas the anonymous cursors you typically create using the DB-API are often client side. A server side cursor keeps most of the resources on the server, e.g. it only transfers data to the client when requested. A client side cursor puts the resources mostly on the client side, e.g. the database will try to send the complete result set to the client in one go to allow for fast scrolling through the result set. Both have advantages and disadvantages, e.g. http://msdn.microsoft.com/en-us/library/aa266531%28v=vs.60%29.aspx Whether or not a cursor is client or server side depends on many things and is generally database backend specific. MS SQL Server for example will default to server side cursors and revert to client side for some special cases: http://msdn.microsoft.com/en-us/library/ms131331.aspx -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 21 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Sun Dec 21 20:46:19 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sun, 21 Dec 2014 14:46:19 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54971B2F.2020000@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <5492C97E.3030303@egenix.com> <54971B2F.2020000@egenix.com> Message-ID: <7A025CD4-8258-4F45-85D3-6615B9E3F9E4@zzzcomputing.com> M.-A. Lemburg wrote: > > Just to clarify: both MS SQL Server and IBM DB2 use ODBC as their > native database interface C API. > > They have both extended it to include some special features which > are not part of the ODBC standard, but in way that's compatible > with ODBC. Just look at the source code. > > So while you may not see that you're using ODBC, you in fact > are under the hood :-) > > The only difference is that these database modules are directly > linked against the driver libraries. We did that as well in > subpackages of older versions of mxODBC, but found that getting > things to work in such a direct setup is much more difficult (for our > users) than using an ODBC manager in between and registering the > drivers with this subsystem. > > The ODBC manager also makes it possible to plug in different > drivers easily without having to change the application, and > it can provide transparent connection pooling, so you stay > flexible. I understand the rationale for the ODBC layer of abstraction, but in practice, this is too many layers of abstraction; a driver like pymssql can essentially ?curate? all the quirks and surprises that one has to deal with explicitly when using multiple generic ODBC connection layers. Switching between FreeTDS on Unix and the Microsoft native drivers on windows for the same SQL Server database, pyodbc acts *completely* differently and far worse on linux and even more poorly on OSX. My experience of ODBC includes much plugging of drivers for sure, but in no case has it ever been ?easy? if one isn?t on Microsoft Windows. That ODBC serves as the native C API for SQL Server and DB2 only supports the assertion that ODBC is very detailed and close to the implementation details. > >> However, getting back to the issue of prepared statements, I will note that >> specific to MySQL, MySQL?s own DBAPI driver, MySQL-connector-Python, does in >> fact tie the prepared statement to the cursor: >> http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html, >> most expediently as a flag to the connection.cursor() call, ?cursor = >> connection.cursor(prepared=True)?, which is a shortcut for using their >> CursorPrepared cursor class. >> >> I actually find MySQL-connector?s API quite appealing here: "The first time >> you pass a statement to the cursor's execute() method, it prepares the >> statement. For subsequent invocations of execute(), the preparation phase is >> skipped if the statement is the same.? > > That's a standard DB-API optimization feature, not special to MySQL. I see no mention of the ?prepare? keyword at https://www.python.org/dev/peps/pep-0249/#cursor ? > > Michael, I think you've misunderstood the proposal. The idea is > to expose the prepare step which normally happens implicitly, > as an explicit optional intermediate step. The main DB-API > functionality is not affected by the proposal. > > We're only trying to find a standard for database modules to adapt > to, which are already exposing the prepare step. Just like we've > done several times in the past with other extensions which > several modules implemented in sometimes slightly different > ways. Right. I?m referring to the explicit API of MySQL-connector which introduces a ?prepare? keyword argument to connection.cursor() as one possible option, one that already works well without the need for extra method calls and bigger changes in use. > > The aim here is to prevent different semantics from causing > surprises with the database module users. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Dec 21 2014) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Sun Dec 21 20:56:24 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sun, 21 Dec 2014 14:56:24 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <549722CD.3090907@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> <54935B76.5070805@egenix.com> <549722CD.3090907@egenix.com> Message-ID: <77ABE69F-CBCF-4DB5-9CDB-D2F39388CE7F@zzzcomputing.com> M.-A. Lemburg wrote: > > Whether or not a cursor is client or server side depends on many > things and is generally database backend specific. MS SQL Server > for example will default to server side cursors and revert to > client side for some special cases: > > http://msdn.microsoft.com/en-us/library/ms131331.aspx this is why I?m so troubled by the concept that a user of the DBAPI needs to be aware of this backend-specific issue as a guide as to whether or not the complicated and itself-expensive pooling of cursors on the application side is necessary in order to gain performance (performance gains that need to be greater than the performance lost by the complexity of pooling the cursors behind some facade); not just at the per-database level, but at the use-case level, as you state here with SQL Server?s driver choosing server- or client- side cursors based on different conditions. I would much prefer that DBAPIs handle the optimization of this underlying backend detail for us. It is too low-level an issue to be exposed in a Python API as a normal matter of course. One of the jobs of SQLAlchemy is to try to abstract over issues like this in the DBAPI, as the DBAPI makes so many of them apparent; the six parameter formats, the totally idiosyncratic behavior of unicode objects, the many ways to set up isolation levels, the many ways to set up autocommit, the great lack of common, specific datatypes, etc. I?d like there to be a DBAPI someday that tries to reduce the number of idiosyncrasies exposed in favor of the drivers themselves using standard, documented techniques to handle them behind the scenes. > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Dec 21 2014) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mal at egenix.com Sun Dec 21 20:59:33 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 21 Dec 2014 20:59:33 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <7A025CD4-8258-4F45-85D3-6615B9E3F9E4@zzzcomputing.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <5492C97E.3030303@egenix.com> <54971B2F.2020000@egenix.com> <7A025CD4-8258-4F45-85D3-6615B9E3F9E4@zzzcomputing.com> Message-ID: <549726A5.6080103@egenix.com> On 21.12.2014 20:46, Michael Bayer wrote: > M.-A. Lemburg wrote: >>> However, getting back to the issue of prepared statements, I will note that >>> specific to MySQL, MySQL?s own DBAPI driver, MySQL-connector-Python, does in >>> fact tie the prepared statement to the cursor: >>> http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html, >>> most expediently as a flag to the connection.cursor() call, ?cursor = >>> connection.cursor(prepared=True)?, which is a shortcut for using their >>> CursorPrepared cursor class. >>> >>> I actually find MySQL-connector?s API quite appealing here: "The first time >>> you pass a statement to the cursor's execute() method, it prepares the >>> statement. For subsequent invocations of execute(), the preparation phase is >>> skipped if the statement is the same.? >> >> That's a standard DB-API optimization feature, not special to MySQL. > > I see no mention of the ?prepare? keyword at https://www.python.org/dev/peps/pep-0249/#cursor ? I was referring to this part: >>> "The first time you pass a statement to the cursor's execute() method, it prepares the >>> statement. For subsequent invocations of execute(), the preparation phase is >>> skipped if the statement is the same.? This is the same as standard DB-API: c = connection.cursor() stmt = 'select * from mytable where id = ?' c.execute(stmt, [8989]) result = c.fetchone() c.execute(stmt, [1212]) result = c.fetchone() The first execute will prepare and run the stmt. The second will reuse the already perpared stmt and simply run it against a different set of parameters. >> Michael, I think you've misunderstood the proposal. The idea is >> to expose the prepare step which normally happens implicitly, >> as an explicit optional intermediate step. The main DB-API >> functionality is not affected by the proposal. >> >> We're only trying to find a standard for database modules to adapt >> to, which are already exposing the prepare step. Just like we've >> done several times in the past with other extensions which >> several modules implemented in sometimes slightly different >> ways. > > Right. I?m referring to the explicit API of MySQL-connector which introduces a ?prepare? keyword argument to connection.cursor() as one possible option, one that already works well without the need for extra method calls and bigger changes in use. >From what I read, the keyword parameter merely enables the above optimization. It also doesn't provide a way to prepare a statement without executing it and again, we're not trying to make things more complicated, only provide access to intermediate implicit steps. >> The aim here is to prevent different semantics from causing >> surprises with the database module users. I would like to hear back from some other module authors, esp. those who are not targeting MySQL/MariaDB, since we've already had lots of feedback for those databases. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 21 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Sun Dec 21 21:17:23 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 21 Dec 2014 21:17:23 +0100 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <77ABE69F-CBCF-4DB5-9CDB-D2F39388CE7F@zzzcomputing.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> <54935B76.5070805@egenix.com> <549722CD.3090907@egenix.com> <77ABE69F-CBCF-4DB5-9CDB-D2F39388CE7F@zzzcomputing.com> Message-ID: <54972AD3.2040609@egenix.com> On 21.12.2014 20:56, Michael Bayer wrote: > > > M.-A. Lemburg wrote: > >> >> Whether or not a cursor is client or server side depends on many >> things and is generally database backend specific. MS SQL Server >> for example will default to server side cursors and revert to >> client side for some special cases: >> >> http://msdn.microsoft.com/en-us/library/ms131331.aspx > > this is why I?m so troubled by the concept that a user of the DBAPI needs to > be aware of this backend-specific issue as a guide as to whether or not the > complicated and itself-expensive pooling of cursors on the application side > is necessary in order to gain performance (performance gains that need to be > greater than the performance lost by the complexity of pooling the cursors > behind some facade); not just at the per-database level, but at the use-case > level, as you state here with SQL Server?s driver choosing server- or > client- side cursors based on different conditions. I would much prefer that > DBAPIs handle the optimization of this underlying backend detail for us. It > is too low-level an issue to be exposed in a Python API as a normal matter > of course. This is hardly something a Python DB-API module could manage or abstract away. As you can see in the above article, the database itself decides which variant to use. AFAIK, the driver does not provide a way to tell the database to use one or the other. > One of the jobs of SQLAlchemy is to try to abstract over issues like this in > the DBAPI, as the DBAPI makes so many of them apparent; the six parameter > formats, the totally idiosyncratic behavior of unicode objects, the many > ways to set up isolation levels, the many ways to set up autocommit, the > great lack of common, specific datatypes, etc. I?d like there to be a DBAPI > someday that tries to reduce the number of idiosyncrasies exposed in favor > of the drivers themselves using standard, documented techniques to handle > them behind the scenes. Some of those things should be standardized, yes, but please do note that the DB-API is still meant as API guideline, which is meant to be adaptable to database features. Database module authors are free to add features to their module. We'll then have a look every now and then and if several modules implement similar things, discuss a standard way of doing this. Some things from your list which we should tackle in the coming months (using new threads :-)): * standard way of enabling/disabled auto-commit * standard way to set transaction isolation level * reduce the number of parameter formats * standard way to support Unicode for parameters and statements I think the first two can be done as standard optional DB-API extensions. The last two will likely have to put into DB-API 3.0, since they may require changes to database module default behavior or additional support. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 21 2014) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-12-11: Released mxODBC Plone/Zope DA 2.2.0 http://egenix.com/go67 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mike_mp at zzzcomputing.com Mon Dec 22 06:59:13 2014 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 22 Dec 2014 00:59:13 -0500 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: <54972AD3.2040609@egenix.com> References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <54933B91.9030408@egenix.com> <6D66F4CF-8E99-41B4-8E86-4BA7A9889976@zzzcomputing.com> <54935B76.5070805@egenix.com> <549722CD.3090907@egenix.com> <77ABE69F-CBCF-4DB5-9CDB-D2F39388CE7F@zzzcomputing.com> <54972AD3.2040609@egenix.com> Message-ID: <91CC1856-6AFE-4A37-A271-77B6FF88E8B1@zzzcomputing.com> M.-A. Lemburg wrote: > On 21.12.2014 20:56, Michael Bayer wrote: >> M.-A. Lemburg wrote: >> >>> Whether or not a cursor is client or server side depends on many >>> things and is generally database backend specific. MS SQL Server >>> for example will default to server side cursors and revert to >>> client side for some special cases: >>> >>> http://msdn.microsoft.com/en-us/library/ms131331.aspx >> >> this is why I?m so troubled by the concept that a user of the DBAPI needs to >> be aware of this backend-specific issue as a guide as to whether or not the >> complicated and itself-expensive pooling of cursors on the application side >> is necessary in order to gain performance (performance gains that need to be >> greater than the performance lost by the complexity of pooling the cursors >> behind some facade); not just at the per-database level, but at the use-case >> level, as you state here with SQL Server?s driver choosing server- or >> client- side cursors based on different conditions. I would much prefer that >> DBAPIs handle the optimization of this underlying backend detail for us. It >> is too low-level an issue to be exposed in a Python API as a normal matter >> of course. > > This is hardly something a Python DB-API module could manage or > abstract away. As you can see in the above article, the database itself > decides which variant to use. AFAIK, the driver does not provide a way to > tell the database to use one or the other. In practice, this aspect of the driver is considered by most to be an implementation detail that people usually don?t want to have to deal with except in optimization scenarios. The example of SQL Server here is an outlier; most of the DBAPIs use only client- or server- side cursors up front and that?s it. JDBC exposes this concept via the Statement/PreparedStatement->ResultSet objects. They don?t call any of these things a ?cursor?, even though in practice you?d probably say (I think you did say this earlier) that Statement->ResultSet or PreparedStatement->ResultSet is really the same as a ?cursor? (so in that sense I will concede that since DBAPI has an explicit cursor already, that is obviously where the ?prepared? statement has to start). Though I like that JDBC changed the names and made roles more agnostic and detached from the underlying details, leaving it up to the driver as to how things will run behind the scenes. Nobody is worried about re-using cursors or not with JDBC as they are not directly exposed, they are perhaps worried about re-using PreparedStatement objects, though in practice I think this is not typical. The issue of whether things are invoked server or client side is not exposed very directly either, it is somewhat exposed though the setFetchSize() method. What I liked about JDBC is that when using it, we aren?t nearly as often making these driver-specific decisions in application code all the time; JDBC?s exposure of PreparedStatement while not exposing ?cursor? directly seemed to hit this ?sweet spot? nicely. The ?cursor? in DBAPI seems a little awkward, in that as we talk about it, we continuously refer to the very specific cursor behaviors of ODBC, which is an old and very finely detailed low level API, as rationales for some of its odder corners, yet in pep-249 itself I see even a note that the entire ?cursor? concept may have to be emulated for some backends that don?t have a real cursor. I?ve no doubt that DBAPI 3 is still going to have a cursor front-and-center. But I?d at least like it if in a DBAPI 3 that these concepts of server and client side cursors as well as that of prepared statements can be discussed and exposed in much greater and consistent detail, from the perspective of which should be used by implementors and when, and how the presence of these behaviors and capabilities can be exposed in a consistent way to consuming applications. As it stands, DBAPIs generally choose one or the other, rarely provide both, and hardly (with notable exceptions) make it at all clear which one it actually is, yet I?m tasked with having to know when the ?cursor? resource is either practically free or significantly expensive, whether the object is reusable or whether it isn?t. From tlocke at tlocke.org.uk Sun Dec 21 20:05:55 2014 From: tlocke at tlocke.org.uk (Tony Locke) Date: Sun, 21 Dec 2014 19:05:55 +0000 Subject: [DB-SIG] Improved support for prepared SQL statements In-Reply-To: References: <5484C15C.8010003@users.sourceforge.net> <5486C39C.2040608@egenix.com> <54918596.1000807@egenix.com> <54918A37.2040001@users.sourceforge.net> <54919177.70205@egenix.com> <5491956D.9000503@users.sourceforge.net> <54919B2D.7010306@egenix.com> <54919EC2.7040007@users.sourceforge.net> <5491A418.4030300@egenix.com> <5492AEDC.6060400@egenix.com> <5492C97E.3030303@egenix.com> Message-ID: Mike, I agree. You quote: "The first time you pass a statement to the cursor's execute() method, it prepares the statement. For subsequent invocations of execute(), the preparation phase is skipped if the statement is the same.? This is basically what pg8000 does but I've found you need a couple of extra rules in practice: * Any statements changing the structure of the database should cause the cache of prepared statements to be invalidated. * The key for the cache needs to include the parameter types as well as the query. Regards, Tony. On 21 Dec 2014 18:40, "Michael Bayer" wrote: > > > M.-A. Lemburg wrote: > > > > > My experience is from working with ODBC and ODBC drivers. > > IBM DB2 and MS SQL Server use ODBC as their native database > > interface API. In ODBC, cursors are called "statements" and > > you have two modes of operation: > > > > a) direct execution, which sends the SQL straight to the > > database > > > > b) prepare + execute, which separates the prepare from the > > execute step > > > > You can configure both connections and cursors in ODBC, > > e.g. the cursor type setting is configured on a per > > cursor basis and this has direct influence on what locking > > mechanisms need to be enabled in the database to run the > > SQL statement: > > > > http://msdn.microsoft.com/en-us/library/ms712631%28v=vs.85%29.aspx > > > > Note that the DB-API is modeled in many aspects after the ODBC > > API and its concepts, since ODBC is an industry standard and > > provides a good common denominator for how database APIs work > > and which concepts they can support. > > /rant-on > > In general, while ODBC is a good enough standard to serve as the basis for > general concepts, I hope it doesn?t remain the final arbiter for most/all > design decisions in the DBAPI, which I hope to see progress at some point > to > even a DBAPI 3. ODBC, while an industry standard, IMHO is slowly becoming a > relic of the commercial closed-source software industry, of which Python > itself decidedly stands in contrast towards as a key member of open source > ecosystem. ODBC is of course widely used for Microsoft SQL Server, the > commercial database for which it was ultimately designed for, but for the > ever-more-popular open source databases such as Postgresql and MySQL, while > ODBC drivers certainly exist, their use is almost unheard of in the > non-Windows community. I?ve worked heavily with Oracle with Java and Perl > starting in the 1990s, and even then, ODBC use on non-windows platforms was > unheard of (specifically with Java we had started with the JDBC-ODBC bridge > which was terribly unstable, and long ago was trumped by the type-4 drivers > for JDBC/Oracle). Even the SQL Server and DB2 shops that I work with, again > because they are hosting their applications from a linux platform, don?t > use > ODBC either, on Python they instead favor the pymssql driver and IBM?s own > non-ODBC-based IBM-DB driver (https://code.google.com/p/ibm-db/wiki/README > ). > ODBC is definitely a standard, but in my experience it?s a stogy old > standard created by and dominated by closed source solutions, and is > generally the last choice of the open source developer as specific open > source drivers are invariably always very difficult and buggy to deal with, > even hiding behind the DBAPI. IMO it does tend to expose a little too much > of underlying mechanics, as it is an API that is largely a product of the > 80?s and 90?s before higher-level VM-based and/or scripting languages had > achieved wide dominance. To the degree that the DBAPI has already drawn > upon > it is fine, but I would be concerned if the DBAPI is to continue to attempt > to mold itself as an ongoing ODBC-compatibility layer going forward. > > /rant-off > > However, getting back to the issue of prepared statements, I will note that > specific to MySQL, MySQL?s own DBAPI driver, MySQL-connector-Python, does > in > fact tie the prepared statement to the cursor: > > http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html > , > most expediently as a flag to the connection.cursor() call, ?cursor = > connection.cursor(prepared=True)?, which is a shortcut for using their > CursorPrepared cursor class. > > I actually find MySQL-connector?s API quite appealing here: "The first time > you pass a statement to the cursor's execute() method, it prepares the > statement. For subsequent invocations of execute(), the preparation phase > is > skipped if the statement is the same.? > > That is, the cursor establishes that it will use prepared statements in the > background, and as long as you pass the same string each time, it uses the > same prepared statement. The moment you send a different string, now a new > prepared statement is produced which is used specific to that string. This > eliminates the need for an awkward ?prepare()? step and the introduction of > new and potentially more awkward APIs. Does that appeal to anyone else > here? > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dieter at handshake.de Sat Dec 27 18:37:39 2014 From: dieter at handshake.de (Dieter Maurer) Date: Sat, 27 Dec 2014 17:37:39 -0000 Subject: [DB-SIG] DBAPI 3.0: Unified parameter style In-Reply-To: References: Message-ID: <21046.55895.840406.403600@localhost.localdomain> Tony Locke wrote at 2013-9-13 18:54 +0100: > ... >I thought I'd give a suggestion for unified parameters. Apologies if >this has been suggested before, I did have a look at the archives :-) > >In DBAPI 2.0, the parameters to execute() can be a sequence or a >mapping. In DBAPI 3.0, the paramstyle attribute should be removed, and >the style determined by whether the parameters are a sequence or >mapping. For a sequence the 'numeric' style is expected, and for a >mapping the 'named' style. For example: > >execute("select * from emp where name = :1", ['horatio']) > >execute("select * from emp where name = :name", {'name': 'horatio'}) > >The reason I like this solution is that if someone asked me how >parameters work in DBAPI, this one would be the easiest to explain! > >Any thoughts? This has been suggested and found one objection: there is no reliable way to distinguish a mapping from a sequence -- or put in different words: it is easy to define a class which supports both the sequence as well as the mapping protocol. -- Dieter From dieter at handshake.de Sat Dec 27 19:59:40 2014 From: dieter at handshake.de (Dieter Maurer) Date: Sat, 27 Dec 2014 18:59:40 -0000 Subject: [DB-SIG] API 3.0 limiting paramstyle to ['named', 'qmark'] is okay. ('format' is not desirable) In-Reply-To: References: <519609A8.6080407@egenix.com> <2F26E0FF-18A6-4D57-B0F1-B4930FD9C0CF@zzzcomputing.com> <51964872.30603@online.de> Message-ID: <20886.29152.64121.627087@localhost.localdomain> Daniele Varrazzo wrote at 2013-5-17 16:33 +0100: > ... >Please, to everybody involved in this discussion: stop thinking that >giving more choices makes people's life easier. No: it makes it more >complicated, whatever the person in question is a database driver >author, a single database user or a multi-library database author. I have thousands of SQL statements using "params" style. I would not be happy if I would need to change them all for cosmetics sake. -- Dieter From vernondcole at gmail.com Sun Dec 28 16:55:22 2014 From: vernondcole at gmail.com (Vernon D. Cole) Date: Sun, 28 Dec 2014 08:55:22 -0700 Subject: [DB-SIG] DBAPI 3.0: Unified parameter style In-Reply-To: <21046.55895.840406.403600@localhost.localdomain> References: <21046.55895.840406.403600@localhost.localdomain> Message-ID: It happens that the statement "there is no reliable way to distinguish a mapping from a sequence" is not true. I submit the following example. Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] > on win > 32 > Type "help", "copyright", "credits" or "license" for more information. > >>> from collections import Mapping > >>> seq = [1, 2] > >>> map = {'one':1, 'two':2} > >>> params = seq > >>> isinstance(params, Mapping) > False > >>> params = map > >>> isinstance(params, Mapping) > True > >>> > The current production version of adodbapi does exactly what Tony proposes, if the programmer sets the paramstye to "dynamic". Yes, it does work reliably. I challenge anyone who claims otherwise to prove their assertion. On Mon, Sep 16, 2013 at 4:15 AM, Dieter Maurer wrote: > Tony Locke wrote at 2013-9-13 18:54 +0100: > > ... > >I thought I'd give a suggestion for unified parameters. Apologies if > >this has been suggested before, I did have a look at the archives :-) > > > >In DBAPI 2.0, the parameters to execute() can be a sequence or a > >mapping. In DBAPI 3.0, the paramstyle attribute should be removed, and > >the style determined by whether the parameters are a sequence or > >mapping. For a sequence the 'numeric' style is expected, and for a > >mapping the 'named' style. For example: > > > >execute("select * from emp where name = :1", ['horatio']) > > > >execute("select * from emp where name = :name", {'name': 'horatio'}) > > > >The reason I like this solution is that if someone asked me how > >parameters work in DBAPI, this one would be the easiest to explain! > > > >Any thoughts? > > This has been suggested and found one objection: there is no reliable > way to distinguish a mapping from a sequence -- or put in > different words: it is easy to define a class which supports both > the sequence as well as the mapping protocol. > > > > -- > Dieter > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: