From nbhamady at gmail.com Thu Oct 11 14:48:51 2018 From: nbhamady at gmail.com (Nader Hamady) Date: Thu, 11 Oct 2018 14:48:51 -0400 Subject: [DB-SIG] PEP 249 Message-ID: Hello. Reaching out to find out if there have been any discusdions around a 3.0 Database API Specification - Or something broader around async specifications for dabatase transactions. Thanks in advance, -N -- Nader B. Hamady -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tom.Cornell at morganstanley.com Thu Oct 11 16:03:05 2018 From: Tom.Cornell at morganstanley.com (Cornell, Tom) Date: Thu, 11 Oct 2018 20:03:05 +0000 Subject: [DB-SIG] PEP 249 implementation Message-ID: Hi, I was curious if there were any implementations of PEP 249 Python Database API v2.0 that I can use as a reference. I'd like to try and implement it for the DBMS we use here at work (Sybase, DB2, Postgres). I will happily donate them to PSF, once cleared by my company. TC Tom Cornell, Vice President Morgan Stanley | Corporate & Funding Technology 750 Seventh Avenue, 20th Floor | New York, NY 10019 Phone: +1 212 762-1619 Tom.Cornell at morganstanley.com Be carbon conscious. Please consider our environment before printing this email. ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent required and/or permitted under applicable law, to monitor electronic communications, including telephone calls with Morgan Stanley personnel. This message is subject to the Morgan Stanley General Disclaimers available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access the links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you acknowledge that you have read, understand and consent, (where applicable), to the foregoing and the Morgan Stanley General Disclaimers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Oct 16 06:00:44 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 16 Oct 2018 13:00:44 +0300 Subject: [DB-SIG] PEP 249 In-Reply-To: References: Message-ID: Hi Nadar, we do have a set of things scheduled for a version 3.0 of the DB API (see the python.org wiki), but nothing related to async I/O yet. FWIW: I think the existing APIs can be used in an async form as well, without having to specify new ones, by simply providing an async entry point at the module level or even making the DB module itself be async only. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ On 11.10.2018 21:48, Nader Hamady wrote: > Hello. > > Reaching out to find out if there have been any discusdions around a 3.0 > Database API Specification - Or something broader around async > specifications for dabatase transactions. > > Thanks in advance, > > -N > > > > Hello. > > Reaching out to find out if there have been any discusdions around a 3.0 > Database API Specification - Or something broader around async > specifications for dabatase transactions. > > Thanks in advance, > > -N > ?? > -- > Nader B. Hamady > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > From mal at egenix.com Tue Oct 16 06:02:38 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 16 Oct 2018 13:02:38 +0300 Subject: [DB-SIG] PEP 249 implementation In-Reply-To: References: Message-ID: <449c1aa1-9e36-8e09-9d22-24e0c58af750@egenix.com> Hi Tom, there are already a number of standard DB-API compatible modules for Sybase, DB2 and PostgreSQL which you can readily use. Please check the python.org wiki for links: https://wiki.python.org/moin/DatabaseInterfaces Regarding donations to the PSF, please contact psf at python.org for details. Many thanks. Best Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ On 11.10.2018 23:03, Cornell, Tom wrote: > Hi, > > I was curious if there were any implementations of PEP 249 Python Database API v2.0 that I can use as a reference. I'd like to try and implement it for the DBMS we use here at work (Sybase, DB2, Postgres). I will happily donate them to PSF, once cleared by my company. > > TC > > Tom Cornell, Vice President > Morgan Stanley | Corporate & Funding Technology > 750 Seventh Avenue, 20th Floor | New York, NY 10019 > Phone: +1 212 762-1619 > Tom.Cornell at morganstanley.com > > > Be carbon conscious. Please consider our environment before printing this email. > > > ________________________________ > NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent required and/or permitted under applicable law, to monitor electronic communications, including telephone calls with Morgan Stanley personnel. This message is subject to the Morgan Stanley General Disclaimers available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access the links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you acknowledge that you have read, understand and consent, (where applicable), to the foregoing and the Morgan Stanley General Disclaimers. > > > > Hi, > > ? > > I was curious if there were any implementations of PEP 249 Python > Database API v2.0 ?that I can use as a reference.? I?d like to try and > implement it for the ?DBMS we use here at work (Sybase, DB2, Postgres).? > I will happily donate them to PSF, once cleared by my company. > > ? > > TC > > ? > > Tom Cornell, Vice President??? > *Morgan Stanley | Corporate & Funding Technology??? > *750 Seventh Avenue, 20th Floor | New York, NY??10019??? > Phone: +1 212 762-1619??? > Tom.Cornell at morganstanley.com ??? > ??? > ??? > Be carbon conscious. Please consider our environment before printing > this email. ??? > ??? > > > ------------------------------------------------------------------------ > NOTICE: Morgan Stanley is not acting as a municipal advisor and the > opinions or views contained herein are not intended to be, and do not > constitute, advice within the meaning of Section 975 of the Dodd-Frank > Wall Street Reform and Consumer Protection Act. If you have received > this communication in error, please destroy all electronic and paper > copies and notify the sender immediately. Mistransmission is not > intended to waive confidentiality or privilege. Morgan Stanley reserves > the right, to the extent required and/or permitted under applicable law, > to monitor electronic communications, including telephone calls with > Morgan Stanley personnel. This message is subject to the Morgan Stanley > General Disclaimers available at the following link: > http://www.morganstanley.com/disclaimers.? If you cannot access the > links, please notify us by reply message and we will send the contents > to you. By communicating with Morgan Stanley you acknowledge that you > have read, understand and consent, (where applicable), to the foregoing > and the Morgan Stanley General Disclaimers. > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > From mal at egenix.com Tue Oct 16 06:09:39 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 16 Oct 2018 13:09:39 +0300 Subject: [DB-SIG] PEP 249 In-Reply-To: References: Message-ID: On 16.10.2018 13:00, M.-A. Lemburg wrote: > Hi Nadar, > > we do have a set of things scheduled for a version 3.0 of > the DB API (see the python.org wiki), but nothing related to > async I/O yet. https://wiki.python.org/moin/DbApi3 > FWIW: I think the existing APIs can be used in an async form > as well, without having to specify new ones, by simply > providing an async entry point at the module level or > even making the DB module itself be async only. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mike_mp at zzzcomputing.com Tue Oct 16 09:37:32 2018 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Tue, 16 Oct 2018 09:37:32 -0400 Subject: [DB-SIG] PEP 249 In-Reply-To: References: Message-ID: AFAIK , Postgresql is the only major relational database that has an actual non-blocking protocol. The other databases all require that "async" be simulated in the context of blocking IO. On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg wrote: > > Hi Nadar, > > we do have a set of things scheduled for a version 3.0 of > the DB API (see the python.org wiki), but nothing related to > async I/O yet. > > FWIW: I think the existing APIs can be used in an async form > as well, without having to specify new ones, by simply > providing an async entry point at the module level or > even making the DB module itself be async only. > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > > > On 11.10.2018 21:48, Nader Hamady wrote: > > Hello. > > > > Reaching out to find out if there have been any discusdions around a 3.0 > > Database API Specification - Or something broader around async > > specifications for dabatase transactions. > > > > Thanks in advance, > > > > -N > > > > > > > > Hello. > > > > Reaching out to find out if there have been any discusdions around a 3.0 > > Database API Specification - Or something broader around async > > specifications for dabatase transactions. > > > > Thanks in advance, > > > > -N > > > > -- > > Nader B. Hamady > > > > > > _______________________________________________ > > DB-SIG maillist - DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig From mal at egenix.com Thu Oct 18 05:46:20 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 18 Oct 2018 12:46:20 +0300 Subject: [DB-SIG] PEP 249 In-Reply-To: References: Message-ID: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> On 16.10.2018 16:37, Mike Bayer wrote: > AFAIK , Postgresql is the only major relational database that has an > actual non-blocking protocol. The other databases all require that > "async" be simulated in the context of blocking IO. ODBC does have an async I/O API as well, but I don't know how wide spread the implementation of this API is among ODBC drivers. Python modules talking wire protocol and thus having full control over sockets could certainly support an async variant of the DB-API as well. E.g. we have added async support to mxODBC Connect (which uses its own wire protocol) via gevent a long time ago and it works well in that context. We did not introduce any API changes for this, only a config option to enable support: https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 > On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg wrote: >> >> Hi Nadar, >> >> we do have a set of things scheduled for a version 3.0 of >> the DB API (see the python.org wiki), but nothing related to >> async I/O yet. >> >> FWIW: I think the existing APIs can be used in an async form >> as well, without having to specify new ones, by simply >> providing an async entry point at the module level or >> even making the DB module itself be async only. >> >> Thanks, >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Experts >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>> Python Database Interfaces ... http://products.egenix.com/ >>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ >> ________________________________________________________________________ >> >> ::: We implement business ideas - efficiently in both time and costs ::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> http://www.malemburg.com/ >> >> >> >> On 11.10.2018 21:48, Nader Hamady wrote: >>> Hello. >>> >>> Reaching out to find out if there have been any discusdions around a 3.0 >>> Database API Specification - Or something broader around async >>> specifications for dabatase transactions. >>> >>> Thanks in advance, >>> >>> -N >>> >>> >>> >>> Hello. >>> >>> Reaching out to find out if there have been any discusdions around a 3.0 >>> Database API Specification - Or something broader around async >>> specifications for dabatase transactions. >>> >>> Thanks in advance, >>> >>> -N >>> >>> -- >>> Nader B. Hamady >>> >>> >>> _______________________________________________ >>> DB-SIG maillist - DB-SIG at python.org >>> https://mail.python.org/mailman/listinfo/db-sig >>> >> >> _______________________________________________ >> DB-SIG maillist - DB-SIG at python.org >> https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mike_mp at zzzcomputing.com Thu Oct 18 09:07:26 2018 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Thu, 18 Oct 2018 09:07:26 -0400 Subject: [DB-SIG] PEP 249 In-Reply-To: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> References: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> Message-ID: On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg wrote: > > On 16.10.2018 16:37, Mike Bayer wrote: > > AFAIK , Postgresql is the only major relational database that has an > > actual non-blocking protocol. The other databases all require that > > "async" be simulated in the context of blocking IO. > > ODBC does have an async I/O API as well, but I don't know > how wide spread the implementation of this API is among ODBC > drivers. > > Python modules talking wire protocol and thus having full > control over sockets could certainly support an async variant > of the DB-API as well. right, unusual though that given async's intense popularity in Python right now, I haven't seen any drivers doing this with say, MySQL, which you would think would be the most obvious target, considering there are multiple pure-Python drivers already. > > E.g. we have added async support to mxODBC Connect (which uses > its own wire protocol) via gevent a long time ago and it works > well in that context. We did not introduce any API changes > for this, only a config option to enable support: Well, "gevent" unfortunately does not count with the "async" crowd these days, I agree it's the easiest way to get at non-blocking behaviors but Python's asyncio is all the rage right now and it requires very dramatic API changes. > > https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 > > > > On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg wrote: > >> > >> Hi Nadar, > >> > >> we do have a set of things scheduled for a version 3.0 of > >> the DB API (see the python.org wiki), but nothing related to > >> async I/O yet. > >> > >> FWIW: I think the existing APIs can be used in an async form > >> as well, without having to specify new ones, by simply > >> providing an async entry point at the module level or > >> even making the DB module itself be async only. > >> > >> Thanks, > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > >> Professional Python Services directly from the Experts > >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>>>> Python Database Interfaces ... http://products.egenix.com/ > >>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > >> ________________________________________________________________________ > >> > >> ::: We implement business ideas - efficiently in both time and costs ::: > >> > >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > >> Registered at Amtsgericht Duesseldorf: HRB 46611 > >> http://www.egenix.com/company/contact/ > >> http://www.malemburg.com/ > >> > >> > >> > >> On 11.10.2018 21:48, Nader Hamady wrote: > >>> Hello. > >>> > >>> Reaching out to find out if there have been any discusdions around a 3.0 > >>> Database API Specification - Or something broader around async > >>> specifications for dabatase transactions. > >>> > >>> Thanks in advance, > >>> > >>> -N > >>> > >>> > >>> > >>> Hello. > >>> > >>> Reaching out to find out if there have been any discusdions around a 3.0 > >>> Database API Specification - Or something broader around async > >>> specifications for dabatase transactions. > >>> > >>> Thanks in advance, > >>> > >>> -N > >>> > >>> -- > >>> Nader B. Hamady > >>> > >>> > >>> _______________________________________________ > >>> DB-SIG maillist - DB-SIG at python.org > >>> https://mail.python.org/mailman/listinfo/db-sig > >>> > >> > >> _______________________________________________ > >> DB-SIG maillist - DB-SIG at python.org > >> https://mail.python.org/mailman/listinfo/db-sig > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > From mal at egenix.com Sat Oct 20 05:02:20 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 20 Oct 2018 12:02:20 +0300 Subject: [DB-SIG] PEP 249 In-Reply-To: References: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> Message-ID: <284139de-3920-e12e-c9ef-672293ad0643@egenix.com> On 18.10.2018 16:07, Mike Bayer wrote: > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg wrote: >> >> On 16.10.2018 16:37, Mike Bayer wrote: >>> AFAIK , Postgresql is the only major relational database that has an >>> actual non-blocking protocol. The other databases all require that >>> "async" be simulated in the context of blocking IO. >> >> ODBC does have an async I/O API as well, but I don't know >> how wide spread the implementation of this API is among ODBC >> drivers. >> >> Python modules talking wire protocol and thus having full >> control over sockets could certainly support an async variant >> of the DB-API as well. > > right, unusual though that given async's intense popularity in Python > right now, I haven't seen any drivers doing this with say, MySQL, > which you would think would be the most obvious target, considering > there are multiple pure-Python drivers already. > >> >> E.g. we have added async support to mxODBC Connect (which uses >> its own wire protocol) via gevent a long time ago and it works >> well in that context. We did not introduce any API changes >> for this, only a config option to enable support: > > Well, "gevent" unfortunately does not count with the "async" crowd > these days, I agree it's the easiest way to get at non-blocking > behaviors but Python's asyncio is all the rage right now and it > requires very dramatic API changes. There are certainly multiple ways to add async operations to database interactions. In ODBC, you can kick off an operation and then poll for completion. This has been in place for several years. More recent versions also have a notification model. I just brought up gevent, because it's certainly the easiest way to add async behavior to an existing application. It's also possible to integrate gevent into Python's asyncio, but I'm not sure how far that has been followed through... https://github.com/gevent/gevent/issues/982 The native Python support requires deep changes in the code. In particular, you have to make sure that all methods down to the socket calls are async. Problem is that many database modules eventually interface to a database C lib and those don't necessarily give you back control in case an operation takes longer, so it's not always possible to implement this, unless you work with threads to mimic async behavior - which is kind of besides the point :-) >> >> https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 >> >> >>> On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg wrote: >>>> >>>> Hi Nadar, >>>> >>>> we do have a set of things scheduled for a version 3.0 of >>>> the DB API (see the python.org wiki), but nothing related to >>>> async I/O yet. >>>> >>>> FWIW: I think the existing APIs can be used in an async form >>>> as well, without having to specify new ones, by simply >>>> providing an async entry point at the module level or >>>> even making the DB module itself be async only. >>>> >>>> Thanks, >>>> -- >>>> Marc-Andre Lemburg >>>> eGenix.com >>>> >>>> Professional Python Services directly from the Experts >>>>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>>>> Python Database Interfaces ... http://products.egenix.com/ >>>>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ >>>> ________________________________________________________________________ >>>> >>>> ::: We implement business ideas - efficiently in both time and costs ::: >>>> >>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>>> http://www.egenix.com/company/contact/ >>>> http://www.malemburg.com/ >>>> >>>> >>>> >>>> On 11.10.2018 21:48, Nader Hamady wrote: >>>>> Hello. >>>>> >>>>> Reaching out to find out if there have been any discusdions around a 3.0 >>>>> Database API Specification - Or something broader around async >>>>> specifications for dabatase transactions. >>>>> >>>>> Thanks in advance, >>>>> >>>>> -N >>>>> >>>>> >>>>> >>>>> Hello. >>>>> >>>>> Reaching out to find out if there have been any discusdions around a 3.0 >>>>> Database API Specification - Or something broader around async >>>>> specifications for dabatase transactions. >>>>> >>>>> Thanks in advance, >>>>> >>>>> -N >>>>> >>>>> -- >>>>> Nader B. Hamady >>>>> >>>>> >>>>> _______________________________________________ >>>>> DB-SIG maillist - DB-SIG at python.org >>>>> https://mail.python.org/mailman/listinfo/db-sig >>>>> >>>> >>>> _______________________________________________ >>>> DB-SIG maillist - DB-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/db-sig >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Experts >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>> Python Database Interfaces ... http://products.egenix.com/ >>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ >> ________________________________________________________________________ >> >> ::: We implement business ideas - efficiently in both time and costs ::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> http://www.malemburg.com/ >> > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mike_mp at zzzcomputing.com Sat Oct 20 12:13:25 2018 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Sat, 20 Oct 2018 12:13:25 -0400 Subject: [DB-SIG] PEP 249 In-Reply-To: <284139de-3920-e12e-c9ef-672293ad0643@egenix.com> References: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> <284139de-3920-e12e-c9ef-672293ad0643@egenix.com> Message-ID: On Sat, Oct 20, 2018, 5:02 AM M.-A. Lemburg wrote: > On 18.10.2018 16:07, Mike Bayer wrote: > > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg wrote: > >> > >> On 16.10.2018 16:37, Mike Bayer wrote: > >>> AFAIK , Postgresql is the only major relational database that has an > >>> actual non-blocking protocol. The other databases all require that > >>> "async" be simulated in the context of blocking IO. > >> > >> ODBC does have an async I/O API as well, but I don't know > >> how wide spread the implementation of this API is among ODBC > >> drivers. > >> > >> Python modules talking wire protocol and thus having full > >> control over sockets could certainly support an async variant > >> of the DB-API as well. > > > > right, unusual though that given async's intense popularity in Python > > right now, I haven't seen any drivers doing this with say, MySQL, > > which you would think would be the most obvious target, considering > > there are multiple pure-Python drivers already. > > > >> > >> E.g. we have added async support to mxODBC Connect (which uses > >> its own wire protocol) via gevent a long time ago and it works > >> well in that context. We did not introduce any API changes > >> for this, only a config option to enable support: > > > > Well, "gevent" unfortunately does not count with the "async" crowd > > these days, I agree it's the easiest way to get at non-blocking > > behaviors but Python's asyncio is all the rage right now and it > > requires very dramatic API changes. > > There are certainly multiple ways to add async operations to > database interactions. In ODBC, you can kick off an operation > and then poll for completion. This has been in place for several > years. More recent versions also have a notification model. > I just brought up gevent, because it's certainly the easiest > way to add async behavior to an existing application. > > It's also possible to integrate gevent into Python's > asyncio, but I'm not sure how far that has been followed > through... https://github.com/gevent/gevent/issues/982 > > The native Python support requires deep changes in the code. > In particular, you have to make sure that all methods down > to the socket calls are async. > > Problem is that many database modules eventually interface to > a database C lib and those don't necessarily give you back > control in case an operation takes longer, so it's not > always possible to implement this, unless you work with threads > to mimic async behavior - which is kind of besides the point :-) > That was my original point :) if the DBAPI is pure Python then it can do asyncio / gevent without background threads, but if not, unless it is using Posgresql's async API or this thing that ODBC seems to have, using background threads will often be a factor, which IMO eliminates any even theoretical advantage of using asyncio. I spend time regularly trying to show the world that non-blocking IO (even using the phrase "non-blocking IO", because a lot of the people so crazy over asyncio don't even understand what non-blocking IO even means) gets us next to nothing in the real world and especially not for the vast majority of database use. My original blog post on this still holds: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and I have a side-hobby of looking for asyncio/greenlet related threads on reddit and elsewhere, finding the posts that say "Async is faster!" and proving them wrong, example: https://stackoverflow.com/a/51932442/34549 . Most people aren't running hundreds of concurrent multi-second queries at once against their database. Therefore they don't need non-blocking IO. > >> > >> https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 > >> > >> > >>> On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg wrote: > >>>> > >>>> Hi Nadar, > >>>> > >>>> we do have a set of things scheduled for a version 3.0 of > >>>> the DB API (see the python.org wiki), but nothing related to > >>>> async I/O yet. > >>>> > >>>> FWIW: I think the existing APIs can be used in an async form > >>>> as well, without having to specify new ones, by simply > >>>> providing an async entry point at the module level or > >>>> even making the DB module itself be async only. > >>>> > >>>> Thanks, > >>>> -- > >>>> Marc-Andre Lemburg > >>>> eGenix.com > >>>> > >>>> Professional Python Services directly from the Experts > >>>>>>> Python Projects, Coaching and Consulting ... > http://www.egenix.com/ > >>>>>>> Python Database Interfaces ... > http://products.egenix.com/ > >>>>>>> Plone/Zope Database Interfaces ... > http://zope.egenix.com/ > >>>> > ________________________________________________________________________ > >>>> > >>>> ::: We implement business ideas - efficiently in both time and costs > ::: > >>>> > >>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > >>>> Registered at Amtsgericht Duesseldorf: HRB 46611 > >>>> http://www.egenix.com/company/contact/ > >>>> http://www.malemburg.com/ > >>>> > >>>> > >>>> > >>>> On 11.10.2018 21:48, Nader Hamady wrote: > >>>>> Hello. > >>>>> > >>>>> Reaching out to find out if there have been any discusdions around a > 3.0 > >>>>> Database API Specification - Or something broader around async > >>>>> specifications for dabatase transactions. > >>>>> > >>>>> Thanks in advance, > >>>>> > >>>>> -N > >>>>> > >>>>> > >>>>> > >>>>> Hello. > >>>>> > >>>>> Reaching out to find out if there have been any discusdions around a > 3.0 > >>>>> Database API Specification - Or something broader around async > >>>>> specifications for dabatase transactions. > >>>>> > >>>>> Thanks in advance, > >>>>> > >>>>> -N > >>>>> > >>>>> -- > >>>>> Nader B. Hamady > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> DB-SIG maillist - DB-SIG at python.org > >>>>> https://mail.python.org/mailman/listinfo/db-sig > >>>>> > >>>> > >>>> _______________________________________________ > >>>> DB-SIG maillist - DB-SIG at python.org > >>>> https://mail.python.org/mailman/listinfo/db-sig > >> > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > >> Professional Python Services directly from the Experts > >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>>>> Python Database Interfaces ... http://products.egenix.com/ > >>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > >> ________________________________________________________________________ > >> > >> ::: We implement business ideas - efficiently in both time and costs ::: > >> > >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > >> Registered at Amtsgericht Duesseldorf: HRB 46611 > >> http://www.egenix.com/company/contact/ > >> http://www.malemburg.com/ > >> > > _______________________________________________ > > DB-SIG maillist - DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > > > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sun Oct 21 05:53:24 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 21 Oct 2018 12:53:24 +0300 Subject: [DB-SIG] PEP 249 In-Reply-To: References: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> <284139de-3920-e12e-c9ef-672293ad0643@egenix.com> Message-ID: <8e623251-22cd-aa41-5b38-6db8c2a61f62@egenix.com> On 20.10.2018 19:13, Mike Bayer wrote: > > > On Sat, Oct 20, 2018, 5:02 AM M.-A. Lemburg > wrote: > > On 18.10.2018 16:07, Mike Bayer wrote: > > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg > wrote: > >> > >> On 16.10.2018 16:37, Mike Bayer wrote: > >>> AFAIK , Postgresql is the only major relational database that has an > >>> actual non-blocking protocol.? ? The other databases all require > that > >>> "async" be simulated in the context of blocking IO. > >> > >> ODBC does have an async I/O API as well, but I don't know > >> how wide spread the implementation of this API is among ODBC > >> drivers. > >> > >> Python modules talking wire protocol and thus having full > >> control over sockets could certainly support an async variant > >> of the DB-API as well. > > > > right, unusual though that given async's intense popularity in Python > > right now, I haven't seen any drivers doing this with say, MySQL, > > which you would think would be the most obvious target, considering > > there are multiple pure-Python drivers already. > > > >> > >> E.g. we have added async support to mxODBC Connect (which uses > >> its own wire protocol) via gevent a long time ago and it works > >> well in that context. We did not introduce any API changes > >> for this, only a config option to enable support: > > > > Well, "gevent" unfortunately does not count with the "async" crowd > > these days, I agree it's the easiest way to get at non-blocking > > behaviors but Python's asyncio is all the rage right now and it > > requires very dramatic API changes. > > There are certainly multiple ways to add async operations to > database interactions. In ODBC, you can kick off an operation > and then poll for completion. This has been in place for several > years. More recent versions also have a notification model. > I just brought up gevent, because it's certainly the easiest > way to add async behavior to an existing application. > > It's also possible to integrate gevent into Python's > asyncio, but I'm not sure how far that has been followed > through... https://github.com/gevent/gevent/issues/982 > > The native Python support requires deep changes in the code. > In particular, you have to make sure that all methods down > to the socket calls are async. > > Problem is that many database modules eventually interface to > a database C lib and those don't necessarily give you back > control in case an operation takes longer, so it's not > always possible to implement this, unless you work with threads > to mimic async behavior - which is kind of besides the point :-) > > > > That was my original point :)?? if the DBAPI is pure Python then it can > do asyncio / gevent without background threads, but if not, unless it is > using Posgresql's async API or this thing that ODBC seems to have, using > background threads will often be a factor, which IMO eliminates any even > theoretical advantage of using asyncio.? I spend time regularly trying > to show the world that non-blocking IO (even using the phrase > "non-blocking IO", because a lot of the people so crazy over asyncio > don't even understand what non-blocking IO even means) gets us next to > nothing in the real world and especially not for the vast majority of > database use.??? My original blog post on this still holds: > http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/? and > I have a side-hobby of looking for asyncio/greenlet related threads on > reddit and elsewhere, finding the posts that say "Async is faster!" and > proving them wrong, example: https://stackoverflow.com/a/51932442/34549 > .????? Most people aren't running hundreds of concurrent multi-second > queries at once against their database.?? Therefore they don't need > non-blocking IO. I think the main advantage of using asyncio is saturating a single core as much as possible given the GIL. It probably also helps when using multiple threads, where asyncio can help making most use of the cores in each thread, but it complicates things since you need to carefully manage the event loop - probably even having the asyncio loop managing them as David Beazley does in his thredo package (https://www.youtube.com/watch?v=xOyJiN3yGfU). Now, the GIL is rarely a problem in Python database programming using C database client libs, since most of times, the GIL is released when running queries, so not blocking other threads. It also has to be noted that database libs typically use background threads already, so might as well use them in Python as well. Anyway, we're drifting off topic I think :-) > > >> > >> > https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 > >> > >> > >>> On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg > wrote: > >>>> > >>>> Hi Nadar, > >>>> > >>>> we do have a set of things scheduled for a version 3.0 of > >>>> the DB API (see the python.org wiki), but > nothing related to > >>>> async I/O yet. > >>>> > >>>> FWIW: I think the existing APIs can be used in an async form > >>>> as well, without having to specify new ones, by simply > >>>> providing an async entry point at the module level or > >>>> even making the DB module itself be async only. > >>>> > >>>> Thanks, > >>>> -- > >>>> Marc-Andre Lemburg > >>>> eGenix.com > >>>> > >>>> Professional Python Services directly from the Experts > >>>>>>> Python Projects, Coaching and Consulting ...? > http://www.egenix.com/ > >>>>>>> Python Database Interfaces ...? ? ? ? ? > ?http://products.egenix.com/ > >>>>>>> Plone/Zope Database Interfaces ...? ? ? ? ? > ?http://zope.egenix.com/ > >>>> > ________________________________________________________________________ > >>>> > >>>> ::: We implement business ideas - efficiently in both time and > costs ::: > >>>> > >>>>? ? eGenix.com Software, Skills and Services GmbH? > Pastor-Loeh-Str.48 > >>>>? ? ?D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > >>>>? ? ? ? ? ? Registered at Amtsgericht Duesseldorf: HRB 46611 > >>>>? ? ? ? ? ? ? ? http://www.egenix.com/company/contact/ > >>>>? ? ? ? ? ? ? ? ? ? ? ?http://www.malemburg.com/ > >>>> > >>>> > >>>> > >>>> On 11.10.2018 21:48, Nader Hamady wrote: > >>>>> Hello. > >>>>> > >>>>> Reaching out to find out if there have been any discusdions > around a 3.0 > >>>>> Database API Specification - Or something broader around async > >>>>> specifications for dabatase transactions. > >>>>> > >>>>> Thanks in advance, > >>>>> > >>>>> -N > >>>>> > >>>>> > >>>>> > >>>>> Hello. > >>>>> > >>>>> Reaching out to find out if there have been any discusdions > around a 3.0 > >>>>> Database API Specification - Or something broader around async > >>>>> specifications for dabatase transactions. > >>>>> > >>>>> Thanks in advance, > >>>>> > >>>>> -N > >>>>> > >>>>> -- > >>>>> Nader B. Hamady > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> DB-SIG maillist? -? DB-SIG at python.org > >>>>> https://mail.python.org/mailman/listinfo/db-sig > >>>>> > >>>> > >>>> _______________________________________________ > >>>> DB-SIG maillist? -? DB-SIG at python.org > >>>> https://mail.python.org/mailman/listinfo/db-sig > >> > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > >> Professional Python Services directly from the Experts > >>>>> Python Projects, Coaching and Consulting ...? > http://www.egenix.com/ > >>>>> Python Database Interfaces ...? ? ? ? ? > ?http://products.egenix.com/ > >>>>> Plone/Zope Database Interfaces ...? ? ? ? ? > ?http://zope.egenix.com/ > >> > ________________________________________________________________________ > >> > >> ::: We implement business ideas - efficiently in both time and > costs ::: > >> > >>? ? eGenix.com Software, Skills and Services GmbH? Pastor-Loeh-Str.48 > >>? ? ?D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > >>? ? ? ? ? ? Registered at Amtsgericht Duesseldorf: HRB 46611 > >>? ? ? ? ? ? ? ? http://www.egenix.com/company/contact/ > >>? ? ? ? ? ? ? ? ? ? ? ?http://www.malemburg.com/ > >> > > _______________________________________________ > > DB-SIG maillist? -? DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > > > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts > >>> Python Projects, Coaching and Consulting ...? http://www.egenix.com/ > >>> Python Database Interfaces ...? ? ? ? ? ?http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ...? ? ? ? ? ?http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > ? ?eGenix.com Software, Skills and Services GmbH? Pastor-Loeh-Str.48 > ? ? D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > ? ? ? ? ? ?Registered at Amtsgericht Duesseldorf: HRB 46611 > ? ? ? ? ? ? ? ?http://www.egenix.com/company/contact/ > ? ? ? ? ? ? ? ? ? ? ? http://www.malemburg.com/ > > > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mike_mp at zzzcomputing.com Sun Oct 21 09:02:39 2018 From: mike_mp at zzzcomputing.com (Mike Bayer) Date: Sun, 21 Oct 2018 09:02:39 -0400 Subject: [DB-SIG] PEP 249 In-Reply-To: <8e623251-22cd-aa41-5b38-6db8c2a61f62@egenix.com> References: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> <284139de-3920-e12e-c9ef-672293ad0643@egenix.com> <8e623251-22cd-aa41-5b38-6db8c2a61f62@egenix.com> Message-ID: On Sun, Oct 21, 2018 at 5:53 AM M.-A. Lemburg wrote: > > On 20.10.2018 19:13, Mike Bayer wrote: > > > > > > On Sat, Oct 20, 2018, 5:02 AM M.-A. Lemburg > > wrote: > > > > On 18.10.2018 16:07, Mike Bayer wrote: > > > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg > > wrote: > > >> > > >> On 16.10.2018 16:37, Mike Bayer wrote: > > >>> AFAIK , Postgresql is the only major relational database that has an > > >>> actual non-blocking protocol. The other databases all require > > that > > >>> "async" be simulated in the context of blocking IO. > > >> > > >> ODBC does have an async I/O API as well, but I don't know > > >> how wide spread the implementation of this API is among ODBC > > >> drivers. > > >> > > >> Python modules talking wire protocol and thus having full > > >> control over sockets could certainly support an async variant > > >> of the DB-API as well. > > > > > > right, unusual though that given async's intense popularity in Python > > > right now, I haven't seen any drivers doing this with say, MySQL, > > > which you would think would be the most obvious target, considering > > > there are multiple pure-Python drivers already. > > > > > >> > > >> E.g. we have added async support to mxODBC Connect (which uses > > >> its own wire protocol) via gevent a long time ago and it works > > >> well in that context. We did not introduce any API changes > > >> for this, only a config option to enable support: > > > > > > Well, "gevent" unfortunately does not count with the "async" crowd > > > these days, I agree it's the easiest way to get at non-blocking > > > behaviors but Python's asyncio is all the rage right now and it > > > requires very dramatic API changes. > > > > There are certainly multiple ways to add async operations to > > database interactions. In ODBC, you can kick off an operation > > and then poll for completion. This has been in place for several > > years. More recent versions also have a notification model. > > I just brought up gevent, because it's certainly the easiest > > way to add async behavior to an existing application. > > > > It's also possible to integrate gevent into Python's > > asyncio, but I'm not sure how far that has been followed > > through... https://github.com/gevent/gevent/issues/982 > > > > The native Python support requires deep changes in the code. > > In particular, you have to make sure that all methods down > > to the socket calls are async. > > > > Problem is that many database modules eventually interface to > > a database C lib and those don't necessarily give you back > > control in case an operation takes longer, so it's not > > always possible to implement this, unless you work with threads > > to mimic async behavior - which is kind of besides the point :-) > > > > > > > > That was my original point :) if the DBAPI is pure Python then it can > > do asyncio / gevent without background threads, but if not, unless it is > > using Posgresql's async API or this thing that ODBC seems to have, using > > background threads will often be a factor, which IMO eliminates any even > > theoretical advantage of using asyncio. I spend time regularly trying > > to show the world that non-blocking IO (even using the phrase > > "non-blocking IO", because a lot of the people so crazy over asyncio > > don't even understand what non-blocking IO even means) gets us next to > > nothing in the real world and especially not for the vast majority of > > database use. My original blog post on this still holds: > > http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and > > I have a side-hobby of looking for asyncio/greenlet related threads on > > reddit and elsewhere, finding the posts that say "Async is faster!" and > > proving them wrong, example: https://stackoverflow.com/a/51932442/34549 > > . Most people aren't running hundreds of concurrent multi-second > > queries at once against their database. Therefore they don't need > > non-blocking IO. > > I think the main advantage of using asyncio is saturating a single > core as much as possible given the GIL. I've yet to see a single demonstration showing this to be more efficient than an equivalent threaded application. This is the theory everyone cites but misses the fact that the context switching in gevent, asyncio is actually more expensive than that of threads even though it happens much less frequently. asyncio does not save you from the GIL in any way, you still have the GIL just as much as if you didn't, and since asyncio applications use only one thread anyway, the point is moot, unless you're saying the GIL can be disabled altogether (is that possible?) and save on the time spent locking. I've written multiple test suites trying to show this is the case for gevent and asyncio without success. if someone can show me a workload that completes more quickly with asyncio than threads then I'd finally have something to work with, but after all of my loud outspokenness, which I thought would attract people to prove me wrong and teach me something, nobody has done so as of yet. > > It probably also helps when using multiple threads, where asyncio can > help making most use of the cores in each thread, but it complicates > things since you need to carefully manage the event loop - probably > even having the asyncio loop managing them as David Beazley does in > his thredo package (https://www.youtube.com/watch?v=xOyJiN3yGfU). > > Now, the GIL is rarely a problem in Python database programming > using C database client libs, since most of times, the GIL is released > when running queries, so not blocking other threads. It also has > to be noted that database libs typically use background threads > already, so might as well use them in Python as well. > > Anyway, we're drifting off topic I think :-) the topic is how useful is it to have an asyncio API for pep-249. I think the only usefulness is just that people want it due to all the attention asyncio is getting, and that it would make integration with asyncio applications more straightforward from a programming point of view. That's a valid reason. Just IMO a sad one :) > > > > > >> > > >> > > https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 > > >> > > >> > > >>> On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg > > wrote: > > >>>> > > >>>> Hi Nadar, > > >>>> > > >>>> we do have a set of things scheduled for a version 3.0 of > > >>>> the DB API (see the python.org wiki), but > > nothing related to > > >>>> async I/O yet. > > >>>> > > >>>> FWIW: I think the existing APIs can be used in an async form > > >>>> as well, without having to specify new ones, by simply > > >>>> providing an async entry point at the module level or > > >>>> even making the DB module itself be async only. > > >>>> > > >>>> Thanks, > > >>>> -- > > >>>> Marc-Andre Lemburg > > >>>> eGenix.com > > >>>> > > >>>> Professional Python Services directly from the Experts > > >>>>>>> Python Projects, Coaching and Consulting ... > > http://www.egenix.com/ > > >>>>>>> Python Database Interfaces ... > > http://products.egenix.com/ > > >>>>>>> Plone/Zope Database Interfaces ... > > http://zope.egenix.com/ > > >>>> > > ________________________________________________________________________ > > >>>> > > >>>> ::: We implement business ideas - efficiently in both time and > > costs ::: > > >>>> > > >>>> eGenix.com Software, Skills and Services GmbH > > Pastor-Loeh-Str.48 > > >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > >>>> Registered at Amtsgericht Duesseldorf: HRB 46611 > > >>>> http://www.egenix.com/company/contact/ > > >>>> http://www.malemburg.com/ > > >>>> > > >>>> > > >>>> > > >>>> On 11.10.2018 21:48, Nader Hamady wrote: > > >>>>> Hello. > > >>>>> > > >>>>> Reaching out to find out if there have been any discusdions > > around a 3.0 > > >>>>> Database API Specification - Or something broader around async > > >>>>> specifications for dabatase transactions. > > >>>>> > > >>>>> Thanks in advance, > > >>>>> > > >>>>> -N > > >>>>> > > >>>>> > > >>>>> > > >>>>> Hello. > > >>>>> > > >>>>> Reaching out to find out if there have been any discusdions > > around a 3.0 > > >>>>> Database API Specification - Or something broader around async > > >>>>> specifications for dabatase transactions. > > >>>>> > > >>>>> Thanks in advance, > > >>>>> > > >>>>> -N > > >>>>> > > >>>>> -- > > >>>>> Nader B. Hamady > > >>>>> > > >>>>> > > >>>>> _______________________________________________ > > >>>>> DB-SIG maillist - DB-SIG at python.org > > >>>>> https://mail.python.org/mailman/listinfo/db-sig > > >>>>> > > >>>> > > >>>> _______________________________________________ > > >>>> DB-SIG maillist - DB-SIG at python.org > > >>>> https://mail.python.org/mailman/listinfo/db-sig > > >> > > >> -- > > >> Marc-Andre Lemburg > > >> eGenix.com > > >> > > >> Professional Python Services directly from the Experts > > >>>>> Python Projects, Coaching and Consulting ... > > http://www.egenix.com/ > > >>>>> Python Database Interfaces ... > > http://products.egenix.com/ > > >>>>> Plone/Zope Database Interfaces ... > > http://zope.egenix.com/ > > >> > > ________________________________________________________________________ > > >> > > >> ::: We implement business ideas - efficiently in both time and > > costs ::: > > >> > > >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > >> Registered at Amtsgericht Duesseldorf: HRB 46611 > > >> http://www.egenix.com/company/contact/ > > >> http://www.malemburg.com/ > > >> > > > _______________________________________________ > > > DB-SIG maillist - DB-SIG at python.org > > > https://mail.python.org/mailman/listinfo/db-sig > > > > > > > -- > > Marc-Andre Lemburg > > eGenix.com > > > > Professional Python Services directly from the Experts > > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > > >>> Python Database Interfaces ... http://products.egenix.com/ > > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > > ________________________________________________________________________ > > > > ::: We implement business ideas - efficiently in both time and costs ::: > > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > Registered at Amtsgericht Duesseldorf: HRB 46611 > > http://www.egenix.com/company/contact/ > > http://www.malemburg.com/ > > > > > > > > _______________________________________________ > > DB-SIG maillist - DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > > > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > From nbhamady at gmail.com Fri Oct 19 17:52:54 2018 From: nbhamady at gmail.com (Nader Hamady) Date: Fri, 19 Oct 2018 17:52:54 -0400 Subject: [DB-SIG] PEP 249 In-Reply-To: References: <0e78e725-d733-d60d-881d-c123ef8a3e36@egenix.com> Message-ID: This is actually a lot of good info, thanks a lot for following up. Wasn?t aware of the new db api version, but will follow along. Just to clarify, I?m coming from the background of building an in house orm that attempts to abstract some of the async features provided by different db modules (i.e pyodbc/oracle_cx). In the process of doing so, we noticed that different modules had different implementations for connection pooling and making async calls. Made me wonder if the new db api version was intended to standarize this. Happy to get involved if there?s an apetite for it. Nader On Thu, Oct 18, 2018 at 09:07 Mike Bayer wrote: > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg wrote: > > > > On 16.10.2018 16:37, Mike Bayer wrote: > > > AFAIK , Postgresql is the only major relational database that has an > > > actual non-blocking protocol. The other databases all require that > > > "async" be simulated in the context of blocking IO. > > > > ODBC does have an async I/O API as well, but I don't know > > how wide spread the implementation of this API is among ODBC > > drivers. > > > > Python modules talking wire protocol and thus having full > > control over sockets could certainly support an async variant > > of the DB-API as well. > > right, unusual though that given async's intense popularity in Python > right now, I haven't seen any drivers doing this with say, MySQL, > which you would think would be the most obvious target, considering > there are multiple pure-Python drivers already. > > > > > E.g. we have added async support to mxODBC Connect (which uses > > its own wire protocol) via gevent a long time ago and it works > > well in that context. We did not introduce any API changes > > for this, only a config option to enable support: > > Well, "gevent" unfortunately does not count with the "async" crowd > these days, I agree it's the easiest way to get at non-blocking > behaviors but Python's asyncio is all the rage right now and it > requires very dramatic API changes. > > > > > > https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695 > > > > > > > On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg wrote: > > >> > > >> Hi Nadar, > > >> > > >> we do have a set of things scheduled for a version 3.0 of > > >> the DB API (see the python.org wiki), but nothing related to > > >> async I/O yet. > > >> > > >> FWIW: I think the existing APIs can be used in an async form > > >> as well, without having to specify new ones, by simply > > >> providing an async entry point at the module level or > > >> even making the DB module itself be async only. > > >> > > >> Thanks, > > >> -- > > >> Marc-Andre Lemburg > > >> eGenix.com > > >> > > >> Professional Python Services directly from the Experts > > >>>>> Python Projects, Coaching and Consulting ... > http://www.egenix.com/ > > >>>>> Python Database Interfaces ... > http://products.egenix.com/ > > >>>>> Plone/Zope Database Interfaces ... > http://zope.egenix.com/ > > >> > ________________________________________________________________________ > > >> > > >> ::: We implement business ideas - efficiently in both time and costs > ::: > > >> > > >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > >> Registered at Amtsgericht Duesseldorf: HRB 46611 > > >> http://www.egenix.com/company/contact/ > > >> http://www.malemburg.com/ > > >> > > >> > > >> > > >> On 11.10.2018 21:48, Nader Hamady wrote: > > >>> Hello. > > >>> > > >>> Reaching out to find out if there have been any discusdions around a > 3.0 > > >>> Database API Specification - Or something broader around async > > >>> specifications for dabatase transactions. > > >>> > > >>> Thanks in advance, > > >>> > > >>> -N > > >>> > > >>> > > >>> > > >>> Hello. > > >>> > > >>> Reaching out to find out if there have been any discusdions around a > 3.0 > > >>> Database API Specification - Or something broader around async > > >>> specifications for dabatase transactions. > > >>> > > >>> Thanks in advance, > > >>> > > >>> -N > > >>> > > >>> -- > > >>> Nader B. Hamady > > >>> > > >>> > > >>> _______________________________________________ > > >>> DB-SIG maillist - DB-SIG at python.org > > >>> https://mail.python.org/mailman/listinfo/db-sig > > >>> > > >> > > >> _______________________________________________ > > >> DB-SIG maillist - DB-SIG at python.org > > >> https://mail.python.org/mailman/listinfo/db-sig > > > > -- > > Marc-Andre Lemburg > > eGenix.com > > > > Professional Python Services directly from the Experts > > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > > >>> Python Database Interfaces ... http://products.egenix.com/ > > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > > ________________________________________________________________________ > > > > ::: We implement business ideas - efficiently in both time and costs ::: > > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > Registered at Amtsgericht Duesseldorf: HRB 46611 > > http://www.egenix.com/company/contact/ > > http://www.malemburg.com/ > > > -- Nader B. Hamady -------------- next part -------------- An HTML attachment was scrubbed... URL: