[DB-SIG] PEP 249
M.-A. Lemburg
mal at egenix.com
Sun Oct 21 05:53:24 EDT 2018
On 20.10.2018 19:13, Mike Bayer wrote:
>
>
> On Sat, Oct 20, 2018, 5:02 AM M.-A. Lemburg <mal at egenix.com
> <mailto:mal at egenix.com>> wrote:
>
> On 18.10.2018 16:07, Mike Bayer wrote:
> > On Thu, Oct 18, 2018 at 5:46 AM M.-A. Lemburg <mal at egenix.com
> <mailto:mal at egenix.com>> wrote:
> >>
> >> On 16.10.2018 16:37, Mike Bayer wrote:
> >>> AFAIK , Postgresql is the only major relational database that has an
> >>> actual non-blocking protocol. The other databases all require
> that
> >>> "async" be simulated in the context of blocking IO.
> >>
> >> ODBC does have an async I/O API as well, but I don't know
> >> how wide spread the implementation of this API is among ODBC
> >> drivers.
> >>
> >> Python modules talking wire protocol and thus having full
> >> control over sockets could certainly support an async variant
> >> of the DB-API as well.
> >
> > right, unusual though that given async's intense popularity in Python
> > right now, I haven't seen any drivers doing this with say, MySQL,
> > which you would think would be the most obvious target, considering
> > there are multiple pure-Python drivers already.
> >
> >>
> >> E.g. we have added async support to mxODBC Connect (which uses
> >> its own wire protocol) via gevent a long time ago and it works
> >> well in that context. We did not introduce any API changes
> >> for this, only a config option to enable support:
> >
> > Well, "gevent" unfortunately does not count with the "async" crowd
> > these days, I agree it's the easiest way to get at non-blocking
> > behaviors but Python's asyncio is all the rage right now and it
> > requires very dramatic API changes.
>
> There are certainly multiple ways to add async operations to
> database interactions. In ODBC, you can kick off an operation
> and then poll for completion. This has been in place for several
> years. More recent versions also have a notification model.
> I just brought up gevent, because it's certainly the easiest
> way to add async behavior to an existing application.
>
> It's also possible to integrate gevent into Python's
> asyncio, but I'm not sure how far that has been followed
> through... https://github.com/gevent/gevent/issues/982
>
> The native Python support requires deep changes in the code.
> In particular, you have to make sure that all methods down
> to the socket calls are async.
>
> Problem is that many database modules eventually interface to
> a database C lib and those don't necessarily give you back
> control in case an operation takes longer, so it's not
> always possible to implement this, unless you work with threads
> to mimic async behavior - which is kind of besides the point :-)
>
>
>
> That was my original point :) if the DBAPI is pure Python then it can
> do asyncio / gevent without background threads, but if not, unless it is
> using Posgresql's async API or this thing that ODBC seems to have, using
> background threads will often be a factor, which IMO eliminates any even
> theoretical advantage of using asyncio. I spend time regularly trying
> to show the world that non-blocking IO (even using the phrase
> "non-blocking IO", because a lot of the people so crazy over asyncio
> don't even understand what non-blocking IO even means) gets us next to
> nothing in the real world and especially not for the vast majority of
> database use. My original blog post on this still holds:
> http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and
> I have a side-hobby of looking for asyncio/greenlet related threads on
> reddit and elsewhere, finding the posts that say "Async is faster!" and
> proving them wrong, example: https://stackoverflow.com/a/51932442/34549
> . Most people aren't running hundreds of concurrent multi-second
> queries at once against their database. Therefore they don't need
> non-blocking IO.
I think the main advantage of using asyncio is saturating a single
core as much as possible given the GIL.
It probably also helps when using multiple threads, where asyncio can
help making most use of the cores in each thread, but it complicates
things since you need to carefully manage the event loop - probably
even having the asyncio loop managing them as David Beazley does in
his thredo package (https://www.youtube.com/watch?v=xOyJiN3yGfU).
Now, the GIL is rarely a problem in Python database programming
using C database client libs, since most of times, the GIL is released
when running queries, so not blocking other threads. It also has
to be noted that database libs typically use background threads
already, so might as well use them in Python as well.
Anyway, we're drifting off topic I think :-)
>
> >>
> >>
> https://www.egenix.com/products/python/mxODBCConnect/doc/#_Toc433125695
> >>
> >>
> >>> On Tue, Oct 16, 2018 at 6:02 AM M.-A. Lemburg <mal at egenix.com
> <mailto:mal at egenix.com>> wrote:
> >>>>
> >>>> Hi Nadar,
> >>>>
> >>>> we do have a set of things scheduled for a version 3.0 of
> >>>> the DB API (see the python.org <http://python.org> wiki), but
> nothing related to
> >>>> async I/O yet.
> >>>>
> >>>> FWIW: I think the existing APIs can be used in an async form
> >>>> as well, without having to specify new ones, by simply
> >>>> providing an async entry point at the module level or
> >>>> even making the DB module itself be async only.
> >>>>
> >>>> Thanks,
> >>>> --
> >>>> Marc-Andre Lemburg
> >>>> eGenix.com
> >>>>
> >>>> Professional Python Services directly from the Experts
> >>>>>>> Python Projects, Coaching and Consulting ...
> http://www.egenix.com/
> >>>>>>> Python Database Interfaces ...
> http://products.egenix.com/
> >>>>>>> Plone/Zope Database Interfaces ...
> http://zope.egenix.com/
> >>>>
> ________________________________________________________________________
> >>>>
> >>>> ::: We implement business ideas - efficiently in both time and
> costs :::
> >>>>
> >>>> eGenix.com Software, Skills and Services GmbH
> Pastor-Loeh-Str.48
> >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> >>>> Registered at Amtsgericht Duesseldorf: HRB 46611
> >>>> http://www.egenix.com/company/contact/
> >>>> http://www.malemburg.com/
> >>>>
> >>>>
> >>>>
> >>>> On 11.10.2018 21:48, Nader Hamady wrote:
> >>>>> Hello.
> >>>>>
> >>>>> Reaching out to find out if there have been any discusdions
> around a 3.0
> >>>>> Database API Specification - Or something broader around async
> >>>>> specifications for dabatase transactions.
> >>>>>
> >>>>> Thanks in advance,
> >>>>>
> >>>>> -N
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hello.
> >>>>>
> >>>>> Reaching out to find out if there have been any discusdions
> around a 3.0
> >>>>> Database API Specification - Or something broader around async
> >>>>> specifications for dabatase transactions.
> >>>>>
> >>>>> Thanks in advance,
> >>>>>
> >>>>> -N
> >>>>>
> >>>>> --
> >>>>> Nader B. Hamady
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> DB-SIG maillist - DB-SIG at python.org <mailto:DB-SIG at python.org>
> >>>>> https://mail.python.org/mailman/listinfo/db-sig
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> DB-SIG maillist - DB-SIG at python.org <mailto:DB-SIG at python.org>
> >>>> https://mail.python.org/mailman/listinfo/db-sig
> >>
> >> --
> >> Marc-Andre Lemburg
> >> eGenix.com
> >>
> >> Professional Python Services directly from the Experts
> >>>>> Python Projects, Coaching and Consulting ...
> http://www.egenix.com/
> >>>>> Python Database Interfaces ...
> http://products.egenix.com/
> >>>>> Plone/Zope Database Interfaces ...
> http://zope.egenix.com/
> >>
> ________________________________________________________________________
> >>
> >> ::: We implement business ideas - efficiently in both time and
> costs :::
> >>
> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> >> Registered at Amtsgericht Duesseldorf: HRB 46611
> >> http://www.egenix.com/company/contact/
> >> http://www.malemburg.com/
> >>
> > _______________________________________________
> > DB-SIG maillist - DB-SIG at python.org <mailto:DB-SIG at python.org>
> > https://mail.python.org/mailman/listinfo/db-sig
> >
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts
> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
> >>> Python Database Interfaces ... http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/
> http://www.malemburg.com/
>
>
>
> _______________________________________________
> DB-SIG maillist - DB-SIG at python.org
> https://mail.python.org/mailman/listinfo/db-sig
>
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> Python Database Interfaces ... http://products.egenix.com/
>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
________________________________________________________________________
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
http://www.malemburg.com/
More information about the DB-SIG
mailing list