[Python-ideas] Proposal: Query language extension to Python (PythonQL)

[David Mertz]

I think it's extraordinarily unlikely that a big change in Python syntax to
support query syntax will ever happen.  Moreover, I would oppose such a
change myself.

But just a change also really is not necessary.  Pandas already abstracts
all the things mentioned using only Python methods.  It is true that Pandas
sometimes does some black magic within those methods to get there; and it
also uses somewhat non-Pythonic style of long chains of method calls.  But
it does everything PythonQL does, as well as much, much more.  Pandas
builds in DataFrame readers for every data source you are likely to
encounter, including leveraging all the abstractions provided by RDBMS
drivers, etc.  It does groupby, join, etc.

See, e.g.:


http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html

Now there's one reasonable objection to Pandas: It doesn't handle
larger-than-memory datasets well.  I don't see that PythonQL is better in
that regard.  But there is an easy next step for that larger data.  Blaze
provides generic interfaces to many, many larger-than-memory data sources.
It is largely a subset of the Pandas API, although not precisely that.
See, e.g.:

    http://blaze.readthedocs.io/en/latest/rosetta-sql.html

Moreover, within the Blaze "orbit" is Dask.  This is a framework for
parallel computation, one of whose abstractions is a DataFrame based on
Pandas.  This gives you 90% of those methods for slicing-and-dicing data
that Pandas does, but deals seamlessly with larger-than-memory datasets.
See, e.g.:

    http://dask.pydata.org/en/latest/dataframe.html

So I think your burden is even higher than showing the usefulness of
PythonQL.  You have to show why it's worth adding new syntax to do somewhat
LESS than is available in very widely used 3rd party tools that avoid new
syntax.

On Fri, Mar 24, 2017 at 8:10 AM, Pavel Velikhov <pavel.velikhov at gmail.com>
wrote:

> Hi folks!
>
>   We started a project to extend Python with a full-blown query language
> about a year ago. The project is call PythonQL, the links are given below
> in the references section. We have implemented what is kind of an alpha
> version now, and gained some experience and insights about why and where
> this is really useful. So I’d like to share those with you and gather some
> opinions whether you think we should try to include these extensions in the
> Python core.
>
> *Intro*
>
>   What we have done is (mostly) extended Python’s comprehensions with
> group by, order by, let and window clauses, which can come in any order,
> thus comprehensions become a query language a bit cleaner and more powerful
> than SQL. And we added a couple small convenience extensions, like a  We
> have identified three top motivations for folks to use these extensions:
>
> *Our Motivations*
>
> 1. This can become a standard for running queries against database
> systems. Instead of learning a large number of different SQL dialects (the
> pain point here are libraries of functions and operators that are different
> for each vendor), the Python developer needs only to learn PythonQL and he
> can query any SQL and NoSQL database.
>
> 2. A single PythonQL expression can integrate a number of
> databases/files/memory structures seamlessly, with the PythonQL optimizer
> figuring out which pieces of plans to ship to which databases. This is a
> cool virtual database integration story that can be very convenient,
> especially now, when a lot of data scientists use Python to wrangle the
> data all day long.
>
> 3. Querying data structures inside Python with the full power of SQL (and
> a bit more) is also really convenient on its own. Usually folks that are
> well-versed in SQL have to resort to completely different means when they
> need to run a query in Python on top of some data structures.
>
> *Current Status*
>
> We have PythonQL running, its installed via pip and an encoding hack, that
> runs our preprocessor. We currently compile PythonQL into Python using our
> executor functions and execute Python subexpressions via eval. We don’t do
> any optimization / rewriting of queries into languages of underlying
> systems. And the query processor is basic too, with naive implementations
> of operators. But we’ve build DBMS systems before, so if there is a good
> amount of support for this project, we’ll be able to build a real system
> here.
>
> *Your take on this*
>
> Extending Python’s grammar is surely a painful thing for the community.
> We’re now convinced that it is well worth it, because of all the wonderful
> functionality and convenience this extension offers. We’d like to get your
> feedback on this and maybe you’ll suggest some next steps for us.
>
> *References*
>
> PythonQL GitHub page: https://github.com/pythonql/pythonql
> PythonQL Intro and Tutorial (this is all User Documentation we have right
> now): https://github.com/pythonql/pythonql/wiki/
> PythonQL-Intro-and-Tutorial
> A use-case of querying Event Logs and doing Process Mining with PythonQL:
> https://github.com/pythonql/pythonql/wiki/Event-Log-Querying-and-Process-
> Mining-with-PythonQL
> PythonQL demo site: www.pythonql.org
>
> Best regards,
> PythonQL Team
>
>
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170325/df5c7565/attachment.html>