[Python-ideas] Proposal: Query language extension to Python (PythonQL)

Sun Mar 26 06:50:22 EDT 2017

Hi David

> On 25 Mar 2017, at 20:08, David Mertz <mertz at gnosis.cx> wrote:
> 
> I think it's extraordinarily unlikely that a big change in Python syntax to support query syntax will ever happen.  Moreover, I would oppose such a change myself.
> 
> But just a change also really is not necessary.  Pandas already abstracts all the things mentioned using only Python methods.  It is true that Pandas sometimes does some black magic within those methods to get there; and it also uses somewhat non-Pythonic style of long chains of method calls.  But it does everything PythonQL does, as well as much, much more.  Pandas builds in DataFrame readers for every data source you are likely to encounter, including leveraging all the abstractions provided by RDBMS drivers, etc.  It does groupby, join, etc.
> 

I work daily with pandas, so of course it does have the functionality that PythonQL introduces, its a completely different beast. One of the reasons I started with
PythonQL is because pandas is so difficult to master (just like any function-based database API would).

The key benefit of PythonQL is that with minimal grammar extensions you get the power of a real query language. So a Python programmer who knows 
comprehensions well and has a good idea of SQL or other query languages can start writing complex queries right away.

With pandas you need to read the docs all the time and complex data transformations become incredibly cryptic.

> See, e.g.:
> 
>     http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html>
> 
> Now there's one reasonable objection to Pandas: It doesn't handle larger-than-memory datasets well.  I don't see that PythonQL is better in that regard.  But there is an easy next step for that larger data.  Blaze provides generic interfaces to many, many larger-than-memory data sources.  It is largely a subset of the Pandas API, although not precisely that.  See, e.g.:
> 
>     http://blaze.readthedocs.io/en/latest/rosetta-sql.html <http://blaze.readthedocs.io/en/latest/rosetta-sql.html>
> 
> Moreover, within the Blaze "orbit" is Dask.  This is a framework for parallel computation, one of whose abstractions is a DataFrame based on Pandas.  This gives you 90% of those methods for slicing-and-dicing data that Pandas does, but deals seamlessly with larger-than-memory datasets.  See, e.g.:
> 
>     http://dask.pydata.org/en/latest/dataframe.html <http://dask.pydata.org/en/latest/dataframe.html>
> 
> So I think your burden is even higher than showing the usefulness of PythonQL.  You have to show why it's worth adding new syntax to do somewhat LESS than is available in very widely used 3rd party tools that avoid new syntax.
> 
> On Fri, Mar 24, 2017 at 8:10 AM, Pavel Velikhov <pavel.velikhov at gmail.com <mailto:pavel.velikhov at gmail.com>> wrote:
> Hi folks!
> 
>   We started a project to extend Python with a full-blown query language about a year ago. The project is call PythonQL, the links are given below in the references section. We have implemented what is kind of an alpha version now, and gained some experience and insights about why and where this is really useful. So I’d like to share those with you and gather some opinions whether you think we should try to include these extensions in the Python core.
> 
> Intro
> 
>   What we have done is (mostly) extended Python’s comprehensions with group by, order by, let and window clauses, which can come in any order, thus comprehensions become a query language a bit cleaner and more powerful than SQL. And we added a couple small convenience extensions, like a  We have identified three top motivations for folks to use these extensions:
> 
> Our Motivations
> 
> 1. This can become a standard for running queries against database systems. Instead of learning a large number of different SQL dialects (the pain point here are libraries of functions and operators that are different for each vendor), the Python developer needs only to learn PythonQL and he can query any SQL and NoSQL database.
> 
> 2. A single PythonQL expression can integrate a number of databases/files/memory structures seamlessly, with the PythonQL optimizer figuring out which pieces of plans to ship to which databases. This is a cool virtual database integration story that can be very convenient, especially now, when a lot of data scientists use Python to wrangle the data all day long.
> 
> 3. Querying data structures inside Python with the full power of SQL (and a bit more) is also really convenient on its own. Usually folks that are well-versed in SQL have to resort to completely different means when they need to run a query in Python on top of some data structures.
> 
> Current Status
> 
> We have PythonQL running, its installed via pip and an encoding hack, that runs our preprocessor. We currently compile PythonQL into Python using our executor functions and execute Python subexpressions via eval. We don’t do any optimization / rewriting of queries into languages of underlying systems. And the query processor is basic too, with naive implementations of operators. But we’ve build DBMS systems before, so if there is a good amount of support for this project, we’ll be able to build a real system here.
> 
> Your take on this
> 
> Extending Python’s grammar is surely a painful thing for the community. We’re now convinced that it is well worth it, because of all the wonderful functionality and convenience this extension offers. We’d like to get your feedback on this and maybe you’ll suggest some next steps for us.
> 
> References
> 
> PythonQL GitHub page: https://github.com/pythonql/pythonql <https://github.com/pythonql/pythonql>
> PythonQL Intro and Tutorial (this is all User Documentation we have right now): https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial <https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial>
> A use-case of querying Event Logs and doing Process Mining with PythonQL: https://github.com/pythonql/pythonql/wiki/Event-Log-Querying-and-Process-Mining-with-PythonQL <https://github.com/pythonql/pythonql/wiki/Event-Log-Querying-and-Process-Mining-with-PythonQL>
> PythonQL demo site: www.pythonql.org <http://www.pythonql.org/>
> 
> Best regards,
> PythonQL Team
> 
> 
> 
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <mailto:Python-ideas at python.org>
> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas>
> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
> 
> 
> 
> 
> -- 
> Keeping medicines from the bloodstreams of the sick; food 
> from the bellies of the hungry; books from the hands of the 
> uneducated; technology from the underdeveloped; and putting 
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170326/270d23fa/attachment.html>