Python grammar extension via encoding (pyxl style) questions

[Pavel Velikhov]

Hi everybody!

    We’re building an experimental extension to Python, we’re extending Python’s comprehensions into a full-scale query language. 

And we’d love to use the trick that was done in pyxl, where a special encoding of the file will trigger the preprocessor to run and compile our extended language into Python and execute it.

The actual encoding business is a bit tricky, but we also really really would like the whole thing to be installable via pip, otherwise only a tiny percentage of our potential users (we're targeting data scientists) would go through the pain of installing the extension. So I have some question, would really appreciate any help:

  - Are there any resources to get up the speed on custom encodings? The pyxl code looks fine, just not clear what some of the steps are. 

  - We’d love to have a pip install for our module, but even pyxl didn't do it (i.e. you clone the repository, then do a local pip install, and then run a script). Is it impossible to just do it all with pip? Would it be a terrible/horrible hack? Any hints on how to do this?

  - Last question - we’d love to enable support for our extension in interactive python and especially in Jupyter notebooks. Would you have any advise on how to approach these?


Thanks in advance!
Pavel Velikhov
P.S. Here's the demo site for our project,if you're interested: www.pythonql.org