[Python-Dev] Language Summit notes

[Nick Coghlan]

On 18 April 2014 08:04, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:
> Here, a week later, are some of my thoughts from the summit, for the record:
> 1.       An aid in the conversion from 2.x series to 3.x series.  Enabling a
> bunch of warnings and such by default.  Perhaps allowing 3.x syntax in some
> places without fuss.  The problem with this idea is that it is pointless.
> Why would anyone want to upgrade from 2.7 to 2.8 if all they get is some new
> warnings for 3.x?  If people are willing to make a version upgrade just to
> get new warnings (i.e. no immediate feature benefit) they might as well go
> directly to 3.x and be done with it.

Right, this is much better handled through tools like linters, import
hooks, custom interpreters, additional support modules, etc. I think
http://python-future.org/ represents the current state of the art, but
there are still a lot of opportunities for improvements here.

> 2.       Feature enhancement to 2.8.  Take a robust and popular version of
> python and add some of the language goodies that have been added to 3.x and
> that don’t have an inherent 3.x aspect.  Yield from.  New exception model.
> Stdlib enhancements such as futures.   The argument goes like this:  We have
> a very popular platform out there with lots of momentum.  People want
> incremental enhancements to it.  Why not give them what they want?  Bread
> and games and all that?  A Rockband cannot stay cooped up in a studio
> producing experimental concept albums all the time.  That is death.
> Sometimes it needs to go on tour and play old hits for the fans!

Do you know how much work a new Python 2.x release creates for people?
All the redistributors have to update, books get outdated, a new
wrinkle gets added to the compatibility matrix for everyone. A new
Python 2.x release is simple untenable at this point in the transition
- it's a *massively* expensive way to achieve things that can be
achieved more cheaply in other ways.

Take yield from, for example. Hy is able to compile *LISP* syntax to
Python AST structures. PEP 380 includes a semantic expansion of yield
from in terms of yield. Is it *really* impossible to get "yield from"
based code running in Python 2.6? Or have people just assumed it's not
possible and never even tried, because the idea of using import hooks
to backport syntax to earlier feature releases is too novel?

(importlib includes tools to make this kind of thing relatively
straightforward, and there's nothing inherently impossible about using
importlib to write import hooks in Python 2 - it's just a matter of
having to do the backport first, and Eric Snow is now seriously
considering doing exactly that. Hit us up on import-sig if you think
that might be interesting to you)

> 3.5 features
>
> When asked what should we aim for in 3.5, there were mostly some very minor
> incremental changes suggested, IIRC.  In my opinion, the reason 3.x has not
> caught on is that there is no real carrot there.  There is no new vision, no
> killer feature.  Nothing that a programmer sees and makes him say “Yeah! I
> want to program my next project using this feature, it will be super!”.

I *really* wish folks from North America, Europe and other regions
where 8-bit encodings can handle their native language and where
Anglicisation of terms to fit them into the ASCII identifier
restriction poses no barrier to communication would stop trotting out
this "no killer feature in Python 3" canard. While it is *possible* to
write internationalised and localised applications in it, Python 2's
Unicode support is so broken that some people can't even run the
interpreter from their home directory because it can't cope with their
username. Python 3 makes Unicode support in the interpreter pervasive
(just as it is in the JVM and CLR), and we've spent the last 5+ years
dealing with the latent Unicode handling defects this uncovered in the
standard library, fixing mistakes we made in the initial transition,
and figuring out how to get the new system to play nice with the POSIX
"text is just bytes in a particular encoding" model, as well as
figuring out what we really lost in the transition when it comes to
binary data manipulation.

If anyone is *ever* tempted to utter the words "Python 3 has no killer
feature" without immediately following it up with the "for me"
qualifier, please go read this post about the creation of a Portuguese
version of Stack Overflow:
http://blog.stackoverflow.com/2014/02/cant-we-all-be-reasonable-and-speak-english/

Late last year, Alex Gaynor made the insightful observation that we
write software either for the users we have or for the users we want.
Python 3 is the software we're writing for the users we want. Python
2.7 long term maintenance, PyPy, standard library backports,
transition libraries and tools, packaging ecosystem improvements,
Python 3 features designed primarily to easy migration from Python 2,
PyPI modules that run on both Python 2 & 3 - that's the software we
and many other people are writing for the users we have.

Sure, there's not yet a killer Python 3 feature for many developers
that *were already happy with Python 2.7* (although matrix
multiplication syntax in 3.5 will likely reach that bar for a subset
of users, and chained exceptions are already pretty amazing for
debugging obscure failures that then run into broken error handlers).
That's fine - that's why Python 2.7 is now being supported upstream
out to 2020. It's also why lots of things are already available as
backports on PyPI (if any of us as core devs identify a Python 3
library that *we* think is a killer feature, we'll often maintain a
backport for it, since many of us are still maintaining Python 2
systems in our day jobs), and we're going to work on ensuring that
those are easier to access for new users. Two of the best projects
people can actually work on that front are actually
https://github.com/python/pythondotorg and
http://packaging.python.org/, to help enable them to become effective
gateways to the entire Python *ecosystem*, rather than having people
downloading CPython and assuming the reference interpreter and the
standard library are all they have available to them.

> In my opinion we should be thinking more boldly.  Either for 3.x or for a
> version 4.  We should be taking the language to a new level.  Thinking about
> evolving the language.  New paradigms.   Look at what C# is doing, with each
> language revision.  Look at Go.  I’m no CS but here are some ideas on stuff
> we could visit:
>
> 1.       Code blocks as a core language construct.  Re-implement context
> managers as block executors.  We shouldn’t let details such as syntax
> questions distract us.   That’s like saying that we can’t  eat spaghetti
> because our Italian is so poor.

When it comes to blocks, the syntax isn't a distraction - it's the
fundamental framing of what the construct is and how it should be
used. How does a block get used to define a sorting key function? How
does it get broadcast over a NumPy array? How does it get distributed
to multiple threads, processes or greenlets, or passed to an IO
scheduler? How does it differ from an ordinary closure? How does it
relate to loops and other constructs?

There is a large existing constellation of concepts that blocks need
to be fitted into, and any successful proposal for adding blocks to
the language will require giving that process the respect and
attention it deserves.

> Proper code blocks would open up new
> avenues for exploration of expressability and paradigms.

This won't happen without a full usability study, but I've been
speaking to Fernando Perez and Greg Wilson about some vague concepts
for doing exactly that. I hope to be able to explore that possibility
further at SciPy in July - Greg laments the lack of "evidence based
language design" and I think the strong scientific presence in the
Python community provides us with a genuine opportunity to move in
that direction when it comes to controversial changes like code
blocks.

> 2.       Concurrency primitives built into the language.  Again, see C# with
> its “async” keyword  (a feature we’ve experimented with in stacklesslib, see
> e.g. stacklesslib.async in https://bitbucket.org/stackless-dev/stacklesslib
> ).  Look at Go with its channels and more importantly, the select feature.
> ( see goless, http://goless.readthedocs.org/en/latest/index.html a 2014
> sprint project).  Don’t get distracted by the GIL.  Concurrency is as much
> about orchestration of operations as it is about parallel execution of code.
> Let’s learn from the success of stackless, gevent, go, and build on top of
> it by absorbing tried and tested research from more than 30 years of CS.

I'm personally letting Armin Rigo spearhead that effort (via his STM
work), as I believe PyPy provides a better foundation for that
exploration.

It also doesn't make sense to me to do this at a language level until
we resolve the blocks question, since that opens up many new
possibilities for parallelisation that won't even need new syntax at
that point.

> These are the immediate ideas rolling off the top of my head.  Notice how I
> don’t mention “removing the GIL” here since that is not a “language feature”
> as such, not something inspiring new thinking and invention.  Of course a
> non-GIL implementation is also desirable, even if it would involve
> completely rethinking the C API.  For a version 4 of python.  But I think we
> thinking beyond that, even.

If people are genuinely interested in building and supporting the
Python *community*, stop thinking about tinkering with the *language*.
Tinkering with the language is *fun* (that's why I still do it), but
the most interesting innovations in the Python world with the largest
near term impact *aren't going to happen in the reference
interpreter*. Our cycle times are too long, especially once commercial
redistributors are taken into account, and the ongoing popularity of
Python 2.7 shows that the language itself reached "good enough" status
for most use cases years ago. As technologists we like to tinker with
out toys, but it's important to remember that *Excel spreadsheets* are
far more dominant as a programming paradigm than any open source
programming language.

So folks looking to core development as the engine room of Python's
growth are *looking in the wrong place*. We laid the foundation (and
will continue to sustain and evolve it), but the real growth drivers
have now moved further out to things like the work the PyPA is doing
to make upstream packaging easier to consume, the work Travis Oliphant
and others are doing to make the scientific Python stack easier to
consume, the work Fernando Perez is coordinating around IPython and
IPython notebook, the work Titus Brown et al are doing around
reproducable science, the work Greg Wilson and Mozilla Science Labs
are doing around Software Carpentry, the work Van Lindberg and others
are putting into opening up the PSF, the work Jesse Noller has been
driving around opening up python.org to more contributors, the work
Jessica McKellar and more are doing around getting Python into high
school curricula in the US, the work the Raspberry Pi Foundation are
doing in the UK, the work around the core mentorship program, GSoC and
the Gnome Outreach Program for Women, the work to build the regional
network of Python conferences (I'm told we're currently running at a
rate of something like 2 new PyCons per week!), the work folks like
me, Guido, Steve Dower, Jesse and more are doing to better engage with
corporate users and inviting them to more directly support the
sustainability of the upstream Python community (on *our* terms), the
work PyLadies is doing around education and outreach, the work Steve
Holden is now doing around improving the ready availability of Python
training courses, and on, and on, and on.

We are *far* past the point where any specific language feature will
be the defining aspect of language adoption - it's all about the
ecosystem, and has been for years. Discoverability and usability and
building relationships with external groups - that is how you make the
leap across the chasm from early adopters to mass market popularity.
Not tinkering with syntactic details - do the latter for fun and to
help people years down the track, but do other things (like helping
with packaging or the core development workflow tools or
communications projects like Mailman 3 and the HyperKitty archiver) if
the long game doesn't excite you and you would like to make a more
immediate impact on the world.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia