[Python-3000] [Python-ideas] Namespaces are one honking great idea -- let's do more of those!

[Nick Coghlan]

Ralf W. Grosse-Kunstleve wrote:
> Nick Coghlan wrote:
> 
>> I personally haven't seen anything to convince me that the 2.x -> 3.0 
>> upgrade cycle is going to be significantly worse from a deployment point 
>> of view than a 2.x -> 2.(x+2) upgrade cycle where breakages are also 
>> possible (e.g. code using 'with' or 'as' as identifiers runs just fine 
>> on 2.4 or 2.5, but that code is going to break in 2.6).
> 
> The 2.x -> 3.0 transition will be *very* different. We're using Python
> since 2000, version 1.5.2. I've always upgraded to the latest release
> and got it working with a few fairly minor tweaks. Importantly, I
> could do this without breaking backward compatibility if I wanted to.
> At some point we had to give up on 1.5.2 compatibility, but that was
> just because of Boost.Python requirements (only works with new-style
> classes).
> 
> IIUC, there is no conservative upgrade path for Python 3.0. Once the
> transition is done, the code will only work with 3.0. The bridge to
> older Python versions is completely broken. Python 2.x will not even
> parse most Python 3 scripts and vice versa. That's a situation we never
> had before.

I still see this as a difference in degree rather than a difference in 
kind. As soon as a developer puts "from __future__ import 
absolute_imports" or "from __future__ import with_statement" in their 
script every version of Python prior to 2.5 is going to refuse to even 
compile it. If they provide an old script which uses an identifier which 
is now a keyword, it is the more recent versions which are going to 
object violently.

Either way, the solution is the same: run that particular script with a 
version of the interpreter that can handle it correctly. Make that 
happen by whatever means your platform makes available (on Windows it 
usually means batch files, on Unix'y systems a version specific shebang 
line or a shell script).

> Even though the scripts look very similar at face value,
> at the parsing level they are as incompatible as e.g. perl and Python.

While there is almost certainly going to be incompatibility at the 
module level, it is perfectly possible to have compatibility at the 
application level. Granted, obtaining it won't be anywhere near as easy 
as it is when staying within the 2.x series, but it is far from a lost 
cause.

Specifically, the application may need a launcher module that fiddles 
with sys.path or package __path__ entries based on the running Python 
version. Alternatively, the application may be installed into 
site-packages and executed using the -m switch so that the interpreter 
automatically picks up the correct version of the application for the 
interpreter which is running.

(Note again that all of these issues are only particularly difficult to 
deal with in the case of applications which are written in Python and 
intended to be used on a system someone else controls. When the same 
entity controls both the application and the runtime environment - such 
as web services, embedded systems, or applications that bundle their own 
Python interpreter - it is merely necessary that the environment is kept 
in sync as the needs of the code change)

> As Python developers you have to realize that for many people Python is
> "just" a core around which they build much bigger applications.

Strange as it may seem, at least some of us do exactly that as part of 
our day jobs ;)

> If the
> success of Python is to continue, you have to give your large user
> base a clear path to working with two co-existing Python versions,
> so that installing a Python 3 application doesn't break all Python 2
> applications (some of which may never be converted since the original
> developers have moved on).

You mean the way you can already have Python 1.5, 2.0, 2.1, 2.2, 2.3, 
2.4 and 2.5 all installed on the same machine, and invoke different 
versions for different applications?

Yes, only one of them can be the 'default Python' for the machine, and 
changing that can be an interesting exercise, but that is hardly a 
problem that is limited to Python - it's the case for *any* scripting 
language interpreter. To choose an example which affected me relatively 
recently, how many shell scripts did the Ubuntu folks break when they 
decided to change their default shell from bash to dash "because it is 
faster" instead of explicitly invoking dash for the particular scripts 
they wanted to speed up? That was one configuration change I reverted 
pretty darn quickly, because there are a hell of a lot of shell scripts 
out there which rely on bash extensions, even if the developers didn't 
realise it when they wrote the scripts.

While I'm +1 on the idea of making altinstall the default for setup.py 
and the Windows installer (forcing the administrator to explicitly 
request that the new Python version be made the default Python 
installation for the machine), I'm a pretty strong -1 on officially 
blessing the idea of a python3 alias or a special py3 extension on 
Windows (end users are obviously free to create those if they want, of 
course).

Who knows, with the rate at which virtual machine technology is 
improving, this whole thing may become a non-issue by the time the 3.x 
series is even ready for production use. One VM with a 2.x default 
Python, another VM with a 3.x default Python, various applications 
associated with each, and seamless integration into the underlying 
platform so the user never even notices a VM is involved.

(OK, I don't actually hold out much hope of that particular scenario 
coming to pass in the next few years, but I'm trying to point out here 
that you're pushing for a language level solution to a system level 
problem: how do we reliably associate files with an application that 
understands them? What do we do when different versions of that 
application permit the files to contain different things? Yes, these 
issues matter for the 2.x->3.x transition, but moving to a new 
interpreter name and a new file extension is nothing more than a 
bandaid, and a pretty ugly one that we would have to live with for a 
long time)

> For example, I really need to be able
> to use wxPython based on Python 2 and wxPython based on Python 3
> simultaneously on my Windows machine. I'm thinking this is only
> possible if Python 3 uses a new file extension.

And how much trouble do you have using wxPython with 2.4 and 2.5 on the 
same machine? They're completely different extension modules, and they 
can live side by side on one machine quite happily (it's one of the main 
benefits of installing things to site-packages). That behaviour won't 
change just because one of those Python version numbers now has a 3 at 
the start.

Making heavy use of PYTHONPATH can cause problems, but again that's the 
result of a platform limitation, where it isn't easy to get different 
applications to see different values for environment variables without 
the use of launch scripts.

> Ideally, I'd even want to have both Python 2 and Python 3 scripts in
> the same directory, while incrementally converting our applications.
> I'd want to cp script.py script.py3, then edit the .py3 version
> until it works, while still being able to run the old version during
> the migration period.

Sure, you can do it that way if you really want to, but that isn't the 
recommended way.

And invoking Python 3 for the scripts that need it would be the same as 
invoking any other specific Python version (batch file or shortcut on 
Windows, version-specific shebang line or shell script on POSIX).

> Of course, I could also start a new Python 3
> specific directory structure, but then I'd have to jump back and
> forth between two complex trees all the time, which is very annoying
> and time consuming; and I may have to copy many non-Python files that
> I wouldn't have to maintain in two copies otherwise.

The intent is that you should never need to edit files directly in both 
trees - instead, it should be possible to work almost entirely in the 
Python 2 tree, with the 2to3 tool automatically creating the Python 3 
tree for you (kind of like a cross-compilation step that accepts valid 
Python 2 code and emits the corresponding Python 3 code). Any test 
failures (or other problems) in the Python 3 version would be fixed by 
modifying the Python 2 source and reconverting.

Having an option to ask the 2to3 tool to automatically copy any 
non-Python files it finds might be an interesting idea, although using 
an existing file synchronisation tool like rsync or a version control 
branch (and then running 2to3 to make modifications after the other 
files are in sync) would probably be a better one.

> I think it will be essential to give the Python 3 interpreter a new
> name and the scripts a new extension. I suggest using "py3" for both.

And I think that would be an ugly hack that we would then be stuck with 
for the life of the 3.x series.

Would the renamed python3 still be looking at all the same environment 
variables (PYTHONPATH, PYTHONHOME, etc)? Or would all those have to be 
renamed as well? Should import change to accept .py3 files? If not, why 
not (after all, the line between execution and import gets pretty blurry 
where the -m switch and the runpy module are concerned)? If yes, should 
import be changed to not accept .py files? When python3 was asked to 
execute a directory or zipfile, would it look for __main__.py3, 
__main__.py or either?

Also note that while it currently will be feasible to write quite 
substantial programs that run on both Python 2 and Python 3 without 
modification (albeit only with a total avoidance of the print statement 
and function, as well as taking a great deal of care in remaining 
str/unicode agnostic), some of the possible answers to the questions 
posed in the previous paragraph would make it downright impossible.

The mechanisms that have been used to date to support multiple Python 
versions on a single system have worked pretty well, and I don't believe 
that they're suddenly going to start failing horribly just because the 
permitted degree of backward incompatibility between 2.x and 3.0 is 
higher than that between 2.x and 2.(x+2).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org