bytecode non-backcompatibility

[Mike Meyer]

Maurice LING <mauriceling at acm.org> writes:

>>>5. Unstable bytecodes makes updating to a newer version of Python very
>>>tedious and risk breaking old scripts, if they uses C modules.
>> Unstable bytecodes have nothing to do with these problems. The
>> current CPython installation process puts the python command and
>> the libraries for different versions in different directories. This
>> allows you to have multiple versions installed so you can keep old
>> scripts working with the old version until you've had time to test
>> them. It also makes life much easier on the developers, as they can
>> have a development version installed on the machine at the same time
>> as they have a production version without breaking the old scripts. It
>> also means you have to reinstall all your modules on the new
>> installation - which is what makes the update process tedious for me.
>> Now, this could be mitigated by having Python libraries installed in
>> the same location for all versions. You could fix all the bytecode
>> files by running compileall.py as a script. Sometimes, a module won't
>> work properly on a new version of Python, and will have to be
>> updated. You'll have to find those by trial and error and fix them as
>> you find them. You'll also have to recompile all the C modules, which
>> will break running scripts on the old interpreter.
>> Under the currewnt system, old scripts that are run by the new
>> interpreter will break if all the modules they need aren't installed
>> yet. Once they're installed, the scripts should work.
>> If you have scripts that use C modules that aren't installed in the
>> standard Python search path, they may well break on the new
>> interpreter until you recompile them. Having both versions of the
>> interpreter available makes this tolerable.
>> The cure I proposed seems worse than disease. If you've got a better
>> solution, I'm sure we'd be interested in hearing it.
>>
>
> Perhaps this is another call for Python version of CPAN (CPyAN or
> PYAN). It can be modelled after Fink or Darwinports.
>
> If I remembered correctly, Fink uses apt-get and curl.
>
> What can be done in PYAN is to encourage all 3rd party library
> developers to centralize their libraries in it, which I think all will
> gladly respond. All that is needed to be  deposited in PYAN is a
> description file, like the simplest setup.py file. All that is needed
> in this description file is
> 1. where to download the source codes (e.g. CVS)?
> 2. version number?
> 3. standard stuffs (optional) like authors, descriptions, copyright?

To copy all of what CPAN does, you need to know what other modules it
depends on, so you can automatically download those as well.

> Python can then have a built-in mechanism to read the description file
> and download the source codes and do the standard "sudo python
> setup.py install" to install the library into site-package.

I don't like this - it would make Python depend on sudo being
available. I'd rather it not do that, and let each systems
administrator issue the command according to *their* security policy.

> I think something like this can be set up for Python quite easily. I
> recall some time back, there's a long discussion about setting up
> Python's version of CPAN but I can't recall the contents of the
> discussions.

It's not clear it's easy. It's *very* clear it won't happen until
someone steps up to do it. The latter is a problem. 

>> The dependencies problem is actually pretty easy to solve. In fact,
>> in
>> many Python environments, it's already solved. On my system, if I
>> install Eric3 using the provided installation package, the
>> dependencies will be picked up automatically. Not very helpfull if
>> you're not on such a system, I know. The long-term solution is for
>> PyPI to grow to include this functionality.
>
> I must say that I do not quite understand your system. Please enlighten me.

FreeBSD has a packaging system (like fink on the Mac) called either
"Ports" (if you build from source) or "Packages" (if you install
pre-compiled binaries). I use Ports, so I'm going to talk about
them. But Packages are built from Ports, and all the information in
the port winds up in the package, so you get the same functionality
with a slightly different command set.

Basically, each port has a list of all the other ports it depends
on. When you install a port, the system will check to see if each of
the dependent ports is installed - usually by checking if a file it
should install exists - and if it doesn't, will install the dependent
port. The dependencies of that port are recursively checked. So if I
were to install the Eric3 port, it would note that I need the
qscintilla, py-sip and py-qt (ports of python modules are all named
"py-<something>"). py-sip and py-qt are already installed, so it would
skip thos. qscintilla isn't, so that would get installed. qscintilla
doesn't have any dependencies, so that would stop there. After
qscintilla is installed, eric3 would be installed.

All of this is recorded in the Packages database, and I can query that
to find out what python modules are installed, which module installed
a specific file, etc. I can also use that to cleanly remove the
module, which the python setup.py doesn't do for me.

I'm not familiar with fink, but I suspect it offers much the same
functionality. rpm's and apt-get offer that functionality for many
Linux systems. Solaris has it's own package system as well.

Basically, for common Unix platforms, the dependency problems you ran
into with Eric3 should be solved by the platforms package system,
assuming that Eric3 is available as a package.

        <mike
-- 
Mike Meyer <mwm at mired.org>			http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.