[Python-Dev] Packaging and binary distributions for Python 3.3

[Vinay Sajip]

Paul Moore <p.f.moore <at> gmail.com> writes:

> Interesting. That's not a use case that I have encountered, but I can
> see it could be an issue. I have been working on the basis that a
> bdist_simple format that matches the functionality of bdist_wininst is
> at least good enough for those projects that currently use
> bdist_wininst. The only potential issue right now is with postinstall
> scripts, which bdist_simple doesn't support. It would be easy enough
> to add, and indeed it may be possible to use existing packaging
> functionality already (I haven't looked into this area).

The packaging machinery contains a reasonably good set of hooks which a 
setup.cfg can plug into, which is IMO more flexible than just a post-
installation script (e.g. sometimes you need your code to run before 
installation, to determine where to install things).
 
> I don't disagree, but I'm struggling to see how that would be done.

See example below.
 
> Can you give an example of a setup.cfg, that would work like this?
> Suppose I have two files, foo.py and bar.pyd, which are a pure-python
> module and a compiled C extension. How would I write a setup.cfg and
> lay out the directory structure to allow pysetup install to do the
> right thing? I tried to do this myself, but couldn't get it to work
> the way I expected. It may be I was just hitting bugs, but it felt to
> me like I was going at the problem the wrong way.

It may not work for you, because in the default branch, packaging is currently 
missing some functionality or has bugs (I've raised about a dozen issues since 
trying to get packaging working with virtual environments).

In the pythonv branch (which is pretty up to date with default), I've added the 
missing functionality/fixed some of the issues. Here's an example: I've created 
an empty virtual environment. Here are the contents of it at the moment:

C:\TEMP\VENV
│   env.cfg
│
├───Include
├───Lib
│   └───site-packages
└───Scripts
        activate.bat
        deactivate.bat
        pysetup3-script.py
        pysetup3.exe
        [various stock Python binaries omitted]

Here's an example setup.cfg:

[global]
setup_hooks = hooks.setup

[install_data]
pre-hook.win32 = hooks.pre_install_data

[metadata]
name = dory
version = 0.1
requires_python = >= 3.3
[other metadata omitted]

[files]
modules = dory
packages = apackage
    apackage.asubpackage

scripts = dory

extra_files =
    hooks.py

resources =
    data/data.bin = {data}
    compiled/ spam.pyd = {compiled}

#[extension: spam]
#language = c
#sources = spammodule.c


The extension section is commented out because we're not building the extension 
at installation time. This setup.cfg works hand-in-hand with some hooks, in 
file hooks.py below:

def setup(config):
    # Can do various checks here. For example, platform
    # compatibility checks, or set up different binaries
    # to install for x86 vs. x64, etc.
    # Do this by setting up config['files']['resources']
    # appropriately based on installation time environment.
    pass

def pre_install_data(cmd):
    # assumes os.name == 'nt' for simplicity of this example
    cmd.categories['compiled'] = '%s/Lib/site-packages' % cmd.install_dir

Notice how in setup.cfg, file 'spam.pyd' in 'compiled' is expected to be copied 
to category 'compiled', whose path is set in hooks.pre_install_data.

Here's the project directory:

C:\USERS\VINAY\PROJECTS\DORY
│   dory
│   dory.py
│   hooks.py
│   setup.cfg
│
├───apackage
│   │   __init__.py
│   │
│   └───asubpackage
│           __init__.py
│
├───compiled
│       spam.pyd
│
└───data
        data.bin

At the moment, the "spam" module is of course not installed:

(venv) c:\Users\Vinay\Projects\dory>python -c "import spam"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'spam'

Now, we install the project:

(venv) c:\Users\Vinay\Projects\dory>pysetup3 install .
Installing from source directory: c:\Users\Vinay\Projects\dory
running install_dist
running build
running build_py
running build_scripts
copying and adjusting dory -> build\scripts-3.3
running install_lib
creating C:\temp\venv\Lib\site-packages\apackage
byte-compiling C:\temp\venv\Lib\site-packages\apackage\__init__.py to __init__.c
python-33.pyc
byte-compiling C:\temp\venv\Lib\site-packages\dory.py to dory.cpython-33.pyc
running install_scripts
running pre_hook hooks.pre_install_data for command install_data
running install_data
running install_distinfo
creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info
creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\METADATA
creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\INSTALLER
creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\REQUESTED
creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\RESOURCES
creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\RECORD

Now, the virtualenv looks like this:

C:\TEMP\VENV
│   env.cfg
│
├───data
│       data.bin
│
├───Include
├───Lib
│   └───site-packages
│       │   dory.py
│       │   spam.pyd
│       │
│       ├───apackage
│       │   │   __init__.py
│       │   │
│       │   ├───asubpackage
│       │   │   │   __init__.py
│       │   │   │
│       │   │   └───__pycache__
│       │   │           __init__.cpython-33.pyc
│       │   │
│       │   └───__pycache__
│       │           __init__.cpython-33.pyc
│       │
│       ├───dory-0.1.dist-info
│       │       INSTALLER
│       │       METADATA
│       │       RECORD
│       │       REQUESTED
│       │       RESOURCES
│       │
│       └───__pycache__
│               dory.cpython-33.pyc
│
└───Scripts
        activate.bat
        deactivate.bat
        dory-script.py
        dory.exe
        pysetup3-script.py
        pysetup3.exe
        [various stock Python binaries omitted]

Now we can import "spam" and run something from it:

(venv) c:\Users\Vinay\Projects\dory>python -c "import spam; spam.system('cd')"
c:\Users\Vinay\Projects\dory

> I'd like to see a bdist_xxx command to do the build step
> as you describe, if only to make it trivially simple for developers to
> produce binary distributions. Having to package stuff up manually is
> bound to put at least some developers off. If you can give me the
> example I mentioned above, I could work on modifying the bdist_simple
> code I posted to the tracker today to produce that format rather than
> my custom format based on bdist_wininst.

Example as above, though you may need to use the pythonv branch to actually get 
it working. I can zip up the directory and send it to you, but at the moment 
there's missing functionality in pythonv in terms of the link step when 
building the extension. (I overcame this by linking manually .) If you want, I 
can zip all the project files up and send them to you.

In the more general case, one might want an arrangement with a directory 
structure like compiled/x86/..., compiled/x64/... in the built zip, with the 
hooks.py code setting up the resources appropriately based on the target 
environment as determined at installation time.

> For the installation step, you shouldn't even need to unzip, as
> pysetup3 can do the unpacking for you.

Indeed, with pythonv I could just zip the whole "dory" project directory and 
install with e.g. "pysetup3 install dory-1.0-win32-py3.3.zip".

> Agreed. Personally, as I've said, I'm happy not to use Add/Remove even
> for system installations - pysetup list and pysetup remove do what I
> need without slowing down the Add/Remove list. But I accept that's not
> likely to be the view of many Windows users. Anyone using virtual
> envs, though, is probably by definition comfortable enough with
> command line tools to be willing to use pysetup3.

A fair subset of those who must have ARP integration will probably also want to 
install using MSI, so that would be taken care of by having a good bdist_simple 
-> bdist_msi conversion.

Regards,

Vinay Sajip