[Distutils] libcollect - collecting all the libraries used by a script

[Eli Bendersky]

Hello,

Motivation:

Imagine that you've written a script that uses several libraries, some of
which you've written and some you've downloaded and installed (for example
PyYAML). You want to distribute the script to your friends and co-workers,
who already have Python installed with all the standard library. But your
script won't run on their machines, because they have neither your personal
libraries, nor PyYAML installed. So what can you do ?
* You can ask them to install PyYAML and other libraries your script uses,
  and send them your own libraries. This is a lengthy and inconvenient
  process.
* You can use a tool like py2exe to package your delivery. This has a
  downside, however. py2exe produces large files (several MBs) and you may
  not want that.
* You can painstakingly collect the libraries into a directory where your
  script can find them, and package the directory together with the script.

I've written libcollect - a module that allows you to easily do the
third option. The full docstring of libcollect is pasted in the bottom
of this email. The module is downloadable from
http://eli.thegreenplace.net/files/prog_code/libcollect.py.txt

I wanted to ask this list:

1) Is there something similar already and I've reimplemented the wheel ?
2) If not, then am I the only one who finds such a utility useful ?
3) If it looks useful to others, doesn't it belong with distutils ?
Its operation is in many ways similar to py2exe - it does a subset of
py2exe's work, really.

Thanks in advance
Eli




"""
libcollect.py

Provides the LibCollect class, used for collecting the various libraries
your script uses for delivery as a self-contained distribution package.

Author: Eli Bendersky (http://eli.thegreenplace.net)
License: Same as Python

Motivation:

Imagine that you've written a script that uses several libraries, some of
which you've written and some you've downloaded and installed (for example
PyYAML). You want to distribute the script to your friends and co-workers,
who already have Python installed with all the standard library. But your
script won't run on their machines, because they have neither your personal
libraries, nor PyYAML installed. So what can you do ?
* You can ask them to install PyYAML and other libraries your script uses,
  and send them your own libraries. This is a lengthy and inconvenient
  process.
* You can use a tool like py2exe to package your delivery. This has a
  downside, however. py2exe produces large files (several MBs) and you may
  not want that.
* You can painstakingly collect the libraries into a directory where your
  script can find them, and package the directory together with the script.

LibCollect makes the third option trivial, by doing all the dirty work
for you.

Example:

Suppose your script is named script.py, and is located in directory $DIR
(although I'm using Unix-y notation here, it is for convenience only.
LibCollect works similarly well on Windows platforms). Follow these steps
to prepare a self-contained distribution with LibCollect:

Create a distribution setup script in the same directory. Lets assume
you call it distrib_script.py. You can easily place it in any directory
you like, I'm using the same one to make the example simpler.

Add the following to distrib_script.py (assuming that libcollect.py is
in your sys.path):

**************************************************************

import libcollect

# Create a LibCollect object
lc = libcollect.LibCollect()

# Prepare arguments for do_collect
#

# Path to the script (can be absolute or relative)
scriptname = 'script.py'

# Ask the resulting distribution to be placed in
# directory distrib
targetdir = 'distrib'

# Specify which libraries to exclude from the
# distribution (because you know they're installed
# on the target machine)
excludes = ["wx",
            "pywin",
            "win32api",
            "win32com"]

# Zip the libraries used by the script to reduce
# clutter and save space
zip_lib = True

# This does the actual work
lc.do_collect(  scriptname,
                targetdir,
                excludes,
                zip_lib=zip_lib)

**************************************************************

Now run distrib_script.py.
When it finishes running, you will see that the distrib directory
has been created in $DIR. In $DIR/distrib you will see two files,
script.py and distlib.zip

* script.py is a loader that replaces your original script.py - this
  is the program your users should run. All it does (look at the
  code, it's short!) is prepare the sys.path to include the
  packaged libraries, and runs your own script.py that was also
  packaged into the .zip file
* distlib.zip is the distribution library, containing all the code
  your script needs to run on any machine with Python installed,
  and nothing else (except the modules you specified in the exclusion
  list). You may choose to pass on the zip file creation and leave
  your distribution library as a directory by providing False
  to the zip_lib argument of LibCollect.do_collect (take a look at
  its documentation, there are some other options there)

How to use LibCollect:

* It is most convenient to use LibCollect in the way demonstrated
  in the example above. You may want to update your application from
  time to time, and having a distribution script handy will turn
  the preparation of a new distribution into a 5-second process.
* If you don't want to create a distribution script, you can use
  a more direct method of invoking libcollect.py as a program on
  your script. Call it without arguments and it will print
  a usage string that will explain what you need to do.

How it works:

* LibCollect uses the standard modulefinder module to find out which
  libraries are used by your script. It categorizes them into two
  types: standard libraries that came with Python, and non-standard
  libraries you've installed or written.
* Only libraries of the second type are included in the distribution
  (bar the libraries you've explicitly asked to exclude).
* It then builds a directory with all the included libraries, in a
  way that your script will be able to find them. The script itself
  is also packaged into the same place.
* On request, this directory can be zipped into a single file, to
  employ Python's built-in zip import facility.
* In the distribution directory, a new file with the name of your
  script is created. It is a simple loader that uses the runpy module
  to transparently load your script from the distribution library.
  This way your script is not being modified (sys.path is rigged
  from the loader).

Compatibility:
    Python 2.5
    Tested on Windows and Linux, but should work on other platforms
    where the standard Python distribution works.

Version history:

    1.0  (2008.06.07): initial release

"""