How to create development Python environment on Linux.

[Zachary Ware]

On Mon, May 16, 2016 at 6:22 AM,  <redirect.null at gmail.com> wrote:
> I have a Linux system (Mint 17.3 based in Ubuntu 14.04) on which I wish to do some Python development. The system has Python 2.7.6 installed already (there is a Python 3 installation too but I won't be needing to use that to start with).

Not what you asked for, but I would encourage you to look into whether
it's possible for you to use Python 3 instead of Python 2 for what
you're doing.  If it's possible, starting with Python 3 will save you
several headaches in the future.

> [...]
> 1. I am unsure which of these technologies is most suitable for my needs. Virtualenv sounds like what I need, but two things give me pause for thought. First I came across this github issue
>
> https://github.com/pypa/virtualenv/pull/697
>
> ...which highlights what seem like severe issues in the vitualenv implementation and proposes to rewrite the whole library. This raises two questions.
>
>    i) Is this rewrite complete?
>    ii) Is 'legacy' virtualenv or virtualenv-rewrite now the recommended library?

virtualenv is a bit of a hack, by necessity.  Python 2 doesn't truly
support virtualenv, so virtualenv has to make do with what it has
available.  That said, it has worked well for my needs and for many
others.

(The real solution is Python 3.3+'s venv module :))

> The second thing that gives me pause for thought is the remark in this article
>
> http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers
>
> ...where it says
>
> 'Recent Debian/Ubuntu versions have modified pip to use the “User Scheme” by default, which is a significant behavior change that can be surprising to some users.'
>
> These concerns caused me to try and research more in the area of user installs and package installation, which leads me to my next two issues.

The real issue here is mixing the use of the OS package manager with
the use of pip.  If you only install python packages using the OS
package manager (e.g., `apt-get install python-requests`) or only with
pip (e.g., `pip install requests`), you'll probably be fine.  If you
try to use both, though, you may wind up in a weird situation.

> 2. User install site.py behaviour confusing.
>
> The first thing I wanted to work out was whether 'user install' was indeed enabled by default on my system as described in the article above. Here is the documentation for site.py
>
> https://docs.python.org/2/library/site.html
>
> This is the documenation for 2.7.11 while I only have 2.7.6. I don't see a link to documentation for 2.7.6 and I dont know whether this version disparity is significant or not.

It shouldn't be, but here's the 2.7.6 documentation anyway:
https://docs.python.org/release/2.7.6/library/site.html

> However, the documentation starts by saying that this module is loaded by default. That being so I would expect sys.prefix to be /usr/local.

Why do you expect that?  sys.prefix is baked in at compile time of the
python interpreter, and refers to where Python is installed on the
system.  `/usr/local` is the default prefix for custom builds, but
most Linux distributions install Python under `/usr`.  That the site
module is loaded by default just means that particular locations
outside of the standard library path are automatically added to
sys.path (the list of directories where Python will look when you try
to import something).  Compare the output of `python -c "import
pprint,sys;pprint.pprint(sys.path)"` with and without the -S and -s
options.

> If I fire up Python import sys and dump sys.prefix it outputs /usr.
> If I dump site.USER_BASE I get 'name 'site' is not defined'.

Even though site is imported by default by the interpreter, it doesn't
bind the name 'site' anywhere in the default namespace.  If you want
to use the contents of the site module, you still need to import site
yourself (this second import simply looks up 'site' in the sys.modules
dict and returns the module it finds there).

> If I import site and dump site.USER_BASE I get '/home/<user_name>/.local'.
>
> This confuses the hell out of me. Two documentary sources suggest this behaviour should be enabled by default, yet it doesn't seem to be, I don't know why, and I don't know how to enable it, or even whether I should.
>
> Can anyone either explain why I'm seeing what I'm seeing here, or point the in the direction to some documentation that can.

Hopefully I've cleared things up a bit above.  Please ask again if
it's still murky!

> 3. Installers
>
> My system Python installation doesn't appear to have either easy_install nor pip nor virtualenv installed. This leaves me with what seems like a circular problem.
>
> I'm pretty sure I need pip, but there are (at least) two ways to get it.
>    i) apt-get pip
>    ii) or download and run get-pip.py.
>
> I've come across various credible warnings of the dangers of using apt-get and pip to manage the same packages, but once pip is installed I was under the impression you need to use pip to update itself before it will install any packages as pip must be up to date in order to access PyPI.
>
> This page
>
> http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers
>
> ...suggests I should use apt-get to install pip, but that gives rise to the following sequence of operations
>
> apt-get pip
> pip install -U pip
>
> ..and isn't that exactly what I should be trying to avoid, installing a package using apt-get and then managing that package using pip?
>
> I'd have though that using get-pip.py would be safer, but even then that might only be with the provision that I execute get-pip.py in a Python environment separate from the system core Python installation, otherwise I'm in the same boat of using get-pip.py to upgrade my apt-get managed system Python installation. What to do?

This is where virtualenv helps: create your venv, then use the venv's
pip to update pip and then install the packages you want.

> 4. Python version.
>
> In addition to all this I'm wondering whether I wouldn't be a good idea to have the Python installation I use for development up to 2.7.11 before I start as my system installation of 2.7.6 is 3 years out of date.

This depends somewhat on where your code will be running in
production: if you develop with 2.7.11 but deploy to 2.7.6, you may
run into bugs that have been fixed in the interim, or you may try to
use some of the security-sensitive features that have been added in
more recent 2.7 releases (particularly in the ssl module).  Ideally,
you should probably be using the latest version for both development
and deployment.  The simplest way to do that is likely to compile and
install your own Python.  Here's a quick-start guide for that:

 - `sudo apt-get build-dep python` to install dependencies (like a C
compiler and a few external library headers, like openssl-dev)

 - Either download and unzip the source from
https://python.org/downloads, or clone the repository from
hg.python.org/cpython or github.com/python/cpython.  If you go the
repository route, you'll want to update to the correct revision for
the release you want; for the hg repo it'll be tagged "vM.m.p" where M
is the major version (2), m is the minor version (7), and p is the
patch version (11) (the github repo has no tags yet, it's currently an
unofficial mirror).

 - From the root of the source tree, run `./configure && make
profile-opt && make test && sudo make install`.  This will build and
install a production-ready Python under `/usr/local`.  Note that
depending on how your PATH is set up, `python` may refer to either the
system Python at `/usr/bin/python` or your own Python at
`/usr/local/bin/python`.  You can avoid a bit of ambiguity by doing
`make altinstall` instead of `make install`, which will not create
`/usr/local/bin/python`; you'd have to invoke your Python using
`python2.7` (again subject to your PATH setup), but `python` would
always be the system Python.

> In short I'm hopelessly confused and could really use some guidance.
> P.S. What happened to 'batteries included'?

The standard library is a large complement of high-capacity batteries;
PyPI is the Three Gorges Dam.

-- 
Zach