[Python-Dev] Issue 13524: subprocess on Windows

[Glyph Lefkowitz]

On Mar 21, 2012, at 4:38 PM, Brad Allen wrote:

> I tripped over this one trying to make one of our Python at work
> Windows compatible. We had no idea that a magic 'SystemRoot'
> environment variable would be required, and it was causing issues for
> pyzmq.
> 
> It might be nice to reflect the findings of this email thread on the
> subprocess documentation page:
> 
> http://docs.python.org/library/subprocess.html
> 
> Currently the docs mention this:
> 
> "Note If specified, env must provide any variables required for the
> program to execute. On Windows, in order to run a side-by-side
> assembly the specified env must include a valid SystemRoot."
> 
> How about rewording that to:
> 
> "Note If specified, env must provide any variables required for the
> program to execute. On Windows, a valid SystemRoot environment
> variable is required for some Python libraries such as the 'random'
> module. Also, in order to run a side-by-side assembly the specified
> env must include a valid SystemRoot."

Also, in order to execute in any installation environment where libraries are found in non-default locations, you will need to set LD_LIBRARY_PATH.  Oh, and you will also need to set $PATH on UNIX so that libraries can find their helper programs and %PATH% on Windows so that any compiled dynamically-loadable modules and/or DLLs can be loaded.  And by the way you will also need to relay DYLD_LIBRARY_PATH if you did a UNIX-style build on OS X, not LD_LIBRARY_PATH.  Don't forget that you probably also need PYTHONPATH to make sure any subprocess environments can import the same modules as their parent.  Not to mention SSH_AUTH_SOCK if your application requires access to _remote_ process spawning, rather than just local.  Oh and DISPLAY in case your subprocesses need GUI support from an X11 program (which sometimes you need just to initialize certain libraries which don't actually do anything with a GUI).  Oh and __CF_USER_TEXT_ENCODING is important sometimes too, don't forget that.  And if your subprocess is in Perl or Ruby or Java you may need a couple dozen other variables which your deployment environment has set for you too.  Did I mention CFLAGS or LC_ALL yet?  Let me tell you a story about this one HP/UX machine...

Ahem.

Bottom line: it seems like screwing with the process spawning environment to make it minimal is a good idea for simplicity, for security, and for modularity.  But take it from me, it isn't.  I guarantee you that you don't actually know what is in your operating system's environment, and initializing it is a complicated many-step dance which some vendor or sysadmin or product integrator figured out how to do much better than your hapless Python program can.

%SystemRoot% is just the tip of a very big, very nasty iceberg.  Better not to keep refining why exactly it's required, or someone will eventually be adding a new variable (starting with %APPDATA% and %HOMEPATH%) that can magically cause your subprocess not to spawn properly to this page every six months for eternity.  If you're spawning processes as a regular user, you should just take the environment you're given, perhaps with a few specific light additions whose meaning you understand.  If you're spawning a process as an administrator or root, you should probably initialize the environment for the user you want to spawn that process as using an OS-specific mechanism like login(1).  (Sorry that I don't know the Windows equivalent.)

-glyph