PEP: Adding Priority Scheduling feature to the subprocess

darklord@timehorse.com TimeHorse at gmail.com
Wed Feb 20 11:33:33 EST 2008 

I am working on a PEP and would appreciate comment.  The proposal is
available at

http://python.timehorse.com/PEP_-_Application_Priority.reST

and is repeated below:

---------

:PEP: <Unassigned>
:Title: Adding Priority Scheduling feature to the subprocess module
:Version: $Rev: 93 $
:Last Modified: $Date: 2008-02-20 08:37:00 -0500 (Wed, 20 Feb 2008) $
:Author: Jeffrey C. Jacobs
:Discussions-To: python at timehorse dot com
:Status: Draft
:Type: Standards Track
:Content-Type: text/x-rst
:Created: 19-Feb-2008
:Python-Version: 2.5.1
:Post-History: <None>

----------

Introduction
~~~~~~~~~~~~

The ``subprocess`` module is intended to create a platform-independent
method of *spawning* or *popening* or *execing* an OS-level sub-
process.  This module provides the ability to spawn processes on a
variety of platforms by creating an instance of the
``subprocess.Popen`` class.

Currently, only the Windows version of the ``Popen.__init__`` method
can set the OS priority of a spawned process directly using the
``createFlags`` parameter.  Clients operating under UNIX must use some
version of the ``preexec_fn`` to either wrap a C-call to the
``setpriority`` POSIX function or do so by prepending ``nice <u>`` to
the ``args`` or in the ``executable``, which would only work for sub-
processes spawned from a shell.  Under a Windows environment, however,
the ``preexec_fn`` parameter is ignored and there is no ``nice`` shell
program.  This thus precludes the easy setting of a spawned process's
priority in the UNIX environment, as well as creates a disconnect in
terms of the way priorities are set between Windows and UNIX
platforms, which somewhat defeats the purpose of ``subprocess.py`` in
terms of being relatively platform-independent.

In addition to these issues, because Windows processes are referenced
by handles, not by process IDs, although this would divide the UNIX-
specific and Windows specific code, the Handle for the windows would
none the less be useful in the ``win32process`` module and therefore
it is proposed that this handle be officially exposed from the
``Popen`` instance via the name ``handle``.  Currently, this property
is unofficially accessible via the name ``_handle``, so this proposes
to rename that value and make the name official for python running
under Windows.  The value of ``handle`` for other platforms would then
be ``None``.  In addition to officially exposing the Handle to
Process, it is proposed that the Main Thread Handle, also internally
exposed within the ``Popen.__init__`` process, be externally exposed
via the name ``hthread`` for use by ``win32process`` and the priority
methods.

Because UNIX uses priorities between +20 and -20 and Windows, via
Process and Thread priorities, allows settings between 0 and 31, a
uniform setting for each system should be derived.  This would be
accomplished by giving process priority in terms of a floating-point
value between 0.0 and 1.0 for lowest and highest possible priority,
respectively.  This priority would be added to the ``Popen``
constructor as an optional parameter, with a default value of
``None``.  If and only if something other than ``None`` is set, that
value would override any setting in ``createFlags``, and the Process
Priority flags of ``createFlags`` would be masked off under the
Windows environment.  In all cases a non-``None`` setting would cause
the spawned process to run at the indicated priority converted to the
platform-specific units.

The Macintosh OS X / BSD subsystem should be compatible with the UNIX-
specific sections of this proposal.

Specifics
~~~~~~~~~

The following steps would accomplish this proposal:

1) Add a C-Python wrapper for the POSIX ``getpriority`` /
``setpriority`` methods to the ``os`` module.  These methods would
only be available in a UNIX environment and would mesh well with the
existing file/process functions specified therein.  The
``win32process`` module already exposes the corresponding Windows
methods.

2) Officially expose the Process Handle as ``handle`` and the
Process's Main Thread Handle as ``hThread`` within the ``Popen``
instance interface.

3) Add ``setPriority`` / ``getPriority`` methods to the ``Popen``
class which will use the platform-specific priority function either
from ``os`` for UNIX or ``win32process`` for Windows. The input
priority to ``setPriority`` will be checked and raise a ``ValueError``
exception for values not in the range 0.0 to 1.0.

   The 0.0 to 1.0 scale will be converted to the 32-point Windows
scale by linear mapping, e.g. for a floating-point priority *p*, the
Windows priority is given by:

   ``int(p*31 + .5)``

   Windows is limited to a non-linear, 7-point scale for process
priorities, with thread priorities making up the difference relative
to the 32-point internal priority scale.  Thus, the Windows versions
of these functions will use a combination of ``SetPriorityClass`` and
``SetThreadPriority`` in order to set the requested process priority,
using the ``handle`` and ``hThread`` properties of the ``Popen``
instance.

   The 0.0 to 1.0 scale will be converted to the 41-point UNIX scale
(values between +20 and -20, inclusive) via an inverted linear
mapping, e.g. for a floating-point priority *p*, the UNIX priority is
given by:

   ``20 - int(p*40 + .5)``

   Under the UNIX platform, only the super-user (root) is allowed to
set priorities below 0 (*p* > 0.5).  This condition will not be
checked in the ``setPriority`` code; it is the responsibility of the
caller to ensure the script making the call to ``setpriority`` has
sufficient user privileges, i.e. via the ``os.setuid`` function.

   The conversion from the platform-specific Priority scale to the
floating-point scale are used by ``getPriority`` and are as follows
for a given UNIX priority *u* and Windows priority *w*:

   ``(20 - u) / 40.0``

   ``w / 31.0``

   As with the ``setPriority`` method for windows, the ``getPriority``
method shall use a combination of ``GetPriorityClass`` /
``GetThreadPriority`` to determine the true Windows priority based on
the 32-point scale before converting to floating point.  The specific
numeric conversion for the Windows ``getPriority`` method is given in
`Table 1`_.

   See the `Appendix`_ for details about the 32-point Windows scale.
Note that some priorities have equivalent settings.  In this case, raw
Process Priority Class (without regard for Foreground and Background
Process settings) will have preferential setting and Thread Priority
Levels shall be used to make up the difference.  E.g. priorities from
1 to 4 will have ``IDLE_PRIORITY_CLASS``, 5 to 7 will have
``BELOW_NORMAL_PRIORITY_CLASS``, 8 and 9 will have
``NORMAL_PRIORITY_CLASS``, 10 to 12 will have
``ABOVE_NORMAL_PRIORITY_CLASS``, 13 to 15 will have
``HIGH_PRIORITY_CLASS`` and 16 to 31 will be accessed through
``REALTIME_PRIORITY_CLASS``, potentially with a non-standard Thread
Priority level.  Also note that Priority 0 is reserved for the Zero-
Page Thread, which is responsible for zeroizing memory when no other
process is running.  It is impossible for a user process to have this
priority.  Therefore, on Windows, any priority set request below about
0.016 will be treated as if it was requesting Windows Priority Level
1.

4) Add a priority parameter to the ``Popen.__init__`` method that
shall be the final parameter of the function and default to ``None``
in order to preserve backwards compatibility.  When priority is not
``None``, it shall be used in order to set the priority of the spawned
process either at the time of the creation, or soon after.  The
setting of priority may be accomplished via a ``nice <u>`` prefix to
shell commands (``shell = True``) or via a call to ``setpriority``
after the process is spawned.  When a priority is set via the Windows
version, a mask is applied to the ``createFlags`` parameter to remove
any priority related bits.  The calculated priority is then used to
determine a new priority in the 7-point Windows Process scale and set
the corresponding flag in the ``createFlags`` parameter.  Then
``SetThreadPriority`` shall be called on the returned Main Thread
Handle with the remaining priority points to further adjust the
priority toward the 32-point scale.

Opened Questions
~~~~~~~~~~~~~~~~

1) Should ``Popen.__init__`` for UNIX check for shell-based spawns and
automatically apply ``nice`` or should it always first spawn the
process and then set its priority via ``setpriority`` once created.
Note, the later technique would technically not be available using
``preexec_fn``.

2) The ``startinfo`` parameter may also need adjusting to set whether
the process is spawned as a background or foreground process (See
`Table 1`_).

3) Should windows use the Priority Boost APIs:
``SetProcessPriorityBoost``, ``GetProcessPriorityBoost``,
``SetThreadPriorityBoost`` and ``GetThreatPriorityBoost``?

4) How does one determine if a given Windows process with handle
``handle`` is running as a *Background* or a *Foreground* process at
any given time, for use in the Windows version of ``getPriority``?

5) With the current version of ``subprocess`` I have been unable to
spawn a process with a higher priority than ``NORMAL_PRIORITY_CLASS``
under Windows; python is always running as ``NORMAL_PRIORITY_CLASS``.

Appendix
~~~~~~~~

.. _Table 1:

+----+----------------------------------
+-------------------------------+
| Pt | Process Priority Class           | Thread Priority
Level         |
+====+==================================
+===============================+
|  1 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_IDLE          |
+----+----------------------------------
+-------------------------------+
|  1 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_IDLE          |
+----+----------------------------------
+-------------------------------+
|  1 | NORMAL_PRIORITY_CLASS            |
THREAD_PRIORITY_IDLE          |
+----+----------------------------------
+-------------------------------+
|  1 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_IDLE          |
+----+----------------------------------
+-------------------------------+
|  1 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_IDLE          |
+----+----------------------------------
+-------------------------------+
|  2 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_LOWEST        |
+----+----------------------------------
+-------------------------------+
|  3 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_BELOW_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  4 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_NORMAL        |
+----+----------------------------------
+-------------------------------+
|  4 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_LOWEST        |
+----+----------------------------------
+-------------------------------+
|  5 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_ABOVE_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  5 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_BELOW_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  5 | Background NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST        |
+----+----------------------------------
+-------------------------------+
|  6 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_HIGHEST       |
+----+----------------------------------
+-------------------------------+
|  6 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_NORMAL        |
+----+----------------------------------
+-------------------------------+
|  6 | Background NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  7 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_ABOVE_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  7 | Background NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL        |
+----+----------------------------------
+-------------------------------+
|  7 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_LOWEST        |
+----+----------------------------------
+-------------------------------+
|  8 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_HIGHEST       |
+----+----------------------------------
+-------------------------------+
|  8 | NORMAL_PRIORITY_CLASS            |
THREAD_PRIORITY_ABOVE_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  8 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_BELOW_NORMAL  |
+----+----------------------------------
+-------------------------------+
|  8 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_LOWEST        |
+----+----------------------------------
+-------------------------------+
|  9 | NORMAL_PRIORITY_CLASS            |
THREAD_PRIORITY_HIGHEST       |
+----+----------------------------------
+-------------------------------+
|  9 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_NORMAL        |
+----+----------------------------------
+-------------------------------+
|  9 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_BELOW_NORMAL  |
+----+----------------------------------
+-------------------------------+
| 10 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_ABOVE_NORMAL  |
+----+----------------------------------
+-------------------------------+
| 10 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_NORMAL        |
+----+----------------------------------
+-------------------------------+
| 11 | Foreground NORMAL_PRIORITY_CLASS |
THREAD_PRIORITY_HIGHEST       |
+----+----------------------------------
+-------------------------------+
| 11 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_ABOVE_NORMAL  |
+----+----------------------------------
+-------------------------------+
| 11 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_LOWEST        |
+----+----------------------------------
+-------------------------------+
| 12 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_HIGHEST       |
+----+----------------------------------
+-------------------------------+
| 12 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_BELOW_NORMAL  |
+----+----------------------------------
+-------------------------------+
| 13 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_NORMAL        |
+----+----------------------------------
+-------------------------------+
| 14 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_ABOVE_NORMAL  |
+----+----------------------------------
+-------------------------------+
| 15 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_HIGHEST       |
+----+----------------------------------
+-------------------------------+
| 15 | HIGH_PRIORITY_CLASS              |
THREAD_PRIORITY_TIME_CRITICAL	|
+----+----------------------------------
+-------------------------------+
| 15 | IDLE_PRIORITY_CLASS              |
THREAD_PRIORITY_TIME_CRITICAL	|
+----+----------------------------------
+-------------------------------+
| 15 | BELOW_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_TIME_CRITICAL	|
+----+----------------------------------
+-------------------------------+
| 15 | NORMAL_PRIORITY_CLASS            |
THREAD_PRIORITY_TIME_CRITICAL	|
+----+----------------------------------
+-------------------------------+
| 15 | ABOVE_NORMAL_PRIORITY_CLASS      |
THREAD_PRIORITY_TIME_CRITICAL	|
+----+----------------------------------
+-------------------------------+
| 16 | REALTIME_PRIORITY_CLASS          | THREAD_PRIORITY_IDLE		|
+----+----------------------------------
+-------------------------------+
| 17 | REALTIME_PRIORITY_CLASS          | -7				|
+----+----------------------------------
+-------------------------------+
| 18 | REALTIME_PRIORITY_CLASS          | -6				|
+----+----------------------------------
+-------------------------------+
| 19 | REALTIME_PRIORITY_CLASS          | -5				|
+----+----------------------------------
+-------------------------------+
| 20 | REALTIME_PRIORITY_CLASS          | -4				|
+----+----------------------------------
+-------------------------------+
| 21 | REALTIME_PRIORITY_CLASS          | -3				|
+----+----------------------------------
+-------------------------------+
| 22 | REALTIME_PRIORITY_CLASS          | THREAD_PRIORITY_LOWEST	|
+----+----------------------------------
+-------------------------------+
| 23 | REALTIME_PRIORITY_CLASS          |
THREAD_PRIORITY_BELOW_NORMAL	|
+----+----------------------------------
+-------------------------------+
| 24 | REALTIME_PRIORITY_CLASS          | THREAD_PRIORITY_NORMAL	|
+----+----------------------------------
+-------------------------------+
| 25 | REALTIME_PRIORITY_CLASS          |
THREAD_PRIORITY_ABOVE_NORMAL	|
+----+----------------------------------
+-------------------------------+
| 26 | REALTIME_PRIORITY_CLASS          | THREAD_PRIORITY_HIGHEST	|
+----+----------------------------------
+-------------------------------+
| 27 | REALTIME_PRIORITY_CLASS          | 3				|
+----+----------------------------------
+-------------------------------+
| 28 | REALTIME_PRIORITY_CLASS          | 4				|
+----+----------------------------------
+-------------------------------+
| 29 | REALTIME_PRIORITY_CLASS          | 5				|
+----+----------------------------------
+-------------------------------+
| 30 | REALTIME_PRIORITY_CLASS          | 6				|
+----+----------------------------------
+-------------------------------+
| 31 | REALTIME_PRIORITY_CLASS          |
THREAD_PRIORITY_TIME_CRITICAL	|
+----+----------------------------------
+-------------------------------+

Table 1: 32-Point Windows Priority Scale [#MSPriorityTable]_.

-------

.. [#MSPriorityTable] From `Microsoft Windows Scheduling Priorities
<http://msdn2.microsoft.com/en-us/library/ms685100.aspx>`_.

More information about the Python-list mailing list