[Patches] [ python-Patches-1035498 ] -m option to run a module as a script

Thu Oct 7 00:24:25 CEST 2004

Patches item #1035498, was opened at 2004-09-28 00:22
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1035498&group_id=5470

Category: Core (C code)
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Nick Coghlan (ncoghlan)
Assigned to: Nobody/Anonymous (nobody)
Summary: -m option to run a module as a script

Initial Comment:
Implements the '-m' option recently discussed on
python-dev.

Runs a module 'as if' it had been invoked from the
command line.

E.g., for a build directory, the following two commands
are equivalent:
  ./python -m pdb
  ./python Lib/pdb.py

Note that neither of these does the same thing as
"./python -c "import pdb". (In this latter case, any
"if __name__ == "__main__" code would not be executed,
whereas it will be invoked in the first two cases).

Given the vagaries of sys.path, this is quite handy -
if a module can be imported, it can be easily used as a
script too. Not that all modules will necessarily do
anything _interesting_ when used as a script. . .

The current implementation makes changes to main.c
(handle the new argument), pythonrun.[ch] (new flavour
of PyRun_) and import.[ch] (expose a
PyImport_FindModule C API)

Current limitations / to-do if this is pursued:
  - print sensible errors for non-Python files (e.g.
hotshot)
  - allow 'dotted' names
  - decide what to do with packages (e.g. run __init__.py)
  - check usage messages in main.c are all still under
512 characters (I assume that limit was set for a reason)
  - the 1K limit on absolute file paths seems rather
arbitrary.
  - this is code written in the wee hours of the
morning, so it probably has a few other lurking 'features'.

----------------------------------------------------------------------

>Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 08:24

Message:
Logged In: YES 
user_id=1038590

New version of doc patch which changes the tutorial only
(since PyRun_SimpleModule is no longer part of the patch).

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 07:32

Message:
Logged In: YES 
user_id=1038590

New version (7) attached which does the right thing for
"python -tt -m regrtest -u network". (I really like that
test case btw - very concise)

This one sets argv[0] correctly, but was initially a little
ungraceful in some failure modes (such as "python -m
parser"). I had to add a new private API function to
import.h (_PyImport_IsScript) to clean that up.

Examples of the error handling in this version:

[... at localhost src]$ ./python -m foo
./python: module foo not found
[... at localhost src]$ ./python -m sys
./python: module sys has no associated file
[... at localhost src]$ ./python -m parser
./python: module parser not usable as script

(/home/.../dev/python/dist/src/build/lib.linux-i686-2.4/parser.so)

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 04:56

Message:
Logged In: YES 
user_id=1038590

Nice catch. . .

I figured out what the problem is - regrtest.py uses
sys.argv[0] to delete it's own directory from sys.path. And
-m currently sets sys.argv[0] to just 'regrtest', instead of
the full path to the module.

Obviously, that is going to have to change. Either
PyRun_SImpleModule will have to do it, or I'll get rid of
that  function all together, and put the 'module-finding'
logic directly in main.c.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-10-07 04:09

Message:
Logged In: YES 
user_id=80475

Put the test directory in PYTHONPATH and run the following
to see if you get the same results:

    python -tt -m regrtest  -u network
    python -tt lib\test\regrtest.py -u network

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 01:07

Message:
Logged In: YES 
user_id=1038590

New version attached which should fix the Windows compile
bug (pythonrun.c does the right thing by undefining BYTE
before including windows.h - the inclusion of 'importdl.h'
has been moved to be after that 'fixed' include).

Unfortunately, I won't get access to a Windows build machine
until after the beta 1 deadline, or I'd check it myself :(

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-10-06 20:58

Message:
Logged In: YES 
user_id=80475

The patch does not compile on windows.  AFAICT, importdl.h
is the culprit because it loads windows.h for a second time
resulting a double redinition of BYTE deep in the bowls of
include files.

Wish I had time to help you on this one, but I don't.

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2004-10-05 23:15

Message:
Logged In: YES 
user_id=4771

Sorry, I didn't follow any of this discussion, neither here
nor in python-dev, so I cannot really comment.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-10-05 14:32

Message:
Logged In: YES 
user_id=80475

Armin, do you have time for this one.  I'm buried for a few
days.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-05 09:05

Message:
Logged In: YES 
user_id=1038590

Raymond,

Passing to you to see what you think of the code, rather
than just the feature in the abstract.

Guido settled for not vetoing it, but the discussion with
him allowed me to articulate why I think this is a
worthwhile feature.
(Details at
http://mail.python.org/pipermail/python-dev/2004-October/049236.html)

Extension to allow packages and zip imports can wait until
there is any demand for either feature. I don't really
expect to see any, since I think the main uses will be
"python -m pdb", "python -m profile" and "python -m timeit".

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-05 04:21

Message:
Logged In: YES 
user_id=1038590

Doc patch attached. Take it with a grain of salt, since this
is the first real doc patch I've made.

This doesn't cover anything for 'What's New' - that can wait
until the patch gets accepted (if it does).

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-05 03:48

Message:
Logged In: YES 
user_id=1038590

5th version of patch attached

- restricts option to top-level modules only. The correct
semantics for a module inside a package just aren't clear
enough at this stage. Support for dotted names can be added
later if this patch gets accepted, and it still seems like a
good idea.

- adds a missing check from main.c (if we got the module
option, don't try to open the file).

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-05 00:30

Message:
Logged In: YES 
user_id=1038590

Relevant documentation locations that I can find:
  Tutorial section 2.1 (Invoking the Interpreter)
    Add a very brief description of the '-m' option

  Python/C API section 2 (The Very High Level Layer)
    Document PyRun_SimpleModule

  What's New sections 12 & 13
    I'll do this as a separate patch from the previous two,
as I can't recall if Andrew prefers to write these himself.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-01 14:13

Message:
Logged In: YES 
user_id=1038590

New version (#4)

Trivial fix - tell PyRun_SimpleFile to close the file when
it's done with it.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-01 12:51

Message:
Logged In: YES 
user_id=1038590

New version (run_module_3.diff)

Takes more account of the file description reported by
PyImport. Explictly restricts itself to PY_SOURCE and
PY_COMPILED scripts.

Behaves correctly when a non-package is encountered while
parsing a dotted name (closes the file descriptor and
reports an error).

I also realised that the '-m' option doesn't currently
comprehend "__path__" variables in packages. So maybe we
*should* be loading the packages as we find them, and then
querying them for their __path__ variable.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-09-30 19:48

Message:
Logged In: YES 
user_id=1038590

Barry, kicking this in your direction solely because you
were one of the people to comment on it on py-dev. Feel free
to say 'too busy' :)

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2004-09-30 19:39

Message:
Logged In: YES 
user_id=1038590

New version of patch attached (run_module_2.diff).

This version allows modules within packages to be executed.
It does this *without* loading the packages first (just as
if the script were specified directly on the command line).

It's also a little more responsive when it comes to error
message, and I've confirmed that the usage message
components are now under 512 bytes each.

The FindModule method is now marked as private (i.e. with
leading underscore).

The limit on file paths in pythonrun.c is now taken from
osdefs.h (the same limit that import.c already uses)

The only bug I know of is that it misbehaves if you supply a
name like "a.b.module.d". You will get an ImportError saying
that d was not found, and the file pointer for 'module'
won't get closed properly.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1035498&group_id=5470