From gward@cnri.reston.va.us Mon Jan 11 04:12:04 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 10 Jan 1999 23:12:04 -0500 Subject: [Distutils] Design proposal Message-ID: <19990110231204.A26431@cnri.reston.va.us> Well, things have been a little quiet on this list. Time to shake it up a bit, methinks. So I've written a design proposal -- here it is. (NB. #1: to avoid sounding overly wafflish, I've written this in a fairly imperious way -- "X will be done this way, Y is like that". Don't take it *too* seriously; this is open for discussion, up for grabs, welcomes your comments and criticisms, etc. Of course, I wouldn't have designed it this way if I didn't think it was good idea, so expect a reaction if you tell me I'm full of baloney... ;-) (NB. #2: this file is in the Distutils CVS archive as 'text/design-proposal.txt'. Once it is better firmed up through the discussion that I'm sure is forthcoming, I'll HTMLify it, put it in the SIG's web space, and start coding!) $Id: design-proposal.txt,v 1.1 1999/01/11 04:06:01 gward Exp $ RECAP: User Interface ===================== Recall from the proposed user interface (posted at http://www.python.org/sigs/distutils-sig/interface.html) that the Distutils will operate via a (usually) trivial Python script, conventionally called setup.py. setup.py has the following syntax: setup.py [global_options] cmd [cmd_options] (XXX no provision for multiple commands here! shouldn't be hard, though, as long as we're firm about what a command is and what an option is) The Distutils will define a standard set of global command-line options; each Distutils command will define a set of command options. The module developer (the person who wrote the setup.py for his module distribution) may define a set of distribution-specific command-line options, which will be mixed in with the global options (and extracted from them before they can cause any harm). The first order of business to decide on the set of standard Distutils commands. Again, from the proposed interface, here is my initial list: make_blib - create mockup installation tree ("build library") build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution bdist - create built distribution for current platform test - run test suite install - install on local machine please see the above URL for details on these. REAL STUFF: Proposed Design =========================== Part 1: from the Distutils' point of view ----------------------------------------- setup.py only has to import one module, 'distutils.core'. This module is responsible for parsing all command-line arguments to setup.py (even though the interpretation of options is distributed across the various Distutils commands, and possibly the client setup.py). It also takes care of receiving control from setup.py, and passing it as appropriate to Distutils commands. Most importantly, 'distutils.core' defines the 'Distribution' class, which is the heart and soul of the Distutils. The client (setup.py) exists mainly to provide attributes for a 'Distribution' instance, and all the Distutils commands operate on that instance. Speaking of Distutils commands: each one is implemented as a Python module, e.g. the 'build' command is implemented by the 'distutils.build' module. Each command module is required to define one function, also named for the command -- e.g. 'distutils.build.build'. This function takes a 'Distribution' instance as its only required argument, and then "does its thing" (eg. build extensions, install everything, etc.). All information needed to "do its thing" is contained in the 'Distribution' instance. (XXX this isn't very OO. As long as we can fully parameterize Distutils commands, that's fine: the client (or user) just provides attributes to go into a 'Distribution' instance, and the command modules use those attributes to know what to do. Inevitably, though, someone somewhere will have to override actual code, and this scheme will break down -- either that or we'll have to put a lot of command-specific stuff into the 'Distribution' class, which is bad [makes it harder to add new commands]. A possible alternate formulation: have each command module define a specially-named class, which is then used to "do its thing". If a client setup.py needs to override, say, installation behaviour, then it can create a subclass of distutils.install.Install. It would then need to tell the 'Distribution' class to use its derived class -- not an impossible task!) Part 2: from the client's point of view --------------------------------------- As I said above, the client (setup.py) only has to import 'distutils.core' -- everything else Distutils-ish is taken care of by this core module. However, the client needs a way to communicate its particular options into the Distutils core (and out to the command modules). I have two possible schemes for this: one short and convenient (but not too extensible), and the other a bit verbose and clunky (but more OO and extensible). There's no reason we can't have our cake and eat it too; the convenient interface could just be a wrapper for the full-blown interface for the many module distributions that don't need a lot of fancy customization. First, here's an example of the simple interface, used for a module distribution with a single "pure Python" module (mymod.py). ------------------------------------------------------------------------ from distutils.core import setup setup (name = "mymod", version = "1.2", author = "Greg Ward ", description = "A very simple, one-module distribution") ------------------------------------------------------------------------ Note that we don't explicitly list "mymod.py" anywhere: Distutils assumes that this is a one-horse distribution named after its sole module ('mymod'). Those who enjoy defining subclasses might prefer to phrase this differently: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mymod" version = "1.2", author = "Greg Ward ", description = "A very simple, one-module distribution") setup (distclass = MyDistribution) ------------------------------------------------------------------------ This is overkill for a small distribution: we're defining a new class solely to provide attribute values, when the 'distutils.core.setup' exists mainly to let do this anyways. Nevertheless, OO purists will like this -- and undoubtedly there will be times when the client *will* have to override behaviour -- not just data -- and the OO interface will be necessary. And more complex module distributions, with lots of attributes to customize, might be easier to read/maintain with things broken up like this. Consider a distribution with two pure Python modules ('mymod' and 'my_othermod') and a C extension ('myext'); the C extension must be linked with two ancillary C files and a C library. Oh yeah, this distribution requires Python 1.5 and any version of the 're' module: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mydist", version = "1.3.4", author = "Greg Ward " description = """This is an example module distribution. It provides no useful code, but is an interesting example of the Distutils in action.""" # Dependencies requires = { 'python': '1.5', # I like class-based exceptions 're': '', # and I love Perl-style regexps! ;-) } # (and yes, I *know* that "Python 1.5" # implies 're'...) # Actual files that need to be processed and installed in some form py_modules = ['mymod.py', 'my_othermod.py'], ext_modules = {'myext.c': {'other_c': ['extra1.c', 'extra2.c'], 'c_libraries': ['mylib']} } setup (distclass = MyDistribution) ------------------------------------------------------------------------ A couple of things to note: * I'm not afraid to use deeply nested data structures; if you're writing and distributing Python modules, this shouldn't be a problem! * every attribute has a particular type (string, list, dictionary, ...) * the attributes with complex types (especially dictionaries) will have a well-known and well-documented internal structure: eg. """ext_modules is a hash mapping names of C source files (each containing a Python extension module) to a nested hash of information about how to build that module. The allowed keys to this nested hash are: - other_c: other C files that must be compiled and linked with the main C file to create the module - c_libraries: C libraries that must be included in the link ... """ No doubt the 'ext_modules' nested hashes would have more options, and no doubt other Distribution attributes would have complex, documented structure. Finally, the list of all Distribution attributes must be well-known and well-documented! These seem to fall into a couple of broad categories. Here's an initial attempt at a list: Distribution meta-data name version author description Dependencies requires Files to be processed and installed py_modules ext_modules doc_files [eg. SGML source - or whatever std. we get for documentation] Build directories [all under "./blib" by default] build_lib - where to put platform-independent library files build_platlib - where to put platform-dependent library files build_exe - where to put executable programs (ie. scripts) build_html - where to put processed documentation (HTML) (etc... more documentation formats, at least) Installation directories [under sysconfig.LIBDEST] install_lib install_platlib install_exe install_html C compilation cc ccshared cflags ldflags ...well, that's a start. I still don't know how to make all those Unixish C compilation variables more cross-platform. Part 3: revisiting the Distutils' point of view ----------------------------------------------- To sum up, let's go through what happens when the user runs 'setup.py'. Whether setup.py is written in the simple (call-a-function) or general (define-a-subclass) form doesn't matter too much, so I won't split things up into two streams. * setup.py imports distutils.core * distutils.core startup code parses command-line arguments: processes global options that it knows about, and saves the rest for the client (setup.py) to deal with; saves the command, and saves the command-specific options for passing to the command module * setup.py calls distutils.core.setup (possibly with a 'distclass' argument specifying a subclass of Distribution) * distutils.core.setup instantiates Distribution (or the subclass supplied by the client), and uses its arguments (apart from 'distclass') to override attributes of this instance * distutils.core.setup loads the command module (eg. 'distutils.build') * distutils.core.setup calls the command module's interface function (eg. 'distutils.build.build'), passing it the Distribution instance and any command-specific options from the setup.py command-line * [alternate formulation: distutils.core.setup instantiates the command module's interface class (eg. 'distutils.build.Build', or an alternate supplied by the client using the as-yet-unmentioned 'command_class' attribute. The Distribution instance and all command-specific options are supplied to the command class so it can "do its thing"] (XXX again, no provision for multiple commands, although it shouldn't be too hard. And the OO approach to writing command modules needs to be better fleshed out.) Part 4: Unresolved issues ------------------------- * Where do we take care of platform dependencies? Somewhere, sometime, we'll need a class or function or attribute named 'foo_posix', 'foo_win32', 'foo_mac', etc. We might also need 'foo_linux_i86', 'foo_solaris2, 'foo_irix5', 'foo_winnt', 'foo_win98', 'foo_macos8', etc. * ...And that's just within Distutils itself. What about client code -- what if I have a module that sets itself up differently for different distributions; how do I specify that? * Recusive setup: what if my distribution has subdirectories containing other module distributions, with their own setup.py's? Will this ever be needed. (Undoubtedly.) How does the client specify them, and how does the Distutils run them? -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Thu Jan 14 15:04:27 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 14 Jan 1999 10:04:27 -0500 Subject: [Distutils] Comments on proposed design? Message-ID: <19990114100427.A28618@cnri.reston.va.us> Hey folks -- that Distutils design proposal was meant to provoke some discussion! Don't tell me everybody thinks my proposal was so sublimely perfect that I should just go home and knock off version 0.01 over the weekend. There are known problems, things I need help with, etc. Please take a look at it and post your reactions to the list. However, be warned that I am hoping to revise the proposal and post a revised version tonight/tomorrow. I would really like to have some feedback before I start revising. Thanks -- Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Fri Jan 15 03:56:59 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 14 Jan 1999 22:56:59 -0500 Subject: [Distutils] Revised design proposal Message-ID: <19990114225659.A29195@cnri.reston.va.us> Hi all -- as threatened, here's a revised design proposal. Main changes: I've commited myself to and firmed up the language regarding 1) multiple commands on the setup.py command line, and 2) OO command modules. There're still some unresolved issues, and I'll be posting on them separately soon. BTW if you can't be bothered to read the whole thing, at least skip down to and read "Part 3", a sort of executive summary (for executives who want to know every step the code will take, but not know the background -- that's what the other 250 lines of text are for!). Enjoy. --snip----------------------------------------------------------------- $Id: design-proposal.txt,v 1.2 1999/01/15 03:52:30 gward Exp $ RECAP: User Interface ===================== Recall from the proposed user interface (posted at http://www.python.org/sigs/distutils-sig/interface.html) that the Distutils will operate via a (usually) trivial Python script, conventionally called setup.py. setup.py has the following syntax: setup.py [global_options] cmd1 [cmd_options] [cmd2 [cmd_options]] ... (Distinguishing the end of the option list for cmd1 from cmd2 might be a bit tricky if command options can take arguments, which they certainly should do. If we want getopt-style syntax, it looks like we'll have to reinvent the getopt module. :-( And if we opt for a different syntax, we'll just have to write something to parse that! It's worth the effort in order to be able to say "./setup.py build test install", though.) The Distutils will define a standard set of global command-line options; each Distutils command will define a set of command options. The module developer (the person who wrote the setup.py for this module distribution) may define a set of distribution-specific command-line options, which may be mixed in with the global options (and extracted from them before they can cause any harm). The first order of business is to decide on the set of standard Distutils commands. Again, from the proposed interface, here is my initial list: make_blib - create mockup installation tree ("build library") build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution bdist - create built distribution for current platform test - run test suite install - install on local machine Please see the above URL for details on these. REAL STUFF: Proposed Design =========================== Part 1: from the Distutils' point of view ----------------------------------------- setup.py only has to import one module, 'distutils.core'. This module is responsible for parsing all command-line arguments to setup.py (even though the interpretation of options is distributed across the various Distutils commands, and possibly the client setup.py). It also takes care of receiving control from setup.py, and passing it as appropriate to Distutils commands. Most importantly, 'distutils.core' defines the 'Distribution' class, which is the heart and soul of the Distutils. The client (setup.py) exists mainly to provide attributes for a 'Distribution' instance, and all the Distutils commands operate on that instance. 'distutils.core' also defines the 'Command' class which comes in handy for implementin Distutils commands. Speaking of Distutils commands: each one is implemented as a Python module, e.g. the 'build' command is implemented by the 'distutils.build' module. Each command module is required to define one class, also named for the command -- e.g. 'distutils.build.Build'. These command classes will inherit from the 'Command' class, which (at the very least) will provide a means of dealing with command-specific options. (Probably 'Command' will provide a constructor that takes a 'Distribution' class and an optional list of arguments for this command, and parse the argument list by inspecting getopt-style option specifiers in the 'Command'-derived instance.) Each command class must provide a method ('go'? 'run'? 'doit'?) that uses the information in the 'Distribution' instance and the command options to "do its thing". Well-written command classes will parcel this task out into several well-defined (and documented) methods, so that the client setup.py may inherit from and override specific behaviours of a Distutils command class. This also means that the 'Distribution' class must have a way to communicate overridden command classes to the main dispatcher. Part 2: from the client's point of view --------------------------------------- As I said above, the client (setup.py) only has to import 'distutils.core' -- everything else Distutils-ish is taken care of by this core module. However, the client needs a way to communicate its particular options into the Distutils core (and out to the command modules). I have two possible schemes for this: one short and convenient (but not too extensible), and the other a bit verbose and clunky (but more OO and extensible). There's no reason we can't have our cake and eat it too; the convenient interface could just be a wrapper for the full-blown interface for the many module distributions that don't need a lot of fancy customization. First, here's an example of the simple interface, used for a module distribution with a single "pure Python" module (mymod.py). ------------------------------------------------------------------------ from distutils.core import setup setup (name = "mymod", version = "1.2", author = "Greg Ward ", description = "A very simple, one-module distribution") ------------------------------------------------------------------------ Note that we don't explicitly list "mymod.py" anywhere: Distutils assumes that this is a one-horse distribution named after its sole module ('mymod'). Those who enjoy defining subclasses might prefer to phrase this differently: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mymod" version = "1.2", author = "Greg Ward ", description = "A very simple, one-module distribution") setup (distclass = MyDistribution) ------------------------------------------------------------------------ This is overkill for a small distribution: we're defining a new class solely to provide attribute values, when 'distutils.core.setup' exists mainly to let do this anyways. Nevertheless, OO purists will like this -- and undoubtedly there will be times when the client *will* have to override behaviour, not just data, and the OO interface will be necessary. And more complex module distributions, with lots of attributes to customize, might be easier to read/maintain with things broken up like this. Consider a distribution with two pure Python modules ('mymod' and 'my_othermod') and a C extension ('myext'); the C extension must be linked with two ancillary C files and a C library. Oh yeah, this distribution requires Python 1.5 and any version of the 're' module: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mydist", version = "1.3.4", author = "Greg Ward " description = """This is an example module distribution. It provides no useful code, but is an interesting example of the Distutils in action.""" # Dependencies requires = { 'python': '1.5', # I like class-based exceptions 're': '', # and I love Perl-style regexps! ;-) } # (and yes, I *know* that "Python 1.5" # implies 're'...) # Actual files that need to be processed and installed in some form py_modules = ['mymod.py', 'my_othermod.py'], ext_modules = {'myext.c': {'other_c': ['extra1.c', 'extra2.c'], 'c_libraries': ['mylib']} } setup (distclass = MyDistribution) ------------------------------------------------------------------------ A couple of things to note: * I'm not afraid to use deeply nested data structures; if you're writing and distributing Python modules, this shouldn't be a problem! * every attribute has a particular type (string, list, dictionary, ...) * the attributes with complex types (especially dictionaries) will have a well-known and well-documented internal structure, eg. """ext_modules is a hash mapping names of C source files (each containing a Python extension module) to a nested hash of information about how to build that module. The allowed keys to this nested hash are: - other_c: other C files that must be compiled and linked with the main C file to create the module - c_libraries: C libraries that must be included in the link ... """ No doubt the 'ext_modules' nested hashes would have more options, and no doubt other Distribution attributes would have complex, documented structure. Finally, the list of all Distribution attributes must be well-known and well-documented! These seem to fall into a couple of broad categories. Here's an initial attempt at a list: Distribution meta-data name version author description Dependencies requires Files to be processed and installed py_modules ext_modules doc_files [eg. SGML source - or whatever std. we get for documentation] Build directories [all under "./blib" by default] build_lib - where to put platform-independent library files build_platlib - where to put platform-dependent library files build_exe - where to put executable programs (ie. scripts) build_html - where to put processed documentation (HTML) (etc... more documentation formats, at least) Installation directories [under sysconfig.LIBDEST by default] install_lib install_platlib install_exe install_html C compilation cc ccshared cflags ldflags ...well, that's a start. I still don't know how to make all those Unixish C compilation variables more cross-platform. Part 3: revisiting the Distutils' point of view ----------------------------------------------- To sum up, let's go through what happens when the user runs 'setup.py'. Whether setup.py is written in the simple (call-a-function) or general (define-a-subclass) form doesn't matter too much, so I won't split things up into two streams. * setup.py imports distutils.core * distutils.core startup code parses command-line arguments: processes global options that it knows about, and saves the rest for the client (setup.py) to deal with; figures out the commands and options for each command, saving them all up for later processing * setup.py calls distutils.core.setup (possibly with a 'distclass' argument specifying a subclass of Distribution, probably with a bunch of other named arguments specifying various attributes for the Distribution instance) * distutils.core.setup instantiates Distribution (or the subclass supplied by the client), and uses its arguments (apart from 'distclass') to override attributes of this instance * distutils.core.setup loads the command module (eg. 'distutils.build') * distutils.core.setup determines the command class (usually just named for the command, eg. 'distutils.build.Build, but possibly a class supplied by the client as one of the attributes of the Distribution instance) and instantiates it * the command class constructor takes as arguments the Distribution instance and any command-line arguments specific to this command on the setup.py command line * the command class constructor parses its options to set/override some instance attributes * distutils.core.setup calls the "make it so" method provided by the command class * that method does whatever the command is supposed to do: build modules, process documentation, install files, etc. * distutils.core.setup determines the next command class (if multiple commands were given), and proceeds as before Part 4: Unresolved issues ------------------------- * Where do we take care of platform dependencies? Somewhere, sometime, we'll need a class or function or attribute named 'foo_posix', 'foo_win32', 'foo_mac', etc. We might also need 'foo_linux_i86', 'foo_solaris2, 'foo_irix5', 'foo_winnt', 'foo_win98', 'foo_macos8', etc. * ...And that's just within Distutils itself. What about client code -- what if I have a module that sets itself up differently for different distributions; how do I specify that? * Recusive setup: what if my distribution has subdirectories containing other module distributions, with their own setup.py's? Will this ever be needed? (Undoubtedly.) How does the client specify them, and how do the Distutils run them? -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Fri Jan 15 04:20:57 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 14 Jan 1999 23:20:57 -0500 Subject: [Distutils] setup.py command-line syntax Message-ID: <19990114232057.B29195@cnri.reston.va.us> OK, time for the first of those unresolved issues. This one isn't actually mentioned as such in the "Unresolved issues" section of the design proposal, because if you carefully read and think through what's already there you'll start to see the holes in my proposal. In particular, we need to figure out what the syntax for the 'setup.py' command line is going to be. I have a couple of ideas, none of them wholly satisfactory. In vague terms, I'm currently thinking that the syntax will be like this: ./setup.py [global_options] cmd1 [cmd1_options] [cmd2 [cmd2_options]] ... The specific problem I'm having is: what will options look like, in particular options with arguments, and how do we know when the options for cmd1 are over and we're staring at cmd2? Option 1: getopt-ish Options are either of the '-x' form (single dash, single letter, may be jammed together, arguments may follow with or without intervening space) or of '--long-opt' form (two dashes, dash-or-underscore-separated-words, arguments will follow either '=' or whitespace). Pros: * familar interface: just like the getopt module, except that there are these "command" things intermingled with options, and there are multiple sources for option descriptors (the standard 'Distribution' class provided by distutils.setup.core, the individual command classes provided by the command modules, and possibly setup.py). Cons: * those complications mean that we'd have to reimplement the getopt module for distutils, with hooks to the various sources of option descriptors specific to distutils * due to many ways of specifying option arguments ("-a foo", "-afoo", "--aardvark=foo", "--aardvark foo"), can't tell where option lists ends and next command begins without fully processing the option descriptors and argument lists. This could induce temporal whiplash. Option 2: restricted getopt-ish Options are only of the '--long-opt' form, and option arguments may only be specified as '--option=value' -- no spaces between option and value (ie. they're the same command line word). Pros: * much easier to implement than a getopt knock-off * don't have to fully process options to know where cmd1's options stop and cmd2 starts: just look for first ! /^--/ after cmd1 * easy to make a standard way to negate boolean (no argument) options: just accept "--no-flag" in addition to "--flag" Cons: * perversion of a familiar interface; people will think that because they can say "--option=value" they can equivalently say "--option value" and will get confused when distutils tells them "no such command 'value'" (err, except it will *also* tell them "option '--option' requires an argument"... maybe not so bad then) * needless typing; double-dashes on long options are really only necessary to distinguish them from short options, and if we don't support short options why bother with the double dashes? Option 3: who needs dashes anyways? Options are always of the "option=value" form -- no dashes, must always have a value (even if that value is only 'yes' or 'no'). Pros: * simple to implement, simple to explain * no confusion with existing interfaces (except... see "Cons") * dead easy to spot end of 'option=value' list after each command Cons: * looks like we're running a Makefile! * no nice way to do boolean (argument-less) options Option 4: who needs command options anyways? Change the syntax to ./setup.py [options] cmd1 [cmd2 ...] and make all options the responsibility of distutils.core (or ensure that the client seutp.py and the various command classes get their option descriptors to distutils.core). Pros: * assuming we can get all the option descriptors to distutils.core in time, can just use standard 'getopt' module; the leftover arguments are of course the commands Cons: * unclear from the command-line whose responsibility each option is Anyone have any other schemes? Comments on these? Preferences? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19990114225659.A29195@cnri.reston.va.us> Message-ID: <13987.48079.467165.463343@weyr.cnri.reston.va.us> Greg Ward writes: > (Distinguishing the end of the option list for cmd1 from cmd2 might be a > bit tricky if command options can take arguments, which they certainly > should do. If we want getopt-style syntax, it looks like we'll have to > reinvent the getopt module. :-( And if we opt for a different syntax, Greg, I don't see any need to re-implement getopt here; the existing module should work just fine. > developer (the person who wrote the setup.py for this module > distribution) may define a set of distribution-specific command-line > options, which may be mixed in with the global options (and extracted The need for this is not clear. > ------------------------------------------------------------------------ > from distutils.core import setup > setup (name = "mymod", > version = "1.2", > author = "Greg Ward ", > description = "A very simple, one-module distribution") > ------------------------------------------------------------------------ This is good. > ...well, that's a start. I still don't know how to make all those > Unixish C compilation variables more cross-platform. Meta-comment: I am very concerned that the non-Unix crowd doesn't seem to be involved with this project. Is there any way to make them aware of the effort? > * the command class constructor parses its options to set/override > some instance attributes It should get the result of getopt.getopt(...)[0] as a parameter, pre-parsed, with the class attributes "shortopts" and "longopts" (or some such) used for the parameters to getopt.getopt(). -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." The option processing Greg proposed in his design doc matches that of CVS: there are a bunch of global options, and then each command has its own options. The big difference is that cvs commands take arguments as well (the list of files to operate on), whereas Greg's proposal is to allow multiple commands, each of which has private options. I don't see anywhere in Greg's proposal a requirement that commands take arguments as well as options, so there's no reason not to use the getopt module. The only aspect of it is remotely interesting is switching in different sets of options for each different command class, and that's pretty trivial. Here's a little demo code. It accepts anything as a command, and just keeps the same options for each, but the comments should make it more than clear what needs to be done where to get the expected behaviour. ------------------------------------------------------------------------ """Example of the command line syntax used by distutils, using getopt.""" import getopt import sys LONGOPTS = ["help", "verbose"] SHORTOPTS = "hv" def main(): args = [""] + sys.argv[1:] while args: cmd = args[0] # switch in different SHORTOPTS and LONGOPTS in this call to # support different options for different commands: cmdinfo = sys.modules[__name__] # cmdinfo = load_command(cmd) opts, args = getopt.getopt(args[1:], cmdinfo.SHORTOPTS, cmdinfo.LONGOPTS) # do something with the command here: print cmd, opts # create_command(opts).run() if __name__ == "__main__": main() ------------------------------------------------------------------------ -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From sanner@scripps.edu Mon Jan 18 23:35:08 1999 From: sanner@scripps.edu (Sanner Michel) Date: Mon, 18 Jan 1999 15:35:08 -0800 Subject: [Distutils] packages and Maintaining Python in general Message-ID: <36A3C52B.C94FBDDC@scripps.edu> Following Barry's advice I post my message on this list too. Sorry for those of you you get it twice, but I wasn't aware of that SIG ! now I am a subscriber :) Hello, I have spend my week end moving opn from Python1.5 to Python1.5.2b1 and have come to the conclusion that there OUGHT to be better way !!! It is a lot of work because I try to maintain Python for - Sgi under Irix5.3 Irix6 and Irix646 - Sun - Dec alpha under OSF1.2 and OSF 1.4 - HP - Linux - Cray T3e and we use many "packages", NumPy, PIL, PyOpenGL. In addition I want to build only one Python environment for each architecture/OS and place it on some disk mounted by all these computers. Although scripts coming with some of the packages help greatly (thanks David) there are still a number of things that I have problems with. Here are some of the problems I have been strugling with: 1 - I started to use Makefile.pre.in to generate make file for additional packages. Currently when any user types "python" he gets 1.5 (current public version). So I decided to build Python1.5.2b1 and install it somewhere not in the users path. The problem is that when I use Makefile.pre.in it uses "python" and gets version 1.5. The only work around I found was to edit the copy of Makefile.pre.in and make it point to the 1.5.2b1 binary. 2 - When building Python for several architectures, the installation script allows to separate platform-dependant and platform-independant files. I'd like the same for packages, i.e. when a package has both types of files I'd like to be able to have a package with that name in the platform-independant AND in the platform-dependant tree. This DOES NOT work currently because import stops searching after it find a package matching that name. 3 - Numeric Python should really be part of the standard distribution .. (please :). Actually I would even argue that it should be part of the Kernel such that developers can rely on the fact that this data type, CRUCIAL for any scientific application and key to Python's success in that field is available. This would also allow for instance to have this datatype included in the Corba mappings for Python. 4 - How difficult would it be to provide some more flexibility in the directory sctructure created by "make install": I'd like to have something like: PYTHONROOT = /mgl/tools/python/$VERSION $PYTHONROOT/include $PYTHONROOT/lib $PYTHONROOT/man $PYTHONROOT/packages # arch independent packages $PYTHONROOT/$ARCH_OS/bin $PYTHONROOT/$ARCH_OS/include $PYTHONROOT/$ARCH_OS/lib $PYTHONROOT/$ARCH_OS/packages # arch dependent packages I do not need the Python version "python1.5" below that. In that way I can install any new version in parallele with an existing one. Any advice/help/modification is very welcome :) I am also going to post some messages that are more specific to particular packages ... so bear with me ! Thanks a lot -Michel -- ------------------------------------------------------------------ Sanner Michel Ph.D. The Scripps Research Institute 10550 North Torrey Pines Road La Jolla, CA 92037 Tel. (619) 784-2341 Fax. (619) 784-2860 e-mail sanner@scripps.edu Home Page http://www.scripps.edu/pub/olson-web/people/sanner From gward@cnri.reston.va.us Tue Jan 19 13:57:44 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 19 Jan 1999 08:57:44 -0500 Subject: [Distutils] packages and Maintaining Python in general In-Reply-To: <36A3C52B.C94FBDDC@scripps.edu>; from Sanner Michel on Mon, Jan 18, 1999 at 03:35:08PM -0800 References: <36A3C52B.C94FBDDC@scripps.edu> Message-ID: <19990119085744.A9141@cnri.reston.va.us> Quoth Sanner Michel, on 18 January 1999: > I have spend my week end moving opn from Python1.5 to Python1.5.2b1 and > have > come to the conclusion that there OUGHT to be better way !!! Yeah, I came to that conclusion within about two weeks of starting to use Python (last September). All it took was trying to find the correct version of NumPy (the Matrix SIG web page at that time pointed to Jim Hugunin's old ftp space at MIT, which is a tad out-of-date), and then build it. It wasn't impossible, it was just clunkier than it ought to be. I whined about this in the presence of Fred Drake and Andrew Kuchling, they both agreed, and ultimately this SIG was created. > 1 - I started to use Makefile.pre.in to generate make file for > additional packages. Currently when any user types "python" he gets > 1.5 (current public version). So I decided to build Python1.5.2b1 and > install it somewhere not in the users path. The problem is that when I > use Makefile.pre.in it uses "python" and gets version 1.5. The only > work around I found was to edit the copy of Makefile.pre.in and make > it point to the 1.5.2b1 binary. This will definitely be addressed by the Distutils suite: Makefile.pre.in will be a thing of the past, replaced by a (usually) tiny Python script. Normally this will just be run as "./setup.py", but of course you can always say "python1.5.2b1 setup.py" and the build process will follow the defaults associated with *that* particular binary (eg. compiler flags, installation directories, etc.). > 2 - When building Python for several architectures, the installation > script allows to separate platform-dependant and platform-independant > files. I'd like the same for packages, i.e. when a package has both > types of files I'd like to be able to have a package with that name in > the platform-independant AND in the platform-dependant tree. This DOES > NOT work currently because import stops searching after it find a > package matching that name. I completely agree. I think this will require some cooperation on the part of Python's configure/build process, and probably the startup code that initializes sys.path. So it's only likely to happen when/if a) Distutils comes to pass as planned, b) Python 1.6 goes out. But it's definitely an important thing to keep in mind. > 3 - Numeric Python should really be part of the standard distribution > .. (please :). Actually I would even argue that it should be part of > the Kernel such that developers can rely on the fact that this data > type, CRUCIAL for any scientific application and key to Python's > success in that field is available. This would also allow for > instance to have this datatype included in the Corba mappings for > Python. I disagree. The whole point of the Distutils SIG is to make it trivial to add third-party modules to an existing Python installation; if NumPy being crucial for scientists justifies adding it to the core, then why not add PIL for image-processors, and so forth? Where do you draw the line? > 4 - How difficult would it be to provide some more flexibility in the > directory > sctructure created by "make install": > > I'd like to have something like: > > PYTHONROOT = /mgl/tools/python/$VERSION > > $PYTHONROOT/include > $PYTHONROOT/lib > $PYTHONROOT/man > $PYTHONROOT/packages # arch independent packages > $PYTHONROOT/$ARCH_OS/bin > $PYTHONROOT/$ARCH_OS/include > $PYTHONROOT/$ARCH_OS/lib > $PYTHONROOT/$ARCH_OS/packages # arch dependent packages > > I do not need the Python version "python1.5" below that. In that way I > can > install any new version in parallele with an existing one. Interesting proposal. I think Python should support something like its existing scheme (where its installation is scattered across lib, bin, man, and include directories under $prefix and $exec_prefix) and something like Michel's proposal (where it's concentrated under a single "Python root"). Of course, you can always hack the latter by saying "./configure --prefix=/usr/local/python", but then you end up with the library in "/usr/local/python/lib/python1.5", which is a bit weird. You do get version discrimination on the binary, library, and include files, but at a pretty low level; there's a lot to be said for discriminating early on, as in "/usr/local/python/1.5.2b1/...". Many issues to be discussed there. Obviously have my opinions... anyone else? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From M.Faassen@vet.uu.nl Tue Jan 19 15:41:36 1999 From: M.Faassen@vet.uu.nl (Martijn Faassen) Date: Tue, 19 Jan 1999 16:41:36 +0100 Subject: [Distutils] Revised design proposal References: <19990114225659.A29195@cnri.reston.va.us> <13987.48079.467165.463343@weyr.cnri.reston.va.us> Message-ID: <36A4A7B0.DFD8F51F@pop.vet.uu.nl> [argh..I keep forgetting 'reply' in Netscape doesn't work when posting to a mailing list, only when posting to a newsgroup. Second try. Sorry for Fred who gets it twice.] "Fred L. Drake" wrote: > Greg Ward writes: > > ...well, that's a start. I still don't know how to make all those > > Unixish C compilation variables more cross-platform. You don't expose Unixish C compilation variables to the user at all, perhaps? At least in principle. In practice you'd still allow modification, but it shouldn't be *necessary* to look at these or even know about them, even to write a C extension. (unless you're on some new platform) As we talked about before: * Make an abstract compiler class that can do simple C (and perhaps C++) compilation and linking. Subclass for each desired platform (or platform+compiler combination, such as win95-cygwin, or pentium-linux2-egcs). * Each compiler subclass just has a bunch of plausible default options for optimization (-O2 for gcc, for instance). Mind that we're not trying to build an entire compiler/linker abstraction. We just need enough to compile and link for Python. This way the extension writer does not need to even think about compiler options for a zillion platforms when making the module. The compiler class for the platform knows anyway. Linking in external libraries is a stickier point that we need to talk about some more. We need a way to specify 'external' resources for modules. These would be Python independent C libraries that your module makes use of. The distutils build system wouldn't need to know about building these; it just needs to know how to compile and link C (and C++) files to work as Python modules. The distutils should allow a way for a module distribution to check for presence of the external resources, and to issue build commands to external libraries. The external libraries are expected to be equipped with platform independent build systems anyway. Of course the extension writer should be able to supply special information for each platform *if desired*. If you know --funroll-all-loops or whatever will optimize your module even further using gcc on Linux, you can replace the default linux-gcc compiler class with your own specialized one. > Meta-comment: I am very concerned that the non-Unix crowd doesn't > seem to be involved with this project. Is there any way to make them > aware of the effort? I consider myself part of the Unix crowd, though I'm currently using Python on a Windows machine at work. I imagine there's only the regular way of making people aware of this effort, posting to the newsgroup and announcing on the website. Perhaps a special 'distutils SIG' status report, posted once every while and updating everybody on developments would be useful to attract interest. And what about something in the Python Journal? Regards, Martijn From M.Faassen@vet.uu.nl Tue Jan 19 15:42:42 1999 From: M.Faassen@vet.uu.nl (Martijn Faassen) Date: Tue, 19 Jan 1999 16:42:42 +0100 Subject: [Distutils] setup.py command-line syntax References: <19990114232057.B29195@cnri.reston.va.us> Message-ID: <36A4A7F2.7EBDF5CF@pop.vet.uu.nl> [another arg..I sent this off to Greg Ward alone instead of to the mailing list. Here it is for everybody else :)] Greg Ward wrote: [a whole slew of options on command line options] > Anyone have any other schemes? Comments on these? Preferences? What about combining it with a options(.py?) file? setup.py would simply implement a number of major options, like: # install everything (default: standard dirs) (build if necessary?) setup install # build everything (default: for this platform) setup build # build docs setup build_docs # install docs setup install_docs This would do everything in the default way. Especially installation (for which distutils will be used a lot anyway) will benefit from this, as it's simple. No need for most people to even think about options. Then we have a simple options file, text, or perhaps better python, copiously commented: # build what kind of docs docs_output = ['html', 'txt'] # install Python files to this directory install_dir = '/usr/local/lib/python1.5' The idea of an options file is: * It has comments. You don't need to remember a whole slew of command line options or read through an INSTALL document to see what command line options there are in the first place. * You can have a site default options file (in combo with a package specific options file?). That way you don't have to command line option each package if you want to install it to a 'strange' directory, or whatever. * Perhaps it's also easier to understand for non Unix Pythoneers, who aren't used to command line options. Regards, Martijn From akuchlin@cnri.reston.va.us Tue Jan 19 17:40:25 1999 From: akuchlin@cnri.reston.va.us (Andrew M. Kuchling) Date: Tue, 19 Jan 1999 12:40:25 -0500 (EST) Subject: [Distutils] Revised design proposal In-Reply-To: <36A4A7B0.DFD8F51F@pop.vet.uu.nl> References: <19990114225659.A29195@cnri.reston.va.us> <13987.48079.467165.463343@weyr.cnri.reston.va.us> <36A4A7B0.DFD8F51F@pop.vet.uu.nl> Message-ID: <13988.43225.287501.970396@amarok.cnri.reston.va.us> Martijn Faassen quoted: >Greg Ward writes: >> Meta-comment: I am very concerned that the non-Unix crowd doesn't >> seem to be involved with this project. Is there any way to make them >> aware of the effort? I wouldn't worry too much about that; if a Unix-based effort bears fruit, then users of other platforms will want to benefit from it too, and someone will do the porting work required. I'll post a second message about Martijn's idea of a weekly status report, because I want to cross-post it to the main list and discuss it there. -- A.M. Kuchling http://starship.skyport.net/crew/amk/ When I was a child I used to draw people as animals - elephants, giraffes, mice. Was I retreating into childhood? I kept thinking of my doctor as a big cat. A snow leopard. A beautiful predator. -- The film director in SIGNAL TO NOISE From akuchlin@cnri.reston.va.us Tue Jan 19 18:24:58 1999 From: akuchlin@cnri.reston.va.us (Andrew M. Kuchling) Date: Tue, 19 Jan 1999 13:24:58 -0500 (EST) Subject: [Distutils] Weekly/biweekly development report? Message-ID: <199901191824.NAA03339@amarok.cnri.reston.va.us> In the Distutils-SIG, Martijn Faassen wrote: >Perhaps a special 'distutils SIG' status report, posted once every while >and updating everybody on developments would be useful to attract >interest. And what about something in the Python Journal? A recent trend for development projects has been to issue a summary of activity at periodic intervals. It may have started with the development summary at Linux Weekly News, but it's been picked up by other groups such as the Debian GNU/Linux distribution and Mozilla.org. There are also development diaries from various people, most notably Alan Cox: LWN Development page for this week: http://www.lwn.net/1999/0114/devel.phtml LWN Kernel development page for this week: http://www.lwn.net/1999/0114/kernel.phtml Debian Weekly News: http://www.debian.org/~joeyh/debiannews.html Mozilla.org status update: http://www.mozilla.org/status/ Linux kernel update: http://lynx.neu.edu/z/zbrown/kt.html Alan Cox's diary: http://www.linux.org.uk/diary/ Should we attempt to have a monthly summary of Python development activity? This is slightly different from the Python-URL postings, which aren't focused exclusively on development, and don't have much editorial commentary. I like LWN's habit of occasionally commenting on the news, saying "This flamewar was rather pointless" or "This recent argument was upsetting because...". The Python version of this might look like: Distutils SIG: Greg Ward posted a proposed interface (URL to a copy). Once some people have commented on it, implementation work should begin. Zope: DC released new database adapters X,Y,Z. Skip Montanaro announced thing W. Python-list: Endless discussion of assignment in while statements; GvR said he's not interested. ... ...etc... Pulling together the information requires several volunteers, because no one can possibly be reading every single relevant mailing list and newsgroup. How could that be made workable? Have a central editor? A CGI script which lets anyone add suggested items, which an editor draws upon for each issue and then gets cleaned out when the issue is released? Include it as part of the Python Journal? Suggestions? -- A.M. Kuchling http://starship.skyport.net/crew/amk/ He told me he was having the time of his life and then fell to the floor unconscious. I didn't take this too personally, although he was not the first person to collapse while talking to me; to tell the truth it has happened quite often. -- Tom Baker, in his autobiography From sanner@scripps.edu Tue Jan 19 18:33:03 1999 From: sanner@scripps.edu (Michel Sanner) Date: Tue, 19 Jan 1999 10:33:03 -0800 Subject: [Distutils] packages and Maintaining Python in general In-Reply-To: Greg Ward "Re: [Distutils] packages and Maintaining Python in general" (Jan 19, 8:57am) References: <36A3C52B.C94FBDDC@scripps.edu> <19990119085744.A9141@cnri.reston.va.us> Message-ID: <9901191033.ZM20001@cain.scripps.edu> On Jan 19, 8:57am, Greg Ward wrote: > Subject: Re: [Distutils] packages and Maintaining Python in general > Quoth Sanner Michel, on 18 January 1999: > > > 1 - I started to use Makefile.pre.in to generate make file for > > additional packages. Currently when any user types "python" he gets > > 1.5 (current public version). So I decided to build Python1.5.2b1 and > > install it somewhere not in the users path. The problem is that when I > > use Makefile.pre.in it uses "python" and gets version 1.5. The only > > work around I found was to edit the copy of Makefile.pre.in and make > > it point to the 1.5.2b1 binary. > > This will definitely be addressed by the Distutils suite: > Makefile.pre.in will be a thing of the past, replaced by a (usually) > tiny Python script. Normally this will just be run as "./setup.py", but > of course you can always say "python1.5.2b1 setup.py" and the build > process will follow the defaults associated with *that* particular > binary (eg. compiler flags, installation directories, etc.). > Cool I see you guys already have a plan :). I guess I will have to browse the archive of the mailing list to come up to speed ! Anyways is the plan to have something available under 1.6 ? or maybe 1.5.2 (non beta ?) > > 2 - When building Python for several architectures, the installation > > script allows to separate platform-dependant and platform-independant > > files. I'd like the same for packages, i.e. when a package has both > > types of files I'd like to be able to have a package with that name in > > the platform-independant AND in the platform-dependant tree. This DOES > > NOT work currently because import stops searching after it find a > > package matching that name. > > I completely agree. I think this will require some cooperation on the > part of Python's configure/build process, and probably the startup code > that initializes sys.path. So it's only likely to happen when/if a) > Distutils comes to pass as planned, b) Python 1.6 goes out. But it's > definitely an important thing to keep in mind. > Yeap, the main problem currently is that import stops searching after it find a package with the right name. So you could NOT have an OpenGL package (for instance) under $prefix and another OpenGL package under $exec_prefix. Is someone talking to Guido about this. It seems to me that this separation between acrh-dependant and arch-independant files should be allowed for packages. > > 3 - Numeric Python should really be part of the standard distribution > > .. (please :). Actually I would even argue that it should be part of > > the Kernel such that developers can rely on the fact that this data > > type, CRUCIAL for any scientific application and key to Python's > > success in that field is available. This would also allow for > > instance to have this datatype included in the Corba mappings for > > Python. > > I disagree. The whole point of the Distutils SIG is to make it trivial > to add third-party modules to an existing Python installation; if NumPy > being crucial for scientists justifies adding it to the core, then why > not add PIL for image-processors, and so forth? Where do you draw the > line? > OK ! I think my level of frustration made me loose my objectivity :) .. although there is something special about Numeric: it implements a basic datatype: ARRAYS that are as basic as lists or tuples. This I think makes a difference with PIL for instance that might use Numeric Arrays to store images for instance. Moreover having this datatype be a part of Python itself would allow to define CORBA mappings (which again I am personally interested in :)... Just pushing my luck here :) -Michel From da@skivs.ski.org Tue Jan 19 18:56:45 1999 From: da@skivs.ski.org (David Ascher) Date: Tue, 19 Jan 1999 10:56:45 -0800 (Pacific Standard Time) Subject: [Distutils] Revised design proposal In-Reply-To: <36A4A7B0.DFD8F51F@pop.vet.uu.nl> Message-ID: > Meta-comment: I am very concerned that the non-Unix crowd doesn't > seem to be involved with this project. Is there any way to make them > aware of the effort? I'm keeping an eye on the list, just don't have time to get deeply involved. As long as you don't assume a standard makefile format, and keep flexibility in the installation locations, we can probably fit win32 around it. Someone should also think about whether adding some support for registry manipulations makes sense, in the long run. --david From Fred L. Drake, Jr." References: <36A4A7B0.DFD8F51F@pop.vet.uu.nl> Message-ID: <13988.55670.667893.592503@weyr.cnri.reston.va.us> David Ascher writes: > I'm keeping an eye on the list, just don't have time to get deeply > involved. As long as you don't assume a standard makefile format, and > keep flexibility in the installation locations, we can probably fit win32 Glad to hear it! > Someone should also think about whether adding some support for registry > manipulations makes sense, in the long run. Probably so; I guess this means it'll need to be added to the core Python distribution for Win32. I'm sure a basic registry module wouldn't be hard to write (unless it's o.k. to abscond an existing bit of code?? ;), but this is largely up to Guido (accepting it, not necessarily writing it) since it would have to be core. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From da@skivs.ski.org Tue Jan 19 19:12:18 1999 From: da@skivs.ski.org (David Ascher) Date: Tue, 19 Jan 1999 11:12:18 -0800 (Pacific Standard Time) Subject: [Distutils] Revised design proposal In-Reply-To: <13988.55670.667893.592503@weyr.cnri.reston.va.us> Message-ID: On Tue, 19 Jan 1999, Fred L. Drake wrote: > David Ascher writes: > > Someone should also think about whether adding some support for registry > > manipulations makes sense, in the long run. > > Probably so; I guess this means it'll need to be added to the core > Python distribution for Win32. I'm sure a basic registry module > wouldn't be hard to write (unless it's o.k. to abscond an existing bit > of code?? ;), but this is largely up to Guido (accepting it, not > necessarily writing it) since it would have to be core. Right. The relevant code would have to move from Mark's distribution to Guido's. That's why I said "in the long run". One big advantage is it'd make it possible to do nice installers without having to use WISE, which is very hard to get right. --david From MHammond@skippinet.com.au Wed Jan 20 00:00:16 1999 From: MHammond@skippinet.com.au (Mark Hammond) Date: Wed, 20 Jan 1999 11:00:16 +1100 Subject: [Distutils] Revised design proposal In-Reply-To: <13988.55670.667893.592503@weyr.cnri.reston.va.us> Message-ID: <006c01be4407$dce704e0$0801a8c0@bobcat> > > Someone should also think about whether adding some > support for registry > > manipulations makes sense, in the long run. > > Probably so; I guess this means it'll need to be added to the core > Python distribution for Win32. I'm sure a basic registry module > wouldn't be hard to write (unless it's o.k. to abscond an > existing bit > of code?? ;), but this is largely up to Guido (accepting it, not > necessarily writing it) since it would have to be core. he he - Im here too :-) Guido has said he is happy to accept some of the Win32 stuff into the core for 1.6 - notably the Registry manipulation functions, and a "PyHANDLE" object - a wrapper around a win32 handle that is auto-closed. And as David said, I too am just keeping an eye half-open, and will pipe up if I see anything that wont work on Windows (with just a little porting effort :-) Mark. From gward@cnri.reston.va.us Wed Jan 20 15:17:18 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 20 Jan 1999 10:17:18 -0500 Subject: [Distutils] Responses to proposal Message-ID: <19990120101718.A16893@cnri.reston.va.us> [Fred says the need for application-specific command-line options is not clear] My impetus for this was the need for specific module distributions to be told where to go looking for external C libraries. For instance, the setup.py for zlib would need to know where to find zlib.h and libz.a -- that is, how to generate a -I option for compiling the extension, and a -L option for linking it (modulo the usual Unixisms). Likewise, tkinter needs to be told where to find the Tcl and Tk headers and libraries. Granted, there's already a standard way of doing this: the Setup file. Perhaps we should preserve the Setup file, or perhaps we should mutate it into a Distutils-specific 'options' file as per Martijn's suggestion for replacing my command-line options. However, let's not limit ourselves to "where's this header file, where's that library?" questions: lots of modules could have configuration options that the builder might want to supply at build time. And keep in mind that ultimately we want to be able to write setup.py for a whole Python *applications*, which almost certainly will have configuration options: where to put temporary files? permanent data files? installation directory? etc. [Martijn Faassen berates me for forgetting about abstract compiler classes] > You don't expose Unixish C compilation variables to the user at all, > perhaps? At least in principle. In practice you'd still allow > modification, but it shouldn't be *necessary* to look at these or even > know about them, even to write a C extension. (unless you're on some new > platform) Duh, you're right. I had *Programming Perl* sitting on my lap and open to the section on MakeMaker as I wrote that part of the design proposal. (Mainly I wanted to be sure I didn't forget anything obvious.) Small wonder a few Unixisms creeped in! Anyways, this raises the broader question: how should the class(es) that abstract compilation and linking behave? I'll try to come up with a few ideas and post them separately. (Arg, really must look at Interscript's compilation stuff... presumably John has already thought through this stuff!) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19990120101718.A16893@cnri.reston.va.us> Message-ID: <13989.63872.34871.754037@weyr.cnri.reston.va.us> Greg Ward writes: > My impetus for this was the need for specific module distributions to be > told where to go looking for external C libraries. For instance, the Why do these need to be application specific? Appropriate options can be defined (such as -I and -L;) to do this without making them application specific. > However, let's not limit ourselves to "where's this header file, where's > that library?" questions: lots of modules could have configuration > options that the builder might want to supply at build time. And keep These should be easily handled using command-specific options, which may or may not need to be application-specific: A few basic options might be useful in the standard implementation simply so that the option names are consistent (where to put temp files, config files, etc.). I'm inclined to think we need to elaborate on the command dispatch mechanics a little to make sure we do the right thing to support both application-specific extensibililty and site-specific extensibility. I'll make a mini-proposal: Each command should be defined by a class. What class that is is determined using a registry initialized from a configuration file for the site (to allow site administration hooks) and the contents of a distutils sub-package. The application can update the registry as needed. This might look like this: ------------------------------------------------------------------------ import distutils.registry baseclass = distutils.registry.get_cmd_class("install") class DerivedInstallCmd(baseclass): LONG_OPTIONS = baseclass.LONG_OPTIONS + ["enable-foo", "disable-foo"] use_foo = 0 def __init__(self, options): for item in options[:]: opt, arg = item if opt == "--enable-foo": self.use_foo = 1 options.remove(item) elif opt == "--disable-foo": self.use_foo = 0 options.remove(item) baseclass.__init__(self, options) def run(self): # use self.use_foo somehow... distutils.registry.set_cmd_class("install", DerivedInstallCmd) ------------------------------------------------------------------------ The command dispatcher can use the same distutils.registry.get_cmd_class() function to get the class that should actually be used to implement the command, allowing all the "hooked-in" functionality to remain. Comments? -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From M.Faassen@vet.uu.nl Wed Jan 20 16:08:43 1999 From: M.Faassen@vet.uu.nl (Martijn Faassen) Date: Wed, 20 Jan 1999 17:08:43 +0100 Subject: [Distutils] Responses to proposal References: <19990120101718.A16893@cnri.reston.va.us> Message-ID: <36A5FF8B.53C85949@pop.vet.uu.nl> Greg Ward wrote: > > [Fred says the need for application-specific command-line options is not > clear] > > My impetus for this was the need for specific module distributions to be > told where to go looking for external C libraries. For instance, the > setup.py for zlib would need to know where to find zlib.h and libz.a -- > that is, how to generate a -I option for compiling the extension, and a > -L option for linking it (modulo the usual Unixisms). Likewise, tkinter > needs to be told where to find the Tcl and Tk headers and libraries. I think dependencies on external packages should be abstracted as much as possible. Otherwise extension/application writers still end up having to write specific configuration stuff for each platform Python runs on, which wasn't what we wanted. Perhaps we can distribute 'external package info files' for common external packages, describing how this package works on various platforms (where the files are, what they're called, how to link to them, etc). These external package info files should be independent of the Setup scripts of specific applications. This way someone else than the extension writer can maintain them, and you only need to do write an information file once for each external resource. In the end the writers of libraries may start making these info files by themselves. :) > Granted, there's already a standard way of doing this: the Setup file. > Perhaps we should preserve the Setup file, or perhaps we should mutate > it into a Distutils-specific 'options' file as per Martijn's suggestion > for replacing my command-line options. And I think options file versus command line options warrants some more discussion, too. Not a whole flamewar, but I wouldn't like to settle on a vast slew of command line options if a nicely commented options file can be used instead. Obviously I'm biased more towards options files, see my earlier discussion in the thread. [snip] > [Martijn Faassen berates me for forgetting about abstract compiler classes] > > You don't expose Unixish C compilation variables to the user at all, > > perhaps? At least in principle. In practice you'd still allow > > modification, but it shouldn't be *necessary* to look at these or even > > know about them, even to write a C extension. (unless you're on some new > > platform) > > Duh, you're right. I had *Programming Perl* sitting on my lap and open > to the section on MakeMaker as I wrote that part of the design proposal. > (Mainly I wanted to be sure I didn't forget anything obvious.) Small > wonder a few Unixisms creeped in! > > Anyways, this raises the broader question: how should the class(es) that > abstract compilation and linking behave? I'll try to come up with a few > ideas and post them separately. (Arg, really must look at Interscript's > compilation stuff... presumably John has already thought through this > stuff!) Actually John Skaller and I and others had some dialogue on this topic on this list some time ago. John posted some relevant code, I posted some comments on this (if I didn't reply to sender instead of to list, as I'm about to do now, but I caught myself just in time :) If you want I can probably drag it up in my mailbox. I also posted an early design of what I thought it should look like, and what the requirements are. Not-berating-though-just-reminding-ly yours, Martijn From Fred L. Drake, Jr." References: <19990120101718.A16893@cnri.reston.va.us> <36A5FF8B.53C85949@pop.vet.uu.nl> Message-ID: <13990.814.279589.618013@weyr.cnri.reston.va.us> Martijn Faassen writes: > Perhaps we can distribute 'external package info files' for common > external packages, describing how this package works on various > platforms (where the files are, what they're called, how to link to This is interesting. It sounds like we can go a long with something like this. It is interesting to note that gtk+ installs a script "gtk-config" that allows other makefiles to use something like: CFLAGS=`gtk-config --cflags` LDFLAGS=`gtk-config --libs` This is probably not the right approach for most packages, but it's worth being aware of. Perhaps we could supply (possibly as part of distutils) a script py-config that takes essentially the same arguments as gtk-config, but also a --package to specify what package we're actually interested in, or require a positional parameter that specifies the package: py-config --package zlib --version - or - py-config --version zlib Maintaining a sufficiently large collection of package info files may be difficult, but it's hard to tell. It may be feasible to use native package management databases on systems that use them; something like RPM may provide enough information, but I don't know enough about SysVR4 PKG system yet. > information file once for each external resource. In the end the writers > of libraries may start making these info files by themselves. :) Ideally, installed on package installation. > And I think options file versus command line options warrants some more > discussion, too. Not a whole flamewar, but I wouldn't like to settle on > a vast slew of command line options if a nicely commented options file I don't see any reason both can't be used. Command line options could be used to override the options file if necessary. This may point to a stronger approach than getopt to the option parsing issue. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From gstein@lyra.org Sun Jan 31 16:02:56 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 31 Jan 1999 08:02:56 -0800 Subject: [Distutils] FW: new import mechanism and a "small" distribution Message-ID: <000001be4d33$2a3db2f0$0e2bc0d0@suntzu.lyra.org> This is a multi-part message in MIME format. ------=_NextPart_000_0001_01BE4CF0.1C1CBCE0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit crap... missed the "s" in distutils. what fun now -- divergent threads (here and python-list). sigh. -----Original Message----- From: Greg Stein [mailto:gstein@lyra.org] Sent: Sunday, January 31, 1999 8:00 AM To: distutil-sig@python.org Cc: python-list@python.org Subject: new import mechanism and a "small" distribution A while back, Mark Hammond brought up a "new import architecture" proposal after some discussion with Jack Jansen and Guido. I responded in my usual diplomatic style and said "feh. the two-step import style is bunk." Then, I expounded on a few of the ideas that I had come up with after working with a lot of custom import stuff (for COM support, for sticking .pyc files into resources or structured-storage files, or for creating distribs with very few files). In essence, I maintained that importers should really be a single step find-and-load, and that Python would be best served by having a list of object instances, each performing this find-and-load step. Imagine, if you will, replacing sys.path with a list of "directory importers" where each importer checks a directory for the requested file. Next step in our hypothetical case is to add a new importer into sys.path that checks the Windows Registry for paths and looks there, or an importer that looks inside the resource fork of a Mac file. Heck, or use the typical example: have an importer that grabs a code object from over the web. Okay... enough background and rambling. If you're interested, go look at the four messages in the thread titled "Freeze and new import architecture" in the distutils-sig archive at: http://www.python.org/pipermail/distutils-sig/1998-December/thread.html What I'm posting here is two things: 1) imputil.py : this is a module that defines an import utility class (imputil.Importer). There are a few subclasses in here for different types of importers. More detail below. 2) a "small distribution" : the demo is for Win32, but it the basic mechanism is quite portable. In this small distro, you can place onto somebody's computer a mere 5 files and NO registry keys. Python starts up just fine and you can import all modules from the standard library. Basically, imputil.Importer's logic was derived from knee.py, which is an emulation of Python's algorithm. Importer defines one method that subclasses should override: get_code(). The doc strings and examples in the module cover that method relatively well. The Importer class will install an import hook and "chain" to the previous hook. This allows the use of multiple Importers, each one attempting to perform the import, and finally falling back to the builtin import mechanism. (note that there is no way to deinstall an importer at the moment, as it can't go "up" the chain to detach itself; the "right" way to do this is to define an Importer that contains a manageable list of Importers, and removing the chaining in favor of the manager iterating over the list) The small distribution replaces site.py and uses an archive of the standard library to fetch code objects from. This library is named "py15.pyl" in the demo, and is generated by the batch file in support/regen.bat (with the help of support/easygen.py). The resulting library is portable, as it is merely a giant glom of .pyc files and a marshalled table of contents (mapping module names to file positions). Look in site.py for the SimpleArchive class to get a feel for how to write a simple subclass of Importer. As another demo, imputil.py contains a class named DirectoryImporter. This subclass attempts to emulate Python's behavior for importing from a directory. It is instantiated with a directory to look in, for modules and packages. If a module is found, it will perform the usual .pyc caching if necessary (or use the cached .pyc). There is a demo function named imputil._test_dir() that will read sys.path and construct a chain of DirectoryImporter instances. You can then import away and Python will actually use the Importers rather than sys.path iteration. Lastly, imputil.py contains two (untested) subclasses for importing from a package contained in archive-type files (such as .gz, .zip, .tar, or a Mac file of resources). For example, you could distribute "package.gz" along with a gzip Importer. When somebody said "import package.module", it would look inside package.gz for the "module" module. I think that's it. This is the first release of this stuff, so there are bound to be improvements or bugfixes or just simple comments :-). Please feel free to speak up. thx, -g -- Greg Stein, http://www.lyra.org/ ------=_NextPart_000_0001_01BE4CF0.1C1CBCE0 Content-Type: application/x-zip-compressed; name="Small.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Small.zip" UEsDBBQAAAAIANUGnyPalk5YhwQAAAAQAAAKAAAAcHl0aG9uLmV4Ze1XT2zbVBj/HLc07Zo2QwtH 5qJW2gFFGR5iiibkwTJxaITbLmUCgRsSt07VOpHjrqvUQ9QAa2QqKsSBwg6pygEiDkXqYYIdMq0i CHVakXrYoUI5DCmCHRCnHSoe3/NznaTLNg7TJFB/yc/v/b73fZ/fv9gv0beXgQeANiQhANeAQYLH I4fsOf59D2x03uq7xg3e6rugpbJCxkhPGPFpIRHX9bQpvK8KxowupHTh3JsjwnQ6qQZ9vq5+J4cc ARjk2qHv129O7eetQi93hGv3gAdFF7OF/HihFKiSWN3D+k2xX8JYXfD21c983dItGCSAM7SUAa7y 8MQwLz16DoOmetnEcpcDdyz1QTAIOJqgkYybcYBux2D7PdPsJ+E3yNzYWEKOn/cBv3IwxRyLHDM0 znGTn5E1EuDMCc6N3bfuB/wkOMR/Gta5fq812N8mD9WOosxvem8uN4IMbEgSbLmIWfcmiQaCBNr2 UQmSqzSJnCwMgB1dHRl9q6DWKpE9qic5EvgKw/ObbZX2UEjCJwwhlfZTTo0Eitho+V5EXfCSwFWq Aieo8q/OU+Et+E6jrD2Hu3XVj7UPyzM92jremARyLHHtZ7yR1o1tmpe6fsdGYQX6aZ7YnaXIHdny CSiGlmLV4aXIjrwUvTtEAp/Z8ce0NhoKNPQSCyWBZXrvWLXgtyJV2YruDFmxu8O1X/4mJL/ZXYjs yiTwMbr88YIVuWd5rc5CdBvn76Qzf5Zas2LbwyTwgTP0yB4drBX9M1nw0WlR3nvH+u3dm878LqIX GfjcmWWNPrbwwkHtlVbrIZabdbOi+b62832L138D2rNXcT5HkIc4BPd4l0P8j2BK9fqPWN9Anmmw fXSWnWVa4SVsex65g+27yN+RfyE9aHsW+RpyGDmGNJCfnmVx16UnEz/KyXNKNJ7SQZ4ztbR+8uVg cmoKYBsU9XLKhDdAuZjImOdTU6ZqwHmPbYQEKEpGUZSUnjJV/RJcRD2hmtOYJ25MZKGbs5swZBoW sC2rmjNZFHFTUw0D4EtQ4snJmaypjCdT+MpIsXyJ9DQ92wLoTI8zlWMZlHgmo5hzGbRUaO8SasZU tLienFINESA6Mvr68AXW+w3AXLpppKfGM/BUUD97e+wzs4Dd7sHyNDviN7Rz9vNBfohda2HvbHOM LezLWCtispoHXCyOAByrSzgB7Ngr2VHsgJxr8G9Gjt0sl2MF/VATXsrlMlqI/aV/dOyCfoj9DnxY wjo6ZhuAC3N/sQFXUH9CXNynumHxem29QqtdJVEMMd35RakIIK6JItOwskYNJVd3LvNo4Nf2NawA X6IZmO4Kh0OwVuTDji4h0MFtL4XD4RI67MeLohgOF7tKvKNDIo+GkqtF0fZwNW8bxKLbH2bgRdjv D+4WVODqI+s/rIcadfl6+ZH6gD/N2KRFO72tD2KrCQCzt/fIbbK1h6IyPwtwpd7403yL+AOgy1bF VS5jF3K41Vx2ALfA5bgFGEOGkH6uqf0AF5D5HNzIlzuQvc2sHr/RKobYG5D+hgTBb+9y+ot4Om/i fwBQSwMEFAAAAAgAgKwwJXNJW4VvBQAAQxIAAA0AAABleGNlcHRpb25zLnB5rVhtb9s2EP6uX3Fw UdjObCPtvgXIgHVLsGBrEzTpsC0IMlo621wkUiCpOAL643ekKJmS7MgFaiSxRN7rc8e7Y0aj0S8p 0xqWTGMCy4KnZs4F4EuMueFSwIajYirelIsouttwDfTDQOAWVshMoRC2G1S4LIGlKZgNgjZMJEzt k6ZnkVEs4faZyEuiVVysQS7/w9joGTCSpzBPWUzWbLnZQGzNQ70AcNrX/Bl1dFOajRRjHdpJSlMr K8OYnrnOyMxMkrxK+FwqjsKQ2BViSs58lNoQP9fGcsUyIcs3skgTeha0ViAYCVupngJDKpyinUNk 163MiFTx+KmEgkwFuYKr6wulpIKMlbBUyJ5mhIZx8GSkN4pllpHRjtxrtZosxhISrtkyJZnWYY/y zMKqKgmV92Q7xoVxlM5AuzX/C6QzjCT9RmGpoqUKkcitsIZZosqTXWAB/pYFxEyARW7tFS/LCG2k CJzKEJ5i5QWnHy3GhiJl/UCRYOLig3W0KoMY/HsyjmxIbbp4G3cuVSyli3mCKy4oNkyDKfKUJNgg MC4q9RglqCjySZBKNu+ET6B5lb7BphVK6CaEw0W9OjmZRvA1gh/mc7j1SerC5Dbc56v/diSlNphd vHATLP6O5VIS3xWlklJFHu5dZblUxkkMV6+7KxfXl92lzwWlXIbd5U+sv/azIZcpDL2N25IQe+mu 3pV5XwTFSFlEuht/SPlU5F1M3J/wxblFUW8pa3YIotZ6qFhREmRoeHyUjutnVKtUbveq+QeV/JU/ c93xoyG4TCWz6XsjuTCHLPqTpcUeJF3kO6sfkcqJd200GkVRdY6a9DpztJTJ8PhIiWseHyca09UM Tpha6+mZFwVgVxd2Dc7BfkUBI+Wz5wsY+AqENDu+3Y79KKTTJGA8blYxJY4UxaThmML5Obzby0ca d3T3pw/TQIzGYZZpaP6aYkvQNa7zwIuaudHFH2oM26exQdQz50TSUO6SfNLi8rS2Sgk6NQQtdQMU kh7kaqXR0IPBF/v1SQp0xJle0yuFcufAtwWu3ujD/VMPbrdZaQzhfkUGhex9WwYXK9nif/fQ2jeq bDPYj6OucZlVrxU2/qXCx794jKymlqSqtvalu+DYh1fyt5s2hMK0OT7imSspqIuYw1Gl4/aBKrzv XCtqrEGpNxtmQMZxoUAWRvMEwy6p3VleODE3imdM8dT1addsmGvogdzr26pxkw115V6MvkOGuA3q GC4hmwRsdggadHr3bQY53dr0GdMky4/t6LyhUYC2p+AkV6OTbcA0mRFLYRGnHsrjjR0SLGK1HmrM H77czTrCtAQ/tXAxdxMaFCJn8ZNt0YlENxH4WQcX60XHmI60Kn41xDOYOGzslFMBMX2dfYs08MVP DmNrmcsBmkukoOD60YGc2kqw5aiZHlmqZUdShszOEnZGoZSQNoloUqoTeeoRymnKMNrmVwuprlV2 IiErtpRZusjzlNsBJKTZpcGsHfhZL9St9Gl4g9y6Pz173ysfhyrHmzp4MeX7GR0Owv3xFs2lkrbR CTmZHmfoN/SstkcEv21iNoP39697Zwe81Q/0e0a/Y3gLk8OI9UrR4Q9dKXzpqc2Zhm2utrU6n/bo tzQdYe7rph7RUltVb9HCtK6U/qxMuhVzT5f0VewQKRW0L+66V7pszqXmL3QvSYoUm2IXigun0731 ud2hm+FpkLaehAcJg9F6kLaZeAcpd8PfIGlv7h/kaA/Zw+Sd0XiIvjUbTzrce+j70/ARTL0Z+wie 8A4xGNbmEjEJ2PbDfwxZ+240qL65Xg1SBrP/sQeA7o29SfY7zJhNgT9t1xHH5/6L0ZkSmsvAoXtA yNkdSvv1qksd/Q9QSwMEFAAAAAgAZTs/JpD/XKucAQAATgMAAAcAAABzaXRlLnB5hVJNa9wwEL0v 7H+YxgfbYEQL7SWQQ469hECOcRFae7wWq69IcoL/fUaSWbwtoRcbad6bN2/0quOhgiAjwiQVfayH OCPcBS2UuoPnNc7WwChD9PK0RGkNEY6H40FqZ30E+tGtup4JtwzxetTCh1moRBiUCAFeqKDw0Q+z fMdmY7PfGY2+vT8eAEacgHNpZOS8CaimDkQhcCfibITGAgRIVabkyQu/wgNYh6b5G9tB7U91+y+B BcRL83OrNHaaAsaupT7FBVuME8OlqR9k3d0yPYqRmF82Lb325WgHarztgylL/D2tTSsq1s8Y+WBH 3Kw74dHEDrQdi5vpbb8BOYGx8arBZhH4BdfmFgXgMS7ewJM1WK6cDR3I4C7nZHijvxban698EWkz lSb8j6GdbNbpMikZzS8sTYgUsqbMWFVVck5R8zhES885eashrIEJf35//Z5nouYkWp/v+16vMNph 0bSc0PcuB7Xvc27T8ccv5lZVl7VWsInlcBv8ALklLlVvM0kSLbvOlqOe4pjWwjl8I3XOtZCG8zoP znfYT1BLAwQUAAAACABoPj8mMGpxqG0PAACYMQAACgAAAGltcHV0aWwucHm1WluP28YVfq4B/4eJ FoGoRMvYbh8KtxvDhZ100TYObCMvm4VAkaMVY4pUOEOvhSD/vec6nKG0aydIXCBdkTPnfvnODM8e Pjgzg6+b2tfWGd+ZrW32phu8ua391pSD893O1Lt913uzs+W2aGu3c7Dt4QPc23bePjXF+66uZFXd 3sDT9nw91A38MLuuGhoLO4QI/F/42x3G58733fhmtZL9q9Vfzs7OzO32YPy2dqZunbdFZbqN+QmE M4NDhmG5W63Ms2cs26brYY81L+relr7rD5dE2/Yxz6H04eeu6N22aHB32RTOGd3w9OEDY2b/Kpw1 /AJJ93bfFCVyd75oq6JXC5jN0Ja+7lqXz5CWMZXdkOBF02TONpsFETQG/85XK7AqKCqbL2Ld4SU/ Xq1O7Oht04Ep0h38kFefpgQbmIw82Hbdu+P1gTiv5Z+yFlef/SH/iBL9x3z/+vKH529fmv+9fPvv Vy/ehDdovFhUsuDStMXOLs1N062Lxl1817Xwq+nK8cem73ZN7Tz9VIvP/gtLPIQ6eEzcBX9LkKq7 UJrKgt93dWsphMqu9faDx6jjKKSdvHZf9Lb1wVBho8osezMRdTHyEI8jAxaAkg5yJmK5NBBqqAo9 BFsUQ+P1JRPy3X5pfFE3U8fCixUTzlhIttqCt9Ub3GlAGzSQ2MdAWPuhb0+FZpaYXK09GjrWbGNm Rb7Oy5m5LZzZ9937urLVEpVoDQUWqjPjJVQ2utZU3W1L1JTMGoLullaCpOeNfW8bdRUvWXce65Pq TSGKlsiCTVKhkBTk7g61non5WU4RTbSfFbNgJChxQcVgJax841PdJpJymNjg397aiXU9lrqJXKJL 21UWxSsgsMp3xY0VybRkdTuNFidGOFNCXIyBXO2RAgo+pTJTmWcgysb2VPPRr1hZlQ7/i2SS6OwL eIY1tWinIghvrIuQTfUHNAGRXepKje2pRLcY9U2T8lbXNE3YTgnbOksJ00ElrlsQHYUkPkDUAtNj FYSbxo35/gDbW+Jp9nX5zgx74qOGYXIpnZB/Yo9MiC6E6j/YLURzHdyDhvIpoVsNCluRQtBde4t6 pYpA0qOlQeneGvthD+3LVhPVOuS0763D4lM0PTTFQx6CVuqAW4mk2XvobRmLv1iEME7KhVpAlt2V 2EkOkWUgg9Rj5dJUszSV1pJK8puJj30x7ixS2tl+KmVf1ODtNwfo+7uXfd/1SzOT5oQRfrCEKRq7 A0PYKuq4d5diZqMFWfjMXyQlf4blDcvSTKIaeMLv9+rBfJ5UCCGG1fpHta4+zG+sz+bagG2/Ws2X VHQXmqYoUFRblFjVWX5PuYo+h02YWt3QhygDjuJkzUYk8RZij+xCuA6CyTKvqRpjn6EE8yMFt+2G pmKURYlSY/9j/wfEs4eSELiKh1GzMWLewnLVJ6ihHTcSXtRyubn0nEngw7q1SueoDUO+2Ft1hvnW xo30KyW6+RmTSiTkJrjiZ9A0RKwrcA0+Abdcj3Kz00OtEt1qj64ymUKHoNH+3Q2L13o3NtijNJzE XIweDi4Xl14lgl7rUgCeWP3UmOBOXgdtuqrLABGDH/italQjD8TYeQ9mrbKEx9LM83mU5qN/Bu+g c6PFgzHMtqBIAgq+Loem6CexFNQHnhfm/PEUW6QBIj2D+aUNImqryvvY8q5eN4DC1epTJye/r57W 1/GyT7C7WD2YmiMFFUtWJzUutnwMXyNAxiUogWUpRL0HTJgsJDHabu26ZvB2kStmCZ4mR8f+vcMv W0tIf9TDKJ6cCU3bOHtq+WjQsIUe118+fnod+LGaYX/wzexzl3/uZuZzk03suyQmixPMw2ZcwA8l diboF4IswF5cu5Stoxl4X1R3xwJCsFWtTBgcAl7Lj/0ATRHadpauH+k4LLwHiOwBamqnAey3i2mp 5M2MVO+y1hmiBt/XwHAUqEVc3B+gaAf/a4RkAa0QSyVzt5F4XGITRVY/ZaI7BNdUTu3Rzj2bIMkN 5jaWgDg/UJo0MYAax6/4TnPkUgCiwfG/CeguGuCwSEXyCEIC0BSwtK2e8WIw5NH4E1WFpBzYD6Xd e/MfeyAsEtVx55Q5gDIc0sTU0PwBeFRjME51Co6XfXdMZHHVDM4ETXJog1pVEoondIAd/CTY6ewM 6JcdtXIGEWDUNbT6qt5AgQWBm8OzZzHXPAYyouT0vYN2SC9ZpatHUWPFCGfYuuQat63x6ATQHY89 zlKDf180Ax9JCeoPeF81EKs1ts2YzcJ8bZ4EsxHiFSiZD/sKaqqsu3pyHfU6+8GWg9c5v0qGcN4+ dyM01yiwper2+Bq7Vswt0lUHlhCHpAWgiNAjRItJ0oMRCu/745gJLKLIiF1K6TTOwQKoZRaeZA+K BUoEPIW9ZTlO4dqB0sF7HIyR5FiWCcmRm6T5OMCeflPbpnIZLuUeFJWSU21gF3UAJLcYl+9O1a4d L5vkkgiIyHkXMdRZgtXXWeK7Tn2DFCozM1+aEz09HlmmA1OwcpiYdKbQOtUejBvW55KOOvDR+DQz uAPHiShD5l/MCeDJfCfrlXw4R+nehVNOALEwxgKGxVCsi7VmyBmmEAIBU1RVjQNN0VC2OSPx6haU ZQykg3dJgvb45CPMyRckdxZUBrPRA86EhQ49LNTSXF0vQmKgzGCOk/TDjEnBgRVBKndcNPCsCNsL tsAgOw4oWTzB7OqbracogFFZIeZnYxNGgA5SfHZByuL8g2uhcVDqgTvhbRyw8PP+iMX1yXJgeUfU wsulEjyOWt760dAtixZXi/okG0avkI0DdpxA4hGb5oYwZ8fFhN7cP7fSIIRG+zEWM96Pg5W0Ap2s Pnog/En/iBL9R0+LzdtX5tUPL1+/vnzxMrxF1UP3/TRcMZt9Q6nSIthC0PU+agyaajcAwdRQuca0 DBRub8t6U1s+wKNHUl0gTnmihTeKJJEiWxdgDE6+TGtXIFARqATy1HgSQfcq8dAVE6Gh3BZ9uQ0C iYqJRAlaMhkGT9V5QEKLuOsJFg/Z+vOEDpWioWkO5z8PRYPPqriE5jDz146bebbvcCSuoQocJMyZ IUdraDOzvuv8TBtRRIwP9qgnje2fuDC1S9oRTjfUv1QjZOXFRTgF4D0k3s76LWSnnHIk4LQwT879 sG8Y+Bfmr/wr7P9CuEZYl0+JEewKb6SkxNdW6NsqVwpv0YZ17zwjH1H8yTlyFH7RdnwHyMHewMj7 CJc8XmrSsVsOGB63W+v5gDaIhkeQJB/PIvG5Mg81PpHIARRsRzAWIr9b/2TLMdI0fmqvZ54qjYCp Kg4nPKY5BlKLPDLmvYAQSwm2Laj1cqr8FYFDUKTu41NWE05gGXfRySqebwcB4B3oYI8UAz4qeegO FCV6c4GtqsT7v1F9UIZI8Pks/ACOBSFBAq6ZG8qt0gIH3FKUTqKGvKKWmOmwzaX+9QB5s7Na67WQ Gan4yYHnwwcfL5qf8o/udc/MG6If7jHP16C5Xpfh9emZXpB+Aysu5XGmf3AxnelPbEe8GlwxOCx/ btjjwWQVOKRXC5yZ3Xvb92B7x5CT+tiqbmu/Wkkxx93JVSo+gH6L//f76n98+YVU7pnZHj6IbnVX GO7ZKE9iF3qc6/3v4k/w1+RmGtsAtKjEbVAfIr99zyXgOS/8TR4UsEGVO9vUMIsov+S6+7fZ/dSp R70ZW+hMatbM1Ee3ibeWrkmCmhRf63MUbSQWGo3IGh3ETGd12Ix7xxMp2YIAT8SPT0juGNonY3tY 7g97G8bViwv+nUVXMmFj9miJmJrXjuLSu0fL8JzfnI3Nb3Lj6rpYb+0pdIRFt1Zcl/iQSmnpNfMM T5BmYiulEJlKHmWpWdBxsvbeo4zowWMocLOl+WUeWXv+VP78NY0qZRqA7ATA4U0AHouJCJ0eYrn7 +n/DZ67RPoGAKF3BNR1KeF1uoQ20oeENmGDrg4niBnOELqd01qPQq31ODZYJCb4r+KBQaKGtOTeA QbeHyEZyAkHKDhd+0H5Fbb0syi3NkngbR1dVTAhLwGYAyTUpRs1fj/bXMUv1TUDMJ3ck3X2qKY0u 05Ril8meu3zHUJvuFY3aVJCEQtAqSmPe+y1B8pNrTEZHuYy/1FkaRIvFUnwvQR57/RT0GcnHiFfF eK7IUV2sfJeI1h3kHBYoukyJiSNXxjlMJoE5VHXxU6PoytsXPWgxHUFOeFcN+Pu8q7tPQ45TrSQ7 3VmkoZx+GbUXDOvxu4RCrV0Gf8fXuykYCJ1FC4jm0+RjKx2LLnRH8hY3waspjd9QgfjQnFlcTJge nTKPTJNyOB71niWlha78HNYcLyNYqBo6QYhsfwLIeLmDqVPOScP9b6Xf1iVgY/xSUD4UZPcefYj3 SYDDnuKrWPSegADREtfDb/As/Dd6BuMzfuKEL1aryq4HPLXGbJyXcyy8826eroaF83x/mOMpS0zg jwM+LGYAH/t3N/BE73aTmzBeqbqNSGCPcYHW6pwWKSqn+NEAodAWJ+rCFzliJz5Iq7Cu4LTlYxwQ Dn+XUMvKAsE7fVqEOwqTfuEJZRbshl/zFIBB+LDD9koJO+Lky1BnsOA0OBuRfxG1dA5vc8NsGF/M gDKibCc3bXIV89z7vl7D6JZeyEgkduFaTvYClS7c1ITswYc5fczwU1e3GdhzCvbAU7qkdvA+m9YX o5PfhfkFT2PVc3PzdOTzaxAPz8Vg6WN9cJcoIyyYa4DrTW4SDRHzIyaPVF+/2h/CgSRWe4iG3T7w gKim6F6Mq8v7l2sSRN8UIgvBfZRKTOWjSJD64UX0YEoL0jELtLAjTTh8fUF/jO7YoCkBR52SF6zZ r+cxjt/keMyc/Y1QOQQPAdxdcVOXWQzOaVSgz4bzocWpJJv/83K+HLcv6LJLl6MSBPNBsuQ4V9SV T45zzI9sE+TZ5GXTOZuNZi3ls8DEiEKE/YMQkeeW1JlLNgvv0c9YMDCWtH8pkRPV0dHTWsPSSI/T MsrKHPb4MS2SJH31Js3OuwaBpmtvMnf1d72gI3lUs1Qa0EvlVMFQH3AOFpqJ69Hf8wX7KDWqrMc4 Gn9enT++pnuBH9t5Yu1APfrxJS1Lojj+kFqlH3fEOszx3Cn+9EauNAjcyzkV7Wfxjh1wX5RTawIe t1Gob/LbvvY2m//4iP93/EriO4ru0dJhsUZuNUCccCDF8btpBrfNogfO2ncw0ppHR+wmyXZXEtwR SenFOwURG4oPaFYeBF9hveYgEYCHd+P45xV/pULltodO1ys/xPt06oevhNkxfqkm5zofB1Of8u/h g/8DUEsDBAoAAAAAAA47PyYAAAAAAAAAAAAAAAAIAAAAc3VwcG9ydC9QSwMEFAAAAAgACTA/Jgv0 yMA4BgAARBAAABIAAABzdXBwb3J0L2Vhc3lnZW4ucHmlV1tv2zYUfm6A/IfTBIXkzBGSdg+F0QTI uq4IsHZFk2IbXENgJNohLEuaSNf1iv73fYcXXWK3GLYEiSTyXL9zI48PD45pIUvZCCNJ0FGh7o5o rgqJjcODwwO1qqvGkN7q9n0lGn0viva76ra0adaZ6X+qctF+LqSp6m633qZZtaqtLr9k1Eqy1vTN 1evrl3RB0bs/f/0penQMYnNflQTzGtFsQfHqj1vsp2ku79aLNCVR5hRlEVUNRVXEQnI5J6XTWmRL sZBxPpocHhCpOQxOamHuE/lZaaPjnH6g6OPHNFWlMmma1NvIkRI10qybks7/LWP2nzmrvZz+/Sy4 w/hoI1Z1zNJKsZKOy9MVVbmIoQokpqOYPp+NggB9X62LPOD+QArsDAvT02eTGV0gAAzH0LJAw4sm hdOIwx7DEDRGhAPC/gXqvcTTyek5G2ltsFSXTvR+zZa85/fbqpSthxkyQTVxPqZ6uRiTyH3ga+Si oaMb7JfIysnRmHJmIprDRE55UiUHqkCIrIQQEVYLs33QIjxcgfBeL7ZKMxe/Bka7XVamn4bDfaKs Ko0q17LHAbt7BKXcYAH6+b9TzYYkUSCRhZb76HcpAzZsw9gTOoQCRWeNW1lVOQM+JvnZQGRwVdeF MliKWceoRQIr08mzGT0e5s2u2GNecc2C5uui2J7+tRaFmiuZs8o1DGe1iU2He6UBIJ6SPIhBCFeP At/fwig0B501qjZjpix7CnqMFPLWmuuds2keSrGzud1l4G3GIaO5o9VWommEKpBHfXzBknKoLfzn bnUQnYFMxMZ/7+U/68DSSyi1UMgAz73QJIpGinxLdxL+av6HBBaUq/lcNhLJjlgHEchCOWmxPCrk QhRH4MzEWnPj14jnqa5lhiBkQclKsPAgwv1Un2TTqJw9F25ytPRJiyxSKkG3j9v0QTmORtMzmxp5 C4cvyd+v3r+9fvt6QvkaeZXxJFKu2LlIH2DkeYh+uX5/c8sEUDb1RDOo2CEk+vnDO1/ubms3HblD WUzrRmYyl2Umrae+hfk3R12hgbpy2HUzisbk/uDu01m/MkI3reyM+tgWbNzbw/ziORY0PA4co2+z DDl4MeraSx8a7mBjcqUP0nEv20Lv3IhiGSNp9JiqtanXIMLARmlpJ/IYDXcFMDhQ2uptZIHa4xVw MQkaggFYF/SFvtpvGJ5zOJlg0uLR9Uxxp7tOS6C98MeGpKg2krtw2AKbk54gUOlSbvt830smqJaZ qZotaKpPKpf5IBse9E+nY5ozYG6w7BA9VNVHYb8mh15dCOSVrlYytC48gdDK9S8IrWyN3/72khlM xaP1i9ManfyMhI9oEkAAADoejcPu7fWbV7zLwzUROuNnbD+KKhNF92nfRqPA+TUYx8MBLaVonVBS 2/CJ2gJ5V4TmYwON1NoT5YFtHq/eRObScOPG6fTnEM5un1lsgNcCKIQh0Ug7QrfSBPI8qOy3l/2Z PbItEWWK9rbqmdRkdjpe7DsQdfPMk/Wnua8HW/aR54JTZ7uZGL20u3zUAIEX1SNqT79JUO5pWmIM CD7mjjqmB/XMbg9p3aAO7z0cvpHs3tQbDJh6aCmCFZdS5ghDhSHQYj+KOu5cFgOThnkuRXZPRjSA KhyvfH675vK/o+hGuKd5UJzRVW5nlB/87NhgkOypZ8+yQzrng08tS39sipq7yMM5TzB1l/Fzjv8j P6ZXYoGByL2xPeY6YlTzIHYOhMTIokAZ78bK728aOB3PEx71KFsHsT/3QmRo3Rn2jUxxP4rx5/A8 ca385GTQxJ3c4FNLHG28WwO9/iLGB98z9xt5E1xz+s7EcGR1pVlV31de9nfIJF/jFgBRgd1zeXqL 7o+7VrmDXcKxjaMLhYBBzahPlxWVlnF77VkJVfq0gXkwFlnJdrkbaeIeMS65CTY+Tc8nMwBymkXh SlIAKWYZ0Qs6D7cClzQfbq5ev5rQE03T02xGL5wBl/QCqJxf0tS+PL2kJEloFtETarWEo4oXRHSa TdqOiMlui0bbKwTOJFrj3utrj0XgJmnic4/XctP14uVm2jYmm2jsyBinAnu9gfdMxD19G3dJg0Mz xqV0To4hwwqG6jTl9MD92h2SGchwSHagHh78A1BLAwQUAAAACABiOz8m3x7QkGUAAACMAAAAEQAA AHN1cHBvcnQvcmVnZW4uYmF0c3B19vBX8Hdz4+Xi5dLTiymoLMnIz1NITSyuTE/N0yuoVNBNVgCL G5oCeTkKSulWMcVFyVCFhnqmMTmZSTEFOYkluuWZeUrY5SHCuZUKKfnJpbmpeSXFUHklBTsFkD0l FSW8XABQSwECFAAUAAAACADVBp8j2pZOWIcEAAAAEAAACgAAAAAAAAAAACAA/4EAAAAAcHl0aG9u LmV4ZVBLAQIUABQAAAAIAICsMCVzSVuFbwUAAEMSAAANAAAAAAAAAAEAIQAkga8EAABleGNlcHRp b25zLnB5UEsBAhQAFAAAAAgAZTs/JpD/XKucAQAATgMAAAcAAAAAAAAAAQAgALaBSQoAAHNpdGUu cHlQSwECFAAUAAAACABoPj8mMGpxqG0PAACYMQAACgAAAAAAAAABACAAtoEKDAAAaW1wdXRpbC5w eVBLAQIUAAoAAAAAAA47PyYAAAAAAAAAAAAAAAAIAAAAAAAAAAAAEAD/QZ8bAABzdXBwb3J0L1BL AQIUABQAAAAIAAkwPyYL9MjAOAYAAEQQAAASAAAAAAAAAAEAIAC2gcUbAABzdXBwb3J0L2Vhc3ln ZW4ucHlQSwECFAAUAAAACABiOz8m3x7QkGUAAACMAAAAEQAAAAAAAAABACAA/4EtIgAAc3VwcG9y dC9yZWdlbi5iYXRQSwUGAAAAAAcABwCVAQAAwSIAAAAA ------=_NextPart_000_0001_01BE4CF0.1C1CBCE0-- From gstein@lyra.org Sun Jan 31 18:47:27 1999 From: gstein@lyra.org (Greg Stein) Date: Sun, 31 Jan 1999 10:47:27 -0800 Subject: [Distutils] Re: new import mechanism and a "small" distribution References: Message-ID: <36B4A53F.3ECA6F0B@lyra.org> Just van Rossum wrote: > > At 8:00 AM -0800 1/31/99, Greg Stein wrote: > >... > >Okay... enough background and rambling. If you're interested, go look at the > >four messages in the thread titled "Freeze and new import architecture" in > >the distutils-sig archive at: > >http://www.python.org/pipermail/distutils-sig/1998-December/thread.html > > I'm not on that sig so I missed your post originally. I agree with most you > say here: > http://www.python.org/pipermail/distutils-sig/1998-December/000077.html > Especially that entries in sys.path should be loader instances (or > directory paths). > > Some questions: > - what is the interface of a loader By "loader", I will presume that you mean an instance of an Importer subclass that is defining get_code(). Here is the method as defined by imputil.Importer: def get_code(self, parent, modname, fqname): """Find and retrieve the code for the given module. parent specifies a parent module to define a context for importing. It may be None, indicating no particular context for the search. modname specifies a single module (not dotted) within the parent. fqname specifies the fully-qualified module name. This is a (potentially) dotted name from the "root" of the module namespace down to the modname. If there is no parent, then modname==fqname. This method should return None, a 2-tuple, or a 3-tuple. * If the module was not found, then None should be returned. * The first item of the 2- or 3-tuple should be the integer 0 or 1, specifying whether the module that was found is a package or not. * The second item is the code object for the module (it will be executed within the new module's namespace). * If present, the third item is a dictionary of name/value pairs that will be inserted into new module before the code object is executed. This provided in case the module's code expects certain values (such as where the module was found). """ raise RuntimeError, "get_code not implemented" That method is the sole interface used by Importer subclasses. To define a custom import mechanism, you would just derive from imputil.Importer and override that one method. I'm not sure if that answers your question, however. Please let me know if something is unclear so that I can correct the docstring. Oh, geez. And I totally spaced on one feature of imputil.py. There is a way to define an import mechanism for very simple uses. I created a subclass named FuncImporter that delegates the get_code() method to a user-supplied function. This allows a user to do something like this: import imputil def get_code(parent, modname, fqname): ...do something... imputil.install_with(get_code) The install_with() utility simply creates a FuncImporter with the specified function and then installs the importer. No need to mess with subclasses. > - how are packages identified If get_code() determines that the requested module is a package, then it should return the integer 1 along with the code object for that package's module. In the standard package system, the code object is loaded from __init__.py. An example: let's say that get_code() is called with (None, "mypkg", "mypkg") for its arguments. get_code() finds "mypkg" in whatever path it is configured for, and then determines that it represents a directory. It looks in the directory for __init__.py or .pyc. If it finds it, then mypkg is a real package. It loads code from __init__.py and returns (1, code). The Importer superclass will create a new module for "mypkg" and execute the code within it, and then label it as a package (for future reference). Internally, packages are labelled with a module-level name: __ispkg__. That is set to 0 or 1 accordingly. The Importer that actually imports a module places itself into the module with "__importer__ = self". This latter variable is to allow Importers to only work on modules that they imported themselves, and helps with identifying the context of an importer (whether an import is being performed by a module within a package, or not). > - can it be in Python 1.6 or sooner? I imagine that it could be in Python 1.6, but I doubt that it would go into Python 1.5.2, as it has not had enough exposure yet. Guido's call on all counts, though :-) > PS: I was just wondering, why doesn't reload() use the standard import hook? The standard import hook returns a *new* module object. reload() must repopulate the module object. I'm not sure that I answered your questions precisely, as they were rather non-specific. If I haven't answered well enough, then please ask again and I'll try again :-). Cheers, -g -- Greg Stein, http://www.lyra.org/ From just@letterror.com Sun Jan 31 20:41:10 1999 From: just@letterror.com (Just van Rossum) Date: Sun, 31 Jan 1999 21:41:10 +0100 Subject: [Distutils] Re: new import mechanism and a "small" distribution In-Reply-To: <36B4A53F.3ECA6F0B@lyra.org> References: Message-ID: At 10:47 AM -0800 1/31/99, Greg Stein wrote: >By "loader", I will presume that you mean an instance of an Importer >subclass that is defining get_code(). Here is the method as defined by >imputil.Importer: > > def get_code(self, parent, modname, fqname): I see in your code that you build a chain of import hooks that basically work like a list of loaders. Why don't you use sys.path for this? (Current implementation details, probably?) I'd suggest a loader object should be a callable, ie. get_code() should be called __call__()... Changing your pseudo code from the distutils-sig to: for pathentry in sys.path: if type(pathentry) == StringType: module = old_import(pathentry, modname) else: module = pathentry(modname) # <-- if module: return module else: raise ImportError, modname + " not found." It's the only public method. Makes more sense to me. It also seems that it would make it easier to implement loaders (and the above loop) in C. >That method is the sole interface used by Importer subclasses. To define >a custom import mechanism, you would just derive from imputil.Importer >and override that one method. > >I'm not sure if that answers your question, however. Please let me know >if something is unclear so that I can correct the docstring. The only thing I find unclear is when the parent argument should be used or not. Is it only for importing submodules? >Oh, geez. And I totally spaced on one feature of imputil.py. There is a >way to define an import mechanism for very simple uses. I created a >subclass named FuncImporter that delegates the get_code() method to a >user-supplied function. This allows a user to do something like this: > >import imputil > >def get_code(parent, modname, fqname): > ...do something... > >imputil.install_with(get_code) Cool! Another reason to allow any callable object in sys.path... >The install_with() utility simply creates a FuncImporter with the >specified function and then installs the importer. No need to mess with >subclasses. > >> - how are packages identified > >If get_code() determines that the requested module is a package, then it >should return the integer 1 along with the code object for that >package's module. In the standard package system, the code object is >loaded from __init__.py. > >An example: let's say that get_code() is called with (None, "mypkg", >"mypkg") for its arguments. get_code() finds "mypkg" in whatever path it >is configured for, and then determines that it represents a directory. >It looks in the directory for __init__.py or .pyc. If it finds it, then >mypkg is a real package. It loads code from __init__.py and returns (1, >code). The Importer superclass will create a new module for "mypkg" and >execute the code within it, and then label it as a package (for future >reference). > >Internally, packages are labelled with a module-level name: __ispkg__. >That is set to 0 or 1 accordingly. Right now a package is labeled with a __path__ variable. If it's there, it's a package. Is it neccesary to define something different? >The Importer that actually imports a >module places itself into the module with "__importer__ = self". This >latter variable is to allow Importers to only work on modules that they >imported themselves, and helps with identifying the context of an >importer (whether an import is being performed by a module within a >package, or not). Ok, so __importer__ is there instead of __path__. That makes sense. The current __path__ variable makes it look like it can have its own sys.path-like thingy, but that's not quite what it is (?). It is even abused: if it is a string, it is supposed to be the (sub)package's full name and means that any submodule is to be located like a normal module, but with the full name (eg. mypkg.sub1.foo). This is crucial for freezing and handy in other cases (it allows submodules to be builtin!). While this is cool and handy, it smells really funny. As a case study, could you show how this stuff should work with the new import mechanism? Just