From mscott at goldenspud.com Sat Oct 1 02:28:45 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Fri, 30 Sep 2005 19:28:45 -0500 Subject: [Distutils] script names that are same as top-level package do not work on windows Message-ID: Ran into a problem with setuptools 0.6a5 on Windows where a console_script is named the same as a top-level package. This problem doesn't occur on Linux, presumably because the installed script does not have a .py extension to confuse the import of a same-named package. Excerpt from setup.py: entry_points = { 'console_scripts': ['schevo = schevo.script.main:start'], } Traceback on Windows when running the installed schevo.exe: Traceback (most recent call last): File "c:\python24\scripts\schevo.py", line 7, in ? sys.exit( File "c:\python24\lib\site-packages\setuptools- 0.6a5-py2.4.egg\pkg_resources.p y", line 236, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "c:\python24\lib\site-packages\setuptools- 0.6a5-py2.4.egg\pkg_resources.p y", line 1892, in load_entry_point return ep.load() File "c:\python24\lib\site-packages\setuptools- 0.6a5-py2.4.egg\pkg_resources.p y", line 1625, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File "c:\python24\scripts\schevo.py", line 7, in ? sys.exit( File "c:\python24\lib\site-packages\setuptools- 0.6a5-py2.4.egg\pkg_resources.p y", line 236, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "c:\python24\lib\site-packages\setuptools- 0.6a5-py2.4.egg\pkg_resources.p y", line 1892, in load_entry_point return ep.load() File "c:\python24\lib\site-packages\setuptools- 0.6a5-py2.4.egg\pkg_resources.p y", line 1625, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) ImportError: No module named script.main Thanks, -- Matthew R. Scott -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20050930/ecf22da1/attachment.html From pje at telecommunity.com Sat Oct 1 02:59:41 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 30 Sep 2005 20:59:41 -0400 Subject: [Distutils] script names that are same as top-level package do not work on windows In-Reply-To: Message-ID: <5.1.1.6.0.20050930205339.02c6ee50@mail.telecommunity.com> At 07:28 PM 9/30/2005 -0500, Matthew Scott wrote: >Ran into a problem with setuptools 0.6a5 on Windows where a console_script >is named the same as a top-level package. Hm. Are you installing scripts and modules to the same directory? Usually those are two different directories on Windows (i.e., site-packages and Scripts) Come to think of it, that wouldn't actually help this problem. I'll have to use some other extension for the files, I guess. It's just that I was using .py and .pyw for console and GUI so that people used to running scripts the "old" way wouldn't be surprised. I guess I could just drop the extensions altogether, but that's going to leave the old files behind. (We really need an uninstaller for scripts.) Another possibility would be to add '-script.py' so that it's e.g. 'schevo.exe' and 'schevo-script.py'. That way, if somebody needed it to be a .py file, it'd be there, but it's not an importable name, so it couldn't clash with any modules or packages. Thoughts, anyone? From mscott at goldenspud.com Sat Oct 1 16:40:11 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Sat, 1 Oct 2005 09:40:11 -0500 Subject: [Distutils] including ez_setup in sdist output Message-ID: The following answers the question I was about to post. Consequently, I am going to attempt to create a patch for setuptools that gives it svk support since I rather dislike using svn itself (see http://svk.elixus.org/) :) """ At 03:15 PM 9/20/2005 -0400, Greg Steffensen wrote: >They are included when I do "setup.py bdist_egg", but for now I'd rather >distribute using traditional tarballs. Am I doing something wrong? > >Also, the README is magically included in the sdist tarball no matter what >I do, but I'd like to include COPYING, AUTHORS, etc. Is there a "correct" >way to do this? Thanks, By default, setuptools will include any file under Subversion or CVS revision control in the source distribution. If you're not using one of those revision control tools, you have to use the standard distutils MANIFEST.in rules file to specify what files to include in the source distribution. See here for more details: http://www.python.org/doc/2.4.1/dist/manifest.html """ -- Matthew R. Scott From mscott at goldenspud.com Sat Oct 1 17:44:08 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Sat, 1 Oct 2005 10:44:08 -0500 Subject: [Distutils] 'sdist' command extension that supports svk Message-ID: It's not pretty, and it requires the 'py' lib, but it works for those using SVK instead of Subversion :) from setuptools.command.sdist import sdist as _sdist class sdist(_sdist): def add_defaults(self): _sdist.add_defaults(self) try: import py except ImportError: return try: svk = py.path.local.sysfind('svk').sysexec except py.error.ENOENT: return try: info = svk('info') except py.process.cmdexec.Error: return depot = None for line in info.splitlines(): if line.startswith('Depot Path'): depot = line.split(': ')[-1] break if not depot: return start = len(depot) + 1 paths = [path[start:] for path in svk('ls', '-fR').splitlines()] self.filelist.extend(paths) -- Matthew R. Scott From pje at telecommunity.com Sat Oct 1 18:07:15 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 01 Oct 2005 12:07:15 -0400 Subject: [Distutils] 'sdist' command extension that supports svk In-Reply-To: Message-ID: <5.1.1.6.0.20051001120549.01fb7eb0@mail.telecommunity.com> At 10:44 AM 10/1/2005 -0500, Matthew Scott wrote: >It's not pretty, and it requires the 'py' lib, but it works for those >using SVK instead of Subversion :) Any possibility you could implement this as an additional file scanner like the SVN/CVS ones in setuptools' sdist module now? Or does svk not keep per-directory lists of tracked files? From mscott at goldenspud.com Sat Oct 1 18:05:51 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Sat, 1 Oct 2005 11:05:51 -0500 Subject: [Distutils] script names that are same as top-level package do not work on windows In-Reply-To: <5.1.1.6.0.20050930205339.02c6ee50@mail.telecommunity.com> References: <5.1.1.6.0.20050930205339.02c6ee50@mail.telecommunity.com> Message-ID: On 9/30/05, Phillip J. Eby wrote: > At 07:28 PM 9/30/2005 -0500, Matthew Scott wrote: > >Ran into a problem with setuptools 0.6a5 on Windows where a console_script > >is named the same as a top-level package. No "rush order" to fix this by the way (at least not from me :) -- we opted to use a differently-named script. -- Matthew R. Scott From mscott at goldenspud.com Sat Oct 1 18:13:00 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Sat, 1 Oct 2005 11:13:00 -0500 Subject: [Distutils] 'sdist' command extension that supports svk In-Reply-To: <5.1.1.6.0.20051001120549.01fb7eb0@mail.telecommunity.com> References: <5.1.1.6.0.20051001120549.01fb7eb0@mail.telecommunity.com> Message-ID: On 10/1/05, Phillip J. Eby wrote: > At 10:44 AM 10/1/2005 -0500, Matthew Scott wrote: > >It's not pretty, and it requires the 'py' lib, but it works for those > >using SVK instead of Subversion :) > > Any possibility you could implement this as an additional file scanner like > the SVN/CVS ones in setuptools' sdist module now? Or does svk not keep > per-directory lists of tracked files? > > It keeps that information in a cache separate from the checkout path, and appears to keep it in a manner that makes it efficient for svk to compare checkout paths and repository state, but perhaps not easily grokkable by something that isn't svk. Essentially, the checkout path looks like you did an 'export' to it; the difference is, svk keeps an association between checkout paths and its cache. -- Matthew R. Scott From ronaldoussoren at mac.com Sun Oct 2 12:21:54 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 2 Oct 2005 12:21:54 +0200 Subject: [Distutils] linking modules to a shared library In-Reply-To: References: Message-ID: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> On 30-sep-2005, at 19:57, Stephen Langer wrote: > Hi -- > > I have a problem that is similar to one discussed in a thread from a > year ago, > http://mail.python.org/pipermail/distutils-sig/2004-September/ > 004160.html, but that thread doesn't quite have a resolution in the > archives. > > I'd like to use distutils to build a bunch of python modules that all > link to the same shared library. The shared library is all C++ > code. On Linux, without distutils, I can build the libraries like > this: > > g++ -o libBase.so -shared base.C # builds the common shared > library > g++ -o _mymodule.so -shared mymodule.C -lBase # builds a python > module > # linking to the > shared lib. > g++ -o _othermodule.so -shared othermodule.C -lBase # and so on > > _mymodule.so and _othermodule.so can be imported by Python, and share > a single copy of the code in libBase.so. > > On the Mac I can get the same effect with this: > > g++ -dynamiclib base.C -o libBase.dylib > g++ -o _mymodule.so mymodule.C -lBase -bundle -undefined > dynamic_lookup -flat_namespace > g++ -o _othermodule.so othermodule.C -lBase -bundle -undefined > dynamic_lookup -flat_namespace Don't use -flat_namespace. Search the pythonmac SIG archives if you want a more elaborate rant :-). > > Is it possible to create libBase portably with distutils? It's > possible to do it on Linux by subclassing build_ext.build_ext and > explicitly using self.compiler.compile() and > self.compiler.link_shared_lib() to build the shared library before > calling build_ext.build_ext.build_extensions(). But the same thing > on Mac OS X only creates libBase.so, whereas I need it to create > libBase.dylib. I don't know if this is possible, although I'd guess it is not. > > If it matters, I'm using OS X 10.4.2 and gcc 3.3, with Python 2.3.5. > > Many thanks, > Steve > > -- > -- EMail: stephen.langer at nist.gov Phone: (301) > 975-5423 -- > -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) > 990-4127 -- > -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md > 20899-8910 -- > > -- "I don't think this will work. That's why it's > science." -- > -- Naomi Langer, 17 Feb > 2003 -- > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2105 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051002/400d66d8/smime.bin From mscott at goldenspud.com Mon Oct 3 00:11:02 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Sun, 2 Oct 2005 17:11:02 -0500 Subject: [Distutils] specifying --with-featurename with ez_setup.py / easy_install Message-ID: When building/installing a package using a setuptools-based setup.py, you can do things like this to enable packages for optional features: python --with-featurename setup.py install This is with 'featurename' specified in the setup() call thusly: setup( ..., features={'featurename': Feature(...)}, ..., ) I can't figure out how to use easy_install / ez_setup.py to do this though. This doesn't work: easy_install -f http://example.com/dist PackageName --with-featurename Nor does this: easy_install --with-featurename -f http://example.com/dist PackageName Is this possible with easy_install? -- Matthew R. Scott From pje at telecommunity.com Mon Oct 3 03:07:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 02 Oct 2005 21:07:38 -0400 Subject: [Distutils] specifying --with-featurename with ez_setup.py / easy_install In-Reply-To: Message-ID: <5.1.1.6.0.20051002210402.01f9ff18@mail.telecommunity.com> At 05:11 PM 10/2/2005 -0500, Matthew Scott wrote: >When building/installing a package using a setuptools-based setup.py, >you can do things like this to enable packages for optional features: > > python --with-featurename setup.py install Actually, it's "python setup.py --with-featurename install" >I can't figure out how to use easy_install / ez_setup.py to do this >though. This doesn't work: > > easy_install -f http://example.com/dist PackageName --with-featurename > >Nor does this: > > easy_install --with-featurename -f http://example.com/dist PackageName > >Is this possible with easy_install? Yes and no. What you want is the --editable option, which gets you a source version of the package. Then you use "python setup.py --with-featurename install". In other words, if you need to do a custom build of a package, you can only use easy_install to find/fetch/unpack it. You also have the option of using "--with-featurename bdist_egg" to build an egg, then using easy_install to install the egg. From mscott at goldenspud.com Mon Oct 3 03:19:11 2005 From: mscott at goldenspud.com (Matthew Scott) Date: Sun, 2 Oct 2005 20:19:11 -0500 Subject: [Distutils] specifying --with-featurename with ez_setup.py / easy_install In-Reply-To: <5.1.1.6.0.20051002210402.01f9ff18@mail.telecommunity.com> References: <5.1.1.6.0.20051002210402.01f9ff18@mail.telecommunity.com> Message-ID: On 10/2/05, Phillip J. Eby wrote: > At 05:11 PM 10/2/2005 -0500, Matthew Scott wrote: > >When building/installing a package using a setuptools-based setup.py, > >you can do things like this to enable packages for optional features: > > > > python --with-featurename setup.py install > > Actually, it's "python setup.py --with-featurename install" D'oh! You are entirely right. That's what I get for re-typing it and not copy/pasting it in :) > > easy_install --with-featurename -f http://example.com/dist PackageName > > > >Is this possible with easy_install? > > Yes and no. What you want is the --editable option, which gets you a > source version of the package. Then you use "python setup.py > --with-featurename install". > > In other words, if you need to do a custom build of a package, you can only > use easy_install to find/fetch/unpack it. You also have the option of > using "--with-featurename bdist_egg" to build an egg, then using > easy_install to install the egg. It sounds like the solution, then, would be to not use 'Feature', to include all Python packages when installing, and use extras_require in the call to setup()? I'll try that and see what I can get it to do. It does beg the question though, what is the intended use case of 'Feature' if features cannot be selected when installing using easy_install? -- Matthew R. Scott From pje at telecommunity.com Mon Oct 3 03:32:24 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 02 Oct 2005 21:32:24 -0400 Subject: [Distutils] specifying --with-featurename with ez_setup.py / easy_install In-Reply-To: References: <5.1.1.6.0.20051002210402.01f9ff18@mail.telecommunity.com> <5.1.1.6.0.20051002210402.01f9ff18@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051002212604.01f9fd80@mail.telecommunity.com> At 08:19 PM 10/2/2005 -0500, Matthew Scott wrote: >It does beg the question though, what is the intended use case of >'Feature' if features cannot be selected when installing using >easy_install? "Features" predated even the whisper of a concept of Python eggs by at least 8-10 months, and the idea of having easy_install by about a year and a half. So, they weren't designed together. Features were originally conceived as a way of trimming down mega-distributions like PEAK and especially Zope 3. You'll notice that setuptools' documentation doesn't mention them at all, though, because eggs are more useful. Currently, I'm only using "features" for a few of my packages, mostly to control building of the associated optional C code. However, that feature could be handled using an "optional" extensions ala mxSetup. So, I'd probably say that the "features" facility is kind of deprecated, especially since it has a few quirks that don't work so well. For example, if you build without the feature, and then run "install" or something, the package doesn't "stay configured" for the options you gave it previously. So, it doesn't really work all that well and will probably die out of setuptools as soon as I have a chance to rework my packages that use it. Certainly, I'd like to know if anyone else has found productive uses for it. If there are any, I'd consider keeping it in, but it seems to me like what's really needed is a "configure" command and a way to write the configuration to a file that then is used by subsequent runs to ensure consistency. From stephen.langer at nist.gov Mon Oct 3 16:56:38 2005 From: stephen.langer at nist.gov (Stephen Langer) Date: Mon, 3 Oct 2005 10:56:38 -0400 Subject: [Distutils] linking modules to a shared library In-Reply-To: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> References: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> Message-ID: On Oct 2, 2005, at 6:21 AM, Ronald Oussoren wrote: > > On 30-sep-2005, at 19:57, Stephen Langer wrote: > >> g++ -o _othermodule.so othermodule.C -lBase -bundle -undefined >> dynamic_lookup -flat_namespace >> > > Don't use -flat_namespace. Search the pythonmac SIG archives if you > want a more elaborate rant :-). Ok, I won't. I was just guessing based on what I'd seen elsewhere. >> >> Is it possible to create libBase portably with distutils? It's >> possible to do it on Linux by subclassing build_ext.build_ext and >> explicitly using self.compiler.compile() and >> self.compiler.link_shared_lib() to build the shared library before >> calling build_ext.build_ext.build_extensions(). But the same thing >> on Mac OS X only creates libBase.so, whereas I need it to create >> libBase.dylib. >> > > I don't know if this is possible, although I'd guess it is not. That's too bad. Is there a reason for it? I'd volunteer to work on modifying distutils so that it can build a .dylib, but I am not an expert on library formats. I don't really know the difference between a .so and a .dylib, except that one of them works and the other doesn't. Can someone point me in the direction of a good reference on the topic? Thanks. -- Steve -- -- EMail: stephen.langer at nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 -- From ronaldoussoren at mac.com Mon Oct 3 19:17:17 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 3 Oct 2005 19:17:17 +0200 Subject: [Distutils] linking modules to a shared library In-Reply-To: References: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> Message-ID: On 3-okt-2005, at 16:56, Stephen Langer wrote: >>> >> >> I don't know if this is possible, although I'd guess it is not. >> > > That's too bad. Is there a reason for it? I'd volunteer to work on > modifying distutils so that it can build a .dylib, but I am not an > expert on library formats. I don't really know the difference > between a .so and a .dylib, except that one of them works and the > other doesn't. Can someone point me in the direction of a good > reference on the topic? I'd guess that nobody has needed this functionality badly enough to volunteer extending distutils :-). The main difference between a .so and a .dylib is that the former is a bundle and the latter is a dylib. Dynamic libraries and loadable objects are two different beasts on OS X. See the manpages for ld(1) and Mach- O(5) for more information. W.r.t. extending distutils: extending the 'clib' might be the best way, although I must say that I've never used that command. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2105 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051003/1d2a9696/smime.bin From bob at redivi.com Mon Oct 3 21:15:24 2005 From: bob at redivi.com (Bob Ippolito) Date: Mon, 3 Oct 2005 15:15:24 -0400 Subject: [Distutils] linking modules to a shared library In-Reply-To: References: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> Message-ID: <8E8FCE54-B9C3-4382-9E26-F4DBC83DED2D@redivi.com> On Oct 3, 2005, at 10:56 AM, Stephen Langer wrote: >>> Is it possible to create libBase portably with distutils? It's >>> possible to do it on Linux by subclassing build_ext.build_ext and >>> explicitly using self.compiler.compile() and >>> self.compiler.link_shared_lib() to build the shared library before >>> calling build_ext.build_ext.build_extensions(). But the same thing >>> on Mac OS X only creates libBase.so, whereas I need it to create >>> libBase.dylib. >>> >>> >> >> I don't know if this is possible, although I'd guess it is not. >> > > That's too bad. Is there a reason for it? I'd volunteer to work on > modifying distutils so that it can build a .dylib, but I am not an > expert on library formats. I don't really know the difference > between a .so and a .dylib, except that one of them works and the > other doesn't. Can someone point me in the direction of a good > reference on the topic? The most portable way is to not build a libBase at all. Build a Python extension that has all the functionality from libBase available in it as a data structure with function pointers, and get a reference to that from everything that depends on libBase functionality. Take a look at what Numeric's C API does, for example. -bob From ianb at colorstudy.com Tue Oct 4 18:08:01 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Oct 2005 11:08:01 -0500 Subject: [Distutils] easy_install, find-links, etc Message-ID: <4342A8E1.90405@colorstudy.com> The whole find-links thing is getting confusing -- right now each system has to give its own set of find-links, and they aren't well inherited. So, for instance, if Paste has some custom packages on an index page, you can't just require Paste -- you have to require Paste plus put whatever find-links settings are needed. What would be mightily useful is if you could at least advise -- from within a package -- on places to find some of the requirements for that package. E.g., .egg-info/find_links.txt. I think it is inevitable that packages will require custom builds, and we can't always push those changes into PyPI or track those changes in aggregate package indexes. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Oct 4 18:18:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Oct 2005 12:18:19 -0400 Subject: [Distutils] easy_install, find-links, etc In-Reply-To: <4342A8E1.90405@colorstudy.com> Message-ID: <5.1.1.6.0.20051004121403.01f5e9a8@mail.telecommunity.com> At 11:08 AM 10/4/2005 -0500, Ian Bicking wrote: >The whole find-links thing is getting confusing -- right now each system >has to give its own set of find-links, and they aren't well inherited. >So, for instance, if Paste has some custom packages on an index page, >you can't just require Paste -- you have to require Paste plus put >whatever find-links settings are needed. > >What would be mightily useful is if you could at least advise -- from >within a package -- on places to find some of the requirements for that >package. E.g., .egg-info/find_links.txt. I think it is inevitable that >packages will require custom builds, and we can't always push those >changes into PyPI or track those changes in aggregate package indexes. Patches to implement this gratefully accepted. :) Please include: * argument handler entry point to allow them to be specified as a setup() keyword (I suggest "dependency_links") * documentation in setuptools.txt for the keyword * a metadata writer entry point to write the file (you might be able to reuse one of the existing entry points * code in easy_install's process_distribution() function to read and process the metadata before doing dependency processing Thanks. :) From stephen.langer at nist.gov Tue Oct 4 19:46:41 2005 From: stephen.langer at nist.gov (Stephen Langer) Date: Tue, 4 Oct 2005 13:46:41 -0400 Subject: [Distutils] linking modules to a shared library In-Reply-To: <8E8FCE54-B9C3-4382-9E26-F4DBC83DED2D@redivi.com> References: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> <8E8FCE54-B9C3-4382-9E26-F4DBC83DED2D@redivi.com> Message-ID: On Oct 3, 2005, at 3:15 PM, Bob Ippolito wrote: > > On Oct 3, 2005, at 10:56 AM, Stephen Langer wrote: > > >>>> Is it possible to create libBase portably with distutils? It's >>>> possible to do it on Linux by subclassing build_ext.build_ext and >>>> explicitly using self.compiler.compile() and >>>> self.compiler.link_shared_lib() to build the shared library before >>>> calling build_ext.build_ext.build_extensions(). But the same thing >>>> on Mac OS X only creates libBase.so, whereas I need it to create >>>> libBase.dylib. >>>> >>>> >>>> >>> >>> I don't know if this is possible, although I'd guess it is not. >>> >>> >> >> That's too bad. Is there a reason for it? I'd volunteer to work on >> modifying distutils so that it can build a .dylib, but I am not an >> expert on library formats. I don't really know the difference >> between a .so and a .dylib, except that one of them works and the >> other doesn't. Can someone point me in the direction of a good >> reference on the topic? >> > > The most portable way is to not build a libBase at all. Build a > Python extension that has all the functionality from libBase > available in it as a data structure with function pointers, and get > a reference to that from everything that depends on libBase > functionality. Take a look at what Numeric's C API does, for example. Will such a scheme allow me to define a C++ base class in one module, and have other modules define derived classes? I want users to be able to define derived classes without having to recompile the whole program. I also want to be able to choose which parts of the program are loaded, so the base classes and some of the derived classes will be in one library, but other derived classes will be separate. Thanks. -- Steve -- -- EMail: stephen.langer at nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 -- From roberto at dealmeida.net Tue Oct 4 22:51:26 2005 From: roberto at dealmeida.net (Roberto De Almeida) Date: Tue, 04 Oct 2005 17:51:26 -0300 Subject: [Distutils] Include (.h) files not installed? Message-ID: <4342EB4E.50406@dealmeida.net> Hi, I'm using setuptools to distribute my DAP (Data Access Protocol) module. The module depends on Numeric Python and Scientific Python, so I added the following to the setup.py script: install_requires = [ 'Numeric>=22.0', 'scientificpython>=2.4.9' ], There's no download URL for Numeric python in PyPI, so I created a package index at http://pydap.org/package_index.html. I then try to install my module with: easy_install -f http://pydap.org/package_index.html dap Easy_install downloads and install Numeric, and then proceeds to install Scientific. Scientific also depends on Numeric, and requires the include file "Numeric/arrayobject.h" to be compiled. The problem is that this file is not being installed when easy_install installs Numeric. Any hints? TIA, Roberto From robert.kern at gmail.com Wed Oct 5 00:11:08 2005 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 04 Oct 2005 15:11:08 -0700 Subject: [Distutils] Include (.h) files not installed? In-Reply-To: <4342EB4E.50406@dealmeida.net> References: <4342EB4E.50406@dealmeida.net> Message-ID: Roberto De Almeida wrote: > Hi, > > I'm using setuptools to distribute my DAP (Data Access Protocol) module. > The module depends on Numeric Python and Scientific Python, so I added > the following to the setup.py script: > > install_requires = [ > 'Numeric>=22.0', > 'scientificpython>=2.4.9' > ], > > There's no download URL for Numeric python in PyPI, so I created a > package index at http://pydap.org/package_index.html. I then try to > install my module with: > > easy_install -f http://pydap.org/package_index.html dap > > Easy_install downloads and install Numeric, and then proceeds to install > Scientific. Scientific also depends on Numeric, and requires the include > file "Numeric/arrayobject.h" to be compiled. The problem is that this > file is not being installed when easy_install installs Numeric. > > Any hints? eggs don't contain headers which are installed by the install_headers distutils command. I've tried to work around this in scipy_core, the coming Numeric replacement, by placing the headers in the package itself and providing a function for dependant packages' setup.py's to call to locate the headers. For source installations of older Numeric, though, you're SOL unless someone implements header installation for eggs. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From bob at redivi.com Wed Oct 5 00:41:45 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 4 Oct 2005 15:41:45 -0700 Subject: [Distutils] linking modules to a shared library In-Reply-To: References: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> <8E8FCE54-B9C3-4382-9E26-F4DBC83DED2D@redivi.com> Message-ID: <35A4A208-053E-4044-BC64-2AE8340B5B54@redivi.com> On Oct 4, 2005, at 10:46 AM, Stephen Langer wrote: > > On Oct 3, 2005, at 3:15 PM, Bob Ippolito wrote: > > >> >> On Oct 3, 2005, at 10:56 AM, Stephen Langer wrote: >> >> >> >>>>> Is it possible to create libBase portably with distutils? It's >>>>> possible to do it on Linux by subclassing build_ext.build_ext and >>>>> explicitly using self.compiler.compile() and >>>>> self.compiler.link_shared_lib() to build the shared library before >>>>> calling build_ext.build_ext.build_extensions(). But the same >>>>> thing >>>>> on Mac OS X only creates libBase.so, whereas I need it to create >>>>> libBase.dylib. >>>>> >>>>> >>>>> >>>>> >>>> >>>> I don't know if this is possible, although I'd guess it is not. >>>> >>>> >>>> >>> >>> That's too bad. Is there a reason for it? I'd volunteer to work on >>> modifying distutils so that it can build a .dylib, but I am not an >>> expert on library formats. I don't really know the difference >>> between a .so and a .dylib, except that one of them works and the >>> other doesn't. Can someone point me in the direction of a good >>> reference on the topic? >>> >>> >> >> The most portable way is to not build a libBase at all. Build a >> Python extension that has all the functionality from libBase >> available in it as a data structure with function pointers, and get >> a reference to that from everything that depends on libBase >> functionality. Take a look at what Numeric's C API does, for >> example. >> > > Will such a scheme allow me to define a C++ base class in one module, > and have other modules define derived classes? I want users to be > able to define derived classes without having to recompile the whole > program. I also want to be able to choose which parts of the program > are loaded, so the base classes and some of the derived classes will > be in one library, but other derived classes will be separate. I don't know enough about C++ implementation details to answer that.. try it and see if it works. It would definitely work for anything C or Objective-C based, but C++ is crazy so I don't know. -bob From jimmy at retzlaff.com Wed Oct 5 02:01:20 2005 From: jimmy at retzlaff.com (Jimmy Retzlaff) Date: Tue, 4 Oct 2005 17:01:20 -0700 Subject: [Distutils] py2exe has a new maintainer Message-ID: I am taking over the maintenance and support of py2exe from Thomas Heller. As he announced a few weeks ago he is looking to focus on other things. py2exe has been very useful to me over the years and I look forward to keeping it every bit as useful in the future. I plan to make the transition as smooth as possible for users of py2exe. I don't plan to make changes to the license other than adding my name to the list of people not to sue. I will try to be as helpful as Thomas has been in supporting py2exe on the py2exe mailing list and comp.lang.python. The mailing list, the SourceForge project, and the Wiki will continue in their current locations. The web site is moving to http://www.py2exe.org and the old site will forward to the new one so any bookmarks should still work. I will be releasing version 0.6.3 very soon with a few changes Thomas and others have made over the last few weeks. After that my priorities for py2exe will be: - Support - Documentation (which should help familiarize me with the code) - Automated tests (to point out when I haven't familiarized myself enough) - Bug fixes Any help on any of these fronts will be greatly appreciated. After I feel comfortable with things, I hope to work with other projects in the Python packaging community (e.g., cx_Freeze, PyInstaller/McMillan, py2app, setuptools, etc.) to see if we can't find synergies that will make all of them better. I recognize that different packagers are better for different audiences because of licensing, platform, Python versions, and module support among other things. Working together on the common parts (identifying dependencies, customized handling of modules with unique needs, etc.) should make all of the packagers serve their niches better. I'd like to thank Thomas for the great work he's done with py2exe over the years. He's set a very high standard for me to try and maintain. Jimmy From bob at redivi.com Wed Oct 5 02:16:48 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 4 Oct 2005 17:16:48 -0700 Subject: [Distutils] py2exe has a new maintainer In-Reply-To: References: Message-ID: <6F3D2CAB-DA1A-4E8B-B86A-96B7F9E6998E@redivi.com> On Oct 4, 2005, at 5:01 PM, Jimmy Retzlaff wrote: > After I feel comfortable with things, I hope to work with other > projects > in the Python packaging community (e.g., cx_Freeze, > PyInstaller/McMillan, py2app, setuptools, etc.) to see if we can't > find > synergies that will make all of them better. I recognize that > different > packagers are better for different audiences because of licensing, > platform, Python versions, and module support among other things. > Working together on the common parts (identifying dependencies, > customized handling of modules with unique needs, etc.) should make > all > of the packagers serve their niches better. py2app has code for the common parts that are relatively abstracted out of py2app itself and should be platform agnostic, but I just don't have the time (or need) to work on py2app right now let alone make contributions to the general python packaging infrastructure... However, it's there and the license is compatible with just about anything, so feel free to make use it of it or ask questions about it. On the other hand, setuptools/eggs solves a lot of these problems (given appropriate metadata), so I'd target integration with that first and then bring in the "legacy" support that py2app does pretty well. Requiring appropriate metadata from the packages themselves has a much brighter future than maintaining an external "quirks" infrastructure in (each) application packager. -bob From b88zhou at gmail.com Wed Oct 5 04:42:19 2005 From: b88zhou at gmail.com (Brian Zhou) Date: Tue, 4 Oct 2005 19:42:19 -0700 Subject: [Distutils] setuptools in a cross-compilation packaging environment Message-ID: <000701c5c956$692db220$0301a8c0@YLTENGO> Hi all, I am the package maintainer of python and a couple of python packages for the nslu2 optware platform, see http://www.nslu2-linux.org and http://ipkg.nslu2-linux.org/feeds/unslung/cross/ There were some distutils features I really appreciate: 1) setup.py install --prefix This allows us to install into a separate staging area. 2) setup.cfg [build_scripts] executable option 3) setup.cfg [build_ext] include-dirs, library-dirs and rpath options The new setuptools is all nice and easy for end user, but as a package maintainer, I'd like to have the option of building a binary package without all the dependencies. Is it possible? How about the above distutils features, are there any equivalent? Thanks in advance for any advice, -Brian Zhou From pje at telecommunity.com Wed Oct 5 04:52:35 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Oct 2005 22:52:35 -0400 Subject: [Distutils] py2exe has a new maintainer In-Reply-To: <6F3D2CAB-DA1A-4E8B-B86A-96B7F9E6998E@redivi.com> References: Message-ID: <5.1.1.6.0.20051004224328.01fe4518@mail.telecommunity.com> At 05:16 PM 10/4/2005 -0700, Bob Ippolito wrote: >On the other hand, setuptools/eggs solves a lot of these problems >(given appropriate metadata), so I'd target integration with that >first and then bring in the "legacy" support that py2app does pretty >well. Requiring appropriate metadata from the packages themselves >has a much brighter future than maintaining an external "quirks" >infrastructure in (each) application packager. Also, setuptools supports adding setup() keywords and commands in a more extensible way than subclassing Distribution, so there's no need to maintain a bunch of code to hook into the distutils if you just plug in with setuptools. See: http://peak.telecommunity.com/DevCenter/setuptools#creating-distutils-extensions At this point, the easy_install command is capable of dumping all the needed eggs and any generated wrapper scripts into a target directory. What it doesn't have is: * frontend wrappers/packaging to actually build the .exe or app and resources * module-finding or stdlib packaging to locate and bundle the needed portion of the standard library It would be cool if these things could be integrated in such a way that a single setup() could be used to create both an OS X application and a Windows .exe. As it happens, setuptools supports specifying additional eggs needed to run a particular setup command, so we can actually delay the requirement of the egg containing py2exe or py2app until somebody actually requests the command. On the other hand, it's also possible that the metadata could be bundled into the egg metadata directory, which would then make it possible to have external bundling tools that just use the eggs to build the application, without needing to run the setup script at all. From jimmy at retzlaff.com Wed Oct 5 05:21:12 2005 From: jimmy at retzlaff.com (Jimmy Retzlaff) Date: Tue, 4 Oct 2005 20:21:12 -0700 Subject: [Distutils] py2exe has a new maintainer Message-ID: Phillip J. Eby wrote: > At 05:16 PM 10/4/2005 -0700, Bob Ippolito wrote: > >On the other hand, setuptools/eggs solves a lot of these problems > >(given appropriate metadata), so I'd target integration with that > >first and then bring in the "legacy" support that py2app does pretty > >well. Requiring appropriate metadata from the packages themselves > >has a much brighter future than maintaining an external "quirks" > >infrastructure in (each) application packager. I agree with Bob here. I know that in the past I've wanted to add metadata to packages I distribute to make them work better with py2exe (e.g. automatically including DLLs accessed through ctypes), so I think this is the right approach. > Also, setuptools supports adding setup() keywords and commands in a more > extensible way than subclassing Distribution, so there's no need to > maintain a bunch of code to hook into the distutils if you just plug in > with setuptools. See: > > http://peak.telecommunity.com/DevCenter/setuptools#creating-distutils- > extensions > > At this point, the easy_install command is capable of dumping all the > needed eggs and any generated wrapper scripts into a target > directory. What it doesn't have is: > > * frontend wrappers/packaging to actually build the .exe or app and > resources > > * module-finding or stdlib packaging to locate and bundle the needed > portion of the standard library > > It would be cool if these things could be integrated in such a way that a > single setup() could be used to create both an OS X application and a > Windows .exe. As it happens, setuptools supports specifying additional > eggs needed to run a particular setup command, so we can actually delay > the > requirement of the egg containing py2exe or py2app until somebody actually > requests the command. That would be awesome. There might be a few platform specific bits that need to be specified, but it would be nice to let the executable packagers focus on the job of creating a platform specific executable. > On the other hand, it's also possible that the metadata could be bundled > into the egg metadata directory, which would then make it possible to have > external bundling tools that just use the eggs to build the application, > without needing to run the setup script at all. One of my thoughts for py2exe has been a small GUI tool that helps people new to distutils/py2exe to create (and maybe maintain) a setup script graphically. A lot of people who just want to bundle one small utility seem to get confused about options on the command line vs. in parameters and that type of thing. With what you are describing, this could degenerate into just a builder tool, and any GUI could be applied further upstream where more of the metadata lives. If enough of the metadata lives in the individual packages, then the options needed for the executable builder part might be simple enough that the confusion goes away. Alas I won't have time for much of this for a while. I expect I'll have my hands full getting up to speed on py2exe's internals and providing support until I get a break from my current deadlines at work (which is looking like December). When I get to that point, I'll try to reopen the discussion. Regards, Jimmy From pje at telecommunity.com Wed Oct 5 05:53:02 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Oct 2005 23:53:02 -0400 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <000701c5c956$692db220$0301a8c0@YLTENGO> Message-ID: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> At 07:42 PM 10/4/2005 -0700, Brian Zhou wrote: >Hi all, > >I am the package maintainer of python and a couple of python packages for >the nslu2 optware platform, see http://www.nslu2-linux.org and >http://ipkg.nslu2-linux.org/feeds/unslung/cross/ > >There were some distutils features I really appreciate: > >1) setup.py install --prefix >This allows us to install into a separate staging area. > >2) setup.cfg [build_scripts] executable option > >3) setup.cfg [build_ext] include-dirs, library-dirs and rpath options All of these options should work just fine; if they don't, it's almost certainly a setuptools bug and you should report it here. >The new setuptools is all nice and easy for end user, but as a package >maintainer, I'd like to have the option of building a binary package without >all the dependencies. In the long run, this should be done by packaging the result of bdist_egg, and by default doing bdist_rpm will do this now. In the short term, unless you're switching to an all-egg distribution, you'll probably want to use legacy/unmanaged mode. >Is it possible? How about the above distutils >features, are there any equivalent? All of the old 'install' options are available, if you use the appropriate command line flags. Try "setup.py install --help" and look for the rather long option name about doing legacy "unmanaged" installation. With that option, a "classic" installation will be done, without an egg being built. However, even if you don't specify that option, all of the options you listed above should work anyway. Please note, by the way, that some packages simply *cannot* work using the "unmanaged" mode, and never will. Such packages *must* be installed as eggs, period. Among these are setuptools itself, and any packages that are plugins for Trac or Python Paste, or any other extensible system using setuptools to manage plugins. What's more, packages that explicitly use pkg_resources to request their dependencies will not recognize unmanaged packages as fulfilling the dependency, which means that over time there will be increasing demand for packages to be installed as eggs to start with. The simplest way to deal with this is to install vendor-packaged eggs to wherever the distribution normally installs packaged Python packages, and install a corresponding '.pth' file that points to the installed egg. This will ensure that the package is available and can be imported as if it were installed the old way, so it doesn't break any non-setuptools-based packages. It also does not require pkg_resources or setuptools to be installed unless a given package actually uses them. (If the package contains C extensions, the .egg can be expanded as a directory, so that it isn't necessary to extract the .so files at runtime.) From mal at egenix.com Wed Oct 5 10:27:57 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 05 Oct 2005 10:27:57 +0200 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> References: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> Message-ID: <43438E8D.9070304@egenix.com> [Some comments on your strategy...] Phillip J. Eby wrote: >>The new setuptools is all nice and easy for end user, but as a package >>maintainer, I'd like to have the option of building a binary package without >>all the dependencies. > > In the long run, this should be done by packaging the result of bdist_egg, > and by default doing bdist_rpm will do this now. In the short term, unless > you're switching to an all-egg distribution, you'll probably want to use > legacy/unmanaged mode. I think you are missing his point here: As package maintainer you *have* to be able to build a distribution package without all the dependency checks being applied - how else would you be able to bootstrap the package in case you have circular dependencies ? >>Is it possible? How about the above distutils >>features, are there any equivalent? > > All of the old 'install' options are available, if you use the appropriate > command line flags. Try "setup.py install --help" and look for the rather > long option name about doing legacy "unmanaged" installation. With that > option, a "classic" installation will be done, without an egg being > built. However, even if you don't specify that option, all of the options > you listed above should work anyway. > > Please note, by the way, that some packages simply *cannot* work using the > "unmanaged" mode, and never will. Such packages *must* be installed as > eggs, period. Among these are setuptools itself, and any packages that are > plugins for Trac or Python Paste, or any other extensible system using > setuptools to manage plugins. > > What's more, packages that explicitly use pkg_resources to request their > dependencies will not recognize unmanaged packages as fulfilling the > dependency, which means that over time there will be increasing demand for > packages to be installed as eggs to start with. I don't think that eggs are the solution to everything, so you should at least extend the dependency checking code to have it detect already installed packages (by trying import and looking at __version__ strings) or having an option to tell the system: "this dependency is satisfied, trust me". BTW, I haven't looked at your bdist_egg, but since we're currently fighting through binaries releases for our eGenix packages, I thought I'd drop in a note: Please make sure that your eggs catch all possible Python binary build dimensions: * Python version * Python Unicode variant (UCS2, UCS4) * OS name * OS version * Platform architecture (e.g. 32-bit vs. 64-bit) and please also make this scheme extendable, so that it is easy to add more dimensions should they become necessary in the future. To make things easier for the user, the install system should be capable of detecting all these dimensions and use appropriate defaults when looking for an egg. BTW, have you had a look at the ActiveState ppm system for add-on packages ? It looks a lot like your egg system. > The simplest way to deal with this is to install vendor-packaged eggs to > wherever the distribution normally installs packaged Python packages, and > install a corresponding '.pth' file that points to the installed egg. This > will ensure that the package is available and can be imported as if it were > installed the old way, so it doesn't break any non-setuptools-based > packages. It also does not require pkg_resources or setuptools to be > installed unless a given package actually uses them. (If the package > contains C extensions, the .egg can be expanded as a directory, so that it > isn't necessary to extract the .so files at runtime.) Please reconsider your use of .pth files - these cause the Python interpreter startup time to increase significantly. If you just have one of those files pointing to your managed installation path used for eggs, that should be fine (although adding that path to PYTHONPATH still beats having a .pth to parse everytime the interpreter fires up). If you however install a .pth file for every egg, you'll soon end up with an unreasonable startup time which slows down your whole Python installation - including applications that don't use setuptools or any of the eggs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From p.f.moore at gmail.com Wed Oct 5 12:45:50 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 5 Oct 2005 11:45:50 +0100 Subject: [Distutils] py2exe has a new maintainer In-Reply-To: References: Message-ID: <79990c6b0510050345y3ea04cf3xa5b1996af9fada53@mail.gmail.com> On 10/5/05, Jimmy Retzlaff wrote: > I am taking over the maintenance and support of py2exe from Thomas > Heller. As he announced a few weeks ago he is looking to focus on other > things. py2exe has been very useful to me over the years and I look > forward to keeping it every bit as useful in the future. [...] > I'd like to thank Thomas for the great work he's done with py2exe over > the years. He's set a very high standard for me to try and maintain. And I'd like to thank *you* for taking over. I strongly feel that py2exe is an important part of Python for Windows, and it would have been a great loss if it withered away like Gordon McMillan's Installer seems to have. Kudos to Thomas for passing the torch on before that happened. Best of luck with your plans for py2exe. Paul. From pje at telecommunity.com Wed Oct 5 18:37:44 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Oct 2005 12:37:44 -0400 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <43438E8D.9070304@egenix.com> References: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> At 10:27 AM 10/5/2005 +0200, M.-A. Lemburg wrote: >[Some comments on your strategy...] > >Phillip J. Eby wrote: > >>The new setuptools is all nice and easy for end user, but as a package > >>maintainer, I'd like to have the option of building a binary package > without > >>all the dependencies. > > > > In the long run, this should be done by packaging the result of bdist_egg, > > and by default doing bdist_rpm will do this now. In the short term, > unless > > you're switching to an all-egg distribution, you'll probably want to use > > legacy/unmanaged mode. > >I think you are missing his point here: > >As package maintainer you *have* to be able to build a distribution >package without all the dependency checks being applied - how else >would you be able to bootstrap the package in case you have circular >dependencies ? In legacy/unmanaged mode, setuptools' "install" command behaves the way the standard distutils "install" does today, without creating an egg or searching for dependencies. >I don't think that eggs are the solution to everything, so >you should at least extend the dependency checking code to >have it detect already installed packages (by trying import >and looking at __version__ strings) or having an option >to tell the system: "this dependency is satisfied, trust me". There are plans to have a feature like that, and in fact setuptools already has code to hunt down __version__ strings and the like, without even needing to import the packages. It isn't integrated with the rest of the system yet, though. One reason for that is that early feedback suggests that package developers and users would rather have the assurance of having the exact version required by something, as long as the installation process doesn't impose any additional burden on them. Local detection hacks have been primarily requested by packagers, who (quite reasonably) do not want to have to repackage everything as eggs. There is a simple trick that packagers can use to make their legacy packages work as eggs: build .egg-info directories for them in the sys.path directory where the package resides, so that the necessary metadata is present. This does not require the use of .pth files, but it does slow down the process of package discovery for things that do use pkg_resources to locate their dependencies. It also still requires them to repackage existing packages, but doesn't require changing the layout. Also, such packages will currently cause easy_install to warn about conflicting packages if you try to install a different version of the same package, but this will be alleviated soon, as I'm working on a better conflict management mechanism that will allow egg directories on PYTHONPATH to override things in the standard directories. (Currently, eggs are only ever added to the end of sys.path, so if the local packaging system puts .egg-info directories in site-packages, there would be no way to locally override that for an individual user's packages. A future version of setuptools will resolve that issue soon, hopefully in the next few weeks.) As for eggs being the "solution to everything", I would like to point out that what precisely constitutes an egg is an extensible concept. See e.g.: http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html which shows that there are actually three formats that are "eggs" at the moment: 1. .egg zipfiles 2. .egg directories 3. .egg-info marker directories The key requirements for a format to be a pluggable distribution or "egg" are: * Adding it to sys.path must make it importable * It must be possible to discover its PyPI project name (and preferably version and platform) from the filename * It must allow arbitrary data files and directories to be included within packages, and allow arbitrary metadata files and directories to be included for the project as a whole * It must include the standard PKG-INFO metadata These are the absolute minimums, but there are additional specific metadata files and directories that easy_install requires in order to detect possible conflicts, create scripts, etc. Anyway, the point is that what constitutes an "egg" is flexible, but the "add to sys.path and make it importable" requirement certainly limits what formats are practically meaningful. Nonetheless, further extensibility is certainly possible if there's need. >Please make sure that your eggs catch all possible >Python binary build dimensions: > >* Python version >* Python Unicode variant (UCS2, UCS4) >* OS name >* OS version >* Platform architecture (e.g. 32-bit vs. 64-bit) As far as I know, all of this except the Unicode variant is captured in distutils' get_platform(). And if it's not, it should be, since it affects any other kind of bdist mechanism. >and please also make this scheme extendable, so that >it is easy to add more dimensions should they become >necessary in the future. It's extensible by changing the get_platform() and compatible_platform() functions in pkg_resources. By the way, I've issued requests on this list at least twice over the past year for people to provide input about how the platform strings should work; I got no response to either call, so I gave up. Later, when an OS X upgrade created a compatibility problem, somebody finally chipped in with info about what good OS X platform strings might be. I suspect that basically we'll get good platform strings once there are enough people encountering problems with the current ones to suggest a better scheme. :( If you have suggestions, please make them known, and let's get them into the distutils in general, not just our own offshoots thereof. >To make things easier for the user, the install system >should be capable of detecting all these dimensions >and use appropriate defaults when looking for an egg. That's done for those dimensions currently handled by get_platform(), and can be changed by changes to get_platform() and compatible_platforms() in pkg_resources. >Please reconsider your use of .pth files - these cause the >Python interpreter startup time to increase significantly. >If you just have one of those files pointing to your >managed installation path used for eggs, that should >be fine (although adding that path to PYTHONPATH still >beats having a .pth to parse everytime the interpreter >fires up). EasyInstall uses at most one .pth file, to allow packages to be on the path at runtime without needing an explicit 'require()'. However, a vendor creating packages probably doesn't want to have to edit that .pth file, so a trivial alternative is to install a .pth for each package. The tradeoff is startup time versus packager convenience in that case. Having a tool to edit a single .pth file would be good, but not all packaging systems have the ability to run a program at install or uninstall time. If they do, then editing easy-install.pth to add or remove eggs is a better option. Eggs can of course be installed in multi-version mode, in which case no .pth is necessary, but then an explicit require() or a dependency declaration in a setup script is necessary in order to use the package. >If you however install a .pth file for every >egg, you'll soon end up with an unreasonable startup time >which slows down your whole Python installation - including >applications that don't use setuptools or any of the eggs. A single .pth file is certainly an option, and it's what easy_install itself uses. From ianb at colorstudy.com Thu Oct 6 01:08:26 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 05 Oct 2005 18:08:26 -0500 Subject: [Distutils] find_package_data function Message-ID: <43445CEA.6090209@colorstudy.com> I wrote a simple find_package_data function: http://svn.pythonpaste.org/Paste/trunk/paste/util/finddata.py (if you want to use it, you should copy it into your own source) At first I made it just generate a list of wildcards with all the (non-package) subdirectories enumerated, e.g., 'mypackage': ['*.pt', 'templates/*.pt', 'templates/admin/*.pt']. But I decided it's too easy to miss files that way. So now it enumerates everything except files and directories you exclude with patterns (or with explicit paths). There's still no way to include empty directories. You use it like: ... package_data=find_package_data() Which should give a reasonable set. Pass show_ignored=True to see more of what it is doing. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hawk78_it at yahoo.it Thu Oct 6 02:31:18 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Thu, 6 Oct 2005 02:31:18 +0200 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> References: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> Message-ID: <200510060231.18191.hawk78_it@yahoo.it> Alle 18:37, mercoled? 05 ottobre 2005, Phillip J. Eby ha scritto: > > >>The new setuptools is all nice and easy for end user, but as a package > > >>maintainer, I'd like to have the option of building a binary package > > without all the dependencies. > > > > > > In the long run, this should be done by packaging the result of bdist_egg, > > > and by default doing bdist_rpm will do this now. ?In the short term, unless > > > you're switching to an all-egg distribution, you'll probably want to use > > > legacy/unmanaged mode. > > > >I think you are missing his point here: > > > >As package maintainer you *have* to be able to build a distribution > >package without all the dependency checks being applied - how else > >would you be able to bootstrap the package in case you have circular > >dependencies ? > > In legacy/unmanaged mode, setuptools' "install" command behaves the way the > standard distutils "install" does today, without creating an egg or > searching for dependencies. > I think that eggs are a good addition to python distutils as long as they can be created by packagers too. If legacy/unmanaged mode is mandatory to build a package we will never see Fedora/Debian/Mandriva and the other big distros shipping eggs. The ability to install a package without installing its dependencies is crucial to let setuptools beeing widely used IMHO. As I previosly wrote here, another requisite would be very good for packagers and setuptools users too: extra egg file installation from setup.py. If setuptools setup.py need to install an .h file or some /etc/config or some /usr/share/heavy.tar.gz by hand... everything more complex than a simple python module will need its distributor to reinvent the weel every time! I think setuptools is a really good package as is, but I always try to add my 0.0001? when I can. Regards Vincenzo ___________________________________ Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB http://mail.yahoo.it From mal at egenix.com Fri Oct 7 14:01:25 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 07 Oct 2005 14:01:25 +0200 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> References: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> Message-ID: <43466395.2010106@egenix.com> Phillip J. Eby wrote: > At 10:27 AM 10/5/2005 +0200, M.-A. Lemburg wrote: > >>[Some comments on your strategy...] >> >>Phillip J. Eby wrote: >> >>>>The new setuptools is all nice and easy for end user, but as a package >>>>maintainer, I'd like to have the option of building a binary package >> >>without >> >>>>all the dependencies. >>> >>>In the long run, this should be done by packaging the result of bdist_egg, >>>and by default doing bdist_rpm will do this now. In the short term, >> >>unless >> >>>you're switching to an all-egg distribution, you'll probably want to use >>>legacy/unmanaged mode. >> >>I think you are missing his point here: >> >>As package maintainer you *have* to be able to build a distribution >>package without all the dependency checks being applied - how else >>would you be able to bootstrap the package in case you have circular >>dependencies ? > > In legacy/unmanaged mode, setuptools' "install" command behaves the way the > standard distutils "install" does today, without creating an egg or > searching for dependencies. Sorry, maybe I wasn't clear: a package builder needs to *build* a package (rpm, egg, .tar.gz drop in place archive, etc.) without the dependency checks. For the user to be able to turn off the dependency checks when installing an egg using an option is also an often needed feature. rpm often requires this when you want to install packages in different order, in automated installs or due to conflicts in the way different packages name the dependencies. I guess, eggs will exhibit the same problems over time. >>I don't think that eggs are the solution to everything, so >>you should at least extend the dependency checking code to >>have it detect already installed packages (by trying import >>and looking at __version__ strings) or having an option >>to tell the system: "this dependency is satisfied, trust me". > > There are plans to have a feature like that, and in fact setuptools already > has code to hunt down __version__ strings and the like, without even > needing to import the packages. It isn't integrated with the rest of the > system yet, though. > > One reason for that is that early feedback suggests that package developers > and users would rather have the assurance of having the exact version > required by something, as long as the installation process doesn't impose > any additional burden on them. Local detection hacks have been primarily > requested by packagers, who (quite reasonably) do not want to have to > repackage everything as eggs. > > There is a simple trick that packagers can use to make their legacy > packages work as eggs: build .egg-info directories for them in the sys.path > directory where the package resides, so that the necessary metadata is > present. This does not require the use of .pth files, but it does slow > down the process of package discovery for things that do use pkg_resources > to locate their dependencies. It also still requires them to repackage > existing packages, but doesn't require changing the layout. Where would you have to put these directories and what do they contain ? > Also, such > packages will currently cause easy_install to warn about conflicting > packages if you try to install a different version of the same package, but > this will be alleviated soon, as I'm working on a better conflict > management mechanism that will allow egg directories on PYTHONPATH to > override things in the standard directories. (Currently, eggs are only > ever added to the end of sys.path, so if the local packaging system puts > .egg-info directories in site-packages, there would be no way to locally > override that for an individual user's packages. A future version of > setuptools will resolve that issue soon, hopefully in the next few weeks.) I must admit that I haven't followed the discussions about these .egg-info directories. Is there a good reason not to use the already existing PKG-INFO files that distutils builds and which are used by PyPI (aka cheeseshop) ? > As for eggs being the "solution to everything", I would like to point out > that what precisely constitutes an egg is an extensible concept. See e.g.: > > http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html > > which shows that there are actually three formats that are "eggs" at the > moment: > > 1. .egg zipfiles > 2. .egg directories > 3. .egg-info marker directories > > The key requirements for a format to be a pluggable distribution or "egg" are: > > * Adding it to sys.path must make it importable > * It must be possible to discover its PyPI project name (and preferably > version and platform) from the filename > * It must allow arbitrary data files and directories to be included > within packages, and allow arbitrary metadata files and directories to be > included for the project as a whole > * It must include the standard PKG-INFO metadata > > These are the absolute minimums, but there are additional specific metadata > files and directories that easy_install requires in order to detect > possible conflicts, create scripts, etc. > > Anyway, the point is that what constitutes an "egg" is flexible, but the > "add to sys.path and make it importable" requirement certainly limits what > formats are practically meaningful. Nonetheless, further extensibility is > certainly possible if there's need. Hmm, you seem to be making things unnecessarily complicated. Why not just rely on the import mechanism and put all eggs into a common package, e.g. pythoneggs ?! Your EasyInstall script could then modify a file in that package called e.g. database.py which includes all the necessary information about all the installed packages in form of a dictionary. This would have the great advantage of allowing introspection without too much fuzz and reduces the need to search paths, directories and so-on which causes a lot of I/O overhead and slows down startup times for applications needing to check dependency requirements a lot. >>Please make sure that your eggs catch all possible >>Python binary build dimensions: >> >>* Python version >>* Python Unicode variant (UCS2, UCS4) >>* OS name >>* OS version >>* Platform architecture (e.g. 32-bit vs. 64-bit) > > > As far as I know, all of this except the Unicode variant is captured in > distutils' get_platform(). And if it's not, it should be, since it affects > any other kind of bdist mechanism. Agreed. So you use get_platform() for the egg names ? >>and please also make this scheme extendable, so that >>it is easy to add more dimensions should they become >>necessary in the future. > > > It's extensible by changing the get_platform() and compatible_platform() > functions in pkg_resources. Ah, that's monkey patching. Isn't there some better way ? > By the way, I've issued requests on this list at least twice over the past > year for people to provide input about how the platform strings should > work; I got no response to either call, so I gave up. Later, when an OS X > upgrade created a compatibility problem, somebody finally chipped in with > info about what good OS X platform strings might be. I suspect that > basically we'll get good platform strings once there are enough people > encountering problems with the current ones to suggest a better scheme. :( > > If you have suggestions, please make them known, and let's get them into > the distutils in general, not just our own offshoots thereof. This is what we use: def py_version(unicode_aware=1, include_patchlevel=0): """ Return the Python version as short string. If unicode_aware is true (default), the function also tests whether a UCS2 or UCS4 built is running and modifies the version accordingly. If include_patchlevel is true (default is false), the patch level is also included in the version string. """ if include_patchlevel: version = sys.version[:5] else: version = sys.version[:3] if unicode_aware and version > '2.0': # UCS4 builds were introduced in Python 2.1; Note: RPM doesn't # like hyphens to be used in the Python version string which is # why we append the UCS information using an underscore. try: unichr(100000) except ValueError: # UCS2 build (standard) version = version + '_ucs2' else: # UCS4 build (most recent Linux distros) version = version + '_ucs4' return version and then patch the various commands in distutils, e.g.: class mx_build(build): """ build command which knows about our distutils extensions. This build command builds extensions in properly separated directories (which includes building different Unicode variants in different directories). """ ... def finalize_options(self): # Make sure different Python versions are built in separate # directories python_platform = '.%s-%s' % (get_platform(), py_version()) if self.build_platlib is None: self.build_platlib = os.path.join(self.build_base, 'lib' + python_platform) if self.build_temp is None: self.build_temp = os.path.join(self.build_base, 'temp' + python_platform) # Call the base method build.finalize_options(self) class mx_bdist(bdist): """ Generic binary distribution command. """ def finalize_options(self): # Default to - on all platforms if self.plat_name is None: self.plat_name = '%s-py%s' % (get_platform(), py_version()) bdist.finalize_options(self) The result is a build system that can be used to build all binaries for a single platform without getting conflicts and binaries that include a proper platform string, e.g. egenix-mxodbc-zopeda-1.0.9.darwin-8.2.0-Power_Macintosh-py2.3_ucs2.zip egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs2.zip egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs4.zip >>To make things easier for the user, the install system >>should be capable of detecting all these dimensions >>and use appropriate defaults when looking for an egg. > > > That's done for those dimensions currently handled by get_platform(), and > can be changed by changes to get_platform() and compatible_platforms() in > pkg_resources. > > > >>Please reconsider your use of .pth files - these cause the >>Python interpreter startup time to increase significantly. >>If you just have one of those files pointing to your >>managed installation path used for eggs, that should >>be fine (although adding that path to PYTHONPATH still >>beats having a .pth to parse everytime the interpreter >>fires up). > > > EasyInstall uses at most one .pth file, to allow packages to be on the path > at runtime without needing an explicit 'require()'. However, a vendor > creating packages probably doesn't want to have to edit that .pth file, so > a trivial alternative is to install a .pth for each package. The tradeoff > is startup time versus packager convenience in that case. Having a tool to > edit a single .pth file would be good, but not all packaging systems have > the ability to run a program at install or uninstall time. If they do, > then editing easy-install.pth to add or remove eggs is a better option. > > Eggs can of course be installed in multi-version mode, in which case no > .pth is necessary, but then an explicit require() or a dependency > declaration in a setup script is necessary in order to use the package. > > > >>If you however install a .pth file for every >>egg, you'll soon end up with an unreasonable startup time >>which slows down your whole Python installation - including >>applications that don't use setuptools or any of the eggs. > > > A single .pth file is certainly an option, and it's what easy_install > itself uses. Fair enough. Could this be enforced and maybe also removed completely by telling people to add the egg directory to PYTHONPATH ? Note that the pythonegg package approach would pretty much remove the need for these .pth files. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Fri Oct 7 19:11:58 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Oct 2005 13:11:58 -0400 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <43466395.2010106@egenix.com> References: <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> At 02:01 PM 10/7/2005 +0200, M.-A. Lemburg wrote: >Sorry, maybe I wasn't clear: a package builder needs >to *build* a package (rpm, egg, .tar.gz drop in place >archive, etc.) without the dependency checks. bdist_egg simply builds an egg. Dependency checking is a function of *installing* the egg, not building it. >For the user to be able to turn off the dependency checks >when installing an egg using an option is also an often >needed feature. Yes, and it has been on my to-do list for some time. However, the majority of packages in eggs today don't have any dependencies declared anyway, because they're not packages that use setuptools. So the option, if it existed, wouldn't have been very useful until quite recently. In any case, the main refactoring I needed to do before that option could be added is done, so I'll probably add it in the next non-bugfix release. > rpm often requires this when you want >to install packages in different order, in automated >installs or due to conflicts in the way different >packages name the dependencies. I guess, eggs will >exhibit the same problems over time. I'm not sure I follow you here, but in any case there's nothing stopping people from installing eggs by just dropping them in a directory on sys.path without doing any installation steps at all. It's only if you want the egg to be on sys.path at startup without manually munging PYTHONPATH or a .pth file or calling require(), or if you want to install any scripts that you need to run easy_install on the egg. > > There is a simple trick that packagers can use to make their legacy > > packages work as eggs: build .egg-info directories for them in the > sys.path > > directory where the package resides, so that the necessary metadata is > > present. This does not require the use of .pth files, but it does slow > > down the process of package discovery for things that do use pkg_resources > > to locate their dependencies. It also still requires them to repackage > > existing packages, but doesn't require changing the layout. > >Where would you have to put these directories and what >do they contain ? You put them in the directory where the unmanaged packages are installed. At minimum, they contain a PKG-INFO file, and if the package ordinarily uses setuptools, they should also contain whatever else the egg's EGG-INFO directory contained. The directory name is ProjectName.egg-info, where ProjectName is the project's name on PyPI, with non-alphanumerics condensed by the pkg_resources.safe_name() function. >I must admit that I haven't followed the discussions about >these .egg-info directories. Is there a good reason not to >use the already existing PKG-INFO files that distutils builds >and which are used by PyPI (aka cheeseshop) ? I don't know if there's such a reason or not, but in any case that's what we use as part of the egg-info directories. However, we *also* allow for unlimited metadata resources to be provided in egg-info, as this is what allows us to carry things like plugin metadata and scripts in the egg. There are other metadata files listing the C extensions in the package, the "namespace packages" that the egg participates in, and so on. >Hmm, you seem to be making things unnecessarily complicated. That probably just means you're not familiar with the requirements. My first post here about the issues was about this time last year, discussing application plugins and their packaging. The use of eggs for general Python libraries as well as plugins only came into play this January, at Bob Ippolito's urging. So, while there may potentially exist solutions that might be somewhat simpler for certain kinds of Python library packaging, they don't even begin to address the issues for application plugin packaging, which is the raison d'etre of eggs. Trac, for example, lets you simply drop eggs into a plugin directory in order to use them. At some point, Chandler should be allowing this as well, and maybe someday Zope will support it too. It's primarily for these use cases that eggs exist; it just so happens that they make a fine way to manage installed Python packages as well. >Why not just rely on the import mechanism and put all >eggs into a common package, e.g. pythoneggs ?! >Your EasyInstall script could then modify a file in that >package called e.g. database.py which includes all the >necessary information about all the installed packages >in form of a dictionary. You completely lost me. A major feature of eggs is that for an application needing plugins, it can simply scan a directory of downloaded eggs and plug them into itself. Having a required installation mechanism other than "download the egg and put it here" breaks that. What's more, putting them in a single package makes it impossible to have eggs installed in more than one directory, since packages can't span directories, at least not without using setuptools' namespace package facility. And using that facility would mean the runtime would have to always get imported whenever you used an egg - which is *not* required right now unless you're using a zipped egg with a C extension in it. And even then the runtime only gets imported if you actually try to import the C extension. So, it seems to me your approach creates more I/O overhead for using installed packages. Finally, don't forget that eggs allow simultaneous installation of multiple versions of a package. So, you'd *still* have to have sys.path manipulation. >This would have the great advantage of allowing introspection >without too much fuzz and reduces the need to search paths, >directories and so-on which causes a lot of I/O overhead >and slows down startup times for applications needing >to check dependency requirements a lot. And the disadvantage of absolutely requiring install/uninstall steps, which is anathema. Note that with the exception of .egg-info markers (which aren't really intended for production use, anyway, they're a feature for deploying packages under development without needing to build a "real" egg), eggs can be fully introspected from their *filename* for dependency processing purposes. So, if the needed eggs are all on sys.path already, no additional I/O gets done. Identifying all the eggs available in a given directory is one listdir() operation, but it only happens if a suitable package isn't already on sys.path, and the listdir()s happen at most once during a given instance of dependency processing. > >>Please make sure that your eggs catch all possible > >>Python binary build dimensions: > >> > >>* Python version > >>* Python Unicode variant (UCS2, UCS4) > >>* OS name > >>* OS version > >>* Platform architecture (e.g. 32-bit vs. 64-bit) > > > > > > As far as I know, all of this except the Unicode variant is captured in > > distutils' get_platform(). And if it's not, it should be, since it > affects > > any other kind of bdist mechanism. > >Agreed. > >So you use get_platform() for the egg names ? Yes - except on Mac OS X, which has a changed platform string. > >>and please also make this scheme extendable, so that > >>it is easy to add more dimensions should they become > >>necessary in the future. > > > > It's extensible by changing the get_platform() and compatible_platform() > > functions in pkg_resources. > >Ah, that's monkey patching. Isn't there some better way ? Well, my presumption here is that we're going to get the scheme right for Python at large, and make it standard. Are you saying that some packages should have their own scheme? That's not really workable since in order to import the package and use its scheme, we would have to first know that the package was compatible! > > If you have suggestions, please make them known, and let's get them into > > the distutils in general, not just our own offshoots thereof. > >This is what we use: > >def py_version(unicode_aware=1, include_patchlevel=0): > >[snip] >The result is a build system that can be used to build >all binaries for a single platform without getting >conflicts and binaries that include a proper platform >string, e.g. > >egenix-mxodbc-zopeda-1.0.9.darwin-8.2.0-Power_Macintosh-py2.3_ucs2.zip >egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs2.zip >egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs4.zip eggs put the Python version before the platform, because "pure" eggs that don't contain any C code don't include the platform string. We also don't have a UCS flag, but if we did it should be part of the platform string rather than the Python version, since "pure" eggs don't care about the UCS mode, and even if they did, that'd be a requirement of the package rather than the egg itself being platform specific. > > A single .pth file is certainly an option, and it's what easy_install > > itself uses. > >Fair enough. > >Could this be enforced and maybe also removed >completely by telling people to add the egg directory to >PYTHONPATH ? If by "egg directory" you mean a single .egg directory (or zipfile) for a particular package, then yes, for that particular package you could do that. If you mean, can you just put the directory *containing* eggs on PYTHONPATH, then the answer is no, if you want the package to be on sys.path without any special action taken (like calling pkg_resources.require()). >Note that the pythonegg package approach would pretty much >remove the need for these .pth files. Only in the sense that it would require reinventing them in a different form. :) From MKRodriguez at lbl.gov Fri Oct 7 20:22:24 2005 From: MKRodriguez at lbl.gov (Matthew Rodriguez DSD staff) Date: Fri, 07 Oct 2005 11:22:24 -0700 Subject: [Distutils] install_lib lib versus lib64 Message-ID: <4346BCE0.8050400@lbl.gov> Greetings, I've been working a few different 64 bit machines with a few different python installations. I'm trying to determine a foolproof way to see if when install_lib is run libraries are installed in a lib or lib64 directory. Here is what is the best that I've come up with so far, and it doesn't always work. import distutils.sysconfig libdir = distutils.sysconfig.get_python_lib() libdir.find('lib64') Has anyone else dealt with this, and if so, how did they handle it? Thanks, Matt Rodriguez From mal at egenix.com Tue Oct 11 16:46:20 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 11 Oct 2005 16:46:20 +0200 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> References: <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> Message-ID: <434BD03C.1070209@egenix.com> Phillip J. Eby wrote: >>I must admit that I haven't followed the discussions about >>these .egg-info directories. Is there a good reason not to >>use the already existing PKG-INFO files that distutils builds >>and which are used by PyPI (aka cheeseshop) ? > > I don't know if there's such a reason or not, but in any case that's what > we use as part of the egg-info directories. However, we *also* allow for > unlimited metadata resources to be provided in egg-info, as this is what > allows us to carry things like plugin metadata and scripts in the > egg. There are other metadata files listing the C extensions in the > package, the "namespace packages" that the egg participates in, and so on. > >>Hmm, you seem to be making things unnecessarily complicated. > > That probably just means you're not familiar with the requirements. I did read your posting, but still don't understand why you need a multitude of meta-data files in a special directory. PKG-INFO is general and extensible enough to hold all that information, IMHO. >>Why not just rely on the import mechanism and put all >>eggs into a common package, e.g. pythoneggs ?! >>Your EasyInstall script could then modify a file in that >>package called e.g. database.py which includes all the >>necessary information about all the installed packages >>in form of a dictionary. > > > You completely lost me. A major feature of eggs is that for an application > needing plugins, it can simply scan a directory of downloaded eggs and plug > them into itself. Having a required installation mechanism other than > "download the egg and put it here" breaks that. While I don't find a non-managed Python installation mechanism a particularly useful goal to have, you could still have the same thing by using and scanning a sub-directory of the pythoneggs package directory or directories listed in an environment variable PYTHONEGGS as fallback solution (if the egg was not found in the database.py module). > What's more, putting them in a single package makes it impossible to have > eggs installed in more than one directory, since packages can't span > directories, at least not without using setuptools' namespace package > facility. And using that facility would mean the runtime would have to > always get imported whenever you used an egg - which is *not* required > right now unless you're using a zipped egg with a C extension in it. And > even then the runtime only gets imported if you actually try to import the > C extension. So, it seems to me your approach creates more I/O overhead > for using installed packages. If your application wants to support drop-in eggs for plugins, I don't see the need to call some startup code in that application as a problem. If the application does not need drop-in eggs, the package approach would be more effective. > Finally, don't forget that eggs allow simultaneous installation of multiple > versions of a package. So, you'd *still* have to have sys.path manipulation. Nope - the .__path__ attribute of Python packages makes this easy: http://www.python.org/doc/essays/packages.html >>This would have the great advantage of allowing introspection >>without too much fuzz and reduces the need to search paths, >>directories and so-on which causes a lot of I/O overhead >>and slows down startup times for applications needing >>to check dependency requirements a lot. > > > And the disadvantage of absolutely requiring install/uninstall steps, which > is anathema. Oops. I disagree on that one. Not only does install/uninstall make system administration a whole lot easier, it also prevents accidental misconfigurations, permission problems, problems with finding the right paths and locations, etc. etc. Also note that the pythoneggs package approach would still allow you to use unmanaged eggs - albeit as fallback solution or specifically for plugins. >>>>Please make sure that your eggs catch all possible >>>>Python binary build dimensions: >>>> >>>>* Python version >>>>* Python Unicode variant (UCS2, UCS4) >>>>* OS name >>>>* OS version >>>>* Platform architecture (e.g. 32-bit vs. 64-bit) > > Well, my presumption here is that we're going to get the scheme right for > Python at large, and make it standard. Are you saying that some packages > should have their own scheme? That's not really workable since in order to > import the package and use its scheme, we would have to first know that the > package was compatible! We're talking about filenames here - they are intended to be read and understood by humans, not machines (these can use the PKG-INFO data inside the archives or from PyPI). That said, yes, the way platforms are setup, it does sometimes make it necessary to add extra information to such a filename. E.g. say you write a plugin for Zope that only works in Zope3 and not Zope2. Such a plugin would use the "zope3" distinguisher in its archive name. >>>If you have suggestions, please make them known, and let's get them into >>>the distutils in general, not just our own offshoots thereof. >> >>This is what we use: >> >>def py_version(unicode_aware=1, include_patchlevel=0): >> >>[snip] >>The result is a build system that can be used to build >>all binaries for a single platform without getting >>conflicts and binaries that include a proper platform >>string, e.g. >> >>egenix-mxodbc-zopeda-1.0.9.darwin-8.2.0-Power_Macintosh-py2.3_ucs2.zip >>egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs2.zip >>egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs4.zip > > > eggs put the Python version before the platform, because "pure" eggs that > don't contain any C code don't include the platform string. They should really use "noarch" like everybody else does :-) > We also don't > have a UCS flag, but if we did it should be part of the platform string > rather than the Python version, since "pure" eggs don't care about the UCS > mode, and even if they did, that'd be a requirement of the package rather > than the egg itself being platform specific. This is not correct: unichr(100000) won't work in UCS2 builds - it will in UCS4 builds, so even though the .pyc files run on both builds unchanged, the application may very well require the used Python version to be a UCS4 build in order to be able to use UCS4 features. >>>A single .pth file is certainly an option, and it's what easy_install >>>itself uses. >> >>Fair enough. >> >>Could this be enforced and maybe also removed >>completely by telling people to add the egg directory to >>PYTHONPATH ? > > If by "egg directory" you mean a single .egg directory (or zipfile) for a > particular package, then yes, for that particular package you could do > that. If you mean, can you just put the directory *containing* eggs on > PYTHONPATH, then the answer is no, if you want the package to be on > sys.path without any special action taken (like calling > pkg_resources.require()). Calling such an API is OK for applications supporting eggs. I don't see that as a problem. >>Note that the pythonegg package approach would pretty much >>remove the need for these .pth files. > > Only in the sense that it would require reinventing them in a different > form. :) Not really - but we seem to have different views on whether installers are good thing or not, so there's little point in argueing over this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 11 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From stephen.langer at nist.gov Tue Oct 11 16:56:28 2005 From: stephen.langer at nist.gov (Stephen Langer) Date: Tue, 11 Oct 2005 10:56:28 -0400 Subject: [Distutils] linking modules to a shared library In-Reply-To: References: <00E13F0C-1DCA-405B-996D-45B6FF25B783@mac.com> Message-ID: <8EF73A04-96F6-413A-B761-51CC9985300D@nist.gov> On Oct 3, 2005, at 1:17 PM, Ronald Oussoren wrote: > > I'd guess that nobody has needed this functionality badly enough to > volunteer extending distutils :-). > Since I do need it, I've been working on build_shlib and install_shlib distutils commands for building and installing shared libraries that extensions can link to. The commands are working on Linux, OS X 10.4, and (sort-of) on SGI Irix. I have a few questions: Why does UnixCCompiler._compile spawn self.compiler_so? This seems to be wrong (even for regular extensions). It works satisfactorily when the C compiler can also handle C++ code, but fails when the C and C++ compilers are separate, as when using the SGI MIPSpro compiler. Shouldn't _compile use CCompiler.detect_language and spawn compiler_cxx if necessary? (I think this is a problem not just with build_shlib, but with build_ext and build_clib as well.) py2app is the only other package I know that adds commands to distutils. It modifies the Distribution class like this: class Distribution(distutils.dist.Distribution): def __init__(self, attrs): etc... distutils.dist.Distribution.__init__(self, attrs) distutils.core.Distribution = Distribution The problem with this is that any other package (such as mine) that also modifies Distribution will either overwrite or be overwritten by the py2app modifications. Shouldn't Distribution be modified like this: oldDist = distutils.dist.Distribution class Distribution(oldDist): def __init__(self, attrs): etc... oldDist.__init__(self, attrs) distutils.dist.Distribution = Distribution Then if two different packages both modify the same class, the __init__ calls are chained together. What's the best way of testing my code on a variety of platforms? I only have easy access to a few different architectures and operating systems. There are spots in the code where I've pretty much just had to guess at what to do, based on other parts of distutils. Would someone who's more familiar with distutils than I am like to take a look at what I've done and give me some feedback? I can either e-mail the files or put them on a web site. They're not really ready for release yet, I don't think. Thanks, Steve -- -- EMail: stephen.langer at nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 -- From p.f.moore at gmail.com Tue Oct 11 17:56:47 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 11 Oct 2005 16:56:47 +0100 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <434BD03C.1070209@egenix.com> References: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> <434BD03C.1070209@egenix.com> Message-ID: <79990c6b0510110856s6d394a2cjff11010e12888e2f@mail.gmail.com> On 10/11/05, M.-A. Lemburg wrote: > > Only in the sense that it would require reinventing them in a different > > form. :) > > Not really - but we seem to have different views on whether > installers are good thing or not, so there's little point in > argueing over this. I have to say that although I like the egg concept in general, I too am uncomfortable with the "just drop the file in" approach to package management. I'm completely fine with it for plugins, but as you say, I'd expect applications that support plugins to set things up to make drop-in plugins work - standardising that mechanism's fine. But for general packages (PIL, pywin32, cx_Oracle, whatever), I actively want a managed installation. I have made noises about building management tools for eggs, but I think I've been hitting the "absolutely requiring install/uninstall steps, which is anathema" philosophy, which makes such tools awkward to define. So I'd like to see this discussion continue, because I suspect MAL has some important points - which may clash with PJE's goals somewhat, but at least if we get the clashes out in the open, it may help people like me, who can't quite understand why we're having difficulty understanding the concepts, to see where the mental blocks lie :-) Paul. From pje at telecommunity.com Tue Oct 11 18:11:25 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Oct 2005 12:11:25 -0400 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <434BD03C.1070209@egenix.com> References: <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051011112526.01f80df0@mail.telecommunity.com> At 04:46 PM 10/11/2005 +0200, M.-A. Lemburg wrote: >Phillip J. Eby wrote: > >>I must admit that I haven't followed the discussions about > >>these .egg-info directories. Is there a good reason not to > >>use the already existing PKG-INFO files that distutils builds > >>and which are used by PyPI (aka cheeseshop) ? > > > > I don't know if there's such a reason or not, but in any case that's what > > we use as part of the egg-info directories. However, we *also* allow for > > unlimited metadata resources to be provided in egg-info, as this is what > > allows us to carry things like plugin metadata and scripts in the > > egg. There are other metadata files listing the C extensions in the > > package, the "namespace packages" that the egg participates in, and so on. > > > >>Hmm, you seem to be making things unnecessarily complicated. > > > > That probably just means you're not familiar with the requirements. > >I did read your posting, but still don't understand why you >need a multitude of meta-data files in a special directory. > >PKG-INFO is general and extensible enough to hold all that information, >IMHO. And I suppose you have a plan for embedding site.zcml files in PKG-INFO too? How about Trac plugin specifiers? Paste template definitions? Eggs are a format to suppport applications and their plugins. They support arbitrary Python projects as well, because that makes it easy for applications and their plugins to depend on them. They are not merely a distribution format for installing systemwide Python packages. We have plenty of those formats already. > > You completely lost me. A major feature of eggs is that for an > application > > needing plugins, it can simply scan a directory of downloaded eggs and > plug > > them into itself. Having a required installation mechanism other than > > "download the egg and put it here" breaks that. > >While I don't find a non-managed Python installation >mechanism a particularly useful goal to have, It's incredibly useful for application distributors, especially extensible applications and app servers like Zope, Trac, Chandler, etc. They simply cannot afford to rely on the system Python or native packaging system to meet their requirements or provide a quality user experience. >you could still >have the same thing by using and scanning a sub-directory of >the pythoneggs package directory or directories listed in >an environment variable PYTHONEGGS as fallback solution (if the >egg was not found in the database.py module). This approach doesn't allow any eggs to be on sys.path by default, nor does it allow simply importing and using target packages. The current system allows us to create eggs for packages that know nothing about eggs, without making any changes to their code. (We even automatically detect potential __file__ manipulation code, and mark such eggs as needing to be installed in unzipped form.) > > And the disadvantage of absolutely requiring install/uninstall steps, > which > > is anathema. > >Oops. I disagree on that one. Not only does install/uninstall >make system administration a whole lot easier, Eggs are not a system administration tool. They're for people making software and people using it. If a vendor wants to package eggs for the convenience of their users, great. If not, that's okay, because eggs are specifically intended to not require system administrator support. System administrator involvement in this process is a *bug*, not a feature. Users having to beg the sysadmin to get Python packages installed is a Bad Thing. Applications having to rely on what's in site-packages is a Bad Thing. Eggs allow users and applications to manage their own needs, independent of what the site or vendor does or doesn't provide. > >>>>Please make sure that your eggs catch all possible > >>>>Python binary build dimensions: > >>>> > >>>>* Python version > >>>>* Python Unicode variant (UCS2, UCS4) > >>>>* OS name > >>>>* OS version > >>>>* Platform architecture (e.g. 32-bit vs. 64-bit) > > > > Well, my presumption here is that we're going to get the scheme right for > > Python at large, and make it standard. Are you saying that some packages > > should have their own scheme? That's not really workable since in > order to > > import the package and use its scheme, we would have to first know that > the > > package was compatible! > >We're talking about filenames here - they are intended >to be read and understood by humans, not machines (these >can use the PKG-INFO data inside the archives or from >PyPI). If you read the specification, you'll see that this is not the case. Eggs require machine-parseable filenames, as this allows them to be rapidly discovered at runtime for dynamic dependency resolution, with a simple listdir(). Unlike your database.py concept or PEP 262, it is impossible for the "index" to become out-of-sync with the actual state of the installation, because it *is* the current state of the installation. That said, let's do what we can to get the distutils platform strings to be more useful indicators of whether the contained native code can be linked and run by a given Python installation. >That said, yes, the way platforms are setup, it does sometimes >make it necessary to add extra information to such a filename. > >E.g. say you write a plugin for Zope that only works in Zope3 >and not Zope2. Such a plugin would use the "zope3" distinguisher >in its archive name. The purpose of including platform information in an egg's filename is to avoid attempting to link or run "foreign" native code that might cause a hard crash of the Python process. A Zope 2 vs. 3 distinction would not be required as an external designation, since the version dependencies declared by the package will either be resolvable or not. > > We also don't > > have a UCS flag, but if we did it should be part of the platform string > > rather than the Python version, since "pure" eggs don't care about the UCS > > mode, and even if they did, that'd be a requirement of the package rather > > than the egg itself being platform specific. > >This is not correct: unichr(100000) won't work in UCS2 >builds - it will in UCS4 builds, so even though the >.pyc files run on both builds unchanged, the application >may very well require the used Python version to be a >UCS4 build in order to be able to use UCS4 features. As I said, that would be a requirement of the package, rather than the egg itself being platform-specific. Again, the platform string is just a filter to avoid trying to import things that could crash the interpreter (as opposed to merely raising an exception). > >>>A single .pth file is certainly an option, and it's what easy_install > >>>itself uses. > >> > >>Fair enough. > >> > >>Could this be enforced and maybe also removed > >>completely by telling people to add the egg directory to > >>PYTHONPATH ? > > > > If by "egg directory" you mean a single .egg directory (or zipfile) for a > > particular package, then yes, for that particular package you could do > > that. If you mean, can you just put the directory *containing* eggs on > > PYTHONPATH, then the answer is no, if you want the package to be on > > sys.path without any special action taken (like calling > > pkg_resources.require()). > >Calling such an API is OK for applications supporting >eggs. I don't see that as a problem. "applications supporting eggs" is not the same thing as "people using eggs". People using eggs would like, in the general case, to be able to just fire up the Python interpreter and use the packages they've installed, without any special steps. This is especially important for users who are simply using the easy_install toolchain to install arbitrary distutils-based packages. > >>Note that the pythonegg package approach would pretty much > >>remove the need for these .pth files. > > > > Only in the sense that it would require reinventing them in a different > > form. :) > >Not really - but we seem to have different views on whether >installers are good thing or not, so there's little point in >argueing over this. We disagree on whether *requiring* an install step is a good thing. Good installer support is important, which is why EasyInstall can search PyPI, and supports download/extract/build/install for most distutils-based packages, and handles dependency resolution for setuptools-based packages. Being able to provide good installer support is actually an important feature of eggs! *Requiring* installation, however, is a no-no. It should be possible to ship a Python application by just dumping the application script and a bunch of eggs in a single directory. Having to then "install" those eggs somewhere on the target system is a nuisance. It's also nice that there is no way to "corrupt" your index of eggs except by tampering with the eggs themselves. It's hard to mess it up in some unrecoverable way, and everything is simple enough to inspect by hand with common tools like 'ls' and 'less' and 'unzip -v'. From pje at telecommunity.com Tue Oct 11 18:30:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Oct 2005 12:30:38 -0400 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <79990c6b0510110856s6d394a2cjff11010e12888e2f@mail.gmail.co m> References: <434BD03C.1070209@egenix.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> <434BD03C.1070209@egenix.com> Message-ID: <5.1.1.6.0.20051011121229.030744f0@mail.telecommunity.com> At 04:56 PM 10/11/2005 +0100, Paul Moore wrote: >But for general packages (PIL, pywin32, cx_Oracle, whatever), I >actively want a managed installation. I have made noises about >building management tools for eggs, but I think I've been hitting the >"absolutely requiring install/uninstall steps, which is anathema" >philosophy, which makes such tools awkward to define. I don't think that's really the case, it's just that implementing management tools is lower on my to-do list; I don't see them coming in until 0.7 or 0.8, and at the moment I don't have a strong vision of what those commands will look like, except that they'll be things like 'nest rm', 'nest ls', etc. (abbreviated forms) or 'nest uninstall', 'nest list', etc. (unabbreviated). Since I don't have a particularly rigid vision of what those tools will do, I'm wide open to suggestions and especially use cases for what you want to be able to do with them. In the meantime, some of the things that are on my list ahead of the "nest" command subsystem are: * Making it possible for wrapper scripts on Windows to not conflict with a same-named module they import * --no-deps option for easy_install * changing the activation mechanism so eggs get added with higher precedence on sys.path than their containing directory, and that package name conflicts are checked at runtime * Automatic support for creating new "site" directories that allow .pth files, so that Unix users can install eggs anywhere without needing to have root access or create a virtual Python installation. Also, the --allow-hosts option, and maybe even signature checking, may end up landing before the nest command. Meanwhile, there's plenty of room for someone else to create management tools using the existing API, as most everything I've got on my list will not be changing the API or file formats or anything like that. From p.f.moore at gmail.com Tue Oct 11 21:45:52 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 11 Oct 2005 20:45:52 +0100 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <5.1.1.6.0.20051011121229.030744f0@mail.telecommunity.com> References: <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> <434BD03C.1070209@egenix.com> <5.1.1.6.0.20051011121229.030744f0@mail.telecommunity.com> Message-ID: <79990c6b0510111245i5021e879ob7de62956d997edc@mail.gmail.com> On 10/11/05, Phillip J. Eby wrote: > At 04:56 PM 10/11/2005 +0100, Paul Moore wrote: > >But for general packages (PIL, pywin32, cx_Oracle, whatever), I > >actively want a managed installation. I have made noises about > >building management tools for eggs, but I think I've been hitting the > >"absolutely requiring install/uninstall steps, which is anathema" > >philosophy, which makes such tools awkward to define. [...] > Since I don't have a particularly rigid vision of what those tools will do, > I'm wide open to suggestions and especially use cases for what you want to > be able to do with them. OK. The hard one first, and the one I think is most likely to clash with your ideas of what eggs are: Integrate with platform-specific installers (in my specific case, bdist_wininst) so that the "list all packages" command (for example) lists *everything* installed, egg or not. I see this as vital, as eggs are not going to replace bdist_wininst installers for a long time, if ever (get pywin32 and mod_python available as eggs, and I'll think again...) I need one, single, place where I can list all Python packages installed on my machine (and one command that does an uninstall, for that matter). Assume that a command prompt (or Windows explorer window) on a directory under C:\Python24 is disallowed - grubbing round in installation-managed directories manually is a definite no. (This is the one that I view as a key requirement, and the one that I've never managed to get a decent start on - if I had, I might have delivered on my offer to produce some management scripts by now!) By the way, although the support for wrapping bdist_wininst installers as eggs is good, it's not perfect. I know, for example, that the cx_Oracle documentation directory gets lost. Until that's corrected (I know, it won't be if I don't report it...) or the author supplies an official egg version, I'll continue to install cx_Oracle as a bdist_wininst package. And a couple of cases like that, where the egg conversion is imperfect, are enough to leave me thinking that life's too short, and go back to using the installer direct, and only even considering eggs when they are offered by the author, "officially". I *know* these items probably don't match your priorities for eggs. We've had this discussion before. All I'm really trying to point out is that there's a fairly deep difference in our ideas. I'd stopped going on about it, as I don't have the time to provide code in support of my ideas, so I applied the "put up or shut up" principle :-) MAL's comments seemed to echo the same sentiments as me[1], so I thought I'd make one more post. I'll go back to lurking, now... Paul. [1] MAL, if I've misinterpreted your comments, I apologise. PS I'm very interested in trying out TurboGears, which comes as an egg-enabled package, automatically grabbing its dependencies. But I'm scared to do so until I can set up a sandbox machine, as *all* of the machines I use regularly have at least one of TurboGears' dependencies installed as a non-egg, and I don't want the easy_install stuff to mess those up, nor do I want to uninstall my existing packages. Ironic that a system designed to make installations with dependencies *easier* has resulted in me being unable to "just try it out"... From pje at telecommunity.com Tue Oct 11 22:21:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Oct 2005 16:21:53 -0400 Subject: [Distutils] setuptools in a cross-compilation packaging environment In-Reply-To: <79990c6b0510111245i5021e879ob7de62956d997edc@mail.gmail.co m> References: <5.1.1.6.0.20051011121229.030744f0@mail.telecommunity.com> <5.1.1.6.0.20051004234128.01fef4d8@mail.telecommunity.com> <5.1.1.6.0.20051005115453.01f6a8b8@mail.telecommunity.com> <5.1.1.6.0.20051007122809.01f7cfd8@mail.telecommunity.com> <434BD03C.1070209@egenix.com> <5.1.1.6.0.20051011121229.030744f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051011160621.01f35750@mail.telecommunity.com> At 08:45 PM 10/11/2005 +0100, Paul Moore wrote: >Integrate with platform-specific installers (in my specific case, >bdist_wininst) so that the "list all packages" command (for example) >lists *everything* installed, egg or not. Feel free to implement that; I just don't offer an API that knows about arbitrarily installed things, since there isn't any reliable way (that I know of) to know that. :) >(This is the one that I view as a key requirement, and the one that >I've never managed to get a decent start on - if I had, I might have >delivered on my offer to produce some management scripts by now!) My take on it is that if there are compelling reasons to use eggs (such as great package management tools), then sooner or later everyone will use eggs, and the legacy problem will take care of itself. EasyInstall also gives you the option to get rid of legacy packages when you install an egg of the same package. >And a couple of cases like that, where the egg >conversion is imperfect, are enough to leave me thinking that life's >too short, and go back to using the installer direct, and only even >considering eggs when they are offered by the author, "officially". Which is quite reasonable, and certainly doesn't make you any worse off than you already are. When enough packages use eggs, and more package management tools for eggs exist, this balance will shift. It's just going to take time. EasyInstall didn't even exist six months ago, and the pkg_resources runtime was still under construction. >PS I'm very interested in trying out TurboGears, which comes as an >egg-enabled package, automatically grabbing its dependencies. But I'm >scared to do so until I can set up a sandbox machine, as *all* of the >machines I use regularly have at least one of TurboGears' dependencies >installed as a non-egg, and I don't want the easy_install stuff to >mess those up, nor do I want to uninstall my existing packages. Ironic >that a system designed to make installations with dependencies >*easier* has resulted in me being unable to "just try it out"... If you're using a Unix machine, just use the "Non-Root Installation" procedure, or Ian Bicking's sandbox creation script, to set up a "virtual Python". Unfortunately, on Windows this is less than practical due to the lack of symlinks. Later 0.6 releases of setuptools, however, are going to make it trivial to install packages to other directories than site-packages, while avoiding any conflict with already-installed packages. The target directory will have to be either a script/launch directory or else a PYTHONPATH directory, but other than that you'll have the full flexibility that you get when installing to site-packages now. The issue right now is that pkg_resources always adds eggs to the *end* of sys.path, but future versions will add eggs in front of their parent directory on sys.path. There will also be a facility to create a dummy site.py to force Python to pay attention to easy-install.pth in the target directory, no matter what the target directory is. From robbyd at u20.org Thu Oct 13 01:14:36 2005 From: robbyd at u20.org (Robby Dermody) Date: Wed, 12 Oct 2005 19:14:36 -0400 Subject: [Distutils] resource_filename() breaks cxfreeze (py2exe?) Message-ID: <434D98DC.4040007@u20.org> Hi guys, The subject line no doubt tells you all something you already know. This morning, I didn't even know that setuputils existed, so excuse my ignorance on the topic. Basically, I have an application that makes use of a python web application development library called nevow. The current version of nevow (0.5.0) makes use of resource_filename() to get the paths of certain static files that it uses. Things work fine when my app is not frozen, but after I freeze it with cx_freeze resource_filename() turns out to be a breakage point. I would think the same kind of thing would happen with py2exe. See the traceback at the end of this message for more info. Once I freeze the application and run it, my sys.path is set to ['/home/robbyd/dev/inquest/dist/inquest-director-0.7.0-1/inquest-director', '/home/robbyd/dev/inquest/dist/inquest-director-0.7.0-1'] (Meaning the frozen zip file executable, 'inquest-director', and its parent path.) Normally this works fine, due to zipimporter. However, with the static files nevow is attempting to locate with resource_filename(), it seems that the problem is that, resource_filename() wants to look into the zip file but it doesn't know how to do so. (I have not done too much research into how this function works, so I may be missing the point here.) What I'd like to know is why this is the case, and what may happen when resource_filename() gets used in more libraries whose users freeze their applications with cxfreeze/py2exe/macmillian/etc. Is this a move by setuptools to take over the world (of python packaging utilities)? ;) Thanks in advance for any feedback that is offered. At this point I'm just unsure if this is more of a setuptools issue, or more of a cx_Freeze issue. I did take a look at setuptools and I was impressed with it, but my app has certain requirements that make cx_freeze a good choice. Robby ------ Here's the traceback that I get, with the specific error (note that util.resource_filename in webform.py is just an alias to pkg_resource.resource_filename): Traceback (most recent call last): File "/home/robbyd/downloads/cx_Freeze-3.0.1/initscripts/Console.py", line 26, in ? exec code in m.__dict__ File "dist/src/inquest/director/director.py", line 33, in ? File "/home/robbyd/dev/inquest/dist/src/inquest/director/util/twistednet.py", line 22, in ? from nevow import inevow, static File "/usr/lib/python2.3/site-packages/nevow/__init__.py", line 140, in ? load(basic_adapters) File "/usr/lib/python2.3/site-packages/nevow/__init__.py", line 30, in load compy.registerAdapter(a, clean(o), i) File "/usr/lib/python2.3/site-packages/nevow/compy.py", line 32, in registerAdapter adapterFactory = _namedAnyWithBuiltinTranslation(adapterFactory) File "/usr/lib/python2.3/site-packages/nevow/util.py", line 225, in _namedAnyWithBuiltinTranslation return namedAny(name) File "/usr/lib/python2.3/site-packages/nevow/util.py", line 78, in namedAny topLevelPackage = __import__(trialname) File "/usr/lib/python2.3/site-packages/formless/webform.py", line 42, in ? defaultCSS = File(util.resource_filename('formless', 'freeform-default.css'), 'text/css') File "/usr/lib/python2.3/site-packages/setuptools-0.5a13-py2.3.egg/pkg_resources.py", line 676, in resource_filename return get_provider(package_or_requirement).get_resource_filename( File "/usr/lib/python2.3/site-packages/setuptools-0.5a13-py2.3.egg/pkg_resources.py", line 1056, in get_resource_filename raise NotImplementedError( NotImplementedError: resource_filename() only supported for .egg, not .zip From pje at telecommunity.com Thu Oct 13 01:37:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 12 Oct 2005 19:37:49 -0400 Subject: [Distutils] resource_filename() breaks cxfreeze (py2exe?) In-Reply-To: <434D98DC.4040007@u20.org> Message-ID: <5.1.1.6.0.20051012191935.02b8cc98@mail.telecommunity.com> At 07:14 PM 10/12/2005 -0400, Robby Dermody wrote: >Hi guys, > >The subject line no doubt tells you all something you already know. This >morning, I didn't even know that setuputils existed, so excuse my >ignorance on the topic. > >Basically, I have an application that makes use of a python web >application development library called nevow. The current version of >nevow (0.5.0) makes use of resource_filename() to get the paths of >certain static files that it uses. Things work fine when my app is not >frozen, but after I freeze it with cx_freeze resource_filename() turns >out to be a breakage point. I would think the same kind of thing would >happen with py2exe. See the traceback at the end of this message for >more info. > >Once I freeze the application and run it, my sys.path is set to >['/home/robbyd/dev/inquest/dist/inquest-director-0.7.0-1/inquest-director', >'/home/robbyd/dev/inquest/dist/inquest-director-0.7.0-1'] >(Meaning the frozen zip file executable, 'inquest-director', and its >parent path.) > >Normally this works fine, due to zipimporter. However, with the static >files nevow is attempting to locate with resource_filename(), it seems >that the problem is that, resource_filename() wants to look into the zip >file but it doesn't know how to do so. (I have not done too much >research into how this function works, so I may be missing the point >here.) What I'd like to know is why this is the case, and what may >happen when resource_filename() gets used in more libraries whose users >freeze their applications with cxfreeze/py2exe/macmillian/etc. Is this >a move by setuptools to take over the world (of python packaging >utilities)? ;) > >Thanks in advance for any feedback that is offered. At this point I'm >just unsure if this is more of a setuptools issue, or more of a >cx_Freeze issue. I did take a look at setuptools and I was impressed >with it, but my app has certain requirements that make cx_freeze a good >choice. You could say that it's a collision between the program's desire to use the more convenient resource_filename() function in place of the more-portable resource_stream() and resource_string() functions. The latter two functions will work just fine with arbitrary importable zip files, but resource_filename() requires either an extracted zipfile or a zipped egg. The issue here is that resource_filename() requires the static file to be extracted to the filesystem. To be extracted to the filesystem, there needs to be a place to extract. For eggs, the extract location is partly determined by the egg's project name and version number, which are assumed to be "relatively unique", allowing the resource to be extracted to a unique location. The runtime complains in this case because it doesn't find any data that tells it what the name of the egg is that it's doing the extraction on behalf of, so it doesn't know where it should extract to. There are two possible solutions: 1. Lobby the nevow author(s) to use resource_stream() or resource_string() instead 2. Package the .egg in the zipfile as a subdirectory The first one is self-explanatory; it will make the package work even if it's not packaged as an egg. I suggest submitting a patch, if you can. The second one is straightforward in principle, but perhaps not in practice, depending on the constraints of your tools. What you want to do is have a subdirectory *in your zipfile* that is named 'nevow-0.5.0-py2.4.egg', or whatever the correct name is. Inside that subdirectory, you include the original contents of the egg. Then, pkg_resources will know what egg contains the package, and thus do the extraction correctly. A zipfile that contains multiple eggs is sometimes called a "basket", as in "putting all your eggs in one basket". :) In general, pkg_resources tries to treat zipfiles and directories inside zipfiles as equivalents to "real" directories, which means that if you have a .zip on sys.path (or even an executable with a zipfile appended to it), then pkg_resources will happily search the zipfile's root for .egg subdirectories when it's trying to resolve dependencies. As for your world domination question, I suspect that eventually most freezing tools will just dump all the necessary eggs into a basket, since this is simpler and less likely to break than scouting out individual module dependencies. There's also an opportunity for installation-oriented tools to create "download only the bits I need" web installers, etc. From elvelind at gmail.com Thu Oct 13 09:08:09 2005 From: elvelind at gmail.com (Elvelind Grandin) Date: Thu, 13 Oct 2005 09:08:09 +0200 Subject: [Distutils] pkg_resources.iter_entry_points() and pkg_resources.require Message-ID: <5035593a0510130008y160d9774q7c5124a1b3292f93@mail.gmail.com> I'm having a go at using entry points for the plugin system for an app I'm working on. All is working good .. exept when the plugins are installed localy, then I need to require them before they will turn up in iter_entry_points(), which ofcourse breaks everything. Is this a good way around this? -- cheers elvelind grandin From pje at telecommunity.com Thu Oct 13 19:28:35 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 13 Oct 2005 13:28:35 -0400 Subject: [Distutils] pkg_resources.iter_entry_points() and pkg_resources.require In-Reply-To: <5035593a0510130008y160d9774q7c5124a1b3292f93@mail.gmail.co m> Message-ID: <5.1.1.6.0.20051013132000.01f430d8@mail.telecommunity.com> At 09:08 AM 10/13/2005 +0200, Elvelind Grandin wrote: >I'm having a go at using entry points for the plugin system for an app >I'm working on. All is working good .. exept when the plugins are >installed localy, then I need to require them before they will turn up >in iter_entry_points(), which ofcourse breaks everything. >Is this a good way around this? Some applications handle plugins by adding every egg in a "plugins" directory to the working set, e.g.: working_set.add_entry(plugins_dir) env = Environment([plugins_dir]) working_set.require(*env.keys()) This first adds the plugins directory to the directories in the working set, so that items in the plugin directory can be found. It then scans the plugin directory to create an environment, which is a table mapping project names to available distributions for that project. Finally, it require()'s all the projects that have eggs in the plugins directory. At this point, the working set will be populated with all the plugins, although nothing will have been imported. You can then use iter_entry_points() or any other APIs to access your plugins. The alternative strategy is to require plugins to be installed in a "site" directory using easy_install, in which case they will be listed in a .pth file and thus be part of the working set as soon as pkg_resources is imported. From raul at murciano.net Thu Oct 13 21:38:21 2005 From: raul at murciano.net (Raul Murciano) Date: Thu, 13 Oct 2005 21:38:21 +0200 Subject: [Distutils] Using setuptools without root permissions Message-ID: <434EB7AD.2040808@murciano.net> Hi all, I'm trying to install TurboGears by using EasyInstall [1] on a FreeBSD shared host (Textdrive). I have not root permissions, so following PEAK's instructions [2] I've executed Ian Bicking's non_root_python.py script. The script seems to work perfectly, it outputs no errors and: Setuptools version 0.6a5 or greater has been installed. (Run "ez_setup.py -U setuptools" to reinstall or upgrade.) I've checked that ~/bin/ contains python (my custom python) and easy_install executables. Now I try: ~/bin/python >>> import setuptools Traceback (most recent call last): File "", line 1, in ? ImportError: No module named setuptools I've checked the path: ~/bin/python >>> import sys >>> sys.path ['', '/home/murciano/lib', '/home/murciano/lib/python2.4/site-packages', '/home/murciano/lib/python24.zip', '/home/murciano/lib/python2.4', '/home/murciano/lib/python2.4/plat-freebsd5', '/home/murciano/lib/python2.4/lib-tk', '/home/murciano/lib/python2.4/lib-dynload', '/home/murciano/lib/python2.4/site-packages/Numeric', '/home/murciano/lib/python2.4/site-packages/PIL'] And if I insert the setuptools-egg path by hand things works: >>>sys.path.insert(0,"/home/murciano/lib/python2.4/site-packages/setuptools-0.6a5-py2.4.egg") >>>import setuptools >>> I have the same problem with pkg_resources, so I can't execute TurboGears properly. Any help/idea/suggestion would be welcome :-) Greetings, Raul Murciano [1] TurboGears installation for *nix: http://turbogears.org/download/nix.html [2] PEAK EasyInstall non-root installation: http://peak.telecommunity.com/DevCenter/EasyInstall#non-root-installation From pje at telecommunity.com Fri Oct 14 05:29:30 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 13 Oct 2005 23:29:30 -0400 Subject: [Distutils] Using setuptools without root permissions In-Reply-To: <434EB7AD.2040808@murciano.net> Message-ID: <5.1.1.6.0.20051013232744.01f1b6f8@mail.telecommunity.com> At 09:38 PM 10/13/2005 +0200, Raul Murciano wrote: >I have the same problem with pkg_resources, so I can't execute >TurboGears properly. Any help/idea/suggestion would be welcome :-) Is there an easy-install.pth in /home/murciano/lib/python2.4/site-packages? If so, does it contain a line reading '/home/murciano/lib/python2.4/site-packages/setuptools-0.6a5-py2.4.egg'? If not, what does it contain? From raul at murciano.net Fri Oct 14 06:43:19 2005 From: raul at murciano.net (Raul Murciano) Date: Fri, 14 Oct 2005 06:43:19 +0200 Subject: [Distutils] Using setuptools without root permissions In-Reply-To: <5.1.1.6.0.20051013232744.01f1b6f8@mail.telecommunity.com> References: <5.1.1.6.0.20051013232744.01f1b6f8@mail.telecommunity.com> Message-ID: <434F3767.50601@murciano.net> Hi Phillip, > Is there an easy-install.pth in > /home/murciano/lib/python2.4/site-packages? If so, does it contain a > line reading > '/home/murciano/lib/python2.4/site-packages/setuptools-0.6a5-py2.4.egg'? > If not, what does it contain? I think you've found where my problem is, line contains /users/home/murciano/lib/python2.4/site-packages/setuptools-0.6a5-py2.4.egg instead /home/murciano/lib/python2.4/site-packages/setuptools-0.6a5-py2.4.egg Correcting the line makes setuptools and pkg_resources work :-) I still now nothing about my hosting filesystem, pwd returns /users/home/murciano but it seems I must use /home/murciano to put anything to work. I'll ask my admins about that. Thank you very much Phillip, my headache is over now :-) Greetings, Raul From dangoor at gmail.com Mon Oct 17 01:56:39 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Sun, 16 Oct 2005 19:56:39 -0400 Subject: [Distutils] egg_info needs to hit the net? Message-ID: <3f085ecd0510161656o686ca83dm8fa7d49bc95117e7@mail.gmail.com> TurboGears quickstart now does an "egg_info" on the newly-created project. Someone just reported that he got a "Connection Refused" exception and was reasonably wondering why it needed to hit the net in order to start a new project. I've gathered that the cheeseshop was down today and this is why the connection was refused, but I am interested to know why egg_info requires net access. Thanks, Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From pje at telecommunity.com Mon Oct 17 02:33:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 16 Oct 2005 20:33:20 -0400 Subject: [Distutils] egg_info needs to hit the net? In-Reply-To: <3f085ecd0510161656o686ca83dm8fa7d49bc95117e7@mail.gmail.co m> Message-ID: <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> At 07:56 PM 10/16/2005 -0400, Kevin Dangoor wrote: >TurboGears quickstart now does an "egg_info" on the newly-created >project. Someone just reported that he got a "Connection Refused" >exception and was reasonably wondering why it needed to hit the net in >order to start a new project. > >I've gathered that the cheeseshop was down today and this is why the >connection was refused, but I am interested to know why egg_info >requires net access. It's not egg_info that's the problem. The issue is running a setup.py that has a 'setup_requires' argument. If the required thing isn't installed, then setup.py goes looking for it. It just so happens in this case that 'egg_info' is the command, but you could've run 'setup.py --help' and the same thing would happen. You should make sure that tg-admin quickstart installs any setup_requires targets that are in the quickstart template, or else this will happen with every new project. From dangoor at gmail.com Mon Oct 17 03:20:40 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Sun, 16 Oct 2005 21:20:40 -0400 Subject: [Distutils] egg_info needs to hit the net? In-Reply-To: <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> References: <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> Message-ID: <3f085ecd0510161820n31b8915bsdf5fed4f2696f685@mail.gmail.com> On 10/16/05, Phillip J. Eby wrote: > At 07:56 PM 10/16/2005 -0400, Kevin Dangoor wrote: > >TurboGears quickstart now does an "egg_info" on the newly-created > >project. Someone just reported that he got a "Connection Refused" > >exception and was reasonably wondering why it needed to hit the net in > >order to start a new project. > > It's not egg_info that's the problem. The issue is running a setup.py that > has a 'setup_requires' argument. If the required thing isn't installed, > then setup.py goes looking for it. It just so happens in this case that > 'egg_info' is the command, but you could've run 'setup.py --help' and the > same thing would happen. You should make sure that tg-admin quickstart > installs any setup_requires targets that are in the quickstart template, or > else this will happen with every new project. I believe TurboGears itself has the same setup_requires. Wouldn't installing TurboGears itself get you the required package? Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From pje at telecommunity.com Mon Oct 17 04:06:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 16 Oct 2005 22:06:28 -0400 Subject: [Distutils] egg_info needs to hit the net? In-Reply-To: <3f085ecd0510161820n31b8915bsdf5fed4f2696f685@mail.gmail.co m> References: <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051016220352.02e498b0@mail.telecommunity.com> At 09:20 PM 10/16/2005 -0400, Kevin Dangoor wrote: >On 10/16/05, Phillip J. Eby wrote: > > At 07:56 PM 10/16/2005 -0400, Kevin Dangoor wrote: > > >TurboGears quickstart now does an "egg_info" on the newly-created > > >project. Someone just reported that he got a "Connection Refused" > > >exception and was reasonably wondering why it needed to hit the net in > > >order to start a new project. > > > > It's not egg_info that's the problem. The issue is running a setup.py that > > has a 'setup_requires' argument. If the required thing isn't installed, > > then setup.py goes looking for it. It just so happens in this case that > > 'egg_info' is the command, but you could've run 'setup.py --help' and the > > same thing would happen. You should make sure that tg-admin quickstart > > installs any setup_requires targets that are in the quickstart template, or > > else this will happen with every new project. > >I believe TurboGears itself has the same setup_requires. Wouldn't >installing TurboGears itself get you the required package? setup_requires never causes anything to be *installed* on the target system. It's just downloaded to the project directory so the setup script can use it. It sounds like you need to also list it in TurboGears' "install_requires" in this case. Then it'll be installed on the system, and new projects won't need to download it. From dangoor at gmail.com Mon Oct 17 04:17:16 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Sun, 16 Oct 2005 22:17:16 -0400 Subject: [Distutils] egg_info needs to hit the net? In-Reply-To: <5.1.1.6.0.20051016220352.02e498b0@mail.telecommunity.com> References: <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> <5.1.1.6.0.20051016220352.02e498b0@mail.telecommunity.com> Message-ID: <3f085ecd0510161917g39d87117y69a73cf23d947cf7@mail.gmail.com> On 10/16/05, Phillip J. Eby wrote: > setup_requires never causes anything to be *installed* on the target > system. It's just downloaded to the project directory so the setup script > can use it. It sounds like you need to also list it in TurboGears' > "install_requires" in this case. Then it'll be installed on the system, > and new projects won't need to download it. That explains a lot. Thanks! Kevin From pje at telecommunity.com Mon Oct 17 04:48:12 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 16 Oct 2005 22:48:12 -0400 Subject: [Distutils] 0.6a6 in CVS (was Re: egg_info needs to hit the net?) In-Reply-To: <3f085ecd0510161917g39d87117y69a73cf23d947cf7@mail.gmail.co m> References: <5.1.1.6.0.20051016220352.02e498b0@mail.telecommunity.com> <5.1.1.6.0.20051016203051.01ee6da8@mail.telecommunity.com> <5.1.1.6.0.20051016220352.02e498b0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051016224508.02e779b8@mail.telecommunity.com> At 10:17 PM 10/16/2005 -0400, Kevin Dangoor wrote: >On 10/16/05, Phillip J. Eby wrote: > > setup_requires never causes anything to be *installed* on the target > > system. It's just downloaded to the project directory so the setup script > > can use it. It sounds like you need to also list it in TurboGears' > > "install_requires" in this case. Then it'll be installed on the system, > > and new projects won't need to download it. > >That explains a lot. Thanks! No problem. I've added a note about this to the docs in CVS now. By the way, I've also just checked in 0.6a6, which I think should be able to handle all the custom Unix install stuff you can throw at it. It's a major update to the installation instructions, and a 'virtual-python' script based on Ian's contribution is included. And of course there are various bugfixes, etc. I'd appreciate it if you could take it for a spin on a few platforms before I release it, since TurboGears users seem to be my biggest group of "customers" at the moment, especially for non-root Unix installs. And of course, I'd love to get feedback from anybody else, not just Kevin. :) From aknuds-1 at broadpark.no Mon Oct 17 20:41:07 2005 From: aknuds-1 at broadpark.no (Arve Knudsen) Date: Mon, 17 Oct 2005 20:41:07 +0200 Subject: [Distutils] Bugginess in distutils.core.run_setup? Message-ID: Sorry if this has been asked before, but I'm having difficulties in using the distutils.core.run_setup function. Setup scripts that will work happily when run from the commandline ("python setup.py") bomb out when under run_setup. What happens as far as I can tell is that run_setup provides a "locals" argument for execfile, which is used for "import" statements. What happens then is that imports on the global level are not visible to functions, so an exception is raised when a function tries to access a globally imported module. Is this due to faulty logic in run_setup (explicitly specifying the "locals" argument)? Thanks Arve Knudsen From pje at telecommunity.com Mon Oct 17 21:18:06 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Oct 2005 15:18:06 -0400 Subject: [Distutils] Bugginess in distutils.core.run_setup? In-Reply-To: Message-ID: <5.1.1.6.0.20051017151653.01edf810@mail.telecommunity.com> At 08:41 PM 10/17/2005 +0200, Arve Knudsen wrote: >Sorry if this has been asked before, but I'm having difficulties in using >the distutils.core.run_setup function. Setup scripts that will work >happily when run from the commandline ("python setup.py") bomb out when >under run_setup. What happens as far as I can tell is that run_setup >provides a "locals" argument for execfile, which is used for "import" >statements. What happens then is that imports on the global level are not >visible to functions, so an exception is raised when a function tries to >access a globally imported module. Is this due to faulty logic in >run_setup (explicitly specifying the "locals" argument)? The run_setup() is broken in more ways than that. I've usually found it simpler to just write my own, e.g. setuptools.sandbox.run_setup(). From aknuds-1 at broadpark.no Tue Oct 18 00:51:23 2005 From: aknuds-1 at broadpark.no (Arve Knudsen) Date: Tue, 18 Oct 2005 00:51:23 +0200 Subject: [Distutils] Bugginess in distutils.core.run_setup? In-Reply-To: <5.1.1.6.0.20051017151653.01edf810@mail.telecommunity.com> References: <5.1.1.6.0.20051017151653.01edf810@mail.telecommunity.com> Message-ID: On Mon, 17 Oct 2005 21:18:06 +0200, Phillip J. Eby wrote: > At 08:41 PM 10/17/2005 +0200, Arve Knudsen wrote: >> Sorry if this has been asked before, but I'm having difficulties in >> using >> the distutils.core.run_setup function. Setup scripts that will work >> happily when run from the commandline ("python setup.py") bomb out when >> under run_setup. What happens as far as I can tell is that run_setup >> provides a "locals" argument for execfile, which is used for "import" >> statements. What happens then is that imports on the global level are >> not >> visible to functions, so an exception is raised when a function tries to >> access a globally imported module. Is this due to faulty logic in >> run_setup (explicitly specifying the "locals" argument)? > > The run_setup() is broken in more ways than that. I've usually found it > simpler to just write my own, e.g. setuptools.sandbox.run_setup(). > Shame to hear that .. anyway, cheers, I'll most definitely have a look at setuptools.sandbox.run_setup. Arve From robertc at robertcollins.net Wed Oct 19 03:12:05 2005 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 19 Oct 2005 11:12:05 +1000 Subject: [Distutils] distutils with package_dir and scripts gets confused (python 2.4.2) Message-ID: <1129684325.9477.15.camel@localhost.localdomain> I have the following tree: / /foo.py /lib /lib/package (package is a dir with __init__ etc. When, in setup.py I do : setup(packages=['package'], package_dir = {'':lib}), it Just Works. When I add in the script - (packages=['package'], package_dir = {'':lib}, scripts=['foo.py']), it starts installing 'package' twice - once into site-packages/package (where it should), and once into site-packages/lib/package (where it should not) running with debug on, there is no sign of the confusion until 'install_lib' occurs. Cheers, Rob -- GPG key available at: . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/distutils-sig/attachments/20051019/dad4e2f5/attachment.pgp From ekpenyong_moses at yahoo.com Wed Oct 19 12:16:01 2005 From: ekpenyong_moses at yahoo.com (Ekpenyong Moses) Date: Wed, 19 Oct 2005 03:16:01 -0700 (PDT) Subject: [Distutils] Can't find graphics package for Python Message-ID: <20051019101601.36812.qmail@web32906.mail.mud.yahoo.com> Dear Greg, I can't find the graphics.py and pypar.py files for Python, could you help me with the files or point me to a link I can download them. Thanx. Moses E. __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com From dangoor at gmail.com Wed Oct 19 21:03:46 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 19 Oct 2005 15:03:46 -0400 Subject: [Distutils] Making a sane non-root egg installation environment Message-ID: <3f085ecd0510191203i6a7ee114w1b67317ca980df18@mail.gmail.com> Hi, Is there any easy way to emulate the nice behavior we get with Mac OS X's ~/Library/Python/2.4/site-packages on other unix-like systems? Is it possible to use the hacks Bob suggests here: http://bob.pythonmac.org/archives/2005/02/06/using-pth-files-for-python-development/ to just import site and addsitedir to add a similar directory under the user's home directory? After doing that, would it be as simple as pointing easy_install at that directory? Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From pje at telecommunity.com Wed Oct 19 22:12:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Oct 2005 16:12:29 -0400 Subject: [Distutils] Making a sane non-root egg installation environment In-Reply-To: <3f085ecd0510191203i6a7ee114w1b67317ca980df18@mail.gmail.co m> Message-ID: <5.1.1.6.0.20051019160637.01faaf30@mail.telecommunity.com> At 03:03 PM 10/19/2005 -0400, Kevin Dangoor wrote: >Hi, > >Is there any easy way to emulate the nice behavior we get with Mac OS >X's ~/Library/Python/2.4/site-packages on other unix-like systems? > >Is it possible to use the hacks Bob suggests here: >http://bob.pythonmac.org/archives/2005/02/06/using-pth-files-for-python-development/ > >to just import site and addsitedir to add a similar directory under >the user's home directory? Yes. But note that you have to be root to do that! (See also the instructions in EasyInstall.txt, under the --site-dirs option, which explains the same idea.) >After doing that, would it be as simple as pointing easy_install at >that directory? And adding the directory to --site-dirs, which by default reflects only the standard Python .pth-supporting directories. By the way, with 0.6a6 (currently in CVS), there's another option. If you put the setuptools egg directly on PYTHONPATH, it will process .pth files in all PYTHONPATH directories. I added this specifically to make more "traditional" non-root installs easier. You still have to add the directories to --site-dirs, however. 0.6a6 also includes a vastly trimmed-down version of Ian's non-root install script, with fewer options and bells and whistles. You simply give it a '--prefix', which defaults to '~', and it doesn't munge siteconfig or distutils.cfg or any of that. Please give 0.6a6 a whirl and let me know how it works for you, as I'd like to be sure these changes (including the dependency resolution algorithm change) work well for TurboGears. Right now your package is the biggest source of fresh test subjects uncontaminated by prior knowledge of how this stuff is supposed to work. :) But the flip side is that I want to make sure nothing I do results in a huge breakage that causes a backlash against the tool. From steve at refcursdemo.sql Fri Oct 21 09:54:57 2005 From: steve at refcursdemo.sql (steve@refcursdemo.sql) Date: Fri, 21 Oct 2005 13:24:57 +0530 Subject: [Distutils] **VIRUS** Error Message-ID: <20051020072457.711551E4004@bag.python.org> The message contains Unicode characters and has been sent as a binary attachment. -------------- next part -------------- KWF Email scanner found a virus in following attachment: Name: message.zip Content type: application/octet-stream Additional information from antivirus: Generic Malware.a!zip Attachment has been removed by firewall. From dangoor at gmail.com Fri Oct 21 15:10:13 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Fri, 21 Oct 2005 09:10:13 -0400 Subject: [Distutils] Fwd: eggs and py2exe Message-ID: <3f085ecd0510210610y665186f4n359f08d2a81f2eae@mail.gmail.com> Phillip suggested that this would be a good addition to the py2exe wiki, but it appears that the py2exe wiki is closed off. This message is my recipe for using py2exe (and py2app with basically the same code) with eggs. Kevin ---------- Forwarded message ---------- From: Kevin Dangoor Date: Oct 20, 2005 9:20 PM Subject: Re: [TurboGears] TurboGears and py2exe To: turbogears at googlegroups.com This is cut from a couple spots in my script. The basic flow is: 1. unpack zipped eggs, because I believe py2exe chokes on them when resolving dependencies 2. keep track of the top level packages in the eggs 3. add all of the files in the eggs to the data_files, so that the eggs are installed along side the main exe 4. build the exe 5. generate a new library.zip that does not include anything in the top level packages found in step 2 here's steps 1 and 2: import pkg_resources eggs = pkg_resources.require("TurboGears") from setuptools.archive_util import unpack_archive for egg in eggs: if os.path.isdir(egg.location): sys.path.insert(0, egg.location) continue unpack_archive(egg.location, eggdir) eggpacks = set() eggspth = open("build/eggs.pth", "w") for egg in eggs: print egg eggspth.write(os.path.basename(egg.location)) eggspth.write("\n") eggpacks.update(egg.get_metadata_lines("top_level.txt")) eggspth.close() eggpacks.remove("pkg_resources") and here's step 5: import zipfile oldzipfile = "dist/exe/library.zip" newzipfile = "dist/exe/small-library.zip" oldzip = zipfile.ZipFile(oldzipfile, "r") newzip = zipfile.ZipFile(newzipfile, "w", zipfile.ZIP_STORED) for entry in oldzip.infolist(): delim = entry.filename.find("/") if delim == -1: delim = entry.filename.find(".") if delim > -1: if entry.filename[0:delim] in eggpacks: print "Skipping %s, it's in the egg" % (entry.filename) continue newzip.writestr(entry, oldzip.read(entry.filename)) newzip.close() oldzip.close() os.remove(oldzipfile) os.rename(newzipfile, oldzipfile) As I mentioned in that thread from August, this is ugly and it would be nice to fix... the only plus to going this route over fixing it properly was that this route works for both py2exe and py2app. Fixing it properly would've required me to dive into both of those packages. Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From theller at python.net Fri Oct 21 16:16:10 2005 From: theller at python.net (Thomas Heller) Date: Fri, 21 Oct 2005 16:16:10 +0200 Subject: [Distutils] Fwd: eggs and py2exe References: <3f085ecd0510210610y665186f4n359f08d2a81f2eae@mail.gmail.com> Message-ID: Kevin Dangoor writes: > Phillip suggested that this would be a good addition to the py2exe > wiki, but it appears that the py2exe wiki is closed off. This message > is my recipe for using py2exe (and py2app with basically the same > code) with eggs. The wiki should be open once you login: http://starship.python.net/crew/theller/moin.cgi/UserPreferences or click on the UserPreferences link on the upper right edge. Thomas From pau at smsarena.com Fri Oct 21 20:22:40 2005 From: pau at smsarena.com (Pau Aliagas) Date: Fri, 21 Oct 2005 20:22:40 +0200 (CEST) Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode Message-ID: I'm trying to create rpms for sqlobject, as I've been doing for a while, but since it works with setuptools, I'm unable to do it properly. sqlobject requires formencode and also requires setuptools. If I build he rpm with the attached specs, it downloads both of them and packs them in the same rpm. This is part of the output: + /usr/bin/python setup.py build --------------------------------------------------------------------------- This script requires setuptools version 0.6a5 to run (even to display help). I will attempt to download it for you (from http://cheeseshop.python.org/packages/2.3/s/setuptools/), but you may need to enable firewall access for this script first. I will start the download in 15 seconds. --------------------------------------------------------------------------- Downloading http://cheeseshop.python.org/packages/2.3/s/setuptools/setuptools-0.6a5-py2.3.egg But it is installed in: /usr/lib/python2.3/site-packages/setuptools-0.6a6-py2.3.egg What do I want? I want to create an rpm of setuptools, install it and then, I'd expect setuptools to find it. I've tried it, I've built the setupttols rpm, installed it (it goes in the site-packages dir), but still setup.py does not find it :( and adds it to the rpm. The same happens with formencode. My question is: how can I create an rpm of setuptools that sqlobject can find when setup.py build is executed? What am I doing wrong? Thanks for your help. -- ____________________________________ Pau Aliagas (((smsARENA smsarena.com Tel. 902010292 ____________________________________ -------------- next part -------------- # TemplateVer: 0.4 %{?!python:%define python python} %{?!pybasever:%{expand:%%define pybasever %(%{__python} -c "import sys ; print sys.version[:3]")}} %{!?python_sitearch: %define python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)")} %{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")} %define origname FormEncode Name: %{python}-formencode Version: 0.2.2 Release: 1026.1 Summary: HTML form validation, generation, and convertion package Group: Development/Libraries License: Python Software Foundation License URL: http://formencode.org/ Source0: %{origname}-%{version}.tar.bz2 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch BuildRequires: %{python}-devel Requires: python-abi = %{pybasever} %description FormEncode performs look-before-you-leap validation. The idea being that you check all the data related to an operation, then apply it. %prep %setup -n %{origname}-%{version} -q %build CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build %install rm -rf $RPM_BUILD_ROOT %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT --record=INSTALLED_FILES.tmp # Ghost optimized sed 's/\(.*\.pyo\)/%ghost \1/' < INSTALLED_FILES.tmp > INSTALLED_FILES find $RPM_BUILD_ROOT%{python_sitearch} -type d \ | sed "s|^$RPM_BUILD_ROOT|%dir |" >> INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES %defattr(-,root,root,-) %doc docs/* %changelog * Sat Oct 1 2005 Pau Aliagas 0.2.3-1026.1 - updated version from svn trunk * Tue Sep 13 2005 Pau Aliagas 0.2.2-2088.1 - updated version from svn trunk * Mon Aug 29 2005 Pau Aliagas 1.0-763 - first version from svn trunk -------------- next part -------------- # TemplateVer: 0.4 %{?!python:%define python python} %{?!pybasever:%{expand:%%define pybasever %(%{__python} -c "import sys ; print sys.version[:3]")}} %{!?python_sitearch: %define python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)")} %{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")} %define origname SQLObject Name: %{python}-sqlobject Version: 0.7.0 Release: 1.pau Summary: SQLObject is an object-relational mapper. Group: Development/Libraries License: LGPL URL: http://sqlobject.org Source0: %{origname}-%{version}.tar.bz2 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch BuildRequires: %{python}-devel Requires: python-abi = %{pybasever} %description SQLObject is an *object-relational mapper*. It allows you to translate RDBMS table rows into Python objects, and manipulate those objects to transparently manipulate the database. In using SQLObject, you will create a class definition that will describe how the object translates to the database table. SQLObject will produce the code to access the database, and update the database with your changes. The generated interface looks similar to any other interface, and callers need not be aware of the database backend. %prep %setup -n %{origname}-%{version} -q %build CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build %install rm -rf $RPM_BUILD_ROOT %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT --record=INSTALLED_FILES.tmp # Ghost optimized sed 's/\(.*\.pyo\)/%ghost \1/' < INSTALLED_FILES.tmp > INSTALLED_FILES find $RPM_BUILD_ROOT%{python_sitearch} -type d \ | sed "s|^$RPM_BUILD_ROOT|%dir |" >> INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES %defattr(-,root,root,-) %doc README.txt docs/* %changelog * Thu Oct 20 2005 Pau Aliagas 0.7.0-1.pau - latest CVS - require FormEncode * Sat Oct 1 2005 Pau Aliagas 0.7b1-2.pau - latest CVS - require FormEncode * Mon Aug 29 2005 Pau Aliagas 0.6.1-94.pau - latest CVS - require FormEncode * Mon Jul 4 2005 Pau Aliagas 0.6.1-93.pau - latest CVS * Wed Jun 1 2005 Pau Aliagas 0.6.1-92.pau - latest CVS with SingleJoin - update spec file * Wed May 25 2005 Pau Aliagas 0.6.1-91.pau - modify installed files to fix x86_64 version * Tue May 24 2005 Pau Aliagas 0.6.1-91 - svn version - try to update to MySQL-python-1.2.0 to access mysql-4.1 * Wed Apr 24 2005 Pau Aliagas 0.6.1-90 - svn version * Wed Apr 20 2005 Pau Aliagas 0.6.1-1 - new version * Wed Sep 22 2004 Jeff Pitman 0.6-1 - new version * Tue Sep 07 2004 Jeff Pitman 0.5.2-1 - new version -------------- next part -------------- # TemplateVer: 0.4 %{?!python:%define python python} %{?!pybasever:%{expand:%%define pybasever %(%{__python} -c "import sys ; print sys.version[:3]")}} %{!?python_sitearch: %define python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)")} %{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")} %define origname setuptools Name: %{python}-setuptools Version: 0.6a5 Release: 1 Summary: HTML form validation, generation, and convertion package Group: Development/Libraries License: Python Software Foundation License URL: http://setuptools.org/ Source0: %{origname}-%{version}.tar.bz2 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch BuildRequires: %{python}-devel Requires: python-abi = %{pybasever} %description FormEncode performs look-before-you-leap validation. The idea being that you check all the data related to an operation, then apply it. %prep %setup -n %{origname}-%{version} -q %build CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build %install rm -rf $RPM_BUILD_ROOT %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT --record=INSTALLED_FILES.tmp # Ghost optimized sed 's/\(.*\.pyo\)/%ghost \1/' < INSTALLED_FILES.tmp > INSTALLED_FILES find $RPM_BUILD_ROOT%{python_sitearch} -type d \ | sed "s|^$RPM_BUILD_ROOT|%dir |" >> INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES %defattr(-,root,root,-) %doc *.txt %changelog * Sat Oct 1 2005 Pau Aliagas 0.2.3-1026.1 - updated version from svn trunk * Tue Sep 13 2005 Pau Aliagas 0.2.2-2088.1 - updated version from svn trunk * Mon Aug 29 2005 Pau Aliagas 1.0-763 - first version from svn trunk From pje at telecommunity.com Fri Oct 21 21:27:26 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Oct 2005 15:27:26 -0400 Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: Message-ID: <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> At 08:22 PM 10/21/2005 +0200, Pau Aliagas wrote: >I'm trying to create rpms for sqlobject, as I've been doing for a while, >but since it works with setuptools, I'm unable to do it properly. > >sqlobject requires formencode and also requires setuptools. If I build he >rpm with the attached specs, it downloads both of them and packs them in >the same rpm. > >This is part of the output: > >+ /usr/bin/python setup.py build >--------------------------------------------------------------------------- >This script requires setuptools version 0.6a5 to run (even to display >help). I will attempt to download it for you (from >http://cheeseshop.python.org/packages/2.3/s/setuptools/), but >you may need to enable firewall access for this script first. >I will start the download in 15 seconds. >--------------------------------------------------------------------------- >Downloading >http://cheeseshop.python.org/packages/2.3/s/setuptools/setuptools-0.6a5-py2.3.egg > >But it is installed in: >/usr/lib/python2.3/site-packages/setuptools-0.6a6-py2.3.egg Is /usr/bin/python the same Python 2.3? Try this: /usr/bin/python -c "import sys; print sys.path" If the output of this command doesn't include the setuptools egg, then /usr/bin/python isn't the right Python to use. From pau at newtral.org Sat Oct 22 18:30:13 2005 From: pau at newtral.org (Pau Aliagas) Date: Sat, 22 Oct 2005 18:30:13 +0200 (CEST) Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> References: <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> Message-ID: On Fri, 21 Oct 2005, Phillip J. Eby wrote: >> + /usr/bin/python setup.py build >> --------------------------------------------------------------------------- >> This script requires setuptools version 0.6a5 to run (even to display >> help). I will attempt to download it for you (from >> http://cheeseshop.python.org/packages/2.3/s/setuptools/), but >> you may need to enable firewall access for this script first. >> I will start the download in 15 seconds. >> --------------------------------------------------------------------------- >> Downloading >> http://cheeseshop.python.org/packages/2.3/s/setuptools/setuptools-0.6a5-py2.3.egg >> >> But it is installed in: >> /usr/lib/python2.3/site-packages/setuptools-0.6a5-py2.3.egg Everything is installed in: /usr/lib/python2.3/site-packages/setuptools-0.6a5-py2.3.egg/ > Is /usr/bin/python the same Python 2.3? Try this: > > /usr/bin/python -c "import sys; print sys.path" > > If the output of this command doesn't include the setuptools egg, then > /usr/bin/python isn't the right Python to use. This is it, there's the path to site-packages, not directly to the egg. $ /usr/bin/python -c "import sys; print sys.path" ['', '/home/pau/include', '/var/www/html/include_parametres', '/var/www/html/include', '/usr/lib/python23.zip', '/usr/lib/python2.3', '/usr/lib/python2.3/plat-linux2', '/usr/lib/python2.3/lib-tk', '/usr/lib/python2.3/lib-dynload', '/usr/lib/python2.3/site-packages', '/usr/lib/python2.3/site-packages/Numeric', '/usr/lib/python2.3/site-packages/PIL', '/usr/lib/python2.3/site-packages/gtk-2.0'] What am I doing wrong? -- Pau From pje at telecommunity.com Sat Oct 22 19:55:13 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 22 Oct 2005 13:55:13 -0400 Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: References: <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> At 06:30 PM 10/22/2005 +0200, Pau Aliagas wrote: >On Fri, 21 Oct 2005, Phillip J. Eby wrote: > >>>+ /usr/bin/python setup.py build >>>--------------------------------------------------------------------------- >>>This script requires setuptools version 0.6a5 to run (even to display >>>help). I will attempt to download it for you (from >>>http://cheeseshop.python.org/packages/2.3/s/setuptools/), but >>>you may need to enable firewall access for this script first. >>>I will start the download in 15 seconds. >>>--------------------------------------------------------------------------- >>>Downloading >>>http://cheeseshop.python.org/packages/2.3/s/setuptools/setuptools-0.6a5-py2.3.egg >>>But it is installed in: >>>/usr/lib/python2.3/site-packages/setuptools-0.6a5-py2.3.egg > >Everything is installed in: >/usr/lib/python2.3/site-packages/setuptools-0.6a5-py2.3.egg/ How did you install it there? Via easy_install, or using some other process? I suspect that you installed it some way that didn't include the setuptools.pth file. Check for a /usr/lib/python2.3/site-packages/setuptools.pth file. It's probably missing. Alternately, you might see if there is an easy-install.pth, or some other .pth file that points to the setuptools egg. If there is none, I'm guessing you installed a broken RPM or other broken packaging of setuptools, and you should report the problem to whoever packaged it. From pau at newtral.org Sat Oct 22 20:08:33 2005 From: pau at newtral.org (Pau Aliagas) Date: Sat, 22 Oct 2005 20:08:33 +0200 (CEST) Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> References: <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> Message-ID: On Sat, 22 Oct 2005, Phillip J. Eby wrote: >> Everything is installed in: >> /usr/lib/python2.3/site-packages/setuptools-0.6a5-py2.3.egg/ > > How did you install it there? Via easy_install, or using some other process? > I suspect that you installed it some way that didn't include the > setuptools.pth file. I'm the one creating the rpm :) I enclose the spec file. What it does is this: python setup.py build python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT \ --record=INSTALLED_FILES.tmp In INSTALLED_FILES.tmp I cannot see any pth file, so it is not included in the rpm, as thes are the files it will include. > Check for a /usr/lib/python2.3/site-packages/setuptools.pth file. It's > probably missing. Alternately, you might see if there is an > easy-install.pth, or some other .pth file that points to the setuptools egg. You are right, no pth file included, not a single one. > If there is none, I'm guessing you installed a broken RPM or other broken > packaging of setuptools, and you should report the problem to whoever > packaged it. How can I make the pth appear when executing the "python setup.py install" command? Should I execute something else? I'm using the standard Fedora template for creating python rpms. Thanks a lot for your help! -- Pau -------------- next part -------------- # TemplateVer: 0.4 %{?!python:%define python python} %{?!pybasever:%{expand:%%define pybasever %(%{__python} -c "import sys ; print sys.version[:3]")}} %{!?python_sitearch: %define python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)")} %{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")} %define origname setuptools Name: %{python}-setuptools Version: 0.6a5 Release: 1 Summary: Download, build, install, upgrade, and uninstall Python packages -- easily Group: Development/Libraries License: Python Software Foundation License URL: http://peak.telecommunity.com/DevCenter/setuptools Source0: %{origname}-%{version}.tar.bz2 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch BuildRequires: %{python}-devel Requires: python-abi = %{pybasever} %description setup tools is acollection of enhancements to the Python ``distutils`` (for Python 2.3 and up) that allow you to more easily build and distribute Python packages, especially ones that have dependencies on other packages. %prep %setup -n %{origname}-%{version} -q %build CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build %install rm -rf $RPM_BUILD_ROOT %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT --record=INSTALLED_FILES.tmp # Ghost optimized sed 's/\(.*\.pyo\)/%ghost \1/' < INSTALLED_FILES.tmp > INSTALLED_FILES find $RPM_BUILD_ROOT%{python_sitearch} -type d \ | sed "s|^$RPM_BUILD_ROOT|%dir |" >> INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES %defattr(-,root,root,-) %doc *.txt %changelog * Mon Aug 29 2005 Pau Aliagas 0.6a5-1 - first version from svn trunk From pje at telecommunity.com Sat Oct 22 20:39:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 22 Oct 2005 14:39:38 -0400 Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: References: <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051022142045.01fa3380@mail.telecommunity.com> At 08:08 PM 10/22/2005 +0200, Pau Aliagas wrote: >On Sat, 22 Oct 2005, Phillip J. Eby wrote: > >>>Everything is installed in: >>>/usr/lib/python2.3/site-packages/setuptools-0.6a5-py2.3.egg/ >> >>How did you install it there? Via easy_install, or using some other >>process? I suspect that you installed it some way that didn't include the >>setuptools.pth file. > >I'm the one creating the rpm :) I enclose the spec file. >What it does is this: > >python setup.py build >python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT \ > --record=INSTALLED_FILES.tmp > >In INSTALLED_FILES.tmp I cannot see any pth file, so it is not included in >the rpm, as thes are the files it will include. Hm. Looks like easy_install isn't writing them to the --record file, even if it's updating them on disk. But it also may not be writing them on disk, due to the use of the --root option. There's actually a bigger issue, too, which is that the contents of those .pth files are going to be based on the --root, rather than where they'll be really installed on the target system. Ugh. I think the best thing to do is going to be to follow the experimental Gentoo egg installation strategy: http://eggs.gentooexperimental.org/wiki/Overview which adds install/uninstall scripts that add or remove eggs from a 'gentoo-eggs.pth' file. I'd happily accept patches for the bdist_rpm command provided by setuptools, so that setuptools-using packages (including setuptools itself) can build rpms that update an 'rpm-eggs.pth' file. From pje at telecommunity.com Sat Oct 22 21:48:47 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 22 Oct 2005 15:48:47 -0400 Subject: [Distutils] setuptools 0.6a6 released Message-ID: <5.1.1.6.0.20051022154449.01fa3758@mail.telecommunity.com> Release notes: * Added support for "traditional" PYTHONPATH-based non-root installation, and also the convenient ``virtual-python.py`` script, based on a contribution by Ian Bicking. The setuptools egg now contains a hacked ``site`` module that makes the PYTHONPATH-based approach work with .pth files, so that you can get the full EasyInstall feature set on such installations. * Added ``--no-deps`` and ``--allow-hosts`` options to easy_install. * Improved Windows ``.exe`` script wrappers so that the script can have the same name as a module without confusing Python. * Changed dependency processing so that it's breadth-first, allowing a depender's preferences to override those of a dependee, to prevent conflicts when a lower version is acceptable to the dependee, but not the depender. Also, ensure that currently installed/selected packages aren't given precedence over ones desired by a package being installed, which could cause conflict errors. * Activated distributions are now inserted in ``sys.path`` (and the working set) just before the directory that contains them, instead of at the end. This allows e.g. eggs in ``site-packages`` to override unmanaged modules in the same location, and allows eggs found earlier on ``sys.path`` to override ones found later. * When a distribution is activated, it now checks whether any contained non-namespace modules have already been imported and issues a warning if a conflicting module has already been imported. * Fixed a problem extracting zipped files on Windows, when the egg in question has had changed contents but still has the same version number. From pau at newtral.org Mon Oct 24 08:41:14 2005 From: pau at newtral.org (Pau Aliagas) Date: Mon, 24 Oct 2005 08:41:14 +0200 (CEST) Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: <5.1.1.6.0.20051022142045.01fa3380@mail.telecommunity.com> References: <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> <5.1.1.6.0.20051022142045.01fa3380@mail.telecommunity.com> Message-ID: On Sat, 22 Oct 2005, Phillip J. Eby wrote: >> I'm the one creating the rpm :) I enclose the spec file. >> What it does is this: >> >> python setup.py build >> python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT \ >> --record=INSTALLED_FILES.tmp >> >> In INSTALLED_FILES.tmp I cannot see any pth file, so it is not included in >> the rpm, as thes are the files it will include. > > Hm. Looks like easy_install isn't writing them to the --record file, even if > it's updating them on disk. But it also may not be writing them on disk, due > to the use of the --root option. I've seen what happens, after reading carefully the documentation... and the output of the program: when callling: $ python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT \ --record=INSTALLED_FILES.tmp I add --root, so it considers it's a multi installation and does not create a pth file. To overcome this, I create a pth file manually and add it to INSTALLED_FILES.tmp. This works. But for the rpms that depend on setuptools, the best way I've found is to use the --old-and-unmanageable option that does not use the eggs dir structure. One comment here: the name "--old-and-unmanageable" is not very polite adn I think it should be something like --no-eggs. There's a lot of people putting effort in building rpms for several distributions and I think we should not be offensive. Having a package in rpm form in a distribution like Fedora, make it much more successful and wider used. I like the idea of being able to install multiple versions of a program, the idea of installing in a private directory, but the most common use is to install only one version in the system, at least when things are stable. This should be made easy and, in my opinion, it is not. The trick I've used is to install setuptools with the egg dirs and the pth file, and the rest without. I don't like the install system downloading eggs and installing them in the global dirs if you do it like root! Package tools like rpm should be used instead if we want mantainable systems. > There's actually a bigger issue, too, which is that the contents of those > .pth files are going to be based on the --root, rather than where they'll be > really installed on the target system. Ugh. I created it by hand inside the spec file, as explained above. > I think the best thing to do is going to be to follow the experimental Gentoo > egg installation strategy: > > http://eggs.gentooexperimental.org/wiki/Overview > > which adds install/uninstall scripts that add or remove eggs from a > 'gentoo-eggs.pth' file. It sounds interesting, but in an rpm system you only can have one version of each package :( > I'd happily accept patches for the bdist_rpm command provided by setuptools, > so that setuptools-using packages (including setuptools itself) can build > rpms that update an 'rpm-eggs.pth' file. I'll see if I can do something, the spec files can be automatically created from the egg info. Thanks Pau -- Pau From ianb at colorstudy.com Tue Oct 25 23:30:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 25 Oct 2005 16:30:39 -0500 Subject: [Distutils] setuptools: .egg-info/ and PKG-INFO Message-ID: <435EA3FF.7020104@colorstudy.com> I'm trying to figure out how I should deal with the .egg-info directory, and PKG-INFO. Right now PKG-INFO is being generated from setup.py. So I don't want to put it in my source control repository, since it is derivative. But I'm maintaining by hand some other files in the .egg-info directory, usually custom package metadata that we consume internally, so I need those files (and .egg-info) in the repository, and hence in the checkout. However, without PKG-INFO lots of things break, as pkg_resources gives a ValueError during some operations (e.g.: ValueError: ("Missing 'Version:' header and/or PKG-INFO file", ProjectName [unknown version] (/usr/home/ianb/co/IscapeLib/tests/appsetup/output/Proj-05))) -- this happens anytime I iterate over entry points, including when setuptools iterates over entry points, which means lots of setup.py commands are broken. They are broken simply because the incomplete .egg-info directory is on the path, even though I don't need direct access to it. I can run setup.py egg_info (creating PKG-INFO) for some reason (that I have not yet determined), but not my own custom setup.py extensions. This includes stuff like "python setup.py egg_info my_custom_command", so I have to write PKG-INFO in a completely separate command. If the .egg-info directory doesn't exist then pkg_resources will be happy, because it won't try to read the package as an egg at all. But it does, and so it's just awkward. Frankly I'd be 100% okay putting PKG-INFO in source control if it was the authoritative source, and I don't particularly care to use setup.py for this data. But that's a cop-out, I should really figure this out as it is. So... what do I do here? Should pkg_resources really just ignore packages without a PKG-INFO file? At least during scan? If I have to live with running setup.py egg_info until 0.6a7, that's fine. BTW, here's the full traceback I get: Traceback (most recent call last): File "setup.py", line 18, in ? package_data={ File "/usr/local/lib/python2.4/distutils/core.py", line 135, in setup ok = dist.parse_command_line() File "/usr/home/ianb/co/setuptools/setuptools/dist.py", line 227, in parse_command_line result = _Distribution.parse_command_line(self) File "/usr/local/lib/python2.4/distutils/dist.py", line 432, in parse_command_line args = self._parse_command_opts(parser, args) File "/usr/home/ianb/co/setuptools/setuptools/dist.py", line 549, in _parse_command_opts nargs = _Distribution._parse_command_opts(self, parser, args) File "/usr/local/lib/python2.4/distutils/dist.py", line 490, in _parse_command_opts cmd_class = self.get_command_class(command) File "/usr/home/ianb/co/setuptools/setuptools/dist.py", line 353, in get_command_class ep.require(installer=self.fetch_build_egg) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 1637, in require working_set.resolve(self.dist.requires(self.extras),env,installer)) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 480, in resolve env = Environment(self.entries) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 558, in __init__ self.scan(search_path) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 588, in scan self.add(dist) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 610, in add if dist not in dists: File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 1760, in __cmp__ def __cmp__(self, other): return cmp(self.hashcmp, other) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 1760, in __cmp__ def __cmp__(self, other): return cmp(self.hashcmp, other) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 1755, in lambda self: ( File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 1782, in parsed_version self._parsed_version = pv = parse_version(self.version) File "/usr/home/ianb/co/setuptools/pkg_resources.py", line 1797, in version raise ValueError( ValueError: ("Missing 'Version:' header and/or PKG-INFO file", ProjectName [unknown version] (/usr/home/ianb/co/IscapeLib/tests/appsetup/output/Proj-05)) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From blais at furius.ca Thu Oct 27 23:59:26 2005 From: blais at furius.ca (Martin Blais) Date: Thu, 27 Oct 2005 17:59:26 -0400 Subject: [Distutils] bdist_wininst and install directory. Message-ID: <8393fff0510271459r5b8bee64pc93035f2e3657e5d@mail.gmail.com> Hi I tried using bdist_wininst to generate a self-extracting executable, and it works great except for one thing: I cannot change the installation root default, and when it runs the user cannot change the value either. My target users --at a client site, intranet, small private tool-- will not have administrative privileges and will not be able to use that. I tracked down the source of this setting and it is hardcoded in the source code for wininst-7.1.exe, in the variable new_scheme. 1. Is there any particular reason for this state of affairs? Why isn't the default value from [install] used? If I made a fix for this to work, would anyone have objections/ 2. Why can't the user change the directory name where the files are going to be installed? Without this ability, this otherwise very pleasant setup target is useless to me, and I would assume to many others. cheers, From dangoor at gmail.com Fri Oct 28 16:35:11 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Fri, 28 Oct 2005 10:35:11 -0400 Subject: [Distutils] find_links when developing Message-ID: <3f085ecd0510280735k39d28626q5a6fe0f9bca53f29@mail.gmail.com> The TurboGears setup.cfg now has this in it: [easy_install] find_links=http://www.turbogears.org/download/index.html http://peak.telecommunity.com/snapshots Trailing slash aside, people seem to be having trouble when they do a setup.py develop or setup.py install with it not finding the snapshot packages at peak.telecommunity. Is there something I'm missing about this? I managed to get it to go on my machine yesterday... Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid at blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com From ianb at colorstudy.com Fri Oct 28 18:09:46 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 28 Oct 2005 11:09:46 -0500 Subject: [Distutils] find_links when developing In-Reply-To: <3f085ecd0510280735k39d28626q5a6fe0f9bca53f29@mail.gmail.com> References: <3f085ecd0510280735k39d28626q5a6fe0f9bca53f29@mail.gmail.com> Message-ID: <43624D4A.8030803@colorstudy.com> Kevin Dangoor wrote: > The TurboGears setup.cfg now has this in it: > [easy_install] > find_links=http://www.turbogears.org/download/index.html > http://peak.telecommunity.com/snapshots > > Trailing slash aside, people seem to be having trouble when they do a > setup.py develop or setup.py install with it not finding the snapshot > packages at peak.telecommunity. Is there something I'm missing about > this? I managed to get it to go on my machine yesterday... Since those options only apply to the easy_install command, then I wouldn't expect them to be used when running setup.py develop. Unless that triggers an easy_install command internally? Anyway, I think I suggested a change earlier, that find_links.txt could be put in the Egg metadata (.egg-info directory) and easy_install (and other installation processes?) could read from there. That might provide an indirect resolution to this issue. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From theller at python.net Fri Oct 28 19:46:07 2005 From: theller at python.net (Thomas Heller) Date: Fri, 28 Oct 2005 19:46:07 +0200 Subject: [Distutils] bdist_wininst and install directory. References: <8393fff0510271459r5b8bee64pc93035f2e3657e5d@mail.gmail.com> Message-ID: Martin Blais writes: > Hi > > I tried using bdist_wininst to generate a self-extracting executable, > and it works great except for one thing: I cannot change the > installation root default, and when it runs the user cannot change the > value either. My target users --at a client site, intranet, small > private tool-- will not have administrative privileges and will not be > able to use that. > > I tracked down the source of this setting and it is hardcoded in the > source code for wininst-7.1.exe, in the variable new_scheme. > > 1. Is there any particular reason for this state of affairs? Why > isn't the default value from [install] used? If I made a fix for this > to work, would anyone have objections/ > > 2. Why can't the user change the directory name where the files are > going to be installed? > > Without this ability, this otherwise very pleasant setup target is > useless to me, and I would assume to many others. Martin, as the bdist_wininst author I should give you a response. I don't have the time (nor any plans) to develop bdist_wininst any further. You could try to cook up a patch and convince a Python developer to look at it and hopefully commit it. One other ideas would be to use Martin v. Loewis bdist_msi instead. Thanks, Thomas From pje at telecommunity.com Sat Oct 29 03:10:13 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 28 Oct 2005 21:10:13 -0400 Subject: [Distutils] creating rpms of setuptools, sqlobject and formencode In-Reply-To: References: <5.1.1.6.0.20051022142045.01fa3380@mail.telecommunity.com> <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051021152459.01fa9620@mail.telecommunity.com> <5.1.1.6.0.20051022135236.029ae058@mail.telecommunity.com> <5.1.1.6.0.20051022142045.01fa3380@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051028202056.01fc8658@mail.telecommunity.com> At 08:41 AM 10/24/2005 +0200, Pau Aliagas wrote: >On Sat, 22 Oct 2005, Phillip J. Eby wrote: >>>I'm the one creating the rpm :) I enclose the spec file. >>>What it does is this: >>>python setup.py build >>>python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT \ >>> --record=INSTALLED_FILES.tmp >>>In INSTALLED_FILES.tmp I cannot see any pth file, so it is not included >>>in the rpm, as thes are the files it will include. >> >>Hm. Looks like easy_install isn't writing them to the --record file, >>even if it's updating them on disk. But it also may not be writing them >>on disk, due to the use of the --root option. > >I've seen what happens, after reading carefully the documentation... and >the output of the program: when callling: >$ python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT \ > --record=INSTALLED_FILES.tmp > >I add --root, so it considers it's a multi installation and does not >create a pth file. > >To overcome this, I create a pth file manually and add it to >INSTALLED_FILES.tmp. This works. Great; ideally, though, this should be a single file for all rpm-wrapped eggs, say 'rpm-eggs.pth' in the site-packages directory. >But for the rpms that depend on setuptools, the best way I've found is to >use the --old-and-unmanageable option that does not use the eggs dir structure. I presume you mean for rpms that do *not* depend on setuptools. >One comment here: the name "--old-and-unmanageable" is not very polite adn >I think it should be something like --no-eggs. There's a lot of people >putting effort in building rpms for several distributions and I think we >should not be offensive. The setuptools 'bdist_rpm' command has a '--no-egg' option already. The '--old-and-unmanageable' option is for 'install', not 'bdist_rpm'. It is deliberately derogatory because many setuptools-based packages (including setuptools itself) simply *will not work* when installed that way, because the distribution's metadata cannot be included. >The trick I've used is to install setuptools with the egg dirs and the pth >file, and the rest without. I don't like the install system downloading >eggs and installing them in the global dirs if you do it like root! >Package tools like rpm should be used instead if we want mantainable systems. The goals of system packagers and system administrators are quite often at odds with the goals of users, application developers and application deployers. However, since the packagers and administrators exist to serve the needs of users, not the other way around, I prefer to make matters better for the latter group first, and then leverage their desire for improved conditions to bring the packagers and administrators into the fold. :) Note in particular that eggs provide application deployers with a *massive* increase in supportability, compared to using native packaging systems. They need only learn one system that can then be used for their Python application across Mac, Windows, and every flavor of Linux or BSD. In contrast, a package like TurboGears, that depends on almost a dozen other Python packages, would be unsupportable if the author had to know the ins and outs of every Linux flavor's packaging system in order to check for the availability of dependencies. Of course, I'm always happy to help packagers and administrators develop ways to have their existing packaging systems and packages provide eggs! I just am dealing with it on the passive side. In the long run, it will be easier for packagers to provide all Python packages as eggs, because there will eventually be so many packages that either must be an egg to function at all, or which require enough other dependencies as eggs that it makes more sense to just bite the bullet and build all Python packages as eggs. (Note that any distutils-packaged Python code may be packaged in egg form, but not every egg-packaged application can be installed in --old-and-unmanageable form.) >>I think the best thing to do is going to be to follow the experimental >>Gentoo egg installation strategy: >> >> http://eggs.gentooexperimental.org/wiki/Overview >> >>which adds install/uninstall scripts that add or remove eggs from a >>'gentoo-eggs.pth' file. > >It sounds interesting, but in an rpm system you only can have one version >of each package :( It would still be better to use a single .pth file for all rpm-deployed eggs, to minimize the impact on interpreter start time. The fact that only one version is present at a time is moot because the .pth file only ever lists one version at a time anyway. From pje at telecommunity.com Sat Oct 29 03:14:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 28 Oct 2005 21:14:20 -0400 Subject: [Distutils] setuptools: .egg-info/ and PKG-INFO In-Reply-To: <435EA3FF.7020104@colorstudy.com> Message-ID: <5.1.1.6.0.20051028211034.02c47680@mail.telecommunity.com> At 04:30 PM 10/25/2005 -0500, Ian Bicking wrote: >I'm trying to figure out how I should deal with the .egg-info directory, >and PKG-INFO. > >Right now PKG-INFO is being generated from setup.py. So I don't want to >put it in my source control repository, since it is derivative. But I'm >maintaining by hand some other files in the .egg-info directory, usually >custom package metadata that we consume internally, so I need those >files (and .egg-info) in the repository, and hence in the checkout. > >However, without PKG-INFO lots of things break, as pkg_resources gives a >ValueError during some operations (e.g.: ValueError: ("Missing >'Version:' header and/or PKG-INFO file", ProjectName [unknown version] >(/usr/home/ianb/co/IscapeLib/tests/appsetup/output/Proj-05))) -- this >happens anytime I iterate over entry points, including when setuptools >iterates over entry points, which means lots of setup.py commands are >broken. They are broken simply because the incomplete .egg-info >directory is on the path, even though I don't need direct access to it. > I can run setup.py egg_info (creating PKG-INFO) for some reason (that >I have not yet determined), but not my own custom setup.py extensions. >This includes stuff like "python setup.py egg_info my_custom_command", >so I have to write PKG-INFO in a completely separate command. 'setup.py develop' should create the info and put the directory on the path for you. Likewise, so does 'setup.py test'. Which other commands are giving you problems? >So... what do I do here? Should pkg_resources really just ignore >packages without a PKG-INFO file? At least during scan? If I have to >live with running setup.py egg_info until 0.6a7, that's fine. How would pkg_resources then determine what version your package is, if it's under development? It has to know the version in order to be able to detect duplicates during scanning, and also to order available packages by version number. From p.f.moore at gmail.com Sat Oct 29 13:48:46 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 29 Oct 2005 12:48:46 +0100 Subject: [Distutils] Announcing bdist_msi 1.0b1 In-Reply-To: <4310BBB5.2020109@v.loewis.de> References: <4310BBB5.2020109@v.loewis.de> Message-ID: <79990c6b0510290448h352b46c2ge95c9d8d28611595@mail.gmail.com> On 8/27/05, "Martin v. L?wis" wrote: > I'm happy to announce the first beta release of my long-planned > distutils command bdist_msi, at > > http://www.dcl.hpi.uni-potsdam.de/home/loewis/bdist_msi/ > http://www.dcl.hpi.uni-potsdam.de/home/loewis/bdist_msi/bdist_msi-1.0b1.win32-py2.4.msi > http://www.dcl.hpi.uni-potsdam.de/home/loewis/bdist_msi/bdist_msi-1.0b1.zip > > Please let me know what you think; it may take some weeks before I can > respond, though. It took me far longer than that to get round to checking it out, sorry :-) One extremely minor point: - bdist_wininst installs a package (in "Add/Remove Programs" under the name "Python 2.4 whatever n.nn", but bdist_msi misses out the "Python 2.4" part. I find the inclusion of "Python" in the name is very useful to group Python packages together in the list - would it be possible to make bdist_msi match bdist_wininst in this? Regards, Paul. From Richard at artsalliancemedia.com Mon Oct 31 12:22:22 2005 From: Richard at artsalliancemedia.com (Richard Cooper) Date: Mon, 31 Oct 2005 11:22:22 -0000 Subject: [Distutils] cli.exe missing from py2.3 egg? Message-ID: I've just tried installing setuptools 0.6a6 so that I could install nose (http://somethingaboutorange.com/mrl/projects/nose/) via easy_install but I've hit a problem. It looks like cli.exe and gui.exe are not getting installed. To check I manually downloaded both setuptools-0.6a6-py2.4.egg and setuptools-0.6a6-py2.3.egg from the cheeseshop and opened them with a Zip viewer. Sure enough the py2.4 egg contains both exes but the py2.3 one doesn't. Should the py2.3 egg include these files? Thanks, Richard PS: Here is a transcript of my installation attempt. --- C:\data>python ez_setup.py Downloading http://cheeseshop.python.org/packages/2.3/s/setuptools/setuptools-0. 6a6-py2.3.egg Processing setuptools-0.6a6-py2.3.egg creating c:\progra~1\python23\lib\site-packages\setuptools-0.6a6-py2.3.egg Extracting setuptools-0.6a6-py2.3.egg to c:\progra~1\python23\lib\site-packages Adding setuptools 0.6a6 to easy-install.pth file Installing easy_install-script.py script to C:\progra~1\python23\Scripts error: None C:\data>python "c:\Program Files\Python23\Scripts\easy_install-script.py" nose Searching for nose Reading http://www.python.org/pypi/nose/ Reading http://somethingaboutorange.com/mrl/projects/nose/ Best match: nose 0.6 Downloading http://somethingaboutorange.com/mrl/projects/nose/nose-0.6.tar.gz Processing nose-0.6.tar.gz Running nose-0.6\setup.py -q bdist_egg --dist-dir c:\docume~1\rcooper\locals~1\t emp\easy_install-ijtcn6\nose-0.6\egg-dist-tmp-sfmqvp zip_safe flag not set; analyzing archive contents... nose.core: module references __file__ nose.inspect_assert: module MAY be using inspect.getframeinfo Adding nose 0.6 to easy-install.pth file Installing nosetests-script.py script to C:\progra~1\python23\Scripts error: c:\progra~1\python23\lib\site-packages\setuptools-0.6a6-py2.3.egg\setupt o ols\cli.exe: No such file or directory From jjl at pobox.com Mon Oct 31 21:37:41 2005 From: jjl at pobox.com (John J Lee) Date: Mon, 31 Oct 2005 20:37:41 +0000 (GMT Standard Time) Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name Message-ID: Hi all, I'm trying to get some Egg/setuptools/easy_install-ified packages working. I've run into three problems. I'm running on Windows XP SP2 with Python 2.4.2. I believe I have the latest easy_install. I have setuptools 0.6a6. First, here's what I'm doing (readers should be able to duplicate this, since the SVN URL that this invocation makes use of is up on the Cheese Shop): C:\jjlee\code>easy_install -m ClientForm==dev Searching for ClientForm==dev Reading http://www.python.org/pypi/ClientForm/ Reading http://wwwsearch.sourceforge.net/ClientForm/ Best match: ClientForm dev Downloading http://codespeak.net/svn/wwwsearch/ClientForm/trunk#egg=ClientForm-dev Doing subversion checkout from http://codespeak.net/svn/wwwsearch/ClientForm/trunk to c:\docume~1\johnle~1\locals~1\temp\easy_install-38wlkt\trunk svn: Unknown hostname 'svn.eby-sarna.com' Processing trunk Running setup.py -q bdist_egg --dist-dir c:\docume~1\johnle~1\locals~1\temp\easy_install-38wlkt\trunk\egg-dist-tmp-aglabc Installed c:\python24\lib\site-packages\clientform-0.2.1a-py2.4.egg Because this distribution was installed --multi-version or --install-dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("clientform") # latest installed version pkg_resources.require("clientform==0.2.1a") # this exact version pkg_resources.require("clientform>=0.2.1a") # this version or higher error: c:\docume~1\johnle~1\locals~1\temp\easy_install-38wlkt\trunk\.svn\dir-prop-base: Permission denied C:\jjlee\code> Problems: 1. Failure to remove temporary SVN working copy on XP SP2. I get the same issue with my own manual checkouts, and use rmdir /q /s working_copy to get rid of them. 2. I don't get any warnings about conflicting unmanaged packages when I install my package, contrary to these docs: http://peak.telecommunity.com/DevCenter/EasyInstall#dealing-with-installation-conflicts Guesses: a. Problem 1. above prevented the warning being displayed b. The package is named 'ClientForm', but the egg base name is 'clientform' so I wonder if this is a case sensitivity issue? 3. It said: Installed c:\python24\lib\site-packages\clientform-0.2.1a-py2.4.egg But I asked for version "dev", and the Cheese shop entry I uploaded for testing purposes (just to get the SVN URL into the long-description) was different again: "0.2.1a-pre1"! Why don't I get an egg with version "dev"? Thanks in advance for any help with all this, John From pje at telecommunity.com Mon Oct 31 22:26:25 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 31 Oct 2005 16:26:25 -0500 Subject: [Distutils] setuptools: Missing unmanaged module warning; Failed SVN cleanup; Wrong version in .egg name In-Reply-To: Message-ID: <5.1.1.6.0.20051031155551.01fa6578@mail.telecommunity.com> At 08:37 PM 10/31/2005 +0000, John J Lee wrote: >svn: Unknown hostname 'svn.eby-sarna.com' This is a DNS problem that I just fixed (I think), please try again. >Problems: > >1. Failure to remove temporary SVN working copy on XP SP2. I get the >same issue with my own manual checkouts, and use rmdir /q /s working_copy >to get rid of them. Check and see if this happens when you're *not* getting the 'Unknown hostname' bit. If so, then I'd guess there's a problem with how SVN is setting the permissions. I'd need more info about this to debug. (I use cygwin subversion and don't have any problems with this on Win2K.) >2. I don't get any warnings about conflicting unmanaged packages when I >install my package, contrary to these docs: > >http://peak.telecommunity.com/DevCenter/EasyInstall#dealing-with-installation-conflicts > > >Guesses: > > a. Problem 1. above prevented the warning being displayed > > b. The package is named 'ClientForm', but the egg base name is > 'clientform' so I wonder if this is a case sensitivity issue? The names that are checked are the ones in the egg's 'EGG-INFO/top_level.txt'. The conflict checker doesn't do any case mangling. So I'm not sure what's going on here. Are you sure you have a conflicting package or module present? If so, where? >3. It said: > >Installed c:\python24\lib\site-packages\clientform-0.2.1a-py2.4.egg > >But I asked for version "dev", and the Cheese shop entry I uploaded for >testing purposes (just to get the SVN URL into the long-description) was >different again: "0.2.1a-pre1"! Why don't I get an egg with version >"dev"? Because when building from source, the version number is controlled by the source setup.py. From pje at telecommunity.com Mon Oct 31 22:40:02 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 31 Oct 2005 16:40:02 -0500 Subject: [Distutils] cli.exe missing from py2.3 egg? In-Reply-To: Message-ID: <5.1.1.6.0.20051031163738.036b5ad0@mail.telecommunity.com> At 11:22 AM 10/31/2005 +0000, Richard Cooper wrote: >I've just tried installing setuptools 0.6a6 so that I could install nose >(http://somethingaboutorange.com/mrl/projects/nose/) via easy_install >but I've hit a problem. > >It looks like cli.exe and gui.exe are not getting installed. To check I >manually downloaded both setuptools-0.6a6-py2.4.egg and >setuptools-0.6a6-py2.3.egg from the cheeseshop and opened them with a >Zip viewer. Sure enough the py2.4 egg contains both exes but the py2.3 >one doesn't. Should the py2.3 egg include these files? Yes it should. I've just released 0.6a7 to fix this. It was mostly a problem in my release build process, which dynamically generated different entry points for 2.3 and 2.4. Unfortunately, using dynamic entry points in setuptools' own setup.py gets into some head-exploding circularities that led to data files not being included in the 2.3 releases. I believe I've fixed this now.